While this is good advice, best practice is to always get your yubikey in pairs and keep them synchronized. One should remain in your home, in a safe place (as you described) while the other should remain on your person or outside the home (e.g. in a safe deposit box)
It’s more of a pain in the ass for sure, but handles the theft scenario more effectively
Fun fact: the infamous “cripple fight” from South Park is a shot-for-shot remake of the fight scene from They Live