Yes, we should try to get our offices, server rooms, and so on up to a high standard of physical security.

But something far more important for us to do is always to behave as if the physical security of all those things is totally broken.

The physical security measures that most companies can afford are so easily hackable and/or bypassable that if an attack by a competent and well resourced attacker takes place it will almost certainly succeed. Confidential information lying around on desk and table tops, PCs left logged in and unlocked overnight are bad; confidential information in ordinary desk drawers or cupboards isn't a lot better; nor is unshredded confidential information in waste bins. Confidential information held on portable storage without really effective encryption is worse. Relying on physical security to prevent any such confidential information from getting to those whom you don't want to have it is extremely foolish, unless you know that the risk of attack is extremely low because the "enemy" doesn't care enough about the information to spend two cents on obtaining it. Many people have been badly bitten by thinking they knew that.