As a consultant I've seen many scenarios.

In some cases the card the card is only necessary to get in. In other cases you need the card as well to go to different floors and use the elevator.

I keep the card usually in my wallet which makes sure I almost never forgot them.

The few times I did forget them (and thus also my wallet), I just checked in at the reception and they gave me a visitors pass. Not really top secret facilities I was working in 🙂