I'd also note your logic is always going to be split this way. Since someone has access with no rows, you have to account for that separately in your logic and query without restrictions if the user doesn't have rows. If they have rows, as Sean noted, you'll need to find better ways to query, or index better. You never mentioned indexes, which might solve the problem.

I've done this before, but we usually ensured that we added rows to the table for all departments/offices/employees so that things were always joined.