I think it depends on how you look at it.

Sure, nobody has the final regret "I wish I'd spent more time at the office". (Well, it's not usual. Given the right [wrong] family life, maybe more time in the office would have been good for a few people.)

But plenty of people can rightfully take pride in "I worked hard and provided the best service I knew how to my co-workers/customers/clients/whatever" at the end of the workday or the end of their career.

There's definitely a sense of positive self-worth that comes from knowing that what you do professionally is top-quality.

Service to others is often our most important legacy. That includes family, but it can also include working for an employer you can be proud of and doing work you can be certain is the best you are capable of. If, at the end of the day, your only thought about work is "Thank God it's 5PM and I can finally go home! I wish it were Friday!", and it's never "Wow! I got a lot done today! It was a good day!", then seriously look for a better job. Might not be an option, but keep some hope if at all possible and keep looking. (Sure, even in the best jobs at the best companies, there are days that just can't end soon enough. It's a question of which end-of-day/end-of-week is most common. If ten days of work contain 6 "It was a good day"s and 4 "Today sucked"s, you're probably on the right track, but might need to work on improving the ratio.)

Just something to keep in mind.