Yes it's a single point of something, but losing the witness will only remove the ability of the mirroring to automatically fail over. It'll have no other effect.
Gail Shaw
Microsoft Certified Master: SQL Server, MVP, M.Sc (Comp Sci)
SQL In The Wild: Discussions on DB performance with occasional diversions into recoverability