To be clear, my response was based on the OP's claim of having a single file per filegroup. I absolutely agree that having a shedload of files per filegroup can cause performance problems when there are too many.
--Jeff Moden
Change is inevitable... Change for the better is not.