Did you check that the filegroup isn't set as the default? If not, set a different filegroup as the default and then you will be able to drop the filegroup.