Do you have any partition schemes left over from the partitioning? If so, you won't be able to remove the filegroup unless you drop the schemes.