Yes, at the very beginning, I have removed all the datafiles on the filegroup.

So I said there should be no objects in the filegroup except PRIMARY.

But still I can not remove it.