I think I had the same issue. I was able to remove all the file groups using the method i explained but one.

Ty this query. Replace the FileGroup0 with your filegroup name.

select distinct so.name

from sys.objects so

inner join sys.indexes si

on so.object_ID = si.object_ID

inner join sys.data_spaces ds

on si.data_space_id = ds.data_space_id

inner join sys.filegroups fg

on ds.data_space_id = fg.data_space_id

where fg.name in ('FileGroup0')

In my case i came up with following results.

queue_messages_1395848385

queue_messages_1427848499

queue_messages_1459848613

Let me know if you face the same situation. I am still investigating a way to remove these objects which belongs to this file group even when there are no files present. I will let you know if I managet to remove them and the filegroup.