Hmph. Since the question asked how to remove the filegroup from the database, I didn't consider dropping the database a viable option.

"Memory-optimized filegroups cannot be deleted."

http://msdn.microsoft.com/en-us/library/dn284308(v=sql.120).aspx