Nice question. I didn't know the answer, so had to look it up in BOL. I didn't use the reference provided in the explanation, because tehobvios place to look for this question seemed to be the BOL page called The Memory Optimized Filegroup because the page title looks right for this question. Using that page, like Raghu I came across that statement
In a production environment, it is unlikely that you will need to remove the memory-optimized filegroup
and started wondering (a) what I would do if that ever happened (that's easy if I can have sufficient down-time for a restore, but OLTP systems sometimes don't allow that) and (b) under what imaginable circumstances would I want to remove the filegroup (as opposed to moving the containers in it).
Tom