Could it be that people are trying to use sys.sp_ms_marksystemobject in a later version of SQL Server than the one it worked in? That would account for its not working quite nicely.
Sometimes I think it's a great pity that SQL Server has abandoned the idea of building a relational database system based on Ted Codd's model* - if it hadn't, we would never have needed an obscure system stored procedure to mark an SP as "system" (which is a very useful concept); on the other hand, I've seen what some developers can achieve by writing directly to system tables (system rendered irrevocably fubar - reinstall from clean media
and recover data from backups) so even if we had the relational catalog as described by Codd we would have to make very sure that most people had no update access to it (not even by obscure SPs :hehe
and couldn't create system stored procedures.
* in particular, conforming to Codd's catalog rule, which he expressed in various ways, including the following (this paragraph is taken from his RM2 book)
"An important property of the relational model is that both the database and its description are perceived by users as a collection of relations. Thus, with very few exceptions, the same relational language that is used to interrogate and modify the database can be used to interrogate and modify the database description. No new training is needed."