I didn't actually know that only sysadmins can run it, but I believe one can give other users the right to execute xp_cmdshell, or at least this was possible in earlier versions.
I think why the security community doesn't like it, is simply because the benefits don't outweigh the risks, and because a lot of bad things happened because of it.
Here is what I think is a good example. Say yo have a SQL Server which has SQL authentication enabled, and the "sa" account gets compromised. With xp_cmdshell enabled (I'm not sure if the "sa" can enable that), you can now compromise the entire server, and potentially more. Without xp_cmdshell, the incident is limited to the SQL server.
But also, i just opens up a can of worms. You have jobs that depend on it, maybe somebody granted some specific user the right to execute cmdshell, and so forth. Just like you said, as soon as you have a server with multiple admins, it gets tricky.
IMHO it's better to run jobs through the Task Scheduler if they need to run commands in the OS, and then just add/update data in SQL from there if needed.
I don't know when xp_cmdshell was introduced (might have been before SQL 2000), but it comes from a time when security was not taken as seriously as it is today.