you cannot rename an instance, it has to be uninstalled and a new instance with the correct name needs to be created. that means backing up non-system databases, and then restoring them in the new instance.
I haven't seen a registry hack or anything that successfully changed an instance name yet.
for example, if you had SERVER\SQL2005 as an instance, and you wanted to rename the \SQL2005 for any reason, you have to uninstall the instance, and add the new one (ie SERVER\DEVELOPER
if you rename the server, it will automatically change to NEWSERVERNAME\SQL2005
Lowell