you cannot rename an instance.
that means, for example, if you installed machinename\SQLEXPRESS, and you want to rename it to be machinename\SQL2008R2 that is not possible. you have to backup whatever was on the instance, remove the instance, and reinstall.
that can get pretty involved, depending on the instance...SSIS packages, linked servers, certificates, logins, mail settings, and a lot of other items might need to be migrated.
Lowell