Indeed. I use localhost for my SQL server connections while developing and that is what gets checked in.
After deploying to another server, it is straightforward (once you know what you are doing) to map the contents of a variable in an SSISDB environment to the connection string within the package.
When executing the package, you reference the environment whose (mapped) variables you wish to use.
Regarding this text: "I should be able to create variables in the connection string, set them on the respective servers and then simply deploy the result without needing to change the server." You don't need to create any variables at all. SQL Server automatically exposes project and package connection strings (assuming you are not creating them programmatically), ready for them to be overridden.
I think that the SSIS stairway demonstrates this process, but I can write out a more detailed set of instructions if that's not clear enough.