If you change the port to a different one from that which it's using currently, then yes, you'll need to restart. If you plan to keep using the same port, you don't really need to restart - although you may want to anyway, just to check that it works in its new configuration.
John