if the two instances are on the same domain, then just set up the endpoints with Windows Authentication:

CREATE ENDPOINT BrokerEndpoint

STATE = STARTED

AS TCP(LISTENER_PORT = <port_number_goes_here>, LISTENER_IP = (<ip_address_goes_here>))

FOR SERVICE_BROKER(AUTHENTICATION = WINDOWS)

that lets you dodge all the certificate/login stuff

-Eddie