Target instance agent service needs to have access to your master, so the error you're getting is most likely caused by the target not being able to connect to your MSX instance (master).

Please ensure that:

1. Target instance agent service is running as AD user (can't be Network service or Local System account)

2. Add target instance agent service account into following role on master msdb: TargetServersRole

You can enlist a target from master as well as from target. I find the enlistment from target more reliable.

I'm assuming you already changed the encryption settings for the target, if not, you need to run that first:

--CHANGE ENCRYPTION

DECLARE @encryptvalue int

DECLARE @keyinput varchar(200)

SET @keyinput = 'SOFTWARE\Microsoft\MSSQLSERVER\SQLServerAgent'

EXECUTE xp_instance_regread @rootkey='HKEY_LOCAL_MACHINE',@key=@keyinput,@value_name='MsxEncryptChannelOptions',@value =@encryptvalue OUTPUT

SELECT @encryptvalue

IF @encryptvalue = 2

BEGIN

PRINT 'SQL agent encryption level set to '+CONVERT(VARCHAR(1),@encryptvalue)+'. This will be changed to 1 for multi-server administration.'

EXECUTE xp_instance_regwrite @rootkey='HKEY_LOCAL_MACHINE', @key=@keyinput, @value_name='MsxEncryptChannelOptions',@type='REG_DWORD', @value=1

END

ELSE

BEGIN

PRINT 'SQL agent encryption level already set to '+CONVERT(VARCHAR(1),@encryptvalue)+'. Ready for multi-server administration.'

END

--ENLIST TARGET SERVER

USE msdb

GO

sp_msx_enlist @msx_server_name = 'yourmasterinstance'