You need access on both the source and the target.

You need access on the source to get to your starting point of the linked server. You need access on the target to access the objects on the other end of the linked server. The credentials for the target are defined in the linked server. The access on the source are the credantials to login to the instance. These credentials doesn't have to match the credentials in the linked server.

1. A linked server from Server X to Server Y is defined using credentials of User B

2. User A has access on Server X and is able to use the linked server

3. User A opens a connection to the target Server Y through the linked server

4. User A can access objects on Server Y using the credentials of User B