I wouldn't worry much about his behavior. As it has been said this is normal behavior. Over time SQL Server will allocate otherwise unused (wasted) memory. It will give it back when other processes request for more memory. You should always expect this behavior unless your database is much smaller than available memory. (I have seen this a few years ago with SQL Server 2000 on a machine with 4 GB memory and a single 200 MB database.)

As this server is used only for mirroring, how does this memory usage compare to the memory usage on the principle server?

SQL Server will populate the buffer cache on the mirror, so it won't need to load those pages from disk when you fail over. This is one of the reasons why mirroring fails over more quickly than a traditional SQL Server cluster setup.

You should configure the memory settings on the mirror server the same way as on the principle server. In fact the mirror server should be identical in hardware and software configuration. See http://support.microsoft.com/kb/2001270 for a few more things to consider when setting up database mirroring in SQL Server.

Also if you use an SAN the database files on the principle server should be on different spindles than the mirror server. This is very important for both performance and availability.

As long this server is only used only for mirroring you shouldn't worry about the memory usage. However the sluggish response time of the mirror server is alarming. Is this same behavior also noticeable on the principle server? If not, something might be wrong at the mirror server. It could be either underpowered or faulty hardware, incorrect settings, or incorrect firmware and/or software versions.