There's no real way to limit access of a given database to memory or cpu on the server. If you have a badly performing database, you'll need to isolate it from the others by moving it to a different server or a different instance of SQL Server.
"The credit belongs to the man who is actually in the arena, whose face is marred by dust and sweat and blood"
- Theodore Roosevelt
Author of:
SQL Server Execution Plans
SQL Server Query Performance Tuning