Before I get started, I'm well aware that this type of setup is bizarre and I definitely did not recommend. I work in a corporation where the application vendor demanded 32-bit SQL 2005 and now they are paying for their decision because the instance is only able to use 2GB of address space on a physical server that has 120GB of RAM. The instance is now constantly swapping pages of memory every 200 seconds.
This was one of many sql installs that we had to do for a large project on a strict deadline, so we didn't have as much time to look into this install. I had assumed the the Windows team and Application team had the correct server configuration when we did the install. I knew that the AWE switch was available for 32-bit SQL, but overlooked that we were installing on a x64 OS.
With that said, I've done some research and found that you can allow SQL to use up to 4GB RAM by setting the \LARGEADDRESSAWARE:YES flag. This makes sense because I've done this with video editing applications years back, but what I've not been able to find is where I set this flag for sql server. I've also read somewhere else that sql server already has the flag built into the application, so this search has gotten pretty confusing.
Is this an option that needs to be setup on the service as a startup parameter or is this a boot.ini configuration? I've researched both options and neither seem to be the answer for setting the \LARGEADDRESSAWARE:YES flag. I understand it's an odd setup so many of us SQL DBA's are going to have limited to no experience with this configuration, but I'm hoping someone out there can shed some light on this issue.