Actually, the use of PAE alone should not cause any issues. All you are doing is tell the OS to load a 36-bit version of the kernel (as opposed to a 32-bit). This allows the OS to recognize up to 64GB of memory (although only 32GB is supported by Microsoft on Datacenter).
The use of /3GB shrinks the memory available to the OS down to 1GB. One thing to watch for is the PTEs (Page Table Extensions). If not enough are available, the system can become unstable or even blue screen.
Since PAE loads a 36-bit OS, managing memory addresses requires more memory (a 32-bit address is shorter than a 36-bit address). So not only the OS is now reduced to working within a 1GB space, using PAE requires the OS to consume more memory to manage itself.
Usually, I have seen that using both PAE and 3GB at the same time is fine with systems with up to 8GB when the server is only running SQL Server and the requires services. When running more than this, or running multiple instances of SQL Server, the OS may be starving memory for the reasons I explained above.
You cannot use PAE and 3GB together with systems with 16GB or more. In this case, you can only use the PAE switch.
Finally, I confirm that using AWE on Datacenter gives the same results/behavior than on Advanced Server. I am not a big fan of AWE for a specific reason: it requires the memory to be zeroed out on startup of the SQL service, which is a single threaded operation and could take minutes (I have seen instances taking up to 15 minutes when allocating 8GB to SQL). I only use it when absolutely required.
Just a comment on the use of AWE. I have a server with 8gb or ram. I used the /3gb and /pae together and this caused me a problem. When using both parms in the boot.ini, I would have problems installing software. When I would install it would give me a message stating that I didn't have enough memory and would not install. If that is all that happend I was lucky.
Sometimes the install with the above parms would wipe out parts of the registry or corrupt OS dlls. Then I would need to reinstall the OS and resetup the entire server.
Now it seems if I only have the /3gb I will get Not Enough Memory errors but no corruption. If I just have the /pae parm loaded, I get no messages and have no problems.
I now only run with the /pae parm.
I am running on a Dell 8450 with 8 gb of ram. My OS is W2k AS and my DBMS is SQL 2000 Enterprise.
Has anyone ever heard of this or had this problem. I did not check into it to far. But I think the installs of older programs may have caused this corruption. Newer programs would only give me Not Enough Memory Errors.
MCDBA, MCSE, MCSD
SQL Server Database Proxy/Firewall and Auditing