InstantForum.NET v2.9.0SQLServerCentralhttp://www.sqlservercentral.com/Forums/notifications@sqlservercentral.comSat, 25 May 2013 16:37:56 GMT20http://www.sqlservercentral.com/Forums/Topic1431662-2799-1.aspxThanks for all the help guys. I'm going to do a bunch of reading, and maybe a bit of troubleshooting and testing this weekend and I'll let you all know how it turns out next week. Cheers!JonFri, 15 Mar 2013 15:59:09 GMTcryovenomhttp://www.sqlservercentral.com/Forums/Topic1431662-2799-1.aspxEmpty post, I made a mistake in my thinking about something here.Fri, 15 Mar 2013 15:48:48 GMTJonathan Kehayiashttp://www.sqlservercentral.com/Forums/Topic1431662-2799-1.aspxGiven the hardware budget squeeze I didn't expect "management" to understand the real value of a good consultant/mentor. Damn shame is about all you can say to that.Good luck finding much useful guidance on SQL Server and NUMA. There is a dearth of good information out there, especially deep/technical guidance, although it is certainly better than it used to be. IIRC there was a pretty good session at one of the SQL Bits conferences that you might be able to download.Fri, 15 Mar 2013 15:36:55 GMTTheSQLGuruhttp://www.sqlservercentral.com/Forums/Topic1431662-2799-1.aspx[quote][b]cryovenom (3/15/2013)[/b][hr]And less than two minutes later I realize how silly my last post seems. For anybody finding this thread through google, NUMA (or Non-Uniform Memory Access) definitely has a lot to do with both. See here for a start:http://msdn.microsoft.com/en-ca/library/ms178144%28v=sql.105%29.aspx[/quote]To make this one easier: [b][url]http://msdn.microsoft.com/en-ca/library/ms178144%28v=sql.105%29.aspx[/url][/b]Fri, 15 Mar 2013 15:36:17 GMTLynn Pettishttp://www.sqlservercentral.com/Forums/Topic1431662-2799-1.aspxPull the cover on the Dev box and look at where the RDIMMs are located and see if they are all populated on one socket or balanced across both of them. To get started on learning about NUMA:[url=http://www.sqlskills.com/blogs/jonathan/understanding-non-uniform-memory-accessarchitectures-numa/]http://www.sqlskills.com/blogs/jonathan/understanding-non-uniform-memory-accessarchitectures-numa/[/url]and then read the following and all the links at the bottom of it:[url=http://blogs.msdn.com/b/psssql/archive/2011/11/11/sql-server-clarifying-the-numa-configuration-information.aspx]http://blogs.msdn.com/b/psssql/archive/2011/11/11/sql-server-clarifying-the-numa-configuration-information.aspx[/url]Fri, 15 Mar 2013 15:34:23 GMTJonathan Kehayiashttp://www.sqlservercentral.com/Forums/Topic1431662-2799-1.aspxAnd less than two minutes later I realize how silly my last post seems. For anybody finding this thread through google, NUMA (or Non-Uniform Memory Access) definitely has a lot to do with both. See here for a start:http://msdn.microsoft.com/en-ca/library/ms178144%28v=sql.105%29.aspxFri, 15 Mar 2013 15:32:03 GMTcryovenomhttp://www.sqlservercentral.com/Forums/Topic1431662-2799-1.aspx[quote][b]TheSQLGuru (3/15/2013)[/b][hr]In my experience the best way BY FAR to become an expert (certainly at such a complex system as SQL Server) is to be MENTORED by an expert. So I stand by my recommendation 100%. Your company would come out with an optimal environment and you would learn along the way. Win-win.[/quote]I enjoy mentoring under people with more experience, and I don't disagree with your recommendation. Trying to sell it to management is always another story. It would take too long here to get into details, but suffice it to say that convincing management to do anything from sending you on a day course, to actual training, to bringing in outside experts is a difficult proposition at best, so I make do with what I've got. Booksonline, friends in the industry, and whatever research I can do on the internet or at the bookstore. [quote][b]TheSQLGuru (3/15/2013)[/b][hr]Oh, I did think of another thing, although it is reaching a bit: maybe try reseating the memory. I have experienced or heard about some very weird things happening with poorly-seated memory.[/quote][/quote]I've noticed that memory keeps getting mentioned along side the NUMA nodes and cores. I haven't read up yet, but are these things connected? Could a memory issue or configuration cause problems with setting core preference? I really have to take a quick few minutes to google what the heck a NUMA node is.Fri, 15 Mar 2013 15:28:29 GMTcryovenomhttp://www.sqlservercentral.com/Forums/Topic1431662-2799-1.aspx[quote][b]Jonathan Kehayias (3/15/2013)[/b][hr]The configuration you are trying to obtain doesn't make sense. With 4 NUMA nodes and 4 instances I would just use ALTER SERVER CONFIGURATION and set the process affinity for each instance to a specific node. The theory behind reserving CPU 0 and 15 for the OS is good in theory, but not in the reality of how OS scheduling occurs or other process scheduling is going to occur. In all my time working with SQL Server I've only once had to reserve CPUs for the OS and it was for a separate application that was running on the server and it had it's own process affinity settings to make it run on those CPUs. You are over complicating things here for no real net gain other than you promised something that I wouldn't have recommended promising.[/quote][/quote]Makes sense. 4 instances, 4 NUMA nodes, I'll give it a shot and see if that works on DEV while I'm researching a better solution long-term, be it virtualization, Windows System Resource Manager, or something else. And I'll still check BIOS settings between the two boxes at my next maintenance window just to make sure.Fri, 15 Mar 2013 15:15:30 GMTcryovenomhttp://www.sqlservercentral.com/Forums/Topic1431662-2799-1.aspxIn my experience the best way BY FAR to become an expert (certainly at such a complex system as SQL Server) is to be MENTORED by an expert. So I stand by my recommendation 100%. Your company would come out with an optimal environment and you would learn along the way. Win-win.Oh, I did think of another thing, although it is reaching a bit: maybe try reseating the memory. I have experienced or heard about some very weird things happening with poorly-seated memory.Fri, 15 Mar 2013 15:13:08 GMTTheSQLGuruhttp://www.sqlservercentral.com/Forums/Topic1431662-2799-1.aspx[b]Production Server\NCR:[/b]affinity mask 14 141 1 1 15 3 14 02 2 1 15 3 14 03 3 1 15 3 14 0[b]Development Server\NCR:[/b]affinity mask 14 141 1 1 15 3 14 02 2 1 15 3 14 03 3 1 15 3 14 0[b]Production Server\GTA:[/b]affinity mask 224 2245 5 1 240 3 224 16 6 1 240 3 224 17 7 1 240 3 224 1[b]Development Server\GTA:[/b]affinity mask 224 2245 5 1 240 3 224 16 6 1 240 3 224 17 7 1 240 3 224 1[b]Production Server\ENT:[/b]affinity mask 3584 35849 9 1 3840 3 3584 210 10 1 3840 3 3584 211 11 1 3840 3 3584 2[b]Development Server\ENT:[/b]affinity mask 3584 35849 9 1 3840 3 3584 210 10 1 3840 3 3584 211 11 1 3840 3 3584 2[b]Production Server\DOCINDEX:[/b]affinity mask 8192 819213 13 1 61440 1 8192 3[b]Development Server\DOCINDEX:[/b]affinity mask 8192 819213 13 1 61440 1 8192 3And this is with the settings where if I stop and restart GTA or DOCINDEX on the Development Server, it will not start.Fri, 15 Mar 2013 15:10:06 GMTcryovenomhttp://www.sqlservercentral.com/Forums/Topic1431662-2799-1.aspx[quote][b]cryovenom (3/15/2013)[/b][hr]Then, for fun, it drives the point home with an exaggerated comparison and copious amounts of exclamation marks.[/quote]It actually isn't an exaggerated comparison at all. My Dell Precision M6700 laptop would outperform your server head to head. My M6700 has the i7-3940XM, 32GB RAM and 1.25TB of SSD storage and scores over 16000 for a GeekBench benchmark, where a AMD 6212 system scores in the 12000 range. [quote]Yes, I'm not an expert. Yes, my company and I may not have made all the right decisions. I would much prefer to learn what's wrong, why, what do to going forward, and even maybe get some help with the actual problem as opposed to reading the kind of unhelpful and negative post you just made. [/quote]This was exactly what the recommendation was geared at, you've just taken it as a personal attack. Above you admit that you aren't an expert here and some decisions may not have been the right ones. Bringing an industry expert in can help you understand the what and why and can be the fastest way to solve your problem while also obtaining knowledge transfer. I used to bring consultants in as a DBA all the time because no one can be an expert in everything and having the right resource available prevents problems from occurring. If you are dedicated to doing it on your own, you have to expect that at some points people are going to point out mistakes you are making or have made, and that you might hit a limit to what information is practical to put into a forum thread. I used to write multi-page explanations of SQL Server internals on forum threads, I won't do that anymore because it requires to much time.The configuration you are trying to obtain doesn't make sense. With 4 NUMA nodes and 4 instances I would just use ALTER SERVER CONFIGURATION and set the process affinity for each instance to a specific node. The theory behind reserving CPU 0 and 15 for the OS is good in theory, but not in the reality of how OS scheduling occurs or other process scheduling is going to occur. In all my time working with SQL Server I've only once had to reserve CPUs for the OS and it was for a separate application that was running on the server and it had it's own process affinity settings to make it run on those CPUs. You are over complicating things here for no real net gain other than you promised something that I wouldn't have recommended promising.Fri, 15 Mar 2013 15:00:39 GMTJonathan Kehayiashttp://www.sqlservercentral.com/Forums/Topic1431662-2799-1.aspxPost the results of the following two queries from BOTH servers for each instance:[code="sql"]select name, value, value_in_use from sys.configurationswhere name = 'affinity mask';select dos.scheduler_id, dos.cpu_id, dos.is_online, don.cpu_affinity_mask, don.online_scheduler_count, don.online_scheduler_mask, don.memory_node_idfrom sys.dm_os_schedulers as dosinner join sys.dm_os_nodes as don on dos.parent_node_id = don.node_idwhere dos.status = 'visible online';[/code]Fri, 15 Mar 2013 14:42:28 GMTJonathan Kehayiashttp://www.sqlservercentral.com/Forums/Topic1431662-2799-1.aspx[quote][b]TheSQLGuru (3/15/2013)[/b][hr]Thought just occurred to me: have you checked the BIOS settings in both servers to see if there is a difference there that could be causing "identical" machines to not behave identically?? NUMA (or at least memory interleaving) may be disabled in the BIOS on one of them and not the other...[/quote]This could definitely be the case. We had some BIOS issues with the machines when they came in and had to make changes to a few of them to get them working. I'm going to read up on NUMA (seeing as I know so little about it) and at my next maintenance window I'll compare BIOS settings between the two. Thanks!Fri, 15 Mar 2013 14:37:49 GMTcryovenomhttp://www.sqlservercentral.com/Forums/Topic1431662-2799-1.aspx[quote][b]TheSQLGuru (3/15/2013)[/b][hr]cryovenom: based on many things in your various posts you (and your company) REALLY REALLY need to do themselves a favor and get a good professional on board to help you get things set up optimally. You have made some VERY poor decisions so far, and if you continue down this path even if you get stuff working you will likely be very unhappy. As an aside I am simply astounded that you ponied up the cost for Enterprise Edition and then got such a crappy piece of hardware to run your SQL Server(s) on!!! My LAPTOP will process data MUCH better than that "server" you bought!![/quote]I realize that this is the internet, and that in all forums there's a tendency to make replies like this. But all your post does is denigrate my decisions, makes no attempt to educate me on which ones were bad and why, and also makes no attempt to help with the problem at hand. Then, for fun, it drives the point home with an exaggerated comparison and copious amounts of exclamation marks.Yes, I'm not an expert. Yes, my company and I may not have made all the right decisions. I would much prefer to learn what's wrong, why, what do to going forward, and even maybe get some help with the actual problem as opposed to reading the kind of unhelpful and negative post you just made. People aren't born experts, and often they get thrown into situations where they have to make decisions based on the information they can gather and the resources they have at hand. I came here to learn, troubleshoot, fix, and become better, not to be given crap. Fri, 15 Mar 2013 14:35:43 GMTcryovenomhttp://www.sqlservercentral.com/Forums/Topic1431662-2799-1.aspxThought just occurred to me: have you checked the BIOS settings in both servers to see if there is a difference there that could be causing "identical" machines to not behave identically?? NUMA (or at least memory interleaving) may be disabled in the BIOS on one of them and not the other...Fri, 15 Mar 2013 14:27:53 GMTTheSQLGuruhttp://www.sqlservercentral.com/Forums/Topic1431662-2799-1.aspxIn SQL Server 2008, SSMS didn't separate things into NUMA nodes at all. It just presented a list of Cores, 0 to 15, one column for Processor Affinity, one for I/O Affinity. In 2012 it breaks it into 4 NUMA nodes, but I didn't know what those were, so I just carried my settings over. So on this server, the GTA instance (for example) has its processor affinity set on three cores in NUMA Node 2, and its I/O affinity set on a core in NUMA node 3. The ENT and DOCINDEX instances both use cores in NUMA Node 4. If there are restrictions or limitations that SQL Server has related to which cores it can use across which nodes, I would like to know what they are. I just assumed that it was a hardware construct like "sockets" that it was exposing to me, but didn't think that it made a difference when it came to assigning affinity.Fri, 15 Mar 2013 14:23:38 GMTcryovenomhttp://www.sqlservercentral.com/Forums/Topic1431662-2799-1.aspxcryovenom: based on many things in your various posts you (and your company) REALLY REALLY need to do themselves a favor and get a good professional on board to help you get things set up optimally. You have made some VERY poor decisions so far, and if you continue down this path even if you get stuff working you will likely be very unhappy. As an aside I am simply astounded that you ponied up the cost for Enterprise Edition and then got such a crappy piece of hardware to run your SQL Server(s) on!!! My LAPTOP will process data MUCH better than that "server" you bought!!Fri, 15 Mar 2013 14:21:58 GMTTheSQLGuruhttp://www.sqlservercentral.com/Forums/Topic1431662-2799-1.aspx[quote][b]cryovenom (3/15/2013)[/b][hr]This may sound like a rather basic question, but what do you mean by "you've mixed NUMA nodes in your settings"? Can two instances not use cores in the same NUMA node? or can one instance not use cores in separate NUMA nodes? I noticed that in 2012 it separated my cores into 4 nodes, while 2008 didn't do that, but I didn't realize there were things I couldn't split across nodes.[/quote]If you only have 2 nodes in 2008 you have different hardware. While the 2012 SQLOS memory manager was rewritten, the NUMA layout presented is from the hardware and SQLOS creates memory nodes that map to the hardware nodes directly, so you have to have different configurations for hardware.[quote]I was setting them in SSMS. The only reason that chunk of SQL using sp_configure 'affinity mask' appears there is because that's what I found when I was looking for a way to un-do the setting without an uninstall/reinstall of the instance. I'm assuming SSMS uses ALTER SERVER CONFIGURATION behind the scenes to set things.[/quote]That is a dangerous and incorrect assumption. Just because SSMS does something one way doesn't mean that is the best, or even correct, way of doing things.Fri, 15 Mar 2013 14:12:48 GMTJonathan Kehayiashttp://www.sqlservercentral.com/Forums/Topic1431662-2799-1.aspx[quote][b]Jonathan Kehayias (3/15/2013)[/b][hr]My guess without being able to query the system is that you've mixed NUMA nodes in your settings.[/quote]This may sound like a rather basic question, but what do you mean by "you've mixed NUMA nodes in your settings"? Can two instances not use cores in the same NUMA node? or can one instance not use cores in separate NUMA nodes? I noticed that in 2012 it separated my cores into 4 nodes, while 2008 didn't do that, but I didn't realize there were things I couldn't split across nodes.[quote][b]Jonathan Kehayias (3/15/2013)[/b][hr] The AMD 6212 would have been my last choice for a SQL Server processor today, the single threaded performance of those is substantially lower than the current Intel E5 series processors. [/quote]That was purely a cost restriction. We were able to get Supermicro servers with 2x Opteron 6212s, 32GB of RAM and 6x240GB SSDs, with hotswap PSUs, hotswap drive bays and Adaptec RAID cards for under $4k ea, taxes inc. We couldn't find anything close on the Intel side, or through large vendors like HP, Dell, IBM, etc... So we built these whitebox servers. They're way more performant than we even need, considering that each of these instances used to be a server with 2x single-core Opteron 2000-series processors and 4GB of DDR2 on Tyan boards in Antec cases with no hotswap anything and Seagate 7,200rpm drives. We're happy with the price/performance ratio of these new boxes and have no complaints about the Opterons[quote][b]Jonathan Kehayias (3/15/2013)[/b][hr]I don't know why you are using affinity mask settings here which are deprecated, instead of using ALTER SERVER CONFIGURATION ([url=http://msdn.microsoft.com/en-us/library/ee210585.aspx]http://msdn.microsoft.com/en-us/library/ee210585.aspx[/url]) which is how you should be setting your process affinity per instance in SQL Server 2012. [/quote]I was setting them in SSMS. The only reason that chunk of SQL using sp_configure 'affinity mask' appears there is because that's what I found when I was looking for a way to un-do the setting without an uninstall/reinstall of the instance. I'm assuming SSMS uses ALTER SERVER CONFIGURATION behind the scenes to set things.[quote][b]Jonathan Kehayias (3/15/2013)[/b][hr]As Gail already pointed out, I wouldn't recommend setting a server up in the manner that you are doing here. If you want isolated workloads, make that server a VM host and run 4 different VMs so you have the type of isolation you are trying to get here.[/quote]I appreciate the recommendation. Adding VM licensing and three more OS licenses to the mix is not something I will easily be able to sell management on, and our chosen VM host (VMWare ESXi 5) has a bit of trouble with the RAID cards on these servers so I'd likely have to hack something together by installing an OS and running virtualization software within the OS, and lose a bunch of resources to overhead along the way. The Windows System Resource Manager that Gail mentioned seems like a much more attractive option if that results in similar resource isolation guarantees. And as I said, until I get that working, I'd like to solve this mystery if possible.Fri, 15 Mar 2013 13:59:26 GMTcryovenomhttp://www.sqlservercentral.com/Forums/Topic1431662-2799-1.aspxMy guess without being able to query the system is that you've mixed NUMA nodes in your settings. The AMD 6212 would have been my last choice for a SQL Server processor today, the single threaded performance of those is substantially lower than the current Intel E5 series processors. The 6212 presents 2 NUMA nodes per socket, so you have 4 total NUMA nodes each with 4 cores and 8GB of RAM in your given configuration.I don't know why you are using affinity mask settings here which are deprecated, instead of using ALTER SERVER CONFIGURATION ([url=http://msdn.microsoft.com/en-us/library/ee210585.aspx]http://msdn.microsoft.com/en-us/library/ee210585.aspx[/url]) which is how you should be setting your process affinity per instance in SQL Server 2012. As Gail already pointed out, I wouldn't recommend setting a server up in the manner that you are doing here. If you want isolated workloads, make that server a VM host and run 4 different VMs so you have the type of isolation you are trying to get here.Fri, 15 Mar 2013 13:39:53 GMTJonathan Kehayiashttp://www.sqlservercentral.com/Forums/Topic1431662-2799-1.aspx[quote][b]cryovenom (3/15/2013)[/b][hr]I'll try setting it to default for memory allocation and see if that goes away. [/quote]Will be interesting to see, but memory allocation is the one thing you do need to do on a server with multiple instances, leaving one or all at default's not a good idea, they will compete for memory and with the OSFri, 15 Mar 2013 13:28:12 GMTGilaMonsterhttp://www.sqlservercentral.com/Forums/Topic1431662-2799-1.aspxI'll try setting it to default for memory allocation and see if that goes away. I'm also going to try allocating just processor and not I/O affinity, as that was an earlier config that we were using but changed away from because we thought it might have been the cause of some performance issues.I'll report back shortly with logs.Fri, 15 Mar 2013 13:20:11 GMTcryovenomhttp://www.sqlservercentral.com/Forums/Topic1431662-2799-1.aspx[quote]2013-03-15 15:10:15.10 Server Failed allocate pages: FAIL_PAGE_ALLOCATION 1[/quote]That's memory related, might have something to do with the NUMA node configuration, not sure though. Going to have to consult an expert on this one. Let me see if I can find someone awake at this hour of a Friday.Fri, 15 Mar 2013 13:16:31 GMTGilaMonsterhttp://www.sqlservercentral.com/Forums/Topic1431662-2799-1.aspxHere is the ERRORLOG for DOCINDEX---2013-03-15 15:07:27.05 Server Microsoft SQL Server 2012 (SP1) - 11.0.3128.0 (X64) Dec 28 2012 20:23:12 Copyright (c) Microsoft Corporation Enterprise Edition (64-bit) on Windows NT 6.1 <X64> (Build 7601: Service Pack 1)2013-03-15 15:07:27.05 Server (c) Microsoft Corporation.2013-03-15 15:07:27.05 Server All rights reserved.2013-03-15 15:07:27.05 Server Server process ID is 2084.2013-03-15 15:07:27.05 Server System Manufacturer: 'Supermicro', System Model: 'H8DGG'.2013-03-15 15:07:27.05 Server Authentication mode is MIXED.2013-03-15 15:07:27.06 Server Logging SQL Server messages in file 'E:\SQLData\DOCINDEX\MSSQL11.DOCINDEX\MSSQL\Log\ERRORLOG'.2013-03-15 15:07:27.06 Server The service account is 'STAFFNET\sqladmin'. This is an informational message; no user action is required.2013-03-15 15:07:27.06 Server Registry startup parameters: -d E:\SQLData\DOCINDEX\MSSQL11.DOCINDEX\MSSQL\DATA\master.mdf -e E:\SQLData\DOCINDEX\MSSQL11.DOCINDEX\MSSQL\Log\ERRORLOG -l E:\SQLData\DOCINDEX\MSSQL11.DOCINDEX\MSSQL\DATA\mastlog.ldf2013-03-15 15:07:27.06 Server Command Line Startup Parameters: -s "DOCINDEX"2013-03-15 15:07:27.40 Server SQL Server detected 2 sockets with 4 cores per socket and 8 logical processors per socket, 16 total logical processors; using 16 logical processors based on SQL Server licensing. This is an informational message; no user action is required.2013-03-15 15:07:27.40 Server SQL Server is starting at normal priority base (=7). This is an informational message only. No user action is required.2013-03-15 15:07:27.40 Server Detected 32766 MB of RAM. This is an informational message; no user action is required.2013-03-15 15:07:27.40 Server Using locked pages in the memory manager.2013-03-15 15:07:27.42 Server Large Page Allocated: 32MB 2013-03-15 15:07:27.43 Server Large Page Allocated: 32MB 2013-03-15 15:07:27.45 Server Large Page Allocated: 32MB 2013-03-15 15:07:27.48 Server Large Page Allocated: 32MB 2013-03-15 15:07:27.97 Server Processor affinity turned on: node 0, processor mask 0x0000000000000000. Threads will execute on CPUs per affinity settings. This is an informational message; no user action is required.2013-03-15 15:07:27.97 Server Processor affinity turned on: node 1, processor mask 0x0000000000000000. Threads will execute on CPUs per affinity settings. This is an informational message; no user action is required.2013-03-15 15:07:27.97 Server Processor affinity turned on: node 2, processor mask 0x0000000000000000. Threads will execute on CPUs per affinity settings. This is an informational message; no user action is required.2013-03-15 15:07:27.97 Server Processor affinity turned on: node 3, processor mask 0x0000000000002000. Threads will execute on CPUs per affinity settings. This is an informational message; no user action is required.2013-03-15 15:07:27.97 Server I/O affinity turned on, processor mask 0x0000000000004000. Disk I/Os will execute on CPUs per affinity I/O mask/affinity64 mask config option. This is an informational message only; no user action is required.2013-03-15 15:07:28.07 Server This instance of SQL Server last reported using a process ID of 1412 at 3/15/2013 3:07:21 PM (local) 3/15/2013 7:07:21 PM (UTC). This is an informational message only; no user action is required.2013-03-15 15:07:28.07 Server Node configuration: node 0: CPU mask: 0x000000000000000f:0 Active CPU mask: 0x0000000000000000:0. This message provides a description of the NUMA configuration for this computer. This is an informational message only. No user action is required.2013-03-15 15:07:28.07 Server Node configuration: node 1: CPU mask: 0x00000000000000f0:0 Active CPU mask: 0x0000000000000000:0. This message provides a description of the NUMA configuration for this computer. This is an informational message only. No user action is required.2013-03-15 15:07:28.07 Server Node configuration: node 2: CPU mask: 0x0000000000000f00:0 Active CPU mask: 0x0000000000000000:0. This message provides a description of the NUMA configuration for this computer. This is an informational message only. No user action is required.2013-03-15 15:07:28.07 Server Node configuration: node 3: CPU mask: 0x000000000000f000:0 Active CPU mask: 0x0000000000002000:0. This message provides a description of the NUMA configuration for this computer. This is an informational message only. No user action is required.2013-03-15 15:07:28.09 Server Using dynamic lock allocation. Initial allocation of 2500 Lock blocks and 5000 Lock Owner blocks per node. This is an informational message only. No user action is required.2013-03-15 15:07:28.09 Server Lock partitioning is enabled. This is an informational message only. No user action is required.2013-03-15 15:10:15.10 Server Failed allocate pages: FAIL_PAGE_ALLOCATION 12013-03-15 15:10:15.10 Server Process/System Counts Value---------------------------------------- ----------Available Physical Memory 21823102976Available Virtual Memory 8747790282752Available Paging File 37028200448Working Set 131485696Percent of Committed Memory in WS 100Page Faults 44643System physical memory high 1System physical memory low 0Process physical memory low 0Process virtual memory low 02013-03-15 15:10:15.10 Server Memory Manager KB---------------------------------------- ----------VM Reserved 46859968VM Committed 322916Locked Pages Allocated 3902916Large Pages Allocated 215040Emergency Memory 1024Emergency Memory In Use 16Target Committed 4194304Current Committed 4225832Pages Allocated 40056Pages Reserved 0Pages Free 3830912Pages In Use 394792Page Alloc Potential 3589800NUMA Growth Phase 1Last OOM Factor 3Last OS Error 22013-03-15 15:10:15.10 Server Memory node Id = 0 KB---------------------------------------- ----------VM Reserved 46761536VM Committed 224532Locked Pages Allocated 68876Pages Allocated 36928Pages Free 0Target Committed 293384Current Committed 293408Foreign Committed 0Away Committed 0Taken Away Committed 02013-03-15 15:10:15.10 Server Memory node Id = 1 KB---------------------------------------- ----------VM Reserved 32768VM Committed 32788Locked Pages Allocated 1040Pages Allocated 1040Pages Free 0Target Committed 33832Current Committed 33832Foreign Committed 0Away Committed 0Taken Away Committed 02013-03-15 15:10:15.10 Server Memory node Id = 2 KB---------------------------------------- ----------VM Reserved 32768VM Committed 32788Locked Pages Allocated 1040Pages Allocated 1040Pages Free 0Target Committed 33832Current Committed 33832Foreign Committed 0Away Committed 3830912Taken Away Committed 02013-03-15 15:10:15.10 Server Memory node Id = 3 KB---------------------------------------- ----------VM Reserved 32768VM Committed 32788Locked Pages Allocated 3831960Pages Allocated 1048Pages Free 3830912Target Committed 3833232Current Committed 3864752Foreign Committed 8Away Committed 0Taken Away Committed 38309122013-03-15 15:10:15.10 Server Memory node Id = 64 KB---------------------------------------- ----------VM Reserved 0VM Committed 20Locked Pages Allocated 02013-03-15 15:10:15.10 Server MEMORYCLERK_SQLGENERAL (node 0) KB---------------------------------------- ----------VM Reserved 0VM Committed 0Locked Pages Allocated 0SM Reserved 0SM Committed 0Pages Allocated 10882013-03-15 15:10:15.10 Server MEMORYCLERK_SQLBUFFERPOOL (node 0) KB---------------------------------------- ----------VM Reserved 12999744VM Committed 65536Locked Pages Allocated 132SM Reserved 0SM Committed 0Pages Allocated 82013-03-15 15:10:15.10 Server MEMORYCLERK_SQLQUERYEXEC (node 0) KB---------------------------------------- ----------VM Reserved 0VM Committed 0Locked Pages Allocated 0SM Reserved 0SM Committed 0Pages Allocated 1522013-03-15 15:10:15.10 Server MEMORYCLERK_SQLOPTIMIZER (node 0) KB---------------------------------------- ----------VM Reserved 0VM Committed 0Locked Pages Allocated 0SM Reserved 0SM Committed 0Pages Allocated 10082013-03-15 15:10:15.10 Server MEMORYCLERK_SQLUTILITIES (node 0) KB---------------------------------------- ----------VM Reserved 0VM Committed 0Locked Pages Allocated 0SM Reserved 0SM Committed 0Pages Allocated 242013-03-15 15:10:15.10 Server MEMORYCLERK_SQLSTORENG (node 0) KB---------------------------------------- ----------VM Reserved 0VM Committed 0Locked Pages Allocated 0SM Reserved 0SM Committed 0Pages Allocated 5362013-03-15 15:10:15.10 Server MEMORYCLERK_SQLCONNECTIONPOOL (node 0) KB---------------------------------------- ----------VM Reserved 0VM Committed 0Locked Pages Allocated 0SM Reserved 0SM Committed 0Pages Allocated 322013-03-15 15:10:15.10 Server MEMORYCLERK_SQLCLR (node 0) KB---------------------------------------- ----------VM Reserved 0VM Committed 0Locked Pages Allocated 0SM Reserved 0SM Committed 0Pages Allocated 82013-03-15 15:10:15.10 Server MEMORYCLERK_SQLSERVICEBROKER (node 0) KB---------------------------------------- ----------VM Reserved 0VM Committed 0Locked Pages Allocated 0SM Reserved 0SM Committed 0Pages Allocated 2882013-03-15 15:10:15.10 Server MEMORYCLERK_SNI (node 0) KB---------------------------------------- ----------VM Reserved 0VM Committed 0Locked Pages Allocated 0SM Reserved 0SM Committed 0Pages Allocated 162013-03-15 15:10:15.10 Server MEMORYCLERK_SNI (node 1) KB---------------------------------------- ----------VM Reserved 0VM Committed 0Locked Pages Allocated 0SM Reserved 0SM Committed 0Pages Allocated 162013-03-15 15:10:15.10 Server MEMORYCLERK_SNI (node 2) KB---------------------------------------- ----------VM Reserved 0VM Committed 0Locked Pages Allocated 0SM Reserved 0SM Committed 0Pages Allocated 162013-03-15 15:10:15.10 Server MEMORYCLERK_SNI (node 3) KB---------------------------------------- ----------VM Reserved 0VM Committed 0Locked Pages Allocated 0SM Reserved 0SM Committed 0Pages Allocated 162013-03-15 15:10:15.10 Server MEMORYCLERK_SNI (node 64) KB---------------------------------------- ----------VM Reserved 0VM Committed 0Locked Pages Allocated 0SM Reserved 0SM Committed 0Pages Allocated 162013-03-15 15:10:15.10 Server MEMORYCLERK_SNI (Total) KB---------------------------------------- ----------VM Reserved 0VM Committed 0Locked Pages Allocated 0SM Reserved 0SM Committed 0Pages Allocated 802013-03-15 15:10:15.10 Server MEMORYCLERK_SQLXP (node 0) KB---------------------------------------- ----------VM Reserved 0VM Committed 0Locked Pages Allocated 0SM Reserved 0SM Committed 0Pages Allocated 162013-03-15 15:10:15.10 Server MEMORYCLERK_XTP (node 0) KB---------------------------------------- ----------VM Reserved 0VM Committed 0Locked Pages Allocated 0SM Reserved 0SM Committed 0Pages Allocated 162013-03-15 15:10:15.10 Server MEMORYCLERK_HOST (node 3) KB---------------------------------------- ----------VM Reserved 0VM Committed 0Locked Pages Allocated 0SM Reserved 0SM Committed 0Pages Allocated 82013-03-15 15:10:15.10 Server MEMORYCLERK_SOSNODE (node 0) KB---------------------------------------- ----------VM Reserved 0VM Committed 0Locked Pages Allocated 0SM Reserved 0SM Committed 0Pages Allocated 179682013-03-15 15:10:15.10 Server MEMORYCLERK_SOSNODE (node 1) KB---------------------------------------- ----------VM Reserved 0VM Committed 0Locked Pages Allocated 0SM Reserved 0SM Committed 0Pages Allocated 33522013-03-15 15:10:15.10 Server MEMORYCLERK_SOSNODE (node 2) KB---------------------------------------- ----------VM Reserved 0VM Committed 0Locked Pages Allocated 0SM Reserved 0SM Committed 0Pages Allocated 33522013-03-15 15:10:15.10 Server MEMORYCLERK_SOSNODE (node 3) KB---------------------------------------- ----------VM Reserved 0VM Committed 0Locked Pages Allocated 0SM Reserved 0SM Committed 0Pages Allocated 35442013-03-15 15:10:15.10 Server MEMORYCLERK_SOSNODE (node 64) KB---------------------------------------- ----------VM Reserved 0VM Committed 0Locked Pages Allocated 0SM Reserved 0SM Committed 0Pages Allocated 24242013-03-15 15:10:15.10 Server MEMORYCLERK_SOSNODE (Total) KB---------------------------------------- ----------VM Reserved 0VM Committed 0Locked Pages Allocated 0SM Reserved 0SM Committed 0Pages Allocated 306402013-03-15 15:10:15.10 Server MEMORYCLERK_SOSOS (node 0) KB---------------------------------------- ----------VM Reserved 0VM Committed 0Locked Pages Allocated 0SM Reserved 0SM Committed 0Pages Allocated 2002013-03-15 15:10:15.10 Server MEMORYCLERK_SOSMEMMANAGER (node 0) KB---------------------------------------- ----------VM Reserved 138496VM Committed 106496Locked Pages Allocated 31816SM Reserved 0SM Committed 0Pages Allocated 02013-03-15 15:10:15.10 Server MEMORYCLERK_SQLSERVICEBROKERTRANSPORT (node 0) KB---------------------------------------- ----------VM Reserved 0VM Committed 0Locked Pages Allocated 0SM Reserved 0SM Committed 0Pages Allocated 1922013-03-15 15:10:15.10 Server MEMORYCLERK_XE (node 0) KB---------------------------------------- ----------VM Reserved 0VM Committed 0Locked Pages Allocated 0SM Reserved 0SM Committed 0Pages Allocated 3762013-03-15 15:10:15.10 Server MEMORYCLERK_SQLLOGPOOL (node 0) KB---------------------------------------- ----------VM Reserved 0VM Committed 0Locked Pages Allocated 0SM Reserved 0SM Committed 0Pages Allocated 20722013-03-15 15:10:15.10 Server CACHESTORE_OBJCP (node 0) KB---------------------------------------- ----------VM Reserved 0VM Committed 0Locked Pages Allocated 0SM Reserved 0SM Committed 0Pages Allocated 82013-03-15 15:10:15.10 Server CACHESTORE_SQLCP (node 0) KB---------------------------------------- ----------VM Reserved 0VM Committed 0Locked Pages Allocated 0SM Reserved 0SM Committed 0Pages Allocated 17042013-03-15 15:10:15.10 Server CACHESTORE_PHDR (node 0) KB---------------------------------------- ----------VM Reserved 0VM Committed 0Locked Pages Allocated 0SM Reserved 0SM Committed 0Pages Allocated 82013-03-15 15:10:15.10 Server CACHESTORE_XPROC (node 0) KB---------------------------------------- ----------VM Reserved 0VM Committed 0Locked Pages Allocated 0SM Reserved 0SM Committed 0Pages Allocated 162013-03-15 15:10:15.10 Server CACHESTORE_TEMPTABLES (node 0) KB---------------------------------------- ----------VM Reserved 0VM Committed 0Locked Pages Allocated 0SM Reserved 0SM Committed 0Pages Allocated 162013-03-15 15:10:15.10 Server CACHESTORE_NOTIF (node 0) KB---------------------------------------- ----------VM Reserved 0VM Committed 0Locked Pages Allocated 0SM Reserved 0SM Committed 0Pages Allocated 162013-03-15 15:10:15.10 Server CACHESTORE_VIEWDEFINITIONS (node 0) KB---------------------------------------- ----------VM Reserved 0VM Committed 0Locked Pages Allocated 0SM Reserved 0SM Committed 0Pages Allocated 162013-03-15 15:10:15.10 Server CACHESTORE_XMLDBTYPE (node 0) KB---------------------------------------- ----------VM Reserved 0VM Committed 0Locked Pages Allocated 0SM Reserved 0SM Committed 0Pages Allocated 82013-03-15 15:10:15.10 Server CACHESTORE_XMLDBELEMENT (node 0) KB---------------------------------------- ----------VM Reserved 0VM Committed 0Locked Pages Allocated 0SM Reserved 0SM Committed 0Pages Allocated 82013-03-15 15:10:15.10 Server CACHESTORE_XMLDBATTRIBUTE (node 0) KB---------------------------------------- ----------VM Reserved 0VM Committed 0Locked Pages Allocated 0SM Reserved 0SM Committed 0Pages Allocated 82013-03-15 15:10:15.10 Server CACHESTORE_STACKFRAMES (node 0) KB---------------------------------------- ----------VM Reserved 0VM Committed 0Locked Pages Allocated 0SM Reserved 0SM Committed 0Pages Allocated 82013-03-15 15:10:15.10 Server CACHESTORE_STACKFRAMES (node 1) KB---------------------------------------- ----------VM Reserved 0VM Committed 0Locked Pages Allocated 0SM Reserved 0SM Committed 0Pages Allocated 82013-03-15 15:10:15.10 Server CACHESTORE_STACKFRAMES (node 2) KB---------------------------------------- ----------VM Reserved 0VM Committed 0Locked Pages Allocated 0SM Reserved 0SM Committed 0Pages Allocated 82013-03-15 15:10:15.10 Server CACHESTORE_STACKFRAMES (node 3) KB---------------------------------------- ----------VM Reserved 0VM Committed 0Locked Pages Allocated 0SM Reserved 0SM Committed 0Pages Allocated 82013-03-15 15:10:15.10 Server CACHESTORE_STACKFRAMES (node 64) KB---------------------------------------- ----------VM Reserved 0VM Committed 0Locked Pages Allocated 0SM Reserved 0SM Committed 0Pages Allocated 82013-03-15 15:10:15.10 Server CACHESTORE_STACKFRAMES (Total) KB---------------------------------------- ----------VM Reserved 0VM Committed 0Locked Pages Allocated 0SM Reserved 0SM Committed 0Pages Allocated 402013-03-15 15:10:15.10 Server CACHESTORE_BROKERTBLACS (node 0) KB---------------------------------------- ----------VM Reserved 0VM Committed 0Locked Pages Allocated 0SM Reserved 0SM Committed 0Pages Allocated 162013-03-15 15:10:15.10 Server CACHESTORE_BROKERKEK (node 0) KB---------------------------------------- ----------VM Reserved 0VM Committed 0Locked Pages Allocated 0SM Reserved 0SM Committed 0Pages Allocated 82013-03-15 15:10:15.10 Server CACHESTORE_BROKERDSH (node 0) KB---------------------------------------- ----------VM Reserved 0VM Committed 0Locked Pages Allocated 0SM Reserved 0SM Committed 0Pages Allocated 82013-03-15 15:10:15.10 Server CACHESTORE_BROKERUSERCERTLOOKUP (node 0) KB---------------------------------------- ----------VM Reserved 0VM Committed 0Locked Pages Allocated 0SM Reserved 0SM Committed 0Pages Allocated 82013-03-15 15:10:15.10 Server CACHESTORE_BROKERRSB (node 0) KB---------------------------------------- ----------VM Reserved 0VM Committed 0Locked Pages Allocated 0SM Reserved 0SM Committed 0Pages Allocated 82013-03-15 15:10:15.10 Server CACHESTORE_BROKERREADONLY (node 0) KB---------------------------------------- ----------VM Reserved 0VM Committed 0Locked Pages Allocated 0SM Reserved 0SM Committed 0Pages Allocated 322013-03-15 15:10:15.10 Server CACHESTORE_BROKERTO (node 0) KB---------------------------------------- ----------VM Reserved 0VM Committed 0Locked Pages Allocated 0SM Reserved 0SM Committed 0Pages Allocated 82013-03-15 15:10:15.10 Server CACHESTORE_EVENTS (node 0) KB---------------------------------------- ----------VM Reserved 0VM Committed 0Locked Pages Allocated 0SM Reserved 0SM Committed 0Pages Allocated 162013-03-15 15:10:15.10 Server CACHESTORE_SEHOBTCOLUMNATTRIBUTE (node 0) KB---------------------------------------- ----------VM Reserved 0VM Committed 0Locked Pages Allocated 0SM Reserved 0SM Committed 0Pages Allocated 1282013-03-15 15:10:15.10 Server CACHESTORE_COLUMNSTOREOBJECTPOOL (node 0) KB---------------------------------------- ----------VM Reserved 0VM Committed 0Locked Pages Allocated 0SM Reserved 0SM Committed 0Pages Allocated 1282013-03-15 15:10:15.10 Server CACHESTORE_XML_SELECTIVE_DG (node 0) KB---------------------------------------- ----------VM Reserved 0VM Committed 0Locked Pages Allocated 0SM Reserved 0SM Committed 0Pages Allocated 82013-03-15 15:10:15.10 Server USERSTORE_SCHEMAMGR (node 0) KB---------------------------------------- ----------VM Reserved 0VM Committed 0Locked Pages Allocated 0SM Reserved 0SM Committed 0Pages Allocated 1522013-03-15 15:10:15.10 Server USERSTORE_TOKENPERM (node 0) KB---------------------------------------- ----------VM Reserved 0VM Committed 0Locked Pages Allocated 0SM Reserved 0SM Committed 0Pages Allocated 1682013-03-15 15:10:15.10 Server USERSTORE_SXC (node 0) KB---------------------------------------- ----------VM Reserved 0VM Committed 0Locked Pages Allocated 0SM Reserved 0SM Committed 0Pages Allocated 82013-03-15 15:10:15.10 Server USERSTORE_SXC (node 1) KB---------------------------------------- ----------VM Reserved 0VM Committed 0Locked Pages Allocated 0SM Reserved 0SM Committed 0Pages Allocated 82013-03-15 15:10:15.10 Server USERSTORE_SXC (node 2) KB---------------------------------------- ----------VM Reserved 0VM Committed 0Locked Pages Allocated 0SM Reserved 0SM Committed 0Pages Allocated 82013-03-15 15:10:15.10 Server USERSTORE_SXC (node 3) KB---------------------------------------- ----------VM Reserved 0VM Committed 0Locked Pages Allocated 0SM Reserved 0SM Committed 0Pages Allocated 82013-03-15 15:10:15.10 Server USERSTORE_SXC (node 64) KB---------------------------------------- ----------VM Reserved 0VM Committed 0Locked Pages Allocated 0SM Reserved 0SM Committed 0Pages Allocated 82013-03-15 15:10:15.10 Server USERSTORE_SXC (Total) KB---------------------------------------- ----------VM Reserved 0VM Committed 0Locked Pages Allocated 0SM Reserved 0SM Committed 0Pages Allocated 402013-03-15 15:10:15.10 Server OBJECTSTORE_SNI_PACKET (node 0) KB---------------------------------------- ----------VM Reserved 0VM Committed 0Locked Pages Allocated 0SM Reserved 0SM Committed 0Pages Allocated 562013-03-15 15:10:15.10 Server OBJECTSTORE_SNI_PACKET (node 1) KB---------------------------------------- ----------VM Reserved 0VM Committed 0Locked Pages Allocated 0SM Reserved 0SM Committed 0Pages Allocated 562013-03-15 15:10:15.10 Server OBJECTSTORE_SNI_PACKET (node 2) KB---------------------------------------- ----------VM Reserved 0VM Committed 0Locked Pages Allocated 0SM Reserved 0SM Committed 0Pages Allocated 562013-03-15 15:10:15.10 Server OBJECTSTORE_SNI_PACKET (node 3) KB---------------------------------------- ----------VM Reserved 0VM Committed 0Locked Pages Allocated 0SM Reserved 0SM Committed 0Pages Allocated 562013-03-15 15:10:15.10 Server OBJECTSTORE_SNI_PACKET (node 64) KB---------------------------------------- ----------VM Reserved 0VM Committed 0Locked Pages Allocated 0SM Reserved 0SM Committed 0Pages Allocated 562013-03-15 15:10:15.10 Server OBJECTSTORE_SNI_PACKET (Total) KB---------------------------------------- ----------VM Reserved 0VM Committed 0Locked Pages Allocated 0SM Reserved 0SM Committed 0Pages Allocated 2802013-03-15 15:10:15.10 Server OBJECTSTORE_SERVICE_BROKER (node 0) KB---------------------------------------- ----------VM Reserved 0VM Committed 0Locked Pages Allocated 0SM Reserved 0SM Committed 0Pages Allocated 3682013-03-15 15:10:15.10 Server OBJECTSTORE_LOCK_MANAGER (node 0) KB---------------------------------------- ----------VM Reserved 16384VM Committed 16384Locked Pages Allocated 0SM Reserved 0SM Committed 0Pages Allocated 6482013-03-15 15:10:15.10 Server OBJECTSTORE_LOCK_MANAGER (node 1) KB---------------------------------------- ----------VM Reserved 0VM Committed 0Locked Pages Allocated 0SM Reserved 0SM Committed 0Pages Allocated 6482013-03-15 15:10:15.10 Server OBJECTSTORE_LOCK_MANAGER (node 2) KB---------------------------------------- ----------VM Reserved 0VM Committed 0Locked Pages Allocated 0SM Reserved 0SM Committed 0Pages Allocated 6482013-03-15 15:10:15.10 Server OBJECTSTORE_LOCK_MANAGER (node 3) KB---------------------------------------- ----------VM Reserved 0VM Committed 0Locked Pages Allocated 0SM Reserved 0SM Committed 0Pages Allocated 6482013-03-15 15:10:15.10 Server OBJECTSTORE_LOCK_MANAGER (node 64) KB---------------------------------------- ----------VM Reserved 0VM Committed 0Locked Pages Allocated 0SM Reserved 0SM Committed 0Pages Allocated 242013-03-15 15:10:15.10 Server OBJECTSTORE_LOCK_MANAGER (Total) KB---------------------------------------- ----------VM Reserved 16384VM Committed 16384Locked Pages Allocated 0SM Reserved 0SM Committed 0Pages Allocated 26162013-03-15 15:10:15.10 Server OBJECTSTORE_XACT_CACHE (node 0) KB---------------------------------------- ----------VM Reserved 0VM Committed 0Locked Pages Allocated 0SM Reserved 0SM Committed 0Pages Allocated 162013-03-15 15:10:15.10 Server OBJECTSTORE_XACT_CACHE (node 1) KB---------------------------------------- ----------VM Reserved 0VM Committed 0Locked Pages Allocated 0SM Reserved 0SM Committed 0Pages Allocated 162013-03-15 15:10:15.10 Server OBJECTSTORE_XACT_CACHE (node 2) KB---------------------------------------- ----------VM Reserved 0VM Committed 0Locked Pages Allocated 0SM Reserved 0SM Committed 0Pages Allocated 162013-03-15 15:10:15.10 Server OBJECTSTORE_XACT_CACHE (node 3) KB---------------------------------------- ----------VM Reserved 0VM Committed 0Locked Pages Allocated 0SM Reserved 0SM Committed 0Pages Allocated 162013-03-15 15:10:15.10 Server OBJECTSTORE_XACT_CACHE (node 64) KB---------------------------------------- ----------VM Reserved 0VM Committed 0Locked Pages Allocated 0SM Reserved 0SM Committed 0Pages Allocated 162013-03-15 15:10:15.10 Server OBJECTSTORE_XACT_CACHE (Total) KB---------------------------------------- ----------VM Reserved 0VM Committed 0Locked Pages Allocated 0SM Reserved 0SM Committed 0Pages Allocated 802013-03-15 15:10:15.10 Server Buffer Pool Value---------------------------------------- ----------Database 0Simulated 0Target 65503232Dirty 0In IO 0Latched 0Page Life Expectancy 1672013-03-15 15:10:15.10 Server Memory Pool Manager Pages---------------------------------------- ----------Reserved Current 0Reserved Limit 4749202013-03-15 15:10:15.10 Server Memory Pool (internal) Pages---------------------------------------- ----------Allocations 49368Predicted 97702Private Target 0Private Limit 0Total Target 524288Total Limit 524288OOM Count 02013-03-15 15:10:15.10 Server MEMORYBROKER_FOR_CACHE (internal) Pages---------------------------------------- ----------Allocations 998Rate 4587Target Allocations 337189Future Allocations 0Overall 425984Last Notification 12013-03-15 15:10:15.10 Server MEMORYBROKER_FOR_STEAL (internal) Pages---------------------------------------- ----------Allocations 4013Rate 12480Target Allocations 348097Future Allocations 0Overall 425984Last Notification 12013-03-15 15:10:15.10 Server MEMORYBROKER_FOR_RESERVE (internal) Pages---------------------------------------- ----------Allocations 0Rate 0Target Allocations 331604Future Allocations 0Overall 425984Last Notification 12013-03-15 15:10:15.10 Server MEMORYBROKER_FOR_COMMITTED (internal) Pages---------------------------------------- ----------Allocations 44355Rate 31258Target Allocations 407217Future Allocations 0Overall 425984Last Notification 12013-03-15 15:10:15.10 Server MEMORYBROKER_FOR_XTP (internal) Pages---------------------------------------- ----------Allocations 2Rate 9Target Allocations 331615Future Allocations 0Overall 425984Last Notification 12013-03-15 15:10:15.10 Server Memory Pool (default) Pages---------------------------------------- ----------Allocations 0Predicted 94982Private Target 0Private Limit 0Total Target 524288Total Limit 524288OOM Count 02013-03-15 15:10:15.10 Server MEMORYBROKER_FOR_CACHE (default) Pages---------------------------------------- ----------Allocations 0Rate 0Target Allocations 331604Future Allocations 0Overall 425984Last Notification 12013-03-15 15:10:15.10 Server MEMORYBROKER_FOR_STEAL (default) Pages---------------------------------------- ----------Allocations 0Rate 0Target Allocations 331604Future Allocations 0Overall 425984Last Notification 12013-03-15 15:10:15.10 Server MEMORYBROKER_FOR_RESERVE (default) Pages---------------------------------------- ----------Allocations 0Rate 0Target Allocations 425984Future Allocations 94848Overall 425984Last Notification 12013-03-15 15:10:15.10 Server MEMORYBROKER_FOR_XTP (default) Pages---------------------------------------- ----------Allocations 0Rate 0Target Allocations 331604Future Allocations 0Overall 425984Last Notification 12013-03-15 15:10:15.12 Server Exception raised in Startup::CommunicationsManager.Fri, 15 Mar 2013 13:11:19 GMTcryovenomhttp://www.sqlservercentral.com/Forums/Topic1431662-2799-1.aspx[quote][b]cryovenom (3/15/2013)[/b][hr]Can you offer any assistance, despite the fact that you don't recommend this configuration?[/quote]If you post the error log that I asked for, maybe.Fri, 15 Mar 2013 13:05:19 GMTGilaMonsterhttp://www.sqlservercentral.com/Forums/Topic1431662-2799-1.aspxSure, I'll get the Errorlog up shortlyFri, 15 Mar 2013 13:04:37 GMTcryovenomhttp://www.sqlservercentral.com/Forums/Topic1431662-2799-1.aspxOkay, I will read up on the Windows System Resource Manager and start the ball rolling to figure out how that works and get it into testing so that we can eventually roll to our production environment.Until that time, I would still really like to find out why I'm able to run this configuration on one server and not the other. Can you offer any assistance, despite the fact that you don't recommend this configuration?Fri, 15 Mar 2013 13:02:50 GMTcryovenomhttp://www.sqlservercentral.com/Forums/Topic1431662-2799-1.aspx[quote][b]GilaMonster (3/15/2013)[/b][hr]If you want to balance the CPU load of the instances, the windows resource manager's a better choice[url]http://technet.microsoft.com/en-us/library/cc755056.aspx[/url][/quote]Firstly you can do more granular limits, second, because if the other instances are doing nothing at all, why shouldn't the one get most of the CPU, it's not preventing the others from working, they're idle. If all 4 are working heavily, then the limits kick in.Can you post the error log of one of the times the startup failed? The entire error log please.Also, 4GB for the OS might be a little on the low side for a 32 GB server. Monitor, make sure that the available memory (perfmon counter) doesn't drop below 600MB-1GB.Fri, 15 Mar 2013 12:54:17 GMTGilaMonsterhttp://www.sqlservercentral.com/Forums/Topic1431662-2799-1.aspxAll four instances are SQL Server 2012 EnterpriseWe've configured the affinity like that because these instances used to be completely separate (but much less powerful) servers, and when we negotiated the purchase of the larger servers with management we had to guarantee that any one instance would not be able to affect the performance of any other instance by encroaching on its resources. In other words, if someone does something crazy in NCR, it should affect only NCR. It should not be able to max out more than its 3 CPUs or eat up more than its 8gb of RAM. Is there another way to limit which cores a given instance will use?JonFri, 15 Mar 2013 12:48:51 GMTcryovenomhttp://www.sqlservercentral.com/Forums/Topic1431662-2799-1.aspxAre those two instances (the ones failing) SQL Express by any chance?btw, generally configuring the cores as you tried is not usually a good idea. Those are fairly advanced settings that usually should be left at default unless there's a really, really good reason. If you want to balance the CPU load of the instances, the windows resource manager's a better choice[url]http://technet.microsoft.com/en-us/library/cc755056.aspx[/url]Fri, 15 Mar 2013 12:43:45 GMTGilaMonsterhttp://www.sqlservercentral.com/Forums/Topic1431662-2799-1.aspxJust to add some information/confirmation:I started the DOCINDEX instance in minimal configuration mode using the following command:"C:\Program Files\Microsoft SQL Server\MSSQL11.DOCINDEX\MSSQL\Binn\sqlserver.exe" -sDOCINDEX -fThen I connected to the database and ran the following:sp_configure 'show advanced options', 1 GORECONFIGURE WITH OVERRIDEGOsp_configure 'affinity mask', 0x00000000GORECONFIGURE GOWhich resulted in the following output:Configuration option 'show advanced options' changed from 0 to 1. Run the RECONFIGURE statement to install.Configuration option 'affinity mask' changed from 8192 to 0. Run the RECONFIGURE statement to install.Then I stopped the server and restarted it from services.msc . It started and ran with no issues and the process and IO affinity were both set back to automatic. I changed them back to Process on CPU 13 and IO on CPU 14. I went into services.msc, stopped the service, and upon restart it did the same "The service started then stopped" thing. So it's DEFINITELY the affinity that's doing this, I just wish I knew why. At least now I have a way to get things working again without having to uninstall and reinstall the whole instance but it's still annoying.Fri, 15 Mar 2013 12:12:29 GMTcryovenomhttp://www.sqlservercentral.com/Forums/Topic1431662-2799-1.aspxI have a server which I'm trying to setup identically to our running production server and I'm running into issues. The hardware is identical, as follows:2x 8-Core Opteron 621232gb RAM670GB of Disk Space split 60 C: / 610 E:OS is Windows Server 2008 R2 EnterpriseSQL Server is 2012 SP1 with rollup (11.0.3128)The server has four instances installed, across which we're trying to split resources by setting Max RAM and Process/IO Affinity, like this:\NCR - 8GB, Process on Cores 1,2,3, I/O on Core 4\GTA - 8GB, Process on Cores 5,6,7, I/O on Core 8\ENT - 8GB, Process on Cores 9,10,11, I/O on Core 12\DOCINDEX - 4GB, Process on Core 13, I/O on Core 14Leaving 4GB, and Cores 0 and 15 free for the OS. The production server is setup like this and running fine. The development server is fine right up until I change the Process and IO affinity. As soon as I do that and restart the SQL Server Services or the whole box, GTA and DOCINDEX's services will start, then immediately stop. Since I can't get them started, I end up uninstalling and reinstalling the instance. Then they run fine with no process or IO affinity set. I can change the Max RAM, but if I change the Process or IO affinity again they do the same thing. I've tried stopping all the SQL services and just turning on DOCINDEX or just turning on GTA, thinking that it might be a problem with resource availability, but they still won't start. I've even gone as far as reformatting the entire box again from scratch, making sure I follow the procedure I wrote up when I built the production box and run into the same issue.The system event logs have no helpful information at all. The only event related to the services in question are a bunch of "This is informational" events describing the startup, then one saying "The service has entered the stopped state". The ERRORLOG has the same lack of information. I can post both if needed. Help!JonFri, 15 Mar 2013 11:55:31 GMTcryovenom