January 17, 2018 at 4:07 pm
After migration from 2008R2 to 2016, we are plugged with unexplained SQL errors like "cannot continue the execution because the session is in the kill state", "Internal Query Processor Error: The query processor could not produce a query plan.". In the Windows log, there are many events going in patterns like this:
6299 - AppDomain created.
10311 - AppDomain is marked for unload due to memory pressure.
6260 - AppDomain unloaded.
The interval between messages is around 5 seconds. I saw resources pointing to possible memory leaks in CLR assemblies. We do have a CLR method called frequently on large XML documents.
The question is if the memory pressure can be the reason for the errors "cannot continue the execution because the session is in the kill state", "Internal Query Processor Error: The query processor could not produce a query plan."?
Another suspicious thing is many errors in the Windows System log like this:
The application-specific permission settings do not grant Local Activation permission for the COM Server application with CLSID
{3185A766-B338-11E4-A71E-12E3F512A338}
and APPID
{7006698D-2974-4091-A424-85DD0B909E23}
to the user PCG-03\SQLSERVER SID (S-1-5-21-2657967316-168907442-3091181946-1009) from address LocalHost (Using LRPC) running in the application container Unavailable SID (Unavailable). This security permission can be modified using the Component Services administrative tool.
event id 10016
The server is an VM, if it is relevant.
January 17, 2018 at 4:48 pm
Eliyahu - Wednesday, January 17, 2018 4:07 PMAfter migration from 2008R2 to 2016, we are plugged with unexplained SQL errors like "cannot continue the execution because the session is in the kill state", "Internal Query Processor Error: The query processor could not produce a query plan.". In the Windows log, there are many events going in patterns like this:
6299 - AppDomain created.
10311 - AppDomain is marked for unload due to memory pressure.
6260 - AppDomain unloaded.
The interval between messages is around 5 seconds. I saw resources pointing to possible memory leaks in CLR assemblies. We do have a CLR method called frequently on large XML documents.
The question is if the memory pressure can be the reason for the errors "cannot continue the execution because the session is in the kill state", "Internal Query Processor Error: The query processor could not produce a query plan."?
Another suspicious thing is many errors in the Windows System log like this:The application-specific permission settings do not grant Local Activation permission for the COM Server application with CLSID
{3185A766-B338-11E4-A71E-12E3F512A338}
and APPID
{7006698D-2974-4091-A424-85DD0B909E23}
to the user PCG-03\SQLSERVER SID (S-1-5-21-2657967316-168907442-3091181946-1009) from address LocalHost (Using LRPC) running in the application container Unavailable SID (Unavailable). This security permission can be modified using the Component Services administrative tool.
event id 10016The server is an VM, if it is relevant.
1) Wow. I would not have thought 2008R2 to 2016 would be a supported upgrade.
2) Is EVERYTHING you can turn off/disable off on the server?
3) VM is very relevant. So many ways you can mess those up.
4) What is the host and VM configurations?
5) I have STRONGLY STRONGLY STRONGLY advised clients against upgrading SQL Server for as long as SQL Server has existed.
I know you don't want to hear this but if you want this resolved (quickly) it is a virtual IMPERATIVE that you hire a qualified SQL Server tuning professional. We could go back and forth for hours/days/weeks (I have seen it happen actually) and still be no closer to getting you functional (assuming that is even possible, which it might not be).
Best,
Kevin G. Boles
SQL Server Consultant
SQL MVP 2007-2012
TheSQLGuru on googles mail service
January 17, 2018 at 4:55 pm
I suggest you post SQL Server error log. I've seem situation where a VM's memory was larger than physical host's memory because one of the memory module was faulty and it brought the VM and SQL server to a halt. Make sure the total memory assigned to the VM does not exceed the physical memory.
January 18, 2018 at 5:28 am
TheSQLGuru - Wednesday, January 17, 2018 4:48 PMEliyahu - Wednesday, January 17, 2018 4:07 PMAfter migration from 2008R2 to 2016, we are plugged with unexplained SQL errors like "cannot continue the execution because the session is in the kill state", "Internal Query Processor Error: The query processor could not produce a query plan.". In the Windows log, there are many events going in patterns like this:
6299 - AppDomain created.
10311 - AppDomain is marked for unload due to memory pressure.
6260 - AppDomain unloaded.
The interval between messages is around 5 seconds. I saw resources pointing to possible memory leaks in CLR assemblies. We do have a CLR method called frequently on large XML documents.
The question is if the memory pressure can be the reason for the errors "cannot continue the execution because the session is in the kill state", "Internal Query Processor Error: The query processor could not produce a query plan."?
Another suspicious thing is many errors in the Windows System log like this:The application-specific permission settings do not grant Local Activation permission for the COM Server application with CLSID
{3185A766-B338-11E4-A71E-12E3F512A338}
and APPID
{7006698D-2974-4091-A424-85DD0B909E23}
to the user PCG-03\SQLSERVER SID (S-1-5-21-2657967316-168907442-3091181946-1009) from address LocalHost (Using LRPC) running in the application container Unavailable SID (Unavailable). This security permission can be modified using the Component Services administrative tool.
event id 10016The server is an VM, if it is relevant.
1) Wow. I would not have thought 2008R2 to 2016 would be a supported upgrade.
2) Is EVERYTHING you can turn off/disable off on the server?
3) VM is very relevant. So many ways you can mess those up.
4) What is the host and VM configurations?
5) I have STRONGLY STRONGLY STRONGLY advised clients against upgrading SQL Server for as long as SQL Server has existed.I know you don't want to hear this but if you want this resolved (quickly) it is a virtual IMPERATIVE that you hire a qualified SQL Server tuning professional. We could go back and forth for hours/days/weeks (I have seen it happen actually) and still be no closer to getting you functional (assuming that is even possible, which it might not be).
No, the server is a VM on a new box. The database migrated.
January 18, 2018 at 6:00 am
TheSQLGuru - Wednesday, January 17, 2018 4:48 PM5) I have STRONGLY STRONGLY STRONGLY advised clients against upgrading SQL Server for as long as SQL Server has existed.
Just so we clarify, you mean an in-place upgrade. You're telling people to not run a new install of an instance on an existing instance to change that from SQL Server X to SQL Server X+1. Right?
Upgrading databases through a side-by-side upgrade, new version of SQL Server on a new VM/Server is fine though, right?
I just want to clarify this, because, as written, it looks like I should still be running SQL Server 6.5. I'm 100% positive that's not what you're saying.
"The credit belongs to the man who is actually in the arena, whose face is marred by dust and sweat and blood"
- Theodore Roosevelt
Author of:
SQL Server Execution Plans
SQL Server Query Performance Tuning
January 18, 2018 at 6:02 am
What you've posted is not a normal result of a standard upgrade. Something is seriously hosed on the system. It could be the SQL Server install itself. It could be the OS install. It could be the Virtual Machine. You're going to have to tear the whole thing down and start again. It sounds seriously messed up.
I'm going to agree 100% with Kevin (a very common occurrence) and suggest in the strongest possible terms that you bring in an expert.
"The credit belongs to the man who is actually in the arena, whose face is marred by dust and sweat and blood"
- Theodore Roosevelt
Author of:
SQL Server Execution Plans
SQL Server Query Performance Tuning
January 18, 2018 at 10:06 pm
Grant Fritchey - Thursday, January 18, 2018 6:00 AMTheSQLGuru - Wednesday, January 17, 2018 4:48 PM5) I have STRONGLY STRONGLY STRONGLY advised clients against upgrading SQL Server for as long as SQL Server has existed.Just so we clarify, you mean an in-place upgrade. You're telling people to not run a new install of an instance on an existing instance to change that from SQL Server X to SQL Server X+1. Right?
Upgrading databases through a side-by-side upgrade, new version of SQL Server on a new VM/Server is fine though, right?
I just want to clarify this, because, as written, it looks like I should still be running SQL Server 6.5. I'm 100% positive that's not what you're saying.
Yeah. I left out the "in-place" part there in my efforts to get people to not do it! DOH!! :blink:
Best,
Kevin G. Boles
SQL Server Consultant
SQL MVP 2007-2012
TheSQLGuru on googles mail service
January 19, 2018 at 1:43 am
I've run into to this issue of AppDomain is marked for unload and it also involved execution of CLR assemblies. I tried to increase server memory and the 'max server memory' option several times but it didn't resolve the problem.
What I found out was that execution of CLR assemblies is using memory outside of SQL Server memory and I ended up decreasing 'max server memory'. This gave more memory to the CLR assemblies.
It also found out that spatial functions in SQL Server actually runs outside of the SQL Server memory. This is in SQL Server 2012 and 2014. I think some of them "moved into" SQL Server memory in 2016.
January 19, 2018 at 12:47 pm
Does your HOST and VM have a good bit more than 128GB of RAM available/reserved for the VM? If not that would be priority #1 in my book (unless you have much less than 128GB of actual data in SQL Server).
To point out the importance of this, I am at a new client yesterday and today for a first engagement. The top priority for the sysadmin since about an hour after I arrived (after he addressed an issue with backups) has been working to figure out exactly what RAM they need to get for their each of their SQL Servers and make the purchases before he leaves today. Saturday delivery if possible with installation this weekend.
Best,
Kevin G. Boles
SQL Server Consultant
SQL MVP 2007-2012
TheSQLGuru on googles mail service
January 22, 2018 at 4:31 am
Thank you for replies.
The first thing we did was upgrading from RTM GDR 13.0.1742.0 to the latest build SP1 CU7 13.0.4466.4. It improved things, we are watching if it is enough.
January 22, 2018 at 10:16 am
Eliyahu - Wednesday, January 17, 2018 4:07 PMAfter migration from 2008R2 to 2016, we are plugged with unexplained SQL errors like "cannot continue the execution because the session is in the kill state", "Internal Query Processor Error: The query processor could not produce a query plan.". In the Windows log, there are many events going in patterns like this:
6299 - AppDomain created.
10311 - AppDomain is marked for unload due to memory pressure.
6260 - AppDomain unloaded.
These are strong memory issue messages.
What is the memory allocated to the VM and what is the max server memory set to in sql, are there any other apps on this server?
TheSQLGuru - Wednesday, January 17, 2018 4:48 PM1) Wow. I would not have thought 2008R2 to 2016 would be a supported upgrade.
Yes it is, but must be minimum of SQL Server 2008 SP4
-----------------------------------------------------------------------------------------------------------
"Ya can't make an omelette without breaking just a few eggs" 😉
Viewing 11 posts - 1 through 10 (of 10 total)
You must be logged in to reply to this topic. Login to reply