Click here to monitor SSC
SQLServerCentral is supported by Redgate
 
Log in  ::  Register  ::  Not logged in
 
 
 


SQL 2012 sp1 Crash ntdll.dll


SQL 2012 sp1 Crash ntdll.dll

Author
Message
Tom Van Harpen
Tom Van Harpen
Right there with Babe
Right there with Babe (726 reputation)Right there with Babe (726 reputation)Right there with Babe (726 reputation)Right there with Babe (726 reputation)Right there with Babe (726 reputation)Right there with Babe (726 reputation)Right there with Babe (726 reputation)Right there with Babe (726 reputation)

Group: General Forum Members
Points: 726 Visits: 1099
For the 3rd time in as many weeks, we've had a SQL 2012 sp1 instance go down.

Couple changes that were made:
Add a linked server type:SQLSERVER and running Sp_blitz and using the linked server to send data back to our management SQL Server. (this job was not running so i guess i cant blame Brent)
Added CommVault backup agent and started running CommVault backups, however the first crash occurrence was before we implemented this.

There is no information in the SQL Logs. The last event was a tranlog backup then the next log entry was the service starting back up.

Event viewer had a single entry for the crash (aside from the one saying it terminated unexpectedly)

Faulting application name: sqlservr.exe, version: 2011.110.3000.0, time stamp: 0x5082086a
Faulting module name: ntdll.dll, version: 6.1.7601.17725, time stamp: 0x4ec4aa8e
Exception code: 0xc0000374
Fault offset: 0x00000000000c40f2
Faulting process id: 0x1178
Faulting application start time: 0x01d29c04eedd742a
Faulting application path: C:\Program Files\Microsoft SQL Server\MSSQL11.MSSQLSERVER\MSSQL\Binn\sqlservr.exe
Faulting module path: C:\Windows\SYSTEM32\ntdll.dll
Report Id: eb0621d9-0cce-11e7-a90d-005056a90e7f


Looking for any thoughts, suggestions, or direction on how to further troubleshoot this or capture more data if it happens again.

Thanks
Sue_H
Sue_H
SSCrazy
SSCrazy (2.5K reputation)SSCrazy (2.5K reputation)SSCrazy (2.5K reputation)SSCrazy (2.5K reputation)SSCrazy (2.5K reputation)SSCrazy (2.5K reputation)SSCrazy (2.5K reputation)SSCrazy (2.5K reputation)

Group: General Forum Members
Points: 2547 Visits: 2928
Tom Van Harpen - Monday, March 20, 2017 11:48 AM
For the 3rd time in as many weeks, we've had a SQL 2012 sp1 instance go down.

Couple changes that were made:
Add a linked server type:SQLSERVER and running Sp_blitz and using the linked server to send data back to our management SQL Server. (this job was not running so i guess i cant blame Brent)
Added CommVault backup agent and started running CommVault backups, however the first crash occurrence was before we implemented this.

There is no information in the SQL Logs. The last event was a tranlog backup then the next log entry was the service starting back up.

Event viewer had a single entry for the crash (aside from the one saying it terminated unexpectedly)

Faulting application name: sqlservr.exe, version: 2011.110.3000.0, time stamp: 0x5082086a
Faulting module name: ntdll.dll, version: 6.1.7601.17725, time stamp: 0x4ec4aa8e
Exception code: 0xc0000374
Fault offset: 0x00000000000c40f2
Faulting process id: 0x1178
Faulting application start time: 0x01d29c04eedd742a
Faulting application path: C:\Program Files\Microsoft SQL Server\MSSQL11.MSSQLSERVER\MSSQL\Binn\sqlservr.exe
Faulting module path: C:\Windows\SYSTEM32\ntdll.dll
Report Id: eb0621d9-0cce-11e7-a90d-005056a90e7f


Looking for any thoughts, suggestions, or direction on how to further troubleshoot this or capture more data if it happens again.

Thanks

That's unusual to not have anything at all in the SQL Server error logs. Do you have any dump files being generated when this happens? Mini-dumps or full dumps?

Sue




Tom Van Harpen
Tom Van Harpen
Right there with Babe
Right there with Babe (726 reputation)Right there with Babe (726 reputation)Right there with Babe (726 reputation)Right there with Babe (726 reputation)Right there with Babe (726 reputation)Right there with Babe (726 reputation)Right there with Babe (726 reputation)Right there with Babe (726 reputation)

Group: General Forum Members
Points: 726 Visits: 1099
I dont see any dump files (.mdmp) files.
for the time in question there are multiple .LOG.1 files(from full text indexing operation), a SQLAGENT.1(shown below) and a system_health_.... extended events file.
There is also a trace file (.trc) which ended about 6 minutes before the crash and shows a normal log backup as the last entry.

SQLAGENT file:
2017-03-19 13:07:37 - ! [012] The MSSQLSERVER service terminated unexpectedly
2017-03-19 13:07:37 - + [139] AutoRestart: Attempting to restart the MSSQLSERVER service (attempt #1)...
2017-03-19 13:07:42 - ! [368] AutoRestart: Unable to restart the MSSQLSERVER service (reason: Access is denied)
2017-03-19 13:07:47 - + [139] AutoRestart: Attempting to restart the MSSQLSERVER service (attempt #2)...
2017-03-19 13:07:47 - ! [368] AutoRestart: Unable to restart the MSSQLSERVER service (reason: Access is denied)
2017-03-19 13:07:52 - + [139] AutoRestart: Attempting to restart the MSSQLSERVER service (attempt #3)...
2017-03-19 13:07:52 - ! [368] AutoRestart: Unable to restart the MSSQLSERVER service (reason: Access is denied)
2017-03-19 13:07:52 - ! [140] AutoRestart: The MSSQLSERVER service could not be restarted after 3 attempts
2017-03-19 13:07:52 - + [360] SQLServerAgent initiating shutdown following MSSQLSERVER shutdown
2017-03-19 13:07:54 - ! [359] The local host server is not running
2017-03-19 13:07:54 - ! [240] 1 engine thread(s) failed to stop after 2 seconds of waiting
2017-03-19 13:07:54 - ! [311] Thread 'JobInvocationEngine' (ID 2552) is still running
2017-03-19 13:07:54 - ! [359] The local host server is not running
2017-03-19 13:07:54 - + [098] SQLServerAgent terminated (forcefully)

Sue_H
Sue_H
SSCrazy
SSCrazy (2.5K reputation)SSCrazy (2.5K reputation)SSCrazy (2.5K reputation)SSCrazy (2.5K reputation)SSCrazy (2.5K reputation)SSCrazy (2.5K reputation)SSCrazy (2.5K reputation)SSCrazy (2.5K reputation)

Group: General Forum Members
Points: 2547 Visits: 2928

I'd probably still search for any log, dmp or mdmp file extensions during that time. It could be a lot of things and with only getting one entry in the event log when the crashes happen, it won't be too easy to figure out. I'd also keep track of what you see logged before the crash.
It's looks like it's heap corruption of some sort which can sometimes happen with linked servers - usually old or faulty drivers though.
And if the backup agent was installed that could possibly be the issue even if there weren't any backups running.
There a way to monitor heap corruption but if you aren't getting any dump files, that would be pointless.

Seems the only thing you could do for now and with the crashes being fairly consistent with just the two changes to the server, I'd try uninstalling the backup agent and get rid of the linked servers for now.

Sue




Tom Van Harpen
Tom Van Harpen
Right there with Babe
Right there with Babe (726 reputation)Right there with Babe (726 reputation)Right there with Babe (726 reputation)Right there with Babe (726 reputation)Right there with Babe (726 reputation)Right there with Babe (726 reputation)Right there with Babe (726 reputation)Right there with Babe (726 reputation)

Group: General Forum Members
Points: 726 Visits: 1099
Thanks Sue,
I've removed the linked server since I've found some posts regarding issues that reference linked servers.
It appears so far that its been the last 3 Sundays, but since the event viewer is cutoff last Monday i cant confirm that. There are no jobs running during those times.
I'll dig around some more and see what i can find, will update if i find anything useful.

Thanks,
Tom
Octavian C.
Octavian C.
Grasshopper
Grasshopper (19 reputation)Grasshopper (19 reputation)Grasshopper (19 reputation)Grasshopper (19 reputation)Grasshopper (19 reputation)Grasshopper (19 reputation)Grasshopper (19 reputation)Grasshopper (19 reputation)

Group: General Forum Members
Points: 19 Visits: 32
Tom Van Harpen - Monday, March 20, 2017 4:42 PM
Thanks Sue,
I've removed the linked server since I've found some posts regarding issues that reference linked servers.
It appears so far that its been the last 3 Sundays, but since the event viewer is cutoff last Monday i cant confirm that. There are no jobs running during those times.
I'll dig around some more and see what i can find, will update if i find anything useful.

Thanks,
Tom


Hi Tom.
The exception has not been cached by the SQL dumper. I assume that the event log entry was from Windows error reporting (WER). You should find the mini dump under the WER directory.
Your susspiscion that this is caused by the linked server provider might be correct. Analyzing the dump would confirm that.
To avoid the instance crash, you should set the provider to run outside of the SQL server process (instance restart required- in some rare cases OS reboot is needed for the change to become affected). This way, next time a failure in the linked server provider will result just in an error message and will not affect the SQL Server process.
Please.S. some providers work only inprosess of the SQL Server.
Hth

Jeff Moden
Jeff Moden
SSC Guru
SSC Guru (70K reputation)SSC Guru (70K reputation)SSC Guru (70K reputation)SSC Guru (70K reputation)SSC Guru (70K reputation)SSC Guru (70K reputation)SSC Guru (70K reputation)SSC Guru (70K reputation)

Group: General Forum Members
Points: 70758 Visits: 40738
Tom Van Harpen - Monday, March 20, 2017 11:48 AM
For the 3rd time in as many weeks, we've had a SQL 2012 sp1 instance go down.


To be blunt, 2012 sucked until sp3 came out. It even had a bug where if you rebuilt a cluster index online under certain conditions, it would corrupt the table.

I strongly recommend that you upgrade your 2012 to the latest SP/CU NOW! Don't wait, just do it.

--Jeff Moden

RBAR is pronounced ree-bar and is a Modenism for Row-By-Agonizing-Row.
First step towards the paradigm shift of writing Set Based code:
Stop thinking about what you want to do to a row... think, instead, of what you want to do to a column.
If you think its expensive to hire a professional to do the job, wait until you hire an amateur. -- Red Adair

Helpful Links:
How to post code problems
How to post performance problems
Forum FAQs
Tom Van Harpen
Tom Van Harpen
Right there with Babe
Right there with Babe (726 reputation)Right there with Babe (726 reputation)Right there with Babe (726 reputation)Right there with Babe (726 reputation)Right there with Babe (726 reputation)Right there with Babe (726 reputation)Right there with Babe (726 reputation)Right there with Babe (726 reputation)

Group: General Forum Members
Points: 726 Visits: 1099
Hey Jeff,
I feel that is good advice. I'd like to know why it happened but i guess if i can get to the point where it doesnt happen again that would be acceptable. Also i appreciate the bluntness Smile
Tom
Go


Permissions

You can't post new topics.
You can't post topic replies.
You can't post new polls.
You can't post replies to polls.
You can't edit your own topics.
You can't delete your own topics.
You can't edit other topics.
You can't delete other topics.
You can't edit your own posts.
You can't edit other posts.
You can't delete your own posts.
You can't delete other posts.
You can't post events.
You can't edit your own events.
You can't edit other events.
You can't delete your own events.
You can't delete other events.
You can't send private messages.
You can't send emails.
You can read topics.
You can't vote in polls.
You can't upload attachments.
You can download attachments.
You can't post HTML code.
You can't edit HTML code.
You can't post IFCode.
You can't post JavaScript.
You can post emoticons.
You can't post or upload images.

Select a forum

































































































































































SQLServerCentral


Search