September 11, 2019 at 8:09 pm
Our SQL Agent has been randomly crashing for the last month and a half.
The last time it crashed we had procmon running on the server monitoring SQLAGENT.EXE and prior to it shutting down there was a bunch of Buffer Overflow events, see screen shot attached. There was a probably a 1000 of these all doing the same operation "QueryNameInformationFile" then it went back to what appeared to be normal operation then did a bunch of closing key/files/threads and shutdown, this all happens over the course of a couple seconds, but during that couple seconds there are thousands of operations.
The buffer overflows to occur at other times just not a group of this many at once. I'm confident they are the issue or are a result of the issue.
There are also SQL Agent dmp files:
I attached the memory.dmp results that i got from windbg, i'm not really sure what is helpful i can add any of the other files.
Not sure where to go from here. My next thought was to build a new server and just migrate everything.
It also seems unrelated to the jobs being run, its been all over the place and times seem random. But consistently at least once per week.
Dump file:
Microsoft (R) Windows Debugger Version 10.0.18972.1001 AMD64
Copyright (c) Microsoft Corporation. All rights reserved.
Loading Dump File [..\memory.hdmp]
User Mini Dump File: Only registers, stack and portions of memory are available
************* Path validation summary **************
Response Time (ms) Location
Deferred srv*
Symbol search path is: srv*
Executable search path is:
Windows 8.1 Version 9600 MP (2 procs) Free x64
Product: Server, suite: TerminalServer SingleUserTS
6.3.9600.18217 (winblue_ltsb.160124-0053)
Machine Name:
Debug session time: Sun Sep 1 05:09:08.000 2019 (UTC - 7:00)
System Uptime: 20 days 15:19:28.495
Process Uptime: 4 days 17:32:38.000
................................................................
................................................................
Loading unloaded module list
................................................................
This dump file has an exception of interest stored in it.
The stored exception information can be accessed via .ecxr.
(a438.30bc0): C++ EH exception - code e06d7363 (first/second chance not available)
For analysis of this file, run !analyze -v
ntdll!NtGetContextThread+0xa:
00007ff9`d526156a c3 ret
0:021> !analyze -v
*******************************************************************************
* *
* Exception Analysis *
* *
*******************************************************************************
*** WARNING: Unable to verify timestamp for msvcr90.dll
*** WARNING: Unable to verify timestamp for SQLAGENT.EXE
*** WARNING: Unable to verify timestamp for xe.dll
Unable to load image C:\Windows\assembly\NativeImages_v4.0.30319_64\mscorlib\2ef49acbb43c068f6ddf1587283b5f29\mscorlib.ni.dll, Win32 error 0n2
*** WARNING: Unable to verify timestamp for msvcr120.dll
*** WARNING: Unable to verify timestamp for msodbcsql13.dll
*** WARNING: Unable to verify timestamp for SQLDTSSS.DLL
Unable to load image C:\Windows\assembly\NativeImages_v4.0.30319_64\System\35d23b050c2c026e1270f73d41cdc3f2\System.ni.dll, Win32 error 0n2
*** WARNING: Unable to verify checksum for System.ni.dll
Unable to load image C:\Windows\WinSxS\amd64_microsoft.vc90.mfc_1fc8b3b9a1e18e3b_9.0.30729.6161_none_044aad0bab1eb146\mfc90u.dll, Win32 error 0n2
*** WARNING: Unable to verify timestamp for mfc90u.dll
Failed to request MethodData, not in JIT code range
Failed to request MethodData, not in JIT code range
Failed to request MethodData, not in JIT code range
KEY_VALUES_STRING: 1
Key : Analysis.CPU.Sec
Value: 31
Key : Analysis.DebugAnalysisProvider.CPP
Value: Create: 8007007e on 1-016910
Key : Analysis.DebugData
Value: CreateObject
Key : Analysis.DebugModel
Value: CreateObject
Key : Analysis.Elapsed.Sec
Value: 39
Key : Analysis.Memory.CommitPeak.Mb
Value: 163
Key : Analysis.System
Value: CreateObject
Key : CLR.Engine
Value: CLR
Key : CLR.Version
Value: 4.0.30319.0
Key : Timeline.OS.Boot.DeltaSec
Value: 1783168
Key : Timeline.Process.Start.DeltaSec
Value: 408758
NTGLOBALFLAG: 0
PROCESS_BAM_CURRENT_THROTTLED: 0
PROCESS_BAM_PREVIOUS_THROTTLED: 0
APPLICATION_VERIFIER_FLAGS: 0
CONTEXT: (.ecxr)
rax=000000b0faa95238 rbx=000000b0faa95910 rcx=000000b0faa95390
rdx=00007ff9d5229bd7 rsi=000000005fd82df8 rdi=000000005fe54ad8
rip=00007ff9d27392fc rsp=000000b0faa95790 rbp=0000000000000000
r8=000000b0faa95270 r9=000000b0faa954f0 r10=000000b0faa95500
r11=000000b0faa954a0 r12=0000000000000011 r13=000000005fe79c58
r14=000000005fe79c08 r15=000000005fe79c48
iopl=0 nv up ei pl nz na pe nc
cs=0033 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000202
KERNELBASE!RaiseException+0x68:
00007ff9d27392fc 488b8c24c0000000 mov rcx,qword ptr [rsp+0C0h] ss:000000b0faa95850=00003285c1ab025b
Resetting default scope
EXCEPTION_RECORD: (.exr -1)
ExceptionAddress: 00007ff9d27392fc (KERNELBASE!RaiseException+0x0000000000000068)
ExceptionCode: e06d7363 (C++ EH exception)
ExceptionFlags: 00000001
NumberParameters: 4
Parameter[0]: 0000000019930520
Parameter[1]: 000000b0faa95910
Parameter[2]: 000000005fe54ad8
Parameter[3]: 000000005fae0000
PROCESS_NAME: SQLAGENT.EXE
ERROR_CODE: (NTSTATUS) 0xe06d7363 - <Unable to get error code text>
EXCEPTION_CODE_STR: e06d7363
EXCEPTION_PARAMETER1: 0000000019930520
EXCEPTION_PARAMETER2: 000000b0faa95910
EXCEPTION_PARAMETER3: 000000005fe54ad8
EXCEPTION_PARAMETER4: 5fae0000
MISSING_CLR_SYMBOL: 0
STACK_TEXT:
000000b0faa95790 00000000614354ca : 000000b0faa95910 000000b0faa958e0 0000000000000001 0000000000000000 : KERNELBASE!RaiseException+0x68
000000b0faa95870 000000b0faa95910 : 000000b0faa958e0 0000000000000001 0000000000000000 00000001e06d7363 : msvcr90!_CxxThrowException+0x86
000000b0faa95878 000000b0faa958e0 : 0000000000000001 0000000000000000 00000001e06d7363 0000000000000000 : 0x000000b0`faa95910
000000b0faa95880 0000000000000001 : 0000000000000000 00000001e06d7363 0000000000000000 0000000000000000 : 0x000000b0`faa958e0
000000b0faa95888 0000000000000000 : 00000001e06d7363 0000000000000000 0000000000000000 0000000000000004 : 0x1
SYMBOL_NAME: KERNELBASE!RaiseException+68
MODULE_NAME: KERNELBASE
IMAGE_NAME: KERNELBASE.dll
STACK_COMMAND: ~21s ; .ecxr ; kb
FAILURE_BUCKET_ID: CPP_EXCEPTION_e06d7363_KERNELBASE.dll!RaiseException
OS_VERSION: 6.3.9600.18217
BUILDLAB_STR: winblue_ltsb
OSPLATFORM_TYPE: x64
OSNAME: Windows 8.1
FAILURE_ID_HASH: {1253aecc-520d-655b-58e3-5eb61e209188}
Followup: MachineOwner
---------
September 12, 2019 at 8:10 pm
Thanks for posting your issue and hopefully someone will answer soon.
This is an automated bump to increase visibility of your question.
Viewing 2 posts - 1 through 1 (of 1 total)
You must be logged in to reply to this topic. Login to reply