SQL Server Agent stops unexpectedly - Buffer Overflow

  • AlphaTangoWhiskey

    SSChampion

    Points: 10786

    Our SQL Agent has been randomly crashing for the last month and a half.

    The last time it crashed we had procmon running on the server monitoring SQLAGENT.EXE and prior to it shutting down there was a bunch of Buffer Overflow events, see screen shot attached. There was a probably a 1000 of these all doing the same operation "QueryNameInformationFile" then it went back to what appeared to be normal operation then did a bunch of closing key/files/threads and shutdown, this all happens over the course of a couple seconds, but during that couple seconds there are thousands of operations.

    The buffer overflows to occur at other times just not a group of this many at once. I'm confident they are the issue or are a result of the issue.

    There are also SQL Agent dmp files:

    • memory.hdmp
    • Report.wer
    • triagedump.dmp
    • wer657.tmp.appcompat.txt
    • weraf6.tmp.werInternalMetadata.xml

    I attached the memory.dmp results that i got from windbg, i'm not really sure what is helpful i can add any of the other files.

    Not sure where to go from here. My next thought was to build a new server and just migrate everything.

    It also seems unrelated to the jobs being run, its been all over the place and times seem random. But consistently at least once per week.

    Dump file:

    Microsoft (R) Windows Debugger Version 10.0.18972.1001 AMD64

    Copyright (c) Microsoft Corporation. All rights reserved.

    Loading Dump File [..\memory.hdmp]

    User Mini Dump File: Only registers, stack and portions of memory are available

    ************* Path validation summary **************

    Response Time (ms) Location

    Deferred srv*

    Symbol search path is: srv*

    Executable search path is:

    Windows 8.1 Version 9600 MP (2 procs) Free x64

    Product: Server, suite: TerminalServer SingleUserTS

    6.3.9600.18217 (winblue_ltsb.160124-0053)

    Machine Name:

    Debug session time: Sun Sep 1 05:09:08.000 2019 (UTC - 7:00)

    System Uptime: 20 days 15:19:28.495

    Process Uptime: 4 days 17:32:38.000

    ................................................................

    ................................................................

    Loading unloaded module list

    ................................................................

    This dump file has an exception of interest stored in it.

    The stored exception information can be accessed via .ecxr.

    (a438.30bc0): C++ EH exception - code e06d7363 (first/second chance not available)

    For analysis of this file, run !analyze -v

    ntdll!NtGetContextThread+0xa:

    00007ff9`d526156a c3 ret

    0:021> !analyze -v

    *******************************************************************************

    * *

    * Exception Analysis *

    * *

    *******************************************************************************

    *** WARNING: Unable to verify timestamp for msvcr90.dll

    *** WARNING: Unable to verify timestamp for SQLAGENT.EXE

    *** WARNING: Unable to verify timestamp for xe.dll

    Unable to load image C:\Windows\assembly\NativeImages_v4.0.30319_64\mscorlib\2ef49acbb43c068f6ddf1587283b5f29\mscorlib.ni.dll, Win32 error 0n2

    *** WARNING: Unable to verify timestamp for msvcr120.dll

    *** WARNING: Unable to verify timestamp for msodbcsql13.dll

    *** WARNING: Unable to verify timestamp for SQLDTSSS.DLL

    Unable to load image C:\Windows\assembly\NativeImages_v4.0.30319_64\System\35d23b050c2c026e1270f73d41cdc3f2\System.ni.dll, Win32 error 0n2

    *** WARNING: Unable to verify checksum for System.ni.dll

    Unable to load image C:\Windows\WinSxS\amd64_microsoft.vc90.mfc_1fc8b3b9a1e18e3b_9.0.30729.6161_none_044aad0bab1eb146\mfc90u.dll, Win32 error 0n2

    *** WARNING: Unable to verify timestamp for mfc90u.dll

    Failed to request MethodData, not in JIT code range

    Failed to request MethodData, not in JIT code range

    Failed to request MethodData, not in JIT code range

    KEY_VALUES_STRING: 1

    Key : Analysis.CPU.Sec

    Value: 31

    Key : Analysis.DebugAnalysisProvider.CPP

    Value: Create: 8007007e on 1-016910

    Key : Analysis.DebugData

    Value: CreateObject

    Key : Analysis.DebugModel

    Value: CreateObject

    Key : Analysis.Elapsed.Sec

    Value: 39

    Key : Analysis.Memory.CommitPeak.Mb

    Value: 163

    Key : Analysis.System

    Value: CreateObject

    Key : CLR.Engine

    Value: CLR

    Key : CLR.Version

    Value: 4.0.30319.0

    Key : Timeline.OS.Boot.DeltaSec

    Value: 1783168

    Key : Timeline.Process.Start.DeltaSec

    Value: 408758

    NTGLOBALFLAG: 0

    PROCESS_BAM_CURRENT_THROTTLED: 0

    PROCESS_BAM_PREVIOUS_THROTTLED: 0

    APPLICATION_VERIFIER_FLAGS: 0

    CONTEXT: (.ecxr)

    rax=000000b0faa95238 rbx=000000b0faa95910 rcx=000000b0faa95390

    rdx=00007ff9d5229bd7 rsi=000000005fd82df8 rdi=000000005fe54ad8

    rip=00007ff9d27392fc rsp=000000b0faa95790 rbp=0000000000000000

    r8=000000b0faa95270 r9=000000b0faa954f0 r10=000000b0faa95500

    r11=000000b0faa954a0 r12=0000000000000011 r13=000000005fe79c58

    r14=000000005fe79c08 r15=000000005fe79c48

    iopl=0 nv up ei pl nz na pe nc

    cs=0033 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000202

    KERNELBASE!RaiseException+0x68:

    00007ff9d27392fc 488b8c24c0000000 mov rcx,qword ptr [rsp+0C0h] ss:000000b0faa95850=00003285c1ab025b

    Resetting default scope

    EXCEPTION_RECORD: (.exr -1)

    ExceptionAddress: 00007ff9d27392fc (KERNELBASE!RaiseException+0x0000000000000068)

    ExceptionCode: e06d7363 (C++ EH exception)

    ExceptionFlags: 00000001

    NumberParameters: 4

    Parameter[0]: 0000000019930520

    Parameter[1]: 000000b0faa95910

    Parameter[2]: 000000005fe54ad8

    Parameter[3]: 000000005fae0000

    PROCESS_NAME: SQLAGENT.EXE

    ERROR_CODE: (NTSTATUS) 0xe06d7363 - <Unable to get error code text>

    EXCEPTION_CODE_STR: e06d7363

    EXCEPTION_PARAMETER1: 0000000019930520

    EXCEPTION_PARAMETER2: 000000b0faa95910

    EXCEPTION_PARAMETER3: 000000005fe54ad8

    EXCEPTION_PARAMETER4: 5fae0000

    MISSING_CLR_SYMBOL: 0

    STACK_TEXT:

    000000b0faa95790 00000000614354ca : 000000b0faa95910 000000b0faa958e0 0000000000000001 0000000000000000 : KERNELBASE!RaiseException+0x68

    000000b0faa95870 000000b0faa95910 : 000000b0faa958e0 0000000000000001 0000000000000000 00000001e06d7363 : msvcr90!_CxxThrowException+0x86

    000000b0faa95878 000000b0faa958e0 : 0000000000000001 0000000000000000 00000001e06d7363 0000000000000000 : 0x000000b0`faa95910

    000000b0faa95880 0000000000000001 : 0000000000000000 00000001e06d7363 0000000000000000 0000000000000000 : 0x000000b0`faa958e0

    000000b0faa95888 0000000000000000 : 00000001e06d7363 0000000000000000 0000000000000000 0000000000000004 : 0x1

    SYMBOL_NAME: KERNELBASE!RaiseException+68

    MODULE_NAME: KERNELBASE

    IMAGE_NAME: KERNELBASE.dll

    STACK_COMMAND: ~21s ; .ecxr ; kb

    FAILURE_BUCKET_ID: CPP_EXCEPTION_e06d7363_KERNELBASE.dll!RaiseException

    OS_VERSION: 6.3.9600.18217

    BUILDLAB_STR: winblue_ltsb

    OSPLATFORM_TYPE: x64

    OSNAME: Windows 8.1

    FAILURE_ID_HASH: {1253aecc-520d-655b-58e3-5eb61e209188}

    Followup: MachineOwner

    ---------

    Attachments:
    You must be logged in to view attached files.
  • Site Owners

    SSC Guru

    Points: 80380

    Thanks for posting your issue and hopefully someone will answer soon.

    This is an automated bump to increase visibility of your question.

Viewing 2 posts - 1 through 2 (of 2 total)

You must be logged in to reply to this topic. Login to reply