Arctic Wolf flooding SQL error log

Question

Arctic Wolf flooding SQL error log

Mr. Brian Gale

SSC-Insane

Points: 24692
More actions
December 14, 2023 at 2:30 pm

#4345924
Hello experts,
I am not sure if anyone uses Arctic Wolf for a security tool, but we recently implemented it and one thing it does is penetration testing and vulnerability scanning on our SQL Servers. Our IT team has this turned this feature on as they want to make sure things are secure, but it is making my SQL Server error logs, and thus RedGate SQL Monitor, unhappy as I am getting a TON of errors like:
Error: 17836, Severity: 20, State: 14.
Length specified in network packet payload did not match number of bytes read; the connection has been closed. Please contact the vendor of the client library. [CLIENT: xxx.xxx.xxx.xxx]
I am just curious if anyone else has set up Arctic Wolf and had a similar experience and what they ended up doing? At the moment, I am tempted to turn off the error log monitoring from SQL Monitor as I am not monitoring it at the moment - too much "junk" data coming in for me to verify if there is any "non-junk" data.
- This topic was modified 4 months, 2 weeks ago by Mr. Brian Gale.
The above is all just my opinion on what you should do.
As with all advice you find on a random internet forum - you shouldn't blindly follow it. Always test on a test server to see if there is negative side effects before making changes to live!
I recommend you NEVER run "random code" you found online on any system you care about UNLESS you understand and can verify the code OR you don't care if the code trashes your system.

Viewing 9 posts - 1 through 8 (of 8 total)

You must be logged in to reply to this topic. Login to reply

Steve Collins SSCrazy Points: 2069 More actions · Answer 1

Error severity 20-25 are system errors so I wouldn't want to turn off logging and consider that the solution :). There's a packet size system setting for which there's a default, 4096.

EXEC sp_configure 'network packet size';

Maybe start by confirming the packet sizes the app sends agree with the setting. I don't know how to do that tho. Maybe you have a network admin who might know? Or ask the vendor

Aus dem Paradies, das Cantor uns geschaffen, soll uns niemand vertreiben können

Mr. Brian Gale SSC-Insane Points: 24692 More actions · Answer 2

Thanks for the tip! I'll send a question out to the IT team as I would LOVE to have that fixed... although I am a bit concerned that changing it to fix these errors may result in errors in other places. I am pretty sure that we don't have any apps or reports or SSIS packages set up to use a specific packet size, but I know that if changing that impacted end users in any way rather than just annoying the DBA's, I'll be on a phone call I'd rather avoid.

Going to do a little research on the setting and see the potential harm in changing it as well as reach out to the IT team about what Arctic Wolf is doing and what packet size they are using... I have an odd feeling that the tool is just throwing random or very specific but non-standard packet sizes at the server in an attempt to see if a specific CVE is vulnerable on our system and in the process is annoying the DBA's who are no longer getting those alerts. We are still tracking them in SQL Monitor (they are not disabled), but the email alerts about it are disabled. My thought is if the tool is sending out a bunch of different packet sizes to check for specific CVE vulnerabilities, changing the packet size will fix 1 of them but the others will still trigger. On one of my servers, I had 6 of these alerts in under a minute.

Also, I'm going to edit the original post as it has the IP in there. The IP is an internal only IP, but better safe than sorry.

The above is all just my opinion on what you should do.
As with all advice you find on a random internet forum - you shouldn't blindly follow it. Always test on a test server to see if there is negative side effects before making changes to live!
I recommend you NEVER run "random code" you found online on any system you care about UNLESS you understand and can verify the code OR you don't care if the code trashes your system.

frederico_fonseca SSCoach Points: 16019 More actions · Answer 3

https://blog.sqlauthority.com/2018/08/08/sql-server-fix-error-17836-length-specified-in-network-packet-payload-did-not-match-number-of-bytes-read-the-connection-has-been-closed/

also have a look at this in case Docker is being used - or if it is related to encryption protocols.

https://github.com/microsoft/mssql-docker/issues/478

and you really should go to the vendor with this issue as they should know what is causing it and how to solve it.

Mr. Brian Gale SSC-Insane Points: 24692 More actions · Answer 4

Thanks for the links Frederico_fonseca - unfortunately it is not a docker thing. In my case it looks like it is a vulnerability scan and it is just driving me nuts LOL.

And your advice about reaching out to the vendor is actually advice I often give people and for whatever reason I didn't think to try that yet. I had reached out to my IT team and they told me "that IP is for Arctic Wolf, so it is just the tool doing scans" and I left it at that. But the way it is flooding SQL Monitor (roughly 100 errors per day) is driving me nuts.

The above is all just my opinion on what you should do.
As with all advice you find on a random internet forum - you shouldn't blindly follow it. Always test on a test server to see if there is negative side effects before making changes to live!
I recommend you NEVER run "random code" you found online on any system you care about UNLESS you understand and can verify the code OR you don't care if the code trashes your system.

frederico_fonseca SSCoach Points: 16019 More actions · Answer 5

tell your IT that you need to have a technical answer from the vendor - or that they need to block their IP (or do it yourself if you can) on your DB servers so the scanner does not affect your database server - raise it as an issue up the food chain if needed.

I do not believe that the error is due to they attempting to do scanning, but rather to their client settings - TLS, encryption protocols or even client version - all these can be addressed at the client level and will then allow the scanner to work as expected and that particular error will go away (but you should have OTHER errors like invalid password/login for some of the well know logins like SA, repl_subscriber, probe, repl_publisher and others, as any good scanning tool will try to login with "well known users/passwords" to see if they were set at default values.

and thinking of this it may well be what is happening - it is trying to login with such a list, but due to the client connection error it cant - so you get this error instead of the invalid login error you would/should be expecting

Eddie Wuerch SSChampion Points: 12532 More actions · Answer 6

Also check what the product is doing - I've been places where threat scanning involved a product throwing a laundry list of known exploits at each of the SQL Servers to test if they were patched. This threw a rash of high-severity errors into the logs every week when the tests would fire, triggering a flood of high-severity alerts to the DBAs. The security people who set up the tests just picked everything SQL Server in their tool and let it rip. This initially included stuff that had been patched out 20 years earlier, like Code Red.

If they are pen testing, then you could work with them to limit the tests to at least the versions you're running or may run in the future. I would also encourage going through the list of tests and with them and compare each test with any logged response, so your team learns to recognize that an actual attack is occurring compared to alerts raised due to other non-threat problems with your enterprise.

If the errors are due to intentional activity by the tool, then:

1. This is actually part of a sound approach to defending your systems: find all the holes attackers would use, then close them. Server configuration can change over time, so pen testing must be a recurring effort.

2. Frequency and communication are important - left alone, those tests are just teaching the DBAs to ignore evidence of an actual attack after responding too many times to alerts that were just the tool doing normal work.

Eddie Wuerch
MCM: SQL

Jeff Moden SSC Guru Points: 1003870 More actions · Answer 7

Perhaps call the folks at Arctic Wolf and ask what they suggest? Maybe their website has some amplifying information?

--Jeff Moden

RBAR is pronounced "ree-bar" and is a "Modenism" for Row-By-Agonizing-Row.
First step towards the paradigm shift of writing Set Based code:
________Stop thinking about what you want to do to a ROW... think, instead, of what you want to do to a COLUMN.

Change is inevitable... Change for the better is not.

Helpful Links:
How to post code problems
How to Post Performance Problems
Create a Tally Function (fnTally)

EdVassie SSC Guru Points: 60445 More actions · Answer 8

At the moment it looks like AW is not doing its job. Instead of alerting you about problems it is just giving errors. You definitely should escalate this to get the problems fixed.

When it does work you need a situation where you only get alerted about real problems. If you get loads of false positives then, to quote, 'those tests are just teaching the DBAs to ignore evidence of an actual attack'

Original author: https://github.com/SQL-FineBuild/Common/wiki/ 1-click install and best practice configuration of SQL Server 2019, 2017 2016, 2014, 2012, 2008 R2, 2008 and 2005.

When I give food to the poor they call me a saint. When I ask why they are poor they call me a communist - Archbishop Hélder Câmara