January 29, 2019 at 2:49 pm
Interesting, please let us know if this seems to fix things.
January 30, 2019 at 8:58 am
I'm sad to report this did not resolve the issue.. it lasted about 12 hours before the DB became corrupted again.. gonna have to wait to hear back from Microsoft.
January 30, 2019 at 10:44 am
oogibah - Wednesday, January 30, 2019 8:58 AMI'm sad to report this did not resolve the issue.. it lasted about 12 hours before the DB became corrupted again.. gonna have to wait to hear back from Microsoft.
Sorry to hear that didn't do it. I'm still curious about the issue and would still guess hardware related but who knows. Hope you can post back after hearing from MS -
Sue
March 13, 2019 at 2:53 pm
ALRIGHT! Its been a while but I'm back!
So we believe we've found the solution, its been almost a week with no issues now. This issue had to do with basically our old infrastructure engineer suppressing a bunch of alerts on our cisco hyperflex system and just completely ignoring them. We found out that the backups were all going through one non load balanced server, which was part of the issue, the physical controllers also were super out of date so we got those updated and got the software side of things updated.. after that .. no issues. Interesting that we only noticed the corruption happening after TDE was implemented but hopefully nobody else has to go through this and if they do maybe this information will lead them down the right path.
March 13, 2019 at 3:25 pm
Are you saying this was older hardware/firmware that was being used as part of infrastructure? Was this to a SAN from the db server? Is it coincidence or is there some conflict with TDE that occurs with older firmware?
If you can, it would be nice to know what was changed from which version to which to help others.
March 13, 2019 at 3:40 pm
HXDP 2.5(1c) to 3.5(2a) and UCS 3.2(1d) to 4.0(1c)
here is a TLDR of the upgrade
"The UCS Infrastructure will be upgraded, a bootstrap upgrade to the HX Data Platform Plug-In will be performed, then a combined upgrade of the HX Data Platform UCS Firmware will be performed."
the following articles provided by Cisco to upgrade the UCS Infrastructure, Hyperflex Data Platform and Plug-In
https://www.cisco.com/c/en/us/td/docs/hyperconverged_systems/HyperFlex_HX_DataPlatformSoftware/HyperFlex_upgrade_guide/3-5/b_HyperFlexSystems_Upgrade_Guide_for_VMware_ESXi_3_5/b_HyperFlexSystems_Upgrade_Guide_3_5_chapter_0101.html#task_69D7667F2BD24F3F95373BD8ADBF47AD
https://www.cisco.com/c/en/us/td/docs/hyperconverged_systems/HyperFlex_HX_DataPlatformSoftware/HyperFlex_upgrade_guide/3-5/b_HyperFlexSystems_Upgrade_Guide_for_VMware_ESXi_3_5/b_HyperFlexSystems_Upgrade_Guide_3_5_chapter_0100.html#id_59981
The backup thing was a little different.. basically there was a primary management IP lets call it .50 and there were 10 other secondary management ip addresses.. the previous sysadmin couldnt access .50 because of the drive controller issue that was updated above. and instead would access the admin console from .55 .. he also decided to send the traffic through .55 which was also where our production sql server traffic was.. so all the backups for all the servers in the company were going through this one hyperflex without being load balanced because only the main .50 is load balanced.
Viewing 6 posts - 16 through 20 (of 20 total)
You must be logged in to reply to this topic. Login to reply