July 8, 2011 at 8:30 am
Hello Friends...
Windows version : Windows Server 2008 R2 Enterprise
SQL server version :SQL Serve 2008 R2 (RTM) Enterprise edition
After system engineer applied some Security patches( thats what they say), SQL server database engine failed to start on both Active and passive nodes. All the SQL drive resources were online and accessible but the SQL server service in failover cluster manager did not come up in both nodes.
Attempting to bring the SQL server online recorded the below error in SQL logs.
Script level upgrade for database 'master' failed because upgrade step 'sqlagent100_msdb_upgrade.sql' encountered error 598, state 1, severity 25. This is a
serious error condition which might interfere with regular operation and the database will be taken offline. If the error happened during upgrade of the
'master' database, it will prevent the entire SQL Server instance from starting. Examine the previous errorlog entries for errors, take the appropriate
corrective actions and re-start the database so that the script upgrade steps run to completion.”
During the deployment stage of the server we had the Sytem datbases moved from one drive to another, but we did not touch any registry settings.
Reading from msdn and SQL blogs we decided that this was registry error and we went in to find that the default data and default log in registry settings were not were the current master database was present adn we changed the registry setting,.
The panicked system engineer rolled back the installed windows patches in the same time, after restarts the SQL server came up in the passive node but the active node faced the same issue, however we did not record the initial error and the active node had some odbc related errors.
We then initiated the Repair tool of setup.exe on the Active node ensuring the resources were moved to active node and the repair tool ran and completed with some errors like below, but the Active node came up, we did a lot of fail overs and SQL continued to listen well, but we are perplexed as to how to support this RCA;
...not over yet please continue to read
=================
Feature: SQL Server Replication
Status: Failed: see logs for details
MSI status: Passed
Configuration status: Failed: see details below
Configuration error code: 0xFDBE3E6D@1360@3
Configuration error description: File C:\Program Files\Microsoft SQL Server\MSSQL10_50.MSSQLSERVER\MSSQL\Binn\DllTmp64\sqlagentctr100.dll could not be copied to file path C:\Windows\system32\perf-MSSQL10_50.MSSQLSERVER-sqlagtctr.dll.
Configuration log: C:\Program Files\Microsoft SQL Server\100\Setup Bootstrap\Log\20110629_133106\Detail.txt
Feature: Full-Text Search
Status: Failed: see logs for details
MSI status: Passed
Configuration status: Failed: see details below
Configuration error code: 0xFDBE3E6D@1360@3
Configuration error description: File C:\Program Files\Microsoft SQL Server\MSSQL10_50.MSSQLSERVER\MSSQL\Binn\DllTmp64\sqlagentctr100.dll could not be copied to file path C:\Windows\system32\perf-MSSQL10_50.MSSQLSERVER-sqlagtctr.dll.
Configuration log: C:\Program Files\Microsoft SQL Server\100\Setup Bootstrap\Log\20110629_133106\Detail.txt
Rules report file: C:\Program Files\Microsoft SQL Server\100\Setup Bootstrap\Log\20110629_133106\SystemConfigurationCheck_Report.htm
The following warnings were encountered while configuring settings on your SQL Server. These resources / settings were missing or invalid so default values were used in recreating the missing resources. Please review to make sure they don’t require further customization for your applications:
Service SID support has been enabled on the service.
Service SID support has been enabled on the service.
===============================
We are not certain as how the security patches were applied by the SE whether SQL was taken to passive node while active node was pacthed or not.
As of now we have to rerun these patches which were Roll backed and we are not sure if the issue may happen again.
The same system file migration happened in test server which has the same exact setup but the SE went with patching without issues.
Can someone say what was the problem & is my system currently stable (as now SQL listens after failing over a couple of times ), is the system ready to be applied with the windows security patches which the SE insists to apply.
Active node patches :
KB2544893
KB2536276
KB2536275
KB2535512
KB2511455
KB2510531
KB2509533
KB2507618
KB2506212
KB2503665
KB2476490
KB2525694
Passive node patches:
KB2544893
KB2536276
KB2536275
KB2535512
KB2525694
KB2511455
KB2510531
KB2509553
KB2507618
KB2506212
KB2503665
KB2476490
Thanks
Eben
Viewing post 1 (of 1 total)
You must be logged in to reply to this topic. Login to reply