InstantForum.NET v2.9.0SQLServerCentralhttp://www.sqlservercentral.com/Forums/notifications@sqlservercentral.comTue, 18 Jun 2013 01:32:52 GMT20http://www.sqlservercentral.com/Forums/Topic1429835-2893-1.aspx[quote][b]GilaMonster (3/20/2013)[/b][hr]That could well be it. Just keep an eye on things, run regular CheckDB and make sure that things remain OK.[/quote]Thanks for all your help.Wed, 20 Mar 2013 12:10:52 GMTauratohttp://www.sqlservercentral.com/Forums/Topic1429835-2893-1.aspxThat could well be it. Just keep an eye on things, run regular CheckDB and make sure that things remain OK.Wed, 20 Mar 2013 12:09:07 GMTGilaMonsterhttp://www.sqlservercentral.com/Forums/Topic1429835-2893-1.aspx[quote][b]GilaMonster (3/13/2013)[/b][hr]Yeah, most people aren't convinced until something fails catastrophically. Make sure your backups are good.Also, it isn't necessarily the physical drive. Could be the firmware, drives, switch, cache, any filter drivers in the IO stack, etc. In short, anything between SQL and the physical disk platters. In most systems these days, that's a lot of parts.[/quote]The hardware support team concluded that because the temperature was unusually high on the drive (84 C), that was probably the cause. I'm told that we've installed some kind of extra cooling on it and now it's back within typical operational temperatures.Wed, 20 Mar 2013 08:33:58 GMTauratohttp://www.sqlservercentral.com/Forums/Topic1429835-2893-1.aspxYeah, most people aren't convinced until something fails catastrophically. Make sure your backups are good.Also, it isn't necessarily the physical drive. Could be the firmware, drives, switch, cache, any filter drivers in the IO stack, etc. In short, anything between SQL and the physical disk platters. In most systems these days, that's a lot of parts.Wed, 13 Mar 2013 01:36:31 GMTGilaMonsterhttp://www.sqlservercentral.com/Forums/Topic1429835-2893-1.aspx[quote][b]GilaMonster (3/12/2013)[/b][hr][quote][b]aurato (3/12/2013)[/b][hr]Would you recommend migrating the databases on this disk to a different drive?[/quote]Well, ask yourself, would you leave important information on a drive that's giving repeated problems?[/quote]Migrating the database that was having issues tonight, but there are many production-critical databases on the suspect drive. We currently have a ticket open with the IO vendor. I'm not sure our team is treating this with enough urgency. No mention was made yet of migrating the other critical DBs (my reference to this thread has not fully convinced anyone that hardware is the issue). I will make the recommendation to move our most critical data to another drive.Tue, 12 Mar 2013 20:25:36 GMTauratohttp://www.sqlservercentral.com/Forums/Topic1429835-2893-1.aspx[quote][b]aurato (3/12/2013)[/b][hr]Would you recommend migrating the databases on this disk to a different drive?[/quote]Well, ask yourself, would you leave important information on a drive that's giving repeated problems?Tue, 12 Mar 2013 14:38:10 GMTGilaMonsterhttp://www.sqlservercentral.com/Forums/Topic1429835-2893-1.aspxI would move to different physical drives, but I'd also do diagnostics. Depending on your situation, you could have firmware/filter drivers (as Gail mentioned) causing issues.These are strange, hard to pin down errors, but almost all the time they are related to hardware. Most of the hardware appears to work, which is what is frustrating.Tue, 12 Mar 2013 11:57:31 GMTSteve Jones - SSC Editorhttp://www.sqlservercentral.com/Forums/Topic1429835-2893-1.aspx[quote][b]Steve Jones - SSC Editor (3/12/2013)[/b][hr]Corruption in the clustered index is likely a call to Microsoft. This is the data, and so you can't rebuild.If I had this, I'd try to BCP out the data from the table first, see if I could recover things. Likely the corruption is in certain pages, so if this doesn't work all at once, you may be able to find the ranges around the corruption and recover most data.This is serious, and if you aren't sure of what is wrong, call Microsoft.[/quote]I'm dealing with nonclustered indexes at the moment. Asked the question about clustered ones as a hypothetical. Dropping the two indexes without recreating (they don't get used very much) allowed the failing SP to run. Would you recommend migrating the databases on this disk to a different drive?Tue, 12 Mar 2013 11:21:59 GMTauratohttp://www.sqlservercentral.com/Forums/Topic1429835-2893-1.aspxCorruption in the clustered index is likely a call to Microsoft. This is the data, and so you can't rebuild.If I had this, I'd try to BCP out the data from the table first, see if I could recover things. Likely the corruption is in certain pages, so if this doesn't work all at once, you may be able to find the ranges around the corruption and recover most data.This is serious, and if you aren't sure of what is wrong, call Microsoft.Tue, 12 Mar 2013 11:18:56 GMTSteve Jones - SSC Editorhttp://www.sqlservercentral.com/Forums/Topic1429835-2893-1.aspxIf the problem is still there, you have IO issues. It's not a SQL problem, but your hardware people are not running low enough level diagnostics, they don't know what to do, or they have drivers/firmware that has a bug and needs patching.Contact your IO vendor.Tue, 12 Mar 2013 11:05:41 GMTSteve Jones - SSC Editorhttp://www.sqlservercentral.com/Forums/Topic1429835-2893-1.aspxActually it seems like the problem is still present. The SP that first indicated there was a corrupt index worked once after the drop/create and failed again after.Tue, 12 Mar 2013 10:50:28 GMTauratohttp://www.sqlservercentral.com/Forums/Topic1429835-2893-1.aspxI do have one further question. If there had been corruption in the clustered index, would I have needed to do a restore?Tue, 12 Mar 2013 10:48:22 GMTauratohttp://www.sqlservercentral.com/Forums/Topic1429835-2893-1.aspx[quote][b]GilaMonster (3/12/2013)[/b][hr]IO subsystem problems in almost all cases. Could be as simple as a write cache that's not battery backed, could be misbehaving filter drives in the IO stack, etc.[/quote]DROP and CREATE worked where REBUILD failed. Thank you.EDIT: I think I'll have to do some reading to understand your answer here though hahaTue, 12 Mar 2013 10:46:37 GMTauratohttp://www.sqlservercentral.com/Forums/Topic1429835-2893-1.aspxIO subsystem problems in almost all cases. Could be as simple as a write cache that's not battery backed, could be misbehaving filter drives in the IO stack, etc.Tue, 12 Mar 2013 10:45:07 GMTGilaMonsterhttp://www.sqlservercentral.com/Forums/Topic1429835-2893-1.aspx[quote][b]GilaMonster (3/12/2013)[/b][hr]Drop the indexes 'IX_Advertise_BrandCopy_Price_Stock' and 'IX_ITEMS_ADVFLAG_PARETN_PRICE_STOCK' (on the table 'Items') and recreate them. Don't rebuild them, drop and recreate.[/quote]Doing so now. Our sysops guys checked the server and said there doesn't appear to be any issue with the hardware. How else can corruption like this occur?Tue, 12 Mar 2013 10:43:47 GMTauratohttp://www.sqlservercentral.com/Forums/Topic1429835-2893-1.aspxDrop the indexes 'IX_Advertise_BrandCopy_Price_Stock' and 'IX_ITEMS_ADVFLAG_PARETN_PRICE_STOCK' (on the table 'Items') and recreate them. Don't rebuild them, drop and recreate.Tue, 12 Mar 2013 10:41:53 GMTGilaMonsterhttp://www.sqlservercentral.com/Forums/Topic1429835-2893-1.aspx[quote][b]GilaMonster (3/12/2013)[/b][hr]Please run the following and post the full, complete and unedited output.[code]DBCC CHECKDB (<Database Name>) WITH NO_INFOMSGS, ALL_ERRORMSGS[/code][/quote]Msg 8951, Level 16, State 1, Line 1Table error: table 'Items' (ID 1954106002). Data row does not have a matching index row in the index 'IX_Advertise_BrandCopy_Price_Stock' (ID 69). Possible missing or invalid keys for the index row matching:Msg 8955, Level 16, State 1, Line 1Data row (1:10834009:15) identified by (Id = 8885719) with index values 'AdvertiseFlag = 1 and BrandCopyParentId = 0 and NSFPPrice = 9.30 and NSFPQtyInStock = 0 and Id = 8885719'.Msg 8951, Level 16, State 1, Line 1Table error: table 'Items' (ID 1954106002). Data row does not have a matching index row in the index 'IX_Advertise_BrandCopy_Price_Stock' (ID 69). Possible missing or invalid keys for the index row matching:Msg 8955, Level 16, State 1, Line 1Data row (1:10945035:10) identified by (Id = 9750456) with index values 'AdvertiseFlag = 1 and BrandCopyParentId = 0 and NSFPPrice = 34.40 and NSFPQtyInStock = 0 and Id = 9750456'.Msg 8951, Level 16, State 1, Line 1Table error: table 'Items' (ID 1954106002). Data row does not have a matching index row in the index 'IX_Advertise_BrandCopy_Price_Stock' (ID 69). Possible missing or invalid keys for the index row matching:Msg 8955, Level 16, State 1, Line 1Data row (1:11226494:9) identified by (Id = 11078215) with index values 'AdvertiseFlag = 1 and BrandCopyParentId = 0 and NSFPPrice = 137.50 and NSFPQtyInStock = 0 and Id = 11078215'.Msg 8951, Level 16, State 1, Line 1Table error: table 'Items' (ID 1954106002). Data row does not have a matching index row in the index 'IX_ITEMS_ADVFLAG_PARETN_PRICE_STOCK' (ID 70). Possible missing or invalid keys for the index row matching:Msg 8955, Level 16, State 1, Line 1Data row (1:10834009:15) identified by (Id = 8885719) with index values 'AdvertiseFlag = 1 and BrandCopyParentId = 0 and NSFPPrice = 9.30 and NSFPQtyInStock = 0 and Id = 8885719'.Msg 8951, Level 16, State 1, Line 1Table error: table 'Items' (ID 1954106002). Data row does not have a matching index row in the index 'IX_ITEMS_ADVFLAG_PARETN_PRICE_STOCK' (ID 70). Possible missing or invalid keys for the index row matching:Msg 8955, Level 16, State 1, Line 1Data row (1:10945035:10) identified by (Id = 9750456) with index values 'AdvertiseFlag = 1 and BrandCopyParentId = 0 and NSFPPrice = 34.40 and NSFPQtyInStock = 0 and Id = 9750456'.Msg 8951, Level 16, State 1, Line 1Table error: table 'Items' (ID 1954106002). Data row does not have a matching index row in the index 'IX_ITEMS_ADVFLAG_PARETN_PRICE_STOCK' (ID 70). Possible missing or invalid keys for the index row matching:Msg 8955, Level 16, State 1, Line 1Data row (1:11226494:9) identified by (Id = 11078215) with index values 'AdvertiseFlag = 1 and BrandCopyParentId = 0 and NSFPPrice = 137.50 and NSFPQtyInStock = 0 and Id = 11078215'.CHECKDB found 0 allocation errors and 6 consistency errors in table 'Items' (object ID 1954106002).CHECKDB found 0 allocation errors and 6 consistency errors in database 'NewCatalog'.repair_rebuild is the minimum repair level for the errors found by DBCC CHECKDB (NewCatalog).Tue, 12 Mar 2013 09:50:22 GMTauratohttp://www.sqlservercentral.com/Forums/Topic1429835-2893-1.aspx[quote][b]GilaMonster (3/12/2013)[/b][hr]Please run the following and post the full, complete and unedited output.[code]DBCC CHECKDB (<Database Name>) WITH NO_INFOMSGS, ALL_ERRORMSGS[/code][/quote]Running this now.EDIT: This may take some time. It's only 5% done with its initial @BlobEater step.Tue, 12 Mar 2013 08:41:41 GMTauratohttp://www.sqlservercentral.com/Forums/Topic1429835-2893-1.aspxPlease run the following and post the full, complete and unedited output.[code]DBCC CHECKDB (<Database Name>) WITH NO_INFOMSGS, ALL_ERRORMSGS[/code]Tue, 12 Mar 2013 08:38:59 GMTGilaMonsterhttp://www.sqlservercentral.com/Forums/Topic1429835-2893-1.aspxAs always I would suggest backups before any work is carried out but it should be ok to drop the index and recreate, you will have to perform an offline rebuild. Have a read of the blog below before you do anything:http://www.sqlskills.com/blogs/paul/teched-demo-nonclustered-index-corruption/Also would have a look at your disk subsystem as well, corruption would suggest an issue.Tue, 12 Mar 2013 08:37:14 GMTkyleheath33http://www.sqlservercentral.com/Forums/Topic1429835-2893-1.aspxAlso, neither is the PK or a FK or clustered.Tue, 12 Mar 2013 08:07:57 GMTauratohttp://www.sqlservercentral.com/Forums/Topic1429835-2893-1.aspxTable error: table 'Items' (ID 1954106002). Data row does not have a matching index row in the index 'IX_Advertise_BrandCopy_Price_Stock' (ID 69). Possible missing or invalid keys for the index row matching:Msg 8955, Level 16, State 1, Line 1Data row (1:11226494:9) identified by (Id = 11078215) with index values 'AdvertiseFlag = 1 and BrandCopyParentId = 0 and NSFPPrice = 137.50 and NSFPQtyInStock = 0 and Id = 11078215'.Msg 8951, Level 16, State 1, Line 1Table error: table 'Items' (ID 1954106002). Data row does not have a matching index row in the index 'IX_ITEMS_ADVFLAG_PARETN_PRICE_STOCK' (ID 70). Possible missing or invalid keys for the index row matching:Msg 8955, Level 16, State 1, Line 1Data row (1:10834009:15) identified by (Id = 8885719) with index values 'AdvertiseFlag = 1 and BrandCopyParentId = 0 and NSFPPrice = 9.30 and NSFPQtyInStock = 0 and Id = 8885719'.These indexes were created days ago and are rarely used. Is it safe to drop and recreate corrupted indexes?Tue, 12 Mar 2013 08:06:56 GMTaurato