Click here to monitor SSC
SQLServerCentral is supported by Red Gate Software Ltd.
 
Log in  ::  Register  ::  Not logged in
 
 
 
        
Home       Members    Calendar    Who's On


Add to briefcase

it takes too long to insert data to different tables Expand / Collapse
Author
Message
Posted Wednesday, February 20, 2013 8:54 AM
Grasshopper

GrasshopperGrasshopperGrasshopperGrasshopperGrasshopperGrasshopperGrasshopperGrasshopper

Group: General Forum Members
Last Login: Thursday, February 21, 2013 11:39 AM
Points: 17, Visits: 46

I was trying to dump data from a Temp table to 4 different tables (Computers, ComputerInfo, Vulnerabilities, and AuditInfo) in C# with different methods I created in classes.
if (_NetBIOSName != string.Empty)
{
//Only add the computers that are not existed in Computers table
AddCompSucc = cc.AddComputer(_NetBIOSName, _IP);

//add compputer's details
CompId = cc.GetComputer(_NetBIOSName).ID;
AddCompInfoSucc = cci.AddComputerInfo(CompId, _DNSName, _MAC, _OS, _NetBIOSDomain);

//add vulneribilities
AddIavSucc = ciav.AddIAVulnerability(CompId, _IAV, _Name, _AuditDate, 0);

//add audit details
IavId = ciav.GetIAVulnerability(CompId, _IAV, _Name, _AuditDate).ID;
AddAuditInfoSucc = cai.AddAuditInfo(IavId, _AuditID, _AuditDate, _SevCode, _Risk, _PCILevel, _Exploit, _Context);
}

In each method to insert data to tables, I used “if not exists…” statement to prevent duplicated data to be inserted.
IF NOT EXISTS (
SELECT DISTINCT NetBIOSName
FROM Computers
WHERE NetBIOSName = @NetBIOSName)
BEGIN
INSERT
INTO [dbo].[Computers]([NetBIOSName], [IP])
VALUES (@NetBIOSName,@IP)
END
Issue: it takes forever to insert data to 4 tables when the Temp database is big, like having 10000 rows.
Any suggestion on how to do this differently to cut down the processing time.
Very Respectfully,
Post #1422131
Posted Wednesday, February 20, 2013 5:37 PM


Hall of Fame

Hall of FameHall of FameHall of FameHall of FameHall of FameHall of FameHall of FameHall of FameHall of Fame

Group: General Forum Members
Last Login: Tuesday, January 28, 2014 8:15 AM
Points: 3,065, Visits: 4,639
huy1002 (2/20/2013)

I was trying to dump data from a Temp table to 4 different tables (Computers, ComputerInfo, Vulnerabilities, and AuditInfo) in C# with different methods I created in classes.
if (_NetBIOSName != string.Empty)
{
//Only add the computers that are not existed in Computers table
AddCompSucc = cc.AddComputer(_NetBIOSName, _IP);

//add compputer's details
CompId = cc.GetComputer(_NetBIOSName).ID;
AddCompInfoSucc = cci.AddComputerInfo(CompId, _DNSName, _MAC, _OS, _NetBIOSDomain);

//add vulneribilities
AddIavSucc = ciav.AddIAVulnerability(CompId, _IAV, _Name, _AuditDate, 0);

//add audit details
IavId = ciav.GetIAVulnerability(CompId, _IAV, _Name, _AuditDate).ID;
AddAuditInfoSucc = cai.AddAuditInfo(IavId, _AuditID, _AuditDate, _SevCode, _Risk, _PCILevel, _Exploit, _Context);
}

In each method to insert data to tables, I used “if not exists…” statement to prevent duplicated data to be inserted.
IF NOT EXISTS (
SELECT DISTINCT NetBIOSName
FROM Computers
WHERE NetBIOSName = @NetBIOSName)
BEGIN
INSERT
INTO [dbo].[Computers]([NetBIOSName], [IP])
VALUES (@NetBIOSName,@IP)
END
Issue: it takes forever to insert data to 4 tables when the Temp database is big, like having 10000 rows.
Any suggestion on how to do this differently to cut down the processing time.
Very Respectfully,


Define a PK constraint - or at least a unique index on NetBIOSName column, get rid of the "IF NOT EXIST" block, handle the duplicate key exception.

Hope this helps.


_____________________________________
Pablo (Paul) Berzukov

Author of Understanding Database Administration available at Amazon and other bookstores.

Disclaimer: Advice is provided to the best of my knowledge but no implicit or explicit warranties are provided. Since the advisor explicitly encourages testing any and all suggestions on a test non-production environment advisor should not held liable or responsible for any actions taken based on the given advice.
Post #1422366
Posted Thursday, February 21, 2013 6:44 AM
SSCarpal Tunnel

SSCarpal TunnelSSCarpal TunnelSSCarpal TunnelSSCarpal TunnelSSCarpal TunnelSSCarpal TunnelSSCarpal TunnelSSCarpal TunnelSSCarpal Tunnel

Group: General Forum Members
Last Login: Today @ 12:05 PM
Points: 4,240, Visits: 3,674
Agreed. Make sure you have a PK on the table. If it were me, I would make an integer field as the primary key and add a unique nonclustered index to the computer name field so SQL won't worry about sort order based on a string field.

One more tip - use an appropriate data type for the computer name before you index it. If you're storing computer names, you probably don't need an nvarchar(max) or anything like that. More companies have strict policies on computer names, so get yours and go 2 bytes beyond it for when they exceed their own limit.



Tally Tables - Performance Personified
String Splitting with True Performance
Best practices on how to ask questions
Post #1422546
Posted Thursday, February 21, 2013 8:33 AM
Grasshopper

GrasshopperGrasshopperGrasshopperGrasshopperGrasshopperGrasshopperGrasshopperGrasshopper

Group: General Forum Members
Last Login: Thursday, February 21, 2013 11:39 AM
Points: 17, Visits: 46
thank you for tips. Very respectful.
Post #1422628
Posted Thursday, February 21, 2013 1:05 PM
Forum Newbie

Forum NewbieForum NewbieForum NewbieForum NewbieForum NewbieForum NewbieForum NewbieForum Newbie

Group: General Forum Members
Last Login: Tuesday, July 23, 2013 9:10 AM
Points: 6, Visits: 18
Why do 1 insert call per temp table row on a 10,000 row temp table?

just
INSERT
INTO [dbo].[Computers]([NetBIOSName], [IP])
select netbiosname,ip from temptable
where temptable.netbiosname not in ( select netbiosname from computers))

Post #1422762
« Prev Topic | Next Topic »

Add to briefcase

Permissions Expand / Collapse