Click here to monitor SSC
SQLServerCentral is supported by Redgate
 
Log in  ::  Register  ::  Not logged in
 
 
 


it takes too long to insert data to different tables


it takes too long to insert data to different tables

Author
Message
huy1002
huy1002
Grasshopper
Grasshopper (18 reputation)Grasshopper (18 reputation)Grasshopper (18 reputation)Grasshopper (18 reputation)Grasshopper (18 reputation)Grasshopper (18 reputation)Grasshopper (18 reputation)Grasshopper (18 reputation)

Group: General Forum Members
Points: 18 Visits: 50
I was trying to dump data from a Temp table to 4 different tables (Computers, ComputerInfo, Vulnerabilities, and AuditInfo) in C# with different methods I created in classes.
if (_NetBIOSName != string.Empty)
{
//Only add the computers that are not existed in Computers table
AddCompSucc = cc.AddComputer(_NetBIOSName, _IP);

//add compputer's details
CompId = cc.GetComputer(_NetBIOSName).ID;
AddCompInfoSucc = cci.AddComputerInfo(CompId, _DNSName, _MAC, _OS, _NetBIOSDomain);

//add vulneribilities
AddIavSucc = ciav.AddIAVulnerability(CompId, _IAV, _Name, _AuditDate, 0);

//add audit details
IavId = ciav.GetIAVulnerability(CompId, _IAV, _Name, _AuditDate).ID;
AddAuditInfoSucc = cai.AddAuditInfo(IavId, _AuditID, _AuditDate, _SevCode, _Risk, _PCILevel, _Exploit, _Context);
}

In each method to insert data to tables, I used “if not exists…” statement to prevent duplicated data to be inserted.
IF NOT EXISTS (
SELECT DISTINCT NetBIOSName
FROM Computers
WHERE NetBIOSName = @NetBIOSName)
BEGIN
INSERT
INTO [dbo].[Computers]([NetBIOSName], [IP])
VALUES (@NetBIOSName,@IP)
END
Issue: it takes forever to insert data to 4 tables when the Temp database is big, like having 10000 rows.
Any suggestion on how to do this differently to cut down the processing time.
Very Respectfully,
PaulB-TheOneAndOnly
PaulB-TheOneAndOnly
Hall of Fame
Hall of Fame (3.1K reputation)Hall of Fame (3.1K reputation)Hall of Fame (3.1K reputation)Hall of Fame (3.1K reputation)Hall of Fame (3.1K reputation)Hall of Fame (3.1K reputation)Hall of Fame (3.1K reputation)Hall of Fame (3.1K reputation)

Group: General Forum Members
Points: 3093 Visits: 4639
huy1002 (2/20/2013)

I was trying to dump data from a Temp table to 4 different tables (Computers, ComputerInfo, Vulnerabilities, and AuditInfo) in C# with different methods I created in classes.
if (_NetBIOSName != string.Empty)
{
//Only add the computers that are not existed in Computers table
AddCompSucc = cc.AddComputer(_NetBIOSName, _IP);

//add compputer's details
CompId = cc.GetComputer(_NetBIOSName).ID;
AddCompInfoSucc = cci.AddComputerInfo(CompId, _DNSName, _MAC, _OS, _NetBIOSDomain);

//add vulneribilities
AddIavSucc = ciav.AddIAVulnerability(CompId, _IAV, _Name, _AuditDate, 0);

//add audit details
IavId = ciav.GetIAVulnerability(CompId, _IAV, _Name, _AuditDate).ID;
AddAuditInfoSucc = cai.AddAuditInfo(IavId, _AuditID, _AuditDate, _SevCode, _Risk, _PCILevel, _Exploit, _Context);
}

In each method to insert data to tables, I used “if not exists…” statement to prevent duplicated data to be inserted.
IF NOT EXISTS (
SELECT DISTINCT NetBIOSName
FROM Computers
WHERE NetBIOSName = @NetBIOSName)
BEGIN
INSERT
INTO [dbo].[Computers]([NetBIOSName], [IP])
VALUES (@NetBIOSName,@IP)
END
Issue: it takes forever to insert data to 4 tables when the Temp database is big, like having 10000 rows.
Any suggestion on how to do this differently to cut down the processing time.
Very Respectfully,


Define a PK constraint - or at least a unique index on NetBIOSName column, get rid of the "IF NOT EXIST" block, handle the duplicate key exception.

Hope this helps.

_____________________________________
Pablo (Paul) Berzukov

Author of Understanding Database Administration available at Amazon and other bookstores.

Disclaimer: Advice is provided to the best of my knowledge but no implicit or explicit warranties are provided. Since the advisor explicitly encourages testing any and all suggestions on a test non-production environment advisor should not held liable or responsible for any actions taken based on the given advice.
Ed Wagner
Ed Wagner
SSChampion
SSChampion (10K reputation)SSChampion (10K reputation)SSChampion (10K reputation)SSChampion (10K reputation)SSChampion (10K reputation)SSChampion (10K reputation)SSChampion (10K reputation)SSChampion (10K reputation)

Group: General Forum Members
Points: 10215 Visits: 9549
Agreed. Make sure you have a PK on the table. If it were me, I would make an integer field as the primary key and add a unique nonclustered index to the computer name field so SQL won't worry about sort order based on a string field.

One more tip - use an appropriate data type for the computer name before you index it. If you're storing computer names, you probably don't need an nvarchar(max) or anything like that. More companies have strict policies on computer names, so get yours and go 2 bytes beyond it for when they exceed their own limit.


Tally Tables - Performance Personified
String Splitting with True Performance
Best practices on how to ask questions
huy1002
huy1002
Grasshopper
Grasshopper (18 reputation)Grasshopper (18 reputation)Grasshopper (18 reputation)Grasshopper (18 reputation)Grasshopper (18 reputation)Grasshopper (18 reputation)Grasshopper (18 reputation)Grasshopper (18 reputation)

Group: General Forum Members
Points: 18 Visits: 50
thank you for tips. Very respectful.
bdloving 4446
bdloving 4446
Forum Newbie
Forum Newbie (6 reputation)Forum Newbie (6 reputation)Forum Newbie (6 reputation)Forum Newbie (6 reputation)Forum Newbie (6 reputation)Forum Newbie (6 reputation)Forum Newbie (6 reputation)Forum Newbie (6 reputation)

Group: General Forum Members
Points: 6 Visits: 18
Why do 1 insert call per temp table row on a 10,000 row temp table?

just
INSERT
INTO [dbo].[Computers]([NetBIOSName], [IP])
select netbiosname,ip from temptable
where temptable.netbiosname not in ( select netbiosname from computers))
Go


Permissions

You can't post new topics.
You can't post topic replies.
You can't post new polls.
You can't post replies to polls.
You can't edit your own topics.
You can't delete your own topics.
You can't edit other topics.
You can't delete other topics.
You can't edit your own posts.
You can't edit other posts.
You can't delete your own posts.
You can't delete other posts.
You can't post events.
You can't edit your own events.
You can't edit other events.
You can't delete your own events.
You can't delete other events.
You can't send private messages.
You can't send emails.
You can read topics.
You can't vote in polls.
You can't upload attachments.
You can download attachments.
You can't post HTML code.
You can't edit HTML code.
You can't post IFCode.
You can't post JavaScript.
You can post emoticons.
You can't post or upload images.

Select a forum

































































































































































SQLServerCentral


Search