SQLServerCentral / Article Discussions / Article Discussions by Author / Discuss content posted by Peter Kierstead / T-SQL Data Processing / Latest Posts

RE: T-SQL Data Processing

chenthor — Wed, 30 Jul 2008 14:02:22 GMT

Great article! very cool!

RE: T-SQL Data Processing

John Mitchell-245523 — Mon, 07 Jul 2008 09:00:45 GMT

[quote][b]Frances L (6/25/2008)[/b][hr]Source is the column name. I still do not understand Source&1 or Source&2 here.[/quote]For enlightenment, read this article and then wade through the discussion:http://www.sqlservercentral.com/articles/Miscellaneous/2748/John

RE: T-SQL Data Processing

Peter E. Kierstead — Wed, 25 Jun 2008 14:03:59 GMT

Source is the name of a TinyInt column in the table.Source&4<>0 is a method of determining a binary bit's value, in this case the 3rd bit from the right.Source&4 is a boolean AND operaration using 4 as the mask, so if:Source = 00000111 (decimal 7)opcode = &(mask) = 00000100 (decimal 4)--------------------------------yields 00000100which is not equal to zero therefore the CASE statement would return 1.

RE: T-SQL Data Processing

Frances L — Wed, 25 Jun 2008 12:01:18 GMT

Source is the column name. I still do not understand Source&1 or Source&2 here.

RE: T-SQL Data Processing

Peter E. Kierstead — Wed, 25 Jun 2008 10:22:21 GMT

Our current database represents data source as a TinyInt column containing bit values to indicate source type; 1=Source1, 2=Source2, 4=Source3, etc... where the decimal numbers 1, 2, & 4 represent binary bits 00000001, 00000010, 00000100, respectively.In order to process these values within then constraints of SQL (SQL not having an aggregate OR function) I break the TinyInt into 3 separate columns Source1, Source2 & Source3 (as seen in the code). The 3 case statements, you've identified, serve this purpose.

RE: T-SQL Data Processing

Frances L — Wed, 25 Jun 2008 09:16:01 GMT

Select PartyId, FirstName, LastName,[sup] Case when Source&1<>0 then 1 else 0 End[Source1], Case when Source&2<>0 then 2 else 0 End[Source2], Case when Source&4<>0 then 4 else 0 End[Source3],[/sup] AddDate, ModDate from dbo.ExistingMasterWill you please let me know what Case when Source&1<>0 then 1 else 0 for in this code ? Thx.

RE: T-SQL Data Processing

Peter E. Kierstead — Mon, 23 Jun 2008 23:19:49 GMT

This process was designed on a SQL 2000 system and ported to SQL 2005 with minimal changes. When I refactor it for SQL 2005 that would be the way to go.

RE: T-SQL Data Processing

Jacob Luebbers — Mon, 23 Jun 2008 19:36:25 GMT

Small suggestion: maybe I missed a reference to this, but wouldn't the Bit datatype be perfectly suited to the Data Source column (one bit column per source)? No need for explicit bitwise operations, acheives the same thing as you're doing against a TinyInt under the covers and the resulting code would be a little more human-readable...?Regards,Jacob

RE: T-SQL Data Processing

Peter E. Kierstead — Mon, 23 Jun 2008 09:33:18 GMT

I'm not a fan of dedup either, and didn't expect a harty round of approbation from this article...I've found that you should ALWAYS consider your alternatives when it comes to SQL programming. Things frequently don't "work" as we think they "should".I use this code to dedup databases of over 1 billion (yep, the B word) rows, a quarter of which (~250,000,000) are duplicates of some kind (exact or based on some "fuzzy" logic). One database processes in about 3 days, the other in about 18 hours (one has a larger record size than the other).This represented a significant reduction in processing times for both these databases over the previous methodology written using a hybrid of external and internal (SQL w/cursors) coding; and its all done in SQL.I am constantly looking for betterprocessing techniques that perform the required functions AND run quicker than existing procedures. So far, this is it.As for SSIS, I will look into it, having not used it before.

RE: T-SQL Data Processing

Jerry Hung — Mon, 23 Jun 2008 08:28:35 GMT

I don't like dedupe, let me tell you that :)But I can't imagine how this WHILE loop would work on millions of rows of data comparisonI had issues with comparing between 400K records vs 8000 inserts using SET operations (still acceptable if I separate by country, etc..)started looking to SSIS Fuzzy Matching and Fuzzy LookupFuzzy Matching - It's pretty cool, dedupe within say the Master table without any T-SQL work, takes a while but it even gives confidence scoreFuzzy Lookup - I am still working on it, supposedly I can lookup those 8000 inserts in the 400K Master table

RE: T-SQL Data Processing

GSquared — Mon, 23 Jun 2008 07:15:14 GMT

I may be missing something here, but why use a While loop for this kind of thing? Do a set-based merge of all the update data, then upsert it into the master table. Two steps, very simple, very clean, no While loop.

T-SQL Data Processing

Peter E. Kierstead — Sat, 21 Jun 2008 11:12:40 GMT

Comments posted to this topic are about the item [B]T-SQL Data Processing[/B]