Click here to monitor SSC
SQLServerCentral is supported by Red Gate Software Ltd.
 
Log in  ::  Register  ::  Not logged in
 
 
 
        
Home       Members    Calendar    Who's On


Add to briefcase

Word analysis with SQL Expand / Collapse
Author
Message
Posted Friday, January 18, 2013 10:54 AM
SSC Rookie

SSC RookieSSC RookieSSC RookieSSC RookieSSC RookieSSC RookieSSC RookieSSC Rookie

Group: General Forum Members
Last Login: Wednesday, May 01, 2013 11:47 AM
Points: 39, Visits: 264
I did a blog post earlier called PASS Summit Twitter Dashboard. It shows all tweets by people at the PASS summit.

I now want to expand that by measuring the sentiment score. Logic to measure the score is below:

there are a list of pre defined good and bad words with a score of 1 and -1 respectively.
for each tweet, remove punctuation marks from text.
compare words from each tweet with the predefined words list.
get score based on matched words.
sentiment score is the sum of scores from the above match.

The logic may be inaccurate, but from what i learnt this model is currently being researched and is acceptable.

Here is the sample DDL.

Predefined words and score:

CREATE TABLE #Words
(
Id int identity(1,1) primary key
, Word char(10)
, Score int) ;

INSERT #Words
(Word, Score)
VALUES ('Good',1)
, ('Awesome', 1)
, ('Super', 1)
, ('Bad', -1)
, ('Fail', -1)
, ('Dirty', -1) ;


Tweets:

CREATE TABLE #Text
(Id int identity(1,1) primary key
, [Text] varchar(140))

INSERT #Text
([Text])
VALUES
('New Bond movie is #awesome!')
, ('I hear dirty reviews. Product X is a fail. #fail')
, ('I am neutral!!!')

Result:

CREATE TABLE #Result
([Text] varchar(140), Score int)

INSERT #Result
VALUES
('New Bond movie is #awesome!',1)
, ('I hear dirty reviews. Product X is a fail. #fail',3)
, ('I am neutral!!!',0)

SELECT *
FROM #Result

For example, score for 'New Bond movie is #awesome!' is 1 because after removing punctuation mark (!) word awesome matches with a word in the Words table and score is 1.
Score for 'I hear dirty reviews. Product X is a fail. #fail' = 3 because of the words dirty, fail, and fail (after removing #).

Query should be able to perform with a huge data set, approximately 100K rows.

Many thanks for your time and input.

Sam.


~Sam.

http://svangasql.wordpress.com
Post #1409016
Posted Friday, January 18, 2013 11:18 AM


Ten Centuries

Ten CenturiesTen CenturiesTen CenturiesTen CenturiesTen CenturiesTen CenturiesTen CenturiesTen Centuries

Group: General Forum Members
Last Login: Today @ 3:29 AM
Points: 1,296, Visits: 3,874
you might want to take a look at the proc dm_fts_parser

It might just provide the word analysis you are looking for...

Link : http://msdn.microsoft.com/en-us/library/cc280463(v=sql.100).aspx


MM




Post #1409032
Posted Sunday, January 20, 2013 5:43 PM


SSCrazy

SSCrazySSCrazySSCrazySSCrazySSCrazySSCrazySSCrazySSCrazy

Group: General Forum Members
Last Login: Today @ 12:39 AM
Points: 2,345, Visits: 3,189
If Mr. Magoo's link doesn't give you any ideas, you might want to take a look at the 4th article in my signature links to retrieve the FUNCTION PatternSplitCM and then try it this way.

SELECT a.ID, a.[Text], TotalScore=ISNULL(SUM(c.Score), 0)
FROM #Text a
CROSS APPLY PatternSplitCM([Text], '[A-Za-z]') b
LEFT JOIN #Words c ON b.Item = c.Word
GROUP BY a.ID, a.[Text]


If you're working in a case sensitive database, you'll want to UPPER the Item and Word in the ON clause of the LEFT JOIN.



No loops! No CURSORs! No RBAR! Hoo-uh!

INDEXing a poor-performing query is like putting sugar on cat food. Yeah, it probably tastes better but are you sure you want to eat it?

Need to UNPIVOT? Why not CROSS APPLY VALUES instead?
Since random numbers are too important to be left to chance, let's generate some!
Are you too recursively challenged?
Splitting strings based on patterns can be fast!
Post #1409332
Posted Monday, January 21, 2013 11:52 AM
SSC Rookie

SSC RookieSSC RookieSSC RookieSSC RookieSSC RookieSSC RookieSSC RookieSSC Rookie

Group: General Forum Members
Last Login: Wednesday, May 01, 2013 11:47 AM
Points: 39, Visits: 264
Dwain - Thank you very much. It works like a charm!

~Sam.

http://svangasql.wordpress.com
Post #1409665
Posted Wednesday, January 23, 2013 10:32 AM
Valued Member

Valued MemberValued MemberValued MemberValued MemberValued MemberValued MemberValued MemberValued Member

Group: General Forum Members
Last Login: Yesterday @ 1:20 PM
Points: 54, Visits: 277
@sam, can you test this one too, just curious, not sure if this helps performance wise..


;with cte1(part,leftover,textid) as
(
select left([text],CHARINDEX(' ',[text])),RIGHT([text],len([text])-CHARINDEX(' ',[text])), id from #Text
union all
select left(leftover,case when CHARINDEX(' ',leftover)=0 then LEN(leftover) else CHARINDEX(' ',leftover) end ),RIGHT(leftover,len(leftover)-case when CHARINDEX(' ',leftover)=0 then len(leftover) else CHARINDEX(' ',leftover) end ),textid from cte1 where leftover!=''
),
cte2(trimpart,id) as
(
select cast(part as varchar(140)),textid from cte1
union all
select cast(replace(trimpart,substring(trimpart,patindex('%[^a-z]%',rtrim(ltrim(trimpart))),1),'') as varchar(140)),id from cte2 where patindex('%[^a-z]%',rtrim(ltrim(trimpart)))!=0
)
select t.id,t.[text],isnull(recurrence,0) from(select id,COUNT(id) as recurrence from(select id,trimpart from cte2 where patindex('%[^a-z]%',rtrim(ltrim(trimpart)))=0 and trimpart in (select [word] from #Words))temp
group by id)temp1
right join
#Text t
on t.id=temp1.id
Post #1410703
Posted Wednesday, January 23, 2013 5:29 PM


SSCrazy

SSCrazySSCrazySSCrazySSCrazySSCrazySSCrazySSCrazySSCrazy

Group: General Forum Members
Last Login: Today @ 12:39 AM
Points: 2,345, Visits: 3,189
sqlbi.vvamsi (1/23/2013)
@sam, can you test this one too, just curious, not sure if this helps performance wise..


It is a good suggestion to do a performance test. I like general tools like PatternSplitCM for the ease with which they allow you to solve a problem quickly. Most of the time a focused, working solution will outperform a general tool. But the general tool may be a good, bootstrap approach to get your code working.



No loops! No CURSORs! No RBAR! Hoo-uh!

INDEXing a poor-performing query is like putting sugar on cat food. Yeah, it probably tastes better but are you sure you want to eat it?

Need to UNPIVOT? Why not CROSS APPLY VALUES instead?
Since random numbers are too important to be left to chance, let's generate some!
Are you too recursively challenged?
Splitting strings based on patterns can be fast!
Post #1410848
« Prev Topic | Next Topic »

Add to briefcase

Permissions Expand / Collapse