April 18, 2013 at 7:15 am
I have a table column contains English and French. My FTS always returns something unexpected. I am not sure because of the multi lingual or not. Can someone tell? Thanks
April 18, 2013 at 8:57 am
Hi,
Full Text Search supports more than 50 languages - http://msdn.microsoft.com/en-us/library/ms142571(v=sql.105).aspx
Regards
IgorMi
Igor Micev,My blog: www.igormicev.com
April 18, 2013 at 9:11 am
Thanks. I read that but sounds like each column can have one specific language. My question was can ONE column stores multi lingual (English&French) for FTS? Because I can't get the expected result because the column I have the FTS index on containing English and French
April 18, 2013 at 9:17 am
What version of SQL Server you use. Check it in msdn library.
I presume you use FREETEXT() and/or CANTAINS(). Have you tried both?
Regards
IgorMi
Igor Micev,My blog: www.igormicev.com
April 18, 2013 at 9:26 am
SQL 2008 R2. I am using FREETEXTTABLE
April 22, 2013 at 3:58 am
My understanding is that a single column can support a single FT index in a single language.
This is because the parsing function for a FT index needs to know about how to break up words, and what token are considered as noise in a particular language.
Further, when you query a FT index, you would get less meaningful answers if the FT data was in multiple languages.
At the place where I work, we always get a small proportion of data in a non-english language (the public gives us the data, we take what is given), but all of it is treated as English within our FT indexing. Because the non-English data is under 5% of total data this very seldom gives false positives on a search, and we asume it give the same low amount of false negatives.
If you really need to accept data in multiple languages, my advice is to segregate this at the input stage, and store data for each language in its own database column and FT index.
Original author: https://github.com/SQL-FineBuild/Common/wiki/ 1-click install and best practice configuration of SQL Server 2019, 2017 2016, 2014, 2012, 2008 R2, 2008 and 2005.
When I give food to the poor they call me a saint. When I ask why they are poor they call me a communist - Archbishop Hélder Câmara
April 22, 2013 at 6:56 am
does anyone has experience to handle multi languages in a single column for FTS? If so, how to do that? I knew exactly there are 2 languages in the column English&French. But can this be resolved by programming? Is there a way to detect languages in programming level? Hope someone can help. Thanks in Advance!
Jim
April 22, 2013 at 7:05 am
If you know that the input is in English, add a token to the input in the form LANG:ENU, use a different language code with documents you know are in French. Then in your FT query always include the criteria LANG:ENU to get results from known English inputs.
You will still be using the wrong word-breakers and niose filters for one of the languages, but you should get good results for both languages.
Original author: https://github.com/SQL-FineBuild/Common/wiki/ 1-click install and best practice configuration of SQL Server 2019, 2017 2016, 2014, 2012, 2008 R2, 2008 and 2005.
When I give food to the poor they call me a saint. When I ask why they are poor they call me a communist - Archbishop Hélder Câmara
Viewing 8 posts - 1 through 7 (of 7 total)
You must be logged in to reply to this topic. Login to reply