One of our users complained that the search functionality was broken. Searches that should have returned results came back empty. The cause turned out to be a single word: 'both'. If you are new to Full-Text Search, start with my previous post. It explains the building blocks I use below. The problem The search in our application is built on SQL Server Full-Text Search. The terms a user types are combined into a CONTAINS query. A simplified version looks like this: SELECT ProductId, Name FROM dbo.Products WHERE CONTAINS(Description, '"bike" AND "both"'); This returned no rows, even though we knew there were products matching both terms. Searching for "bike" alone worked fine. Looking for the cause The cause is simple: Stop words Stop words are words that are not indexed because they are considered too common to be useful. SQL Server ships with a system stoplist per language, and 'both' is on the English one: S...
Although we have ElasticSearch running in our environment, our most important search functionality is still handled by SQL Server. And no, it is not by writing a statement like WHERE Name LIKE '%term%' . That would probably work on a small table, but we have a lot of data to dig through, so we need something smarter than substring matching. SQL Server has a built-in answer for this: Full-Text Search . In this post I walk through the building blocks and the main features. In the next post I'll show what happened when a customer ran into a surprising side effect. Why do we need this? A LIKE with a leading wildcard can't use an index seek. SQL Server has to scan every row and compare strings. On top of that, LIKE knows nothing about language: searching for 'ride' won't find 'riding', and there is no concept of relevance. Full-text search works differently. It builds an inverted index: a list of words (tokens) and, for every word, the rows wh...