There is a lot of reinvention of the wheel. I've worked with full-text in both Oracle (9i, 10g) and SQL Server (2000, 2005, 2008) as well as external, i.e., not-integrated, full-text systems such as AltaVista, etc.

Now that SQL Server, in 2008, has an integrated full-text engine similar to what Oracle had in 9i (circa 2000), full-text in SQL Server is pretty robust. However, it still lacks a lot of Oracle's functionality.

We use it for plain text, HTML, and binary (Microsoft Office files, PDFs, etc.).

Eliminate the Stop (noise) word filtering as it is so 90's. I.e., index everything.

As for performance and capacity, just look at what Iron Mountain is doing.