People generally love the speed of index searches, but there are a few power users who wish they could also run regular expression (regex) searches against the index. The power of regex with the speed of the index, what’s not to like? From the support desk this week:
“I’d like to use a regular expression in an index-based search to find lines that simultaneously contain the keywords 230 and 2015, with the requirement that 230 must be surrounded by whitespace characters on both sides, and 2015 must be preceded by a whitespace character.”
The trouble is that indexes aren’t designed to handle regular expression searches. To understand why, let’s think about how an index works.
How does an index work?
Just like the index at the back of a book, the index for search data is basically a sorted list of terms taken from the source data. Anything that doesn’t look like a term, eg, a lone % sign, is stripped from the text. There are some additional data, such as term frequency, and term positions, but the terms aren’t stored in sentence order.
For example, the line of text:
file_locator work-efficiency calculation:(normal-speed*50%)=lots
Results in the following term database:
50
calculation
efficiency
file_locator
lots
normal
speed
work
An index search can quickly scan the term database for the individual search terms and link them back to the original document.
However, a regex, which is typically implemented as a directed graph of nodes and arcs representing transitions based on the input stream, requires the whole stream of text to work. It won’t work on a database of terms. Now, you could run a regex over the individual terms in the database (which we may implement in the future), but a regex spanning multiple terms is not possible.
Looking back at the original question, the end-user’s request doesn’t necessarily need to be a regex to work. You can use angle brackets to define the word boundaries and the NEAR operator for proximity. So, this search term would probably suffice:
<230> NEAR <2015>
Note: Notice how certain characters are stripped from the text, such as *, %, and -. That has the unfortunate side effect of preventing searches for those characters. The program will, however, compensate, so a search for ‘work-efficiency’ still works, but it will also match ‘work efficiency’ or even ‘work#efficiency’.
I REALLY want to use a regex
Not all is lost. Most of the effort spent searching through PDFs and Word docs is spent converting the file formats into text. The actual string search is very fast. That conversion effort can be avoided with the Caching functionality, which stores extracted text in a local database for super quick retrieval in future searches.
So, if you need to run regex searches on a group of documents consider switching on Caching for lightning-fast searches, it’s not as quick as Index search but it’s pretty damn fast.
I’m pleased to report that Agent Ransack 2019 is now available for download.
