Building a search index, and querying it from the command line

Question

Does a tool with the following features exist?

Command Line Interface (for indexing and searching)

We have grep for command line searching of txt files. It’s quick as long as the file is at most a few megabytes.

Maintains an index (for searching with immediate results)

We have mlocate, which uses an index. Unlike grep, it maintains a database (i.e. index), rather than searches the content directly which improves performance.

However, it is meant only for file paths, that too paths that actually are valid in your file system rather than just corpuses (corpi?) of free form text.

Unconstrained in content (for indexing)

For searching of giant volumes of no restrictions on what items (i.e. documents) we store, we have Apache Solr. But those are meant mostly for web clients, not for the command line (though I know you can curl to its server and parse the XML response - yuck).

So my question is - does (the concept of) a tool that does the following exist?

1. the search interface is the command line (like grep)
2. it stores an index of plaintext items (like mlocate)
3. Has no restrictions on what that plaintext semantically represents (like Apache Solr)

I know I could write my own, but a) I’m not going to live that long b) surely someone in the software domain will have had this requirement in their life before?

Answer 1

I am not sure what giant volumes means to you, but http://sary.sourceforge.net/ might be a place to start.

On one of our systems we run glimpseindex every night. It produces an index and has a few options for setting the size of the index. The companion search command, glimpse, has numerous options for controlling the search and output.

Answer 2

Recoll seems to do what I'm looking for: http://www.lesbonscomptes.com/recoll/

Installation

sudo apt-get install recoll

Build an index

cat videos.txt | recollindex -if

Manpage:

   recollindex -i will index individual files into the database. The stem expansion and aspell databases will not be updated. The skippedPaths and skippedNames configuration variables will be used, so that some files  may  be
   skipped.  You  can  tell recollindex to ignore skippedPaths and skippedNames by setting the -f option. This allows fully custom file selection for a given subtree, for which you would add the top directory to skippedPaths,
   and use any custom tool to generate the file list (ie: a tool from a source code control system).

Search an index

recoll -t world cup

Output

video/webm  [file:///Large/Videos/documentary/Documentaries2/documentaries/The 1966 World Cup Documentary.webm] [The 1966 World Cup Documentary.webm]   43323557    bytes
video/x-ms-asf  [file:///Large/Videos/soccer/1986 World Cup Mexico /film/World Cup Mexico 1986 Highlights.wmv]  [World Cup Mexico 1986 Highlights.wmv]  75115064    bytes
video/mp4   [file:///Large/Videos/soccer/1990 World Cup Italia/Italia 90 World Cup 1990.mp4]    [Italia 90 World Cup 1990.mp4]  44752690    bytes
video/webm  [file:///Large/Videos/soccer/1994 World Cup USA/1994 FIFA World Cup Highlights.webm]    [1994 FIFA World Cup Highlights.webm]   99297071    bytes
video/webm  [file:///Large/Videos/soccer/1994 World Cup USA/WHITNEY HOUSTON IN WORLD CUP 1994.webm] [WHITNEY HOUSTON IN WORLD CUP 1994.webm]    209027435   bytes

Data location

0       ~/.recoll/xapiandb/flintlock
4.0K    ~/.recoll/xapiandb/iamchert
4.0K    ~/.recoll/xapiandb/position.baseA
4.0K    ~/.recoll/xapiandb/position.baseB
4.0K    ~/.recoll/xapiandb/postlist.baseA
4.0K    ~/.recoll/xapiandb/postlist.baseB
4.0K    ~/.recoll/xapiandb/record.baseA
4.0K    ~/.recoll/xapiandb/record.baseB
4.0K    ~/.recoll/xapiandb/termlist.baseA
4.0K    ~/.recoll/xapiandb/termlist.baseB
2.5M    ~/.recoll/xapiandb/record.DB
7.6M    ~/.recoll/xapiandb/termlist.DB
13M     ~/.recoll/xapiandb/postlist.DB
14M     ~/.recoll/xapiandb/position.DB
36M     ~/.recoll/xapiandb

Notes

• WRONG: If the input record doesn't exist on the file system, the record will still get added to the index but its content won't get indexed.
  • Actually, it won't index non-existing files. So there isn't much benefit in using recoll over mlocate
• Credits https://unix.stackexchange.com/a/76198/7000