Software to search text by regular expression with analytic report

Question

I have many text files that contains syntax such as <Example><EG1>...<Eg1>. Well all of these can be captured by regular expression.

Most text editors can do finding words by regular expression.

I want to search multiple regular expressions and instead of search results, I want to find out the reports of word count.

For example, if i find word start with < and ends with >, the result should like this:

Search | Counts
<Example> 100
<Eg1> 100
<Antoerh> 

instead of showing a list of highlighted search results with text.

Are you any text search analytic software?

Answer 1

If it is a one-off task I would stick with grep -R '<[^>]*>' /path/to/files and store the result in a text file, to do the count afterwards. If this is a task that is going to happen multiple times, I would be tempted to use xapian or swish-e.

xapian can index all of the files into a database and then you can search that database using a regex. The other advantage of this, is that by indexing all of the files first, you don't need to know what regex you are going to need to apply, (and updating is relatively quick, and much faster than running another full grep over all of the files.)

When I last had to do this, I wrote a quick python script that would build/update the database and a second script that would let me search, that looked something like:

#!/usr/bin/env python
# xapian-search ver. 20180118111418 Copyright 2018 alexx, MIT License
# Import the os module for file system management
import os,sys
# Set the database path
databasePath = os.path.abspath('xapian-database')
# Import Xapian's Python bindings
import xapian
# Create the Xapian database
database = xapian.Database(databasePath)

xapian_file_name = 0
xapian_file_path = 1

def search(queryString):
    # Parse query string
    queryParser = xapian.QueryParser()
    queryParser.set_stemmer(xapian.Stem('english'))
    queryParser.set_database(database)
    queryParser.set_stemming_strategy(xapian.QueryParser.STEM_SOME)
    query = queryParser.parse_query(queryString)
    # Set offset and limit for pagination
    offset, limit = 0, database.get_doccount()
    # Start query session
    enquire = xapian.Enquire(database)
    enquire.set_query(query)
    # Display matches
    matches = enquire.get_mset(offset, limit)
    for match in matches:
        delim = ''
        path = match.document.get_value(xapian_file_path)
        if not str(path).endswith('/'):
            delim = '/'
        fileName='unknown'
        print('%s%s%s' % (path.decode("utf-8"), delim, match.document.get_value(xapian_file_name)))
    #print 'Number of documents matching query: %s' % matches.get_matches_estimated()
    #print 'Number of documents returned: %s' % matches.size()

search(str(sys.argv[1:]))

I can't release the script that created the database, (due to license) but I don't remember it being very difficult to write. Being python, you can have it output in the form that is most helpful to you.