3

I have recently converted ~30GB of old Outlook .pst files into text files using the tool, readpst.

The tool creates a new directory for each email thread, inside which it places a text file (for the text associated with the thread), and any attachments.

This has allowed me to quickly find attachments, but now I'd like to be able to quickly search the many tens of thousands of text files (i.e. email threads) for strings of various lengths. Using find and grep is far too slow, so I'm presumably looking for a way to index the files first.

I am based on Linux and am comfortable using the command line. I would prefer to keep the data offline, rather than uploading to an online tool. I don't have access to an instance of Outlook, so can't just load the original .pst files.

Finally, any recommended tool should be open source/gratis.

  • 1
    Would it be an option for you to convert the PST files to mbox format? You could then import those to your IMAP account (if you have such), where the IMAP server should index them, and thus be able to search them using your mail client. Just a raw idea, not tested. – Izzy Jan 6 '16 at 14:48
  • 1
    Here is a thought. If you swap your hard drive for an SSD you will get 5x+ speed increase for any solution. How many cores and what speed is your CPU? – cybernard Jan 7 '16 at 0:37
1

I use a Windows program called Everything, from voidtools.com. It is not what you want (it's Windows only, and folder and filenames only), but it is fast and flexible, so I used it as a starting point to see what else was available. A search at alternativeto.net for Everything, for Linux, gives 5 total hits. Search on Alternativeto.net

Of these, all 5 are open source. 3 claim to search contents as well: Regain, SearchMonkey, and Tracker. A fourth, Catfish, is unclear as to whether it searches content; it's described as a search front end.

Edit: The link to the regain website on alternativeto.net is obsolete, I believe. The one I found that works is: Regain website Note that is cross platform. Claimed to be very fast because it uses an index. On Windows, it has to be installed. Don't know about Linux.

The link to the SearchMonkey website is: SearchMonkey website It is also cross platform. It does not require installation on Windows. It does not seem to use an index, so may not be faster than grep.

I have not used Regain, but did try SearchMonkey. Seems OK. Again, on Windows.

Edit2 (1/19/16): I rediscovered another program that I had previously downloaded, DocFetcher. I ran some tests on it under windows and I believe it does what you want. It is free, open source, and cross platform. It will search inside files and display the relevant plain text. It supports multiple file types, including the Outlook pst. It will generate an index, with user control of the path being indexed, so you could index just your pst file. The indexing may take some time, but it is then quite fast. It needs Java. It's available at: DocFetcher on Sourceforge Its website, also on Sourceforge, is at DocFetcher website. The comments on Sourceforge are generally very positive.

  • 1
    VoidTools Everything searches by file name, not by file content. The answer seems to be a Google search result, which is discouraged. You have little to no experience with the programs you recommend, which is also discouraged. – Thomas Weller Jan 8 '16 at 14:04
  • @Thomas - I explicitly and correctly identified Everything as Windows and file/folders name search only. I explicitly and clearly indicated I used Alternativeto.net to do a search. I identified programs that might meet the OP's needs. I did not "recommend" them. So far, my response, as inadequate as you may find it, is the only one provided. The OP can certainly try these programs and comment back. – JKEngineer Jan 11 '16 at 13:46
  • 1
    I don't understand the concept of this approach: you know a program that can search for file names, then google for alternatives and hop you find programs that search for file content? Why not start with a program that searches file content and search for alternatives to that? This site is about software "recommendations", so why don't you recommend something? – Thomas Weller Jan 12 '16 at 7:21
  • DocFetcher looks promising - I'll take a look. If it does what it says it does, I'll accept this as the best answer. – Richard Horrocks Jan 20 '16 at 21:35
  • Search Monkey was my first thought, aslo I think that it does not build an index. For Windows, Copernicus desktop used to be the best (and does build an index), althoguh I prefer Agent Ransack – Mawg May 2 '17 at 8:31
4

As you're on linux and using grep/find already, let's pipe them to xargs and use it's concurrency option to speed things up. It'll depend on how much compute you have on your machine but should be quicker. For example, you're probably doing something similar to:

grep -i "<string>" /dir/path

Pipe the command to xargs and use the -n to state how many concurrent processes you want to run (4 is just an example here, could be 2 or could be 200):

grep -i "<string>" /dir/path | xargs -n 4

It will speed things up for you. If you're unsure as to how much it will speed things up you can do the following and compare completion times:

time grep -i "<string>" /dir/path

time grep -i "<string>" /dir/path | xargs -n 4
  • How does that xargs -n 4 work? – wb9688 Jan 20 '16 at 20:32
  • 3
    @wb9688 it runs the command concurrently the number of times specified after the -n, so ultimately faster. – ynnekkram Jan 20 '16 at 21:02
  • 1
    Thanks for the answer. I'd tried this suggestion already, but the impact was limited by the creakingly old hardware I'm running it on :) – Richard Horrocks Jan 20 '16 at 21:33
  • What about mounting the files in memory and doing it? – ynnekkram Jan 20 '16 at 21:41
  • What about runnig it on some other, not so creakingly old, hardware? :-) – Mawg May 2 '17 at 8:14
4

I don't have any experience converting outlook's PST files, but if you can get those emails into maildir format, there is an excellent tool called notmuch which seems to do exactly what you are asking. It will index your mail and allow you to quickly search through it. There are several front ends and clients designed to work with it as well. Read about it here: NotMuchMail.org

  • The separate file mode of readpst gives essentially something close to a Maildir. (The missing tags must not be a problems if your task is to search.) – imz -- Ivan Zakharyaschev Aug 26 '16 at 15:06
0

Since you are on a proper operating system and aren't afraid of the command line, can we assume you have some coding skills as well?

If so... I used to run a rouge archive of a super-specific-topic email listserv. Would run jobs every so often converting mbox mail to maildir and a file for each message, then scan the files, get rid of the headers, and store a keyword index in a mysql database.

You could do the similar - the individual file for each message is done, you'd just need to write the code to 1) keep track of the identifier/location of the message thread you are processing and 2) insert all of the words into a keyword/message id db table using mysql, mariadb, mongo, whatever you feel comfortable with.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.