2

I am looking for a personal search engine to index all my data, be it local or remote, so that I can find my own stuff easily.

Folders, tags, bookmarks, these are all annoying, a search engine is what I need. Is this possible already? Or should I start a company for this? :D

  • I seriously doubt there are personal use webcrawler – Braiam Apr 29 '14 at 22:13
  • @Braiam Why not? - easy to set one up for many use cases, I'll have to query qed a bit more on situation to get the details but I think I have a good match. – Nick Wilde Apr 29 '14 at 23:51
  • 1
    qed: I think I have a match, however need to know a few bits of data; do any of the remote locations you want to index require logging in? (if so would you be interested in a solution that couldn't do those ones but could do the rest of it?) What OS do you want it for? Would you be open to installing a web-server on your system? – Nick Wilde Apr 29 '14 at 23:53
  • 1
    @NickWilde I am on kubuntu 13.10 and apache running, so no problem with web servers at all. Some target sites like google drive do require login, but they have a nice search interface already, so no worries there, either. I want it on Linux, that's for sure. – qed Apr 30 '14 at 7:39
1

Sphider or (better) Sphider Plus will do pretty well for that.

I've found them both great. They are pretty much the same except that Sphider Plus is a thousand times better - think of the original as a 'lite' version. This is really a rec. for Sphider plus and that's the feature set I'll be primarily talking about.

To set that up in your case you'll need to install a [L]AMP stack - I'm not a regular Linux user but I believe that XAMPP is a good single download option. You'll need to enable directory listing and adjust the Apache VirtualHosts file (which will be located somewhere like this: Apache<versionnumber>\conf\extra\httpd-vhosts.conf).

The code you'll want will depend on what files you want searchable; I'd suggest something like this (using Windows paths but just change the paths as desired/required): - I'll be refering to these throughout these instructions but they are example only really.

<VirtualHost *:80>
    DocumentRoot "C:/Users/Nick/Desktop"
    ServerName www.desktop.loc
    ServerAlias desktop.loc
    <Directory "C:/Users/Nick/Desktop">
        Options Indexes FollowSymLinks
        AllowOverride All
        Order allow,deny
        Allow from all
    </Directory>
</VirtualHost>
<VirtualHost *:80>
    DocumentRoot "C:/Users/Nick/Pictures"
    ServerName www.pictures.loc
    ServerAlias pictures.loc
    <Directory "C:/Users/Nick/Pictures">
        Options Indexes FollowSymLinks
        AllowOverride All
        Order allow,deny
        Allow from all
    </Directory>
</VirtualHost>

and add to your OS hosts files:

127.0.0.1 desktop.loc
127.0.0.1 www.desktop.loc
127.0.0.1 pictures.loc
127.0.0.1 www.pictures.loc

Then install Sphider - Plus is probably better since it has easily set-up automated re-indexing. Probably want to make a new vhost location for search.loc or something for easy access.

Next add picture.loc and desktop.loc and any external locations to the Sphider index; it'll take a while depending on number of files etc. then you are good to go :D

Features:

  • no maximum number of files to index
  • Indexes HTML, and through built in converters can index PDF, DOCX, XLSX, ODT, ODS, CSV and XLS files - it will index filename only for images and such that it can't get contents from.
  • Quite reasonably priced (25 EURO for Plus, donation/free for original)
  • powerful admin interface
  • quite a bit of automation (ie auto-reindexing via CRON etc. for Plus only)
1

I have found YaCy to be a wonderful tool.

Web Search by the people, for the people

YaCy is a free search engine that anyone can use to build a search portal for their intranet or to help search the public internet. When contributing to the world-wide peer network, the scale of YaCy is limited only by the number of users in the world and can index billions of web pages. It is fully decentralized, all users of the search engine network are equal, the network does not store user search requests and it is not possible for anyone to censor the content of the shared index. We want to achieve freedom of information through a free, distributed web search which is powered by the world's users.

So far so good, at least much much better than Google custom search, since you can index virtually every page of your target domain.

One drawback is it's sensitive to word suffixes, i.e. you may not get a result when you search for "matrix" if the actual word is "matrices". Maybe there is a configuration for this somewhere, but I don't know how to do it yet.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.