Intranet crawler and search engine

Question

I am looking for any really free alternatives for implementing an intranet web-search engine.

I know that Google Search Appliance (GSA) would be probably the best but it's extremely expensive and I do want to be able to crawl millions of pages.

I tried SearchBlox and in addition to the fact that I found it poorly documented and counter-intuive it also have a limit of 25.000 documents, which is almost nothing compared to the level I want to scale it to. Maybe if it was better I would have considered upgrading to commercial license but based on the experience, I wouldn't pay for it.

So, now I am looking for other approaches

Answer 1

Solr can do this. With it you can define a datasource to crawl it and it interprets the data. Solr is gratis and open source.

Solr has an extremely fast search engine (Lucene) and can import a lot of data. It is optimized for "field like" data like XML, JSON or HTML but there are converters of everything that contains text (like Word, PDF, etc.)

If you just need a few intranet sites (<100.000) to be indexed AND you can access the database you should just set up the Data Import Handler to do the work for you.

Else you will need to code some own thing that sends the data to the server.

Keep in mind that this is a development tool and not an end user program. You will need to create some interface. (It is quite easy, in my experience)

If you store the information in various fields (like meta, title, url, content, language, filetype...) you can search through this fields specifically, thous having the possibility to narrow your search down. If for example all your sites in the intranet have an author and you can access that field and index it, you can search for all documents that are from this author, but ignore all that are about him. It also supports fuzzy search ("seach" finding "search")

I used Solr in one (and the underling search engine Lucene in many) projects and was very impressed by it. The high flexibility in the data processing engines are incredible. The searching part is so fast that I have it on my list to one day understand how it works :)

If all you need is a search crawler & search interface, then the configuration overhead of Solr might not be what you need. But if you need a tool that chews through 30.000.000 documents than this is the tool to go. In the project I used it (with said amount of documents) we had more trouble with network latency then with the Solr search time. You can replicate the index and use a load balancing Solr instance that distributives the search requests to the other ones. And and and... the amount of different optimizations of this tool are staggering. That of course comes with a bit of necessary configuration that might not be super intuitive.

As hinted above, Solr is a wrapper around Lucene so if you already have a CMS to do the site creation for you there might be already a Lucene plugin for it that you can tap into.

Answer 2

Duckduckgo is an Internet crawler and search engine.

Here is its source code: https://github.com/duckduckgo

You can compile it, configure it to crawl only your intranet websites, and host it on a server in your intranet.

That being said, crawling web interfaces is not very efficient, you can do much better (with Solr for instance) if you have back access to where the content is actually stored. This will also allow you to show different results to people who have different access rights (for instance, only HR people can search resumes).

Answer 3

Sphider or (better) Sphider Plus

So far I've found them both great. They are pretty much the same except that Sphider Plus is a thousand times better - think of the original as a 'lite' version. This is really a rec. for Sphider plus and that's the feature set I'll be primarily talking about.

• Requires PHP 5 or 6.
• Database (MySQL - IIRC can also use a few other DB Engines)

Features to match your requirements:

• no maximum number of files to index
• Indexes HTML, and through built in converters can index PDF, DOCX, XLSX, ODT, ODS, CSV and XLS files.
• Quite reasonably priced (25 EURO)
• powerful admin interface
• quite a bit of automation (ie auto-reindexing via CRON etc.)

Answer 4

It sounds like you'll be needing to crawl over the available sites as well? A free web crawler I've set up is Heritrix. It is open source and is what the Internet Archive's "Wayback Machine" runs on.

https://webarchive.jira.com/wiki/display/Heritrix/Heritrix

You would then need I suppose need to combine it with something like Solr to do the actual indexing for the searches.