0

Guys is there a software or library I can use to parse contents of whole website and then perform analysis like for example which is the most frequent word used on the site? The most active user? etc. etc.

I want to parse the whole site, so say if it is forum I want to get access to ALL data on the site. All posts made.

  • Do you have control of or access to the web server? Or are you asking about analyzing a web site from outside? – Basil Bourque Jul 9 '17 at 19:55
1

You will need multiple tools to accomplish your goal.

Screen-scraping

First you need to retrieve the web page, and parse to extract the meaningful content removed from the chaff of the HTML markup.

jsoup project

For Java, I suggest the jsoup library. The word 'soup' is a cute reference to some incorrect HTML markup being "tag soup". This library has worked well for me in some projects, and is used in major software such as Vaadin.

The jsoup library handles both tasks of retrieving a page and parsing that page's contents. This process is commonly called "web scraping" or "screen scraping", by the way.

You could choose from many other HTML parsers as well.

Full text indexing

To track the usage of words, you need full text indexing.

logo of Apache Lucene project

A leading product for this is the Apache Lucene project. Built in Java, but also ported to several other languages.

To quote Wikipedia:

While suitable for any application that requires full text indexing and searching capability, Lucene has been widely recognized for its utility in the implementation of Internet search engines and local, single-site searching.

Lucene includes a feature to perform a fuzzy search based on edit distance.

logo of PostgreSQL database management system

Another approach is to use a powerful database such as Postgres that is capable of full text indexing and searching.

Log parsing

To determine active users, you would need to analyze the web server's logs.

See the Wikipedia page for Web log analysis software.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.