Does anybody know if there's any program/library able to extract "main contents" (like the Readability experiment) from a forum-like HTML page?

  • open-source;
  • should be able to extract:
    • posts contents (and properly separate each post),
    • "prev/next page" link,
    • optionally also: authors, title, date(s);
  • at minimum, should be able to properly process "most phpBB sites" (fuzzy, I know), and any Stack Exchange Q+A page;
    • bonus points for a Stack Exchange "hot"/"new"/… listing page;
    • bonus points for handling http://news.ycombinator.com comments pages, or main ("listing") page, or reddit main/comments pages;
  • as to point above, should still be "generic", i.e. not specially-tailored to parse exact HTML layout of Stack Exchange/phpBB;
  • using ?mode=print is allowed, but solutions working without this get more "points".

edit: just to make it clear, please note I don't need help with the initial step of downloading the HTML, so answers like "use wget/curl" won't help me a bit.

  • akavel, I removed your last sentence asking for papers about this topic, as it might invite off-topic answers. – unor Sep 11 '14 at 10:06
  • @unor, didn't having that sentence make them explicitly on-topic? that was my intent, at least; I'm really interested in papers (as research-quality software) too, as I have somewhat low expectations on there being such "production-quality" software already – akavel Sep 11 '14 at 10:22
  • Questions on SR have to ask for software (and nothing else). While it can be appropriate for an answer to provide links to papers (for example, when the answerer knows that there exists no solution, and recommends implementing it yourself), explicitly asking for papers is problematic because it would invite answers from people knowing about papers but nothing about available software. – unor Sep 11 '14 at 10:44
  • @unor, big thanks for the explanation. Sounds acceptable compromise for keeping balance on the site. Now, this sparked interest in me whether after SR hopefully matures, a Paper Recommendations SE will be born out of it :) – akavel Sep 11 '14 at 12:34
  • I am not sure which sort of papers you were looking for if there isn't already one you could check out area51.stackexchange.com where you can suggest possible new sites. But if you were after Computer Science Theory papers there is cstheory.stackexchange.com as a possibility. – Steve Barnes Sep 11 '14 at 16:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.