I am looking for a free tool (website, software) to do the following:

  • Find all of the pages on a given domain name - if absolutely necessary, I can provide a list myself.
  • Count the total number of words on all pages combined.
    • Filter out any repeated sentences from the word count (e.g count the footer only once, then ignore it for all other cases).
  • Option to ignore elements with certain IDs or classes.

I need to give an estimate to a translation service about the number of words they need to translate for my website, but it would be too much work to do by hand (>2000 pages).

  • This would be simple to do if you program Python, which is easy to learn :-) Just use Scrapy. I am sorry, but I don't know of an off the shelf app to do so. If you don't find one, your choices seem to be 1) learn Python or 20 pay someone, on a site live Fiverr Good luck – Mawg says reinstate Monica Jul 8 '19 at 15:45
  • 1
    @Mawg I should probably clarify: I've tried already to do this is JS and Python - they both ended badly, because of buffer/RAM overload (and once accidently DDoSing my server). I just used request library in Python though - I might try Scrapy (I've not heard of it before). Thanks for the suggestion. – Geza Kerecsenyi Jul 8 '19 at 15:48
  • There are two very well known Python scrapers - Beautiful Soup and Scrapy. Scrapy generally;y gets the vote, but feel free to try both. If you add the Python tag, it will act like the Bat Singal to Steve Barnes, our resident Python guru :-) If you have buffer/RAM overload, consider restructuring your code, possiblt even writing interim results to text files on disk and freeing the data structures in RAM. But, try Scrapy first. Good luck :-) – Mawg says reinstate Monica Jul 9 '19 at 8:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.