2

I'm a little distressed, for my job I need to get some info from a website, specifically http://www.museumsusa.org/museums/ , I need for every museum in America to get the name, address, phone number and e-mail address, if they exist.

I would prefer not to copy-paste 20000 strings; does anyone know of a tool to help me?

1

The answer depends on your skills.

I would tell the problem to my friend ruby. She probably will use mechanize to do the job.

If you need some guidance, consult the search engine of your choice for a book or video with the term "web scraping" in it's title. I checked Safari, and guides for JavaScript, Python, Java turned up in the search results.

If you go the coding route, remember to be nice with the server operators.

If coding is not your thing, there used to be software for clicking together bots in a GUI like Kapow out there, but I am not up to date regarding the state of the art here. Kapow seems to have turned into this.

| improve this answer | |
  • Welcome aboard. Good answer (+1). For Python, look at scrapy or beautiful soup – Mawg says reinstate Monica Feb 1 '19 at 11:27
  • There is a Python version of mechanize as well. Both the Ruby and Python versions are inspired by (if not directly derived from) Perl's mechanize. – John Y Feb 1 '19 at 22:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.