2

I'm a little distressed, for my job I need to get some info from a website, specifically http://www.museumsusa.org/museums/ , I need for every museum in America to get the name, address, phone number and e-mail address, if they exist.

I would prefer not to copy-paste 20000 strings; does anyone know of a tool to help me?

Improve this question
2
  • 1
    Welcome. if you use the search facility on the forum, you will find a few scraping tools. See softwarerecs.stackexchange.com/search?q=scraping you caan also add more keywords to narrow down search.
    – Z Z
    Commented Feb 1, 2019 at 9:35
  • Have you looked on opendata.stackexchange.com ? (but DO NOT post the same question to both sites, or it WILL be closed). Good luck
    – Mawg
    Commented Feb 1, 2019 at 11:30

1 Answer 1

Reset to default
1

The answer depends on your skills.

I would tell the problem to my friend ruby. She probably will use mechanize to do the job.

If you need some guidance, consult the search engine of your choice for a book or video with the term "web scraping" in it's title. I checked Safari, and guides for JavaScript, Python, Java turned up in the search results.

If you go the coding route, remember to be nice with the server operators.

If coding is not your thing, there used to be software for clicking together bots in a GUI like Kapow out there, but I am not up to date regarding the state of the art here. Kapow seems to have turned into this.

Improve this answer
2
  • Welcome aboard. Good answer (+1). For Python, look at scrapy or beautiful soup
    – Mawg
    Commented Feb 1, 2019 at 11:27
  • There is a Python version of mechanize as well. Both the Ruby and Python versions are inspired by (if not directly derived from) Perl's mechanize.
    – John Y
    Commented Feb 1, 2019 at 22:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.