I have a rather large list of domain names, and a rather large set of checks and audits that I have developed to review them all. I intend to attempt to train a machine learning model to make predictions once data mining is complete, but for the purposes of this question, say that all I need to do is download the html of each site and regex test it.

The issue I am having comes in with the sheer volume of websites that I have in my list as needing to be checked, which is over 10 million. Even just simple script to ping each site and save to a csv the hosting IP addresses will take weeks. (Python. Socket library calls are slow, as are the list of checks and audits I need to do when processing.)

I have been brain-storming, and thought that maybe a cluster of raspberry pi single board computers might be able to break up the task. Is that feasible?

I looked into Google's App Engine, but that, kubernetes, docker and the like are all beyond my skill set, and I am not even sure if they are a solution for this problem.

What recommendations can be made to accelerate the process?

Is a small cluster of Pi's a realistic solution?

Is there a better method to get through this list faster?

I am keen to learn, just need some help to point me in the right direction.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.