I have a rather large list of domain names, and a rather large set of checks and audits that I have developed to review them all. I intend to attempt to train a machine learning model to make predictions once data mining is complete, but for the purposes of this question, say that all I need to do is download the html of each site and regex test it.
The issue I am having comes in with the sheer volume of websites that I have in my list as needing to be checked, which is over 10 million. Even just simple script to ping each site and save to a csv the hosting IP addresses will take weeks. (Python. Socket library calls are slow, as are the list of checks and audits I need to do when processing.)
I have been brain-storming, and thought that maybe a cluster of raspberry pi single board computers might be able to break up the task. Is that feasible?
I looked into Google's App Engine, but that, kubernetes, docker and the like are all beyond my skill set, and I am not even sure if they are a solution for this problem.
What recommendations can be made to accelerate the process?
Is a small cluster of Pi's a realistic solution?
Is there a better method to get through this list faster?
I am keen to learn, just need some help to point me in the right direction.