I'm developing a script that, using a third-party API service, gets information about the football matches. The data is taken for each match separately (by match id). There is also a limit of requests for method of API (1000 requests per hour). And because of there are 14 thousand of matches, this process will take more than 14 hours.
What is the best solution to run this process in the background: common Cron or Message Broker?
Thanks in advance!
I would strongly suggest against either one of these, on account of your batch processing's bottleneck being the data ingestion. Specifically, attempting to do both the query, and the relevant processing, if your processing hypothesis is wrong, you will have to repeat data ingestion.
Rather, I'd suggest writing a script, which does nothing else but query the API, and store the results as-is (into either plain JSON, SQL, what have you) in a format that is the exact replica of the original data; then add eg. a 3.6 seconds delay after each query. This way you can start hacking on the data as it arrives, have the confidence that all data will be at least be downloaded, while still remaining within the API query limits.
After the data have been downloaded, and to process ongoing ingestion, you can hook up your processing routines at the end of the query directly; this keeps your architecture parsimonious, and dynamic to changes.
