I'm developing a script that, using a third-party API service, gets information about the football matches. The data is taken for each match separately (by match id). There is also a limit of requests for method of API (1000 requests per hour). And because of there are 14 thousand of matches, this process will take more than 14 hours.

What is the best solution to run this process in the background: common Cron or Message Broker?

Thanks in advance!

  • The thing to remember is that there are not 14,000 matches in any given hour, I would be surprised if there were 1,000 matches finishing in a given hour so first find one or more sources that you can query for which matches are on a given day and their start times then you only need to query the matches that are on, after their start time and probably near/after the finish. – Steve Barnes Oct 28 '16 at 5:43
  • @Steve Barnes First of all I need to get all matches data from their DB – jekahm Oct 28 '16 at 23:08
  • Surely this is not the only source of match schedules? Not knowing the service I can't be sure but check the API for some method such as Today's Matches, Scheduled Matches or Matches in Progress. – Steve Barnes Oct 29 '16 at 4:54

I would strongly suggest against either one of these, on account of your batch processing's bottleneck being the data ingestion. Specifically, attempting to do both the query, and the relevant processing, if your processing hypothesis is wrong, you will have to repeat data ingestion.

Rather, I'd suggest writing a script, which does nothing else but query the API, and store the results as-is (into either plain JSON, SQL, what have you) in a format that is the exact replica of the original data; then add eg. a 3.6 seconds delay after each query. This way you can start hacking on the data as it arrives, have the confidence that all data will be at least be downloaded, while still remaining within the API query limits.

After the data have been downloaded, and to process ongoing ingestion, you can hook up your processing routines at the end of the query directly; this keeps your architecture parsimonious, and dynamic to changes.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.