I need to build a competitors prices tracking tool for 100,000 products from around 15 competitors? What would be the best way to do it? Please recommend?
Read the web scraping chapter of the excellent (and free) eBook "Automate the boring stuff with Python".
That will tell you everything that you need to know.