There was a Web Crawler called sparkler but today is defunct.
I'm looking for an alternative that could:
- crawl the web and list top URLs with number of iterations
- have a dashboard that could show the big picture like sparkler was doing:
In alternative, if it doesn't comes with a dashboard it would be great at least to output the results to CSV and I will take care of that through Power BI.
But at least I need to save the output somewhere.
Everything points to nutch as an alternative but it really looks mediocre to me and I could not even pass the 4th installation step.