1

Can someone recommend a software for searching links on Google and other search engines?

E.g. you would specify a regex for a link and the software would run on a schedule. Then it would email you the list of pages containing URLs it found.

Especially I am interested in finding links similar to the following:

https://www.youtube.com/watch?v=J6vIS8jb6Fs

where the URL before = is always the same, then followed by some unique ID.

  • You are libable to receive a B I G email, if you don't specify further. Can you code? If so, I can suggest an approach – Mawg Jan 12 '17 at 11:06
  • 1
    @Mawg Sure I can code, the problem is Google API is deprecated... Email isn't going to be big in my case, I am not including all YouTube videos, this is just an example. My company has an application and people post links online from time to time, we need to detect those. – Alex Jan 12 '17 at 16:43
  • What about Pyton and scrapy.org (or beutiful soup)? The problem is that you will have to repeatedly fetch pages until you have all results, but that's just a matter of adding &start=X to the URL. – Mawg Jan 12 '17 at 16:46
  • @Mawg the problem is I don't know where people would post these hyperlinks. Meaning, it's a two stage process - 1) Find matches in search engine 2) Scrap pages from the results and find the links on the page to confirm it's there. – Alex Jan 12 '17 at 17:48
  • 1
    Have you considered tracking the visitors and seeing where the traffic is coming from? Google Analytics would be one way to do this... – rovr138 Jan 14 '17 at 12:27
1

Since the Google API is no more, perhaps you could try another search engine?

DuckDuckGo's API might help, but might not ...

Our Instant Answer API gives you free access to many of our instant answers like: topic summaries, categories, disambiguation, and !bang redirects.

This API does not include all of our links, however.

That is, it is not a full search results API or a way to get DuckDuckGo results into your applications beyond our instant answers.

Because of the way we generate our search results, we unfortunately do not have the rights to fully syndicate our results. For the same reason, we cannot allow framing our results without our branding.

.

The Bing API looks more promising

Bring intelligent search to your apps and harness the ability to comb billions of webpages, images, videos, and news with a single API call.

BUT, see the pricing and decide if are you willing to pay.

.

What about Yahoo! (as was; what will happen now that it has been bought out?)

Ooops,

we will discontinue the BOSS JSON Search API on March 31, 2016.

Check out other search engines?

Or search for other APIs which might be useful? For instance on Programmable Web, which has thousands.

  • 1
    Current solution uses Bing API (which doesn't seem to return all the results we get using Google manual search). Also, the following is used instead of Google Web Search API googleapis.com/customsearch/v1 but it neither returns satisfactory results... – Alex Jan 12 '17 at 19:50
  • All that I can think of it using a combination of sites – Mawg Jan 13 '17 at 8:58
  • Or coding your own spider – Mawg Jan 13 '17 at 8:58
  • Surely my crawler wouldn't beat Bing, not to mention Google. Thanks for your suggestions though, maybe the question has no real answer to it at this moment in time. Need another search giant with API :) – Alex Jan 13 '17 at 17:00
  • 1
    Agreed, but you could do it. You only have to handle one page at a time and discard those that don't suit you. Hire processing power from AWS or Google maybe? It's that or incomplete results. Only you can decide – Mawg Jan 13 '17 at 17:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.