Does anyone know how one can go about obtaining Whois data in bulk? I know there exist paid services (see links below), which offer exactly what I'm looking for, but I find it frustrating that there doesn't appear to be a legitimate way to go about grabbing this data. The reason I find it frustrating, is because the data itself is open and free.

I've thought about automating and parsing the results from running a command line whois but this goes against the disclaimers listed at the bottom of the command result output. I've also reached out to IANA and the best they could do is provide me with zone files. I've additionally reached out to PIR who manages the .org domain who also pointed me towards the DNS zone files. I'd appreciate any feedback or guidance into areas I can further research.

https://domainnamestat.com/ and https://www.whoisxmlapi.com/


"I find it frustrating that there doesn't appear to be a legitimate way to go about grabbing this data. The reason I find it frustrating, is because the data itself is open and free."

This part is wrong on multiple points.

First, there are numerous questions like this already on StackExchange, like in the Webmasters site or on StackOverflow (unfortunately, as offtopic there). Please take a moment to search.

But to summarize things:

First, stop saying "whois data". Everyone does this, agreed, but it is wrong. It is registration data, that you can (typically) access using the whois protocol on port 43 (or web interface on top of it). It may soon be through RDAP instead (another protocol) but it will still be same data. So "Whois data" does not fit.

Then, you need to understand how domain names work, and the fact that there are gTLDs and ccTLDs. All gTLDs are under contract with ICANN so you can find some common things, but ccTLDs are independant.

Registration data is not "free". When a registrant gives this data to a registry it is with some expected scope of use. It is not necessarily expected that this data is then copied "everywhere" and used for whatever other purposes (you are not stating your goals in fact with this data or why do you have this need). This is exactly what the newer GDPR is about.

Nor it is open in fact. Part of it may be public and as you can find out today many whois servers now reply with redacted parts, specifically because of the GDPR. In the future, the new RDAP will allow tiered accesses: typically law enforcement agencies would get all the data while public unprivileged access may just get names and countries for example, if even that.

DNS zonefiles provide list of published domain names (which is a subset of all registered domains names), this is free and public (in the gTLD world), but won't give you "whois data", that is registration data. As you observed yourself, you are not supposed to query registry whois servers in high volume, typically to reconstruct all data.

In the gTLD world, registrars are required to sell that data. See this other reply for hints: https://stackoverflow.com/questions/25057370/where-do-services-like-http-whoisxmlapi-com-get-their-data Of course it does not fit your bill since it is not free as in beer, you will need to enter more than 1000 contracts like that (at $10,000 per contract, that can be costly), and you still could get a lot of "redacted" data.

In the ccTLD world, there is no such contract, because registries are all thick and not thin like COM/NET, and that they are, specially from European countries often more strict on handling of private personal data and hence public access on it.

Some registries started an "open-data" initiative. So part of registration data (you still have privacy issues) can be available, albeit not in real time. See https://opendata.afnic.fr/ for example for the .FR registry.

Now you may be wondering, how did others do since you cite some examples. Probably, and typically, starting with zonefiles, and slowly doing whois query to collect all data. This is of course not without perils, and you can abandon any hope to do that today on a large scale without a lot of resources. Note that there are even lawsuits between registries and websites aggregating data from whois, like http://www.domainpulse.com/2019/02/13/domaintools-violate-nz-harvesting-whois-data/

So, in short, again, why do you need that "bulk" registration data in the first place?

Not the answer you're looking for? Browse other questions tagged or ask your own question.