2

I have a list of HTTP URLs and I want to know if the corresponding HTML pages contain structured data markup, and if yes, which syntaxes and which vocabularies they use.

The tool can be anything: a desktop program (no matter which OS, but ideally for GNU/Linux), a Web app (no matter if self-hosted or hosted), a script, … I’d prefer not to have to script/program something myself, but if all needed parts are figured out and I only have to connect them or adjust something, I’m open for it.

Free/Libre software preferred, but not required. It doesn’t have to be gratis.

Syntaxes

At the minimum, the tool must detect the syntaxes

  • JSON-LD (in script elements with the application/ld+json type),
  • Microdata, and
  • RDFa.

Bonus points for support of other embedded syntaxes (e.g., Turtle in script elements), or even for checking alternative representations (e.g., RDF/XML via content negotiation).

Vocabularies

It should output all used vocabularies (so not only looking for specific ones, like Schema.org).

It would be great if the tool outputs all specified types/classes and properties in addition.

I’m not interested in the full RDF graph or the actual data, I only want to know which elements from vocabularies get used.

Example output

It could be something like this:

URL                        Syntaxes                Vocabularies
------------------------------------------------------------------
http://example.com/foo     Microdata               Schema.org
http://example.com/bar     RDFa, JSON-LD           OGP, Schema.org
http://example.net/i       RDFa                    FOAF
http://example.org/login   Microdata               <no vocabulary>

Or with types:

URL                      Syntaxes     Types
-----------------------------------------------------------------------
http://example.com/foo   Microdata    schema:Organization, schema:Thing
http://example.net/i     RDFa         foaf:Person, foaf:Document

The output should be in a suitable format for gathering statistics.

  • (FWIW, Web Data Commons published their Extraction Framework under an open source license. It’s for getting stats about usage of Microdata/Microformats/RDFa.) – unor Mar 1 '16 at 8:16
  • http: class URLs invariably point to HTML pages; I consider HTML to be "sturctured data markup" so I'm confused about what you are looking for. At best, you appear uninterested in HTML, but rather what's in script tags. Can you give an example of each kind of "structured data", and what you want to extract from that? (If this involves any digging in source code inside the script tags, I think you are asking for rather a lot of machinery). – Ira Baxter Apr 17 '16 at 3:33
  • @IraBaxter: I’m only interested in structured data that uses (at least) the mentioned syntaxes JSON-LD/Microdata/RDFa (so RDF, with the exception of Microdata, but there is a Microdata-to-RDF spec available). Examples can be seen, for example, on Schema.org’s website (for Product, bottom of the page) or on Google’s doc for their Product Rich Snippet. – unor Apr 17 '16 at 13:29
  • I checked out the Microdata spec. To detect the presence of Microdata, one might get away with fetching a page and using regex to hunt for the phrase "itemdata". That's not exactly hard. To extract the microdata (which you seem to want to do) requires that one parse the HTML(5) text and collect the itemdata spread across the tags and their descendents. You presumably can do that with the access procedures the Microdata spec implies exists in some DOM. I'd guess you'd need something similar for the JSON/RDF data. This sounds like you need lots of machinery to handle 3 standards.... – Ira Baxter Apr 17 '16 at 16:01
  • Why can't use the implied DOM features of the standards (at least for Microdata?) If you want a unified engine to do this, you'll need the ability to parse HMTL5 and follow the data extraction rules. I can understand why you don't want to invest the energy it takes to do this. But I'd guess hoping for a single tool that does all 3 standards and extracts the data you want is probably not going to get an answer, so now you likely have to choose among less convenient approaches. You could investigate program transformation engines, which support building custom tools. – Ira Baxter Apr 17 '16 at 16:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.