Command line tool to download via HTTP documents paginated by RFC 5988

Question

I'm using wget at the moment to retrieve a large set of data from a REST API and it is paginated as per RFC 5988, i.e. the body text contains a limited number of records, and the HTTP response header contains a line Link: <https://foo.com/bar?abcdefgh>; rel="next", where the link retrieves the next page of the data set and so on, until the last page which doesn't have a Link: header.

Is there a pre-existing tool that will parse this and keep pulling down the pieces to assemble the full set as one file?

---

Of course it would be possible to write a shell script or similar to do this by repeated calls to wget or curl and parsing header etc. but it seems to me this would be a fairly common task and I couldn't find any existing tool by googling.

Answer 1

In the meantime I managed to get a bash script working, for the particular server I'm trying to read anyway:

#!/bin/bash

function CURL {
    curl -D headers.tmp $1
}

LINK=$1

while [ ! -e $LINK ]
do
    CURL $LINK
    LINK="$(sed -n 's/.*<\([^>]*\)>; rel="next"/\1/p' headers.tmp)"
done

Usage example: ./scriptname https://foo.com/api/v2/bar.json >bar.json

The sed command has to search for rel="next" because the server might send both prev and next links.