2

I'm using wget at the moment to retrieve a large set of data from a REST API and it is paginated as per RFC 5988, i.e. the body text contains a limited number of records, and the HTTP response header contains a line Link: <https://foo.com/bar?abcdefgh>; rel="next", where the link retrieves the next page of the data set and so on, until the last page which doesn't have a Link: header.

Is there a pre-existing tool that will parse this and keep pulling down the pieces to assemble the full set as one file?


Of course it would be possible to write a shell script or similar to do this by repeated calls to wget or curl and parsing header etc. but it seems to me this would be a fairly common task and I couldn't find any existing tool by googling.

0

In the meantime I managed to get a bash script working, for the particular server I'm trying to read anyway:

#!/bin/bash

function CURL {
    curl -D headers.tmp $1
}

LINK=$1

while [ ! -e $LINK ]
do
    CURL $LINK
    LINK="$(sed -n 's/.*<\([^>]*\)>; rel="next"/\1/p' headers.tmp)"
done

Usage example: ./scriptname https://foo.com/api/v2/bar.json >bar.json

The sed command has to search for rel="next" because the server might send both prev and next links.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.