3

Is there a way to extract, from a PDF, the highlighted passages with the corresponding page numbers from the command line, on Linux? I found two tools, but they do not exactly fit my needs: pdf-highlights-extractor allows me to extract the highlighted passages with the page numbers, but it does not have a command line interface, only a graphical interface. And DyAnnotationExtractor has a command line interface, but only gives me the highlighted passages, not the page numbers. Is there a tool that can do what I need?

Thank you in advance for your help!

8
  • 1
    Do you need a command line interface so that you can parse the output? or maybe to call it from scripts?
    – golimar
    Commented Jul 7, 2022 at 11:37
  • @golimar: Both: I would like to call the command from a script and then further process the output.
    – Wolfhart
    Commented Jul 7, 2022 at 17:00
  • 1
    For the first option, many times Java GUI applications look GUI only but they also support command line arguments. Or you could call the Java class that processes the PDF file instead of calling the GUI class.
    – golimar
    Commented Jul 7, 2022 at 18:20
  • 1
    As a last resort, since both options are hosted in open source sites, you can open a ticket or issue asking for that feature, or check the source code to see if you can modify it
    – golimar
    Commented Jul 7, 2022 at 18:21
  • 1
    Are you interested in some custom Python or Go code to handle this? I've got some solutions in Python and Go.
    – Zach Young
    Commented Jul 8, 2022 at 2:05

2 Answers 2

3

Full disclosure: I'm being paid as a consultant by UniDOC/UniPDF to answer this question and spread good will towards PDFs, Go, and UniPDF.

I've written two command-line programs that are equivalent. They both read some number of PDFs that you list, locate highlighted text per page, and then print out CSV of the PDF, page number, and the highlighted text.

For both the PyMuPDF (Python) and the UniPDF (Go) code examples, I'll use these two reference PDFs:

one_page.pdf two_pages.pdf
page 1 enter image description here enter image description here
page 2 enter image description here

main.py

./main.py one_page.pdf two_pages.pdf 
Filename,Page_num,Highlighted_text
one_page.pdf,1,word1
one_page.pdf,1,d2 word3 wo
two_pages.pdf,1,"“and what is the use of a book,” thought Alice “without pictures or conversations?”"
two_pages.pdf,2,when suddenly a White Rabbit with pink eyes ran close by her.

and

main.go

go run main.go one_page.pdf two_pages.pdf 
Filename,Page_num,Highlighted_text
one_page.pdf,1,word1
one_page.pdf,1,d2 word3 wor
two_pages.pdf,1,"“and what is the use of a book,” thought Alice “without pictures or conversations?”"
two_pages.pdf,2,when suddenly a White Rabbit with pink eyes ran close by her.

The Python program uses the PyMuPDF library, which is being actively maintained and updated. Consult the docs for getting the library installed, PyMuPDF: Installation.

The Go program uses the UniPDF library, which has a freemium tier where 100 document reads/writes are free per month:

Both programs also have a VISUALIZE switch in the source code that you can turn on to visualize what the highlight rectangles look like, which can be very handy if you ever start getting weird results. The highlights don't contain the text, they're just graphical objects that are drawn over text. Both programs get the highlight rectangles, then use other APIs in their respective libraries to query a page for text in the rectangle, but the different libraries have different ideas about what text is inside or outside of a particular highlighted region (rectangle):

  • For PyMuPDF I had to grow the highlight rectangle just ever so slightly to get any text.

  • For UniPDF I had to shrink the highlight rectangle vertically, and substantially, to avoid extraneous text (outside of the intended highlight region).

If you're new to using Go, you can download Go for your system (Linux) and then you can run the program interactively with go run main.go pdf1 ..., or you can build an executable and install it on your path and then call it like any command-line utility:

go build -o listhltext main.go
cp listhltext [SOMEWHERE ON YOUR PATH]
10
  • 1
    Change that to page.first_annot... you got a newer version of PyMuPDF than me. I'll work to update it this weekend.
    – Zach Young
    Commented Jul 9, 2022 at 5:17
  • 1
    Oh, try adjusting the SCALE_H and SCALE_W constants. If you’re getting no text, the highlight rectangle might be too small. Change those values to 2.0, just to prove you can get some text, then dial them back down till you get the correct text.
    – Zach Young
    Commented Jul 9, 2022 at 17:06
  • 1
    I fixed page.first_annot and added command-line args to the Python prog. Do a git pull or re-download the project. Now, you'll need to supply the scale factors with --scale-w and --scale-h. I used 1.001 and 1.001 to get the Alice text correct. You'll need to play around with those values for different PDFs (because there's no standard that says how the overlayed rectangle relates to the text underneath it).
    – Zach Young
    Commented Jul 9, 2022 at 18:31
  • 1
    Thank you! As you suggested, I played around with the SCALE_X and SCALE_Y values. I found that I could leave the X value at 1.0 and modify only the Y value. I found the following: PDF1 worked with the original value (as already mentioned). PDF2 required 1.2. PDF3 required 1.6. So I will now use the program with 1.6 and see whether this value works for all my PDFs.
    – Wolfhart
    Commented Jul 9, 2022 at 19:31
  • 1
    I think you’re seeing CSV quoting, and that’s not part of the actual result. When you open the CSV in a spreadsheet app do you still see extraneous quotes?
    – Zach Young
    Commented Jul 9, 2022 at 19:55
3

I would recommend usage of the nifty little Python library pdfannots, which has the very capability you are looking for.

$ pdfannots sample.pdf

If combined with some other Bash commands, it can produce nicely formatted output. For example:

$ pdfannots sample.pdf --no-condense | \
# Removing duplicate lines:
cat -n | sort -uk2 | sort -nk1 | cut -f2- | \
# Improving output formatting:
awk '{$1=$1};1' | sed 's/^\(> \)//g' | sed 's/* Page #/\n&/'

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.