I am looking for a PDF viewer that allows annotations that are stored in an separate file, ideally in an ASCII format that works well when merged by git. Ideally the viewer works on Android, Linux and Mac's.
For Android, I can warmly recommend Moon+ Reader Pro (for PDF, you will need the paid version). (One of) the fastest PDF readers, well supported – I use it myself for years. As a side-effect, you'll have the same features for many other eBook formats.
Annotations (and highlights) are saved into the PDF file directly (so you see them with other readers as well). Additionally, they are saved into a local database which can be exported into a text file (which Git should be able to deal with fine).
Highlights and Annotations, Network libraries in Moon+ (source: Google Play; click images for larger variants)
Plenty of other goodies in there, e.g. direct access to many net libraries, day/night modes, and more. Currently it's USD 5, but Seany often makes discount actions (I wouldn't wonder if there'd be an XMas action soon).