What I am looking for is a library -- or alternatively a CLI tool -- that allows me to sign the "PE hash" along with time-stamping etc., but without having to transmit the full file (which can be several GiB in size) between the signing server and the client. The idea would be to transmit some metadata along with the hash to the signing server, then receive back a valid signature blob and some metadata again, which the client-side would tack onto the previously unsigned binary file.
Preferably it should work both for Linux and Windows as a minimum requirement. Meaning it should not rely on system-specific functionality from either OS.
osslsigncode (under GPL) already does most if not all of what
signtool (proprietary, distributed with Windows SDKs) can do in terms of AuthentiCode code-signing.
However, due to the more stringent requirements placed on CAs by the CA/B forum, it becomes desirable to have a single protected machine doing the actual signing -- a signing server, if you will.
osslsigncode -- and all other tools I am aware of -- requires the full binary for signing. But technically it should be enough to sign just the "PE hash". That's the hash -- using a cryptographic hashing algorithm such as SHA-1 or SHA-2 (256 bit) -- over the contents of the PE file up to the "image size" and excluding the checksum field and the security directory from the PE header.
This differs from the file hash, which take into account the full file contents. The term "PE hash" is borrowed from Microsoft's