Paxmod is Firefox mod/theme with multiple tab rows (and site-dependent tab colors) which I wrote.
Compatible with Firefox 57
It has been tested in latest FF Nightly (currently 61.0a1).
Ideally open source
Sure, it's on GitHub.
Preferably an addon, but if no addon is available then a patch is OK too
It's a WebExtension. However, in order to modify the browser UI it's bundled with an API experiment, an additional unofficial API to apply CSS to the browser UI.
Unfortunately, this means the extension won't be signed by Mozilla and it can't be distributed over the official channels (but you can simply install it from GitHub or build it yourself). Therefore, you also need to run it in either Firefox Developer or Nightly.
As @Izzy noted in a comment, the set of APIs that are available to regular WebExtensions is pretty limited, so you likely won't get UI modifications with any (non-legacy) addon from the store.