I am looking for a music notation software which offers an API to code additional music. For example Id like to write a melody in the GUI and code an algorithm which dynamically plays along with the melody.
If you prefer the GUI part in your question, I would recommend MuseScore. It is Open Source and has a Plugin System, so you may extend it to your needs. MuseScore is based on Qt.
However, if you are looking for an even more programmable approach, you should check LilyPond. This is kind of LaTeX for music notation.