Take a look at Team Mates:
TEAMMATES is an online peer evaluation and feedback system for
students. It is free for use, and all you require is to request for an
account from us. With TEAMMATES you can,
Create new courses and easily upload student details. Create sessions.
There are two types of sessions:
(a) Team peer evaluation sessions with fixed questions: These come
with preset questions optimized to measure contribution of team
members and give peer feedback to team members.
(b) Feedback sessions with customizable questions: With these, you can
set your own questions and set visibility (who can see the responses)
and anonymity level per question. This is suitable for other feedback
paths such as peer feedback between teams and for feedback from
instructors to students.
See here for more details about different session types.
View reports of responses, and moderate them before publishing the
responses for students to view.
It also includes the option of peer review within teams or courses.
Not sure about it being fully open source it is free, already does a lot of what you are asking for and they are open to contributions and enhancements to the system and the source code is available on-line you would have to look into the licence and hosting options.