Please note that I've not tried any of these so I can't be sure that they will do what you want.
I think that calcurse might do the job.
ical - An X based Calendar Program - OK so needs x-windows, not so good for your requirement\
NOTE: The ical software does NOT support the iCalendar format!
These might do it but not clear if they support iCal:
https://www.sococo.com/ is built on similar ideas, though it is designed for work and classes, not for meetups: there is no continuous space where people walk around with others' voices fading when you walk away, there are rooms, and people can jump from room to room to check what's the topic of discussion in another room. In our case when we tried Sococo ...
There are various websites that allow you to make a 'room' and watch youtube videos together with your 'room' participants. There is full control over the video in these rooms. This would require you to put these video files on youtube though, and that might not be ideal. It is, however, free and relatively easy to do.
Rabit and Gaze are examples of this.