WebRTC is, by design, necessarily end-to-end encrypted. And there are dozens of free WebRTC-powered video chat websites available, many for free.
From what I understand this could be accomplished using WebRTC and thus wouldn't even really require a server because everything could happen peer-to-peer.
It's actually not that simple. If the users are behind a NAT or you want it to scale beyond ~4 people, then you'll need to throw up some STUN or SFU or similar servers.
This gets tricky if the server you use terminates the WebRTC on their own server (eg using a Multipoint Conferencing Unit (MCU) or Selective Forwarding Unit (SFU)), which it may do for scalability (mesh WebRTC connections tend to break down with more than ~4 people without some MCU or SFU)
Probably the most popular web-based WebRTC service that provides e2ee that scales is Jitsi. You can use their free service or host your own jitsi meet instance (it's free software):
You can also build your own service using the jitsi video bridge and their other components.
You may also want to look into Janus, which has a less mature (UX) set of WebRTC services, but might be more useful if you actually want to build your own service that just uses its libraries