Preface
So I'm not one to jump aboard Kubernetes just because it's become really popular. I don't have a business, it's literally just me, so I'm not interested in making things as complicated as possible without any real need.
Now I don't mind learning Kubernetes, but my research has told me it's only for very large companies and that many people are jumping aboard because it's the new hype but I've also read about many people quickly leaving because they realize it's designed for a team of dedicated people and not just for any person.
My issue
I want to start a digital ocean cluster of 5 computers each about 2GB of memory until I can bring in revenue from it. I want to roll out about 10 - 15 websites. Some are dead simple with just a static html page served from NGINX, others are full web apps requiring Redis, MySQL or Mongo, static asset storage, etc... I want to just develop an app and throw it out there to run on the cluster and be able to expand the cluster as needed and not have to worry about extra work and potential downtime for everything as a whole for just deploying a new app or site.
I do know that not everything in the computer world is that simple and I know that there's several moving wheels in a cluster so that's perfectly fine. What I mean is I don't want is to needlessly complicate it if I don't have to.
It's important to note that all of these web sites will be in their own domains. It's not going to be under the same domain or sub-domain or a path in the same domain. I've actually bought domains for all these websites ahead of time and they're already linked up to Digital Ocean waiting for me to sort this out.
Tried Solutions
Docker Compose
My first attempt was to avoid a cluster to keep things simple on one machine and use Docker Compose. But I quickly ran into all kinds of issues that made it clear it was not a good solution. I had to juggle multiple compose files running at the same time and access each others networks which got problematic if a very certain startup and shutdown order wasn't used and I had the complication of having to have a separate router compose file that would reverse proxy to the different compose files since I only had one IP and one port for all the websites.
Docker Swarm
I switched to Docker Swarm so that each stack could be managed separately and this was ok for the most part, a bit of hacking around to get data storage right given I run as non-root but I just bind mounted to folders and told swarm to keep it tied to that machine.
The biggest issue I had was still the routing issue and swarm amplified this. The entire cluster only had one IP and one port (Unless I were to spin-up individual nodes for each website and tie them to a host network which defeats the point of even doing all this). So I still had to create a separate image that would configure a reverse proxy to listen to arbitrary ports and reverse proxy to the container. This got even more complicated with SSL and I ran into all kinds of proxying issues.
So I would have to deploy a webapp with a webserver then I would have to go and modify my router image for the new update and deploy that. There was a high chance I'd have to go through several iterations of uploading to a private Docker Hub since swarm doesn't build images which is understandable and then download it to the swarm and see if it works or not.
If I wasn't careful odds are high even when deploying to production the router could fail and there goes every single website and webapp I have running across the whole cluster since it all has to go through the Router. Testing locally first before deploying was incredibly hard for the router because changing a few environment variables wasn't enough. The router almost had to be completely different for a local dev because it's specific to the cluster it's deployed to which feels counter productive to the philosophy behind a container which depends on portability and self-containment.
I'm aware of NGINX router images that work with Docker Sockets
These are images people have created that auto-provision a router based on communication from the Docker process. But not only are these either dead or have many hundreds of open issues but it feels very hacky and doesn't feel like it'd be ok for production especially with such a high issue count. Then it also seems like it'd give me more complications over the reverse proxy details and if SSL certificates were involved and more-so over multiple domains and sub-domains.
I'm seriously considering Kubernetes
I've been avoiding it like the plague!!! But it's looking more and more like the only option for me. I've heard Kubernetes will assign each pod it's own IP meaning I don't have to deal with the BS from a router that reverse proxies to my whole entire cluster and that's so specific to my cluster I have to try and figure out inaccurate ways to creatively test it outside of my cluster before deploying to my cluster where it still may fail in which case everything on my cluster becomes inaccessible externally. It's own IP address will make things so much incredibly better.
But more so it seems like Kubernetes is really integrated deep into Digital Ocean and that could be another huge saver rather than trying to install Docker on a regular linux container and setup a swarm and setup everything myself manually from scratch and then have to deal with creating a cluster specific router.
