1

Is there a software that allows me to 'connect' my local network computers (say a workstation, with high processing power) with a public hosted server and have them communicate pretty effortlessly?

I have docker containers deployed on a public server instance. This containers have very little processing power. I am looking to deploy some heavy data analytic power for the server without paying for dedicated GPUs. I want to set up a system whereby the workstation can be leveraged for processing if it is available, otherwise use the server.

I am not sure that kubernetes fits this need, although I could be wrong. I keep hearing about openstack, but can not seem to find anything concrete on it.

Any software out that allows this, is open source, and is security minded. By the way, I use Python for programming. I could leverage Python to SSH into the computer, process and return the info to the server, but that does not seem well integrated with my stack (docker containers) and seems more hacky. Something that could be used cleanly with a programming language (in my case python, even it is just json passing like we see will containers, kubelets, etc)

  • I can see why you'd want to do this and you should be able to -- in a better world. In this one, in my experience, people rarely do this, and some of the reasons are security and reliable connectivity. The latter may not matter for batch processing, but the former is very hard to fix, IMO. – Croad Langshan Oct 19 at 23:05
  • @CroadLangshan what is the next best alternative? – David Frick Oct 20 at 0:14
  • I don't know if I misread your question yesterday or I'm just feeling a more optimistic person today. I don't have a solid answer for you, but think were I looking for a solution I'd look for one that did not for example directly expose your workstation as a kubernetes ("k8s") node. Instead I'd split the problem (because security) into two: 1. Special-purpose highly restricted thing to fetch processing jobs (including the data needed to run them) and push job results back. 2. Potentially general-purpose tool to execute the jobs. k8s may be overkill for 2, but I have not used it myself. – Croad Langshan Oct 20 at 19:54
  • Are you saing that k8s might work. I have heard of that so I might look into it. As for saying doing the batch job. I could always set up a python ssh program but then make sure I have to provide email and text confirmation for it to be allowed to connect. I definitely can see what you are saying! – David Frick Oct 20 at 21:17
  • There are many tools "like this" in multiple software categories. I don't have a good overview, but my impression is that k8s has a reputation for complexity and configuration effort appropriate to major websites but not a small project. You might look at something like docs.celeryproject.org and configure it to use maybe sqlite as its backend? – Croad Langshan Oct 22 at 20:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.