Cross-platform arbitrary code execution tool for research

Question

I'm working on an academic research topic that requires quite a lot of computation. I'm building the software in C++. When I'm at the university, I will have access to a beefy server, but for my time at home, I would like to execute my code on my Windows gaming PC, which is quite a bit faster than my 5yo MacBook Pro.

So, is there a tool that allows me to take a Mach-O binary (OS X binary) from memory, move it to the Windows PC over the network, and continue execution there. I believe this might be possible, since:

• I'm only relying on header-only libraries (STL, Eigen).
• Not doing multithreading.
• No OS X-specific calls are made (no syscalls)

I think of it as: x86 is x86: just copy the code in memory as-is. On the target machine, allocate an identical virtual address space, transfer x86 processor state (using pushf and popf instructions), and resume computations.

If something like this exists, it would probably involve linking against a library that has two calls like:

• moveToSlave(): pause local computation, and move the entire binary to the other machine.
• moveToMaster(): pause remote execution (which is local on the remote machine), and transfer the updated memory state back to the master (my laptop).

I know it's a long shot, and such software would be highly experimental.

Answer 1

First off compiled C code from one OS, (Windows), is unlikely to be recognised on another (MacOS). Not because of the OP Codes, which will be the same if the processor is the same, but because of the structure of the executable file and also the memory map. Also there are more OS calls than you might realise, e.g. malloc/new will have some OS interaction.

For a lot of this sort of work Python is often used, (with C or Cython extensions for the heavy lifting), partly because of OS independence. It is also excellent for parallel processing including distributed processing across a network.

If you can restructure your processing into a number of parallel tasks and use the pub/sub model you can have any number of nodes, (from 1 upwards), working on your problem and, if it is correctly structured, you will be able to loose a node at any point in time with only the processing of that part of the model being lost and needing to be re-done. Thus if you need to switch machines you would start the new machine before stopping the old one, give it some time to get synchronised on the state of your processing and then power down the old machine.

This is a very large topic and there are a few books and a lot of academic papers on the subject.

To finally get to some specific software for this sort of problem one of the common tools to use is NSQ.

• Cross Platform
• Scale-able - if you have more platforms available at once your processing will be done faster
• Fast
• Support for multiple processing languages including C, Python and Go.

But you will have to restructure your problem and code to make it suitable.

Answer 2

As @JawadAlShaikh suggested, using a virtual machine for the actual crunching might work. However, I just checked, VirtualBox will discard saved state if you try to export the vm.

What may work is to set up VirtualBox on each host machine so that it is pointing to external media (external hard drive via USB3 or eSATA) to store the actual VMs and config info on each machine. Plug drive in, boot machine, run VBox, resume saved state and carry on computing.

A second choice might be to use a "remote" server - either your PC at home or one hosted online by either one of the big cloud providers or one of the smaller ones like linode.com - this way you can work on it from anywhere and your local hardware doesn't matter. Run small scale for development work, then when you need to crunch a large data set upsize the machine and go for it. When data crunch run done, size it back down.