There is too unclear situation for beginner - what is propriate for my case please?

I have few PCs (Linux) with 4-8 cores each. All ssh accessible on 1 GB network. My function will contain a call to some previously compiled code, like ./work < input_file > output_file. One evaluation can take a minute or even much more, I can run it several times in parallel.

I can see jug, scoop, pp, pathos? In the lists there are tens of packages and some may be too old. Only thing I know is I want to use new python - 3.5. Thank you. No answer on similar q: Parallel execution library for Python


I went through several packages and pathos seems to work with my python 3.5.

from pathos.parallel import ParallelPool as Pool
import time
pool = Pool()

def host(id):
    import socket
    from math import sin
    for i in range(1,1000):
        for j in range(1,10000):
    return "Rank: %d -- %s. %f" % (id, socket.gethostname(),sum+id)

pool.ncpus = 4
print("Evaluate on "+str(pool.ncpus)+" cpus")
pool.servers = ('localhost:5653','localhost:5654', 'Fil:5653')

res5 = pool.amap( host, range(16) )
while not res5.ready():
         print(stats()  )
print(  "\n".join(res5.get()) )

Provided I run ppserver.py -p 5653 -w 8 -t 300 2x locally and on Fil node

If there is a better answer, give it please.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.