11

Goal

I am looking for an Open Source Java Solution that can be used in a small (2-4) cluster of linux machines. You can think on this like a farm of processing worker servers that are just listening to a message from a JMS endpoint to start processing.

Requirements

This library/solution/whatever must be able to trigger around 10~20 processes in each machine from the cluster (each one is a JVM). Each process will consume a message from a centralized JMS instance and will save the job results in a centalized DBMS instance. Each process takes several minutes (5 to 50 mins) to complete and has little footprint in terms of network, disk IO, CPU and memory usage. Each job is independent. The library must just help to manage this allocation/deallocation of JVM processes and provide some minimal statistics and control. It's not necessary to pause/resume/cancel jobs. I just need to know when they're running or not, and if it completed successfully or not. Keeping idle worker servers is not a problem.

Important: I am not looking for PaaS or any cloud-based solution.

What do I know

Initially I've considered the idea of just start a bunch of tomcat instances, but it seems overkill and I'd have to provide each one of them different ports. It's not a divide and conquer problem, so I am not looking for map-reduce solutions. It's also not something to be solved using hadoop (I guess). But I confess I know little about this kind of solutions. I've read a little about JavaSpaces and RMI, but it seems these are building blocks for distributed solutions. I've also heard about microservices, but they just look like something more useful for orchestration of different parts of a whole process. I've also checked memcache, hazelcast, terracota, but they're intended to solve a different class of problem.

My Feeling

is that this is some sort of a well known problem with several insteresting solutions, but I just don't know exactly how it's called (and then I can't google it properly).

Improve this question
3
  • 1
    Are't Apache YARN, Mesos etc. trying to do exactly this (and a bit more: manage resource allocation)?
    – Has QUIT--Anony-Mousse
    Commented Jan 4, 2015 at 1:09
  • @Anony-Mousse gonna take a look on that, thanks for the heads up!
    – Leo
    Commented Jan 5, 2015 at 0:33
  • Isn't Jenkins or Apache Airflow what you're looking for?
    – Marian Paździoch
    Commented Jul 10 at 12:51

2 Answers 2

Reset to default
2

I don't know of a ready made solution for this problem (and no fancy name) but would rather set it up myself (in Java). Also my JMS skills are "in development" so there might be better solutions, that combine parts 1&2. And I'm not quite sure if I understood your problem correctly.

I assume the workers handle their Database connections themselves so I wont account for it.

First part: The distributor - A message driven bean that consumes your JMS Messages and handles them. As you don't want an Application server on all machines you now only need one.

Second part: The frontworker - A Java Program that runs on each machine and holds open a port for communication from the distributor. They do need some exchange format, RMI is the straightest solution in my experience for this.

Third part: The worker - Gets started by the frontworker. They are on the same machine so, meh. All information the worker needs is provided by the frontworker in some fashion (Database, console, file, whatever.) The workers insert into the database when they are started, stopped and failed.

Last part: The monitor - reads the Database. The data is displayed with a simple Table. Fancy Reports maybe via JasperReports.

Dataflow would be like this: the JMS data provider sends its messages which are consumed by the distributor. The distributor optionally checks which server has the fewest workers running currently or does just a round-robin. It then opens the RMI connection to the frontworker on this specific server and hands over the JMS information. The frontworker starts a Worker Process with the information. Each worker enters his data to the database independently of each other.

The monitor would be used independently of this and just read the database of the work the workers do.

Improve this answer
1
  • Hi Angelo, it makes a lot of sense. After doing some research, I think I'm going to use a strategy like that, but merging the distributor and the frontworker, because the frontworker can read from the JMS directly. The part I am specially interested in your idea is how the frontworker would start the worker. ProcessBuilder? Thanks for your help.
    – Leo
    Commented Oct 21, 2014 at 19:30
2

I would use the Quartz scheduler for this.

I have used it successfully in the past, and apparently it has a cluster mode (which I haven't tried). It performs load balancing and can use any JDBC database for coordination.

It is open source, written in Java.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.