I'm looking at a challenging design problem. I think I could invent a way to solve it, but I want to see if there's any existing libraries out there so that I don't have to reinvent the wheel.

We're running a server, written in Java, that processes large numbers of requests. The server is running on a fleet of hosts, and there's a thread pool on each host. Processing each request consists of executing steps S1, S2, S3, ..., Sn. Not every step is needed for all requests, and which steps are needed can't be determined up front--that is, the result of Sk may determine whether some later steps are needed.

The issue is that certain steps (let's say S2 and S4) require significant delays--possibly as much as an hour. (In our case, they call downstream services that put low limits on the request rate. We could also imagine a case where they call downstream services that simply take a very long time to complete.) What is happening is that if we simply let the thread wait until it can continue, we quickly reach a point where all threads on all hosts are blocked, and then no requests can be processed. Since a large fraction of the requests do not need to execute S2 and S4, this is a problem because those requests, which could be processed very quickly, cannot start.

I've been thinking about possible solutions, such as having a Java abstract class to define a "step", and some kind of controller that has a list of steps, and will execute a sequence of steps on a thread until it hits a blocking point, at which point it would relinquish the thread, and the controller would either hold onto the context information in an in-memory queue so that it can restart the sequence on some other thread when it becomes unblocked; it's also possible that there might be too many requests even for an in-memory queue, so an external queue might be needed. I can imagine a few possible architectures, but I haven't tried to devise any details for any of them yet.

But before I do, I'd like to know if there's some off-the-shelf Java library to solve this sort of problem.

(I'm not looking for a rate limiter; we already have leads on that. I'm looking for a way to manage the workflow in a way that doesn't block threads when a rate limiter is in use or other very long delays are possible.)

1 Answer 1


How is the thread currently blocked? Are you in a loop waiting for some information to come? Are you checking on some information on a regular basis until it changes its state?

From what I understand I would suggest an open source scheduler.

  • Our current implementation uses Guava RateLimiter. Its acquire() method blocks until a permit is available, but I have no idea how it does the blocking. Thanks for the link, I'll take a look.
    – ajb
    Jul 7, 2017 at 3:30
  • I'm not sure to fully understand the function but I'll look into it, you got me interested. On the other hand, I had a similar issue using an external service. The PEPS platform is doing processing in the background and I'm using a loop in ActiveEon's solution to ask on a regular basis when the file is available. In that case, my resources are free between the different requests and can be used for other purposes.
    – XYZ123
    Jul 7, 2017 at 16:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.