I'm looking at a challenging design problem. I think I could invent a way to solve it, but I want to see if there's any existing libraries out there so that I don't have to reinvent the wheel.
We're running a server, written in Java, that processes large numbers of requests. The server is running on a fleet of hosts, and there's a thread pool on each host. Processing each request consists of executing steps S1, S2, S3, ..., Sn. Not every step is needed for all requests, and which steps are needed can't be determined up front--that is, the result of Sk may determine whether some later steps are needed.
The issue is that certain steps (let's say S2 and S4) require significant delays--possibly as much as an hour. (In our case, they call downstream services that put low limits on the request rate. We could also imagine a case where they call downstream services that simply take a very long time to complete.) What is happening is that if we simply let the thread wait until it can continue, we quickly reach a point where all threads on all hosts are blocked, and then no requests can be processed. Since a large fraction of the requests do not need to execute S2 and S4, this is a problem because those requests, which could be processed very quickly, cannot start.
I've been thinking about possible solutions, such as having a Java abstract class to define a "step", and some kind of controller that has a list of steps, and will execute a sequence of steps on a thread until it hits a blocking point, at which point it would relinquish the thread, and the controller would either hold onto the context information in an in-memory queue so that it can restart the sequence on some other thread when it becomes unblocked; it's also possible that there might be too many requests even for an in-memory queue, so an external queue might be needed. I can imagine a few possible architectures, but I haven't tried to devise any details for any of them yet.
But before I do, I'd like to know if there's some off-the-shelf Java library to solve this sort of problem.
(I'm not looking for a rate limiter; we already have leads on that. I'm looking for a way to manage the workflow in a way that doesn't block threads when a rate limiter is in use or other very long delays are possible.)