Will have to be split into multiple tickets. To do:
- Check with async experts (Clément?) that implementing our own publisher/subscriber is not a bad idea (not the reactive map/filter/etc. bits, just the raw publisher/subscriber)
- Implement/expose result scrolling as a createPublisher operation so that users can wrap that with their reactive library.
- Implement mass indexing with a publisher/subscriber pattern
- Re-consider the default size of the work queues in the Elasticsearch backend; we essentially need the queue to be at least equal to (estimated number of user threads)*(estimated number of works created by each thread).
- Provide a way for users to customize the size of the work queues; allow to set it to "unlimited" for people who'd rather get an OOM error.
- Document how users can determine the ideal size of their work queues
- Make queueing of works throw an exception instead of blocking when the queue is full. Maybe allow the client to choose between failing and blocking? It might make sense to block if the client will block anyway.
- Implement cancellation of works simply by allowing the user to call cancel() on the returned future. In the work consumer thread, if the future is complete with a CancellationException, we'll just ignore the corresponding work. If it's not, we'll send the work to the REST client and add an exception handler (.exceptionally(...) to the user-returned future: if the future is completed with a CancellationException, we'll cancel the HTTP request (is it possible?).
|