Discussing with Jeremy Whiting, we agreed that having an Optimizer which replenishes its pool in a dedicated thread is indeed a good idea but that would probably be better off as an alternative Optimizer implementation.

People having high wait times on this Optimizer could probably minimize the impact by configuring an higher amount for incrementSize.

I'm going to resolve this issue by improving the time spent in the locked block in the case of the fast path (when the pool is not empty).