Re: [infinispan-dev] Improving the performance of index writers

On 20 October 2014 12:59, Emmanuel Bernard <emmanuel(a)hibernate.org&gt; wrote: > HSEARCH-1699 looks good. A few comments. > > Maybe from a user point of you we want to expose the number of ms the user is ok to delay a commit due to indexing. Which would mean that you can wait up to that number before calling it a day and emptying the queue. The big question I have which you elude too is whether this mechanism should have some kind of back pressure mechanism by also caping the queue size. Gustavo is implementing that for the ASYNC backend, but the SYNC backend will always block the user thread until the commit is done (and some commit is going to be done ASAP). About to write a mail to hibernate-dev to discuss the ASYNC backend property name and exact semantics.

> > BTW, in the following paragraph, either you lost me or you are talking non sense: > >> Systems with an high degree of parallelism will benefit from this, and the performance should converge to the performance you would have without every doing a commit; however if the frequency of commits is apparoching to zero, it also means that the average latency of each operation will get significantly higher. Still, in such situations assuming we are for example stacking up a million changesets between each commit, that implies this solution would be approximately a million times faster than the existing design (A million would not be realistic of course as it implies a million of parallel requests). > > I think you can only converge to an average of 1/2 * (commit + configured delay time) latency wise. I am assuming latency is what people are interested in, not the average CPU / memory load of indexing. I'm sorry I'm confused. There is no configured delay time for the SYNC backend discussed on HSEARCH-1699, are you talking about the Async one? But my paragraph above is strictly referring tot the strategy meant to be applied for the Sync one.