Hi Mircea, Seeing Radim's mail makes me think we need a new event. We already have a TopologyChangedEvent but this one is only useful to find out when a new CH starts to be installed and when installation ends. With NBST actual state transfer still continues a while after this and there is no programmatic way to tell when it ends (except for some very ugly polling for pendingCh == null). Wouldn't it make sense to introduce a new event to signal end of state transfer? I think for performance testing we should not rely on log analysis because the logging itself slows everything down quite a lot and has an undesirable impact on the results. We already have the polling I mentioned in our unit tests in TestingUtil.waitForRehashToComplete(). Removing those busy polling loops might even make the tests run faster. Adrian On 10/12/2012 02:52 PM, Radim Vansa wrote:

Rather than (or in addition to) using log parsing, how about using Byteman to inject aspects measuring things such as join time, state transfer time, view installation time, state transfer time / state size and so on ? On 10/12/12 1:52 PM, Radim Vansa wrote: -- Bela Ban, JGroups lead (http://www.jgroups.org)

On Fri, Oct 12, 2012 at 7:05 PM, Sanne Grinovero <sanne(a)infinispan.org&gt;wrote: I have been pointing out that the goal should not to be to minimize Yep, this is indeed the goal of NBST. Mircea's email is a bit confusing, he starts by saying we try to minimize the impact on throughput during state transfer but then he switches to how long state transfer takes. Because not all nodes receive the rebalance start event at the same time, and because the forwarding of commands to new owners is not optimal, I expect to see a visible drop in throughput during state transfer. Maybe we should use something like "total drop in throughput compared to the initial cluster from the moment a node joined to the moment it became part of the read consistent hash (i.e. the joiner finished receiving state + we finished our bookkeeping on the coordinator)". I'm not sure how easy it would be to do this, but I'd also like to see how old transactions (with topologyId < current topology id) affect performance, because they need special handling vis-a-vis locking. So perhaps we can also measure the throughput in the post-join cluster while TransactionTable.minTxViewId (soon to be called minTxTopologyId) < StateConsumerImpl.cacheTopology.topologyId and compare it to the throughput in the cluster after everything has settled. We don't do this (yet). Actually the part about interrupting state transfers seemed the most tricky part in the "tree of cache views" NBST design, so in the last version we don't interrupt state transfer at all. In order to really minimize transfers in such a design you'd have to compute the consistent hash based on the previous consistent hash *and* on what segments the new owners have already received - otherwise each incremental change could undo all the progress made during the previous state transfer. But I don't think we'll do that very soon... Our solution at the moment is to ignore any joins or leaves during state transfer, and only start a new state transfer once the current one is done. This is possible because when the primary owner of a segment leaves the first backup automatically becomes the new primary owner, so transactions can go on without any state transfer. In the end, I think the best solution to avoid wasted transfers is to allow the adiministrator to suspend rebalancing for a period of time. We do have this, but if a second node leaves while we're still transferring state, the user had better configured numOwners > 2... The response time is definitely not going to be the same during state transfers, except maybe for reads. But we do need to measure how bad it is. Cheers Dan

On 12 October 2012 17:07, Mircea Markus <mircea.markus(a)jboss.com&gt; wrote: +1, in the longer term. With Byteman anyone can start measuring it now without needing to wait for new features to be developed, plus who drives this (Radim?) can very easily change the interception points to measure different aspects. We can then see what are the most interesting metrics after some experience and define them as typed events. Sanne