Hi, Some feedback has come saying that it'd be nice to be able to configure externalizers using annotations. Now, in a previous discussion (http://lists.jboss.org/pipermail/infinispan-dev/2010-December/007047.html) it was agreed that for framework developers this is not nice since it makes it hard to abstract the Infinispan layer, but end users might be interested in using annotations rather than having to implement getId(), getTypeClasses() in Externalizer interface - see http://community.jboss.org/docs/DOC-16198 To be able to support this, @Marshallable annotation that would be used wth externalizer implementations would have to be brought back with id and typeClasses attributes. That'd make it a 3rd way to define ids, after XML and getId() implementations. Clearly, getId() and getTypeClasses() would be moved to a different interface, so that people that chose to use @Marshallable could just provide read/writeObject method implementations. The gain from having end users use @Marshallable is not that great IMO cos we don't do annotation scanning, so there would still be a need to register externalizers.

more advanced trivial garbage collection

> > Is it to do with classloader leaks? If so, a weak map could be used for this table... You misunderstood what I meant. By annotation scanning I meant that we won't be scanning the entire classpath for user classes that have @Marshaller, right? I mean, we don't do like an EJB container that goes through deployments checking for all @EJB classes, cos that would be very expensive and it's not our business. So, back to my question, the users will still need to give us the list of user defined externalizers. Sure, we can do annotation scanning on that specific list of classes.

My point is that it might look nice that users can use annotations to define some externalizer properties (id and typeClasses) but they're not getting as much out of it as they'd with @EJBs in an EJB container cos they have to list the externalizers somehow. Again, unless you wanna get into classpath annotation search...

> > Ah I see what you mean. Initially I thought you'd only need to > register the externaliser when you first encounter a new type (i.e., > when a user defined type is first encountered with a put()), but this > may not be the case since on the remote node it may see the a new > magic number which may not be registered, and then you have a > problem. Marshalling will send Externalizers across the wire too, so the receiving side need not know about it in advance. The sending side can then do this kind of discovery safely. In addition, JBMAR already has the @org.jboss.marshalling.Externalize annotation which does just this. Unless you're talking about your own mechanism, anyway. Just thought I'd throw this bit out there.

On Mar 17, 2011, at 3:22 PM, Manik Surtani wrote: > > On 17 Mar 2011, at 14:16, David M. Lloyd wrote: > >>> >>> Ah I see what you mean.  Initially I thought you'd only need to >>> register the externaliser when you first encounter a new type (i.e., >>> when a user defined type is first encountered with a put()), but this >>> may not be the case since on the remote node it may see the a new >>> magic number which may not be registered, and then you have a >>> problem. >> >> Marshalling will send Externalizers across the wire too, so the >> receiving side need not know about it in advance.  The sending side can >> then do this kind of discovery safely.  In addition, JBMAR already has >> the @org.jboss.marshalling.Externalize annotation which does just this. >> >> Unless you're talking about your own mechanism, anyway.  Just thought >> I'd throw this bit out there. > > Sweet.  I think we can make this work then - even reusing JBM's annotations. Hmmm, I wasn't aware of that either. I'll defo look into that for 5.0 - https://issues.jboss.org/browse/ISPN-986 @Eduardo, this might solve your cache startup issue and the current need to define Externalizers in advance. > > -- > Manik Surtani > manik(a)jboss.org > twitter.com/maniksurtani > > Lead, Infinispan > http://www.infinispan.org > > > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev -- Galder Zamarreño Sr. Software Engineer Infinispan, JBoss Cache _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

> Initially we were seeing lots of little clusters form, increasing the number of initial members in the PING section of the jgroups-udp.xml file and increasing the > timeouts seems to have made things better but not great. Even at 20 nodes, we are seeing messages where the coordinator is failing to flush. I do *not* recommend FLUSH in clusters bigger than 10-15 nodes ! FLUSH will definitely kill you if you have clusters of 100s of nodes. IIRC, Infinispan requires FLUSH, but I think this was removed in 4.2.1.FINAL. Maybe Manik or Vladimir can comment ?

> It does not appear that the default configuration scales to a large size. Correct. The default config is geared toward a 4-16 node cluster.

Bela, Manik, Thanks for responding. Will ISPN-83 be included in 4.2.1.FINAL? Yes, a large cluster jgroups config would be great. At this point we have taken the UDP config distributed with 4.2.0 and increased all the timeouts. It took about 30 minutes to get to 150 nodes. The cluster appears stable once its up, the problem is in the startup. Based on what I am seeing and have read in the documentation, every time a node wants to join a FLUSH is sent across the entire system and then a new view is created. We are seeing nodes wait minutes to get the new view. It has been mentioned before that Infinispan was tested at 1000 nodes and there may be a couple of places where Infinispan is in use at the several hundred node size. Can we get a copy of the configuration files for these systems?

Date: Fri, 18 Mar 2011 16:30:46 +0100 From: bban(a)redhat.com To: infinispan-dev(a)lists.jboss.org Subject: Re: [infinispan-dev] Infinispan Large Scale support On 3/18/11 2:19 PM, david marion wrote: > > Bela, Manik, > > Thanks for responding. Will ISPN-83 be included in 4.2.1.FINAL? Yes, a large cluster jgroups config would be great. At this point we have taken the UDP config distributed with 4.2.0 and increased all the timeouts. That's certainly not the way to do it ! If you post your config over on the JGroups mailing list [1], I'll take a look and suggest modifications. If we work on this a bit to get your cluster going, this config could serve as the basis of a large cluster sample config, shipped with JGroups and posted on a wiki. > It took about 30 minutes to get to 150 nodes. The cluster appears stable once its up, the problem is in the startup. Based on what I am seeing and have read in the >documentation, every time a node wants to join a FLUSH is sent across the entire system and then a new view is created. FLUSH can not be part of a large cluster configuration; virtual synchrony was never meant to scale to more than 20 or so nodes ! > We are seeing nodes wait minutes to get the new view. It has been mentioned before that Infinispan was tested at 1000 nodes Where was this mentioned ? I personally have never see such a large cluster... The largest cluster I know of is ca 400 nodes... [1] https://lists.sourceforge.net/mailman/listinfo/javagroups-users -- Bela Ban Lead JGroups / Clustering Team JBoss _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

Date: Sat, 19 Mar 2011 10:50:54 +0100 From: bban(a)redhat.com To: infinispan-dev(a)lists.jboss.org Subject: Re: [infinispan-dev] Infinispan Large Scale support On 3/18/11 10:35 PM, Dave wrote: > Won't be able to get CR4 uploaded, policy dictates that I wait until final > release. However, I was able to get 431 nodes up and running as a replicated > cluster and 115 nodes up as a distributed cluster. For the 430 node cache, I > was able to get it started with no problems about 50% of the time. When they > formed multiple clusters they merged together only some of the time. It > really does appear to be a startup issue at this point. We have not pushed > it hard enough yet to see what happens at this scale under load. > > > > Any idea when CR4 will be FINAL? > > Are there any tools to help diagnose problems / performance at this scale (I > ended up writing my own monitor program)? Yes, there's probe.sh at the JGroups level. I created a JIRA to provide a sample for large clusters. You said you based your config on udp.xml, correct ? [1] https://issues.jboss.org/browse/JGRP-1307 -- Bela Ban Lead JGroups / Clustering Team JBoss _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

Initially yes, but I think we are getting better stability using TCP. I

Dave Marion > Date: Sat, 19 Mar 2011 10:50:54 +0100 > From: bban(a)redhat.com > To: infinispan-dev(a)lists.jboss.org > Subject: Re: [infinispan-dev] Infinispan Large Scale support > > > > On 3/18/11 10:35 PM, Dave wrote: >> Won't be able to get CR4 uploaded, policy dictates that I wait until

>> release. However, I was able to get 431 nodes up and running as a

>> cluster and 115 nodes up as a distributed cluster. For the 430 node

>> was able to get it started with no problems about 50% of the time. When

>> formed multiple clusters they merged together only some of the time. It >> really does appear to be a startup issue at this point. We have not

>> it hard enough yet to see what happens at this scale under load. >> >> >> >> Any idea when CR4 will be FINAL? >> >> Are there any tools to help diagnose problems / performance at this

>> ended up writing my own monitor program)? > > > Yes, there's probe.sh at the JGroups level. I created a JIRA to provide > a sample for large clusters. You said you based your config on udp.xml, > correct ? > > [1] https://issues.jboss.org/browse/JGRP-1307 > > -- > Bela Ban > Lead JGroups / Clustering Team > JBoss > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

I switched back to UDP today based on your feedback. Our config resembles the config below. Like I said we just increased sizes and timeouts. If you ask me why I tweaked a certain parameter my response would be that it seemed like a good idea based on the JGroups documentation. UDP seemed a little more problematic than TCP, not sure why though. <config xmlns="urn:org:jgroups" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="urn:org:jgroups file:schema/JGroups-2.8.xsd"> <UDP mcast_addr="${jgroups.udp.mcast_addr:228.6.7.8}" mcast_port="${jgroups.udp.mcast_port:46655}" tos="8" ucast_recv_buf_size="20000000" ucast_send_buf_size="640000" mcast_recv_buf_size="25000000" mcast_send_buf_size="640000" loopback="true" discard_incompatible_packets="true" max_bundle_size="4000000" max_bundle_timeout="30" ip_ttl="${jgroups.udp.ip_ttl:2}" enable_bundling="true" enable_diagnostics="false" thread_naming_pattern="pl" thread_pool.enabled="true" thread_pool.min_threads="2" thread_pool.max_threads="30" thread_pool.keep_alive_time="5000" thread_pool.queue_enabled="true" thread_pool.queue_max_size="1000" thread_pool.rejection_policy="Discard" oob_thread_pool.enabled="true" oob_thread_pool.min_threads="2" oob_thread_pool.max_threads="30" oob_thread_pool.keep_alive_time="5000" oob_thread_pool.queue_enabled="true" oob_thread_pool.queue_max_size="1000" oob_thread_pool.rejection_policy="Discard" /> <PING timeout="360000" num_initial_members="400" break_on_coord_rsp="false"/> <MERGE2 max_interval="30000" min_interval="10000"/> <FD_SOCK/> <FD_ALL/> <BARRIER /> <pbcast.NAKACK use_stats_for_retransmission="false" exponential_backoff="0" use_mcast_xmit="true" gc_lag="0" retransmit_timeout="300,600,1200,2400,3600,4800" discard_delivered_msgs="true"/> <UNICAST timeout="300,600,1200,2400,3600,4800"/> <pbcast.STABLE stability_delay="1000" desired_avg_gossip="50000" max_bytes="1000000"/> <pbcast.GMS print_local_addr="false" join_timeout="60000" view_bundling="true" use_flush_if_present="false"/> <UFC max_credits="2000000" min_threshold="0.20"/> <MFC max_credits="2000000" min_threshold="0.20"/> <FRAG2 frag_size="2000000" /> <pbcast.STREAMING_STATE_TRANSFER/> <!--<pbcast.STATE_TRANSFER/> --> <pbcast.FLUSH timeout="0"/> </config> -----Original Message----- From: infinispan-dev-bounces(a)lists.jboss.org [mailto:infinispan-dev-bounces@lists.jboss.org] On Behalf Of Bela Ban Sent: Saturday, March 19, 2011 1:15 PM To: infinispan-dev(a)lists.jboss.org Subject: Re: [infinispan-dev] Infinispan Large Scale support Hard to believe that TCP would be better, as TCP creates a mesh of connections; for 400 nodes, with every node sending, you'll have roughly 400*400 connections ! I always had a much better experience with UDP On 3/19/11 2:37 PM, david marion wrote: > > Initially yes, but I think we are getting better stability using TCP. I switched it back to TCP yesterday. I can post specifics of what I did in the TCP configuration, but the short story is I increased a lot of the timeout values to get it to work. > > Dave Marion > > >> Date: Sat, 19 Mar 2011 10:50:54 +0100 >> From: bban(a)redhat.com >> To: infinispan-dev(a)lists.jboss.org >> Subject: Re: [infinispan-dev] Infinispan Large Scale support >> >> >> >> On 3/18/11 10:35 PM, Dave wrote: >>> Won't be able to get CR4 uploaded, policy dictates that I wait until final >>> release. However, I was able to get 431 nodes up and running as a replicated >>> cluster and 115 nodes up as a distributed cluster. For the 430 node cache, I >>> was able to get it started with no problems about 50% of the time. When they >>> formed multiple clusters they merged together only some of the time. It >>> really does appear to be a startup issue at this point. We have not pushed >>> it hard enough yet to see what happens at this scale under load. >>> >>> >>> >>> Any idea when CR4 will be FINAL? >>> >>> Are there any tools to help diagnose problems / performance at this scale (I >>> ended up writing my own monitor program)? >> >> >> Yes, there's probe.sh at the JGroups level. I created a JIRA to provide >> a sample for large clusters. You said you based your config on udp.xml, >> correct ? >> >> [1] https://issues.jboss.org/browse/JGRP-1307 >> >> -- >> Bela Ban >> Lead JGroups / Clustering Team >> JBoss >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev > > > > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev

Date: Tue, 22 Mar 2011 08:44:32 +0100 From: bban(a)redhat.com To: infinispan-dev(a)lists.jboss.org Subject: Re: [infinispan-dev] Infinispan Large Scale support I cross-posted this to the JGroups mailing lists [1] [1] https://sourceforge.net/mail/?group_id=6081 On 3/22/11 2:05 AM, Dave wrote: > I switched back to UDP today based on your feedback. Our config resembles > the config below. Like I said we just increased sizes and timeouts. If you > ask me why I tweaked a certain parameter my response would be that it seemed > like a good idea based on the JGroups documentation. UDP seemed a little > more problematic than TCP, not sure why though. > > <config xmlns="urn:org:jgroups" > xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" > xsi:schemaLocation="urn:org:jgroups file:schema/JGroups-2.8.xsd"> > <UDP > mcast_addr="${jgroups.udp.mcast_addr:228.6.7.8}" > mcast_port="${jgroups.udp.mcast_port:46655}" > tos="8" > ucast_recv_buf_size="20000000" > ucast_send_buf_size="640000" > mcast_recv_buf_size="25000000" > mcast_send_buf_size="640000" > loopback="true" > discard_incompatible_packets="true" > max_bundle_size="4000000" > max_bundle_timeout="30" > ip_ttl="${jgroups.udp.ip_ttl:2}" > enable_bundling="true" > enable_diagnostics="false" > > thread_naming_pattern="pl" > > thread_pool.enabled="true" > thread_pool.min_threads="2" > thread_pool.max_threads="30" > thread_pool.keep_alive_time="5000" > thread_pool.queue_enabled="true" > thread_pool.queue_max_size="1000" > thread_pool.rejection_policy="Discard" > > oob_thread_pool.enabled="true" > oob_thread_pool.min_threads="2" > oob_thread_pool.max_threads="30" > oob_thread_pool.keep_alive_time="5000" > oob_thread_pool.queue_enabled="true" > oob_thread_pool.queue_max_size="1000" > oob_thread_pool.rejection_policy="Discard" > /> > > <PING timeout="360000" num_initial_members="400" > break_on_coord_rsp="false"/> > <MERGE2 max_interval="30000" min_interval="10000"/> > <FD_SOCK/> > <FD_ALL/> > <BARRIER /> > <pbcast.NAKACK use_stats_for_retransmission="false" > exponential_backoff="0" > use_mcast_xmit="true" gc_lag="0" > retransmit_timeout="300,600,1200,2400,3600,4800" > discard_delivered_msgs="true"/> > <UNICAST timeout="300,600,1200,2400,3600,4800"/> > <pbcast.STABLE stability_delay="1000" desired_avg_gossip="50000" > max_bytes="1000000"/> > <pbcast.GMS print_local_addr="false" join_timeout="60000" > view_bundling="true" use_flush_if_present="false"/> > <UFC max_credits="2000000" min_threshold="0.20"/> > <MFC max_credits="2000000" min_threshold="0.20"/> > <FRAG2 frag_size="2000000" /> > <pbcast.STREAMING_STATE_TRANSFER/> > <!--<pbcast.STATE_TRANSFER/> --> > <pbcast.FLUSH timeout="0"/> > </config> > > > -----Original Message----- > From: infinispan-dev-bounces(a)lists.jboss.org > [mailto:infinispan-dev-bounces@lists.jboss.org] On Behalf Of Bela Ban > Sent: Saturday, March 19, 2011 1:15 PM > To: infinispan-dev(a)lists.jboss.org > Subject: Re: [infinispan-dev] Infinispan Large Scale support > > Hard to believe that TCP would be better, as TCP creates a mesh of > connections; for 400 nodes, with every node sending, you'll have roughly > 400*400 connections ! > > I always had a much better experience with UDP > > On 3/19/11 2:37 PM, david marion wrote: >> >> Initially yes, but I think we are getting better stability using TCP. I > switched it back to TCP yesterday. I can post specifics of what I did in the > TCP configuration, but the short story is I increased a lot of the timeout > values to get it to work. >> >> Dave Marion >> >> >>> Date: Sat, 19 Mar 2011 10:50:54 +0100 >>> From: bban(a)redhat.com >>> To: infinispan-dev(a)lists.jboss.org >>> Subject: Re: [infinispan-dev] Infinispan Large Scale support >>> >>> >>> >>> On 3/18/11 10:35 PM, Dave wrote: >>>> Won't be able to get CR4 uploaded, policy dictates that I wait until > final >>>> release. However, I was able to get 431 nodes up and running as a > replicated >>>> cluster and 115 nodes up as a distributed cluster. For the 430 node > cache, I >>>> was able to get it started with no problems about 50% of the time. When > they >>>> formed multiple clusters they merged together only some of the time. It >>>> really does appear to be a startup issue at this point. We have not > pushed >>>> it hard enough yet to see what happens at this scale under load. >>>> >>>> >>>> >>>> Any idea when CR4 will be FINAL? >>>> >>>> Are there any tools to help diagnose problems / performance at this > scale (I >>>> ended up writing my own monitor program)? >>> >>> >>> Yes, there's probe.sh at the JGroups level. I created a JIRA to provide >>> a sample for large clusters. You said you based your config on udp.xml, >>> correct ? >>> >>> [1] https://issues.jboss.org/browse/JGRP-1307 >>> >>> -- >>> Bela Ban >>> Lead JGroups / Clustering Team >>> JBoss >>> _______________________________________________ >>> infinispan-dev mailing list >>> infinispan-dev(a)lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >> >> >> >> >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev > -- Bela Ban Lead JGroups / Clustering Team JBoss _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

Date: Wed, 23 Mar 2011 06:31:31 +0100 From: bban(a)redhat.com To: infinispan-dev(a)lists.jboss.org Subject: Re: [infinispan-dev] Infinispan Large Scale support On 3/22/11 9:02 PM, david marion wrote: > > Bela, > > Tried your new config. I had to leave FLUSH in for Infinispan 4.2.0. This is going to kill performance. > Also, it complained about the cap parameter on pbcast.STABLE, so I removed it. You have to use JGroups 2.12.0.Final, replace the JAR shipped with Infinispan 4.2.0. > When I started everything up, it created 5 groups and did not appear to be merging. The merge was timing out (maybe because the FLUSH was taking too long). I increased some of the timeouts (listed below) and then it started up with three groups. I will test more tomorrow. > > PING timeout="10000" > pbcast.GMS join_timeout="20000" merge_timeout="10000" I suggest take a step back and test only with JGroups. Once we've got a configuration that works for JGroups, you can copy it and use it to start Infinispan. There's no point in starting the entire stack at this time. -- Bela Ban Lead JGroups / Clustering Team JBoss _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

Bela, Is there a way to start up the JGroups stack on every node without using Infinispan?

Is there some functional test that I can run or something? I know I can't remove the FLUSH from Infinispan until 5.0.0 and I don't know if I can upgrade the underlying JGroups jar.

Good news! I was able to use the system property from ISPN-83 and remove the FLUSH from the jgroups config with 4.2.1.FINAL, and start-up times are much much better. We have a replicated cache on about 420+ nodes up in under 2 minutes.

I am seeing an issue with the distributed cache though with as little as 5 nodes. In the coordinator log I see org.infinispan.distribution.DistributionmanagerImpl: Detected a view change. Member list changed....... org.infinispan.distribution.DistributionmanagerImpl: This is a JOIN event! Wait for notification from new joiner<name> In the log from the joining node I see: org.infinispan.distribution.JoinTask: Commencing rehash on node:<name>. Before start, distributionManager.joinComplete=false org.infinispan.distribution.JoinTask: Requesting old consistent hash from coordinator I jstack'd the joiner, the DefaultCacheManager.getCache() method is waiting on org.infinispan.distribution.DistributionManagerImpl.waitForJoinToComplete() and the Rehasher thread is waiting on: at org.infinispan.util.concurrent.ReclosableLatch.await(ReclosableLatch.java:75) at org.infinipsan.remoting.transport.jgroups.JGroupsDistSync.blockUntilNoJoinsInProgress(JGroupsDistSync.java:113) Any thoughts?

> Date: Wed, 23 Mar 2011 15:58:19 +0100 > From: bban(a)redhat.com > To: infinispan-dev(a)lists.jboss.org > Subject: Re: [infinispan-dev] Infinispan Large Scale support > > > > On 3/23/11 2:39 PM, david marion wrote: >> >> Bela, >> >> Is there a way to start up the JGroups stack on every node without using Infinispan? > > > You could use ViewDemo [1] or Draw. Or write your own small test > program; if you take a look at ViewDemo's src, you'll see that it's onyl > a page of code. > > >> Is there some functional test that I can run or something? I know I can't remove the FLUSH from Infinispan until 5.0.0 and I don't know if I can upgrade the underlying >> JGroups jar. > > > I suggest test with the latest JGroups (2.12.0) and +FLUSH and -FLUSH. > The +FLUSH config should be less painful now, with the introduction of > view bundling: we need to run flush fewer times than before. > > > [1] http://community.jboss.org/wiki/TestingJBoss > > -- > Bela Ban > Lead JGroups / Clustering Team > JBoss > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

Bela, Yes, it is a replicated cache and I used your udp-largecluster.xml file and just modified it slightly. It does appear that the distributed cache is in a deadlock (or there is a race condition), the coordinator comes up, but the other caches do not, they sit there and wait. I was able to get a distributed cache up and running on 100+ nodes, now I cannot get 5 of them running. > Date: Tue, 5 Apr 2011 11:09:54 +0200 > From: bban(a)redhat.com > To: infinispan-dev(a)lists.jboss.org > Subject: Re: [infinispan-dev] Infinispan Large Scale support > > > > On 4/4/11 5:45 PM, david marion wrote: >> >> >> Good news! I was able to use the system property from ISPN-83 and remove the FLUSH from the jgroups config with 4.2.1.FINAL, and start-up times are much much better. We have a replicated cache on about 420+ nodes up in under 2 minutes. > > > Great ! Just to confirm: this is 420+ Infinispan instances, with > replication enabled, correct ? > > Did you use a specific JGroups config (e.g. udp-largecluster.xml) ? > > >> I am seeing an issue with the distributed cache though with as little as 5 nodes. >> >> In the coordinator log I see >> >> org.infinispan.distribution.DistributionmanagerImpl: Detected a view change. Member list changed....... >> org.infinispan.distribution.DistributionmanagerImpl: This is a JOIN event! Wait for notification from new joiner<name> >> >> In the log from the joining node I see: >> >> org.infinispan.distribution.JoinTask: Commencing rehash on node:<name>. Before start, distributionManager.joinComplete=false >> org.infinispan.distribution.JoinTask: Requesting old consistent hash from coordinator >> >> I jstack'd the joiner, the DefaultCacheManager.getCache() method is waiting on org.infinispan.distribution.DistributionManagerImpl.waitForJoinToComplete() and the Rehasher thread >> is waiting on: >> >> at org.infinispan.util.concurrent.ReclosableLatch.await(ReclosableLatch.java:75) >> at org.infinipsan.remoting.transport.jgroups.JGroupsDistSync.blockUntilNoJoinsInProgress(JGroupsDistSync.java:113) >> >> Any thoughts? > > > I recently took a look at the distribution code, and this part is very > brittle with respect to parallel startup and merging. Plus, I believe > the (blocking) RPC to fetch the old CH from the coordinator might > deadlock in certain cases... > > I've got a pull request for a push based rebalancing versus pull based > rebalancing pending. It'll likely make it into 5.x, as a matter of fact > I've got a chat about this this afternoon. > > > > >>> Date: Wed, 23 Mar 2011 15:58:19 +0100 >>> From: bban(a)redhat.com >>> To: infinispan-dev(a)lists.jboss.org >>> Subject: Re: [infinispan-dev] Infinispan Large Scale support >>> >>> >>> >>> On 3/23/11 2:39 PM, david marion wrote: >>>> >>>> Bela, >>>> >>>> Is there a way to start up the JGroups stack on every node without using Infinispan? >>> >>> >>> You could use ViewDemo [1] or Draw. Or write your own small test >>> program; if you take a look at ViewDemo's src, you'll see that it's onyl >>> a page of code. >>> >>> >>>> Is there some functional test that I can run or something? I know I can't remove the FLUSH from Infinispan until 5.0.0 and I don't know if I can upgrade the underlying >>>> JGroups jar. >>> >>> >>> I suggest test with the latest JGroups (2.12.0) and +FLUSH and -FLUSH. >>> The +FLUSH config should be less painful now, with the introduction of >>> view bundling: we need to run flush fewer times than before. >>> >>> >>> [1] http://community.jboss.org/wiki/TestingJBoss >>> >>> -- >>> Bela Ban >>> Lead JGroups / Clustering Team >>> JBoss >>> _______________________________________________ >>> infinispan-dev mailing list >>> infinispan-dev(a)lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >> >> >> >> >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev > > -- > Bela Ban > Lead JGroups / Clustering Team > JBoss > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

Hi Bela, I'm interested in your changes as well, as concurrent startup vexes my cache usage as well. Is there a wiki or JIRA I could look at to understand the fundamental differences? Thanks, Erik -----Original Message----- From: infinispan-dev-bounces(a)lists.jboss.org [mailto:infinispan-dev-bounces@lists.jboss.org] On Behalf Of Bela Ban Sent: Wednesday, April 06, 2011 2:46 AM To: infinispan-dev(a)lists.jboss.org Subject: Re: [infinispan-dev] Infinispan Large Scale support Hi David, Dan and I had a talk about integrating my changes to the distribution code to 5.x. As I mentioned below, the current code is quite brittle wrt concurrent startup, so this will get fixed with my changes. I hope we can backport this to the 4.2.x branch as well. As a matter of fact, I actually made my changes on a branch off of 4.2.x. On 4/5/11 2:51 PM, david marion wrote: > > Bela, > > Yes, it is a replicated cache and I used your udp-largecluster.xml file and just modified it slightly. It does appear that the distributed cache is in a deadlock (or there is a race condition), the coordinator comes up, but the other caches do not, they sit there and wait. I was able to get a distributed cache up and running on 100+ nodes, now I cannot get 5 of them running. > >> Date: Tue, 5 Apr 2011 11:09:54 +0200 >> From: bban(a)redhat.com >> To: infinispan-dev(a)lists.jboss.org >> Subject: Re: [infinispan-dev] Infinispan Large Scale support >> >> >> >> On 4/4/11 5:45 PM, david marion wrote: >>> >>> >>> Good news! I was able to use the system property from ISPN-83 and remove the FLUSH from the jgroups config with 4.2.1.FINAL, and start-up times are much much better. We have a replicated cache on about 420+ nodes up in under 2 minutes. >> >> >> Great ! Just to confirm: this is 420+ Infinispan instances, with >> replication enabled, correct ? >> >> Did you use a specific JGroups config (e.g. udp-largecluster.xml) ? >> >> >>> I am seeing an issue with the distributed cache though with as little as 5 nodes. >>> >>> In the coordinator log I see >>> >>> org.infinispan.distribution.DistributionmanagerImpl: Detected a view change. Member list changed....... >>> org.infinispan.distribution.DistributionmanagerImpl: This is a JOIN >>> event! Wait for notification from new joiner<name> >>> >>> In the log from the joining node I see: >>> >>> org.infinispan.distribution.JoinTask: Commencing rehash on >>> node:<name>. Before start, distributionManager.joinComplete=false >>> org.infinispan.distribution.JoinTask: Requesting old consistent hash >>> from coordinator >>> >>> I jstack'd the joiner, the DefaultCacheManager.getCache() method is >>> waiting on org.infinispan.distribution.DistributionManagerImpl.waitForJoinToComplete() and the Rehasher thread is waiting on: >>> >>> at >>> org.infinispan.util.concurrent.ReclosableLatch.await(ReclosableLatch >>> .java:75) at >>> org.infinipsan.remoting.transport.jgroups.JGroupsDistSync.blockUntil >>> NoJoinsInProgress(JGroupsDistSync.java:113) >>> >>> Any thoughts? >> >> >> I recently took a look at the distribution code, and this part is >> very brittle with respect to parallel startup and merging. Plus, I >> believe the (blocking) RPC to fetch the old CH from the coordinator >> might deadlock in certain cases... >> >> I've got a pull request for a push based rebalancing versus pull >> based rebalancing pending. It'll likely make it into 5.x, as a matter >> of fact I've got a chat about this this afternoon. >> >> >> >> >>>> Date: Wed, 23 Mar 2011 15:58:19 +0100 >>>> From: bban(a)redhat.com >>>> To: infinispan-dev(a)lists.jboss.org >>>> Subject: Re: [infinispan-dev] Infinispan Large Scale support >>>> >>>> >>>> >>>> On 3/23/11 2:39 PM, david marion wrote: >>>>> >>>>> Bela, >>>>> >>>>> Is there a way to start up the JGroups stack on every node without using Infinispan? >>>> >>>> >>>> You could use ViewDemo [1] or Draw. Or write your own small test >>>> program; if you take a look at ViewDemo's src, you'll see that it's >>>> onyl a page of code. >>>> >>>> >>>>> Is there some functional test that I can run or something? I know >>>>> I can't remove the FLUSH from Infinispan until 5.0.0 and I don't know if I can upgrade the underlying JGroups jar. >>>> >>>> >>>> I suggest test with the latest JGroups (2.12.0) and +FLUSH and -FLUSH. >>>> The +FLUSH config should be less painful now, with the introduction >>>> of view bundling: we need to run flush fewer times than before. >>>> >>>> >>>> [1] http://community.jboss.org/wiki/TestingJBoss >>>> >>>> -- >>>> Bela Ban >>>> Lead JGroups / Clustering Team >>>> JBoss >>>> _______________________________________________ >>>> infinispan-dev mailing list >>>> infinispan-dev(a)lists.jboss.org >>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>> >>> >>> >>> >>> _______________________________________________ >>> infinispan-dev mailing list >>> infinispan-dev(a)lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >> >> -- >> Bela Ban >> Lead JGroups / Clustering Team >> JBoss >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev > > > > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev -- Bela Ban Lead JGroups / Clustering Team JBoss _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev The information contained in this message is legally privileged and confidential, and is intended for the individual or entity to whom it is addressed (or their designee). If this message is read by anyone other than the intended recipient, please be advised that distribution of this message, in any form, is strictly prohibited. If you have received this message in error, please notify the sender immediately and delete or destroy all copies of this message. _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

From: galder(a)redhat.com Date: Wed, 6 Apr 2011 20:44:44 +0200 To: infinispan-dev(a)lists.jboss.org Subject: Re: [infinispan-dev] Infinispan Large Scale support David, Removing FLUSH is something that causes issues as highlighted by https://issues.jboss.org/browse/ISPN-937 Cheers, On Apr 4, 2011, at 5:45 PM, david marion wrote: > > Good news! I was able to use the system property from ISPN-83 and remove the FLUSH from the jgroups config with 4.2.1.FINAL, and start-up times are much much better. We have a replicated cache on about 420+ nodes up in under 2 minutes. I am seeing an issue with the distributed cache though with as little as 5 nodes. > > In the coordinator log I see > > org.infinispan.distribution.DistributionmanagerImpl: Detected a view change. Member list changed....... > org.infinispan.distribution.DistributionmanagerImpl: This is a JOIN event! Wait for notification from new joiner <name> > > In the log from the joining node I see: > > org.infinispan.distribution.JoinTask: Commencing rehash on node: <name>. Before start, distributionManager.joinComplete=false > org.infinispan.distribution.JoinTask: Requesting old consistent hash from coordinator > > I jstack'd the joiner, the DefaultCacheManager.getCache() method is waiting on org.infinispan.distribution.DistributionManagerImpl.waitForJoinToComplete() and the Rehasher thread > is waiting on: > > at org.infinispan.util.concurrent.ReclosableLatch.await(ReclosableLatch.java:75) > at org.infinipsan.remoting.transport.jgroups.JGroupsDistSync.blockUntilNoJoinsInProgress(JGroupsDistSync.java:113) > > Any thoughts? > > > Date: Wed, 23 Mar 2011 15:58:19 +0100 > > From: bban(a)redhat.com > > To: infinispan-dev(a)lists.jboss.org > > Subject: Re: [infinispan-dev] Infinispan Large Scale support > > > > > > > > On 3/23/11 2:39 PM, david marion wrote: > > > > > > Bela, > > > > > > Is there a way to start up the JGroups stack on every node without using Infinispan? > > > > > > You could use ViewDemo [1] or Draw. Or write your own small test > > program; if you take a look at ViewDemo's src, you'll see that it's onyl > > a page of code. > > > > > > > Is there some functional test that I can run or something? I know I can't remove the FLUSH from Infinispan until 5.0.0 and I don't know if I can upgrade the underlying > > >JGroups jar. > > > > > > I suggest test with the latest JGroups (2.12.0) and +FLUSH and -FLUSH. > > The +FLUSH config should be less painful now, with the introduction of > > view bundling: we need to run flush fewer times than before. > > > > > > [1] http://community.jboss.org/wiki/TestingJBoss > > > > -- > > Bela Ban > > Lead JGroups / Clustering Team > > JBoss > > _______________________________________________ > > infinispan-dev mailing list > > infinispan-dev(a)lists.jboss.org > > https://lists.jboss.org/mailman/listinfo/infinispan-dev > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev -- Galder Zamarreño Sr. Software Engineer Infinispan, JBoss Cache _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

OK, so the question is whether FLUSH removal causes problems with replication or distribution, or both ?

David got 420+ nodes with replication, and FLUSH removed. However, with distribution and FLUSH removed, it failed. Are you suggesting that FLUSH is (currently) required only for *distribution* mode ?

David, maybe worth a try to add <FLUSH timeout="3000".../> to udp-largecluster.xml and try to start the cluster with mode=distribution... Cheers, On 4/6/11 8:44 PM, Galder Zamarreño wrote: > David, > > Removing FLUSH is something that causes issues as highlighted by https://issues.jboss.org/browse/ISPN-937 > > Cheers, > > On Apr 4, 2011, at 5:45 PM, david marion wrote: > >> >> Good news! I was able to use the system property from ISPN-83 and remove the FLUSH from the jgroups config with 4.2.1.FINAL, and start-up times are much much better. We have a replicated cache on about 420+ nodes up in under 2 minutes. I am seeing an issue with the distributed cache though with as little as 5 nodes. >> >> In the coordinator log I see >> >> org.infinispan.distribution.DistributionmanagerImpl: Detected a view change. Member list changed....... >> org.infinispan.distribution.DistributionmanagerImpl: This is a JOIN event! Wait for notification from new joiner<name> >> >> In the log from the joining node I see: >> >> org.infinispan.distribution.JoinTask: Commencing rehash on node:<name>. Before start, distributionManager.joinComplete=false >> org.infinispan.distribution.JoinTask: Requesting old consistent hash from coordinator >> >> I jstack'd the joiner, the DefaultCacheManager.getCache() method is waiting on org.infinispan.distribution.DistributionManagerImpl.waitForJoinToComplete() and the Rehasher thread >> is waiting on: >> >> at org.infinispan.util.concurrent.ReclosableLatch.await(ReclosableLatch.java:75) >> at org.infinipsan.remoting.transport.jgroups.JGroupsDistSync.blockUntilNoJoinsInProgress(JGroupsDistSync.java:113) >> >> Any thoughts? >> >>> Date: Wed, 23 Mar 2011 15:58:19 +0100 >>> From: bban(a)redhat.com >>> To: infinispan-dev(a)lists.jboss.org >>> Subject: Re: [infinispan-dev] Infinispan Large Scale support >>> >>> >>> >>> On 3/23/11 2:39 PM, david marion wrote: >>>> >>>> Bela, >>>> >>>> Is there a way to start up the JGroups stack on every node without using Infinispan? >>> >>> >>> You could use ViewDemo [1] or Draw. Or write your own small test >>> program; if you take a look at ViewDemo's src, you'll see that it's onyl >>> a page of code. >>> >>> >>>> Is there some functional test that I can run or something? I know I can't remove the FLUSH from Infinispan until 5.0.0 and I don't know if I can upgrade the underlying >>>> JGroups jar. >>> >>> >>> I suggest test with the latest JGroups (2.12.0) and +FLUSH and -FLUSH. >>> The +FLUSH config should be less painful now, with the introduction of >>> view bundling: we need to run flush fewer times than before. >>> >>> >>> [1] http://community.jboss.org/wiki/TestingJBoss >>> >>> -- >>> Bela Ban >>> Lead JGroups / Clustering Team >>> JBoss >>> _______________________________________________ >>> infinispan-dev mailing list >>> infinispan-dev(a)lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev > > -- > Galder Zamarreño > Sr. Software Engineer > Infinispan, JBoss Cache > > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev -- Bela Ban Lead JGroups / Clustering Team JBoss _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

However, I'm not assigning blame, I was under the impression that it had been tested at 1000 nodes. I just had it up at 430 nodes, but I can't reliably get it to that size when I restart it. I think the answer is that Infinispan will have to use what will scale in JGroups (remove what does not scale). Until then I will have to scale it down to a size that I can start in a reliable fashion.

Bela, No VMs. Each node has a 3GB JVM, dual GB Ethernet (bonded).

But there are other things running on the nodes. We are starting them all at once.

On 18 Mar 2011, at 07:12, Bela Ban wrote: > >> Initially we were seeing lots of little clusters form, increasing the number of initial members in the PING section of the jgroups-udp.xml file and increasing the >> timeouts seems to have made things better but not great. Even at 20 > nodes, we are seeing messages where the coordinator is failing to flush. > > I do *not* recommend FLUSH in clusters bigger than 10-15 nodes ! FLUSH > will definitely kill you if you have clusters of 100s of nodes. > > IIRC, Infinispan requires FLUSH, but I think this was removed in > 4.2.1.FINAL. Maybe Manik or Vladimir can comment ? 4.2.1.FINAL isn't out yet, but it should be soon. The strategy I'm taking is that FLUSH will still be enabled and required to maintain the same behaviour as 4.2.0, but you will be able to switch this check off with a system property flag, and provide a JGroups config that excludes FLUSH.

>> It does not appear that the default configuration scales to a large size. > > > Correct. The default config is geared toward a 4-16 node cluster. Yes, you will need to tune your JGroups config for larger clusters. Maybe it makes sense to ship a "jgroups-large-cluster.xml" config as well, to help ease the complexity of this in future. Cheers Manik -- Manik Surtani manik(a)jboss.org twitter.com/maniksurtani Lead, Infinispan http://www.infinispan.org _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

On 11-03-18 10:05 AM, Galder Zamarreño wrote: > Hmmm, but we know there're problems with doing this, right? https://issues.jboss.org/browse/ISPN-937 - Vladimir, did you get around to investigating that? I have not found the root cause. Paul detected the problem in AS test suite. I tested his AS configuration in our test suite and it worked flawlessly. After that, I "outsourced" the investigation to Paul. Since all our state transfer tests were passing I thought its worth investigating AS setup. That's were the investigation died.

On 11-03-18 10:05 AM, Galder Zamarreño wrote: > Hmmm, but we know there're problems with doing this, right? https://issues.jboss.org/browse/ISPN-937 - Vladimir, did you get around to investigating that? I have not found the root cause. Paul detected the problem in AS test suite. I tested his AS configuration in our test suite and it worked flawlessly. After that, I "outsourced" the investigation to Paul. Since all our state transfer tests were passing I thought its worth investigating AS setup. That's were the investigation died. _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

The classes have to be present at the destination, just as with any serializable class.  This means that if you have:   @Externalize(Bar.class)   public class Foo {      ...   } in your module you must also have Bar in your class path.  But note that you need Bar present at compile time already; and, this annotation has runtime retention so you already have a runtime dependency anyway. It would be extremely unlikely for someone to use this annotation without including the Externalizer class which is associated in the same module - that would be a very unusual thing to do. On 03/17/2011 01:14 PM, Eduardo Martins wrote: > Hi, so the marshaller is serialized, but what happens to the classes > it marshalls? And even if whole dependencies are serialized too, in > our case there are resources that needed to be initialized for it to > work, in concrete "client" component jars need to be deployed. > > The proper solution, which is the only way to ensure everything will > always work, is to have a proper life cycle where externalizers are > set before cache starts, we have this already. IMHO what is really > missing in Infinispan, is a instantiated cache state before started, > as JBoss Cache has. It's really hard to come up with a final > configuration (including externalizers), before creating cache > instance. I believe it will be a big problem for reusing > configurations created and managed by JBoss AS, for instance it will > probably be very painful to add custom externalizers to such > configurations... > > -- Eduardo > .............................................. > http://emmartins.blogspot.com > http://redhat.com/solutions/telco > > > > On Thu, Mar 17, 2011 at 5:06 PM, Galder Zamarreño&lt;galder(a)redhat.com&gt;  wrote: >> >> On Mar 17, 2011, at 3:22 PM, Manik Surtani wrote: >> >>> >>> On 17 Mar 2011, at 14:16, David M. Lloyd wrote: >>> >>>>> >>>>> Ah I see what you mean.  Initially I thought you'd only need to >>>>> register the externaliser when you first encounter a new type (i.e., >>>>> when a user defined type is first encountered with a put()), but this >>>>> may not be the case since on the remote node it may see the a new >>>>> magic number which may not be registered, and then you have a >>>>> problem. >>>> >>>> Marshalling will send Externalizers across the wire too, so the >>>> receiving side need not know about it in advance.  The sending side can >>>> then do this kind of discovery safely.  In addition, JBMAR already has >>>> the @org.jboss.marshalling.Externalize annotation which does just this. >>>> >>>> Unless you're talking about your own mechanism, anyway.  Just thought >>>> I'd throw this bit out there. >>> >>> Sweet.  I think we can make this work then - even reusing JBM's annotations. >> >> Hmmm, I wasn't aware of that either. I'll defo look into that for 5.0 - https://issues.jboss.org/browse/ISPN-986 >> >> @Eduardo, this might solve your cache startup issue and the current need to define Externalizers in advance. >> >>> >>> -- >>> Manik Surtani >>> manik(a)jboss.org >>> twitter.com/maniksurtani >>> >>> Lead, Infinispan >>> http://www.infinispan.org >>> >>> >>> >>> _______________________________________________ >>> infinispan-dev mailing list >>> infinispan-dev(a)lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >> >> -- >> Galder Zamarreño >> Sr. Software Engineer >> Infinispan, JBoss Cache >> >> >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev >> > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev -- - DML _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

On 17 Mar 2011, at 08:30, Galder Zamarreño wrote: >> >> Is it to do with classloader leaks? If so, a weak map could be used for this table... > > You misunderstood what I meant. By annotation scanning I meant that we won't be scanning the entire classpath for user classes that have @Marshaller, right? I mean, we don't do like an EJB container that goes through deployments checking for all @EJB classes, cos that would be very expensive and it's not our business. So, back to my question, the users will still need to give us the list of user defined externalizers. Sure, we can do annotation scanning on that specific list of classes. Ah I see what you mean. Initially I thought you'd only need to register the externaliser when you first encounter a new type (i.e., when a user defined type is first encountered with a put()), but this may not be the case since on the remote node it may see the a new magic number which may not be registered, and then you have a problem.

> My point is that it might look nice that users can use annotations to define some externalizer properties (id and typeClasses) but they're not getting as much out of it as they'd with @EJBs in an EJB container cos they have to list the externalizers somehow. Again, unless you wanna get into classpath annotation search... Yeah CP scanning does suck, but it is something to think about. Maybe follow a SEAM-like approach and only scan jars that have a certain file present in its META-INF? WDYT? Cheers Manik -- Manik Surtani manik(a)jboss.org twitter.com/maniksurtani Lead, Infinispan http://www.infinispan.org _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev