From rory.odonnell at oracle.com Tue Mar 1 05:14:03 2016 From: rory.odonnell at oracle.com (Rory O'Donnell) Date: Tue, 1 Mar 2016 10:14:03 +0000 Subject: [infinispan-dev] Early Access builds of JDK 9 b107 & JDK 9 with Project Jigsaw b106 (#4540) are available on java.net Message-ID: <56D56B6B.4080305@oracle.com> Hi Galder, Early Access b107 for JDK 9 is available on java.net, summary of changes are listed here . Among other fixes , the following are also included: * Update class file version to 53 for JDK-9, more info here * Add support for ES6 collections , more info here Early Access b106 (#4540) for JDK 9 with Project Jigsaw is available on java.net. The March 2016 Quality Outreach Report is posted on here, thanks again to those who found & logged bugs against Early Access builds. Rgds,Rory -- Rgds,Rory O'Donnell Quality Engineering Manager Oracle EMEA , Dublin, Ireland -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.jboss.org/pipermail/infinispan-dev/attachments/20160301/23485fe8/attachment.html From gustavo at infinispan.org Wed Mar 2 04:42:37 2016 From: gustavo at infinispan.org (Gustavo Fernandes) Date: Wed, 2 Mar 2016 09:42:37 +0000 Subject: [infinispan-dev] Infinispan 8.2.0.CR1 available Message-ID: Dear community, Infinispan 8.2.0.CR1 has just been released! For more information please go to: http://blog.infinispan.org/2016/03/infinispan-820cr1-released.html Cheers, Gustavo -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.jboss.org/pipermail/infinispan-dev/attachments/20160302/0db881ae/attachment.html From gustavo at infinispan.org Wed Mar 2 09:23:36 2016 From: gustavo at infinispan.org (Gustavo Fernandes) Date: Wed, 2 Mar 2016 14:23:36 +0000 Subject: [infinispan-dev] Infinispan 8 presentation on DevConf.cz 2016 In-Reply-To: <2201094.oE6FtqIMlI@dhcp-10-40-5-198.brq.redhat.com> References: <761888892.15555969.1454946567441.JavaMail.zimbra@redhat.com> <956349635.15560183.1454947148356.JavaMail.zimbra@redhat.com> <2201094.oE6FtqIMlI@dhcp-10-40-5-198.brq.redhat.com> Message-ID: On Tue, Feb 9, 2016 at 2:59 AM, Vojtech Juranek wrote: > Hi Sebastian, > > > I'm probably blind - but could you tell me how to find your recording? > All > > I can see on the YT link is some Docker stuff.... > > for some reason it doesn't work in some browsers well, the direct link to > the > presentation is [1]. > (and another ISPN related DevConf presentation is on [2]) > HTH > Vojta > Nice Spark Streaming + Infinispan demo! This would make a great quickstart project! Gustavo > > [1] https://www.youtube.com/watch?v=r0__NEgldzI > [2] https://www.youtube.com/watch?v=TVXeXM2g7So > _______________________________________________ > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.jboss.org/pipermail/infinispan-dev/attachments/20160302/0d1b7594/attachment.html From galder at redhat.com Wed Mar 2 11:23:18 2016 From: galder at redhat.com (=?utf-8?Q?Galder_Zamarre=C3=B1o?=) Date: Wed, 2 Mar 2016 17:23:18 +0100 Subject: [infinispan-dev] HotRod exec operation tight coupling with JBoss Marshaller In-Reply-To: References: <954084CA-04F0-4389-9A09-BB89B7BE8DED@redhat.com> <351961288.55333152.1456752949592.JavaMail.zimbra@redhat.com> Message-ID: <73260886-1B33-41BA-B1AB-830F7DDEB4FB@redhat.com> Just an update: the prototype I worked on to add a 'datatype' metadata parameter to the script seems to be working, so I'll be sending a PR for inclusion in 8.2. Cheers, -- Galder Zamarre?o Infinispan, Red Hat > On 29 Feb 2016, at 19:46, Galder Zamarre?o wrote: > > Hi all, > > I had a chat earlier with Tristan and he pointed me to the metadata information that can be passed in script header. > > Given that so far (until compatibility is supported), Javascript client only supports String key/values, a very simple solution to this problem would be to define a new metadata parameter, e.g. data-type, which can optionally define the type of key, value, parameters and returned object. E.g. in the javascript client case, I could just say: data-type=utf8 in the header of the script, and that would provide enough hints for the server to do its job by interpreting byte arrays dealt with in exec command as UTF-8 strings. > > This method is much more user friendly than having user plug a marshaller since it only requires a small change in the header of the script, as opposed to having to change server side configuration. I'm working on a prototype for this with hopes to include in 8.2. > > Cheers, > -- > Galder Zamarre?o > Infinispan, Red Hat > >> On 29 Feb 2016, at 14:35, Vittorio Rigamonti wrote: >> >> Hi All, Hi Galder >> >> JBasicMarshaller.h is not an intent to go further along the way to implement a c++ JBossMarshaller, it has the only goal to provide a working 'exec' operation. >> The c++ exec currently works only with integer and "small" string data types, user can easily extends the marshaller to other basic types. >> >> My current model of the exec use case is this one (maybe it's too simplistic so correct me if I'm wrong): a "user defined consumer" consumes data produced by a "user defined producer", so maybe it could worth to evaluate a solution where the framework provides a common standard for communication (stringified json? or JSObject for in memory 100% Java) and let the user handle it's own data. >> This is a proposal approach specific to the exec scope, I can't say if it can be extended as a general approach where marshalling is involved. >> >> Cheers, >> Vittorio >> >> >> >> >> ----- Original Message ----- >> From: "Galder Zamarre?o" >> To: "infinispan -Dev List" >> Sent: Monday, February 29, 2016 11:28:55 AM >> Subject: [infinispan-dev] HotRod exec operation tight coupling with JBoss Marshaller >> >> Hi all, >> >> While implement `exec` operation for the JS client, I've encountered an issue with how the exec parameters and return types are marshalled. >> >> The essence of the problem is that the server marshalls these objects instead of having the client drive how these are marshalled. As a result of this, for a JS or C++ client to be able to use `exec` with default configuration, they need to understand JBoss Marshaller format, which is not good. >> >> I'm not sure this would have been unavoidable due to the characteristics of `exec` but I wanted to see if we can find a good way to solve or get around this issue. Long term, we need better encoding handling both for incoming and returning types, but the question is whether we can find a way to better solve this until then. Here are some options: >> >> - For the C++ client, Vittorio has part implemented the JBoss Marshaller format [1], but I'm kinda reluctant to go down this path since that creates a lot of work for us as the number of types that can be discovered in a JBoss Marshaller format byte array are quite considerable [2]. We're bound to miss one of those and since clients could execute any script, the chances are high IMO... >> >> - An alternative would be for the JS/C++ clients to only support exec when the marshaller is one that enables compatibility mode. The idea here is that for compatibility mode to work, all clients involved are going to be set up with a marshaller that can work for all of them. Working on such marshaller is time better spent than on implementing the JBoss Marshaller format. We had a separate discussion on this topic in another dev thread... >> >> Any other ideas someone might have? >> >> Cheers, >> >> [1] https://github.com/infinispan/cpp-client/blob/master/include/infinispan/hotrod/JBasicMarshaller.h >> [2] https://github.com/jboss-remoting/jboss-marshalling/blob/master/river/src/main/java/org/jboss/marshalling/river/Protocol.java >> -- >> Galder Zamarre?o >> Infinispan, Red Hat >> >> >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev at lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev >> >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev at lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev > > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev From galder at redhat.com Wed Mar 2 11:35:14 2016 From: galder at redhat.com (=?utf-8?Q?Galder_Zamarre=C3=B1o?=) Date: Wed, 2 Mar 2016 17:35:14 +0100 Subject: [infinispan-dev] HotRod exec operation tight coupling with JBoss Marshaller In-Reply-To: <73260886-1B33-41BA-B1AB-830F7DDEB4FB@redhat.com> References: <954084CA-04F0-4389-9A09-BB89B7BE8DED@redhat.com> <351961288.55333152.1456752949592.JavaMail.zimbra@redhat.com> <73260886-1B33-41BA-B1AB-830F7DDEB4FB@redhat.com> Message-ID: FYI: https://issues.jboss.org/browse/ISPN-6307 -- Galder Zamarre?o Infinispan, Red Hat > On 2 Mar 2016, at 17:23, Galder Zamarre?o wrote: > > Just an update: the prototype I worked on to add a 'datatype' metadata parameter to the script seems to be working, so I'll be sending a PR for inclusion in 8.2. > > Cheers, > -- > Galder Zamarre?o > Infinispan, Red Hat > >> On 29 Feb 2016, at 19:46, Galder Zamarre?o wrote: >> >> Hi all, >> >> I had a chat earlier with Tristan and he pointed me to the metadata information that can be passed in script header. >> >> Given that so far (until compatibility is supported), Javascript client only supports String key/values, a very simple solution to this problem would be to define a new metadata parameter, e.g. data-type, which can optionally define the type of key, value, parameters and returned object. E.g. in the javascript client case, I could just say: data-type=utf8 in the header of the script, and that would provide enough hints for the server to do its job by interpreting byte arrays dealt with in exec command as UTF-8 strings. >> >> This method is much more user friendly than having user plug a marshaller since it only requires a small change in the header of the script, as opposed to having to change server side configuration. I'm working on a prototype for this with hopes to include in 8.2. >> >> Cheers, >> -- >> Galder Zamarre?o >> Infinispan, Red Hat >> >>> On 29 Feb 2016, at 14:35, Vittorio Rigamonti wrote: >>> >>> Hi All, Hi Galder >>> >>> JBasicMarshaller.h is not an intent to go further along the way to implement a c++ JBossMarshaller, it has the only goal to provide a working 'exec' operation. >>> The c++ exec currently works only with integer and "small" string data types, user can easily extends the marshaller to other basic types. >>> >>> My current model of the exec use case is this one (maybe it's too simplistic so correct me if I'm wrong): a "user defined consumer" consumes data produced by a "user defined producer", so maybe it could worth to evaluate a solution where the framework provides a common standard for communication (stringified json? or JSObject for in memory 100% Java) and let the user handle it's own data. >>> This is a proposal approach specific to the exec scope, I can't say if it can be extended as a general approach where marshalling is involved. >>> >>> Cheers, >>> Vittorio >>> >>> >>> >>> >>> ----- Original Message ----- >>> From: "Galder Zamarre?o" >>> To: "infinispan -Dev List" >>> Sent: Monday, February 29, 2016 11:28:55 AM >>> Subject: [infinispan-dev] HotRod exec operation tight coupling with JBoss Marshaller >>> >>> Hi all, >>> >>> While implement `exec` operation for the JS client, I've encountered an issue with how the exec parameters and return types are marshalled. >>> >>> The essence of the problem is that the server marshalls these objects instead of having the client drive how these are marshalled. As a result of this, for a JS or C++ client to be able to use `exec` with default configuration, they need to understand JBoss Marshaller format, which is not good. >>> >>> I'm not sure this would have been unavoidable due to the characteristics of `exec` but I wanted to see if we can find a good way to solve or get around this issue. Long term, we need better encoding handling both for incoming and returning types, but the question is whether we can find a way to better solve this until then. Here are some options: >>> >>> - For the C++ client, Vittorio has part implemented the JBoss Marshaller format [1], but I'm kinda reluctant to go down this path since that creates a lot of work for us as the number of types that can be discovered in a JBoss Marshaller format byte array are quite considerable [2]. We're bound to miss one of those and since clients could execute any script, the chances are high IMO... >>> >>> - An alternative would be for the JS/C++ clients to only support exec when the marshaller is one that enables compatibility mode. The idea here is that for compatibility mode to work, all clients involved are going to be set up with a marshaller that can work for all of them. Working on such marshaller is time better spent than on implementing the JBoss Marshaller format. We had a separate discussion on this topic in another dev thread... >>> >>> Any other ideas someone might have? >>> >>> Cheers, >>> >>> [1] https://github.com/infinispan/cpp-client/blob/master/include/infinispan/hotrod/JBasicMarshaller.h >>> [2] https://github.com/jboss-remoting/jboss-marshalling/blob/master/river/src/main/java/org/jboss/marshalling/river/Protocol.java >>> -- >>> Galder Zamarre?o >>> Infinispan, Red Hat >>> >>> >>> _______________________________________________ >>> infinispan-dev mailing list >>> infinispan-dev at lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>> >>> _______________________________________________ >>> infinispan-dev mailing list >>> infinispan-dev at lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >> >> >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev at lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev > > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev From mudokonman at gmail.com Thu Mar 3 10:19:51 2016 From: mudokonman at gmail.com (William Burns) Date: Thu, 03 Mar 2016 15:19:51 +0000 Subject: [infinispan-dev] Lambda Serialization In-Reply-To: References:

Message-ID: I now have a working branch that is using this for the new CacheStream interface [1]. With this it allows users to use a stream without needing any casts for any of the intermediate or terminal operations. Note I completely revamped the BaseStreamTest [2] So in that case every example a user can find online can be pretty much copy pasted without additional changes, which to me is HUGE. Unfortunately this causes the API to bloat quite a bit and I had to add a bunch of Serializable* classes (ex. [3]). The former bloat issue seems acceptable to me, I had thought about making a new separate API, but it seems like it is unneeded to me. The latter issue I had tried defining the generics on the method itself but the compiler can't quite figure out which method to invoke still [4]. I am still planning on adding a CacheIntStream, CacheDoubleStream and CacheLongStream interfaces as well. Without those users would need to do casts on the subsequent primitive stream if they used any of the mapTo or flatMapTo methods. Another side benefit of this refactoring is we can easily add new operations to the stream interfaces. We could add approximation methods maybe that return after a certain timeout, histogram specific support among others. I am open to whatever people think they would want added here. Unfortunately, we can't easily add in a Map.Entry stream (similar to spark PairRDD) without redoing a bunch more of the APIs and I don't know if we have time to support that. Any feedback would be great, hoping to get this ironed out soon before API freeze :) Cheers, - Will [1] https://github.com/wburns/infinispan/tree/ISPN-6272 [2] https://github.com/wburns/infinispan/commit/09734d533a445df23df94f7a053b11bc496422ec#diff-170c50a8f618af028f238109f0f1392a [3] https://github.com/wburns/infinispan/blob/ISPN-6272/core/src/main/java/org/infinispan/util/SerializableFunction.java [4] https://gist.github.com/wburns/dffe4f7543f68215f74b On Wed, Feb 17, 2016 at 8:39 AM William Burns wrote: > Actually I have a PR that will go in before the 8.2 Final release that > uses this [1]. Specifically check out the ClusterExecutor interface. It > doesn't have the issues of streams with overloading existing methods, > however it adds both overloaded variants and you can see how the tests > invoke those. > > [1] https://github.com/infinispan/infinispan/pull/4008 > > > On Wed, Feb 17, 2016 at 3:23 AM Galder Zamarre?o > wrote: > >> Hey Will, >> >> A very interesting discovery! >> >> Do you have a branch were you've tried this out? I'd like to play with it >> to see it in action and analyse the downsides more closely. >> >> Cheers, >> -- >> Galder Zamarre?o >> Infinispan, Red Hat >> >> > On 9 Feb 2016, at 17:36, William Burns wrote: >> > >> > I wanted to propose a pretty simple way of making the lambdas >> serializable by default that I stumbled upon while working on another issue. >> > >> > I noticed that in the method resolution of the compiler it does some >> nice things [1]. To be more specific when you have 2 methods with the same >> name but vary by argument types, it will attempt to pick the most >> "specific" one. Specific in this case you can think of if I can cast one >> argument type to the other but it can't be cast to this type, then this one >> is most specific. >> > >> > Here is an example, given the following class >> > >> > interface SerializableFunction extends Serializable, Function> R> >> > >> > The stream interface already defines: >> > >> > Stream map(Function mapper); >> > >> > But we could add this to the CacheStream interface >> > >> > CacheStream map(SerializableFunction mapper); >> > >> > In this case you have 2 different map methods accessible from your >> CacheStream instance. When passing a lambda the Java compiler will >> automatically choose the most specific one (in this case the >> SerializableFunction one since Function can't be cast to >> SerializableFunction). This will then make the lambda automatically >> Serializable. In this way nothing special has to be done (ie. explicit >> cast) to make the instance Serializable. >> > >> > This allows anyone using our Cache interface to immediately get lambdas >> that are Serializable when using Streams. >> > >> > The main problem however would be ambiguity because the Serialization >> would only be applied assuming you are using a defined class of CacheStream >> etc. Also this means there are 2 methods (but that seems fine to me), so >> it could cause a bit of confusion. The non serialization method is still >> helpful if people want to their own Externalizer, since their >> implementation doesn't have to implement Serializable then. >> > >> > What do you guys think? It seems like a decent compromise to me. >> > >> > - Will >> > >> > >> > >> > >> > >> > [1] >> https://docs.oracle.com/javase/specs/jls/se8/html/jls-15.html#jls-15.12.2.5 >> > >> > >> > _______________________________________________ >> > infinispan-dev mailing list >> > infinispan-dev at lists.jboss.org >> > https://lists.jboss.org/mailman/listinfo/infinispan-dev >> >> >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev at lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.jboss.org/pipermail/infinispan-dev/attachments/20160303/890eccee/attachment.html From sanne at infinispan.org Thu Mar 3 10:25:29 2016 From: sanne at infinispan.org (Sanne Grinovero) Date: Thu, 3 Mar 2016 15:25:29 +0000 Subject: [infinispan-dev] Lambda Serialization In-Reply-To: References:

Message-ID: On 3 March 2016 at 15:19, William Burns wrote: > I now have a working branch that is using this for the new CacheStream > interface [1]. > > With this it allows users to use a stream without needing any casts for any > of the intermediate or terminal operations. Note I completely revamped the > BaseStreamTest [2] So in that case every example a user can find online can > be pretty much copy pasted without additional changes, which to me is HUGE. I agree, it's HUGE! Great work! > > Unfortunately this causes the API to bloat quite a bit and I had to add a > bunch of Serializable* classes (ex. [3]). The former bloat issue seems > acceptable to me, I had thought about making a new separate API, but it > seems like it is unneeded to me. The latter issue I had tried defining the > generics on the method itself but the compiler can't quite figure out which > method to invoke still [4]. Rather than making many things serializable, did you consider to extend our collection of JBoss Marshallers? Maybe support for marshalling many of JDK's stream components could be contributed directly to the Marshaller project. > > I am still planning on adding a CacheIntStream, CacheDoubleStream and > CacheLongStream interfaces as well. Without those users would need to do > casts on the subsequent primitive stream if they used any of the > mapTo or flatMapTo methods. > > Another side benefit of this refactoring is we can easily add new operations > to the stream interfaces. We could add approximation methods maybe that > return after a certain timeout, histogram specific support among others. I > am open to whatever people think they would want added here. Unfortunately, > we can't easily add in a Map.Entry stream (similar to spark PairRDD) without > redoing a bunch more of the APIs and I don't know if we have time to support > that. > > Any feedback would be great, hoping to get this ironed out soon before API > freeze :) > > Cheers, > > - Will > > [1] https://github.com/wburns/infinispan/tree/ISPN-6272 > [2] > https://github.com/wburns/infinispan/commit/09734d533a445df23df94f7a053b11bc496422ec#diff-170c50a8f618af028f238109f0f1392a > [3] > https://github.com/wburns/infinispan/blob/ISPN-6272/core/src/main/java/org/infinispan/util/SerializableFunction.java > [4] https://gist.github.com/wburns/dffe4f7543f68215f74b > > > > On Wed, Feb 17, 2016 at 8:39 AM William Burns wrote: >> >> Actually I have a PR that will go in before the 8.2 Final release that >> uses this [1]. Specifically check out the ClusterExecutor interface. It >> doesn't have the issues of streams with overloading existing methods, >> however it adds both overloaded variants and you can see how the tests >> invoke those. >> >> [1] https://github.com/infinispan/infinispan/pull/4008 >> >> >> On Wed, Feb 17, 2016 at 3:23 AM Galder Zamarre?o >> wrote: >>> >>> Hey Will, >>> >>> A very interesting discovery! >>> >>> Do you have a branch were you've tried this out? I'd like to play with it >>> to see it in action and analyse the downsides more closely. >>> >>> Cheers, >>> -- >>> Galder Zamarre?o >>> Infinispan, Red Hat >>> >>> > On 9 Feb 2016, at 17:36, William Burns wrote: >>> > >>> > I wanted to propose a pretty simple way of making the lambdas >>> > serializable by default that I stumbled upon while working on another issue. >>> > >>> > I noticed that in the method resolution of the compiler it does some >>> > nice things [1]. To be more specific when you have 2 methods with the same >>> > name but vary by argument types, it will attempt to pick the most "specific" >>> > one. Specific in this case you can think of if I can cast one argument type >>> > to the other but it can't be cast to this type, then this one is most >>> > specific. >>> > >>> > Here is an example, given the following class >>> > >>> > interface SerializableFunction extends Serializable, Function>> > R> >>> > >>> > The stream interface already defines: >>> > >>> > Stream map(Function mapper); >>> > >>> > But we could add this to the CacheStream interface >>> > >>> > CacheStream map(SerializableFunction mapper); >>> > >>> > In this case you have 2 different map methods accessible from your >>> > CacheStream instance. When passing a lambda the Java compiler will >>> > automatically choose the most specific one (in this case the >>> > SerializableFunction one since Function can't be cast to >>> > SerializableFunction). This will then make the lambda automatically >>> > Serializable. In this way nothing special has to be done (ie. explicit >>> > cast) to make the instance Serializable. >>> > >>> > This allows anyone using our Cache interface to immediately get lambdas >>> > that are Serializable when using Streams. >>> > >>> > The main problem however would be ambiguity because the Serialization >>> > would only be applied assuming you are using a defined class of CacheStream >>> > etc. Also this means there are 2 methods (but that seems fine to me), so it >>> > could cause a bit of confusion. The non serialization method is still >>> > helpful if people want to their own Externalizer, since their implementation >>> > doesn't have to implement Serializable then. >>> > >>> > What do you guys think? It seems like a decent compromise to me. >>> > >>> > - Will >>> > >>> > >>> > >>> > >>> > >>> > [1] >>> > https://docs.oracle.com/javase/specs/jls/se8/html/jls-15.html#jls-15.12.2.5 >>> > >>> > >>> > _______________________________________________ >>> > infinispan-dev mailing list >>> > infinispan-dev at lists.jboss.org >>> > https://lists.jboss.org/mailman/listinfo/infinispan-dev >>> >>> >>> _______________________________________________ >>> infinispan-dev mailing list >>> infinispan-dev at lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev > > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev From mudokonman at gmail.com Thu Mar 3 10:40:07 2016 From: mudokonman at gmail.com (William Burns) Date: Thu, 03 Mar 2016 15:40:07 +0000 Subject: [infinispan-dev] Lambda Serialization In-Reply-To: References:

Message-ID: On Thu, Mar 3, 2016 at 10:26 AM Sanne Grinovero wrote: > On 3 March 2016 at 15:19, William Burns wrote: > > I now have a working branch that is using this for the new CacheStream > > interface [1]. > > > > With this it allows users to use a stream without needing any casts for > any > > of the intermediate or terminal operations. Note I completely revamped > the > > BaseStreamTest [2] So in that case every example a user can find online > can > > be pretty much copy pasted without additional changes, which to me is > HUGE. > > I agree, it's HUGE! Great work! > Oh I forgot to mention the 1 caveat with this approach. If the user defines their Cache or the various collections returned from it as the base type (ie. Map, ConcurrentMap, Set, Collection) this automatic serialization is lost and would require manual casting again. Normal method chaining keeps this benefit though. This seems like an acceptable and unavoidable issue to me. > > > > > Unfortunately this causes the API to bloat quite a bit and I had to add a > > bunch of Serializable* classes (ex. [3]). The former bloat issue seems > > acceptable to me, I had thought about making a new separate API, but it > > seems like it is unneeded to me. The latter issue I had tried defining > the > > generics on the method itself but the compiler can't quite figure out > which > > method to invoke still [4]. > > Rather than making many things serializable, did you consider to extend > our collection of JBoss Marshallers? > Maybe support for marshalling many of JDK's stream components could > be contributed directly to the Marshaller project. > I personally haven't looked at this aspect. To be honest, I was leaning on Galder a bit more here, since he is much more familiar with the Marshalling code. If we can do this instead I think it would be even bigger. Unfortunately I don't know how feasible it is. > > > > > I am still planning on adding a CacheIntStream, CacheDoubleStream and > > CacheLongStream interfaces as well. Without those users would need to do > > casts on the subsequent primitive stream if they used any of the > > mapTo or flatMapTo methods. > > > > Another side benefit of this refactoring is we can easily add new > operations > > to the stream interfaces. We could add approximation methods maybe that > > return after a certain timeout, histogram specific support among > others. I > > am open to whatever people think they would want added here. > Unfortunately, > > we can't easily add in a Map.Entry stream (similar to spark PairRDD) > without > > redoing a bunch more of the APIs and I don't know if we have time to > support > > that. > > > > Any feedback would be great, hoping to get this ironed out soon before > API > > freeze :) > > > > Cheers, > > > > - Will > > > > [1] https://github.com/wburns/infinispan/tree/ISPN-6272 > > [2] > > > https://github.com/wburns/infinispan/commit/09734d533a445df23df94f7a053b11bc496422ec#diff-170c50a8f618af028f238109f0f1392a > > [3] > > > https://github.com/wburns/infinispan/blob/ISPN-6272/core/src/main/java/org/infinispan/util/SerializableFunction.java > > [4] https://gist.github.com/wburns/dffe4f7543f68215f74b > > > > > > > > On Wed, Feb 17, 2016 at 8:39 AM William Burns > wrote: > >> > >> Actually I have a PR that will go in before the 8.2 Final release that > >> uses this [1]. Specifically check out the ClusterExecutor interface. > It > >> doesn't have the issues of streams with overloading existing methods, > >> however it adds both overloaded variants and you can see how the tests > >> invoke those. > >> > >> [1] https://github.com/infinispan/infinispan/pull/4008 > >> > >> > >> On Wed, Feb 17, 2016 at 3:23 AM Galder Zamarre?o > >> wrote: > >>> > >>> Hey Will, > >>> > >>> A very interesting discovery! > >>> > >>> Do you have a branch were you've tried this out? I'd like to play with > it > >>> to see it in action and analyse the downsides more closely. > >>> > >>> Cheers, > >>> -- > >>> Galder Zamarre?o > >>> Infinispan, Red Hat > >>> > >>> > On 9 Feb 2016, at 17:36, William Burns wrote: > >>> > > >>> > I wanted to propose a pretty simple way of making the lambdas > >>> > serializable by default that I stumbled upon while working on > another issue. > >>> > > >>> > I noticed that in the method resolution of the compiler it does some > >>> > nice things [1]. To be more specific when you have 2 methods with > the same > >>> > name but vary by argument types, it will attempt to pick the most > "specific" > >>> > one. Specific in this case you can think of if I can cast one > argument type > >>> > to the other but it can't be cast to this type, then this one is most > >>> > specific. > >>> > > >>> > Here is an example, given the following class > >>> > > >>> > interface SerializableFunction extends Serializable, > Function >>> > R> > >>> > > >>> > The stream interface already defines: > >>> > > >>> > Stream map(Function mapper); > >>> > > >>> > But we could add this to the CacheStream interface > >>> > > >>> > CacheStream map(SerializableFunction > mapper); > >>> > > >>> > In this case you have 2 different map methods accessible from your > >>> > CacheStream instance. When passing a lambda the Java compiler will > >>> > automatically choose the most specific one (in this case the > >>> > SerializableFunction one since Function can't be cast to > >>> > SerializableFunction). This will then make the lambda automatically > >>> > Serializable. In this way nothing special has to be done (ie. > explicit > >>> > cast) to make the instance Serializable. > >>> > > >>> > This allows anyone using our Cache interface to immediately get > lambdas > >>> > that are Serializable when using Streams. > >>> > > >>> > The main problem however would be ambiguity because the Serialization > >>> > would only be applied assuming you are using a defined class of > CacheStream > >>> > etc. Also this means there are 2 methods (but that seems fine to > me), so it > >>> > could cause a bit of confusion. The non serialization method is > still > >>> > helpful if people want to their own Externalizer, since their > implementation > >>> > doesn't have to implement Serializable then. > >>> > > >>> > What do you guys think? It seems like a decent compromise to me. > >>> > > >>> > - Will > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > [1] > >>> > > https://docs.oracle.com/javase/specs/jls/se8/html/jls-15.html#jls-15.12.2.5 > >>> > > >>> > > >>> > _______________________________________________ > >>> > infinispan-dev mailing list > >>> > infinispan-dev at lists.jboss.org > >>> > https://lists.jboss.org/mailman/listinfo/infinispan-dev > >>> > >>> > >>> _______________________________________________ > >>> infinispan-dev mailing list > >>> infinispan-dev at lists.jboss.org > >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev > > > > > > _______________________________________________ > > infinispan-dev mailing list > > infinispan-dev at lists.jboss.org > > https://lists.jboss.org/mailman/listinfo/infinispan-dev > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.jboss.org/pipermail/infinispan-dev/attachments/20160303/958191c4/attachment.html From vjuranek at redhat.com Thu Mar 3 10:54:52 2016 From: vjuranek at redhat.com (Vojtech Juranek) Date: Thu, 03 Mar 2016 16:54:52 +0100 Subject: [infinispan-dev] Infinispan 8 presentation on DevConf.cz 2016 In-Reply-To: References: <761888892.15555969.1454946567441.JavaMail.zimbra@redhat.com> <2201094.oE6FtqIMlI@dhcp-10-40-5-198.brq.redhat.com> Message-ID: <4875847.8Vk2cQq9QQ@localhost.localdomain> On Wednesday 02 March 2016 14:23:36 Gustavo Fernandes wrote: > Nice Spark Streaming + Infinispan demo! This would make a great quickstart > project! thanks. I'll try to improve it little bit and convert to a quickstart. Vojta -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 473 bytes Desc: This is a digitally signed message part. Url : http://lists.jboss.org/pipermail/infinispan-dev/attachments/20160303/62155f25/attachment.bin From galder at redhat.com Fri Mar 4 08:44:35 2016 From: galder at redhat.com (=?utf-8?Q?Galder_Zamarre=C3=B1o?=) Date: Fri, 4 Mar 2016 14:44:35 +0100 Subject: [infinispan-dev] Shade source jar issue? Message-ID: Hi guys, I've been trying to test out a embedded shaded jar snapshot but I'm having issues with discrepancies between jar file and source file. It appears that the source file is not updated, e.g. If I take the jar, expand it and grep, I see this: $ javap org.infinispan.CacheStream | grep collect public abstract R1 collect(java.util.stream.Collector); public abstract R1 collect(org.infinispan.util.SerializableSupplier, org.infinispan.util.SerializableBiConsumer, org.infinispan.util.SerializableBiConsumer); When I look at the source jar, I see: $ grep collect ./org/infinispan/CacheStream.java ... R1 collect(Collector collector); Are you aware of any issues with shade source jars? Cheers, -- Galder Zamarre?o Infinispan, Red Hat From galder at redhat.com Fri Mar 4 08:52:06 2016 From: galder at redhat.com (=?utf-8?Q?Galder_Zamarre=C3=B1o?=) Date: Fri, 4 Mar 2016 14:52:06 +0100 Subject: [infinispan-dev] Shade source jar issue? In-Reply-To: References: Message-ID: Actually, I have same issue building core/ with: $ mvn -DskipTests=true install Looking into it... Cheers, -- Galder Zamarre?o Infinispan, Red Hat > On 4 Mar 2016, at 14:44, Galder Zamarre?o wrote: > > Hi guys, > > I've been trying to test out a embedded shaded jar snapshot but I'm having issues with discrepancies between jar file and source file. > > It appears that the source file is not updated, e.g. > > If I take the jar, expand it and grep, I see this: > > $ javap org.infinispan.CacheStream | grep collect > public abstract R1 collect(java.util.stream.Collector); > public abstract R1 collect(org.infinispan.util.SerializableSupplier, org.infinispan.util.SerializableBiConsumer, org.infinispan.util.SerializableBiConsumer); > > When I look at the source jar, I see: > > $ grep collect ./org/infinispan/CacheStream.java > ... > R1 collect(Collector collector); > > Are you aware of any issues with shade source jars? > > Cheers, > -- > Galder Zamarre?o > Infinispan, Red Hat > > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev From mudokonman at gmail.com Fri Mar 4 08:52:56 2016 From: mudokonman at gmail.com (William Burns) Date: Fri, 04 Mar 2016 13:52:56 +0000 Subject: [infinispan-dev] Lambda Serialization In-Reply-To: References:

Message-ID: On Thu, Mar 3, 2016 at 10:40 AM William Burns wrote: > On Thu, Mar 3, 2016 at 10:26 AM Sanne Grinovero > wrote: > >> On 3 March 2016 at 15:19, William Burns wrote: >> > I now have a working branch that is using this for the new CacheStream >> > interface [1]. >> > >> > With this it allows users to use a stream without needing any casts for >> any >> > of the intermediate or terminal operations. Note I completely revamped >> the >> > BaseStreamTest [2] So in that case every example a user can find >> online can >> > be pretty much copy pasted without additional changes, which to me is >> HUGE. >> >> I agree, it's HUGE! Great work! >> > > Oh I forgot to mention the 1 caveat with this approach. If the user > defines their Cache or the various collections returned from it as the base > type (ie. Map, ConcurrentMap, Set, Collection) this automatic serialization > is lost and would require manual casting again. Normal method chaining > keeps this benefit though. This seems like an acceptable and unavoidable > issue to me. > Sorry also these changes still require the use of the CacheCollectors when using [1] like we have had in the past it. It just removes the need for all of the various casting (ie. (Serializable & Function) for the other methods. [1] https://docs.oracle.com/javase/8/docs/api/java/util/stream/Stream.html#collect-java.util.stream.Collector- > > >> >> > >> > Unfortunately this causes the API to bloat quite a bit and I had to add >> a >> > bunch of Serializable* classes (ex. [3]). The former bloat issue seems >> > acceptable to me, I had thought about making a new separate API, but it >> > seems like it is unneeded to me. The latter issue I had tried defining >> the >> > generics on the method itself but the compiler can't quite figure out >> which >> > method to invoke still [4]. >> >> Rather than making many things serializable, did you consider to extend >> our collection of JBoss Marshallers? >> Maybe support for marshalling many of JDK's stream components could >> be contributed directly to the Marshaller project. >> > > I personally haven't looked at this aspect. To be honest, I was leaning > on Galder a bit more here, since he is much more familiar with the > Marshalling code. If we can do this instead I think it would be even > bigger. Unfortunately I don't know how feasible it is. > > >> >> > >> > I am still planning on adding a CacheIntStream, CacheDoubleStream and >> > CacheLongStream interfaces as well. Without those users would need to >> do >> > casts on the subsequent primitive stream if they used any of the >> > mapTo or flatMapTo methods. >> > >> > Another side benefit of this refactoring is we can easily add new >> operations >> > to the stream interfaces. We could add approximation methods maybe that >> > return after a certain timeout, histogram specific support among >> others. I >> > am open to whatever people think they would want added here. >> Unfortunately, >> > we can't easily add in a Map.Entry stream (similar to spark PairRDD) >> without >> > redoing a bunch more of the APIs and I don't know if we have time to >> support >> > that. >> > >> > Any feedback would be great, hoping to get this ironed out soon before >> API >> > freeze :) >> > >> > Cheers, >> > >> > - Will >> > >> > [1] https://github.com/wburns/infinispan/tree/ISPN-6272 >> > [2] >> > >> https://github.com/wburns/infinispan/commit/09734d533a445df23df94f7a053b11bc496422ec#diff-170c50a8f618af028f238109f0f1392a >> > [3] >> > >> https://github.com/wburns/infinispan/blob/ISPN-6272/core/src/main/java/org/infinispan/util/SerializableFunction.java >> > [4] https://gist.github.com/wburns/dffe4f7543f68215f74b >> > >> > >> > >> > On Wed, Feb 17, 2016 at 8:39 AM William Burns >> wrote: >> >> >> >> Actually I have a PR that will go in before the 8.2 Final release that >> >> uses this [1]. Specifically check out the ClusterExecutor interface. >> It >> >> doesn't have the issues of streams with overloading existing methods, >> >> however it adds both overloaded variants and you can see how the tests >> >> invoke those. >> >> >> >> [1] https://github.com/infinispan/infinispan/pull/4008 >> >> >> >> >> >> On Wed, Feb 17, 2016 at 3:23 AM Galder Zamarre?o >> >> wrote: >> >>> >> >>> Hey Will, >> >>> >> >>> A very interesting discovery! >> >>> >> >>> Do you have a branch were you've tried this out? I'd like to play >> with it >> >>> to see it in action and analyse the downsides more closely. >> >>> >> >>> Cheers, >> >>> -- >> >>> Galder Zamarre?o >> >>> Infinispan, Red Hat >> >>> >> >>> > On 9 Feb 2016, at 17:36, William Burns >> wrote: >> >>> > >> >>> > I wanted to propose a pretty simple way of making the lambdas >> >>> > serializable by default that I stumbled upon while working on >> another issue. >> >>> > >> >>> > I noticed that in the method resolution of the compiler it does some >> >>> > nice things [1]. To be more specific when you have 2 methods with >> the same >> >>> > name but vary by argument types, it will attempt to pick the most >> "specific" >> >>> > one. Specific in this case you can think of if I can cast one >> argument type >> >>> > to the other but it can't be cast to this type, then this one is >> most >> >>> > specific. >> >>> > >> >>> > Here is an example, given the following class >> >>> > >> >>> > interface SerializableFunction extends Serializable, >> Function> >>> > R> >> >>> > >> >>> > The stream interface already defines: >> >>> > >> >>> > Stream map(Function mapper); >> >>> > >> >>> > But we could add this to the CacheStream interface >> >>> > >> >>> > CacheStream map(SerializableFunction >> mapper); >> >>> > >> >>> > In this case you have 2 different map methods accessible from your >> >>> > CacheStream instance. When passing a lambda the Java compiler will >> >>> > automatically choose the most specific one (in this case the >> >>> > SerializableFunction one since Function can't be cast to >> >>> > SerializableFunction). This will then make the lambda automatically >> >>> > Serializable. In this way nothing special has to be done (ie. >> explicit >> >>> > cast) to make the instance Serializable. >> >>> > >> >>> > This allows anyone using our Cache interface to immediately get >> lambdas >> >>> > that are Serializable when using Streams. >> >>> > >> >>> > The main problem however would be ambiguity because the >> Serialization >> >>> > would only be applied assuming you are using a defined class of >> CacheStream >> >>> > etc. Also this means there are 2 methods (but that seems fine to >> me), so it >> >>> > could cause a bit of confusion. The non serialization method is >> still >> >>> > helpful if people want to their own Externalizer, since their >> implementation >> >>> > doesn't have to implement Serializable then. >> >>> > >> >>> > What do you guys think? It seems like a decent compromise to me. >> >>> > >> >>> > - Will >> >>> > >> >>> > >> >>> > >> >>> > >> >>> > >> >>> > [1] >> >>> > >> https://docs.oracle.com/javase/specs/jls/se8/html/jls-15.html#jls-15.12.2.5 >> >>> > >> >>> > >> >>> > _______________________________________________ >> >>> > infinispan-dev mailing list >> >>> > infinispan-dev at lists.jboss.org >> >>> > https://lists.jboss.org/mailman/listinfo/infinispan-dev >> >>> >> >>> >> >>> _______________________________________________ >> >>> infinispan-dev mailing list >> >>> infinispan-dev at lists.jboss.org >> >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >> > >> > >> > _______________________________________________ >> > infinispan-dev mailing list >> > infinispan-dev at lists.jboss.org >> > https://lists.jboss.org/mailman/listinfo/infinispan-dev >> >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev at lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.jboss.org/pipermail/infinispan-dev/attachments/20160304/33e5d4a7/attachment.html From galder at redhat.com Fri Mar 4 09:07:20 2016 From: galder at redhat.com (=?utf-8?Q?Galder_Zamarre=C3=B1o?=) Date: Fri, 4 Mar 2016 15:07:20 +0100 Subject: [infinispan-dev] Shade source jar issue? In-Reply-To: References:

Message-ID: <89425F26-3028-4E46-AA46-E8F76ADAB03E@redhat.com> Needs -Pextras :) -- Galder Zamarre?o Infinispan, Red Hat > On 4 Mar 2016, at 14:52, Galder Zamarre?o wrote: > > Actually, I have same issue building core/ with: > > $ mvn -DskipTests=true install > > Looking into it... > > Cheers, > -- > Galder Zamarre?o > Infinispan, Red Hat > >> On 4 Mar 2016, at 14:44, Galder Zamarre?o wrote: >> >> Hi guys, >> >> I've been trying to test out a embedded shaded jar snapshot but I'm having issues with discrepancies between jar file and source file. >> >> It appears that the source file is not updated, e.g. >> >> If I take the jar, expand it and grep, I see this: >> >> $ javap org.infinispan.CacheStream | grep collect >> public abstract R1 collect(java.util.stream.Collector); >> public abstract R1 collect(org.infinispan.util.SerializableSupplier, org.infinispan.util.SerializableBiConsumer, org.infinispan.util.SerializableBiConsumer); >> >> When I look at the source jar, I see: >> >> $ grep collect ./org/infinispan/CacheStream.java >> ... >> R1 collect(Collector collector); >> >> Are you aware of any issues with shade source jars? >> >> Cheers, >> -- >> Galder Zamarre?o >> Infinispan, Red Hat >> >> >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev at lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev > > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev From galder at redhat.com Fri Mar 4 10:07:39 2016 From: galder at redhat.com (=?utf-8?Q?Galder_Zamarre=C3=B1o?=) Date: Fri, 4 Mar 2016 16:07:39 +0100 Subject: [infinispan-dev] Lambda Serialization In-Reply-To: References:

Message-ID: -- Galder Zamarre?o Infinispan, Red Hat > On 3 Mar 2016, at 16:40, William Burns wrote: > > > > On Thu, Mar 3, 2016 at 10:26 AM Sanne Grinovero wrote: > On 3 March 2016 at 15:19, William Burns wrote: > > I now have a working branch that is using this for the new CacheStream > > interface [1]. > > > > With this it allows users to use a stream without needing any casts for any > > of the intermediate or terminal operations. Note I completely revamped the > > BaseStreamTest [2] So in that case every example a user can find online can > > be pretty much copy pasted without additional changes, which to me is HUGE. > > I agree, it's HUGE! Great work! ^ Awesome stuff! I've just tried it out and it's a huge improvement particularly for out-of-the-box experience and live demos. > > Oh I forgot to mention the 1 caveat with this approach. If the user defines their Cache or the various collections returned from it as the base type (ie. Map, ConcurrentMap, Set, Collection) this automatic serialization is lost and would require manual casting again. Normal method chaining keeps this benefit though. This seems like an acceptable and unavoidable issue to me. > > > > > > Unfortunately this causes the API to bloat quite a bit and I had to add a > > bunch of Serializable* classes (ex. [3]). The former bloat issue seems > > acceptable to me, I had thought about making a new separate API, but it > > seems like it is unneeded to me. The latter issue I had tried defining the > > generics on the method itself but the compiler can't quite figure out which > > method to invoke still [4]. > > Rather than making many things serializable, did you consider to extend > our collection of JBoss Marshallers? > Maybe support for marshalling many of JDK's stream components could > be contributed directly to the Marshaller project. > > I personally haven't looked at this aspect. To be honest, I was leaning on Galder a bit more here, since he is much more familiar with the Marshalling code. If we can do this instead I think it would be even bigger. Unfortunately I don't know how feasible it is. I'm not sure this is that important. With Will's changes we're trying to improve the out-of-the-box experience by being able to marshall lambdas as they are. The user can still provide Externalizers for them to make them ultra fast. Also, an Externalizer has no default read/write object implementation, so not sure how we could implement what you suggest in a generic way. > > > > > > I am still planning on adding a CacheIntStream, CacheDoubleStream and > > CacheLongStream interfaces as well. Without those users would need to do > > casts on the subsequent primitive stream if they used any of the > > mapTo or flatMapTo methods. > > > > Another side benefit of this refactoring is we can easily add new operations > > to the stream interfaces. We could add approximation methods maybe that > > return after a certain timeout, histogram specific support among others. I > > am open to whatever people think they would want added here. Unfortunately, > > we can't easily add in a Map.Entry stream (similar to spark PairRDD) without > > redoing a bunch more of the APIs and I don't know if we have time to support > > that. > > > > Any feedback would be great, hoping to get this ironed out soon before API > > freeze :) > > > > Cheers, > > > > - Will > > > > [1] https://github.com/wburns/infinispan/tree/ISPN-6272 > > [2] > > https://github.com/wburns/infinispan/commit/09734d533a445df23df94f7a053b11bc496422ec#diff-170c50a8f618af028f238109f0f1392a > > [3] > > https://github.com/wburns/infinispan/blob/ISPN-6272/core/src/main/java/org/infinispan/util/SerializableFunction.java > > [4] https://gist.github.com/wburns/dffe4f7543f68215f74b > > > > > > > > On Wed, Feb 17, 2016 at 8:39 AM William Burns wrote: > >> > >> Actually I have a PR that will go in before the 8.2 Final release that > >> uses this [1]. Specifically check out the ClusterExecutor interface. It > >> doesn't have the issues of streams with overloading existing methods, > >> however it adds both overloaded variants and you can see how the tests > >> invoke those. > >> > >> [1] https://github.com/infinispan/infinispan/pull/4008 > >> > >> > >> On Wed, Feb 17, 2016 at 3:23 AM Galder Zamarre?o > >> wrote: > >>> > >>> Hey Will, > >>> > >>> A very interesting discovery! > >>> > >>> Do you have a branch were you've tried this out? I'd like to play with it > >>> to see it in action and analyse the downsides more closely. > >>> > >>> Cheers, > >>> -- > >>> Galder Zamarre?o > >>> Infinispan, Red Hat > >>> > >>> > On 9 Feb 2016, at 17:36, William Burns wrote: > >>> > > >>> > I wanted to propose a pretty simple way of making the lambdas > >>> > serializable by default that I stumbled upon while working on another issue. > >>> > > >>> > I noticed that in the method resolution of the compiler it does some > >>> > nice things [1]. To be more specific when you have 2 methods with the same > >>> > name but vary by argument types, it will attempt to pick the most "specific" > >>> > one. Specific in this case you can think of if I can cast one argument type > >>> > to the other but it can't be cast to this type, then this one is most > >>> > specific. > >>> > > >>> > Here is an example, given the following class > >>> > > >>> > interface SerializableFunction extends Serializable, Function >>> > R> > >>> > > >>> > The stream interface already defines: > >>> > > >>> > Stream map(Function mapper); > >>> > > >>> > But we could add this to the CacheStream interface > >>> > > >>> > CacheStream map(SerializableFunction mapper); > >>> > > >>> > In this case you have 2 different map methods accessible from your > >>> > CacheStream instance. When passing a lambda the Java compiler will > >>> > automatically choose the most specific one (in this case the > >>> > SerializableFunction one since Function can't be cast to > >>> > SerializableFunction). This will then make the lambda automatically > >>> > Serializable. In this way nothing special has to be done (ie. explicit > >>> > cast) to make the instance Serializable. > >>> > > >>> > This allows anyone using our Cache interface to immediately get lambdas > >>> > that are Serializable when using Streams. > >>> > > >>> > The main problem however would be ambiguity because the Serialization > >>> > would only be applied assuming you are using a defined class of CacheStream > >>> > etc. Also this means there are 2 methods (but that seems fine to me), so it > >>> > could cause a bit of confusion. The non serialization method is still > >>> > helpful if people want to their own Externalizer, since their implementation > >>> > doesn't have to implement Serializable then. > >>> > > >>> > What do you guys think? It seems like a decent compromise to me. > >>> > > >>> > - Will > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > [1] > >>> > https://docs.oracle.com/javase/specs/jls/se8/html/jls-15.html#jls-15.12.2.5 > >>> > > >>> > > >>> > _______________________________________________ > >>> > infinispan-dev mailing list > >>> > infinispan-dev at lists.jboss.org > >>> > https://lists.jboss.org/mailman/listinfo/infinispan-dev > >>> > >>> > >>> _______________________________________________ > >>> infinispan-dev mailing list > >>> infinispan-dev at lists.jboss.org > >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev > > > > > > _______________________________________________ > > infinispan-dev mailing list > > infinispan-dev at lists.jboss.org > > https://lists.jboss.org/mailman/listinfo/infinispan-dev > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev > _______________________________________________ > infinispan-dev mailing list > infinispan-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev From slaskawi at redhat.com Mon Mar 7 09:31:34 2016 From: slaskawi at redhat.com (Sebastian Laskawiec) Date: Mon, 7 Mar 2016 15:31:34 +0100 Subject: [infinispan-dev] Uber jars - how do we want to use them Message-ID: Hey! Uber jars have been around for quite a while but some old problems are still biting our ankles. The biggest problem is JBoss Logging... But before diving into the technical discussion, I would like to clarify how do we treat Uber Jars and how other (let's call them "extra") modules should integrate with them... How Uber Jars look like: - All Uber Jar dependencies are declared as: - provided - those are APIs which should be available on your container - optional - all other dependencies - compile - JTA (I'm not sure why, but probably because this is a required API). I think it should be changed to provided at some point. - Uber Jars have (almost) no dependencies (apart from JTA) - Uber Jars contain everything to get a standard application running with ISPN in a single jar Now the biggest question is how to integrate other extra modules with them - for example Spring or Cache Stores? Here are the options: - Not integrate at all... users should use standard (small) jars in such use cases - This one has a big impact on our users, so I would assume no - All extra modules should shade everything they need and provide their own Uber Jars (like Spring Uber Jar = ISPN Core + Spring) - This one might be really hard to maintain, so I would also vote for no. - All extra modules should integrate with small jars and Uber Jars (but not both at the same time). An example application might want to add Spring-embedded and Infinispan-embedded to it's classpath and it's ready to go. Alternatively it could add Spring-embedded + infinispan-core + whatever is needed. Adding both infinispan-embedded and infinispan-core is prohibited. - Easy for the users, a bit harder for us. My vote goes here. Could you please share some thoughts whether this makes sense or not? Once we achieve common understanding - I'll propose some technical solutions. Thanks Sebastian -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.jboss.org/pipermail/infinispan-dev/attachments/20160307/24081d2c/attachment.html From ttarrant at redhat.com Mon Mar 7 15:01:36 2016 From: ttarrant at redhat.com (Tristan Tarrant) Date: Mon, 7 Mar 2016 21:01:36 +0100 Subject: [infinispan-dev] Infinispan Weekly IRC meeting logs 2016-03-07 Message-ID: <56DDDE20.1020505@redhat.com> Hi all, get the meeting logs from http://transcripts.jboss.org/meeting/irc.freenode.org/infinispan/2016/infinispan.2016-03-07-15.01.log.html Cheers Tristan -- Tristan Tarrant Infinispan Lead JBoss, a division of Red Hat From pedro at infinispan.org Tue Mar 8 18:07:46 2016 From: pedro at infinispan.org (Pedro Ruivo) Date: Tue, 8 Mar 2016 23:07:46 +0000 Subject: [infinispan-dev] Infinispan 8.2.0.Final is out! Message-ID: <56DF5B42.5040503@infinispan.org> Dear community, The Infinispan team is proud to announce the release of Infinispan 8.2.0.Final. Checkout what is new in this version here: http://blog.infinispan.org/2016/03/infinispan-820final-is-out.html Cheers, The Infinispan team. From sanne at infinispan.org Fri Mar 11 07:07:55 2016 From: sanne at infinispan.org (Sanne Grinovero) Date: Fri, 11 Mar 2016 12:07:55 +0000 Subject: [infinispan-dev] Removal of auto-detection of indexed entities Message-ID: When booting Infinispan Server v. 8.2.0.Final I see this message being logged: ISPN000403: No indexable classes were defined for this indexed cache; switching to autodetection (support for autodetection will be removed in Infinispan 9.0). Does it mean we can finally remove the infamous, super-complex to maintain and generally hated "feature" in Hibernate Search to dynamically add new entities on the fly? From anistor at redhat.com Fri Mar 11 08:00:05 2016 From: anistor at redhat.com (Adrian Nistor) Date: Fri, 11 Mar 2016 15:00:05 +0200 Subject: [infinispan-dev] Removal of auto-detection of indexed entities In-Reply-To: References: Message-ID: <56E2C155.2000402@redhat.com> Hi Sanne, yes, but there is a small unsolved issue ATM and the implementation needs more polishing before we get to Ispn 9.0; infinispan and probably search too will need a bit more tweaking. Current approach is that the search factory is first created with 0 classes and then after initialization of the cache and the index manager has completed we reconfigure the search factory again, exactly once, and all known indexed classes are added at once. We cannot add the indexed classes at the initial creation time because that would cause infinispan directory to try to create the needed caches (for locking, metadata,...) while the user's cache has not even finished initializing. These leads to deadlock in our component registry. This approach if far more efficient and less error prone than the previous but still not ideal. Adrian On 03/11/2016 02:07 PM, Sanne Grinovero wrote: > When booting Infinispan Server v. 8.2.0.Final I see this message being logged: > > ISPN000403: No indexable classes were defined for this indexed cache; > switching to autodetection (support for autodetection will be removed > in Infinispan 9.0). > > Does it mean we can finally remove the infamous, super-complex to > maintain and generally hated "feature" in Hibernate Search to > dynamically add new entities on the fly? > _______________________________________________ > infinispan-dev mailing list > infinispan-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev From sanne at infinispan.org Fri Mar 11 08:13:48 2016 From: sanne at infinispan.org (Sanne Grinovero) Date: Fri, 11 Mar 2016 13:13:48 +0000 Subject: [infinispan-dev] Removal of auto-detection of indexed entities In-Reply-To: <56E2C155.2000402@redhat.com> References: <56E2C155.2000402@redhat.com> Message-ID: On 11 March 2016 at 13:00, Adrian Nistor wrote: > Hi Sanne, > > yes, but there is a small unsolved issue ATM and the implementation > needs more polishing before we get to Ispn 9.0; infinispan and probably > search too will need a bit more tweaking. > > Current approach is that the search factory is first created with 0 > classes and then after initialization of the cache and the index manager > has completed we reconfigure the search factory again, exactly once, and > all known indexed classes are added at once. We cannot add the indexed > classes at the initial creation time because that would cause infinispan > directory to try to create the needed caches (for locking, metadata,...) > while the user's cache has not even finished initializing. These leads > to deadlock in our component registry. This approach if far more > efficient and less error prone than the previous but still not ideal. Sounds like this would be solved by initializing the SearchIntegrator lazily (only once) ? Let us know what the Search team could do, it would be very nice for us to be able to drop this requirement as it makes any other change we're working on more complex to implement. Thanks! Sanne > > Adrian > > On 03/11/2016 02:07 PM, Sanne Grinovero wrote: >> When booting Infinispan Server v. 8.2.0.Final I see this message being logged: >> >> ISPN000403: No indexable classes were defined for this indexed cache; >> switching to autodetection (support for autodetection will be removed >> in Infinispan 9.0). >> >> Does it mean we can finally remove the infamous, super-complex to >> maintain and generally hated "feature" in Hibernate Search to >> dynamically add new entities on the fly? >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev at lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev From gunnar at hibernate.org Fri Mar 11 08:42:27 2016 From: gunnar at hibernate.org (Gunnar Morling) Date: Fri, 11 Mar 2016 14:42:27 +0100 Subject: [infinispan-dev] Removal of auto-detection of indexed entities In-Reply-To: References: <56E2C155.2000402@redhat.com> Message-ID: What Sanne says :) This is causing great pain in HSEARCH, so if we managed to get rid of this, that'd be just awesome! 2016-03-11 14:13 GMT+01:00 Sanne Grinovero : > On 11 March 2016 at 13:00, Adrian Nistor wrote: >> Hi Sanne, >> >> yes, but there is a small unsolved issue ATM and the implementation >> needs more polishing before we get to Ispn 9.0; infinispan and probably >> search too will need a bit more tweaking. >> >> Current approach is that the search factory is first created with 0 >> classes and then after initialization of the cache and the index manager >> has completed we reconfigure the search factory again, exactly once, and >> all known indexed classes are added at once. We cannot add the indexed >> classes at the initial creation time because that would cause infinispan >> directory to try to create the needed caches (for locking, metadata,...) >> while the user's cache has not even finished initializing. These leads >> to deadlock in our component registry. This approach if far more >> efficient and less error prone than the previous but still not ideal. > > Sounds like this would be solved by initializing the SearchIntegrator > lazily (only once) ? > > Let us know what the Search team could do, it would be very nice for > us to be able to > drop this requirement as it makes any other change we're working on more complex > to implement. > > Thanks! > Sanne > >> >> Adrian >> >> On 03/11/2016 02:07 PM, Sanne Grinovero wrote: >>> When booting Infinispan Server v. 8.2.0.Final I see this message being logged: >>> >>> ISPN000403: No indexable classes were defined for this indexed cache; >>> switching to autodetection (support for autodetection will be removed >>> in Infinispan 9.0). >>> >>> Does it mean we can finally remove the infamous, super-complex to >>> maintain and generally hated "feature" in Hibernate Search to >>> dynamically add new entities on the fly? >>> _______________________________________________ >>> infinispan-dev mailing list >>> infinispan-dev at lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >> >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev at lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev From anistor at redhat.com Fri Mar 11 09:02:45 2016 From: anistor at redhat.com (Adrian Nistor) Date: Fri, 11 Mar 2016 16:02:45 +0200 Subject: [infinispan-dev] Removal of auto-detection of indexed entities In-Reply-To: References: <56E2C155.2000402@redhat.com> Message-ID: <56E2D005.9010408@redhat.com> The laziness could be tried first in Infinispan, then if it does not work we might need some help from hibernate-search. That might save hibernate-search from adding another hack for infinispan. Thanks! On 03/11/2016 03:13 PM, Sanne Grinovero wrote: > On 11 March 2016 at 13:00, Adrian Nistor wrote: >> Hi Sanne, >> >> yes, but there is a small unsolved issue ATM and the implementation >> needs more polishing before we get to Ispn 9.0; infinispan and probably >> search too will need a bit more tweaking. >> >> Current approach is that the search factory is first created with 0 >> classes and then after initialization of the cache and the index manager >> has completed we reconfigure the search factory again, exactly once, and >> all known indexed classes are added at once. We cannot add the indexed >> classes at the initial creation time because that would cause infinispan >> directory to try to create the needed caches (for locking, metadata,...) >> while the user's cache has not even finished initializing. These leads >> to deadlock in our component registry. This approach if far more >> efficient and less error prone than the previous but still not ideal. > Sounds like this would be solved by initializing the SearchIntegrator > lazily (only once) ? > > Let us know what the Search team could do, it would be very nice for > us to be able to > drop this requirement as it makes any other change we're working on more complex > to implement. > > Thanks! > Sanne > >> Adrian >> >> On 03/11/2016 02:07 PM, Sanne Grinovero wrote: >>> When booting Infinispan Server v. 8.2.0.Final I see this message being logged: >>> >>> ISPN000403: No indexable classes were defined for this indexed cache; >>> switching to autodetection (support for autodetection will be removed >>> in Infinispan 9.0). >>> >>> Does it mean we can finally remove the infamous, super-complex to >>> maintain and generally hated "feature" in Hibernate Search to >>> dynamically add new entities on the fly? >>> _______________________________________________ >>> infinispan-dev mailing list >>> infinispan-dev at lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev at lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev > _______________________________________________ > infinispan-dev mailing list > infinispan-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev From anistor at redhat.com Fri Mar 11 09:09:27 2016 From: anistor at redhat.com (Adrian Nistor) Date: Fri, 11 Mar 2016 16:09:27 +0200 Subject: [infinispan-dev] Removal of auto-detection of indexed entities In-Reply-To: <56E2D005.9010408@redhat.com> References: <56E2C155.2000402@redhat.com> <56E2D005.9010408@redhat.com> Message-ID: <56E2D197.30305@redhat.com> BTW, these changes were added a bit late in the Infinispan 8.2 development so I had to keep them at the minimum, otherwise I would have added lazy search factory initialization already. But since we already have some strange random mystery deadlocks at server start (which do not seem related, until I'm proven wrong :)) I was reluctant to do it. On 03/11/2016 04:02 PM, Adrian Nistor wrote: > The laziness could be tried first in Infinispan, then if it does not > work we might need some help from hibernate-search. That might save > hibernate-search from adding another hack for infinispan. > > Thanks! > > On 03/11/2016 03:13 PM, Sanne Grinovero wrote: >> On 11 March 2016 at 13:00, Adrian Nistor wrote: >>> Hi Sanne, >>> >>> yes, but there is a small unsolved issue ATM and the implementation >>> needs more polishing before we get to Ispn 9.0; infinispan and probably >>> search too will need a bit more tweaking. >>> >>> Current approach is that the search factory is first created with 0 >>> classes and then after initialization of the cache and the index manager >>> has completed we reconfigure the search factory again, exactly once, and >>> all known indexed classes are added at once. We cannot add the indexed >>> classes at the initial creation time because that would cause infinispan >>> directory to try to create the needed caches (for locking, metadata,...) >>> while the user's cache has not even finished initializing. These leads >>> to deadlock in our component registry. This approach if far more >>> efficient and less error prone than the previous but still not ideal. >> Sounds like this would be solved by initializing the SearchIntegrator >> lazily (only once) ? >> >> Let us know what the Search team could do, it would be very nice for >> us to be able to >> drop this requirement as it makes any other change we're working on more complex >> to implement. >> >> Thanks! >> Sanne >> >>> Adrian >>> >>> On 03/11/2016 02:07 PM, Sanne Grinovero wrote: >>>> When booting Infinispan Server v. 8.2.0.Final I see this message being logged: >>>> >>>> ISPN000403: No indexable classes were defined for this indexed cache; >>>> switching to autodetection (support for autodetection will be removed >>>> in Infinispan 9.0). >>>> >>>> Does it mean we can finally remove the infamous, super-complex to >>>> maintain and generally hated "feature" in Hibernate Search to >>>> dynamically add new entities on the fly? >>>> _______________________________________________ >>>> infinispan-dev mailing list >>>> infinispan-dev at lists.jboss.org >>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>> _______________________________________________ >>> infinispan-dev mailing list >>> infinispan-dev at lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev at lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev > _______________________________________________ > infinispan-dev mailing list > infinispan-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev From galder at redhat.com Fri Mar 11 12:20:22 2016 From: galder at redhat.com (=?utf-8?Q?Galder_Zamarre=C3=B1o?=) Date: Fri, 11 Mar 2016 19:20:22 +0200 Subject: [infinispan-dev] Uber jars - how do we want to use them In-Reply-To: References: Message-ID: An HTML attachment was scrubbed... URL: http://lists.jboss.org/pipermail/infinispan-dev/attachments/20160311/fd1542e1/attachment.html From ttarrant at redhat.com Mon Mar 14 03:42:53 2016 From: ttarrant at redhat.com (Tristan Tarrant) Date: Mon, 14 Mar 2016 08:42:53 +0100 Subject: [infinispan-dev] Uber jars - how do we want to use them In-Reply-To: References:

Message-ID: <56E66B7D.70901@redhat.com> On 11/03/2016 18:20, Galder Zamarre?o wrote: > Are uber jars really that useful? From my own experience they often get The number of users who don't use a dependency management system (Maven, Ivy, Gradle) is quite a lot higher than you'd expect. Tristan -- Tristan Tarrant Infinispan Lead JBoss, a division of Red Hat From slaskawi at redhat.com Mon Mar 14 06:21:25 2016 From: slaskawi at redhat.com (Sebastian Laskawiec) Date: Mon, 14 Mar 2016 11:21:25 +0100 Subject: [infinispan-dev] Uber jars - how do we want to use them In-Reply-To: <56E66B7D.70901@redhat.com> References:

<56E66B7D.70901@redhat.com> Message-ID: I took a look at Nexus download statistics and Infinispan Uberjars are about 7% of our downloads (of course this calculation has been based on our JBoss Nexus instance and we have no data from other mirrors). So, once we are clear how Uber Jars should work... let's take a look at one of the problems: - Many of our modules use JBoss Logging - Uber jars relocate dependencies (like the one above) into infinispan package. The result for JBoss Logging would be: infinispan.org.jboss.logging* - Most of our modules are compiled with small jars (like Spring integration), so the compile dependency for Spring integration is org.jboss.logging (not infinispan.org.jboss.logging) - If someone wants to use Spring with Uber Jars (which should be absolutely fine), he gets a clash... Spring module can't find JBoss Logging There are several options to solve that: 1. Stop relocating dependencies - Pro: everything should work without problems - Con: our dependencies will clash with client's dependencies 2. Wrap 3rd party dependency with our own wrappers. In case of JBoss Logging - develop our own facade and put it in common. - Pro: 3rd party dependencies will not leak between our modules - Con: A lot of effort and rather small gain 3. Develop 3rd party dependencies as Uber Jars. In case of Spring - we will have a Spring Uber Jar. I think Dan put this idea on the table. - Pro: Lots of Uber Jars for everyone - Con: How to use small jars in this scenario? 4. Mark all module dependencies as provided and let users add small/uber jars themselves. - Pro: It should give us the best results with the smallest amount of time - Cons: Not all scenarios will be solved. We will also need to stop relocating some dependencies (like logging facades which are used almost everywhere) Having in mind our download statistics I would go for #4 (even though it doesn't solve the problem entirely). However #2 seems like a better fit for long term maintenance and we should proceed with this direction if number of Uber Jars users will grow (I can set a reminder in a 3-4 months time). Does it sound reasonable? Thanks Sebastian On Mon, Mar 14, 2016 at 8:42 AM, Tristan Tarrant wrote: > > > On 11/03/2016 18:20, Galder Zamarre?o wrote: > > Are uber jars really that useful? From my own experience they often get > > The number of users who don't use a dependency management system (Maven, > Ivy, Gradle) is quite a lot higher than you'd expect. > > Tristan > -- > Tristan Tarrant > Infinispan Lead > JBoss, a division of Red Hat > _______________________________________________ > infinispan-dev mailing list > infinispan-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.jboss.org/pipermail/infinispan-dev/attachments/20160314/7f0010b9/attachment.html From galder at redhat.com Mon Mar 14 06:47:59 2016 From: galder at redhat.com (=?windows-1252?Q?Galder_Zamarre=F1o?=) Date: Mon, 14 Mar 2016 11:47:59 +0100 Subject: [infinispan-dev] Uber jars - how do we want to use them In-Reply-To: <56E66B7D.70901@redhat.com> References:

<56E66B7D.70901@redhat.com> Message-ID: <9EC129F2-B9C8-44BB-9EEE-F6899F0AC49F@redhat.com> -- Galder Zamarre?o Infinispan, Red Hat > On 14 Mar 2016, at 08:42, Tristan Tarrant wrote: > > > > On 11/03/2016 18:20, Galder Zamarre?o wrote: >> Are uber jars really that useful? From my own experience they often get > > The number of users who don't use a dependency management system (Maven, > Ivy, Gradle) is quite a lot higher than you'd expect. Good point, so the uber jars are mostly for those use cases then? That could be the recommendation: - Using dependency management? Use normal jars, otherwise use uber jars Cheers, > > Tristan > -- > Tristan Tarrant > Infinispan Lead > JBoss, a division of Red Hat > _______________________________________________ > infinispan-dev mailing list > infinispan-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev From sanne at infinispan.org Mon Mar 14 06:50:00 2016 From: sanne at infinispan.org (Sanne Grinovero) Date: Mon, 14 Mar 2016 10:50:00 +0000 Subject: [infinispan-dev] Uber jars - how do we want to use them In-Reply-To: References:

<56E66B7D.70901@redhat.com> Message-ID: Hi, uber jars were introduced as an answer to complaints such as "there are too many jars" but I still think this was the wrong answer.. too many issues so please stop this: it's not helping usability to understand which jars are needed, and it makes things worse with runtime errors not matching source code. It was my impression that uber jars were a temporary solution to improve things, while the real improvements - to actually need less at runtime, or provide better errors pointing out what's needed - would take more time. So rather than stubbornly spend time maintaining this approach which is so obviously broken, I'd rather see time spent in pursuing the better improvements so that we can finally drop this. Sebastian: I din't understand the idea behind proposal 2#: what does it meant to provide our own facade to JBoss Logging? One will still need to have JBoss Logging at runtime, right? Thanks, Sanne On 14 March 2016 at 10:21, Sebastian Laskawiec wrote: > I took a look at Nexus download statistics and Infinispan Uberjars are about > 7% of our downloads (of course this calculation has been based on our JBoss > Nexus instance and we have no data from other mirrors). > > So, once we are clear how Uber Jars should work... let's take a look at one > of the problems: > > Many of our modules use JBoss Logging > Uber jars relocate dependencies (like the one above) into infinispan > package. The result for JBoss Logging would be: > infinispan.org.jboss.logging* > Most of our modules are compiled with small jars (like Spring integration), > so the compile dependency for Spring integration is org.jboss.logging (not > infinispan.org.jboss.logging) > If someone wants to use Spring with Uber Jars (which should be absolutely > fine), he gets a clash... Spring module can't find JBoss Logging > > There are several options to solve that: > > Stop relocating dependencies > > Pro: everything should work without problems > Con: our dependencies will clash with client's dependencies > > Wrap 3rd party dependency with our own wrappers. In case of JBoss Logging - > develop our own facade and put it in common. > > Pro: 3rd party dependencies will not leak between our modules > Con: A lot of effort and rather small gain > > Develop 3rd party dependencies as Uber Jars. In case of Spring - we will > have a Spring Uber Jar. I think Dan put this idea on the table. > > Pro: Lots of Uber Jars for everyone > Con: How to use small jars in this scenario? > > Mark all module dependencies as provided and let users add small/uber jars > themselves. > > Pro: It should give us the best results with the smallest amount of time > Cons: Not all scenarios will be solved. We will also need to stop relocating > some dependencies (like logging facades which are used almost everywhere) > > Having in mind our download statistics I would go for #4 (even though it > doesn't solve the problem entirely). However #2 seems like a better fit for > long term maintenance and we should proceed with this direction if number of > Uber Jars users will grow (I can set a reminder in a 3-4 months time). > > Does it sound reasonable? > > Thanks > Sebastian > > On Mon, Mar 14, 2016 at 8:42 AM, Tristan Tarrant > wrote: >> >> >> >> On 11/03/2016 18:20, Galder Zamarre?o wrote: >> > Are uber jars really that useful? From my own experience they often get >> >> The number of users who don't use a dependency management system (Maven, >> Ivy, Gradle) is quite a lot higher than you'd expect. >> >> Tristan >> -- >> Tristan Tarrant >> Infinispan Lead >> JBoss, a division of Red Hat >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev at lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev > > > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev From sanne at infinispan.org Mon Mar 14 06:57:07 2016 From: sanne at infinispan.org (Sanne Grinovero) Date: Mon, 14 Mar 2016 10:57:07 +0000 Subject: [infinispan-dev] Uber jars - how do we want to use them In-Reply-To: References:

<56E66B7D.70901@redhat.com> Message-ID: To make a more practical proposal: give me a way to start a local, non transactional Cache which doesn't require any of: - JGroups - JBoss Marshalling - JTA API - Infinispan Commons If I don't have - for example - JGroups and am starting a clustered cache, I will see an exception "Enabling Clustering on Cache {1} requires a Transport Module such as org.infinispan:transport-jgroups:9.0.0.Final. No transport module found on classpath." On 14 March 2016 at 10:50, Sanne Grinovero wrote: > Hi, > uber jars were introduced as an answer to complaints such as "there > are too many jars" but I still think this was the wrong answer.. too > many issues so please stop this: it's not helping usability to > understand which jars are needed, and it makes things worse with > runtime errors not matching source code. > > It was my impression that uber jars were a temporary solution to > improve things, while the real improvements - to actually need less at > runtime, or provide better errors pointing out what's needed - would > take more time. > > So rather than stubbornly spend time maintaining this approach which > is so obviously broken, I'd rather see time spent in pursuing the > better improvements so that we can finally drop this. > > Sebastian: I din't understand the idea behind proposal 2#: what does > it meant to provide our own facade to JBoss Logging? One will still > need to have JBoss Logging at runtime, right? > > Thanks, > Sanne > > On 14 March 2016 at 10:21, Sebastian Laskawiec wrote: >> I took a look at Nexus download statistics and Infinispan Uberjars are about >> 7% of our downloads (of course this calculation has been based on our JBoss >> Nexus instance and we have no data from other mirrors). >> >> So, once we are clear how Uber Jars should work... let's take a look at one >> of the problems: >> >> Many of our modules use JBoss Logging >> Uber jars relocate dependencies (like the one above) into infinispan >> package. The result for JBoss Logging would be: >> infinispan.org.jboss.logging* >> Most of our modules are compiled with small jars (like Spring integration), >> so the compile dependency for Spring integration is org.jboss.logging (not >> infinispan.org.jboss.logging) >> If someone wants to use Spring with Uber Jars (which should be absolutely >> fine), he gets a clash... Spring module can't find JBoss Logging >> >> There are several options to solve that: >> >> Stop relocating dependencies >> >> Pro: everything should work without problems >> Con: our dependencies will clash with client's dependencies >> >> Wrap 3rd party dependency with our own wrappers. In case of JBoss Logging - >> develop our own facade and put it in common. >> >> Pro: 3rd party dependencies will not leak between our modules >> Con: A lot of effort and rather small gain >> >> Develop 3rd party dependencies as Uber Jars. In case of Spring - we will >> have a Spring Uber Jar. I think Dan put this idea on the table. >> >> Pro: Lots of Uber Jars for everyone >> Con: How to use small jars in this scenario? >> >> Mark all module dependencies as provided and let users add small/uber jars >> themselves. >> >> Pro: It should give us the best results with the smallest amount of time >> Cons: Not all scenarios will be solved. We will also need to stop relocating >> some dependencies (like logging facades which are used almost everywhere) >> >> Having in mind our download statistics I would go for #4 (even though it >> doesn't solve the problem entirely). However #2 seems like a better fit for >> long term maintenance and we should proceed with this direction if number of >> Uber Jars users will grow (I can set a reminder in a 3-4 months time). >> >> Does it sound reasonable? >> >> Thanks >> Sebastian >> >> On Mon, Mar 14, 2016 at 8:42 AM, Tristan Tarrant >> wrote: >>> >>> >>> >>> On 11/03/2016 18:20, Galder Zamarre?o wrote: >>> > Are uber jars really that useful? From my own experience they often get >>> >>> The number of users who don't use a dependency management system (Maven, >>> Ivy, Gradle) is quite a lot higher than you'd expect. >>> >>> Tristan >>> -- >>> Tristan Tarrant >>> Infinispan Lead >>> JBoss, a division of Red Hat >>> _______________________________________________ >>> infinispan-dev mailing list >>> infinispan-dev at lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >> >> >> >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev at lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev From dan.berindei at gmail.com Mon Mar 14 08:03:52 2016 From: dan.berindei at gmail.com (Dan Berindei) Date: Mon, 14 Mar 2016 14:03:52 +0200 Subject: [infinispan-dev] Uber jars - how do we want to use them In-Reply-To: References:

<56E66B7D.70901@redhat.com> Message-ID: On Mon, Mar 14, 2016 at 12:21 PM, Sebastian Laskawiec wrote: > I took a look at Nexus download statistics and Infinispan Uberjars are about > 7% of our downloads (of course this calculation has been based on our JBoss > Nexus instance and we have no data from other mirrors). > > So, once we are clear how Uber Jars should work... let's take a look at one > of the problems: > > Many of our modules use JBoss Logging > Uber jars relocate dependencies (like the one above) into infinispan > package. The result for JBoss Logging would be: > infinispan.org.jboss.logging* > Most of our modules are compiled with small jars (like Spring integration), > so the compile dependency for Spring integration is org.jboss.logging (not > infinispan.org.jboss.logging) > If someone wants to use Spring with Uber Jars (which should be absolutely > fine), he gets a clash... Spring module can't find JBoss Logging > > There are several options to solve that: > > Stop relocating dependencies > > Pro: everything should work without problems > Con: our dependencies will clash with client's dependencies > > Wrap 3rd party dependency with our own wrappers. In case of JBoss Logging - > develop our own facade and put it in common. > > Pro: 3rd party dependencies will not leak between our modules > Con: A lot of effort and rather small gain > > Develop 3rd party dependencies as Uber Jars. In case of Spring - we will > have a Spring Uber Jar. I think Dan put this idea on the table. > > Pro: Lots of Uber Jars for everyone > Con: How to use small jars in this scenario? > > Mark all module dependencies as provided and let users add small/uber jars > themselves. > > Pro: It should give us the best results with the smallest amount of time > Cons: Not all scenarios will be solved. We will also need to stop relocating > some dependencies (like logging facades which are used almost everywhere) > > Having in mind our download statistics I would go for #4 (even though it > doesn't solve the problem entirely). However #2 seems like a better fit for > long term maintenance and we should proceed with this direction if number of > Uber Jars users will grow (I can set a reminder in a 3-4 months time). > > Does it sound reasonable? +1 for option 4. The embedded uber-jar doesn't expose any class in org.jboss.logging, so it doesn't provide jboss-logging. The Spring module should depend on jboss-logging itself, not assume that it's going to be provided by another module. One complication we have now is that we've made jboss-logging a part of our API: one module's i18n Log interface can extend another module's Log, and that means it extends the BasicLogger from jboss-logging. I believe the Log interfaces should be private, and each module should extend BasicLogger directly. Cheers Dan From emmanuel at hibernate.org Mon Mar 14 10:02:01 2016 From: emmanuel at hibernate.org (Emmanuel Bernard) Date: Mon, 14 Mar 2016 15:02:01 +0100 Subject: [infinispan-dev] Uber jars - how do we want to use them In-Reply-To: References:

<56E66B7D.70901@redhat.com> Message-ID: <20160314140201.GQ47051@hibernate.org> On Mon 2016-03-14 11:21, Sebastian Laskawiec wrote: > I took a look at Nexus download statistics and Infinispan Uberjars are > about 7% of our downloads (of course this calculation has been based on our > JBoss Nexus instance and we have no data from other mirrors). It's actually the opposite, the way nexus stats work, the stats you see from our nexus are actually the maven central stats minus direct hits from our nexus instance. Don't ask. From pedro at infinispan.org Mon Mar 14 15:14:26 2016 From: pedro at infinispan.org (Pedro Ruivo) Date: Mon, 14 Mar 2016 19:14:26 +0000 Subject: [infinispan-dev] Distributed Counter Discussion Message-ID: <56E70D92.1010206@infinispan.org> Hi everybody, Discussion about distributed counters. == Public API == interface Counter String getName() //counter name long get() //current value. may return stale value due to concurrent operations to other nodes. void increment() //async or sync increment. default add(1) void decrement() //async or sync decrement. default add(-1) void add(long) //async or sync add. void reset() //resets to initial value Note: Tried to make the interface as simple as possible with support for sync and async operations. To avoid any confusion, I consider an async operation as happening somewhat in the future, i.e. eventually increments/decrements. The sync operation happens somewhat during the method execution. interface AtomiCounter extends Counter long addAndGet() //adds a returns the new value. sync operation long incrementAndGet() //increments and returns the new value. sync operation. default addAndGet(1) long decrementAndGet() //decrements and returns the new value. sync operation. default addAndGet(-1) interface AdvancedCounter extends Counter long getMin/MaxThreshold() //returns the min and max threshold value void add/removeListener() //adds a listener that is invoked when the value change. Can be extended to notify when it is "reseted" and when the threshold is reached. Note: should this interface be splitted? == Details == This is what I have in mind. Two counter managers: one based on JGroups counter and another one based on Infinispan cache. The first one creates AtomicCounters and it first perfectly. All counters are created with an initial value (zero by default) The second generates counters with all the options available. It can mix sync/async operation and all counters will be in the same cache. The cache will be configure by us and it would be an internal cache. This will use all the features available in the cache. Configuration-wise, I'm thinking about 2 parameters: number of backups and timeout (for sync operations). So, comment bellow and let me know alternatives, improvement or if I missed something. ps. I also consider implement a counter based on JGroups-raft but I believe it is an overkill. ps2. sorry for the long email :( I tried to be shorter as possible. Cheers, Pedro From sanne at infinispan.org Mon Mar 14 17:27:44 2016 From: sanne at infinispan.org (Sanne Grinovero) Date: Mon, 14 Mar 2016 21:27:44 +0000 Subject: [infinispan-dev] Distributed Counter Discussion In-Reply-To: <56E70D92.1010206@infinispan.org> References: <56E70D92.1010206@infinispan.org> Message-ID: Great starting point! Some comments inline: On 14 March 2016 at 19:14, Pedro Ruivo wrote: > Hi everybody, > > Discussion about distributed counters. > > == Public API == > > interface Counter As a user, how do I get a Counter instance? From a the CacheContainer interface? Will they have their own configuration section in the configuration file? > > String getName() //counter name > long get() //current value. may return stale value due to concurrent > operations to other nodes. This is what puzzles me the most. I'm not sure if the feature is actually useful, unless we can clearly state how far outdated the value could be. I think a slightly more formal definition would be in order. For example I think it would be acceptable to say that this will return a value from the range of values the primary owner of this counter was holding in the timeframe between the method is being invoked and the time the value is returned. Could it optionally be integrated with Total Order ? Transactions? > void increment() //async or sync increment. default add(1) > void decrement() //async or sync decrement. default add(-1) > void add(long) //async or sync add. > void reset() //resets to initial value > > Note: Tried to make the interface as simple as possible with support for > sync and async operations. To avoid any confusion, I consider an async > operation as happening somewhat in the future, i.e. eventually > increments/decrements. > The sync operation happens somewhat during the method execution. > > interface AtomiCounter extends Counter > > long addAndGet() //adds a returns the new value. sync operation > long incrementAndGet() //increments and returns the new value. sync > operation. default addAndGet(1) > long decrementAndGet() //decrements and returns the new value. sync > operation. default addAndGet(-1) > > interface AdvancedCounter extends Counter > > long getMin/MaxThreshold() //returns the min and max threshold value "threshold" ?? > void add/removeListener() //adds a listener that is invoked when the > value change. Can be extended to notify when it is "reseted" and when > the threshold is reached. > > Note: should this interface be splitted? I'd prefer a single interface, with reduced redundancy. For example, is there really a benefit in having a "void increment()" and also a "long addAndGet()" ? [Besides The fact that only the first one can benefit of an async option] Besides, I am no longer sure that it's good thing that methods in Infinispan can be async vs sync depending on configuration switches; I'd rather make it explicit in the signature and simplify the configuration by removing such a flag. Making the methods which are async-capable to look "explicitly async" should also allow us to add completeable futures & similar. > > == Details == > > This is what I have in mind. Two counter managers: one based on JGroups > counter and another one based on Infinispan cache. > The first one creates AtomicCounters and it first perfectly. All > counters are created with an initial value (zero by default) > The second generates counters with all the options available. It can mix > sync/async operation and all counters will be in the same cache. The > cache will be configure by us and it would be an internal cache. This > will use all the features available in the cache. Rather than a split organized on the backing implementation details, I'd prefer to see a split based on purpose of the counter. For example JGroups has different styles of counters: one might need one to do an atomic increment which is monotonically increasing across the whole cluster - from the point of view from an omniscent observer - but in many cases we just need the generation of a guarantee of a monotonic unique number: that allows for example to have each node pre-allocate a range of values and pre-fetch a new block while its running out of values. For some systems this is not acceptable because a failing server might result in some of its allocated numbers to not ever be used, so creating gaps; for others it's not acceptable that the figures returned by the "monotonic counter" are not monotonic across each other when confronting results from different cluster nodes, but for many use cases that's acceptable. My point being that we need to be able to choose between these possible semantics. As an Infinispan consumer being able to express such details is essential, while having to choose between "backed by JGroups counters" or not is irrelevant and actually makes it harder. In terms of Infinispan integration, the most important point I'd like to see is persistence: i.e. make sure you can store them in a CacheStore. How we do that efficiently will also affect design; for example I was expecting to see a Counter as something which is stored in a Cache, so inheriting details such as configured CacheStore(s) and number of owners. Thanks for starting this! Sanne > > Configuration-wise, I'm thinking about 2 parameters: number of backups > and timeout (for sync operations). > > So, comment bellow and let me know alternatives, improvement or if I > missed something. > > ps. I also consider implement a counter based on JGroups-raft but I > believe it is an overkill. > ps2. sorry for the long email :( I tried to be shorter as possible. > > Cheers, > Pedro > _______________________________________________ > infinispan-dev mailing list > infinispan-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev From rhauch at redhat.com Mon Mar 14 18:17:25 2016 From: rhauch at redhat.com (Randall Hauch) Date: Mon, 14 Mar 2016 17:17:25 -0500 Subject: [infinispan-dev] Distributed Counter Discussion In-Reply-To: <56E70D92.1010206@infinispan.org> References: <56E70D92.1010206@infinispan.org> Message-ID: <80FC3F6D-F7F8-4CBE-90CC-FEED3ADE86F1@redhat.com> What are the requirements? What are the distributed counters for? Is the counter to be monotonically increasing? Can there be any missed values? Does the counter need to increment and decrement? What is the *smallest* API you need initially? There are two choices when implementing a distributed counter: use central coordination (like JGroups counters), or use independent counters on separate machines that will eventually converge to the correct value (CRDTs). Coordinated counters are expensive and therefore slow, and can suffer from problems during network or cluster problems. For example, what happens during a split brain? OTOH, CRDTs are decentralized so therefore are very fast, easily merged, and fault tolerant; they?re excellent when counting things that are occurring independently and therefore may be more suited for monitoring/metrics/accumulators/etc. Both have very different behaviors under ideal and failure scenarios, have different performance and consistency guarantees, and are useful in different scenarios. Make sure you choose accordingly. For information about CRDTs, make sure you?ve read the CRDT paper by Shapiro: http://hal.upmc.fr/inria-00555588/document Randall > On Mar 14, 2016, at 2:14 PM, Pedro Ruivo wrote: > > Hi everybody, > > Discussion about distributed counters. > > == Public API == > > interface Counter > > String getName() //counter name > long get() //current value. may return stale value due to concurrent > operations to other nodes. > void increment() //async or sync increment. default add(1) > void decrement() //async or sync decrement. default add(-1) > void add(long) //async or sync add. > void reset() //resets to initial value > > Note: Tried to make the interface as simple as possible with support for > sync and async operations. To avoid any confusion, I consider an async > operation as happening somewhat in the future, i.e. eventually > increments/decrements. > The sync operation happens somewhat during the method execution. > > interface AtomiCounter extends Counter > > long addAndGet() //adds a returns the new value. sync operation > long incrementAndGet() //increments and returns the new value. sync > operation. default addAndGet(1) > long decrementAndGet() //decrements and returns the new value. sync > operation. default addAndGet(-1) > > interface AdvancedCounter extends Counter > > long getMin/MaxThreshold() //returns the min and max threshold value > void add/removeListener() //adds a listener that is invoked when the > value change. Can be extended to notify when it is "reseted" and when > the threshold is reached. > > Note: should this interface be splitted? > > == Details == > > This is what I have in mind. Two counter managers: one based on JGroups > counter and another one based on Infinispan cache. > The first one creates AtomicCounters and it first perfectly. All > counters are created with an initial value (zero by default) > The second generates counters with all the options available. It can mix > sync/async operation and all counters will be in the same cache. The > cache will be configure by us and it would be an internal cache. This > will use all the features available in the cache. > > Configuration-wise, I'm thinking about 2 parameters: number of backups > and timeout (for sync operations). > > So, comment bellow and let me know alternatives, improvement or if I > missed something. > > ps. I also consider implement a counter based on JGroups-raft but I > believe it is an overkill. > ps2. sorry for the long email :( I tried to be shorter as possible. > > Cheers, > Pedro > _______________________________________________ > infinispan-dev mailing list > infinispan-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.jboss.org/pipermail/infinispan-dev/attachments/20160314/fcdd1c15/attachment.html From bban at redhat.com Tue Mar 15 03:00:55 2016 From: bban at redhat.com (Bela Ban) Date: Tue, 15 Mar 2016 08:00:55 +0100 Subject: [infinispan-dev] Distributed Counter Discussion In-Reply-To: <56E70D92.1010206@infinispan.org> References: <56E70D92.1010206@infinispan.org> Message-ID: <56E7B327.2050604@redhat.com> What are the requirements? To offer a cluster wide counter from an Infinispan cache? Or is this going to be used internally, by Infinispan? I've implemented 2 counters, in JGroups and in jgroups-raft. Here are their properties: #1 JGroups counter: - Cluster wide _named_ counters with atomicity guarantees - A counter is always managed by the current coordinator - In a cluster split, you can end up with different counter instances (one per coordinator) - When a cluster partition heals, the current coordinator's counter 'wins', ie. values from the other counters are lost, even if they were updated last #2 jgroups-raft counter: - Counter is managed by the leader - A counter will not be accessible when there's no leader (no majority): this is CP in CAP - Counters cannot diverge in split brain scenarios; there's always at most 1 counter (or 0) If you can live with multiple counters being present in light of network partitiond, then pick #1 as this is faster than #2 (which requires disk writes for updates), and is always available, whereas #2 can be non-available. I'm thinking about changing the way #1 works. Currently, it uses the next-in-line to backup a counter value on each update. I'd like to remove this and replace it with a reconciliation round when the coordinator changes: the new coordinator asks all members for their current counter values. Requires a bit more work on a view change, but less in steady state mode. Perhaps I can make it configurable. On 14/03/16 20:14, Pedro Ruivo wrote: > Hi everybody, > > Discussion about distributed counters. > > == Public API == > > interface Counter > > String getName() //counter name > long get() //current value. may return stale value due to concurrent > operations to other nodes. > void increment() //async or sync increment. default add(1) > void decrement() //async or sync decrement. default add(-1) > void add(long) //async or sync add. > void reset() //resets to initial value > > Note: Tried to make the interface as simple as possible with support for > sync and async operations. To avoid any confusion, I consider an async > operation as happening somewhat in the future, i.e. eventually > increments/decrements. > The sync operation happens somewhat during the method execution. > > interface AtomiCounter extends Counter > > long addAndGet() //adds a returns the new value. sync operation > long incrementAndGet() //increments and returns the new value. sync > operation. default addAndGet(1) > long decrementAndGet() //decrements and returns the new value. sync > operation. default addAndGet(-1) > > interface AdvancedCounter extends Counter > > long getMin/MaxThreshold() //returns the min and max threshold value > void add/removeListener() //adds a listener that is invoked when the > value change. Can be extended to notify when it is "reseted" and when > the threshold is reached. > > Note: should this interface be splitted? > > == Details == > > This is what I have in mind. Two counter managers: one based on JGroups > counter and another one based on Infinispan cache. > The first one creates AtomicCounters and it first perfectly. All > counters are created with an initial value (zero by default) > The second generates counters with all the options available. It can mix > sync/async operation and all counters will be in the same cache. The > cache will be configure by us and it would be an internal cache. This > will use all the features available in the cache. > > Configuration-wise, I'm thinking about 2 parameters: number of backups > and timeout (for sync operations). > > So, comment bellow and let me know alternatives, improvement or if I > missed something. > > ps. I also consider implement a counter based on JGroups-raft but I > believe it is an overkill. > ps2. sorry for the long email :( I tried to be shorter as possible. > > Cheers, > Pedro > _______________________________________________ > infinispan-dev mailing list > infinispan-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev > -- Bela Ban, JGroups lead (http://www.jgroups.org) From bban at redhat.com Tue Mar 15 03:12:30 2016 From: bban at redhat.com (Bela Ban) Date: Tue, 15 Mar 2016 08:12:30 +0100 Subject: [infinispan-dev] Distributed Counter Discussion In-Reply-To: <80FC3F6D-F7F8-4CBE-90CC-FEED3ADE86F1@redhat.com> References: <56E70D92.1010206@infinispan.org> <80FC3F6D-F7F8-4CBE-90CC-FEED3ADE86F1@redhat.com> Message-ID: <56E7B5DE.6010606@redhat.com> On 14/03/16 23:17, Randall Hauch wrote: > What are the requirements? What are the distributed counters for? Is the > counter to be monotonically increasing? Can there be any missed values? > Does the counter need to increment and decrement? What is the *smallest* > API you need initially? > > There are two choices when implementing a distributed counter: use > central coordination (like JGroups counters), or use independent > counters on separate machines that will eventually converge to the > correct value (CRDTs). Coordinated counters are expensive and therefore > slow, and can suffer from problems during network or cluster problems. The question is what do you get for this? If your app can't afford duplicate counter values during a network partition, then - yes - there is some overhead. CRDTs won't be able to guarantee this property. OTOH CRDTs are fast when you only care about some sort of eventual consistency, and don't need 'hard' consistency. > For example, what happens during a split brain? OTOH, CRDTs are > decentralized so therefore are very fast, easily merged, and fault > tolerant; Yes, CRDTS are AP whereas jgroups-raft counters are CP. JGroups counters, otoh, are CRAP (consistent, reliable, available and partition-aware. Take the last sentence with a grain of salt :-) > they?re excellent when counting things that are occurring > independently and therefore may be more suited for > monitoring/metrics/accumulators/etc. Both have very different behaviors > under ideal and failure scenarios, have different performance and > consistency guarantees, and are useful in different scenarios. Make sure > you choose accordingly. > > For information about CRDTs, make sure you?ve read the CRDT paper by > Shapiro: http://hal.upmc.fr/inria-00555588/document > > Randall > >> On Mar 14, 2016, at 2:14 PM, Pedro Ruivo > > wrote: >> >> Hi everybody, >> >> Discussion about distributed counters. >> >> == Public API == >> >> interface Counter >> >> String getName() //counter name >> long get()//current value. may return stale value due to concurrent >> operations to other nodes. >> void increment() //async or sync increment. default add(1) >> void decrement() //async or sync decrement. default add(-1) >> void add(long) //async or sync add. >> void reset() //resets to initial value >> >> Note: Tried to make the interface as simple as possible with support for >> sync and async operations. To avoid any confusion, I consider an async >> operation as happening somewhat in the future, i.e. eventually >> increments/decrements. >> The sync operation happens somewhat during the method execution. >> >> interface AtomiCounter extends Counter >> >> long addAndGet() //adds a returns the new value. sync operation >> long incrementAndGet() //increments and returns the new value. sync >> operation. default addAndGet(1) >> long decrementAndGet() //decrements and returns the new value. sync >> operation. default addAndGet(-1) >> >> interface AdvancedCounter extends Counter >> >> long getMin/MaxThreshold() //returns the min and max threshold value >> void add/removeListener() //adds a listener that is invoked when the >> value change. Can be extended to notify when it is "reseted" and when >> the threshold is reached. >> >> Note: should this interface be splitted? >> >> == Details == >> >> This is what I have in mind. Two counter managers: one based on JGroups >> counter and another one based on Infinispan cache. >> The first one creates AtomicCounters and it first perfectly. All >> counters are created with an initial value (zero by default) >> The second generates counters with all the options available. It can mix >> sync/async operation and all counters will be in the same cache. The >> cache will be configure by us and it would be an internal cache. This >> will use all the features available in the cache. >> >> Configuration-wise, I'm thinking about 2 parameters: number of backups >> and timeout (for sync operations). >> >> So, comment bellow and let me know alternatives, improvement or if I >> missed something. >> >> ps. I also consider implement a counter based on JGroups-raft but I >> believe it is an overkill. >> ps2. sorry for the long email :( I tried to be shorter as possible. >> >> Cheers, >> Pedro >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev at lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev > > > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev > -- Bela Ban, JGroups lead (http://www.jgroups.org) From ttarrant at redhat.com Tue Mar 15 03:38:37 2016 From: ttarrant at redhat.com (Tristan Tarrant) Date: Tue, 15 Mar 2016 08:38:37 +0100 Subject: [infinispan-dev] Distributed Counter Discussion In-Reply-To: <56E7B327.2050604@redhat.com> References: <56E70D92.1010206@infinispan.org> <56E7B327.2050604@redhat.com> Message-ID: <56E7BBFD.4080403@redhat.com> On 15/03/2016 08:00, Bela Ban wrote: > What are the requirements? To offer a cluster wide counter from an > Infinispan cache? Or is this going to be used internally, by Infinispan? The starting point is a requirement by apiman as a way to manage throttling/quotas. Currently their requirement is not particularly strict (i.e. synchronous), but they would like the ability to reset a counter and possibly to receive an event when a certain threshold is reached. Obviously we'd like to cater to more use-cases, so the design should take that into account. Tristan -- Tristan Tarrant Infinispan Lead JBoss, a division of Red Hat From rvansa at redhat.com Tue Mar 15 04:26:27 2016 From: rvansa at redhat.com (Radim Vansa) Date: Tue, 15 Mar 2016 09:26:27 +0100 Subject: [infinispan-dev] Distributed Counter Discussion In-Reply-To: References: <56E70D92.1010206@infinispan.org> Message-ID: <56E7C733.8080809@redhat.com> I second Sanne's opinion about the sync/async API: make the API express the synchronicity directly. I would even propose that for synchronous methods, there's only the CompletableFuture or CompletableFuture variant; we are already reaching the concurrency limits of application that lets threads block, so let's give user a hint that he should use such notification API. If anyone prefers the sync variant, he can always use get(). Let's settle on some nomenclature, too, because these JGroups-counters and RAFT-counters don't have commonly known properties. It almost seems that the term *counter* is so overloaded that we shouldn't use it all. There is a widely known j.u.c.AtomicLong, so if we want to implement that (strict JMM-like properties), let's call it AtomicLong. Does not have to follow j.u.c.AtomicLong API, but it's definitely CP. Something providing unique values should be called Sequence. This will be probably the one batching ranges (therefore possibly with gaps). By default non-monotonic, but monotonicity could be a ctor arg (in a similar way as fairness is set for j.u.concurrent.* classes). As for CRDTs, I can't imagine how this could be easily built on top of current 'passive' cache (without any syncing). But as for names *CRDT counter* is explanatory enough. If someone needs quotas, let's create Quota according to their needs (soft and hard threshold, fluent operation below soft, some jitter when above that). It seems that this will be closest to Sequence by reserving some ranges. Don't let them shoot themselves into foot with some liger counter. And I hope you'll build these on top of the functional API, with at most one RPC per operation. Radim On 03/14/2016 10:27 PM, Sanne Grinovero wrote: > Great starting point! > Some comments inline: > > On 14 March 2016 at 19:14, Pedro Ruivo wrote: >> Hi everybody, >> >> Discussion about distributed counters. >> >> == Public API == >> >> interface Counter > As a user, how do I get a Counter instance? From a the CacheContainer interface? > > Will they have their own configuration section in the configuration file? > >> String getName() //counter name >> long get() //current value. may return stale value due to concurrent >> operations to other nodes. > This is what puzzles me the most. I'm not sure if the feature is > actually useful, unless we can clearly state how far outdated the > value could be. > > I think a slightly more formal definition would be in order. For > example I think it would be acceptable to say that this will return a > value from the range of values the primary owner of this counter was > holding in the timeframe between the method is being invoked and the > time the value is returned. > > Could it optionally be integrated with Total Order ? Transactions? > >> void increment() //async or sync increment. default add(1) >> void decrement() //async or sync decrement. default add(-1) >> void add(long) //async or sync add. >> void reset() //resets to initial value >> >> Note: Tried to make the interface as simple as possible with support for >> sync and async operations. To avoid any confusion, I consider an async >> operation as happening somewhat in the future, i.e. eventually >> increments/decrements. >> The sync operation happens somewhat during the method execution. >> >> interface AtomiCounter extends Counter >> >> long addAndGet() //adds a returns the new value. sync operation >> long incrementAndGet() //increments and returns the new value. sync >> operation. default addAndGet(1) >> long decrementAndGet() //decrements and returns the new value. sync >> operation. default addAndGet(-1) >> >> interface AdvancedCounter extends Counter >> >> long getMin/MaxThreshold() //returns the min and max threshold value > "threshold" ?? > >> void add/removeListener() //adds a listener that is invoked when the >> value change. Can be extended to notify when it is "reseted" and when >> the threshold is reached. >> >> Note: should this interface be splitted? > I'd prefer a single interface, with reduced redundancy. > For example, is there really a benefit in having a "void increment()" > and also a "long addAndGet()" ? [Besides The fact that only the first > one can benefit of an async option] > > Besides, I am no longer sure that it's good thing that methods in > Infinispan can be async vs sync depending on configuration switches; > I'd rather make it explicit in the signature and simplify the > configuration by removing such a flag. > > Making the methods which are async-capable to look "explicitly async" > should also allow us to add completeable futures & similar. > >> == Details == >> >> This is what I have in mind. Two counter managers: one based on JGroups >> counter and another one based on Infinispan cache. >> The first one creates AtomicCounters and it first perfectly. All >> counters are created with an initial value (zero by default) >> The second generates counters with all the options available. It can mix >> sync/async operation and all counters will be in the same cache. The >> cache will be configure by us and it would be an internal cache. This >> will use all the features available in the cache. > Rather than a split organized on the backing implementation details, > I'd prefer to see a split based on purpose of the counter. > > For example JGroups has different styles of counters: one might need > one to do an atomic increment which is monotonically increasing across > the whole cluster - from the point of view from an omniscent observer > - but in many cases we just need the generation of a guarantee of a > monotonic unique number: that allows for example to have each node > pre-allocate a range of values and pre-fetch a new block while its > running out of values. > > For some systems this is not acceptable because a failing server might > result in some of its allocated numbers to not ever be used, so > creating gaps; for others it's not acceptable that the figures > returned by the "monotonic counter" are not monotonic across each > other when confronting results from different cluster nodes, but for > many use cases that's acceptable. My point being that we need to be > able to choose between these possible semantics. > > As an Infinispan consumer being able to express such details is > essential, while having to choose between "backed by JGroups counters" > or not is irrelevant and actually makes it harder. > > In terms of Infinispan integration, the most important point I'd like > to see is persistence: i.e. make sure you can store them in a > CacheStore. > How we do that efficiently will also affect design; for example I was > expecting to see a Counter as something which is stored in a Cache, so > inheriting details such as configured CacheStore(s) and number of > owners. > > Thanks for starting this! > > Sanne > >> Configuration-wise, I'm thinking about 2 parameters: number of backups >> and timeout (for sync operations). >> >> So, comment bellow and let me know alternatives, improvement or if I >> missed something. >> >> ps. I also consider implement a counter based on JGroups-raft but I >> believe it is an overkill. >> ps2. sorry for the long email :( I tried to be shorter as possible. >> >> Cheers, >> Pedro >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev at lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev > _______________________________________________ > infinispan-dev mailing list > infinispan-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev -- Radim Vansa JBoss Performance Team From gustavo at infinispan.org Tue Mar 15 05:04:43 2016 From: gustavo at infinispan.org (Gustavo Fernandes) Date: Tue, 15 Mar 2016 09:04:43 +0000 Subject: [infinispan-dev] Distributed Counter Discussion In-Reply-To: <80FC3F6D-F7F8-4CBE-90CC-FEED3ADE86F1@redhat.com> References: <56E70D92.1010206@infinispan.org> <80FC3F6D-F7F8-4CBE-90CC-FEED3ADE86F1@redhat.com> Message-ID: > For information about CRDTs, make sure you?ve read the CRDT paper by > Shapiro: http://hal.upmc.fr/inria-00555588/document > On that topic, I recently came across the paper "Scalable Eventually Consistent Counters over Unreliable Networks" [1] which highlights some of the scalability problems with CRDT counters at large scale: namely the propagation and maintenance of a version vector that keeps growing over time and requires broadcasts across all members. The paper then describes how some implementors of CRDT counters tackled this limitation by using "server side only" counters, and the inherent issues with that approach. Finally it proposes an alternative called "Handoff counters" in order to overcome the scalability issue. [1] http://arxiv.org/pdf/1307.3207v1.pdf _______________________________________________ > infinispan-dev mailing list > infinispan-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.jboss.org/pipermail/infinispan-dev/attachments/20160315/20a88929/attachment.html From pedro at infinispan.org Tue Mar 15 07:58:20 2016 From: pedro at infinispan.org (Pedro Ruivo) Date: Tue, 15 Mar 2016 11:58:20 +0000 Subject: [infinispan-dev] Distributed Counter Discussion In-Reply-To: References: <56E70D92.1010206@infinispan.org> Message-ID: <56E7F8DC.9060000@infinispan.org> Comments inline On 03/14/2016 09:27 PM, Sanne Grinovero wrote: > On 14 March 2016 at 19:14, Pedro Ruivo wrote: > > As a user, how do I get a Counter instance? From a the CacheContainer interface? You can get a Counter from a CounterManager. The CounterManager is associated to the CacheManager. I have two options to get the CounterManager: CacheManager.as(CouterManager.class), as discussed during the Rome meeting (2015). This will change the API in the core but it allows to introduce the other building blocked (locks, sequences, latches, ...) The other alternative I have in mind is CounterMager.get(CacheManager). The only advantage I see is no need to change the API in core. guys... comment what alternative do you prefer... > > Will they have their own configuration section in the configuration file? I have no idea here... probably something like the cache store configurations would work. >> long get() //current value. may return stale value due to concurrent >> operations to other nodes. > > This is what puzzles me the most. I'm not sure if the feature is > actually useful, unless we can clearly state how far outdated the > value could be. > > I think a slightly more formal definition would be in order. For > example I think it would be acceptable to say that this will return a > value from the range of values the primary owner of this counter was > holding in the timeframe between the method is being invoked and the > time the value is returned. > > Could it optionally be integrated with Total Order ? Transactions? That's why we are having this discussion. If someone heard some requirements from an user, it would be good to write them here. IMO, at least, the get() must include all the finished operations in the local node. >> >> long getMin/MaxThreshold() //returns the min and max threshold value > > "threshold" ?? See Tristan's reply. > > I'd prefer a single interface, with reduced redundancy. > For example, is there really a benefit in having a "void increment()" > and also a "long addAndGet()" ? [Besides The fact that only the first > one can benefit of an async option] > > Besides, I am no longer sure that it's good thing that methods in > Infinispan can be async vs sync depending on configuration switches; > I'd rather make it explicit in the signature and simplify the > configuration by removing such a flag. > > Making the methods which are async-capable to look "explicitly async" > should also allow us to add completeable futures & similar. So, lets revamp the API. We can drop the AtomiCounter and create a simple Counter interface like: String getName() CompletableFuture addAndGet() long get() void reset() It is the simplest we can get and it covers all the semantics we need so far. If something does not fit, we can start improving it. > > Rather than a split organized on the backing implementation details, > I'd prefer to see a split based on purpose of the counter. So, what about multiple CounterManager where in each implementation we specified what semantic the Counter created ensures? > > For example JGroups has different styles of counters: one might need > one to do an atomic increment which is monotonically increasing across > the whole cluster - from the point of view from an omniscent observer > - but in many cases we just need the generation of a guarantee of a > monotonic unique number: that allows for example to have each node > pre-allocate a range of values and pre-fetch a new block while its > running out of values. you lost me in the pre-allocate range of values. I think you are talking about sequences (we also discussed that in Rome) but I don't think it should be the primary objective of a counter. > > For some systems this is not acceptable because a failing server might > result in some of its allocated numbers to not ever be used, so > creating gaps; for others it's not acceptable that the figures > returned by the "monotonic counter" are not monotonic across each > other when confronting results from different cluster nodes, but for > many use cases that's acceptable. My point being that we need to be > able to choose between these possible semantics. > > As an Infinispan consumer being able to express such details is > essential, while having to choose between "backed by JGroups counters" > or not is irrelevant and actually makes it harder. > > In terms of Infinispan integration, the most important point I'd like > to see is persistence: i.e. make sure you can store them in a > CacheStore. > How we do that efficiently will also affect design; for example I was > expecting to see a Counter as something which is stored in a Cache, so > inheriting details such as configured CacheStore(s) and number of > owners. I'm ok to support persistence but I don't see any advantage. Does anyone have a use case for it? From rvansa at redhat.com Tue Mar 15 08:57:31 2016 From: rvansa at redhat.com (Radim Vansa) Date: Tue, 15 Mar 2016 13:57:31 +0100 Subject: [infinispan-dev] Distributed Counter Discussion In-Reply-To: <56E7F8DC.9060000@infinispan.org> References: <56E70D92.1010206@infinispan.org> <56E7F8DC.9060000@infinispan.org> Message-ID: <56E806BB.8010803@redhat.com> On 03/15/2016 12:58 PM, Pedro Ruivo wrote: > > On 03/14/2016 09:27 PM, Sanne Grinovero wrote: >> Rather than a split organized on the backing implementation details, >> I'd prefer to see a split based on purpose of the counter. > So, what about multiple CounterManager where in each implementation we > specified what semantic the Counter created ensures? > Please, don't do that. Different implementation = does the same thing but has different internals. If it has different semantics (does something different), use different interface. This way users will just swap in 'more performant' implementation without thinking about the consequences. My 2c R. -- Radim Vansa JBoss Performance Team From rhauch at redhat.com Tue Mar 15 10:15:22 2016 From: rhauch at redhat.com (Randall Hauch) Date: Tue, 15 Mar 2016 09:15:22 -0500 Subject: [infinispan-dev] Distributed Counter Discussion In-Reply-To: <56E7C733.8080809@redhat.com> References: <56E70D92.1010206@infinispan.org> <56E7C733.8080809@redhat.com> Message-ID: <2B729151-6A22-49A9-9FC1-473220A3AF13@redhat.com> > On Mar 15, 2016, at 3:26 AM, Radim Vansa wrote: > > I second Sanne's opinion about the sync/async API: make the API express > the synchronicity directly. I would even propose that for synchronous > methods, there's only the CompletableFuture or > CompletableFuture variant; we are already reaching the concurrency > limits of application that lets threads block, so let's give user a hint > that he should use such notification API. If anyone prefers the sync > variant, he can always use get(). > > Let's settle on some nomenclature, too, because these JGroups-counters > and RAFT-counters don't have commonly known properties. It almost seems > that the term *counter* is so overloaded that we shouldn't use it all. > > There is a widely known j.u.c.AtomicLong, so if we want to implement > that (strict JMM-like properties), let's call it AtomicLong. Does not > have to follow j.u.c.AtomicLong API, but it's definitely CP. > > Something providing unique values should be called Sequence. This will > be probably the one batching ranges (therefore possibly with gaps). By > default non-monotonic, but monotonicity could be a ctor arg (in a > similar way as fairness is set for j.u.concurrent.* classes). +1 for differentiating in the API between a counter that a client increments (or resets) and a sequence generator that a client can use to obtain (non)monotonic values. > > As for CRDTs, I can't imagine how this could be easily built on top of > current 'passive' cache (without any syncing). But as for names *CRDT > counter* is explanatory enough. > > If someone needs quotas, let's create Quota according to their needs > (soft and hard threshold, fluent operation below soft, some jitter when > above that). It seems that this will be closest to Sequence by reserving > some ranges. Don't let them shoot themselves into foot with some liger > counter. +1 for creating a specific interface to enable quotas. Perhaps it is implemented with some more general purpose functionality that might be exposed in the future, but for now why not keep it narrowly focused. > > And I hope you'll build these on top of the functional API, with at most > one RPC per operation. > > Radim > > On 03/14/2016 10:27 PM, Sanne Grinovero wrote: >> Great starting point! >> Some comments inline: >> >> On 14 March 2016 at 19:14, Pedro Ruivo wrote: >>> Hi everybody, >>> >>> Discussion about distributed counters. >>> >>> == Public API == >>> >>> interface Counter >> As a user, how do I get a Counter instance? From a the CacheContainer interface? >> >> Will they have their own configuration section in the configuration file? >> >>> String getName() //counter name >>> long get() //current value. may return stale value due to concurrent >>> operations to other nodes. >> This is what puzzles me the most. I'm not sure if the feature is >> actually useful, unless we can clearly state how far outdated the >> value could be. >> >> I think a slightly more formal definition would be in order. For >> example I think it would be acceptable to say that this will return a >> value from the range of values the primary owner of this counter was >> holding in the timeframe between the method is being invoked and the >> time the value is returned. >> >> Could it optionally be integrated with Total Order ? Transactions? >> >>> void increment() //async or sync increment. default add(1) >>> void decrement() //async or sync decrement. default add(-1) >>> void add(long) //async or sync add. >>> void reset() //resets to initial value >>> >>> Note: Tried to make the interface as simple as possible with support for >>> sync and async operations. To avoid any confusion, I consider an async >>> operation as happening somewhat in the future, i.e. eventually >>> increments/decrements. >>> The sync operation happens somewhat during the method execution. >>> >>> interface AtomiCounter extends Counter >>> >>> long addAndGet() //adds a returns the new value. sync operation >>> long incrementAndGet() //increments and returns the new value. sync >>> operation. default addAndGet(1) >>> long decrementAndGet() //decrements and returns the new value. sync >>> operation. default addAndGet(-1) >>> >>> interface AdvancedCounter extends Counter >>> >>> long getMin/MaxThreshold() //returns the min and max threshold value >> "threshold" ?? >> >>> void add/removeListener() //adds a listener that is invoked when the >>> value change. Can be extended to notify when it is "reseted" and when >>> the threshold is reached. >>> >>> Note: should this interface be splitted? >> I'd prefer a single interface, with reduced redundancy. >> For example, is there really a benefit in having a "void increment()" >> and also a "long addAndGet()" ? [Besides The fact that only the first >> one can benefit of an async option] >> >> Besides, I am no longer sure that it's good thing that methods in >> Infinispan can be async vs sync depending on configuration switches; >> I'd rather make it explicit in the signature and simplify the >> configuration by removing such a flag. >> >> Making the methods which are async-capable to look "explicitly async" >> should also allow us to add completeable futures & similar. >> >>> == Details == >>> >>> This is what I have in mind. Two counter managers: one based on JGroups >>> counter and another one based on Infinispan cache. >>> The first one creates AtomicCounters and it first perfectly. All >>> counters are created with an initial value (zero by default) >>> The second generates counters with all the options available. It can mix >>> sync/async operation and all counters will be in the same cache. The >>> cache will be configure by us and it would be an internal cache. This >>> will use all the features available in the cache. >> Rather than a split organized on the backing implementation details, >> I'd prefer to see a split based on purpose of the counter. >> >> For example JGroups has different styles of counters: one might need >> one to do an atomic increment which is monotonically increasing across >> the whole cluster - from the point of view from an omniscent observer >> - but in many cases we just need the generation of a guarantee of a >> monotonic unique number: that allows for example to have each node >> pre-allocate a range of values and pre-fetch a new block while its >> running out of values. >> >> For some systems this is not acceptable because a failing server might >> result in some of its allocated numbers to not ever be used, so >> creating gaps; for others it's not acceptable that the figures >> returned by the "monotonic counter" are not monotonic across each >> other when confronting results from different cluster nodes, but for >> many use cases that's acceptable. My point being that we need to be >> able to choose between these possible semantics. >> >> As an Infinispan consumer being able to express such details is >> essential, while having to choose between "backed by JGroups counters" >> or not is irrelevant and actually makes it harder. >> >> In terms of Infinispan integration, the most important point I'd like >> to see is persistence: i.e. make sure you can store them in a >> CacheStore. >> How we do that efficiently will also affect design; for example I was >> expecting to see a Counter as something which is stored in a Cache, so >> inheriting details such as configured CacheStore(s) and number of >> owners. >> >> Thanks for starting this! >> >> Sanne >> >>> Configuration-wise, I'm thinking about 2 parameters: number of backups >>> and timeout (for sync operations). >>> >>> So, comment bellow and let me know alternatives, improvement or if I >>> missed something. >>> >>> ps. I also consider implement a counter based on JGroups-raft but I >>> believe it is an overkill. >>> ps2. sorry for the long email :( I tried to be shorter as possible. >>> >>> Cheers, >>> Pedro >>> _______________________________________________ >>> infinispan-dev mailing list >>> infinispan-dev at lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev at lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev > > > -- > Radim Vansa > JBoss Performance Team > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev From rhauch at redhat.com Tue Mar 15 10:17:45 2016 From: rhauch at redhat.com (Randall Hauch) Date: Tue, 15 Mar 2016 09:17:45 -0500 Subject: [infinispan-dev] Distributed Counter Discussion In-Reply-To: <56E7B5DE.6010606@redhat.com> References: <56E70D92.1010206@infinispan.org> <80FC3F6D-F7F8-4CBE-90CC-FEED3ADE86F1@redhat.com> <56E7B5DE.6010606@redhat.com> Message-ID: <69D897A4-54A4-4BE0-A954-996106318CFA@redhat.com> > On Mar 15, 2016, at 2:12 AM, Bela Ban wrote: > > > > On 14/03/16 23:17, Randall Hauch wrote: >> What are the requirements? What are the distributed counters for? Is the >> counter to be monotonically increasing? Can there be any missed values? >> Does the counter need to increment and decrement? What is the *smallest* >> API you need initially? >> >> There are two choices when implementing a distributed counter: use >> central coordination (like JGroups counters), or use independent >> counters on separate machines that will eventually converge to the >> correct value (CRDTs). Coordinated counters are expensive and therefore >> slow, and can suffer from problems during network or cluster problems. > > The question is what do you get for this? If your app can't afford > duplicate counter values during a network partition, then - yes - there > is some overhead. CRDTs won't be able to guarantee this property. OTOH > CRDTs are fast when you only care about some sort of eventual > consistency, and don't need 'hard' consistency. To be clear, I?m not saying they are interchangeable. They have very different properties, which is why the requirements will help determine which of them (if any) are applicable. > > >> For example, what happens during a split brain? OTOH, CRDTs are >> decentralized so therefore are very fast, easily merged, and fault >> tolerant; > > Yes, CRDTS are AP whereas jgroups-raft counters are CP. JGroups > counters, otoh, are CRAP (consistent, reliable, available and > partition-aware. > > Take the last sentence with a grain of salt :-) > >> they?re excellent when counting things that are occurring >> independently and therefore may be more suited for >> monitoring/metrics/accumulators/etc. Both have very different behaviors >> under ideal and failure scenarios, have different performance and >> consistency guarantees, and are useful in different scenarios. Make sure >> you choose accordingly. >> >> For information about CRDTs, make sure you?ve read the CRDT paper by >> Shapiro: http://hal.upmc.fr/inria-00555588/document >> >> Randall >> >>> On Mar 14, 2016, at 2:14 PM, Pedro Ruivo >>> >> wrote: >>> >>> Hi everybody, >>> >>> Discussion about distributed counters. >>> >>> == Public API == >>> >>> interface Counter >>> >>> String getName() //counter name >>> long get()//current value. may return stale value due to concurrent >>> operations to other nodes. >>> void increment() //async or sync increment. default add(1) >>> void decrement() //async or sync decrement. default add(-1) >>> void add(long) //async or sync add. >>> void reset() //resets to initial value >>> >>> Note: Tried to make the interface as simple as possible with support for >>> sync and async operations. To avoid any confusion, I consider an async >>> operation as happening somewhat in the future, i.e. eventually >>> increments/decrements. >>> The sync operation happens somewhat during the method execution. >>> >>> interface AtomiCounter extends Counter >>> >>> long addAndGet() //adds a returns the new value. sync operation >>> long incrementAndGet() //increments and returns the new value. sync >>> operation. default addAndGet(1) >>> long decrementAndGet() //decrements and returns the new value. sync >>> operation. default addAndGet(-1) >>> >>> interface AdvancedCounter extends Counter >>> >>> long getMin/MaxThreshold() //returns the min and max threshold value >>> void add/removeListener() //adds a listener that is invoked when the >>> value change. Can be extended to notify when it is "reseted" and when >>> the threshold is reached. >>> >>> Note: should this interface be splitted? >>> >>> == Details == >>> >>> This is what I have in mind. Two counter managers: one based on JGroups >>> counter and another one based on Infinispan cache. >>> The first one creates AtomicCounters and it first perfectly. All >>> counters are created with an initial value (zero by default) >>> The second generates counters with all the options available. It can mix >>> sync/async operation and all counters will be in the same cache. The >>> cache will be configure by us and it would be an internal cache. This >>> will use all the features available in the cache. >>> >>> Configuration-wise, I'm thinking about 2 parameters: number of backups >>> and timeout (for sync operations). >>> >>> So, comment bellow and let me know alternatives, improvement or if I >>> missed something. >>> >>> ps. I also consider implement a counter based on JGroups-raft but I >>> believe it is an overkill. >>> ps2. sorry for the long email :( I tried to be shorter as possible. >>> >>> Cheers, >>> Pedro >>> _______________________________________________ >>> infinispan-dev mailing list >>> infinispan-dev at lists.jboss.org

> >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >> >> >> >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev at lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev >> > > -- > Bela Ban, JGroups lead (http://www.jgroups.org ) > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.jboss.org/pipermail/infinispan-dev/attachments/20160315/c385e9ac/attachment-0001.html From sanne at infinispan.org Tue Mar 15 15:22:57 2016 From: sanne at infinispan.org (Sanne Grinovero) Date: Tue, 15 Mar 2016 19:22:57 +0000 Subject: [infinispan-dev] XA transactions on a single CacheManager? Message-ID: Hi all, I just noticed that when I'm making changes to multiple caches within the same transaction, the transaction manager will treat this as XA transactions. That seems suboptimal as they are all managed by the same resource; is there a configuration I'm missing or should I open a JIRA to ask for an improvement? I'm not just asking for performance reasons: having to setup a transaction manager is an annoying configuration step for Java SE users. Thanks, Sanne From vjuranek at redhat.com Tue Mar 15 18:15:14 2016 From: vjuranek at redhat.com (Vojtech Juranek) Date: Tue, 15 Mar 2016 23:15:14 +0100 Subject: [infinispan-dev] ISPN embedded tutorial broken Message-ID: <6207433.jbZTtoz8p6@localhost> Hi, I noticed that ISPN embedded tutorial (weather app) [1] is broken. It fails with NPE when fetching weather information from OpenWeatherMap. It's broken as OpenWeatherMap started to request user registration [2]. I checked couple of other free weather service (like Weather underground, Yahoo Weather, World weather online) and all of them seem to require some kind of user registration. Shall we register on OpenWeatherMap (I didn't investigate if API token can be pushed to GitHub) or switch to RandomWeatherService as default service? Thanks Vojta [1] http://infinispan.org/tutorials/embedded/ [2] http://openweathermap.org/faq#error401 -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 473 bytes Desc: This is a digitally signed message part. Url : http://lists.jboss.org/pipermail/infinispan-dev/attachments/20160315/288d318d/attachment.bin From slaskawi at redhat.com Wed Mar 16 02:22:56 2016 From: slaskawi at redhat.com (Sebastian Laskawiec) Date: Wed, 16 Mar 2016 07:22:56 +0100 Subject: [infinispan-dev] ISPN embedded tutorial broken In-Reply-To: <6207433.jbZTtoz8p6@localhost> References: <6207433.jbZTtoz8p6@localhost> Message-ID: +1 for registering a user and putting API key in the repo. On Tue, Mar 15, 2016 at 11:15 PM, Vojtech Juranek wrote: > Hi, > I noticed that ISPN embedded tutorial (weather app) [1] is broken. It fails > with NPE when fetching weather information from OpenWeatherMap. It's > broken as > OpenWeatherMap started to request user registration [2]. I checked couple > of > other free weather service (like Weather underground, Yahoo Weather, World > weather online) and all of them seem to require some kind of user > registration. > > Shall we register on OpenWeatherMap (I didn't investigate if API token can > be > pushed to GitHub) or switch to RandomWeatherService as default service? > > Thanks > Vojta > > [1] http://infinispan.org/tutorials/embedded/ > [2] http://openweathermap.org/faq#error401 > _______________________________________________ > infinispan-dev mailing list > infinispan-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.jboss.org/pipermail/infinispan-dev/attachments/20160316/6ba81151/attachment.html From galder at redhat.com Wed Mar 16 03:56:12 2016 From: galder at redhat.com (=?utf-8?Q?Galder_Zamarre=C3=B1o?=) Date: Wed, 16 Mar 2016 08:56:12 +0100 Subject: [infinispan-dev] Verifying Consistent Hash based routing using stats In-Reply-To: References: Message-ID: Hey Vittorio, Expanding this to dev list in case anyone has interest. Latest commit is in [1]. The relevant code that verifies Consistent Hash (CH) based routing is in [2]. The way it works is this: Assuming I have 3 nodes, A, B and C, with 2 as number of copies for distribution, I do the following: 0. Get stats from nodes A, B, C. 1. Generate a key K1 whose owners are A,B. 2. Generate a key K2 whose owners are B,C. 3. Generate a key K3 whose owners are C,A. 4. The primary node is always the first node of the owners, so if CH routing works fine, K1 would targeted for node A, K2 for node B and K3 for node C. 5. Call puts on keys K1, K2 and K3. 6. Get stats from nodes A, B, C. 7. For all nodes, "stores" value in latest stats should be +1 compared to initial stats. This would pass even if with a round-robin load balance policy. 8. For all nodes, "currentNumberOfEntries" value in latest stats should be (Number of copies) * ("stores"). This is the key assumption that guarantees CH routing works as expected. ^ To be more precise, this can only be guaranteed under certain circumstances: First, the ownership needs to be spread around, so a node does not take owner ship of 3 keys and another of only 1. We can guarantee that based on how we generate the keys. The second part is that if a put happens in a non-owner, this non-owner node will increase its "stores" values but not its "currentNumberOfEntries" since the key does not belong in this node. So, if any put request happened in a non-owner, there would be an inbalance in the ratio of currentNumberOfEntries to "stores". Hope this helps. Cheers, [1] https://github.com/galderz/js-client/commit/c57c285561b73e67b7ebce04de67d99086b5b726 [2] https://github.com/galderz/js-client/blob/t_steps/spec/infinispan_cluster_spec.js#L72 -- Galder Zamarre?o Infinispan, Red Hat > On 16 Mar 2016, at 08:25, Galder Zamarre?o wrote: > > Hey, > > Sorry I missed your reply yesterday about the location of the code. > > I've not pushed my branch yet, I'll do that this morning. As soon as it's on Github I'll let you know :) > > Cheers, > -- > Galder Zamarre?o > Infinispan, Red Hat > From ttarrant at redhat.com Wed Mar 16 04:20:20 2016 From: ttarrant at redhat.com (Tristan Tarrant) Date: Wed, 16 Mar 2016 09:20:20 +0100 Subject: [infinispan-dev] ISPN embedded tutorial broken In-Reply-To: <6207433.jbZTtoz8p6@localhost> References: <6207433.jbZTtoz8p6@localhost> Message-ID: <56E91744.6040401@redhat.com> I'd rather go for RandomWeatherService. Tristan On 15/03/2016 23:15, Vojtech Juranek wrote: > Hi, > I noticed that ISPN embedded tutorial (weather app) [1] is broken. It fails > with NPE when fetching weather information from OpenWeatherMap. It's broken as > OpenWeatherMap started to request user registration [2]. I checked couple of > other free weather service (like Weather underground, Yahoo Weather, World > weather online) and all of them seem to require some kind of user > registration. > > Shall we register on OpenWeatherMap (I didn't investigate if API token can be > pushed to GitHub) or switch to RandomWeatherService as default service? > > Thanks > Vojta > > [1] http://infinispan.org/tutorials/embedded/ > [2] http://openweathermap.org/faq#error401 > > > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev > -- Tristan Tarrant Infinispan Lead JBoss, a division of Red Hat From rvansa at redhat.com Wed Mar 16 04:51:45 2016 From: rvansa at redhat.com (Radim Vansa) Date: Wed, 16 Mar 2016 09:51:45 +0100 Subject: [infinispan-dev] XA transactions on a single CacheManager? In-Reply-To: References: Message-ID: <56E91EA1.4020700@redhat.com> Quite related, though for NON_XA: https://issues.jboss.org/browse/ISPN-5600 R. On 03/15/2016 08:22 PM, Sanne Grinovero wrote: > Hi all, > > I just noticed that when I'm making changes to multiple caches within > the same transaction, the transaction manager will treat this as XA > transactions. > > That seems suboptimal as they are all managed by the same resource; is > there a configuration I'm missing or should I open a JIRA to ask for > an improvement? > > I'm not just asking for performance reasons: having to setup a > transaction manager is an annoying configuration step for Java SE > users. > > Thanks, > Sanne > _______________________________________________ > infinispan-dev mailing list > infinispan-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev -- Radim Vansa JBoss Performance Team From galder at redhat.com Wed Mar 16 06:26:50 2016 From: galder at redhat.com (=?utf-8?Q?Galder_Zamarre=C3=B1o?=) Date: Wed, 16 Mar 2016 11:26:50 +0100 Subject: [infinispan-dev] Infinispan Javascript client 0.1.0 is out now! In-Reply-To: <56CC887B.4090905@redhat.com> References: <36BD5510-94EC-47BD-AA21-54C459165CD9@redhat.com> <56C6E0EA.8050305@redhat.com> <56C6EBBC.3080605@redhat.com> <56CC887B.4090905@redhat.com> Message-ID: AFAIK, we're stuck with `function(client) { return ... ;}` since we need to support ECMAScript 5. ECMAScript 6 brings in lambda syntax which will make this more palatable: https://developer.mozilla.org/en/docs/Web/JavaScript/Reference/Functions/Arrow_functions Your suggestion would not really work since it would mean passing a value instead of a function, and we need the latter to be able to chain functions lazily. Cheers, -- Galder Zamarre?o Infinispan, Red Hat > On 23 Feb 2016, at 17:27, Radim Vansa wrote: > > Thanks, though it was pretty clear even after your first reply. > > One more thing: it seems to me that there is certain redundancy in > declaring > > function(client) { return ... ;} > > where only the ellipsis is the business logic. I would recommend > including helper with higher-order functions, that would give remove the > need for the above when you just called ispn API, please see how I've > modified > > var put = before.then(function() { return client.put('k', 'v'); }); // > from [2] > > into > > var put = before.then(cache().put('k', 'v')); // [3] > > ([3] includes stub of the helper). Or is this too non-JS-ish (TBH I > haven't tested if that is valid JS)? > > Radim > > [3] https://gist.github.com/rvansa/ba5be94ba3bf7ae39a91 > > On 02/23/2016 04:35 PM, Galder Zamarre?o wrote: >> Radim, once again thanks for the invaluable feedback. >> >> As promised, I will ammend the README to make it much more clear but I found a very easy example that I can show here as I was working on the JS client testsuite. >> >> Imagine this scenario: I want to verify that after doing a put 'stores' statistic goes up by 1. The obvious thing to do here is this: >> >> 1. invoke stats to find out initial 'stores' value, then... >> 2. invoke put, then... >> 3. invoke stats again and compare with initial 'stores' value. >> >> Using Promises, this can be coded in two different ways. The first one is using nesting, e.g. [1]. This is a bit ugly but this is how you'd do have to do it if you were using function callbacks instead of Promises. >> >> The alternative is to chain Promises, taking into account that Promise's then() return a Promise or a directly value from within it. Here's an alternative solution that reads more cleanly [2]. >> >> Here we can see how the first then() callback returns a value directly, which are the statistics before the put operation. The next two .then() calls return Promises, first a promise for the put, and the second a promise of the stats retrieved after put operation. >> >> Finally, we call Promise.all() which takes N promises and calls back .then() when all those promises have been fullfilled, and then passes the results of those promises as parameter. Within this last .then() callback, we can do the necessary assertions compare before and after stastistic values. >> >> Hope this clarifies things further. >> >> Cheers, >> >> p.s. 'stats' operation support is work in progress. >> >> [1] https://gist.github.com/galderz/65ed0b2a4e02b8f4e0c1 >> [2] https://gist.github.com/galderz/c349d3aa64a39684d7d7 >> -- >> Galder Zamarre?o >> Infinispan, Red Hat >> >>> On 19 Feb 2016, at 11:17, Radim Vansa wrote: >>> >>> All right, this example looks much better. And the ability to use native >>> variables rather than keys into the session is much more comfortable. >>> Thanks for the explanation! >>> >>> Radim >>> >>> On 02/19/2016 10:53 AM, Galder Zamarre?o wrote: >>>> -- >>>> Galder Zamarre?o >>>> Infinispan, Red Hat >>>> >>>>> On 19 Feb 2016, at 10:31, Radim Vansa wrote: >>>>> >>>>> Just checked the readme, and TBH I wouldn't be too enthusiastic about >>>>> the deep nesting of chained operations (basically each follow-up >>>>> operation requires further level of nesting). Is that the way async >>>>> handlers are usually written in JS? >>>> I think the README I've produce is a bit confusing, I'll work to improve for future versions. In the mean time, let me explain this: >>>> >>>> The reason Promises are used instead of callback parameters is precisely to avoid this kind of deep nesting. >>>> >>>> The code below shows 4 (but could be N really) operations which are chained only 2 levels deep: >>>> >>>> connected.then(function(client) { >>>> ... <- Do something with the client >>>> var putCall = client.put('key', 'value'); >>>> var getCall = putCall.then(function() { >>>> return client.get('key'); >>>> }); >>>> var containsKeyCall = getCall.then(function() { >>>> return client.containsKey('key'); >>>> }); >>>> var removeCall = containsKeyCall.then(function(contains) { >>>> return client.remove('key'); >>>> }) >>>> }); >>>> >>>> The key aspect of this is that no matter how many operations you want to chain together, the nesting levels are constant. You could have 100 operations chained together, you would still only have 2 levels deep. >>>> >>>> The README file might not make this so obvious and hence I'll work to improve on that. >>>> >>>> Also, the API above is using Promise API which will be standard in ES6. Even though Infinispan JS Client is based on ES5, the `promise` module sticks to the forthcoming standard Promise API, and hence we're using something that in a few years will be standard, which is a good thing. Conceptually, Promise API is similar to Scala's Future or Java's CompletableFuture. >>>> >>>> The other alternative that's used in Node.js is passing callbacks as parameters to each put/get call... and that does cause very deep nesting, because if you want to chain 100 operations together, you need 100 nested calls! Plus not centralised error handling, and many other disadvantages. >>>> >>>>> Last few weeks I am working with Gatling and the syntax to describe >>>>> scenarios = chained operations is quite nice (though it's Scala). Would >>>>> it be possible to have a syntax like this? >>>>> >>>>> connected >>>>> .then(function(client) { >>>>> client.put('key', 'value'); >>>>> .then(function(client) { >>>>> client.get('key').use(function(value) { >>>>> console.log(':get(`key`) = ' + value); >>>>> }).save('p0'); // this would be the same as use(function(value) >>>>> { client.save('p0') }) >>>>> }. >>>>> .then(function(client)) { >>>>> client.remove('key', client.load('p0')).use(function(success) { >>>>> console.log(':remove(`key`) = ' + success); >>>>> }); >>>>> }).exec(); >>>> Independent of the correctness of the syntax, this API can be split in several concepts: >>>> >>>> 1. The first one is that for those operations that have not a return, e.g. a put, you're passing on the client instance. I think that sounds fine. >>>> >>>> 2. For your API to work, get() would need to return something other than Promise that has a use/save methods. I'm very reluctant to do anything like that since it's not standard and would require more upfront understanding from the user. Right now, we provide a Promise, a concept that's very well understood in the JS community and that will become standard in ES6. >>>> >>>>> Note that I've used conditional remove to demonstrate load of previously >>>>> saved value. Also, I am mixing the concept of client = cache and >>>>> session, but maybe these could go to single object. >>>>> >>>>> Just my 2c >>>>> >>>>> Radim >>>>> >>>>> On 02/18/2016 12:04 PM, Galder Zamarre?o wrote: >>>>>> Hi, >>>>>> >>>>>> Following on Tristan's anouncement, we've also released Infinispan Javascript client version 0.1.0. You can read all about it here: >>>>>> http://blog.infinispan.org/2016/02/infinispan-javascript-client-010-is-out.html >>>>>> >>>>>> Cheers, >>>>>> -- >>>>>> Galder Zamarre?o >>>>>> Infinispan, Red Hat >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> infinispan-dev mailing list >>>>>> infinispan-dev at lists.jboss.org >>>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>>>> -- >>>>> Radim Vansa >>>>> JBoss Performance Team >>>>> >>>>> _______________________________________________ >>>>> infinispan-dev mailing list >>>>> infinispan-dev at lists.jboss.org >>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>>> _______________________________________________ >>>> infinispan-dev mailing list >>>> infinispan-dev at lists.jboss.org >>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>> >>> -- >>> Radim Vansa >>> JBoss Performance Team >>> >>> _______________________________________________ >>> infinispan-dev mailing list >>> infinispan-dev at lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >> >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev at lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev > > > -- > Radim Vansa > JBoss Performance Team > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev From sanne at infinispan.org Thu Mar 17 07:25:20 2016 From: sanne at infinispan.org (Sanne Grinovero) Date: Thu, 17 Mar 2016 11:25:20 +0000 Subject: [infinispan-dev] Practical example of difficulties with dependencies Message-ID: Since we recently discussed the purpose of the "UberJar"s , I think it would be a great exercise to look at this discussion: - https://developer.jboss.org/message/953081 and ask ourselves what we could have done to make the first steps of this new user less miserable. Personally I think we could go a long way with some clear documentation about our dependencies; it's not hard to inject version properties from the pom files into the docs, so we could have a little reminder about - for example - which version of Hibernate is expected to have the JPACacheStore working. Also, I think it's clear that uber jars - at least in this case - are not even close to the solution, unless you intend to include all of Hibernate ORM and transitive dependencies.. Thanks, Sanne From slaskawi at redhat.com Thu Mar 17 07:48:04 2016 From: slaskawi at redhat.com (Sebastian Laskawiec) Date: Thu, 17 Mar 2016 12:48:04 +0100 Subject: [infinispan-dev] Practical example of difficulties with dependencies In-Reply-To: References: Message-ID: Hey! If Hibernate had an Uber jar - one could add it to the classpath and fix the problem :) Of course I'm not serious here... So what are you proposing? One of the proposed solutions (in "Uber jars - how do we want to use them" email thread) was to limit number of dependencies and throw errors in the runtime if something is missing. I'm slightly positive to the first part but really negative to the second (it only delays the problem until you run the application). Another idea (a bit similar) is to squash most of the ISPN modules into infinispan-core and move all "extras" (like Spring support) in separate repository (infinispan-extras organization?). Such projects would have a dependency to core. But with this approach we get all the bad things from coarse grained layout. The question I'm asking here is - if not Uber Jars, than what? Ideas are more than welcome! Thanks Sebastian On Thu, Mar 17, 2016 at 12:25 PM, Sanne Grinovero wrote: > Since we recently discussed the purpose of the "UberJar"s , I think it > would be a great exercise to look at this discussion: > - https://developer.jboss.org/message/953081 > > and ask ourselves what we could have done to make the first steps of > this new user less miserable. > > Personally I think we could go a long way with some clear > documentation about our dependencies; > it's not hard to inject version properties from the pom files into the > docs, so we could have a little reminder about - for example - which > version of Hibernate is expected to have the JPACacheStore working. > > Also, I think it's clear that uber jars - at least in this case - are > not even close to the solution, unless you intend to include all of > Hibernate ORM and transitive dependencies.. > > Thanks, > Sanne > _______________________________________________ > infinispan-dev mailing list > infinispan-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.jboss.org/pipermail/infinispan-dev/attachments/20160317/3d81f053/attachment.html From ttarrant at redhat.com Thu Mar 17 08:22:58 2016 From: ttarrant at redhat.com (Tristan Tarrant) Date: Thu, 17 Mar 2016 13:22:58 +0100 Subject: [infinispan-dev] Practical example of difficulties with dependencies In-Reply-To: References: Message-ID: <56EAA1A2.4060109@redhat.com> On 17/03/2016 12:25, Sanne Grinovero wrote: > Since we recently discussed the purpose of the "UberJar"s , I think it > would be a great exercise to look at this discussion: > - https://developer.jboss.org/message/953081 > > and ask ourselves what we could have done to make the first steps of > this new user less miserable. > > Personally I think we could go a long way with some clear > documentation about our dependencies; > it's not hard to inject version properties from the pom files into the > docs, so we could have a little reminder about - for example - which > version of Hibernate is expected to have the JPACacheStore working. > > Also, I think it's clear that uber jars - at least in this case - are > not even close to the solution, unless you intend to include all of > Hibernate ORM and transitive dependencies.. While you are right that we could be clearer, and providing a printout of mvn dependency:dependency-[list|tree] would help some users, managing dependencies manually always comes loaded with a number of issues: duplicate jars, multiple versions, incompatible versions, useless jars, etc. So, to me, this is an NP complete problem. Tristan -- Tristan Tarrant Infinispan Lead JBoss, a division of Red Hat From sanne at infinispan.org Thu Mar 17 09:32:06 2016 From: sanne at infinispan.org (Sanne Grinovero) Date: Thu, 17 Mar 2016 13:32:06 +0000 Subject: [infinispan-dev] Practical example of difficulties with dependencies In-Reply-To: <56EAA1A2.4060109@redhat.com> References: <56EAA1A2.4060109@redhat.com> Message-ID: On 17 March 2016 at 12:22, Tristan Tarrant wrote: > On 17/03/2016 12:25, Sanne Grinovero wrote: >> Since we recently discussed the purpose of the "UberJar"s , I think it >> would be a great exercise to look at this discussion: >> - https://developer.jboss.org/message/953081 >> >> and ask ourselves what we could have done to make the first steps of >> this new user less miserable. >> >> Personally I think we could go a long way with some clear >> documentation about our dependencies; >> it's not hard to inject version properties from the pom files into the >> docs, so we could have a little reminder about - for example - which >> version of Hibernate is expected to have the JPACacheStore working. >> >> Also, I think it's clear that uber jars - at least in this case - are >> not even close to the solution, unless you intend to include all of >> Hibernate ORM and transitive dependencies.. > > While you are right that we could be clearer, and providing a printout > of mvn dependency:dependency-[list|tree] would help some users, managing > dependencies manually always comes loaded with a number of issues: > duplicate jars, multiple versions, incompatible versions, useless jars, > etc. So, to me, this is an NP complete problem. Sure, I don't expect neither yourself or the Infinispan team as a whole to have a magic solution. Just wondering what could be "good enough" without being too hard to maintain, maybe environment/goal driven? For example: to run on Websphere, you will need this list. The Wildfly modules are a great way to package; maybe we should provide (and test) packages for other containers? BTW Maven helps a bit but it's not providing the magic solution either, especially not when deploying on containers and it's not clear what is already going to be provided, but also for example it doesn't help with the different versions of JEE API specifications which we, i.e. geronimo and Narayana publish different artifact identifiers, and even within the same project they will publish a different API version as a different artifact. N.B. Uber Jars didn't help at all here, and they potentially add even more confusion or duplication. Sanne > > Tristan > -- > Tristan Tarrant > Infinispan Lead > JBoss, a division of Red Hat > _______________________________________________ > infinispan-dev mailing list > infinispan-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev From eric.wittmann at redhat.com Thu Mar 17 13:06:04 2016 From: eric.wittmann at redhat.com (Eric Wittmann) Date: Thu, 17 Mar 2016 13:06:04 -0400 Subject: [infinispan-dev] Distributed Counter Discussion Message-ID: <56EAE3FC.6030401@redhat.com> Greetings. Apologies for coming in a bit late on this conversation. Tristan pointed me to it a couple of days ago and unfortunately I'm just now getting time to reply. I can try to quickly give an overview of apiman's (JBoss API Management Gateway) requirements. What we're trying to do is implement support for Limiting policies: * Rate Limiting/Throttling (e.g. limit of 100 requests per second) * Quotas (e.g. limit of 100,000,000 requests per month) * Transfer Quotas (e.g. limit of 2.5GB of data downloaded per day) We will need to support multiple backing implementations of the Rate Limiter, and we're trying to get Infinispan to be one of those implementations. In no particular order, we would need the following characteristics: - Can be "squishy" for quotas and transfer quotas: If you get 100,001,017 requests that's OK - Strict would be cool as an option: Hard-fail when the counter reaches the limit - no chance it will go over. - Lots of individual counters: users may publish 100s of APIs to the Gateway, and each API may be consumed by 100s or 1000s of users/client. Depending on configuration of the policy, *each* user/client has a separate limit. - Counters need to be created dynamically: users can add APIs via the Management UI, configure them to add policies (e.g. a Quota policy) and then publish them to a running Gateway, at which point end users can invoke the API through the Gateway, which will use a counter to enforce the Quota. - Counter values reset at the end of a time boundary: for example, at the end of the month the counter value for the example quota above would reset to 0. - Don't care (right now) what the counter value is: at the moment we simply need to know if some counter max value has been reached. In the future we would like to know when a max value is being "approached" (e.g. to notify a user) - Should be persistent: it would not be ideal for e.g. per- month quota values to be lost on server restart. That's all the high level requirements I can think of off the top of my head, and after reading all of the current messages in this thread. :) -Eric From bban at redhat.com Fri Mar 18 02:32:02 2016 From: bban at redhat.com (Bela Ban) Date: Fri, 18 Mar 2016 07:32:02 +0100 Subject: [infinispan-dev] Distributed Counter Discussion In-Reply-To: <56EAE3FC.6030401@redhat.com> References: <56EAE3FC.6030401@redhat.com> Message-ID: <56EBA0E2.7070106@redhat.com> Stupid question: whay do you need a distributed counter for this? Is the service you're monitoring replicated in a cluster? On 17/03/16 18:06, Eric Wittmann wrote: > Greetings. Apologies for coming in a bit late on this conversation. > Tristan pointed me to it a couple of days ago and unfortunately I'm just > now getting time to reply. > > I can try to quickly give an overview of apiman's (JBoss API Management > Gateway) requirements. > > What we're trying to do is implement support for Limiting policies: > > * Rate Limiting/Throttling (e.g. limit of 100 requests per second) > * Quotas (e.g. limit of 100,000,000 requests per month) > * Transfer Quotas (e.g. limit of 2.5GB of data downloaded per day) > > We will need to support multiple backing implementations of the Rate > Limiter, and we're trying to get Infinispan to be one of those > implementations. > > In no particular order, we would need the following characteristics: > > - Can be "squishy" for quotas and transfer quotas: If you > get 100,001,017 requests that's OK > - Strict would be cool as an option: Hard-fail when the > counter reaches the limit - no chance it will go over. > - Lots of individual counters: users may publish 100s of > APIs to the Gateway, and each API may be consumed by > 100s or 1000s of users/client. Depending on configuration > of the policy, *each* user/client has a separate limit. > - Counters need to be created dynamically: users can > add APIs via the Management UI, configure them to add > policies (e.g. a Quota policy) and then publish them to > a running Gateway, at which point end users can invoke > the API through the Gateway, which will use a counter > to enforce the Quota. > - Counter values reset at the end of a time boundary: for > example, at the end of the month the counter value for > the example quota above would reset to 0. > - Don't care (right now) what the counter value is: at the > moment we simply need to know if some counter max value > has been reached. In the future we would like to know > when a max value is being "approached" (e.g. to notify a > user) > - Should be persistent: it would not be ideal for e.g. per- > month quota values to be lost on server restart. > > That's all the high level requirements I can think of off the top of my > head, and after reading all of the current messages in this thread. :) > > -Eric > _______________________________________________ > infinispan-dev mailing list > infinispan-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev > -- Bela Ban, JGroups lead (http://www.jgroups.org) From eric.wittmann at redhat.com Fri Mar 18 09:19:11 2016 From: eric.wittmann at redhat.com (Eric Wittmann) Date: Fri, 18 Mar 2016 09:19:11 -0400 Subject: [infinispan-dev] Distributed Counter Discussion In-Reply-To: <56EBA0E2.7070106@redhat.com> References: <56EAE3FC.6030401@redhat.com> <56EBA0E2.7070106@redhat.com> Message-ID: <56EC004F.7040101@redhat.com> Yes, precisely. The API Gateway itself is clustered. It services a large volume of inbound traffic which it reverse-proxies to appropriate back-end APIs after applying policies such as security, rate limiting, caching, etc. -Eric On 3/18/2016 2:32 AM, Bela Ban wrote: > Stupid question: whay do you need a distributed counter for this? Is the > service you're monitoring replicated in a cluster? > > On 17/03/16 18:06, Eric Wittmann wrote: >> Greetings. Apologies for coming in a bit late on this conversation. >> Tristan pointed me to it a couple of days ago and unfortunately I'm just >> now getting time to reply. >> >> I can try to quickly give an overview of apiman's (JBoss API Management >> Gateway) requirements. >> >> What we're trying to do is implement support for Limiting policies: >> >> * Rate Limiting/Throttling (e.g. limit of 100 requests per second) >> * Quotas (e.g. limit of 100,000,000 requests per month) >> * Transfer Quotas (e.g. limit of 2.5GB of data downloaded per day) >> >> We will need to support multiple backing implementations of the Rate >> Limiter, and we're trying to get Infinispan to be one of those >> implementations. >> >> In no particular order, we would need the following characteristics: >> >> - Can be "squishy" for quotas and transfer quotas: If you >> get 100,001,017 requests that's OK >> - Strict would be cool as an option: Hard-fail when the >> counter reaches the limit - no chance it will go over. >> - Lots of individual counters: users may publish 100s of >> APIs to the Gateway, and each API may be consumed by >> 100s or 1000s of users/client. Depending on configuration >> of the policy, *each* user/client has a separate limit. >> - Counters need to be created dynamically: users can >> add APIs via the Management UI, configure them to add >> policies (e.g. a Quota policy) and then publish them to >> a running Gateway, at which point end users can invoke >> the API through the Gateway, which will use a counter >> to enforce the Quota. >> - Counter values reset at the end of a time boundary: for >> example, at the end of the month the counter value for >> the example quota above would reset to 0. >> - Don't care (right now) what the counter value is: at the >> moment we simply need to know if some counter max value >> has been reached. In the future we would like to know >> when a max value is being "approached" (e.g. to notify a >> user) >> - Should be persistent: it would not be ideal for e.g. per- >> month quota values to be lost on server restart. >> >> That's all the high level requirements I can think of off the top of my >> head, and after reading all of the current messages in this thread. :) >> >> -Eric >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev at lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev >> > From bban at redhat.com Fri Mar 18 09:31:11 2016 From: bban at redhat.com (Bela Ban) Date: Fri, 18 Mar 2016 14:31:11 +0100 Subject: [infinispan-dev] Distributed Counter Discussion In-Reply-To: <56EC004F.7040101@redhat.com> References: <56EAE3FC.6030401@redhat.com> <56EBA0E2.7070106@redhat.com> <56EC004F.7040101@redhat.com> Message-ID: <56EC031F.10809@redhat.com> So actually you don't care if you have multiple counters in case of a network split, but you do care that the numbers of different counters get reconciled when a network partition heals. Example - C1: 1000 - Network split: C1: 1000, C2: 1000 - Different clients update counters on both sides of the partition: C1: 1500 C2: 1600 - Network split disappears, reconciling C1 to 2100: 1000 +500 +600. This means the 500 added to C1 should have been added to C2 as well, and the 600 to C2 should have been added to C1 If such a behavior would be acceptable, then we could do without CP and live with AP On 18/03/16 14:19, Eric Wittmann wrote: > Yes, precisely. The API Gateway itself is clustered. It services a > large volume of inbound traffic which it reverse-proxies to appropriate > back-end APIs after applying policies such as security, rate limiting, > caching, etc. > > -Eric > > On 3/18/2016 2:32 AM, Bela Ban wrote: >> Stupid question: whay do you need a distributed counter for this? Is the >> service you're monitoring replicated in a cluster? >> >> On 17/03/16 18:06, Eric Wittmann wrote: >>> Greetings. Apologies for coming in a bit late on this conversation. >>> Tristan pointed me to it a couple of days ago and unfortunately I'm just >>> now getting time to reply. >>> >>> I can try to quickly give an overview of apiman's (JBoss API Management >>> Gateway) requirements. >>> >>> What we're trying to do is implement support for Limiting policies: >>> >>> * Rate Limiting/Throttling (e.g. limit of 100 requests per second) >>> * Quotas (e.g. limit of 100,000,000 requests per month) >>> * Transfer Quotas (e.g. limit of 2.5GB of data downloaded per day) >>> >>> We will need to support multiple backing implementations of the Rate >>> Limiter, and we're trying to get Infinispan to be one of those >>> implementations. >>> >>> In no particular order, we would need the following characteristics: >>> >>> - Can be "squishy" for quotas and transfer quotas: If you >>> get 100,001,017 requests that's OK >>> - Strict would be cool as an option: Hard-fail when the >>> counter reaches the limit - no chance it will go over. >>> - Lots of individual counters: users may publish 100s of >>> APIs to the Gateway, and each API may be consumed by >>> 100s or 1000s of users/client. Depending on configuration >>> of the policy, *each* user/client has a separate limit. >>> - Counters need to be created dynamically: users can >>> add APIs via the Management UI, configure them to add >>> policies (e.g. a Quota policy) and then publish them to >>> a running Gateway, at which point end users can invoke >>> the API through the Gateway, which will use a counter >>> to enforce the Quota. >>> - Counter values reset at the end of a time boundary: for >>> example, at the end of the month the counter value for >>> the example quota above would reset to 0. >>> - Don't care (right now) what the counter value is: at the >>> moment we simply need to know if some counter max value >>> has been reached. In the future we would like to know >>> when a max value is being "approached" (e.g. to notify a >>> user) >>> - Should be persistent: it would not be ideal for e.g. per- >>> month quota values to be lost on server restart. >>> >>> That's all the high level requirements I can think of off the top of my >>> head, and after reading all of the current messages in this thread. :) >>> >>> -Eric >>> _______________________________________________ >>> infinispan-dev mailing list >>> infinispan-dev at lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>> >> > _______________________________________________ > infinispan-dev mailing list > infinispan-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev > -- Bela Ban, JGroups lead (http://www.jgroups.org) From eric.wittmann at redhat.com Fri Mar 18 10:19:11 2016 From: eric.wittmann at redhat.com (Eric Wittmann) Date: Fri, 18 Mar 2016 10:19:11 -0400 Subject: [infinispan-dev] Distributed Counter Discussion In-Reply-To: <56EC031F.10809@redhat.com> References: <56EAE3FC.6030401@redhat.com> <56EBA0E2.7070106@redhat.com> <56EC004F.7040101@redhat.com> <56EC031F.10809@redhat.com> Message-ID: <56EC0E5F.1020103@redhat.com> Agreed. :) On 3/18/2016 9:31 AM, Bela Ban wrote: > So actually you don't care if you have multiple counters in case of a > network split, but you do care that the numbers of different counters > get reconciled when a network partition heals. > > Example > - C1: 1000 > - Network split: C1: 1000, C2: 1000 > - Different clients update counters on both sides of the partition: C1: > 1500 C2: 1600 > - Network split disappears, reconciling C1 to 2100: 1000 +500 +600. This > means the 500 added to C1 should have been added to C2 as well, and the > 600 to C2 should have been added to C1 > > If such a behavior would be acceptable, then we could do without CP and > live with AP > > On 18/03/16 14:19, Eric Wittmann wrote: >> Yes, precisely. The API Gateway itself is clustered. It services a >> large volume of inbound traffic which it reverse-proxies to appropriate >> back-end APIs after applying policies such as security, rate limiting, >> caching, etc. >> >> -Eric >> >> On 3/18/2016 2:32 AM, Bela Ban wrote: >>> Stupid question: whay do you need a distributed counter for this? Is the >>> service you're monitoring replicated in a cluster? >>> >>> On 17/03/16 18:06, Eric Wittmann wrote: >>>> Greetings. Apologies for coming in a bit late on this conversation. >>>> Tristan pointed me to it a couple of days ago and unfortunately I'm just >>>> now getting time to reply. >>>> >>>> I can try to quickly give an overview of apiman's (JBoss API Management >>>> Gateway) requirements. >>>> >>>> What we're trying to do is implement support for Limiting policies: >>>> >>>> * Rate Limiting/Throttling (e.g. limit of 100 requests per second) >>>> * Quotas (e.g. limit of 100,000,000 requests per month) >>>> * Transfer Quotas (e.g. limit of 2.5GB of data downloaded per day) >>>> >>>> We will need to support multiple backing implementations of the Rate >>>> Limiter, and we're trying to get Infinispan to be one of those >>>> implementations. >>>> >>>> In no particular order, we would need the following characteristics: >>>> >>>> - Can be "squishy" for quotas and transfer quotas: If you >>>> get 100,001,017 requests that's OK >>>> - Strict would be cool as an option: Hard-fail when the >>>> counter reaches the limit - no chance it will go over. >>>> - Lots of individual counters: users may publish 100s of >>>> APIs to the Gateway, and each API may be consumed by >>>> 100s or 1000s of users/client. Depending on configuration >>>> of the policy, *each* user/client has a separate limit. >>>> - Counters need to be created dynamically: users can >>>> add APIs via the Management UI, configure them to add >>>> policies (e.g. a Quota policy) and then publish them to >>>> a running Gateway, at which point end users can invoke >>>> the API through the Gateway, which will use a counter >>>> to enforce the Quota. >>>> - Counter values reset at the end of a time boundary: for >>>> example, at the end of the month the counter value for >>>> the example quota above would reset to 0. >>>> - Don't care (right now) what the counter value is: at the >>>> moment we simply need to know if some counter max value >>>> has been reached. In the future we would like to know >>>> when a max value is being "approached" (e.g. to notify a >>>> user) >>>> - Should be persistent: it would not be ideal for e.g. per- >>>> month quota values to be lost on server restart. >>>> >>>> That's all the high level requirements I can think of off the top of my >>>> head, and after reading all of the current messages in this thread. :) >>>> >>>> -Eric >>>> _______________________________________________ >>>> infinispan-dev mailing list >>>> infinispan-dev at lists.jboss.org >>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>>> >>> >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev at lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev >> > From rory.odonnell at oracle.com Sat Mar 19 03:40:02 2016 From: rory.odonnell at oracle.com (Rory O'Donnell) Date: Sat, 19 Mar 2016 13:10:02 +0530 Subject: [infinispan-dev] Early Access builds of JDK 9 b110 & JDK 9 with Project Jigsaw b110 (#4664) are available on java.net Message-ID: <56ED0252.2090307@oracle.com> Hi Galder, Early Access b110 for JDK 9 is available on java.net, summary of changes are listed here . Among other fixes , the following changes are also included from builds 108 to 110 o removal of the OS X-specific com.apple.concurrent package o update to the JAX-WS RI integration to latest version (2.3.0-SNAPSHOT) o disabling of Diffie-Hellman keys smaller than 1024 bits Early Access b110 (#4664) for JDK 9 with Project Jigsaw is available on java.net. We expect this will be the last EA build before we integrate into JDK 9. This means that this EA build is essentially what will be jdk-9+111 next week. As always, please help out by trying out existing code and libraries to see what works and doesn't work. We have a detailed list of compatibility issues listed in JEP 261 [2]. Also feedback and questions from those that trying the EA build to modularize existing code would be appreciated. We also want to know what works and doesn't work here [3] Rgds,Rory [1] https://jdk9.java.net/jigsaw/ [2] http://openjdk.java.net/jeps/261 [3] http://mail.openjdk.java.net/pipermail/jigsaw-dev/ -- Rgds,Rory O'Donnell Quality Engineering Manager Oracle EMEA, Dublin,Ireland -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.jboss.org/pipermail/infinispan-dev/attachments/20160319/d2e46673/attachment.html From pedro at infinispan.org Mon Mar 21 06:07:34 2016 From: pedro at infinispan.org (Pedro Ruivo) Date: Mon, 21 Mar 2016 10:07:34 +0000 Subject: [infinispan-dev] XA transactions on a single CacheManager? In-Reply-To: References: Message-ID: <56EFC7E6.7040303@infinispan.org> Hi Sanne On 03/15/2016 07:22 PM, Sanne Grinovero wrote: > Hi all, > > I just noticed that when I'm making changes to multiple caches within > the same transaction, the transaction manager will treat this as XA > transactions. > > That seems suboptimal as they are all managed by the same resource; is > there a configuration I'm missing or should I open a JIRA to ask for > an improvement? what are you suggesting? Having a single XA resource per cache manager instead of one per cache? > > I'm not just asking for performance reasons: having to setup a > transaction manager is an annoying configuration step for Java SE > users. I got a little confused by this. You can use the same transaction manager in all the caches, so you have to configured it once. If you don't have one, you could use the DummyTransactionManager (we need to rename it!) in the core that contains the basic to handle transactions. Cheers, Pedro > > Thanks, > Sanne > _______________________________________________ > infinispan-dev mailing list > infinispan-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev > From pedro at infinispan.org Mon Mar 21 06:12:55 2016 From: pedro at infinispan.org (Pedro Ruivo) Date: Mon, 21 Mar 2016 10:12:55 +0000 Subject: [infinispan-dev] Distributed Counter Discussion In-Reply-To: <56EC0E5F.1020103@redhat.com> References: <56EAE3FC.6030401@redhat.com> <56EBA0E2.7070106@redhat.com> <56EC004F.7040101@redhat.com> <56EC031F.10809@redhat.com> <56EC0E5F.1020103@redhat.com> Message-ID: <56EFC927.804@infinispan.org> Hi all, @Eric, thanks for the requirements. @Bela, does JGroups counter supports that semantics (AP)? Infinispan does not have eventually consistency (yet) neither an update log. So, it can't reconcile the counter and you will lose one of the partition updates. On 03/18/2016 02:19 PM, Eric Wittmann wrote: > Agreed. :) > > On 3/18/2016 9:31 AM, Bela Ban wrote: >> So actually you don't care if you have multiple counters in case of a >> network split, but you do care that the numbers of different counters >> get reconciled when a network partition heals. >> >> Example >> - C1: 1000 >> - Network split: C1: 1000, C2: 1000 >> - Different clients update counters on both sides of the partition: C1: >> 1500 C2: 1600 >> - Network split disappears, reconciling C1 to 2100: 1000 +500 +600. This >> means the 500 added to C1 should have been added to C2 as well, and the >> 600 to C2 should have been added to C1 >> >> If such a behavior would be acceptable, then we could do without CP and >> live with AP >> >> On 18/03/16 14:19, Eric Wittmann wrote: >>> Yes, precisely. The API Gateway itself is clustered. It services a >>> large volume of inbound traffic which it reverse-proxies to appropriate >>> back-end APIs after applying policies such as security, rate limiting, >>> caching, etc. >>> >>> -Eric >>> >>> On 3/18/2016 2:32 AM, Bela Ban wrote: >>>> Stupid question: whay do you need a distributed counter for this? Is the >>>> service you're monitoring replicated in a cluster? >>>> >>>> On 17/03/16 18:06, Eric Wittmann wrote: >>>>> Greetings. Apologies for coming in a bit late on this conversation. >>>>> Tristan pointed me to it a couple of days ago and unfortunately I'm just >>>>> now getting time to reply. >>>>> >>>>> I can try to quickly give an overview of apiman's (JBoss API Management >>>>> Gateway) requirements. >>>>> >>>>> What we're trying to do is implement support for Limiting policies: >>>>> >>>>> * Rate Limiting/Throttling (e.g. limit of 100 requests per second) >>>>> * Quotas (e.g. limit of 100,000,000 requests per month) >>>>> * Transfer Quotas (e.g. limit of 2.5GB of data downloaded per day) >>>>> >>>>> We will need to support multiple backing implementations of the Rate >>>>> Limiter, and we're trying to get Infinispan to be one of those >>>>> implementations. >>>>> >>>>> In no particular order, we would need the following characteristics: >>>>> >>>>> - Can be "squishy" for quotas and transfer quotas: If you >>>>> get 100,001,017 requests that's OK >>>>> - Strict would be cool as an option: Hard-fail when the >>>>> counter reaches the limit - no chance it will go over. >>>>> - Lots of individual counters: users may publish 100s of >>>>> APIs to the Gateway, and each API may be consumed by >>>>> 100s or 1000s of users/client. Depending on configuration >>>>> of the policy, *each* user/client has a separate limit. >>>>> - Counters need to be created dynamically: users can >>>>> add APIs via the Management UI, configure them to add >>>>> policies (e.g. a Quota policy) and then publish them to >>>>> a running Gateway, at which point end users can invoke >>>>> the API through the Gateway, which will use a counter >>>>> to enforce the Quota. >>>>> - Counter values reset at the end of a time boundary: for >>>>> example, at the end of the month the counter value for >>>>> the example quota above would reset to 0. >>>>> - Don't care (right now) what the counter value is: at the >>>>> moment we simply need to know if some counter max value >>>>> has been reached. In the future we would like to know >>>>> when a max value is being "approached" (e.g. to notify a >>>>> user) >>>>> - Should be persistent: it would not be ideal for e.g. per- >>>>> month quota values to be lost on server restart. >>>>> >>>>> That's all the high level requirements I can think of off the top of my >>>>> head, and after reading all of the current messages in this thread. :) >>>>> >>>>> -Eric >>>>> _______________________________________________ >>>>> infinispan-dev mailing list >>>>> infinispan-dev at lists.jboss.org >>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>>>> >>>> >>> _______________________________________________ >>> infinispan-dev mailing list >>> infinispan-dev at lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>> >> > _______________________________________________ > infinispan-dev mailing list > infinispan-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev > From sanne at infinispan.org Mon Mar 21 07:19:38 2016 From: sanne at infinispan.org (Sanne Grinovero) Date: Mon, 21 Mar 2016 11:19:38 +0000 Subject: [infinispan-dev] XA transactions on a single CacheManager? In-Reply-To: <56EFC7E6.7040303@infinispan.org> References: <56EFC7E6.7040303@infinispan.org> Message-ID: On 21 March 2016 at 10:07, Pedro Ruivo wrote: > Hi Sanne > > On 03/15/2016 07:22 PM, Sanne Grinovero wrote: >> Hi all, >> >> I just noticed that when I'm making changes to multiple caches within >> the same transaction, the transaction manager will treat this as XA >> transactions. >> >> That seems suboptimal as they are all managed by the same resource; is >> there a configuration I'm missing or should I open a JIRA to ask for >> an improvement? > > what are you suggesting? Having a single XA resource per cache manager > instead of one per cache? Yes, I actually expected that to be the case already. Would that be complex? I am not familiar at all with this code, I was just surprised as I don't see why it is the way it is: I don't need XA transactions when writing to multiple tables on a same RDBMs ;) My guess is that we originally thought of Caches a being used mostly independently, but I think experience is telling us differently. >> I'm not just asking for performance reasons: having to setup a >> transaction manager is an annoying configuration step for Java SE >> users. > > I got a little confused by this. You can use the same transaction > manager in all the caches, so you have to configured it once. > If you don't have one, you could use the DummyTransactionManager (we > need to rename it!) in the core that contains the basic to handle > transactions. To support XA transactions the transaction manager requires both a different configuration and different dependencies. I noticed while working on a "Getting started with Hibernate OGM" which needs to be simple of course, and this example happens to use Infinispan. Since we need writes on multiple caches, this makes the setup phase of the tutorial much harder, to the point that I'm considering to simply switch to something like MongoDB so that I can keep the OGM tutorial focused. > > Cheers, > Pedro > >> >> Thanks, >> Sanne >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev at lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev >> > _______________________________________________ > infinispan-dev mailing list > infinispan-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev From bban at redhat.com Mon Mar 21 07:43:03 2016 From: bban at redhat.com (Bela Ban) Date: Mon, 21 Mar 2016 12:43:03 +0100 Subject: [infinispan-dev] Distributed Counter Discussion In-Reply-To: <56EFC927.804@infinispan.org> References: <56EAE3FC.6030401@redhat.com> <56EBA0E2.7070106@redhat.com> <56EC004F.7040101@redhat.com> <56EC031F.10809@redhat.com> <56EC0E5F.1020103@redhat.com> <56EFC927.804@infinispan.org> Message-ID: <56EFDE47.4000101@redhat.com> On 21/03/16 11:12, Pedro Ruivo wrote: > Hi all, > > @Eric, thanks for the requirements. > > @Bela, does JGroups counter supports that semantics (AP)? No. You'd have to catch the MergeView and do this manually. > Infinispan does not have eventually consistency (yet) neither an update log. So, it > can't reconcile the counter and you will lose one of the partition updates. Same for the JGroups counter service. The jgroups-raft CounterService provides strong consistency, but at the expense of availability. > On 03/18/2016 02:19 PM, Eric Wittmann wrote: >> Agreed. :) >> >> On 3/18/2016 9:31 AM, Bela Ban wrote: >>> So actually you don't care if you have multiple counters in case of a >>> network split, but you do care that the numbers of different counters >>> get reconciled when a network partition heals. >>> >>> Example >>> - C1: 1000 >>> - Network split: C1: 1000, C2: 1000 >>> - Different clients update counters on both sides of the partition: C1: >>> 1500 C2: 1600 >>> - Network split disappears, reconciling C1 to 2100: 1000 +500 +600. This >>> means the 500 added to C1 should have been added to C2 as well, and the >>> 600 to C2 should have been added to C1 >>> >>> If such a behavior would be acceptable, then we could do without CP and >>> live with AP >>> >>> On 18/03/16 14:19, Eric Wittmann wrote: >>>> Yes, precisely. The API Gateway itself is clustered. It services a >>>> large volume of inbound traffic which it reverse-proxies to appropriate >>>> back-end APIs after applying policies such as security, rate limiting, >>>> caching, etc. >>>> >>>> -Eric >>>> >>>> On 3/18/2016 2:32 AM, Bela Ban wrote: >>>>> Stupid question: whay do you need a distributed counter for this? Is the >>>>> service you're monitoring replicated in a cluster? >>>>> >>>>> On 17/03/16 18:06, Eric Wittmann wrote: >>>>>> Greetings. Apologies for coming in a bit late on this conversation. >>>>>> Tristan pointed me to it a couple of days ago and unfortunately I'm just >>>>>> now getting time to reply. >>>>>> >>>>>> I can try to quickly give an overview of apiman's (JBoss API Management >>>>>> Gateway) requirements. >>>>>> >>>>>> What we're trying to do is implement support for Limiting policies: >>>>>> >>>>>> * Rate Limiting/Throttling (e.g. limit of 100 requests per second) >>>>>> * Quotas (e.g. limit of 100,000,000 requests per month) >>>>>> * Transfer Quotas (e.g. limit of 2.5GB of data downloaded per day) >>>>>> >>>>>> We will need to support multiple backing implementations of the Rate >>>>>> Limiter, and we're trying to get Infinispan to be one of those >>>>>> implementations. >>>>>> >>>>>> In no particular order, we would need the following characteristics: >>>>>> >>>>>> - Can be "squishy" for quotas and transfer quotas: If you >>>>>> get 100,001,017 requests that's OK >>>>>> - Strict would be cool as an option: Hard-fail when the >>>>>> counter reaches the limit - no chance it will go over. >>>>>> - Lots of individual counters: users may publish 100s of >>>>>> APIs to the Gateway, and each API may be consumed by >>>>>> 100s or 1000s of users/client. Depending on configuration >>>>>> of the policy, *each* user/client has a separate limit. >>>>>> - Counters need to be created dynamically: users can >>>>>> add APIs via the Management UI, configure them to add >>>>>> policies (e.g. a Quota policy) and then publish them to >>>>>> a running Gateway, at which point end users can invoke >>>>>> the API through the Gateway, which will use a counter >>>>>> to enforce the Quota. >>>>>> - Counter values reset at the end of a time boundary: for >>>>>> example, at the end of the month the counter value for >>>>>> the example quota above would reset to 0. >>>>>> - Don't care (right now) what the counter value is: at the >>>>>> moment we simply need to know if some counter max value >>>>>> has been reached. In the future we would like to know >>>>>> when a max value is being "approached" (e.g. to notify a >>>>>> user) >>>>>> - Should be persistent: it would not be ideal for e.g. per- >>>>>> month quota values to be lost on server restart. >>>>>> >>>>>> That's all the high level requirements I can think of off the top of my >>>>>> head, and after reading all of the current messages in this thread. :) >>>>>> >>>>>> -Eric >>>>>> _______________________________________________ >>>>>> infinispan-dev mailing list >>>>>> infinispan-dev at lists.jboss.org >>>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>>>>> >>>>> >>>> _______________________________________________ >>>> infinispan-dev mailing list >>>> infinispan-dev at lists.jboss.org >>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>>> >>> >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev at lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev >> > _______________________________________________ > infinispan-dev mailing list > infinispan-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev > -- Bela Ban, JGroups lead (http://www.jgroups.org) From pedro at infinispan.org Mon Mar 21 08:18:59 2016 From: pedro at infinispan.org (Pedro Ruivo) Date: Mon, 21 Mar 2016 12:18:59 +0000 Subject: [infinispan-dev] XA transactions on a single CacheManager? In-Reply-To: References: <56EFC7E6.7040303@infinispan.org> Message-ID: <56EFE6B3.3060004@infinispan.org> On 03/21/2016 11:19 AM, Sanne Grinovero wrote: > On 21 March 2016 at 10:07, Pedro Ruivo wrote: >> Hi Sanne >> >> On 03/15/2016 07:22 PM, Sanne Grinovero wrote: >>> Hi all, >>> >>> I just noticed that when I'm making changes to multiple caches within >>> the same transaction, the transaction manager will treat this as XA >>> transactions. >>> >>> That seems suboptimal as they are all managed by the same resource; is >>> there a configuration I'm missing or should I open a JIRA to ask for >>> an improvement? >> >> what are you suggesting? Having a single XA resource per cache manager >> instead of one per cache? > > Yes, I actually expected that to be the case already. > Would that be complex? I am not familiar at all with this code, I was > just surprised as I don't see why it is the way it is: I don't need XA > transactions when writing to multiple tables on a same RDBMs ;) > > My guess is that we originally thought of Caches a being used mostly > independently, but I think experience is telling us differently. It makes sense since the tables in RDBMs has some kind of relation between them. Caches on other hand, they haven't. It will not be impossible to move the XA to the cache manager, although, it would add some complexity to our xa implementation. Randim already opened a JIRA. It would be good to know if someone has any issue with the per-cache-manager xa :) > >>> I'm not just asking for performance reasons: having to setup a >>> transaction manager is an annoying configuration step for Java SE >>> users. >> >> I got a little confused by this. You can use the same transaction >> manager in all the caches, so you have to configured it once. >> If you don't have one, you could use the DummyTransactionManager (we >> need to rename it!) in the core that contains the basic to handle >> transactions. > > To support XA transactions the transaction manager requires both a > different configuration > and different dependencies. > > I noticed while working on a "Getting started with Hibernate OGM" > which needs to be simple of course, > and this example happens to use Infinispan. > Since we need writes on multiple caches, this makes the setup phase of > the tutorial much harder, to the point that I'm considering to simply > switch to something like MongoDB so that I can keep the OGM tutorial > focused. Even if you have a per-cache-manager xa, you still have to configure the TransactionManager. If you don't want to complicate the "getting start" you could use the DummyTransactionManager to achieve the same result. In infinispan, you only need to set the "transaction-manager-lookup" in tag to fetch the DummyTransactionManager. > >> >> Cheers, >> Pedro >> >>> >>> Thanks, >>> Sanne >>> _______________________________________________ >>> infinispan-dev mailing list >>> infinispan-dev at lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>> >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev at lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev > _______________________________________________ > infinispan-dev mailing list > infinispan-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev > From rvansa at redhat.com Mon Mar 21 08:44:32 2016 From: rvansa at redhat.com (Radim Vansa) Date: Mon, 21 Mar 2016 13:44:32 +0100 Subject: [infinispan-dev] XA transactions on a single CacheManager? In-Reply-To: <56EFE6B3.3060004@infinispan.org> References: <56EFC7E6.7040303@infinispan.org> <56EFE6B3.3060004@infinispan.org> Message-ID: <56EFECB0.9060202@redhat.com> On 03/21/2016 01:18 PM, Pedro Ruivo wrote: > > On 03/21/2016 11:19 AM, Sanne Grinovero wrote: >> On 21 March 2016 at 10:07, Pedro Ruivo wrote: >>> Hi Sanne >>> >>> On 03/15/2016 07:22 PM, Sanne Grinovero wrote: >>>> Hi all, >>>> >>>> I just noticed that when I'm making changes to multiple caches within >>>> the same transaction, the transaction manager will treat this as XA >>>> transactions. >>>> >>>> That seems suboptimal as they are all managed by the same resource; is >>>> there a configuration I'm missing or should I open a JIRA to ask for >>>> an improvement? >>> what are you suggesting? Having a single XA resource per cache manager >>> instead of one per cache? >> Yes, I actually expected that to be the case already. >> Would that be complex? I am not familiar at all with this code, I was >> just surprised as I don't see why it is the way it is: I don't need XA >> transactions when writing to multiple tables on a same RDBMs ;) >> >> My guess is that we originally thought of Caches a being used mostly >> independently, but I think experience is telling us differently. > It makes sense since the tables in RDBMs has some kind of relation > between them. Caches on other hand, they haven't. We don't offer any constraints as RDBMs have, but as columns in RDBMs contain FKs to row in another table, I would expect that data in cache also often contains keys for other caches/ keys into the same cache. Either app-driven, or automatically generated (as OGM probably does). It's not just OGM, values in entity collection cache in ORM 2LC contains list of keys in entity cache. > > It will not be impossible to move the XA to the cache manager, although, > it would add some complexity to our xa implementation. Randim already > opened a JIRA. > > It would be good to know if someone has any issue with the > per-cache-manager xa :) Do you expect that people dig as deep as Sanne, check and complain in JIRA? IMO at best, they get surprised when they find out, expect this to be the intent and workaround that. Radim > >>>> I'm not just asking for performance reasons: having to setup a >>>> transaction manager is an annoying configuration step for Java SE >>>> users. >>> I got a little confused by this. You can use the same transaction >>> manager in all the caches, so you have to configured it once. >>> If you don't have one, you could use the DummyTransactionManager (we >>> need to rename it!) in the core that contains the basic to handle >>> transactions. >> To support XA transactions the transaction manager requires both a >> different configuration >> and different dependencies. >> >> I noticed while working on a "Getting started with Hibernate OGM" >> which needs to be simple of course, >> and this example happens to use Infinispan. >> Since we need writes on multiple caches, this makes the setup phase of >> the tutorial much harder, to the point that I'm considering to simply >> switch to something like MongoDB so that I can keep the OGM tutorial >> focused. > Even if you have a per-cache-manager xa, you still have to configure the > TransactionManager. > > If you don't want to complicate the "getting start" you could use the > DummyTransactionManager to achieve the same result. In infinispan, you > only need to set the "transaction-manager-lookup" in tag > to fetch the DummyTransactionManager. > >>> Cheers, >>> Pedro >>> >>>> Thanks, >>>> Sanne >>>> _______________________________________________ >>>> infinispan-dev mailing list >>>> infinispan-dev at lists.jboss.org >>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>>> >>> _______________________________________________ >>> infinispan-dev mailing list >>> infinispan-dev at lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev at lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev >> > _______________________________________________ > infinispan-dev mailing list > infinispan-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev -- Radim Vansa JBoss Performance Team From sanne at infinispan.org Mon Mar 21 08:53:06 2016 From: sanne at infinispan.org (Sanne Grinovero) Date: Mon, 21 Mar 2016 12:53:06 +0000 Subject: [infinispan-dev] XA transactions on a single CacheManager? In-Reply-To: <56EFE6B3.3060004@infinispan.org> References: <56EFC7E6.7040303@infinispan.org> <56EFE6B3.3060004@infinispan.org> Message-ID: On 21 March 2016 at 12:18, Pedro Ruivo wrote: > > > On 03/21/2016 11:19 AM, Sanne Grinovero wrote: >> On 21 March 2016 at 10:07, Pedro Ruivo wrote: >>> Hi Sanne >>> >>> On 03/15/2016 07:22 PM, Sanne Grinovero wrote: >>>> Hi all, >>>> >>>> I just noticed that when I'm making changes to multiple caches within >>>> the same transaction, the transaction manager will treat this as XA >>>> transactions. >>>> >>>> That seems suboptimal as they are all managed by the same resource; is >>>> there a configuration I'm missing or should I open a JIRA to ask for >>>> an improvement? >>> >>> what are you suggesting? Having a single XA resource per cache manager >>> instead of one per cache? >> >> Yes, I actually expected that to be the case already. >> Would that be complex? I am not familiar at all with this code, I was >> just surprised as I don't see why it is the way it is: I don't need XA >> transactions when writing to multiple tables on a same RDBMs ;) >> >> My guess is that we originally thought of Caches a being used mostly >> independently, but I think experience is telling us differently. > > It makes sense since the tables in RDBMs has some kind of relation > between them. Caches on other hand, they haven't. That statement is too broad, and proven wrong with real world examples and practical requirements from our users; in my experience it also is the primary reason to enable transactions at all. I'd rather question why this wasn't done than question which use cases are appropriate, as I don't see what benefit it brings to consider each cache a different resource? Am I missing a trade-off or is it fair to say that there are no drawbacks in improving this? > It will not be impossible to move the XA to the cache manager, although, > it would add some complexity to our xa implementation. Randim already > opened a JIRA. > > It would be good to know if someone has any issue with the > per-cache-manager xa :) > >> >>>> I'm not just asking for performance reasons: having to setup a >>>> transaction manager is an annoying configuration step for Java SE >>>> users. >>> >>> I got a little confused by this. You can use the same transaction >>> manager in all the caches, so you have to configured it once. >>> If you don't have one, you could use the DummyTransactionManager (we >>> need to rename it!) in the core that contains the basic to handle >>> transactions. >> >> To support XA transactions the transaction manager requires both a >> different configuration >> and different dependencies. >> >> I noticed while working on a "Getting started with Hibernate OGM" >> which needs to be simple of course, >> and this example happens to use Infinispan. >> Since we need writes on multiple caches, this makes the setup phase of >> the tutorial much harder, to the point that I'm considering to simply >> switch to something like MongoDB so that I can keep the OGM tutorial >> focused. > > Even if you have a per-cache-manager xa, you still have to configure the > TransactionManager. > > If you don't want to complicate the "getting start" you could use the > DummyTransactionManager to achieve the same result. In infinispan, you > only need to set the "transaction-manager-lookup" in tag > to fetch the DummyTransactionManager. it might just be an example, but that doesn't look good enough. Thanks, Sanne > >> >>> >>> Cheers, >>> Pedro >>> >>>> >>>> Thanks, >>>> Sanne >>>> _______________________________________________ >>>> infinispan-dev mailing list >>>> infinispan-dev at lists.jboss.org >>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>>> >>> _______________________________________________ >>> infinispan-dev mailing list >>> infinispan-dev at lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev at lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev >> > _______________________________________________ > infinispan-dev mailing list > infinispan-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev From pedro at infinispan.org Mon Mar 21 09:30:35 2016 From: pedro at infinispan.org (Pedro Ruivo) Date: Mon, 21 Mar 2016 13:30:35 +0000 Subject: [infinispan-dev] XA transactions on a single CacheManager? In-Reply-To: References: <56EFC7E6.7040303@infinispan.org> <56EFE6B3.3060004@infinispan.org> Message-ID: <56EFF77B.7070906@infinispan.org> On 03/21/2016 12:53 PM, Sanne Grinovero wrote: > > I'd rather question why this wasn't done than question which use cases > are appropriate, as I don't see what benefit it brings to consider > each cache a different resource? Am I missing a trade-off or is it > fair to say that there are no drawbacks in improving this? I'm not aware of any issue with it except code simplicity :) I've added this to 9.x wishlist. From sanne at infinispan.org Mon Mar 21 09:45:01 2016 From: sanne at infinispan.org (Sanne Grinovero) Date: Mon, 21 Mar 2016 13:45:01 +0000 Subject: [infinispan-dev] XA transactions on a single CacheManager? In-Reply-To: <56EFF77B.7070906@infinispan.org> References: <56EFC7E6.7040303@infinispan.org> <56EFE6B3.3060004@infinispan.org> <56EFF77B.7070906@infinispan.org> Message-ID: On 21 March 2016 at 13:30, Pedro Ruivo wrote: > > > On 03/21/2016 12:53 PM, Sanne Grinovero wrote: >> >> I'd rather question why this wasn't done than question which use cases >> are appropriate, as I don't see what benefit it brings to consider >> each cache a different resource? Am I missing a trade-off or is it >> fair to say that there are no drawbacks in improving this? > > I'm not aware of any issue with it except code simplicity :) > > I've added this to 9.x wishlist. Thanks! From emmanuel at hibernate.org Tue Mar 22 03:24:30 2016 From: emmanuel at hibernate.org (Emmanuel Bernard) Date: Tue, 22 Mar 2016 08:24:30 +0100 Subject: [infinispan-dev] Google load balancer design Message-ID: <4DEC6438-C168-4434-9FC2-3449F5E3A6B0@hibernate.org> http://blog.acolyer.org/2016/03/21/maglev-a-fast-and-reliable-software-network-load-balancer/ Interesting read. Some data points I extracted. 350 ns per parquet. Ensuring a 300us during load. Some math I was not aware of: [quote] -- Let N be the size of a VIP?s backend pool. A table is constructed with M entries, where M is a prime number and M > 100 * N, which will ensure at most a 1% difference in hash space assigned to backends. -- Might be interesting in our virtual nodes logic. The segment size might not be too friendly here though. They do use try to ensure relative stability of the targeted backend for a given key even when the pool of backend changes (e.g. one server down) Emmanuel -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.jboss.org/pipermail/infinispan-dev/attachments/20160322/41f173df/attachment.html From rvansa at redhat.com Tue Mar 22 03:56:43 2016 From: rvansa at redhat.com (Radim Vansa) Date: Tue, 22 Mar 2016 08:56:43 +0100 Subject: [infinispan-dev] Google load balancer design In-Reply-To: <4DEC6438-C168-4434-9FC2-3449F5E3A6B0@hibernate.org> References: <4DEC6438-C168-4434-9FC2-3449F5E3A6B0@hibernate.org> Message-ID: <56F0FABB.2060004@redhat.com> On 03/22/2016 08:24 AM, Emmanuel Bernard wrote: > http://blog.acolyer.org/2016/03/21/maglev-a-fast-and-reliable-software-network-load-balancer/ > > Interesting read. Some data points I extracted. > > 350 ns per parquet. Ensuring a 300us during load. Regrettably JGroups does not bypass Linux kernel (yet). It sounds like the forwarder is written for the particular NIC model (or maybe reuses the driver internally), though it's in userspace. > > Some math I was not aware of: > > [quote] > > -- > > Let /N/be the size of a VIP?s backend pool. A table is constructed > with/M/entries, where/M/is a prime number and/M > 100 * N/, which will > ensure at most a 1% difference in hash space assigned to backends. > > -- > > Might be interesting in our virtual nodes logic. The segment size > might not be too friendly here though. No virtual nodes since Infinispan 5.2.0. R. > > They do use try to ensure relative stability of the targeted backend > for a given key even when the pool of backend changes (e.g. one server > down) > > Emmanuel > > > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev -- Radim Vansa JBoss Performance Team From emmanuel at hibernate.org Tue Mar 22 10:59:13 2016 From: emmanuel at hibernate.org (Emmanuel Bernard) Date: Tue, 22 Mar 2016 14:59:13 +0000 Subject: [infinispan-dev] Google load balancer design In-Reply-To: <56F0FABB.2060004@redhat.com> References: <4DEC6438-C168-4434-9FC2-3449F5E3A6B0@hibernate.org> <56F0FABB.2060004@redhat.com> Message-ID: <20160322145913.GA3898@hibernate.org> On Tue 2016-03-22 8:56, Radim Vansa wrote: > On 03/22/2016 08:24 AM, Emmanuel Bernard wrote: > > http://blog.acolyer.org/2016/03/21/maglev-a-fast-and-reliable-software-network-load-balancer/ > > > > Interesting read. Some data points I extracted. > > > > 350 ns per parquet. Ensuring a 300us during load. > > Regrettably JGroups does not bypass Linux kernel (yet). It sounds like > the forwarder is written for the particular NIC model (or maybe reuses > the driver internally), though it's in userspace. Yes I was even thinking they did write their own NIC firmware :) From dan.berindei at gmail.com Tue Mar 22 13:04:09 2016 From: dan.berindei at gmail.com (Dan Berindei) Date: Tue, 22 Mar 2016 19:04:09 +0200 Subject: [infinispan-dev] Distributed Counter Discussion In-Reply-To: <56EFDE47.4000101@redhat.com> References: <56EAE3FC.6030401@redhat.com> <56EBA0E2.7070106@redhat.com> <56EC004F.7040101@redhat.com> <56EC031F.10809@redhat.com> <56EC0E5F.1020103@redhat.com> <56EFC927.804@infinispan.org> <56EFDE47.4000101@redhat.com> Message-ID: On Mon, Mar 21, 2016 at 1:43 PM, Bela Ban wrote: > > > On 21/03/16 11:12, Pedro Ruivo wrote: >> Hi all, >> >> @Eric, thanks for the requirements. >> >> @Bela, does JGroups counter supports that semantics (AP)? > > No. You'd have to catch the MergeView and do this manually. I should also mention that you don't get a "cluster split" event. With when a cluster ABC splits into A and BC and merges back, you could get quite a view sequence like this: A, B, C: A|3 [A, B, C] A: A|4 [A, B] A: A|5 [A] (could be missing) B, C: B|4 [B, C] A, B, C: B|6 [B, C, A] (merge view) So it's not that easy to keep track of counter additions "since the split". > >> Infinispan does not have eventually consistency (yet) neither an update log. So, it >> can't reconcile the counter and you will lose one of the partition updates. > > Same for the JGroups counter service. The jgroups-raft CounterService > provides strong consistency, but at the expense of availability. > >> On 03/18/2016 02:19 PM, Eric Wittmann wrote: >>> Agreed. :) >>> >>> On 3/18/2016 9:31 AM, Bela Ban wrote: >>>> So actually you don't care if you have multiple counters in case of a >>>> network split, but you do care that the numbers of different counters >>>> get reconciled when a network partition heals. >>>> >>>> Example >>>> - C1: 1000 >>>> - Network split: C1: 1000, C2: 1000 >>>> - Different clients update counters on both sides of the partition: C1: >>>> 1500 C2: 1600 >>>> - Network split disappears, reconciling C1 to 2100: 1000 +500 +600. This >>>> means the 500 added to C1 should have been added to C2 as well, and the >>>> 600 to C2 should have been added to C1 >>>> >>>> If such a behavior would be acceptable, then we could do without CP and >>>> live with AP >>>> >>>> On 18/03/16 14:19, Eric Wittmann wrote: >>>>> Yes, precisely. The API Gateway itself is clustered. It services a >>>>> large volume of inbound traffic which it reverse-proxies to appropriate >>>>> back-end APIs after applying policies such as security, rate limiting, >>>>> caching, etc. >>>>> >>>>> -Eric >>>>> >>>>> On 3/18/2016 2:32 AM, Bela Ban wrote: >>>>>> Stupid question: whay do you need a distributed counter for this? Is the >>>>>> service you're monitoring replicated in a cluster? >>>>>> >>>>>> On 17/03/16 18:06, Eric Wittmann wrote: >>>>>>> Greetings. Apologies for coming in a bit late on this conversation. >>>>>>> Tristan pointed me to it a couple of days ago and unfortunately I'm just >>>>>>> now getting time to reply. >>>>>>> >>>>>>> I can try to quickly give an overview of apiman's (JBoss API Management >>>>>>> Gateway) requirements. >>>>>>> >>>>>>> What we're trying to do is implement support for Limiting policies: >>>>>>> >>>>>>> * Rate Limiting/Throttling (e.g. limit of 100 requests per second) >>>>>>> * Quotas (e.g. limit of 100,000,000 requests per month) >>>>>>> * Transfer Quotas (e.g. limit of 2.5GB of data downloaded per day) >>>>>>> >>>>>>> We will need to support multiple backing implementations of the Rate >>>>>>> Limiter, and we're trying to get Infinispan to be one of those >>>>>>> implementations. >>>>>>> >>>>>>> In no particular order, we would need the following characteristics: >>>>>>> >>>>>>> - Can be "squishy" for quotas and transfer quotas: If you >>>>>>> get 100,001,017 requests that's OK >>>>>>> - Strict would be cool as an option: Hard-fail when the >>>>>>> counter reaches the limit - no chance it will go over. >>>>>>> - Lots of individual counters: users may publish 100s of >>>>>>> APIs to the Gateway, and each API may be consumed by >>>>>>> 100s or 1000s of users/client. Depending on configuration >>>>>>> of the policy, *each* user/client has a separate limit. >>>>>>> - Counters need to be created dynamically: users can >>>>>>> add APIs via the Management UI, configure them to add >>>>>>> policies (e.g. a Quota policy) and then publish them to >>>>>>> a running Gateway, at which point end users can invoke >>>>>>> the API through the Gateway, which will use a counter >>>>>>> to enforce the Quota. >>>>>>> - Counter values reset at the end of a time boundary: for >>>>>>> example, at the end of the month the counter value for >>>>>>> the example quota above would reset to 0. >>>>>>> - Don't care (right now) what the counter value is: at the >>>>>>> moment we simply need to know if some counter max value >>>>>>> has been reached. In the future we would like to know >>>>>>> when a max value is being "approached" (e.g. to notify a >>>>>>> user) >>>>>>> - Should be persistent: it would not be ideal for e.g. per- >>>>>>> month quota values to be lost on server restart. >>>>>>> >>>>>>> That's all the high level requirements I can think of off the top of my >>>>>>> head, and after reading all of the current messages in this thread. :) >>>>>>> >>>>>>> -Eric >>>>>>> _______________________________________________ >>>>>>> infinispan-dev mailing list >>>>>>> infinispan-dev at lists.jboss.org >>>>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>>>>>> >>>>>> >>>>> _______________________________________________ >>>>> infinispan-dev mailing list >>>>> infinispan-dev at lists.jboss.org >>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>>>> >>>> >>> _______________________________________________ >>> infinispan-dev mailing list >>> infinispan-dev at lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>> >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev at lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev >> > > -- > Bela Ban, JGroups lead (http://www.jgroups.org) > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev From bban at redhat.com Tue Mar 22 13:12:06 2016 From: bban at redhat.com (Bela Ban) Date: Tue, 22 Mar 2016 18:12:06 +0100 Subject: [infinispan-dev] Distributed Counter Discussion In-Reply-To: References: <56EAE3FC.6030401@redhat.com> <56EBA0E2.7070106@redhat.com> <56EC004F.7040101@redhat.com> <56EC031F.10809@redhat.com> <56EC0E5F.1020103@redhat.com> <56EFC927.804@infinispan.org> <56EFDE47.4000101@redhat.com> Message-ID: <56F17CE6.4020103@redhat.com> On 22/03/16 18:04, Dan Berindei wrote: > On Mon, Mar 21, 2016 at 1:43 PM, Bela Ban wrote: >> >> >> On 21/03/16 11:12, Pedro Ruivo wrote: >>> Hi all, >>> >>> @Eric, thanks for the requirements. >>> >>> @Bela, does JGroups counter supports that semantics (AP)? >> >> No. You'd have to catch the MergeView and do this manually. > > I should also mention that you don't get a "cluster split" event. With > when a cluster ABC splits into A and BC and merges back, you could get > quite a view sequence like this: > > A, B, C: A|3 [A, B, C] > A: A|4 [A, B] > A: A|5 [A] (could be missing) How's that possible? Are you assuming that A still gets heartbeats from B? Otherwise A will definitely get a singleton view A|5=[A], unless the cluster heals in the meantime (before B is suspected and excluded). Note that A|4 on A may or may not be received; if FD_ALL is used then chances are view [A] will be received directly after [ABC] on A. > B, C: B|4 [B, C] > A, B, C: B|6 [B, C, A] (merge view) Note that you could keep the order if you installed a custom view/merge-view creation policy, but by default members are sorted according to the UUIDs. > So it's not that easy to keep track of counter additions "since the split". Agreed. >>> Infinispan does not have eventually consistency (yet) neither an update log. So, it >>> can't reconcile the counter and you will lose one of the partition updates. >> >> Same for the JGroups counter service. The jgroups-raft CounterService >> provides strong consistency, but at the expense of availability. >> >>> On 03/18/2016 02:19 PM, Eric Wittmann wrote: >>>> Agreed. :) >>>> >>>> On 3/18/2016 9:31 AM, Bela Ban wrote: >>>>> So actually you don't care if you have multiple counters in case of a >>>>> network split, but you do care that the numbers of different counters >>>>> get reconciled when a network partition heals. >>>>> >>>>> Example >>>>> - C1: 1000 >>>>> - Network split: C1: 1000, C2: 1000 >>>>> - Different clients update counters on both sides of the partition: C1: >>>>> 1500 C2: 1600 >>>>> - Network split disappears, reconciling C1 to 2100: 1000 +500 +600. This >>>>> means the 500 added to C1 should have been added to C2 as well, and the >>>>> 600 to C2 should have been added to C1 >>>>> >>>>> If such a behavior would be acceptable, then we could do without CP and >>>>> live with AP >>>>> >>>>> On 18/03/16 14:19, Eric Wittmann wrote: >>>>>> Yes, precisely. The API Gateway itself is clustered. It services a >>>>>> large volume of inbound traffic which it reverse-proxies to appropriate >>>>>> back-end APIs after applying policies such as security, rate limiting, >>>>>> caching, etc. >>>>>> >>>>>> -Eric >>>>>> >>>>>> On 3/18/2016 2:32 AM, Bela Ban wrote: >>>>>>> Stupid question: whay do you need a distributed counter for this? Is the >>>>>>> service you're monitoring replicated in a cluster? >>>>>>> >>>>>>> On 17/03/16 18:06, Eric Wittmann wrote: >>>>>>>> Greetings. Apologies for coming in a bit late on this conversation. >>>>>>>> Tristan pointed me to it a couple of days ago and unfortunately I'm just >>>>>>>> now getting time to reply. >>>>>>>> >>>>>>>> I can try to quickly give an overview of apiman's (JBoss API Management >>>>>>>> Gateway) requirements. >>>>>>>> >>>>>>>> What we're trying to do is implement support for Limiting policies: >>>>>>>> >>>>>>>> * Rate Limiting/Throttling (e.g. limit of 100 requests per second) >>>>>>>> * Quotas (e.g. limit of 100,000,000 requests per month) >>>>>>>> * Transfer Quotas (e.g. limit of 2.5GB of data downloaded per day) >>>>>>>> >>>>>>>> We will need to support multiple backing implementations of the Rate >>>>>>>> Limiter, and we're trying to get Infinispan to be one of those >>>>>>>> implementations. >>>>>>>> >>>>>>>> In no particular order, we would need the following characteristics: >>>>>>>> >>>>>>>> - Can be "squishy" for quotas and transfer quotas: If you >>>>>>>> get 100,001,017 requests that's OK >>>>>>>> - Strict would be cool as an option: Hard-fail when the >>>>>>>> counter reaches the limit - no chance it will go over. >>>>>>>> - Lots of individual counters: users may publish 100s of >>>>>>>> APIs to the Gateway, and each API may be consumed by >>>>>>>> 100s or 1000s of users/client. Depending on configuration >>>>>>>> of the policy, *each* user/client has a separate limit. >>>>>>>> - Counters need to be created dynamically: users can >>>>>>>> add APIs via the Management UI, configure them to add >>>>>>>> policies (e.g. a Quota policy) and then publish them to >>>>>>>> a running Gateway, at which point end users can invoke >>>>>>>> the API through the Gateway, which will use a counter >>>>>>>> to enforce the Quota. >>>>>>>> - Counter values reset at the end of a time boundary: for >>>>>>>> example, at the end of the month the counter value for >>>>>>>> the example quota above would reset to 0. >>>>>>>> - Don't care (right now) what the counter value is: at the >>>>>>>> moment we simply need to know if some counter max value >>>>>>>> has been reached. In the future we would like to know >>>>>>>> when a max value is being "approached" (e.g. to notify a >>>>>>>> user) >>>>>>>> - Should be persistent: it would not be ideal for e.g. per- >>>>>>>> month quota values to be lost on server restart. >>>>>>>> >>>>>>>> That's all the high level requirements I can think of off the top of my >>>>>>>> head, and after reading all of the current messages in this thread. :) >>>>>>>> >>>>>>>> -Eric >>>>>>>> _______________________________________________ >>>>>>>> infinispan-dev mailing list >>>>>>>> infinispan-dev at lists.jboss.org >>>>>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>>>>>>> >>>>>>> >>>>>> _______________________________________________ >>>>>> infinispan-dev mailing list >>>>>> infinispan-dev at lists.jboss.org >>>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>>>>> >>>>> >>>> _______________________________________________ >>>> infinispan-dev mailing list >>>> infinispan-dev at lists.jboss.org >>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>>> >>> _______________________________________________ >>> infinispan-dev mailing list >>> infinispan-dev at lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>> >> >> -- >> Bela Ban, JGroups lead (http://www.jgroups.org) >> >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev at lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev > _______________________________________________ > infinispan-dev mailing list > infinispan-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev > -- Bela Ban, JGroups lead (http://www.jgroups.org) From dan.berindei at gmail.com Tue Mar 22 13:23:04 2016 From: dan.berindei at gmail.com (Dan Berindei) Date: Tue, 22 Mar 2016 19:23:04 +0200 Subject: [infinispan-dev] Distributed Counter Discussion In-Reply-To: <56F17CE6.4020103@redhat.com> References: <56EAE3FC.6030401@redhat.com> <56EBA0E2.7070106@redhat.com> <56EC004F.7040101@redhat.com> <56EC031F.10809@redhat.com> <56EC0E5F.1020103@redhat.com> <56EFC927.804@infinispan.org> <56EFDE47.4000101@redhat.com> <56F17CE6.4020103@redhat.com> Message-ID: On Tue, Mar 22, 2016 at 7:12 PM, Bela Ban wrote: > > > On 22/03/16 18:04, Dan Berindei wrote: >> On Mon, Mar 21, 2016 at 1:43 PM, Bela Ban wrote: >>> >>> >>> On 21/03/16 11:12, Pedro Ruivo wrote: >>>> Hi all, >>>> >>>> @Eric, thanks for the requirements. >>>> >>>> @Bela, does JGroups counter supports that semantics (AP)? >>> >>> No. You'd have to catch the MergeView and do this manually. >> >> I should also mention that you don't get a "cluster split" event. With >> when a cluster ABC splits into A and BC and merges back, you could get >> quite a view sequence like this: >> >> A, B, C: A|3 [A, B, C] >> A: A|4 [A, B] >> A: A|5 [A] (could be missing) > > How's that possible? Are you assuming that A still gets heartbeats from B? > > Otherwise A will definitely get a singleton view A|5=[A], unless the > cluster heals in the meantime (before B is suspected and excluded). Exactly, the cluster can heal before B is suspected by A. > > Note that A|4 on A may or may not be received; if FD_ALL is used then > chances are view [A] will be received directly after [ABC] on A. Even with FD_ALL, the heartbeats from B and C will arrive on A at different times, so they may be suspected at different times. True, it's not the most likely scenario, but I wanted to emphasize the corner cases that the counter implementation would have to deal with. > >> B, C: B|4 [B, C] >> A, B, C: B|6 [B, C, A] (merge view) > > Note that you could keep the order if you installed a custom > view/merge-view creation policy, but by default members are sorted > according to the UUIDs. > > >> So it's not that easy to keep track of counter additions "since the split". > > Agreed. > >>>> Infinispan does not have eventually consistency (yet) neither an update log. So, it >>>> can't reconcile the counter and you will lose one of the partition updates. >>> >>> Same for the JGroups counter service. The jgroups-raft CounterService >>> provides strong consistency, but at the expense of availability. >>> >>>> On 03/18/2016 02:19 PM, Eric Wittmann wrote: >>>>> Agreed. :) >>>>> >>>>> On 3/18/2016 9:31 AM, Bela Ban wrote: >>>>>> So actually you don't care if you have multiple counters in case of a >>>>>> network split, but you do care that the numbers of different counters >>>>>> get reconciled when a network partition heals. >>>>>> >>>>>> Example >>>>>> - C1: 1000 >>>>>> - Network split: C1: 1000, C2: 1000 >>>>>> - Different clients update counters on both sides of the partition: C1: >>>>>> 1500 C2: 1600 >>>>>> - Network split disappears, reconciling C1 to 2100: 1000 +500 +600. This >>>>>> means the 500 added to C1 should have been added to C2 as well, and the >>>>>> 600 to C2 should have been added to C1 >>>>>> >>>>>> If such a behavior would be acceptable, then we could do without CP and >>>>>> live with AP >>>>>> >>>>>> On 18/03/16 14:19, Eric Wittmann wrote: >>>>>>> Yes, precisely. The API Gateway itself is clustered. It services a >>>>>>> large volume of inbound traffic which it reverse-proxies to appropriate >>>>>>> back-end APIs after applying policies such as security, rate limiting, >>>>>>> caching, etc. >>>>>>> >>>>>>> -Eric >>>>>>> >>>>>>> On 3/18/2016 2:32 AM, Bela Ban wrote: >>>>>>>> Stupid question: whay do you need a distributed counter for this? Is the >>>>>>>> service you're monitoring replicated in a cluster? >>>>>>>> >>>>>>>> On 17/03/16 18:06, Eric Wittmann wrote: >>>>>>>>> Greetings. Apologies for coming in a bit late on this conversation. >>>>>>>>> Tristan pointed me to it a couple of days ago and unfortunately I'm just >>>>>>>>> now getting time to reply. >>>>>>>>> >>>>>>>>> I can try to quickly give an overview of apiman's (JBoss API Management >>>>>>>>> Gateway) requirements. >>>>>>>>> >>>>>>>>> What we're trying to do is implement support for Limiting policies: >>>>>>>>> >>>>>>>>> * Rate Limiting/Throttling (e.g. limit of 100 requests per second) >>>>>>>>> * Quotas (e.g. limit of 100,000,000 requests per month) >>>>>>>>> * Transfer Quotas (e.g. limit of 2.5GB of data downloaded per day) >>>>>>>>> >>>>>>>>> We will need to support multiple backing implementations of the Rate >>>>>>>>> Limiter, and we're trying to get Infinispan to be one of those >>>>>>>>> implementations. >>>>>>>>> >>>>>>>>> In no particular order, we would need the following characteristics: >>>>>>>>> >>>>>>>>> - Can be "squishy" for quotas and transfer quotas: If you >>>>>>>>> get 100,001,017 requests that's OK >>>>>>>>> - Strict would be cool as an option: Hard-fail when the >>>>>>>>> counter reaches the limit - no chance it will go over. >>>>>>>>> - Lots of individual counters: users may publish 100s of >>>>>>>>> APIs to the Gateway, and each API may be consumed by >>>>>>>>> 100s or 1000s of users/client. Depending on configuration >>>>>>>>> of the policy, *each* user/client has a separate limit. >>>>>>>>> - Counters need to be created dynamically: users can >>>>>>>>> add APIs via the Management UI, configure them to add >>>>>>>>> policies (e.g. a Quota policy) and then publish them to >>>>>>>>> a running Gateway, at which point end users can invoke >>>>>>>>> the API through the Gateway, which will use a counter >>>>>>>>> to enforce the Quota. >>>>>>>>> - Counter values reset at the end of a time boundary: for >>>>>>>>> example, at the end of the month the counter value for >>>>>>>>> the example quota above would reset to 0. >>>>>>>>> - Don't care (right now) what the counter value is: at the >>>>>>>>> moment we simply need to know if some counter max value >>>>>>>>> has been reached. In the future we would like to know >>>>>>>>> when a max value is being "approached" (e.g. to notify a >>>>>>>>> user) >>>>>>>>> - Should be persistent: it would not be ideal for e.g. per- >>>>>>>>> month quota values to be lost on server restart. >>>>>>>>> >>>>>>>>> That's all the high level requirements I can think of off the top of my >>>>>>>>> head, and after reading all of the current messages in this thread. :) >>>>>>>>> >>>>>>>>> -Eric >>>>>>>>> _______________________________________________ >>>>>>>>> infinispan-dev mailing list >>>>>>>>> infinispan-dev at lists.jboss.org >>>>>>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>>>>>>>> >>>>>>>> >>>>>>> _______________________________________________ >>>>>>> infinispan-dev mailing list >>>>>>> infinispan-dev at lists.jboss.org >>>>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>>>>>> >>>>>> >>>>> _______________________________________________ >>>>> infinispan-dev mailing list >>>>> infinispan-dev at lists.jboss.org >>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>>>> >>>> _______________________________________________ >>>> infinispan-dev mailing list >>>> infinispan-dev at lists.jboss.org >>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>>> >>> >>> -- >>> Bela Ban, JGroups lead (http://www.jgroups.org) >>> >>> _______________________________________________ >>> infinispan-dev mailing list >>> infinispan-dev at lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev at lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev >> > > -- > Bela Ban, JGroups lead (http://www.jgroups.org) > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev From dan.berindei at gmail.com Wed Mar 23 06:20:05 2016 From: dan.berindei at gmail.com (Dan Berindei) Date: Wed, 23 Mar 2016 12:20:05 +0200 Subject: [infinispan-dev] PSA: CI test failure links are not permalinks Message-ID: Hi guys It's a good idea to link to CI test failures in JIRA, as the build logs can sometimes shed more information. However, those links will be dead after a few months, so please don't rely on them exclusively. Cheers Dan From galder at redhat.com Wed Mar 23 07:45:39 2016 From: galder at redhat.com (=?utf-8?Q?Galder_Zamarre=C3=B1o?=) Date: Wed, 23 Mar 2016 12:45:39 +0100 Subject: [infinispan-dev] Javascript client 0.2.0 arrives with Cluster Topology support Message-ID: Hi all, You can read all about the release here: http://blog.infinispan.org/2016/03/javascript-client-020-arrives-with.html As promised, the examples in the README file have been improved to make them clearer [1]. Cheers, [1] https://github.com/infinispan/js-client/blob/master/README.md -- Galder Zamarre?o Infinispan, Red Hat From galder at redhat.com Tue Mar 29 03:47:55 2016 From: galder at redhat.com (=?utf-8?Q?Galder_Zamarre=C3=B1o?=) Date: Tue, 29 Mar 2016 09:47:55 +0200 Subject: [infinispan-dev] Should Tree module be deprecated? Message-ID: Re: http://stackoverflow.com/questions/29673123/infinispan-treecache-not-getting-distributed-properly?noredirect=1#comment59906993_29673123 Cheers, -- Galder Zamarre?o Infinispan, Red Hat From galder at redhat.com Tue Mar 29 04:12:36 2016 From: galder at redhat.com (=?utf-8?Q?Galder_Zamarre=C3=B1o?=) Date: Tue, 29 Mar 2016 10:12:36 +0200 Subject: [infinispan-dev] Distributed Counter Discussion In-Reply-To: <56E70D92.1010206@infinispan.org> References: <56E70D92.1010206@infinispan.org> Message-ID: Good start but rather than coming out with a general design for a counter, I'd try to start coming out with functionality needed for the most commonly use cases for counters, e.g. e.g. if you're counting number of visits to your website, you only need an incrementing counter (and maybe a reset to be called at the end of the day/week/month/year?) but you'll never decrement, precision not hugely important? Could you assume you always increment by 1? It might be more efficient to increment by N... e.g. if you're counting number of users logged in at one point to your website, then you need a counter that both increments and decrements, precision not hugely important? You probably don't want reset operation? Moreover, could you assume you always increment/decrement by 1? e.g. if using a counter to generate a unique identifier that always increases, you need only increment but precision must be guaranteed, so after increasing and retrieving an old value can't be retrieved. You probably don't want reset operation? You can probably assume that increment is only by 1. ...etc And based on those, see what kind of API you need for each use case. There might be many more use cases but I'd rather start there than head directly to specifying operations. My main objective is to avoid that someone that wants to count number of visits to a website does not end up calling a method that would not make sense for its use case. Doing this might not be 100% avoidable but hope you see my point. Cheers, -- Galder Zamarre?o Infinispan, Red Hat > On 14 Mar 2016, at 20:14, Pedro Ruivo wrote: > > Hi everybody, > > Discussion about distributed counters. > > == Public API == > > interface Counter > > String getName() //counter name > long get() //current value. may return stale value due to concurrent > operations to other nodes. > void increment() //async or sync increment. default add(1) > void decrement() //async or sync decrement. default add(-1) > void add(long) //async or sync add. > void reset() //resets to initial value > > Note: Tried to make the interface as simple as possible with support for > sync and async operations. To avoid any confusion, I consider an async > operation as happening somewhat in the future, i.e. eventually > increments/decrements. > The sync operation happens somewhat during the method execution. > > interface AtomiCounter extends Counter > > long addAndGet() //adds a returns the new value. sync operation > long incrementAndGet() //increments and returns the new value. sync > operation. default addAndGet(1) > long decrementAndGet() //decrements and returns the new value. sync > operation. default addAndGet(-1) > > interface AdvancedCounter extends Counter > > long getMin/MaxThreshold() //returns the min and max threshold value > void add/removeListener() //adds a listener that is invoked when the > value change. Can be extended to notify when it is "reseted" and when > the threshold is reached. > > Note: should this interface be splitted? > > == Details == > > This is what I have in mind. Two counter managers: one based on JGroups > counter and another one based on Infinispan cache. > The first one creates AtomicCounters and it first perfectly. All > counters are created with an initial value (zero by default) > The second generates counters with all the options available. It can mix > sync/async operation and all counters will be in the same cache. The > cache will be configure by us and it would be an internal cache. This > will use all the features available in the cache. > > Configuration-wise, I'm thinking about 2 parameters: number of backups > and timeout (for sync operations). > > So, comment bellow and let me know alternatives, improvement or if I > missed something. > > ps. I also consider implement a counter based on JGroups-raft but I > believe it is an overkill. > ps2. sorry for the long email :( I tried to be shorter as possible. > > Cheers, > Pedro > _______________________________________________ > infinispan-dev mailing list > infinispan-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev From ttarrant at redhat.com Tue Mar 29 05:16:25 2016 From: ttarrant at redhat.com (Tristan Tarrant) Date: Tue, 29 Mar 2016 11:16:25 +0200 Subject: [infinispan-dev] Distributed Counter Discussion In-Reply-To: References: <56E70D92.1010206@infinispan.org> Message-ID: <56FA47E9.2030702@redhat.com> On 29/03/2016 10:12, Galder Zamarre?o wrote: > Good start but rather than coming out with a general design for a counter, I'd try to start coming out with functionality needed for the most commonly use cases for counters, e.g. > > e.g. if you're counting number of visits to your website, you only need an incrementing counter (and maybe a reset to be called at the end of the day/week/month/year?) but you'll never decrement, precision not hugely important? Could you assume you always increment by 1? It might be more efficient to increment by N... > e.g. if you're counting number of users logged in at one point to your website, then you need a counter that both increments and decrements, precision not hugely important? You probably don't want reset operation? Moreover, could you assume you always increment/decrement by 1? You want an "Adder" for both of these cases [1]. > e.g. if using a counter to generate a unique identifier that always increases, you need only increment but precision must be guaranteed, so after increasing and retrieving an old value can't be retrieved. You probably don't want reset operation? You can probably assume that increment is only by 1. This is a sequence, and, even in this case, there are two possible variants: - one where you just need uniqueness but monotonic increment is not essential (nodes can retrieve preallocated ranges of ids on demand). - one which is fully monotonic Tristan [1] https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/atomic/LongAdder.html -- Tristan Tarrant Infinispan Lead JBoss, a division of Red Hat From pedro at infinispan.org Tue Mar 29 13:25:40 2016 From: pedro at infinispan.org (Pedro Ruivo) Date: Tue, 29 Mar 2016 18:25:40 +0100 Subject: [infinispan-dev] Distributed Counter Discussion In-Reply-To: References: <56E70D92.1010206@infinispan.org> Message-ID: <56FABA94.5010104@infinispan.org> == Status update == Hi all, The work in progress can be found in [1]. The main interface is: CompletableFuture addAndGet(long) The CompletableFuture will allow to cover a more use cases, for example, if you need to count the number of visits, you can increment without waiting it to complete. On other hand, if you need "unique ids", it is possible to increment and wait for the result. * Let me know any concerns with it. Currently, two types of counters can be returned by the counter manager: "unlimited" and limited (i.e. with thresholds). "Unlimited" is actually limited to Long.MAX_VALUE but will never throw any exception when reached. The limited version throws a CounterException when the min or max threshold is reached. In both cases, it doesn't overflow the counter value. Currently, I'm working in the notification mechanism. So, just ignore that part of the code :) Moving to the next problem, I need an opinion how to create the counter manager. The current implementation is using a private cache. It has the advantage of starting the cache when a cache manager starts, however, it is a bit difficult to configure (e.g. persistence). Also, it does not allow different configuration for the counter (replicated vs distributed, persisted or not). Another alternative, it to construct the counter manager with the cache manager and a cache name. This looks simpler and it allows the user to configure it properly and have different counter manager with different configurations. * Comments are welcome. Cheers, Pedro [1] https://github.com/pruivo/infinispan/tree/t_counters/distributed-counter From sanne at infinispan.org Wed Mar 30 06:34:33 2016 From: sanne at infinispan.org (Sanne Grinovero) Date: Wed, 30 Mar 2016 11:34:33 +0100 Subject: [infinispan-dev] Distributed Counter Discussion In-Reply-To: <56FABA94.5010104@infinispan.org> References: <56E70D92.1010206@infinispan.org> <56FABA94.5010104@infinispan.org> Message-ID: Hi Pedro, there have been many questions on this thread pointing out the need to clarify the definition and which requirements it aims to solve. I don't think we can comment on the API proposal without making that clear first? It seems there was some consensus that this doesn't help for the APIMan use case; could we clarify the goal and use cases then? thanks, Sanne On 29 March 2016 at 18:25, Pedro Ruivo wrote: > == Status update == > > Hi all, > > The work in progress can be found in [1]. > > The main interface is: > > CompletableFuture addAndGet(long) > > The CompletableFuture will allow to cover a more use cases, for example, > if you need to count the number of visits, you can increment without > waiting it to complete. On other hand, if you need "unique ids", it is > possible to increment and wait for the result. > > * Let me know any concerns with it. > > Currently, two types of counters can be returned by the counter manager: > "unlimited" and limited (i.e. with thresholds). > > "Unlimited" is actually limited to Long.MAX_VALUE but will never throw > any exception when reached. > The limited version throws a CounterException when the min or max > threshold is reached. > In both cases, it doesn't overflow the counter value. > > Currently, I'm working in the notification mechanism. So, just ignore > that part of the code :) > > Moving to the next problem, I need an opinion how to create the counter > manager. > > The current implementation is using a private cache. It has the > advantage of starting the cache when a cache manager starts, however, it > is a bit difficult to configure (e.g. persistence). Also, it does not > allow different configuration for the counter (replicated vs > distributed, persisted or not). > > Another alternative, it to construct the counter manager with the cache > manager and a cache name. This looks simpler and it allows the user to > configure it properly and have different counter manager with different > configurations. > > * Comments are welcome. > > Cheers, > Pedro > > [1] https://github.com/pruivo/infinispan/tree/t_counters/distributed-counter > _______________________________________________ > infinispan-dev mailing list > infinispan-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev From mudokonman at gmail.com Wed Mar 30 12:04:53 2016 From: mudokonman at gmail.com (William Burns) Date: Wed, 30 Mar 2016 16:04:53 +0000 Subject: [infinispan-dev] Distributed Executor remote cancellation Message-ID: Recently we have been moving a lot of our methods that return Future [1] to CompletableFuture [2]. Unfortunately the latter, CompletableFuture, doesn't allow for cancellation of the future, since there is no thread tied to it. So I am proposing that our DistributedExecutorService [3] no longer allows for interruption of remote threads on a cancellation. This way we can have our distributed executor service return CompletableFuture instances which do not support interruption. Also to note that DistributedExecutorService extends ExecutorService which returns a normal Future which is documented as allowing cancellation. In this case I would just document on DistributedExecutorService that we don't support interruption anyways. Does anyone require the use of interruptable tasks with Distributed Executor? Thanks, - Will [1] https://docs.oracle.com/javase/7/docs/api/java/util/concurrent/Future.html [2] https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/CompletableFuture.html [3] https://docs.jboss.org/infinispan/8.1/apidocs/org/infinispan/distexec/DistributedExecutorService.html -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.jboss.org/pipermail/infinispan-dev/attachments/20160330/4031ff14/attachment.html From sanne at infinispan.org Wed Mar 30 12:19:44 2016 From: sanne at infinispan.org (Sanne Grinovero) Date: Wed, 30 Mar 2016 17:19:44 +0100 Subject: [infinispan-dev] Distributed Executor remote cancellation In-Reply-To: References: Message-ID: The term "Interruption" might have been too specific, but being able to cancel a task seems essential to me. On 30 March 2016 at 17:04, William Burns wrote: > Recently we have been moving a lot of our methods that return Future [1] to > CompletableFuture [2]. Unfortunately the latter, CompletableFuture, doesn't > allow for cancellation of the future, since there is no thread tied to it. > So I am proposing that our DistributedExecutorService [3] no longer allows > for interruption of remote threads on a cancellation. This way we can have > our distributed executor service return CompletableFuture instances which do > not support interruption. > > Also to note that DistributedExecutorService extends ExecutorService which > returns a normal Future which is documented as allowing cancellation. In > this case I would just document on DistributedExecutorService that we don't > support interruption anyways. > > Does anyone require the use of interruptable tasks with Distributed > Executor? > > Thanks, > > - Will > > [1] > https://docs.oracle.com/javase/7/docs/api/java/util/concurrent/Future.html > [2] > https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/CompletableFuture.html > [3] > https://docs.jboss.org/infinispan/8.1/apidocs/org/infinispan/distexec/DistributedExecutorService.html > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev From ttarrant at redhat.com Wed Mar 30 12:29:35 2016 From: ttarrant at redhat.com (Tristan Tarrant) Date: Wed, 30 Mar 2016 18:29:35 +0200 Subject: [infinispan-dev] Distributed Executor remote cancellation In-Reply-To: References:

Message-ID: <56FBFEEF.2030702@redhat.com> I agree with Sanne, we need cancellable tasks. Tristan On 30/03/2016 18:19, Sanne Grinovero wrote: > The term "Interruption" might have been too specific, but being able > to cancel a task seems essential to me. > > On 30 March 2016 at 17:04, William Burns wrote: >> Recently we have been moving a lot of our methods that return Future [1] to >> CompletableFuture [2]. Unfortunately the latter, CompletableFuture, doesn't >> allow for cancellation of the future, since there is no thread tied to it. >> So I am proposing that our DistributedExecutorService [3] no longer allows >> for interruption of remote threads on a cancellation. This way we can have >> our distributed executor service return CompletableFuture instances which do >> not support interruption. >> >> Also to note that DistributedExecutorService extends ExecutorService which >> returns a normal Future which is documented as allowing cancellation. In >> this case I would just document on DistributedExecutorService that we don't >> support interruption anyways. >> >> Does anyone require the use of interruptable tasks with Distributed >> Executor? >> >> Thanks, >> >> - Will >> >> [1] >> https://docs.oracle.com/javase/7/docs/api/java/util/concurrent/Future.html >> [2] >> https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/CompletableFuture.html >> [3] >> https://docs.jboss.org/infinispan/8.1/apidocs/org/infinispan/distexec/DistributedExecutorService.html >> >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev at lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev > _______________________________________________ > infinispan-dev mailing list > infinispan-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev > -- Tristan Tarrant Infinispan Lead JBoss, a division of Red Hat From mudokonman at gmail.com Wed Mar 30 12:40:08 2016 From: mudokonman at gmail.com (William Burns) Date: Wed, 30 Mar 2016 16:40:08 +0000 Subject: [infinispan-dev] Distributed Executor remote cancellation In-Reply-To: <56FBFEEF.2030702@redhat.com> References:

<56FBFEEF.2030702@redhat.com> Message-ID: You can still attempt to cancel a task. This email is specifically about interruption though. Let me explain the nuances in a bit more detail. With the suggestion we would still fully support if the task hasn't yet started that it would be cancellable. However when you cancel a task, there is an mayInterruptIfRunning boolean. I f mayInterruptIfRunning is true and the task is already running it will try to interrupt the thread processing it. This is what this email trail is about. We all know that java interruption can be flaky to begin with and then adding a remote aspect to it, it becomes even more unreliable. So do you guys think we need to support "interrupting" a task in progress? - Will On Wed, Mar 30, 2016 at 12:29 PM Tristan Tarrant wrote: > I agree with Sanne, we need cancellable tasks. > > Tristan > > On 30/03/2016 18:19, Sanne Grinovero wrote: > > The term "Interruption" might have been too specific, but being able > > to cancel a task seems essential to me. > > > > On 30 March 2016 at 17:04, William Burns wrote: > >> Recently we have been moving a lot of our methods that return Future > [1] to > >> CompletableFuture [2]. Unfortunately the latter, CompletableFuture, > doesn't > >> allow for cancellation of the future, since there is no thread tied to > it. > >> So I am proposing that our DistributedExecutorService [3] no longer > allows > >> for interruption of remote threads on a cancellation. This way we can > have > >> our distributed executor service return CompletableFuture instances > which do > >> not support interruption. > >> > >> Also to note that DistributedExecutorService extends ExecutorService > which > >> returns a normal Future which is documented as allowing cancellation. > In > >> this case I would just document on DistributedExecutorService that we > don't > >> support interruption anyways. > >> > >> Does anyone require the use of interruptable tasks with Distributed > >> Executor? > >> > >> Thanks, > >> > >> - Will > >> > >> [1] > >> > https://docs.oracle.com/javase/7/docs/api/java/util/concurrent/Future.html > >> [2] > >> > https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/CompletableFuture.html > >> [3] > >> > https://docs.jboss.org/infinispan/8.1/apidocs/org/infinispan/distexec/DistributedExecutorService.html > >> > >> _______________________________________________ > >> infinispan-dev mailing list > >> infinispan-dev at lists.jboss.org > >> https://lists.jboss.org/mailman/listinfo/infinispan-dev > > _______________________________________________ > > infinispan-dev mailing list > > infinispan-dev at lists.jboss.org > > https://lists.jboss.org/mailman/listinfo/infinispan-dev > > > > -- > Tristan Tarrant > Infinispan Lead > JBoss, a division of Red Hat > _______________________________________________ > infinispan-dev mailing list > infinispan-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.jboss.org/pipermail/infinispan-dev/attachments/20160330/6d27cadb/attachment-0001.html From sanne at infinispan.org Wed Mar 30 13:26:45 2016 From: sanne at infinispan.org (Sanne Grinovero) Date: Wed, 30 Mar 2016 18:26:45 +0100 Subject: [infinispan-dev] Distributed Executor remote cancellation In-Reply-To: References:

<56FBFEEF.2030702@redhat.com> Message-ID: On 30 March 2016 at 17:40, William Burns wrote: > You can still attempt to cancel a task. This email is specifically about > interruption though. Let me explain the nuances in a bit more detail. > > With the suggestion we would still fully support if the task hasn't yet > started that it would be cancellable. > > However when you cancel a task, there is an mayInterruptIfRunning boolean. > If mayInterruptIfRunning is true and the task is already running it will try > to interrupt the thread processing it. This is what this email trail is > about. We all know that java interruption can be flaky to begin with and > then adding a remote aspect to it, it becomes even more unreliable. > > So do you guys think we need to support "interrupting" a task in progress? Yes, absolutely! Resources are precious :) See also the reasoning of our old friend: - https://corner.squareup.com/2016/01/query-sniper.html Take the example of a non-correctly tuned Hibernate Search "rebuild all indexes" task. On a non-trivial data set it might need to run for weeks... you need a way to kill it. Thanks, Sanne > > - Will > > On Wed, Mar 30, 2016 at 12:29 PM Tristan Tarrant > wrote: >> >> I agree with Sanne, we need cancellable tasks. >> >> Tristan >> >> On 30/03/2016 18:19, Sanne Grinovero wrote: >> > The term "Interruption" might have been too specific, but being able >> > to cancel a task seems essential to me. >> > >> > On 30 March 2016 at 17:04, William Burns wrote: >> >> Recently we have been moving a lot of our methods that return Future >> >> [1] to >> >> CompletableFuture [2]. Unfortunately the latter, CompletableFuture, >> >> doesn't >> >> allow for cancellation of the future, since there is no thread tied to >> >> it. >> >> So I am proposing that our DistributedExecutorService [3] no longer >> >> allows >> >> for interruption of remote threads on a cancellation. This way we can >> >> have >> >> our distributed executor service return CompletableFuture instances >> >> which do >> >> not support interruption. >> >> >> >> Also to note that DistributedExecutorService extends ExecutorService >> >> which >> >> returns a normal Future which is documented as allowing cancellation. >> >> In >> >> this case I would just document on DistributedExecutorService that we >> >> don't >> >> support interruption anyways. >> >> >> >> Does anyone require the use of interruptable tasks with Distributed >> >> Executor? >> >> >> >> Thanks, >> >> >> >> - Will >> >> >> >> [1] >> >> >> >> https://docs.oracle.com/javase/7/docs/api/java/util/concurrent/Future.html >> >> [2] >> >> >> >> https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/CompletableFuture.html >> >> [3] >> >> >> >> https://docs.jboss.org/infinispan/8.1/apidocs/org/infinispan/distexec/DistributedExecutorService.html >> >> >> >> _______________________________________________ >> >> infinispan-dev mailing list >> >> infinispan-dev at lists.jboss.org >> >> https://lists.jboss.org/mailman/listinfo/infinispan-dev >> > _______________________________________________ >> > infinispan-dev mailing list >> > infinispan-dev at lists.jboss.org >> > https://lists.jboss.org/mailman/listinfo/infinispan-dev >> > >> >> -- >> Tristan Tarrant >> Infinispan Lead >> JBoss, a division of Red Hat >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev at lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev > > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev From gustavo at infinispan.org Wed Mar 30 13:37:18 2016 From: gustavo at infinispan.org (Gustavo Fernandes) Date: Wed, 30 Mar 2016 18:37:18 +0100 Subject: [infinispan-dev] Distributed Executor remote cancellation In-Reply-To: References:

<56FBFEEF.2030702@redhat.com>

Message-ID: On Wed, Mar 30, 2016 at 6:26 PM, Sanne Grinovero wrote: > On 30 March 2016 at 17:40, William Burns wrote: > > You can still attempt to cancel a task. This email is specifically about > > interruption though. Let me explain the nuances in a bit more detail. > > > > With the suggestion we would still fully support if the task hasn't yet > > started that it would be cancellable. > > > > However when you cancel a task, there is an mayInterruptIfRunning > boolean. > > If mayInterruptIfRunning is true and the task is already running it will > try > > to interrupt the thread processing it. This is what this email trail is > > about. We all know that java interruption can be flaky to begin with and > > then adding a remote aspect to it, it becomes even more unreliable. > > > > So do you guys think we need to support "interrupting" a task in > progress? > > Yes, absolutely! Resources are precious :) > > See also the reasoning of our old friend: > - https://corner.squareup.com/2016/01/query-sniper.html > > Take the example of a non-correctly tuned Hibernate Search "rebuild > all indexes" task. On a non-trivial data set it might need to run for > weeks... you need a way to kill it. > That's the case of our Mass Indexer: one of the motivations of making it return a Future was the possibility of cancelling tasks, capability offered by the Distributed Executor. Gustavo > > Thanks, > Sanne > > > > > - Will > > > > On Wed, Mar 30, 2016 at 12:29 PM Tristan Tarrant > > wrote: > >> > >> I agree with Sanne, we need cancellable tasks. > >> > >> Tristan > >> > >> On 30/03/2016 18:19, Sanne Grinovero wrote: > >> > The term "Interruption" might have been too specific, but being able > >> > to cancel a task seems essential to me. > >> > > >> > On 30 March 2016 at 17:04, William Burns > wrote: > >> >> Recently we have been moving a lot of our methods that return Future > >> >> [1] to > >> >> CompletableFuture [2]. Unfortunately the latter, CompletableFuture, > >> >> doesn't > >> >> allow for cancellation of the future, since there is no thread tied > to > >> >> it. > >> >> So I am proposing that our DistributedExecutorService [3] no longer > >> >> allows > >> >> for interruption of remote threads on a cancellation. This way we > can > >> >> have > >> >> our distributed executor service return CompletableFuture instances > >> >> which do > >> >> not support interruption. > >> >> > >> >> Also to note that DistributedExecutorService extends ExecutorService > >> >> which > >> >> returns a normal Future which is documented as allowing cancellation. > >> >> In > >> >> this case I would just document on DistributedExecutorService that we > >> >> don't > >> >> support interruption anyways. > >> >> > >> >> Does anyone require the use of interruptable tasks with Distributed > >> >> Executor? > >> >> > >> >> Thanks, > >> >> > >> >> - Will > >> >> > >> >> [1] > >> >> > >> >> > https://docs.oracle.com/javase/7/docs/api/java/util/concurrent/Future.html > >> >> [2] > >> >> > >> >> > https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/CompletableFuture.html > >> >> [3] > >> >> > >> >> > https://docs.jboss.org/infinispan/8.1/apidocs/org/infinispan/distexec/DistributedExecutorService.html > >> >> > >> >> _______________________________________________ > >> >> infinispan-dev mailing list > >> >> infinispan-dev at lists.jboss.org > >> >> https://lists.jboss.org/mailman/listinfo/infinispan-dev > >> > _______________________________________________ > >> > infinispan-dev mailing list > >> > infinispan-dev at lists.jboss.org > >> > https://lists.jboss.org/mailman/listinfo/infinispan-dev > >> > > >> > >> -- > >> Tristan Tarrant > >> Infinispan Lead > >> JBoss, a division of Red Hat > >> _______________________________________________ > >> infinispan-dev mailing list > >> infinispan-dev at lists.jboss.org > >> https://lists.jboss.org/mailman/listinfo/infinispan-dev > > > > > > _______________________________________________ > > infinispan-dev mailing list > > infinispan-dev at lists.jboss.org > > https://lists.jboss.org/mailman/listinfo/infinispan-dev > _______________________________________________ > infinispan-dev mailing list > infinispan-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.jboss.org/pipermail/infinispan-dev/attachments/20160330/9cd95e49/attachment.html From mudokonman at gmail.com Wed Mar 30 14:13:44 2016 From: mudokonman at gmail.com (William Burns) Date: Wed, 30 Mar 2016 18:13:44 +0000 Subject: [infinispan-dev] Distributed Executor remote cancellation In-Reply-To: References:

<56FBFEEF.2030702@redhat.com>

Message-ID: Thanks for the feedback. So in that case to try to appease both worlds: interrupting tasks and using CompletableFuture, I will just document the DistributedExecutorService to say that the mayInterruptIfRunning argument is only used on the returned future instance from methods such as submit and it is ignored on any chained CompletableFutures. - Will On Wed, Mar 30, 2016 at 1:37 PM Gustavo Fernandes wrote: > On Wed, Mar 30, 2016 at 6:26 PM, Sanne Grinovero > wrote: > >> On 30 March 2016 at 17:40, William Burns wrote: >> > You can still attempt to cancel a task. This email is specifically >> about >> > interruption though. Let me explain the nuances in a bit more detail. >> > >> > With the suggestion we would still fully support if the task hasn't yet >> > started that it would be cancellable. >> > >> > However when you cancel a task, there is an mayInterruptIfRunning >> boolean. >> > If mayInterruptIfRunning is true and the task is already running it >> will try >> > to interrupt the thread processing it. This is what this email trail is >> > about. We all know that java interruption can be flaky to begin with >> and >> > then adding a remote aspect to it, it becomes even more unreliable. >> > >> > So do you guys think we need to support "interrupting" a task in >> progress? >> >> Yes, absolutely! Resources are precious :) >> >> See also the reasoning of our old friend: >> - https://corner.squareup.com/2016/01/query-sniper.html >> >> Take the example of a non-correctly tuned Hibernate Search "rebuild >> all indexes" task. On a non-trivial data set it might need to run for >> weeks... you need a way to kill it. >> > > > That's the case of our Mass Indexer: one of the motivations of making it > return a Future was > the possibility of cancelling tasks, capability offered by the Distributed > Executor. > > Gustavo > > >> >> Thanks, >> Sanne >> >> > >> > - Will >> > >> > On Wed, Mar 30, 2016 at 12:29 PM Tristan Tarrant >> > wrote: >> >> >> >> I agree with Sanne, we need cancellable tasks. >> >> >> >> Tristan >> >> >> >> On 30/03/2016 18:19, Sanne Grinovero wrote: >> >> > The term "Interruption" might have been too specific, but being able >> >> > to cancel a task seems essential to me. >> >> > >> >> > On 30 March 2016 at 17:04, William Burns >> wrote: >> >> >> Recently we have been moving a lot of our methods that return Future >> >> >> [1] to >> >> >> CompletableFuture [2]. Unfortunately the latter, CompletableFuture, >> >> >> doesn't >> >> >> allow for cancellation of the future, since there is no thread tied >> to >> >> >> it. >> >> >> So I am proposing that our DistributedExecutorService [3] no longer >> >> >> allows >> >> >> for interruption of remote threads on a cancellation. This way we >> can >> >> >> have >> >> >> our distributed executor service return CompletableFuture instances >> >> >> which do >> >> >> not support interruption. >> >> >> >> >> >> Also to note that DistributedExecutorService extends ExecutorService >> >> >> which >> >> >> returns a normal Future which is documented as allowing >> cancellation. >> >> >> In >> >> >> this case I would just document on DistributedExecutorService that >> we >> >> >> don't >> >> >> support interruption anyways. >> >> >> >> >> >> Does anyone require the use of interruptable tasks with Distributed >> >> >> Executor? >> >> >> >> >> >> Thanks, >> >> >> >> >> >> - Will >> >> >> >> >> >> [1] >> >> >> >> >> >> >> https://docs.oracle.com/javase/7/docs/api/java/util/concurrent/Future.html >> >> >> [2] >> >> >> >> >> >> >> https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/CompletableFuture.html >> >> >> [3] >> >> >> >> >> >> >> https://docs.jboss.org/infinispan/8.1/apidocs/org/infinispan/distexec/DistributedExecutorService.html >> >> >> >> >> >> _______________________________________________ >> >> >> infinispan-dev mailing list >> >> >> infinispan-dev at lists.jboss.org >> >> >> https://lists.jboss.org/mailman/listinfo/infinispan-dev >> >> > _______________________________________________ >> >> > infinispan-dev mailing list >> >> > infinispan-dev at lists.jboss.org >> >> > https://lists.jboss.org/mailman/listinfo/infinispan-dev >> >> > >> >> >> >> -- >> >> Tristan Tarrant >> >> Infinispan Lead >> >> JBoss, a division of Red Hat >> >> _______________________________________________ >> >> infinispan-dev mailing list >> >> infinispan-dev at lists.jboss.org >> >> https://lists.jboss.org/mailman/listinfo/infinispan-dev >> > >> > >> > _______________________________________________ >> > infinispan-dev mailing list >> > infinispan-dev at lists.jboss.org >> > https://lists.jboss.org/mailman/listinfo/infinispan-dev >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev at lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev >> > _______________________________________________ > infinispan-dev mailing list > infinispan-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.jboss.org/pipermail/infinispan-dev/attachments/20160330/c1087186/attachment-0001.html From dereed at redhat.com Wed Mar 30 14:32:29 2016 From: dereed at redhat.com (Dennis Reed) Date: Wed, 30 Mar 2016 13:32:29 -0500 Subject: [infinispan-dev] Distributed Executor remote cancellation In-Reply-To: References:

<56FBFEEF.2030702@redhat.com>

Message-ID: <56FC1BBD.4020102@redhat.com> On 03/30/2016 12:26 PM, Sanne Grinovero wrote: > On 30 March 2016 at 17:40, William Burns wrote: >> You can still attempt to cancel a task. This email is specifically about >> interruption though. Let me explain the nuances in a bit more detail. >> >> With the suggestion we would still fully support if the task hasn't yet >> started that it would be cancellable. >> >> However when you cancel a task, there is an mayInterruptIfRunning boolean. >> If mayInterruptIfRunning is true and the task is already running it will try >> to interrupt the thread processing it. This is what this email trail is >> about. We all know that java interruption can be flaky to begin with and >> then adding a remote aspect to it, it becomes even more unreliable. >> >> So do you guys think we need to support "interrupting" a task in progress? > > Yes, absolutely! Resources are precious :) > > See also the reasoning of our old friend: > - https://corner.squareup.com/2016/01/query-sniper.html > > Take the example of a non-correctly tuned Hibernate Search "rebuild > all indexes" task. On a non-trivial data set it might need to run for > weeks... you need a way to kill it. Do you want a way to "kill" it, or a way to "interrupt" it? Your reply appeared to equate these, but the terms are actually quite different. Interruption is more of a "hey, if you don't mind could you please stop?", and requires the code to correctly handle it. The vast majority of code isn't written with interruption in mind. As William mentioned, interruption is not reliable (and there isn't any reliable and safe way to stop a single thread in Java). So the question is whether we want to support "ask it to stop, and it might stop or it might not" functionality. -Dennis > Thanks, > Sanne > >> >> - Will >> >> On Wed, Mar 30, 2016 at 12:29 PM Tristan Tarrant >> wrote: >>> >>> I agree with Sanne, we need cancellable tasks. >>> >>> Tristan >>> >>> On 30/03/2016 18:19, Sanne Grinovero wrote: >>>> The term "Interruption" might have been too specific, but being able >>>> to cancel a task seems essential to me. >>>> >>>> On 30 March 2016 at 17:04, William Burns wrote: >>>>> Recently we have been moving a lot of our methods that return Future >>>>> [1] to >>>>> CompletableFuture [2]. Unfortunately the latter, CompletableFuture, >>>>> doesn't >>>>> allow for cancellation of the future, since there is no thread tied to >>>>> it. >>>>> So I am proposing that our DistributedExecutorService [3] no longer >>>>> allows >>>>> for interruption of remote threads on a cancellation. This way we can >>>>> have >>>>> our distributed executor service return CompletableFuture instances >>>>> which do >>>>> not support interruption. >>>>> >>>>> Also to note that DistributedExecutorService extends ExecutorService >>>>> which >>>>> returns a normal Future which is documented as allowing cancellation. >>>>> In >>>>> this case I would just document on DistributedExecutorService that we >>>>> don't >>>>> support interruption anyways. >>>>> >>>>> Does anyone require the use of interruptable tasks with Distributed >>>>> Executor? >>>>> >>>>> Thanks, >>>>> >>>>> - Will >>>>> >>>>> [1] >>>>> >>>>> https://docs.oracle.com/javase/7/docs/api/java/util/concurrent/Future.html >>>>> [2] >>>>> >>>>> https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/CompletableFuture.html >>>>> [3] >>>>> >>>>> https://docs.jboss.org/infinispan/8.1/apidocs/org/infinispan/distexec/DistributedExecutorService.html >>>>> >>>>> _______________________________________________ >>>>> infinispan-dev mailing list >>>>> infinispan-dev at lists.jboss.org >>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>>> _______________________________________________ >>>> infinispan-dev mailing list >>>> infinispan-dev at lists.jboss.org >>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>>> >>> >>> -- >>> Tristan Tarrant >>> Infinispan Lead >>> JBoss, a division of Red Hat >>> _______________________________________________ >>> infinispan-dev mailing list >>> infinispan-dev at lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >> >> >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev at lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev > _______________________________________________ > infinispan-dev mailing list > infinispan-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev > From rvansa at redhat.com Thu Mar 31 03:24:46 2016 From: rvansa at redhat.com (Radim Vansa) Date: Thu, 31 Mar 2016 09:24:46 +0200 Subject: [infinispan-dev] Distributed Executor remote cancellation In-Reply-To: References:

<56FBFEEF.2030702@redhat.com>

Message-ID: <56FCD0BE.7020803@redhat.com> Can any of the chained futures be already running when the first is not completed? Because when it's not running, there's nothing to interrupt. Btw., if the interface returns CompletableFuture but you inherit from that to override the cancel method, the javadoc will still say that the mayInterruptIfRunning does not apply. In that case, you should either return InterruptableCompletableFuture with javadoc override, or maybe just AnotherFuture implements Future, CompletionStage (names at your discretion). R. On 03/30/2016 08:13 PM, William Burns wrote: > Thanks for the feedback. > > So in that case to try to appease both worlds: interrupting tasks and > using CompletableFuture, I will just document the > DistributedExecutorService to say that the mayInterruptIfRunning > argument is only used on the returned future instance from methods > such as submit and it is ignored on any chained CompletableFutures. > > - Will > > On Wed, Mar 30, 2016 at 1:37 PM Gustavo Fernandes > > wrote: > > On Wed, Mar 30, 2016 at 6:26 PM, Sanne Grinovero > > wrote: > > On 30 March 2016 at 17:40, William Burns > wrote: > > You can still attempt to cancel a task. This email is > specifically about > > interruption though. Let me explain the nuances in a bit > more detail. > > > > With the suggestion we would still fully support if the task > hasn't yet > > started that it would be cancellable. > > > > However when you cancel a task, there is an > mayInterruptIfRunning boolean. > > If mayInterruptIfRunning is true and the task is already > running it will try > > to interrupt the thread processing it. This is what this > email trail is > > about. We all know that java interruption can be flaky to > begin with and > > then adding a remote aspect to it, it becomes even more > unreliable. > > > > So do you guys think we need to support "interrupting" a > task in progress? > > Yes, absolutely! Resources are precious :) > > See also the reasoning of our old friend: > - https://corner.squareup.com/2016/01/query-sniper.html > > Take the example of a non-correctly tuned Hibernate Search > "rebuild > all indexes" task. On a non-trivial data set it might need to > run for > weeks... you need a way to kill it. > > > > That's the case of our Mass Indexer: one of the motivations of > making it return a Future was > the possibility of cancelling tasks, capability offered by the > Distributed Executor. > > Gustavo > > > Thanks, > Sanne > > > > > - Will > > > > On Wed, Mar 30, 2016 at 12:29 PM Tristan Tarrant > > > > wrote: > >> > >> I agree with Sanne, we need cancellable tasks. > >> > >> Tristan > >> > >> On 30/03/2016 18:19, Sanne Grinovero wrote: > >> > The term "Interruption" might have been too specific, but > being able > >> > to cancel a task seems essential to me. > >> > > >> > On 30 March 2016 at 17:04, William Burns > > wrote: > >> >> Recently we have been moving a lot of our methods that > return Future > >> >> [1] to > >> >> CompletableFuture [2]. Unfortunately the latter, > CompletableFuture, > >> >> doesn't > >> >> allow for cancellation of the future, since there is no > thread tied to > >> >> it. > >> >> So I am proposing that our DistributedExecutorService > [3] no longer > >> >> allows > >> >> for interruption of remote threads on a cancellation. > This way we can > >> >> have > >> >> our distributed executor service return > CompletableFuture instances > >> >> which do > >> >> not support interruption. > >> >> > >> >> Also to note that DistributedExecutorService extends > ExecutorService > >> >> which > >> >> returns a normal Future which is documented as allowing > cancellation. > >> >> In > >> >> this case I would just document on > DistributedExecutorService that we > >> >> don't > >> >> support interruption anyways. > >> >> > >> >> Does anyone require the use of interruptable tasks with > Distributed > >> >> Executor? > >> >> > >> >> Thanks, > >> >> > >> >> - Will > >> >> > >> >> [1] > >> >> > >> >> > https://docs.oracle.com/javase/7/docs/api/java/util/concurrent/Future.html > >> >> [2] > >> >> > >> >> > https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/CompletableFuture.html > >> >> [3] > >> >> > >> >> > https://docs.jboss.org/infinispan/8.1/apidocs/org/infinispan/distexec/DistributedExecutorService.html > >> >> > >> >> _______________________________________________ > >> >> infinispan-dev mailing list > >> >> infinispan-dev at lists.jboss.org > > >> >> https://lists.jboss.org/mailman/listinfo/infinispan-dev > >> > _______________________________________________ > >> > infinispan-dev mailing list > >> > infinispan-dev at lists.jboss.org > > >> > https://lists.jboss.org/mailman/listinfo/infinispan-dev > >> > > >> > >> -- > >> Tristan Tarrant > >> Infinispan Lead > >> JBoss, a division of Red Hat > >> _______________________________________________ > >> infinispan-dev mailing list > >> infinispan-dev at lists.jboss.org > > >> https://lists.jboss.org/mailman/listinfo/infinispan-dev > > > > > > _______________________________________________ > > infinispan-dev mailing list > > infinispan-dev at lists.jboss.org > > > https://lists.jboss.org/mailman/listinfo/infinispan-dev > _______________________________________________ > infinispan-dev mailing list > infinispan-dev at lists.jboss.org > > https://lists.jboss.org/mailman/listinfo/infinispan-dev > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev > > > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev -- Radim Vansa JBoss Performance Team From bban at redhat.com Thu Mar 31 04:22:30 2016 From: bban at redhat.com (Bela Ban) Date: Thu, 31 Mar 2016 10:22:30 +0200 Subject: [infinispan-dev] Distributed Counter Discussion In-Reply-To: <56FABA94.5010104@infinispan.org> References: <56E70D92.1010206@infinispan.org> <56FABA94.5010104@infinispan.org> Message-ID: <56FCDE46.5060605@redhat.com> Note that [1] will make the JGroupsCounter completely async. Currently, the wrapper is not async. [1] https://issues.jboss.org/browse/JGRP-2025 On 29/03/16 19:25, Pedro Ruivo wrote: > == Status update == > > Hi all, > > The work in progress can be found in [1]. > > The main interface is: > > CompletableFuture addAndGet(long) > > The CompletableFuture will allow to cover a more use cases, for example, > if you need to count the number of visits, you can increment without > waiting it to complete. On other hand, if you need "unique ids", it is > possible to increment and wait for the result. > > * Let me know any concerns with it. > > Currently, two types of counters can be returned by the counter manager: > "unlimited" and limited (i.e. with thresholds). > > "Unlimited" is actually limited to Long.MAX_VALUE but will never throw > any exception when reached. > The limited version throws a CounterException when the min or max > threshold is reached. > In both cases, it doesn't overflow the counter value. > > Currently, I'm working in the notification mechanism. So, just ignore > that part of the code :) > > Moving to the next problem, I need an opinion how to create the counter > manager. > > The current implementation is using a private cache. It has the > advantage of starting the cache when a cache manager starts, however, it > is a bit difficult to configure (e.g. persistence). Also, it does not > allow different configuration for the counter (replicated vs > distributed, persisted or not). > > Another alternative, it to construct the counter manager with the cache > manager and a cache name. This looks simpler and it allows the user to > configure it properly and have different counter manager with different > configurations. > > * Comments are welcome. > > Cheers, > Pedro > > [1] https://github.com/pruivo/infinispan/tree/t_counters/distributed-counter > _______________________________________________ > infinispan-dev mailing list > infinispan-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev > -- Bela Ban, JGroups lead (http://www.jgroups.org) From dan.berindei at gmail.com Thu Mar 31 08:31:37 2016 From: dan.berindei at gmail.com (Dan Berindei) Date: Thu, 31 Mar 2016 15:31:37 +0300 Subject: [infinispan-dev] Distributed Executor remote cancellation In-Reply-To: <56FCD0BE.7020803@redhat.com> References:

<56FBFEEF.2030702@redhat.com>

<56FCD0BE.7020803@redhat.com> Message-ID: On Thu, Mar 31, 2016 at 10:24 AM, Radim Vansa wrote: > Can any of the chained futures be already running when the first is not > completed? Because when it's not running, there's nothing to interrupt. It's not a matter of timing, but the CF instance returned by any CF method will not pass the cancellation request to the initial future. So this code won't cancel anything: CompletableFuture cf = CompletableFuture.allOf(des.submitEverywhere(() -> null).toArray(new CompletableFuture[0])); cf.cancel(true); > > Btw., if the interface returns CompletableFuture but you inherit from > that to override the cancel method, the javadoc will still say that the > mayInterruptIfRunning does not apply. In that case, you should either > return InterruptableCompletableFuture with javadoc override, or maybe > just AnotherFuture implements Future, CompletionStage (names at your > discretion). Yeah, we should also make it clear that remote tasks are interrupted -- we know that's how it used to work, but it wasn't documented. > > R. > > On 03/30/2016 08:13 PM, William Burns wrote: >> Thanks for the feedback. >> >> So in that case to try to appease both worlds: interrupting tasks and >> using CompletableFuture, I will just document the >> DistributedExecutorService to say that the mayInterruptIfRunning >> argument is only used on the returned future instance from methods >> such as submit and it is ignored on any chained CompletableFutures. >> >> - Will >> >> On Wed, Mar 30, 2016 at 1:37 PM Gustavo Fernandes >> > wrote: >> >> On Wed, Mar 30, 2016 at 6:26 PM, Sanne Grinovero >> > wrote: >> >> On 30 March 2016 at 17:40, William Burns >