From rvansa at redhat.com  Wed Mar  1 03:18:17 2017
From: rvansa at redhat.com (Radim Vansa)
Date: Wed, 1 Mar 2017 09:18:17 +0100
Subject: [infinispan-dev] Calling getCache with a template and defined
 configuration
In-Reply-To: <CA+YCuUNzEo7zxq8gEfAu5Y3G-_YVOYwasXjThOfQPyQOp6t6xw@mail.gmail.com>
References: <CA+YCuUOT0KJNHcg7WNhWkAa_E=NoaSHLXim3+AmR=aMk90aGbw@mail.gmail.com>
	<CA+nfvwTY=JFmspivhHmRyJ+6iWntx_zAEL+L=pqFeK4gqpUj2w@mail.gmail.com>
	<CA+YCuUN_N80u_ybdNGX1kaO6vD6LeK2o7gTXo2QxeXO46-pr1g@mail.gmail.com>
	<CA+YCuUNzEo7zxq8gEfAu5Y3G-_YVOYwasXjThOfQPyQOp6t6xw@mail.gmail.com>
Message-ID: <8ddeca18-7bf5-b240-4b7a-a155dd0d3c74@redhat.com>

I still think that if the cache is already defined, defineConfiguration 
should throw an exception. This javadoc was written 7 years ago [1], 
maybe with a different intention.
Strange and complex combinations don't help. We have made a clear 
separation between templates and cache configurations; you should not 
use regular cache configuration as a template for programmatically 
defined cache anymore, and if you really want to, there are means to 
that (load, undefine, define).

Btw., the javadoc is out of date, too, since it mentions default cache 
which has been removed recently.

R.

[1] 
https://github.com/infinispan/infinispan/commit/73d99d37ebfb8af6b64df6a7579a2448deacbde7#diff-e2b618b97769a45ec42eb5910a8c2119R62

On 02/28/2017 10:51 PM, William Burns wrote:
> So while I was trying to work on this, I have to admit I am even more 
> torn in regards to what to do.  Looking at [1] it looks like the 
> template should only be applied if the cache configuration is not 
> currently defined.  Unfortunately it doesn't work this way and always 
> applies this template to any existing configuration. So I am thinking 
> an alternative is to instead make it work as the documentation states, 
> only using the template if the cache is not currently defined. This 
> seems more logical to me at least.
>
> With that change the getCache(String, String) could stay as long as it 
> is documented that a template is only applied if no cache 
> configuration exists.
>
> What do you guys think?
>
> [1] 
> https://github.com/infinispan/infinispan/blob/master/core/src/main/java/org/infinispan/manager/EmbeddedCacheManager.java#L84 
>
>
> On Mon, Feb 27, 2017 at 10:09 AM William Burns <mudokonman at gmail.com 
> <mailto:mudokonman at gmail.com>> wrote:
>
>     On Mon, Feb 27, 2017 at 9:55 AM Dan Berindei
>     <dan.berindei at gmail.com <mailto:dan.berindei at gmail.com>> wrote:
>
>         I would go for option 2.
>
>
>     Do you think a WARN message will be enough? I am a bit weary about
>     this option myself.
>
>
>         We already started disconnecting the cache definition and
>         retrieval,
>         at least `getCache(name)` doesn't define a new cache based on the
>         default configuration any more. So I don't think it would be
>         too much,
>         even at this point, to deprecate all the overloads of
>         `getCache` that
>         can define a new cache and advise users to use
>         `defineConfiguration`
>         instead.
>
>
>     Hrmm I like the idea of deprecating the overloads :)
>
>
>
>
>         Cheers
>         Dan
>
>
>         On Mon, Feb 27, 2017 at 4:31 PM, William Burns
>         <mudokonman at gmail.com <mailto:mudokonman at gmail.com>> wrote:
>         > When working on another project using Infinispan the code
>         being used was a
>         > bit interesting and I don't think our template configuration
>         handling was
>         > expecting it do so in such a way.
>         >
>         > Essentially the code defined a template for a distributed
>         cache as well as
>         > some named caches.  Then whenever a cache is retrieved it
>         would pass the
>         > given name and always the distributed cache template. 
>         Unfortunately with
>         > the way templates work they essentially redefine a cache
>         first so the actual
>         > cache configuration was wiped out.  In this example I was
>         able to get the
>         > code to change to using a default cache instead, which is
>         the behavior that
>         > is needed.
>         >
>         > The issue though at hand is whether we should allow a user
>         to call getCache
>         > in such a way. My initial thought is to have it throw some
>         sort of
>         > configuration exception when this is invoked. But there are
>         some possible
>         > options.
>         >
>         > 1. Throw a configuration exception not allowing a user to
>         use a template
>         > with an already defined cache. This has a slight disconnect
>         between
>         > configuration and runtime, since if a user adds a new
>         definition it could
>         > cause runtime issues.
>         > 2. Log an error/warning message when this occurs. Is this
>         enough though?
>         > Still could have runtime issues that are possibly undetected.
>         > 3. Merge the configurations together applying the template
>         first.  This
>         > would be akin to how default cache works currently, but you
>         would get to
>         > define your default template configuration at runtime. This
>         sounded like the
>         > best option to me, but the problem is what if someone calls
>         getCache using
>         > the same cache name but a different template. This could get
>         hairy as well.
>         >
>         > Really thinking about the future, disconnecting the cache
>         definition and
>         > retrieval would be the best option, but we can't do that
>         this late in the
>         > game.
>         >
>         > What do you guys think?
>         >
>         >  - Will
>         >
>         > _______________________________________________
>         > infinispan-dev mailing list
>         > infinispan-dev at lists.jboss.org
>         <mailto:infinispan-dev at lists.jboss.org>
>         > https://lists.jboss.org/mailman/listinfo/infinispan-dev
>         _______________________________________________
>         infinispan-dev mailing list
>         infinispan-dev at lists.jboss.org
>         <mailto:infinispan-dev at lists.jboss.org>
>         https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
>
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev


-- 
Radim Vansa <rvansa at redhat.com>
JBoss Performance Team


From slaskawi at redhat.com  Wed Mar  1 03:45:09 2017
From: slaskawi at redhat.com (Sebastian Laskawiec)
Date: Wed, 01 Mar 2017 08:45:09 +0000
Subject: [infinispan-dev] Have a look at the new HealthCheck API blog post
Message-ID: <CAA9R9O6r=vUFOosF6JHF=CPf60skhScpOJ9CB3FMsi0swXeAeQ@mail.gmail.com>

http://blog.infinispan.org/2017/03/checking-infinispan-cluster-health-and.html
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/infinispan-dev/attachments/20170301/f6904ff8/attachment.html 

From vjuranek at redhat.com  Wed Mar  1 04:15:00 2017
From: vjuranek at redhat.com (Vojtech Juranek)
Date: Wed, 01 Mar 2017 10:15:00 +0100
Subject: [infinispan-dev] SASL noanonymous policy
Message-ID: <2909595.g9rjNcEEmC@localhost.localdomain>

Hi,
the behaviour of HR authentication handler has changed recently [1]. Now, if 
HR authentication is enabled, anonymous access over HR is disabled unless SASL 
property NOANONYMOUS is set to false. While I agree that disabling anonymous 
access by default is reasonable, on the other hand I find this approach 
confusing for 2 reasons:

* my understanding of ISPN security mode was that "what is not explicitly 
forbidden it's allowed" (this is itself IMHO quite confusing, but OK) - e.g. 
if the security is defined on cache container level, access to all caches is 
granted to anyone, unless security is defined also for given cache. Recent 
change takes different approach "what is not explicitly allowed is forbidden", 
IMHO making whole security configuration even more confusing for the users.

* SASL policy is actually intended for SASL mechanism negotiation (see e.g. 
"How SASL Mechanisms are Installed and Selected" in [2]) so that it doesn't 
have to be specified explicitly and shouldn't be used in server logic. Using 
it in sever logic can be again quite confusing for users and, even worse, can 
result into security flaws - e.g. admin which expects that EXTERNAL 
authentication will be used will set SASL policy to NOANONYMOUS and the side 
effect would be that any unsecured cache can be access by any (non 
authenticated!) user.

IMHO HR authentication handler should be rewritten so that is just passes SASL 
policy to SASL server and don't use it in sever logic and we also should sick 
with one approach "what is not allowed is forbidden" or "what is not forbidden 
is allowed" and we should use it consistently everywhere where security comes 
into the play.

Thoughts?

Thanks
Vojta

[1] https://github.com/infinispan/infinispan/commit/
edaf39bd09a52a37f28abe9fdc29ee3214d6c256
[2] https://docs.oracle.com/javase/8/docs/technotes/guides/security/sasl/sasl-refguide.html
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 473 bytes
Desc: This is a digitally signed message part.
Url : http://lists.jboss.org/pipermail/infinispan-dev/attachments/20170301/24a093b2/attachment-0001.bin 

From ttarrant at redhat.com  Wed Mar  1 04:56:08 2017
From: ttarrant at redhat.com (Tristan Tarrant)
Date: Wed, 1 Mar 2017 10:56:08 +0100
Subject: [infinispan-dev] SASL noanonymous policy
In-Reply-To: <2909595.g9rjNcEEmC@localhost.localdomain>
References: <2909595.g9rjNcEEmC@localhost.localdomain>
Message-ID: <c7ede54d-7e1e-7d0c-c701-804a4388cd48@redhat.com>



On 01/03/17 10:15, Vojtech Juranek wrote:
> Hi,
> the behaviour of HR authentication handler has changed recently [1]. Now, if
> HR authentication is enabled, anonymous access over HR is disabled unless SASL
> property NOANONYMOUS is set to false. While I agree that disabling anonymous
> access by default is reasonable, on the other hand I find this approach
> confusing for 2 reasons:
> 
> * my understanding of ISPN security mode was that "what is not explicitly
> forbidden it's allowed" (this is itself IMHO quite confusing, but OK) 

I think that we need to make a clear distinction about authentication 
and authorization.
While your statement is true for authorization, it should never have 
been so for authentication. Also, authentication should never depend on 
authorization being set up, as enabling authorization has quite an 
impact on performance (the opposite, i.e. authorization requiring 
authentication, is however true).
If a user enables authentication on the endpoint, the expected behaviour 
should be that non-authenticated clients will fail to connect.

Tristan

- e.g.
> if the security is defined on cache container level, access to all caches is
> granted to anyone, unless security is defined also for given cache. Recent
> change takes different approach "what is not explicitly allowed is forbidden",
> IMHO making whole security configuration even more confusing for the users.
> 
> * SASL policy is actually intended for SASL mechanism negotiation (see e.g.
> "How SASL Mechanisms are Installed and Selected" in [2]) so that it doesn't
> have to be specified explicitly and shouldn't be used in server logic. Using
> it in sever logic can be again quite confusing for users and, even worse, can
> result into security flaws - e.g. admin which expects that EXTERNAL
> authentication will be used will set SASL policy to NOANONYMOUS and the side
> effect would be that any unsecured cache can be access by any (non
> authenticated!) user.
> 
> IMHO HR authentication handler should be rewritten so that is just passes SASL
> policy to SASL server and don't use it in sever logic and we also should sick
> with one approach "what is not allowed is forbidden" or "what is not forbidden
> is allowed" and we should use it consistently everywhere where security comes
> into the play.
> 
> Thoughts?
> 
> Thanks
> Vojta
> 
> [1] https://github.com/infinispan/infinispan/commit/
> edaf39bd09a52a37f28abe9fdc29ee3214d6c256
> [2] https://docs.oracle.com/javase/8/docs/technotes/guides/security/sasl/sasl-refguide.html
> 
> 
> 
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> 

-- 
Tristan Tarrant
Infinispan Lead
JBoss, a division of Red Hat

From tsegismont at gmail.com  Wed Mar  1 06:12:42 2017
From: tsegismont at gmail.com (Thomas SEGISMONT)
Date: Wed, 1 Mar 2017 12:12:42 +0100
Subject: [infinispan-dev] Calling getCache with a template and defined
	configuration
In-Reply-To: <CA+YCuUNzEo7zxq8gEfAu5Y3G-_YVOYwasXjThOfQPyQOp6t6xw@mail.gmail.com>
References: <CA+YCuUOT0KJNHcg7WNhWkAa_E=NoaSHLXim3+AmR=aMk90aGbw@mail.gmail.com>
	<CA+nfvwTY=JFmspivhHmRyJ+6iWntx_zAEL+L=pqFeK4gqpUj2w@mail.gmail.com>
	<CA+YCuUN_N80u_ybdNGX1kaO6vD6LeK2o7gTXo2QxeXO46-pr1g@mail.gmail.com>
	<CA+YCuUNzEo7zxq8gEfAu5Y3G-_YVOYwasXjThOfQPyQOp6t6xw@mail.gmail.com>
Message-ID: <CACiEr_TcU5GAM9GxWjOP9RDXZAg_7qkYdJM0DPikZs2WkPSNeQ@mail.gmail.com>

Hi

2017-02-28 22:51 GMT+01:00 William Burns <mudokonman at gmail.com>:

> So while I was trying to work on this, I have to admit I am even more torn
> in regards to what to do.  Looking at [1]  it looks like the template
> should only be applied if the cache configuration is not currently
> defined.  Unfortunately it doesn't work this way and always applies this
> template to any existing configuration. So I am thinking an alternative is
> to instead make it work as the documentation states, only using the
> template if the cache is not currently defined.  This seems more logical to
> me at least.
>

I guess you started this conversation because of the issue you found in
vertx-infinispan :) Then yes, when I used this method the Javadoc made me
think that the template would be picked if no cache with the provided name
exists.


>
> With that change the getCache(String, String) could stay as long as it is
> documented that a template is only applied if no cache configuration exists.
>
> What do you guys think?
>


Sounds better to me since it's what the doc suggests.


>
> [1] https://github.com/infinispan/infinispan/blob/master/core/
> src/main/java/org/infinispan/manager/EmbeddedCacheManager.java#L84
>
> On Mon, Feb 27, 2017 at 10:09 AM William Burns <mudokonman at gmail.com>
> wrote:
>
>> On Mon, Feb 27, 2017 at 9:55 AM Dan Berindei <dan.berindei at gmail.com>
>> wrote:
>>
>> I would go for option 2.
>>
>>
>> Do you think a WARN message will be enough? I am a bit weary about this
>> option myself.
>>
>>
>>
>> We already started disconnecting the cache definition and retrieval,
>> at least `getCache(name)` doesn't define a new cache based on the
>> default configuration any more. So I don't think it would be too much,
>> even at this point, to deprecate all the overloads of `getCache` that
>> can define a new cache and advise users to use `defineConfiguration`
>> instead.
>>
>>
>> Hrmm I like the idea of deprecating the overloads :)
>>
>>
>>
>>
>>
>> Cheers
>> Dan
>>
>>
>> On Mon, Feb 27, 2017 at 4:31 PM, William Burns <mudokonman at gmail.com>
>> wrote:
>> > When working on another project using Infinispan the code being used
>> was a
>> > bit interesting and I don't think our template configuration handling
>> was
>> > expecting it do so in such a way.
>> >
>> > Essentially the code defined a template for a distributed cache as well
>> as
>> > some named caches.  Then whenever a cache is retrieved it would pass the
>> > given name and always the distributed cache template.  Unfortunately
>> with
>> > the way templates work they essentially redefine a cache first so the
>> actual
>> > cache configuration was wiped out.  In this example I was able to get
>> the
>> > code to change to using a default cache instead, which is the behavior
>> that
>> > is needed.
>> >
>> > The issue though at hand is whether we should allow a user to call
>> getCache
>> > in such a way. My initial thought is to have it throw some sort of
>> > configuration exception when this is invoked. But there are some
>> possible
>> > options.
>> >
>> > 1. Throw a configuration exception not allowing a user to use a template
>> > with an already defined cache. This has a slight disconnect between
>> > configuration and runtime, since if a user adds a new definition it
>> could
>> > cause runtime issues.
>> > 2. Log an error/warning message when this occurs. Is this enough though?
>> > Still could have runtime issues that are possibly undetected.
>> > 3. Merge the configurations together applying the template first.  This
>> > would be akin to how default cache works currently, but you would get to
>> > define your default template configuration at runtime. This sounded
>> like the
>> > best option to me, but the problem is what if someone calls getCache
>> using
>> > the same cache name but a different template. This could get hairy as
>> well.
>> >
>> > Really thinking about the future, disconnecting the cache definition and
>> > retrieval would be the best option, but we can't do that this late in
>> the
>> > game.
>> >
>> > What do you guys think?
>> >
>> >  - Will
>> >
>> > _______________________________________________
>> > infinispan-dev mailing list
>> > infinispan-dev at lists.jboss.org
>> > https://lists.jboss.org/mailman/listinfo/infinispan-dev
>> _______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>
>>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/infinispan-dev/attachments/20170301/b1c35cce/attachment.html 

From vjuranek at redhat.com  Wed Mar  1 06:27:46 2017
From: vjuranek at redhat.com (Vojtech Juranek)
Date: Wed, 01 Mar 2017 12:27:46 +0100
Subject: [infinispan-dev] SASL noanonymous policy
In-Reply-To: <c7ede54d-7e1e-7d0c-c701-804a4388cd48@redhat.com>
References: <2909595.g9rjNcEEmC@localhost.localdomain>
	<c7ede54d-7e1e-7d0c-c701-804a4388cd48@redhat.com>
Message-ID: <2944081.sZ9RhEPBAK@localhost.localdomain>

On st?eda 1. b?ezna 2017 10:56:08 CET Tristan Tarrant wrote:
> I think that we need to make a clear distinction about authentication
> and authorization.

ok, I mixed these two together when I was thinking about it, you are right. 
Thanks for reminder

vojta
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 473 bytes
Desc: This is a digitally signed message part.
Url : http://lists.jboss.org/pipermail/infinispan-dev/attachments/20170301/e2da1954/attachment.bin 

From dan.berindei at gmail.com  Wed Mar  1 07:01:22 2017
From: dan.berindei at gmail.com (Dan Berindei)
Date: Wed, 1 Mar 2017 14:01:22 +0200
Subject: [infinispan-dev] Calling getCache with a template and defined
	configuration
In-Reply-To: <8ddeca18-7bf5-b240-4b7a-a155dd0d3c74@redhat.com>
References: <CA+YCuUOT0KJNHcg7WNhWkAa_E=NoaSHLXim3+AmR=aMk90aGbw@mail.gmail.com>
	<CA+nfvwTY=JFmspivhHmRyJ+6iWntx_zAEL+L=pqFeK4gqpUj2w@mail.gmail.com>
	<CA+YCuUN_N80u_ybdNGX1kaO6vD6LeK2o7gTXo2QxeXO46-pr1g@mail.gmail.com>
	<CA+YCuUNzEo7zxq8gEfAu5Y3G-_YVOYwasXjThOfQPyQOp6t6xw@mail.gmail.com>
	<8ddeca18-7bf5-b240-4b7a-a155dd0d3c74@redhat.com>
Message-ID: <CA+nfvwQV-0WRMRhwekZtk0omKib6vT1tkHjHjfNPUXMhON8=6A@mail.gmail.com>

On Wed, Mar 1, 2017 at 10:18 AM, Radim Vansa <rvansa at redhat.com> wrote:
> I still think that if the cache is already defined, defineConfiguration
> should throw an exception. This javadoc was written 7 years ago [1],
> maybe with a different intention.
> Strange and complex combinations don't help. We have made a clear
> separation between templates and cache configurations; you should not
> use regular cache configuration as a template for programmatically
> defined cache anymore, and if you really want to, there are means to
> that (load, undefine, define).
>
> Btw., the javadoc is out of date, too, since it mentions default cache
> which has been removed recently.
>

That defineConfiguration javadoc is just weird, it says what the
Configuration returned by the method will be but it doesn't say what
the configuration associated with that cache name in the cache manager
will be...

I agree with throwing an exception in defineConfiguration(...) when
that cache is already defined. I would not throw an exception from
getCache(cache, configurationName) when the cache is already defined,
I'd just ignore the new configuration (as we already ignore it when
the cache is runninng) and maybe log a warning telling people to use
defineConfiguration(cacheName, configurationName, new
ConfigurationBuilder().build()) + getCache(cacheName).

Cheers
Dan


> R.
>
> [1]
> https://github.com/infinispan/infinispan/commit/73d99d37ebfb8af6b64df6a7579a2448deacbde7#diff-e2b618b97769a45ec42eb5910a8c2119R62
>
> On 02/28/2017 10:51 PM, William Burns wrote:
>> So while I was trying to work on this, I have to admit I am even more
>> torn in regards to what to do.  Looking at [1] it looks like the
>> template should only be applied if the cache configuration is not
>> currently defined.  Unfortunately it doesn't work this way and always
>> applies this template to any existing configuration. So I am thinking
>> an alternative is to instead make it work as the documentation states,
>> only using the template if the cache is not currently defined. This
>> seems more logical to me at least.
>>
>> With that change the getCache(String, String) could stay as long as it
>> is documented that a template is only applied if no cache
>> configuration exists.
>>
>> What do you guys think?
>>
>> [1]
>> https://github.com/infinispan/infinispan/blob/master/core/src/main/java/org/infinispan/manager/EmbeddedCacheManager.java#L84
>>
>>
>> On Mon, Feb 27, 2017 at 10:09 AM William Burns <mudokonman at gmail.com
>> <mailto:mudokonman at gmail.com>> wrote:
>>
>>     On Mon, Feb 27, 2017 at 9:55 AM Dan Berindei
>>     <dan.berindei at gmail.com <mailto:dan.berindei at gmail.com>> wrote:
>>
>>         I would go for option 2.
>>
>>
>>     Do you think a WARN message will be enough? I am a bit weary about
>>     this option myself.
>>
>>
>>         We already started disconnecting the cache definition and
>>         retrieval,
>>         at least `getCache(name)` doesn't define a new cache based on the
>>         default configuration any more. So I don't think it would be
>>         too much,
>>         even at this point, to deprecate all the overloads of
>>         `getCache` that
>>         can define a new cache and advise users to use
>>         `defineConfiguration`
>>         instead.
>>
>>
>>     Hrmm I like the idea of deprecating the overloads :)
>>
>>
>>
>>
>>         Cheers
>>         Dan
>>
>>
>>         On Mon, Feb 27, 2017 at 4:31 PM, William Burns
>>         <mudokonman at gmail.com <mailto:mudokonman at gmail.com>> wrote:
>>         > When working on another project using Infinispan the code
>>         being used was a
>>         > bit interesting and I don't think our template configuration
>>         handling was
>>         > expecting it do so in such a way.
>>         >
>>         > Essentially the code defined a template for a distributed
>>         cache as well as
>>         > some named caches.  Then whenever a cache is retrieved it
>>         would pass the
>>         > given name and always the distributed cache template.
>>         Unfortunately with
>>         > the way templates work they essentially redefine a cache
>>         first so the actual
>>         > cache configuration was wiped out.  In this example I was
>>         able to get the
>>         > code to change to using a default cache instead, which is
>>         the behavior that
>>         > is needed.
>>         >
>>         > The issue though at hand is whether we should allow a user
>>         to call getCache
>>         > in such a way. My initial thought is to have it throw some
>>         sort of
>>         > configuration exception when this is invoked. But there are
>>         some possible
>>         > options.
>>         >
>>         > 1. Throw a configuration exception not allowing a user to
>>         use a template
>>         > with an already defined cache. This has a slight disconnect
>>         between
>>         > configuration and runtime, since if a user adds a new
>>         definition it could
>>         > cause runtime issues.
>>         > 2. Log an error/warning message when this occurs. Is this
>>         enough though?
>>         > Still could have runtime issues that are possibly undetected.
>>         > 3. Merge the configurations together applying the template
>>         first.  This
>>         > would be akin to how default cache works currently, but you
>>         would get to
>>         > define your default template configuration at runtime. This
>>         sounded like the
>>         > best option to me, but the problem is what if someone calls
>>         getCache using
>>         > the same cache name but a different template. This could get
>>         hairy as well.
>>         >
>>         > Really thinking about the future, disconnecting the cache
>>         definition and
>>         > retrieval would be the best option, but we can't do that
>>         this late in the
>>         > game.
>>         >
>>         > What do you guys think?
>>         >
>>         >  - Will
>>         >
>>         > _______________________________________________
>>         > infinispan-dev mailing list
>>         > infinispan-dev at lists.jboss.org
>>         <mailto:infinispan-dev at lists.jboss.org>
>>         > https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>         _______________________________________________
>>         infinispan-dev mailing list
>>         infinispan-dev at lists.jboss.org
>>         <mailto:infinispan-dev at lists.jboss.org>
>>         https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>
>>
>>
>> _______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
>
> --
> Radim Vansa <rvansa at redhat.com>
> JBoss Performance Team
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev

From ttarrant at redhat.com  Wed Mar  1 09:26:34 2017
From: ttarrant at redhat.com (Tristan Tarrant)
Date: Wed, 1 Mar 2017 15:26:34 +0100
Subject: [infinispan-dev] Calling getCache with a template and defined
 configuration
In-Reply-To: <CA+nfvwQV-0WRMRhwekZtk0omKib6vT1tkHjHjfNPUXMhON8=6A@mail.gmail.com>
References: <CA+YCuUOT0KJNHcg7WNhWkAa_E=NoaSHLXim3+AmR=aMk90aGbw@mail.gmail.com>
	<CA+nfvwTY=JFmspivhHmRyJ+6iWntx_zAEL+L=pqFeK4gqpUj2w@mail.gmail.com>
	<CA+YCuUN_N80u_ybdNGX1kaO6vD6LeK2o7gTXo2QxeXO46-pr1g@mail.gmail.com>
	<CA+YCuUNzEo7zxq8gEfAu5Y3G-_YVOYwasXjThOfQPyQOp6t6xw@mail.gmail.com>
	<8ddeca18-7bf5-b240-4b7a-a155dd0d3c74@redhat.com>
	<CA+nfvwQV-0WRMRhwekZtk0omKib6vT1tkHjHjfNPUXMhON8=6A@mail.gmail.com>
Message-ID: <a5898a51-eab4-fca1-9f98-ba80132de21d@redhat.com>

On 01/03/17 13:01, Dan Berindei wrote:
> On Wed, Mar 1, 2017 at 10:18 AM, Radim Vansa <rvansa at redhat.com> wrote:
>> I still think that if the cache is already defined, defineConfiguration
>> should throw an exception. This javadoc was written 7 years ago [1],

> I agree with throwing an exception in defineConfiguration(...) when
> that cache is already defined.

+1

> getCache(cache, configurationName) when the cache is already defined,
> I'd just ignore the new configuration (as we already ignore it when
> the cache is runninng) and maybe log a warning telling people to use
> defineConfiguration(cacheName, configurationName, new
> ConfigurationBuilder().build()) + getCache(cacheName).

+1

Tristan

>> R.
>>
>> [1]
>> https://github.com/infinispan/infinispan/commit/73d99d37ebfb8af6b64df6a7579a2448deacbde7#diff-e2b618b97769a45ec42eb5910a8c2119R62
>>
>> On 02/28/2017 10:51 PM, William Burns wrote:
>>> So while I was trying to work on this, I have to admit I am even more
>>> torn in regards to what to do.  Looking at [1] it looks like the
>>> template should only be applied if the cache configuration is not
>>> currently defined.  Unfortunately it doesn't work this way and always
>>> applies this template to any existing configuration. So I am thinking
>>> an alternative is to instead make it work as the documentation states,
>>> only using the template if the cache is not currently defined. This
>>> seems more logical to me at least.
>>>
>>> With that change the getCache(String, String) could stay as long as it
>>> is documented that a template is only applied if no cache
>>> configuration exists.
>>>
>>> What do you guys think?
>>>
>>> [1]
>>> https://github.com/infinispan/infinispan/blob/master/core/src/main/java/org/infinispan/manager/EmbeddedCacheManager.java#L84
>>>
>>>
>>> On Mon, Feb 27, 2017 at 10:09 AM William Burns <mudokonman at gmail.com
>>> <mailto:mudokonman at gmail.com>> wrote:
>>>
>>>      On Mon, Feb 27, 2017 at 9:55 AM Dan Berindei
>>>      <dan.berindei at gmail.com <mailto:dan.berindei at gmail.com>> wrote:
>>>
>>>          I would go for option 2.
>>>
>>>
>>>      Do you think a WARN message will be enough? I am a bit weary about
>>>      this option myself.
>>>
>>>
>>>          We already started disconnecting the cache definition and
>>>          retrieval,
>>>          at least `getCache(name)` doesn't define a new cache based on the
>>>          default configuration any more. So I don't think it would be
>>>          too much,
>>>          even at this point, to deprecate all the overloads of
>>>          `getCache` that
>>>          can define a new cache and advise users to use
>>>          `defineConfiguration`
>>>          instead.
>>>
>>>
>>>      Hrmm I like the idea of deprecating the overloads :)
>>>
>>>
>>>
>>>
>>>          Cheers
>>>          Dan
>>>
>>>
>>>          On Mon, Feb 27, 2017 at 4:31 PM, William Burns
>>>          <mudokonman at gmail.com <mailto:mudokonman at gmail.com>> wrote:
>>>          > When working on another project using Infinispan the code
>>>          being used was a
>>>          > bit interesting and I don't think our template configuration
>>>          handling was
>>>          > expecting it do so in such a way.
>>>          >
>>>          > Essentially the code defined a template for a distributed
>>>          cache as well as
>>>          > some named caches.  Then whenever a cache is retrieved it
>>>          would pass the
>>>          > given name and always the distributed cache template.
>>>          Unfortunately with
>>>          > the way templates work they essentially redefine a cache
>>>          first so the actual
>>>          > cache configuration was wiped out.  In this example I was
>>>          able to get the
>>>          > code to change to using a default cache instead, which is
>>>          the behavior that
>>>          > is needed.
>>>          >
>>>          > The issue though at hand is whether we should allow a user
>>>          to call getCache
>>>          > in such a way. My initial thought is to have it throw some
>>>          sort of
>>>          > configuration exception when this is invoked. But there are
>>>          some possible
>>>          > options.
>>>          >
>>>          > 1. Throw a configuration exception not allowing a user to
>>>          use a template
>>>          > with an already defined cache. This has a slight disconnect
>>>          between
>>>          > configuration and runtime, since if a user adds a new
>>>          definition it could
>>>          > cause runtime issues.
>>>          > 2. Log an error/warning message when this occurs. Is this
>>>          enough though?
>>>          > Still could have runtime issues that are possibly undetected.
>>>          > 3. Merge the configurations together applying the template
>>>          first.  This
>>>          > would be akin to how default cache works currently, but you
>>>          would get to
>>>          > define your default template configuration at runtime. This
>>>          sounded like the
>>>          > best option to me, but the problem is what if someone calls
>>>          getCache using
>>>          > the same cache name but a different template. This could get
>>>          hairy as well.
>>>          >
>>>          > Really thinking about the future, disconnecting the cache
>>>          definition and
>>>          > retrieval would be the best option, but we can't do that
>>>          this late in the
>>>          > game.
>>>          >
>>>          > What do you guys think?
>>>          >
>>>          >  - Will
>>>          >
>>>          > _______________________________________________
>>>          > infinispan-dev mailing list
>>>          > infinispan-dev at lists.jboss.org
>>>          <mailto:infinispan-dev at lists.jboss.org>
>>>          > https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>>          _______________________________________________
>>>          infinispan-dev mailing list
>>>          infinispan-dev at lists.jboss.org
>>>          <mailto:infinispan-dev at lists.jboss.org>
>>>          https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>>
>>>
>>>
>>> _______________________________________________
>>> infinispan-dev mailing list
>>> infinispan-dev at lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>
>>
>> --
>> Radim Vansa <rvansa at redhat.com>
>> JBoss Performance Team
>>
>> _______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> 

-- 
Tristan Tarrant
Infinispan Lead
JBoss, a division of Red Hat

From mudokonman at gmail.com  Wed Mar  1 20:31:02 2017
From: mudokonman at gmail.com (William Burns)
Date: Thu, 02 Mar 2017 01:31:02 +0000
Subject: [infinispan-dev] Calling getCache with a template and defined
	configuration
In-Reply-To: <a5898a51-eab4-fca1-9f98-ba80132de21d@redhat.com>
References: <CA+YCuUOT0KJNHcg7WNhWkAa_E=NoaSHLXim3+AmR=aMk90aGbw@mail.gmail.com>
	<CA+nfvwTY=JFmspivhHmRyJ+6iWntx_zAEL+L=pqFeK4gqpUj2w@mail.gmail.com>
	<CA+YCuUN_N80u_ybdNGX1kaO6vD6LeK2o7gTXo2QxeXO46-pr1g@mail.gmail.com>
	<CA+YCuUNzEo7zxq8gEfAu5Y3G-_YVOYwasXjThOfQPyQOp6t6xw@mail.gmail.com>
	<8ddeca18-7bf5-b240-4b7a-a155dd0d3c74@redhat.com>
	<CA+nfvwQV-0WRMRhwekZtk0omKib6vT1tkHjHjfNPUXMhON8=6A@mail.gmail.com>
	<a5898a51-eab4-fca1-9f98-ba80132de21d@redhat.com>
Message-ID: <CA+YCuUP2Uar8Crva4d5XOz=iDi_Ov7XsJL4e84y4-uy31ns0RA@mail.gmail.com>

Alright seems fine to me.

On Wed, Mar 1, 2017, 9:27 AM Tristan Tarrant <ttarrant at redhat.com> wrote:

> On 01/03/17 13:01, Dan Berindei wrote:
> > On Wed, Mar 1, 2017 at 10:18 AM, Radim Vansa <rvansa at redhat.com> wrote:
> >> I still think that if the cache is already defined, defineConfiguration
> >> should throw an exception. This javadoc was written 7 years ago [1],
>
> > I agree with throwing an exception in defineConfiguration(...) when
> > that cache is already defined.
>
> +1
>

I get a lot of tests to change and remove now ?


> > getCache(cache, configurationName) when the cache is already defined,
> > I'd just ignore the new configuration (as we already ignore it when
> > the cache is runninng) and maybe log a warning telling people to use
> > defineConfiguration(cacheName, configurationName, new
> > ConfigurationBuilder().build()) + getCache(cacheName).
>
> +1
>
> Tristan
>
> >> R.
> >>
> >> [1]
> >>
> https://github.com/infinispan/infinispan/commit/73d99d37ebfb8af6b64df6a7579a2448deacbde7#diff-e2b618b97769a45ec42eb5910a8c2119R62
> >>
> >> On 02/28/2017 10:51 PM, William Burns wrote:
> >>> So while I was trying to work on this, I have to admit I am even more
> >>> torn in regards to what to do.  Looking at [1] it looks like the
> >>> template should only be applied if the cache configuration is not
> >>> currently defined.  Unfortunately it doesn't work this way and always
> >>> applies this template to any existing configuration. So I am thinking
> >>> an alternative is to instead make it work as the documentation states,
> >>> only using the template if the cache is not currently defined. This
> >>> seems more logical to me at least.
> >>>
> >>> With that change the getCache(String, String) could stay as long as it
> >>> is documented that a template is only applied if no cache
> >>> configuration exists.
> >>>
> >>> What do you guys think?
> >>>
> >>> [1]
> >>>
> https://github.com/infinispan/infinispan/blob/master/core/src/main/java/org/infinispan/manager/EmbeddedCacheManager.java#L84
> >>>
> >>>
> >>> On Mon, Feb 27, 2017 at 10:09 AM William Burns <mudokonman at gmail.com
> >>> <mailto:mudokonman at gmail.com>> wrote:
> >>>
> >>>      On Mon, Feb 27, 2017 at 9:55 AM Dan Berindei
> >>>      <dan.berindei at gmail.com <mailto:dan.berindei at gmail.com>> wrote:
> >>>
> >>>          I would go for option 2.
> >>>
> >>>
> >>>      Do you think a WARN message will be enough? I am a bit weary about
> >>>      this option myself.
> >>>
> >>>
> >>>          We already started disconnecting the cache definition and
> >>>          retrieval,
> >>>          at least `getCache(name)` doesn't define a new cache based on
> the
> >>>          default configuration any more. So I don't think it would be
> >>>          too much,
> >>>          even at this point, to deprecate all the overloads of
> >>>          `getCache` that
> >>>          can define a new cache and advise users to use
> >>>          `defineConfiguration`
> >>>          instead.
> >>>
> >>>
> >>>      Hrmm I like the idea of deprecating the overloads :)
> >>>
> >>>
> >>>
> >>>
> >>>          Cheers
> >>>          Dan
> >>>
> >>>
> >>>          On Mon, Feb 27, 2017 at 4:31 PM, William Burns
> >>>          <mudokonman at gmail.com <mailto:mudokonman at gmail.com>> wrote:
> >>>          > When working on another project using Infinispan the code
> >>>          being used was a
> >>>          > bit interesting and I don't think our template configuration
> >>>          handling was
> >>>          > expecting it do so in such a way.
> >>>          >
> >>>          > Essentially the code defined a template for a distributed
> >>>          cache as well as
> >>>          > some named caches.  Then whenever a cache is retrieved it
> >>>          would pass the
> >>>          > given name and always the distributed cache template.
> >>>          Unfortunately with
> >>>          > the way templates work they essentially redefine a cache
> >>>          first so the actual
> >>>          > cache configuration was wiped out.  In this example I was
> >>>          able to get the
> >>>          > code to change to using a default cache instead, which is
> >>>          the behavior that
> >>>          > is needed.
> >>>          >
> >>>          > The issue though at hand is whether we should allow a user
> >>>          to call getCache
> >>>          > in such a way. My initial thought is to have it throw some
> >>>          sort of
> >>>          > configuration exception when this is invoked. But there are
> >>>          some possible
> >>>          > options.
> >>>          >
> >>>          > 1. Throw a configuration exception not allowing a user to
> >>>          use a template
> >>>          > with an already defined cache. This has a slight disconnect
> >>>          between
> >>>          > configuration and runtime, since if a user adds a new
> >>>          definition it could
> >>>          > cause runtime issues.
> >>>          > 2. Log an error/warning message when this occurs. Is this
> >>>          enough though?
> >>>          > Still could have runtime issues that are possibly
> undetected.
> >>>          > 3. Merge the configurations together applying the template
> >>>          first.  This
> >>>          > would be akin to how default cache works currently, but you
> >>>          would get to
> >>>          > define your default template configuration at runtime. This
> >>>          sounded like the
> >>>          > best option to me, but the problem is what if someone calls
> >>>          getCache using
> >>>          > the same cache name but a different template. This could get
> >>>          hairy as well.
> >>>          >
> >>>          > Really thinking about the future, disconnecting the cache
> >>>          definition and
> >>>          > retrieval would be the best option, but we can't do that
> >>>          this late in the
> >>>          > game.
> >>>          >
> >>>          > What do you guys think?
> >>>          >
> >>>          >  - Will
> >>>          >
> >>>          > _______________________________________________
> >>>          > infinispan-dev mailing list
> >>>          > infinispan-dev at lists.jboss.org
> >>>          <mailto:infinispan-dev at lists.jboss.org>
> >>>          > https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >>>          _______________________________________________
> >>>          infinispan-dev mailing list
> >>>          infinispan-dev at lists.jboss.org
> >>>          <mailto:infinispan-dev at lists.jboss.org>
> >>>          https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >>>
> >>>
> >>>
> >>> _______________________________________________
> >>> infinispan-dev mailing list
> >>> infinispan-dev at lists.jboss.org
> >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >>
> >>
> >> --
> >> Radim Vansa <rvansa at redhat.com>
> >> JBoss Performance Team
> >>
> >> _______________________________________________
> >> infinispan-dev mailing list
> >> infinispan-dev at lists.jboss.org
> >> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> > _______________________________________________
> > infinispan-dev mailing list
> > infinispan-dev at lists.jboss.org
> > https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >
>
> --
> Tristan Tarrant
> Infinispan Lead
> JBoss, a division of Red Hat
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/infinispan-dev/attachments/20170302/1d332380/attachment-0001.html 

From rory.odonnell at oracle.com  Fri Mar  3 07:15:12 2017
From: rory.odonnell at oracle.com (Rory O'Donnell)
Date: Fri, 3 Mar 2017 12:15:12 +0000
Subject: [infinispan-dev] JDK 9 EA Build 159 and JDK 8u152 is available on
	java.net
Message-ID: <a50e39c0-876a-dbd8-eb71-15be6f31852e@oracle.com>


Hi Galder,

*JDK 8u152 **Early Access b01 <https://jdk8.java.net/download.html> *is 
available on java.net

*JDK 9 Early Access* b159 <https://jdk9.java.net/download/>  is 
available on java.net, summary of  changes are listed here 
<http://download.java.net/java/jdk9/changes/jdk-9+159.html>.

There have been a number of fixes to bugs reported by Open Source 
projects since the last availability email  :

  * b159 - JDK-8175261 : Per-protocol cache setting not working for JAR
    URLConnection
  * b158 - JDK-8173028 : Incorrect processing of supplementary-plane
    characters in text fields
  * b158 - JDK-8172967 : [macosx] Exception while working with layout
    for text containing unmappable character
  * b158 - JDK-8173804 : javadoc throws UnsupportedOperationException:
    should not happen
  * b157 - JDK-8174073 : NPE caused by @link reference to class
  * b156 - JDK-8172726 : ForkJoin common pool retains a reference to the
    thread context class loader

The following changeset is included in jdk-9+158:
http://hg.openjdk.java.net/jdk9/dev/jdk/rev/8b0d55e02f54

If you have a user-defined Policy implementation that grants 
FilePermission on ${user.dir}/-, reading a file in the current directory 
using its base name will fail.  Still the same solution: Ensure that the 
path used in permission granting has the same style as the one how you 
access the file.

Setting -Djdk.security.filePermCompat=true will take you back to the 
jdk-9+140 behavior.
Setting -Djdk.io.permissionsUseCanonicalPath=true will take you back to 
the jdk8 behavior.
Feedback is welcome on jdk9-dev at openjdk.java.net

Other areas of interest

  * JDK 9 Developer Guide [1]
  * JDK 9 Migration Guide [2]
  * JDK Cryptographic Roadmap [3]

Finaly, Dalibor and I gave a presentation at FOSDEM the video is 
available here [*4*]

Rgds,Rory

[1] http://docs.oracle.com/javase/9/javase-docs.htm
[2] 
https://docs.oracle.com/javase/9/migrate/toc.htm#JSMIG-GUID-7744EF96-5899-4FB2-B34E-86D49B2E89B6
[3] https://www.java.com/en/jre-jdk-cryptoroadmap.html
[4] https://fosdem.org/2017/schedule/event/outreach/

-- 
Rgds,Rory O'Donnell
Quality Engineering Manager
Oracle EMEA , Dublin, Ireland

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/infinispan-dev/attachments/20170303/3a976bb6/attachment.html 

From vblagoje at redhat.com  Mon Mar  6 10:17:45 2017
From: vblagoje at redhat.com (Vladimir Blagojevic)
Date: Mon, 6 Mar 2017 10:17:45 -0500
Subject: [infinispan-dev] My weekly report
Message-ID: <6646c030-6777-2322-a675-7aec7df1c5fd@redhat.com>

Guys,

I am on PTO (March 6th-8th). Here is my weekly update:

ISPN-7228 - Fixed a minor issue where an invalid schema got added anyway

ISPN-7544 - Server cards do not indicate correct status in some cases

All server states are properly indicated on the card now. We also 
adjusted an action menu for servers to accommodate all possible states 
(we handle reload-required and restart-required now)

ISPN-7477 Parsing of all endpoints including multi tenant router 
endpoints. Currently on PR queue at 
https://github.com/infinispan/infinispan-management-console/pull/192

I also completed console sketches for initial proposal and basic use 
cases for endpoints. PR is at 
https://github.com/infinispan/infinispan-console-mockup/pull/15

Infinispan 9.1 document plan has been updated to reflect basic endpoints 
use cases. I'll add more use cases this week. Document is at 
https://docs.google.com/document/d/1ot7m0e1lslc7mA1f2ArN4f4YeT8vbFUafXailWNIAik/edit

Vladimir






From sanne at infinispan.org  Wed Mar  8 11:34:47 2017
From: sanne at infinispan.org (Sanne Grinovero)
Date: Wed, 8 Mar 2017 16:34:47 +0000
Subject: [infinispan-dev] Auto-detection of appropriate key2StringMapperClass
Message-ID: <CAFm4XO3Mc2rsthTFZuVSRkTGky-iTtSiq-ePqfkYEFdXwNcCqA@mail.gmail.com>

It seems I keep forgetting that the "key2StringMapperClass" needs to
be explicitly set when using the String based JDBC CacheStore, which I
think is a quite popular store.

http://infinispan.org/docs/dev/user_guide/user_guide.html#storing_the_index_in_a_database

I think we could remove the need to configure this with an appropriate
service discovery?

We could keep the property so to be able to override the discovery process.

Would anyone volunteer to implement this?

Thanks,
Sanne

From gustavo at infinispan.org  Wed Mar  8 13:52:30 2017
From: gustavo at infinispan.org (Gustavo Fernandes)
Date: Wed, 8 Mar 2017 18:52:30 +0000
Subject: [infinispan-dev] Infinispan 9.0.0.CR3 released!
Message-ID: <CAH8Ud1HXR-y89yJTOV97_J0A=tazq=C8t4D_oC2N58L1d=XWtQ@mail.gmail.com>

Find all about it in the blog post [1]

[1] http://blog.infinispan.org/2017/03/infinispan-900cr3-is-out.html

Cheers,
Gustavo
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/infinispan-dev/attachments/20170308/5bf8f21d/attachment.html 

From dan.berindei at gmail.com  Thu Mar  9 04:38:16 2017
From: dan.berindei at gmail.com (Dan Berindei)
Date: Thu, 9 Mar 2017 11:38:16 +0200
Subject: [infinispan-dev] Auto-detection of appropriate
	key2StringMapperClass
In-Reply-To: <CAFm4XO3Mc2rsthTFZuVSRkTGky-iTtSiq-ePqfkYEFdXwNcCqA@mail.gmail.com>
References: <CAFm4XO3Mc2rsthTFZuVSRkTGky-iTtSiq-ePqfkYEFdXwNcCqA@mail.gmail.com>
Message-ID: <CA+nfvwTorPz6ptLwkcYQJ42mhA7416YxxacD8F2za9xt9F_7wA@mail.gmail.com>

Every store can have its own Key2StringMapper, so I really don't think
service discovery is appropriate here.

My favourite solution would be to have a default Key2StringMapper
implementation that uses Java serialization (not our internal
marshalling, because that can change between versions). It could even
do base64 encoding for databases that don't have a VARBINARY/BYTEA
equivalent.

Cheers
Dan


On Wed, Mar 8, 2017 at 6:34 PM, Sanne Grinovero <sanne at infinispan.org> wrote:
> It seems I keep forgetting that the "key2StringMapperClass" needs to
> be explicitly set when using the String based JDBC CacheStore, which I
> think is a quite popular store.
>
> http://infinispan.org/docs/dev/user_guide/user_guide.html#storing_the_index_in_a_database
>
> I think we could remove the need to configure this with an appropriate
> service discovery?
>
> We could keep the property so to be able to override the discovery process.
>
> Would anyone volunteer to implement this?
>
> Thanks,
> Sanne
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev

From sanne at infinispan.org  Thu Mar  9 04:48:51 2017
From: sanne at infinispan.org (Sanne Grinovero)
Date: Thu, 9 Mar 2017 09:48:51 +0000
Subject: [infinispan-dev] Auto-detection of appropriate
	key2StringMapperClass
In-Reply-To: <CA+nfvwTorPz6ptLwkcYQJ42mhA7416YxxacD8F2za9xt9F_7wA@mail.gmail.com>
References: <CAFm4XO3Mc2rsthTFZuVSRkTGky-iTtSiq-ePqfkYEFdXwNcCqA@mail.gmail.com>
	<CA+nfvwTorPz6ptLwkcYQJ42mhA7416YxxacD8F2za9xt9F_7wA@mail.gmail.com>
Message-ID: <CAFm4XO17D+fVOb0_BJZ6cU4NeUjBDUuprFwJb+b7WQe4EtLcFA@mail.gmail.com>

Hi Dan!

On 9 March 2017 at 09:38, Dan Berindei <dan.berindei at gmail.com> wrote:
> Every store can have its own Key2StringMapper, so I really don't think
> service discovery is appropriate here.

I suggested a default, while maintaining the property as an override option.

It won't make any difference to people in the situation you describe,
but it will avoid trouble for many more users.

In my experience when one develops a custom type & a custom
Key2StringMapper, you really wouldn't have interest in creating
multiple implementations for the same type. The only case in which I
can see some use for allowing multiple conflicting registrations to
the same type is for upgrade / migration cases..

>
> My favourite solution would be to have a default Key2StringMapper
> implementation that uses Java serialization (not our internal
> marshalling, because that can change between versions). It could even
> do base64 encoding for databases that don't have a VARBINARY/BYTEA
> equivalent.
>
> Cheers
> Dan
>
>
> On Wed, Mar 8, 2017 at 6:34 PM, Sanne Grinovero <sanne at infinispan.org> wrote:
>> It seems I keep forgetting that the "key2StringMapperClass" needs to
>> be explicitly set when using the String based JDBC CacheStore, which I
>> think is a quite popular store.
>>
>> http://infinispan.org/docs/dev/user_guide/user_guide.html#storing_the_index_in_a_database
>>
>> I think we could remove the need to configure this with an appropriate
>> service discovery?
>>
>> We could keep the property so to be able to override the discovery process.
>>
>> Would anyone volunteer to implement this?
>>
>> Thanks,
>> Sanne
>> _______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev

From dan.berindei at gmail.com  Thu Mar  9 05:22:53 2017
From: dan.berindei at gmail.com (Dan Berindei)
Date: Thu, 9 Mar 2017 12:22:53 +0200
Subject: [infinispan-dev] Auto-detection of appropriate
	key2StringMapperClass
In-Reply-To: <CAFm4XO17D+fVOb0_BJZ6cU4NeUjBDUuprFwJb+b7WQe4EtLcFA@mail.gmail.com>
References: <CAFm4XO3Mc2rsthTFZuVSRkTGky-iTtSiq-ePqfkYEFdXwNcCqA@mail.gmail.com>
	<CA+nfvwTorPz6ptLwkcYQJ42mhA7416YxxacD8F2za9xt9F_7wA@mail.gmail.com>
	<CAFm4XO17D+fVOb0_BJZ6cU4NeUjBDUuprFwJb+b7WQe4EtLcFA@mail.gmail.com>
Message-ID: <CA+nfvwTREQqh3dtKgjQ6nD5H4Up7LKN+v66go6GSRPnF0LTmjA@mail.gmail.com>

On Thu, Mar 9, 2017 at 11:48 AM, Sanne Grinovero <sanne at infinispan.org> wrote:
> Hi Dan!
>

Hey Sanne :)

> On 9 March 2017 at 09:38, Dan Berindei <dan.berindei at gmail.com> wrote:
>> Every store can have its own Key2StringMapper, so I really don't think
>> service discovery is appropriate here.
>
> I suggested a default, while maintaining the property as an override option.
>
> It won't make any difference to people in the situation you describe,
> but it will avoid trouble for many more users.
>
> In my experience when one develops a custom type & a custom
> Key2StringMapper, you really wouldn't have interest in creating
> multiple implementations for the same type. The only case in which I
> can see some use for allowing multiple conflicting registrations to
> the same type is for upgrade / migration cases..
>

The problem I see is that each store can only have one
Key2StringMapper implementation, so you're going to have conflicts
even if your caches use completely different key types. The store
doesn't know which key types the application actually uses, so it
can't pick the appropriate implementation when there are multiple ones
on the classpath.

>>
>> My favourite solution would be to have a default Key2StringMapper
>> implementation that uses Java serialization (not our internal
>> marshalling, because that can change between versions). It could even
>> do base64 encoding for databases that don't have a VARBINARY/BYTEA
>> equivalent.
>>

Do you see any problem with using serialization everywhere?

Dan


>> Cheers
>> Dan
>>
>>
>> On Wed, Mar 8, 2017 at 6:34 PM, Sanne Grinovero <sanne at infinispan.org> wrote:
>>> It seems I keep forgetting that the "key2StringMapperClass" needs to
>>> be explicitly set when using the String based JDBC CacheStore, which I
>>> think is a quite popular store.
>>>
>>> http://infinispan.org/docs/dev/user_guide/user_guide.html#storing_the_index_in_a_database
>>>
>>> I think we could remove the need to configure this with an appropriate
>>> service discovery?
>>>
>>> We could keep the property so to be able to override the discovery process.
>>>
>>> Would anyone volunteer to implement this?
>>>
>>> Thanks,
>>> Sanne
>>> _______________________________________________
>>> infinispan-dev mailing list
>>> infinispan-dev at lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>> _______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev

From sanne at infinispan.org  Thu Mar  9 06:09:13 2017
From: sanne at infinispan.org (Sanne Grinovero)
Date: Thu, 9 Mar 2017 11:09:13 +0000
Subject: [infinispan-dev] Auto-detection of appropriate
	key2StringMapperClass
In-Reply-To: <CA+nfvwTREQqh3dtKgjQ6nD5H4Up7LKN+v66go6GSRPnF0LTmjA@mail.gmail.com>
References: <CAFm4XO3Mc2rsthTFZuVSRkTGky-iTtSiq-ePqfkYEFdXwNcCqA@mail.gmail.com>
	<CA+nfvwTorPz6ptLwkcYQJ42mhA7416YxxacD8F2za9xt9F_7wA@mail.gmail.com>
	<CAFm4XO17D+fVOb0_BJZ6cU4NeUjBDUuprFwJb+b7WQe4EtLcFA@mail.gmail.com>
	<CA+nfvwTREQqh3dtKgjQ6nD5H4Up7LKN+v66go6GSRPnF0LTmjA@mail.gmail.com>
Message-ID: <CAFm4XO3FqfbT_PincguCjDgSD8aQMJkw_pcOQG63A17H3q=D3g@mail.gmail.com>

On 9 March 2017 at 10:22, Dan Berindei <dan.berindei at gmail.com> wrote:
> On Thu, Mar 9, 2017 at 11:48 AM, Sanne Grinovero <sanne at infinispan.org> wrote:
>> Hi Dan!
>>
>
> Hey Sanne :)
>
>> On 9 March 2017 at 09:38, Dan Berindei <dan.berindei at gmail.com> wrote:
>>> Every store can have its own Key2StringMapper, so I really don't think
>>> service discovery is appropriate here.
>>
>> I suggested a default, while maintaining the property as an override option.
>>
>> It won't make any difference to people in the situation you describe,
>> but it will avoid trouble for many more users.
>>
>> In my experience when one develops a custom type & a custom
>> Key2StringMapper, you really wouldn't have interest in creating
>> multiple implementations for the same type. The only case in which I
>> can see some use for allowing multiple conflicting registrations to
>> the same type is for upgrade / migration cases..
>>
>
> The problem I see is that each store can only have one
> Key2StringMapper implementation, so you're going to have conflicts
> even if your caches use completely different key types. The store
> doesn't know which key types the application actually uses, so it
> can't pick the appropriate implementation when there are multiple ones
> on the classpath.

Good point. I was hoping to allow different services to "contribute"
their different converters but I realise now that it would introduce a
risk of ambiguous representations.
They would need some standard scoping to know to which one to delegate
to on read operations. I wonder if we could enforce / validate that?

>>> My favourite solution would be to have a default Key2StringMapper
>>> implementation that uses Java serialization (not our internal
>>> marshalling, because that can change between versions). It could even
>>> do base64 encoding for databases that don't have a VARBINARY/BYTEA
>>> equivalent.
>>>
>
> Do you see any problem with using serialization everywhere?

I guess we could discuss alternative strategies, but it's an
orthogonal discussion right? You're proposing a different CacheStore
implementation, but people would still need a better way to associate
the adaptors to the types.

But to answer your question: yes I wouldn't like using Java
serialization: people need to be able to evolve their domain model.
When we had to "break" the string format in the Lucene mappers (as I
added a new field) we managed to do it in an evolutionary way: it
wasn't too hard to make sure we could read the old string format,
while with Java serialization this would have been a mess.

Also it would be nice to keep the representation as short as possible
for both efficient processing and storage space efficiency: Java
Serialization isn't particularly compact.

Thanks,
Sanne

>
> Dan
>
>
>>> Cheers
>>> Dan
>>>
>>>
>>> On Wed, Mar 8, 2017 at 6:34 PM, Sanne Grinovero <sanne at infinispan.org> wrote:
>>>> It seems I keep forgetting that the "key2StringMapperClass" needs to
>>>> be explicitly set when using the String based JDBC CacheStore, which I
>>>> think is a quite popular store.
>>>>
>>>> http://infinispan.org/docs/dev/user_guide/user_guide.html#storing_the_index_in_a_database
>>>>
>>>> I think we could remove the need to configure this with an appropriate
>>>> service discovery?
>>>>
>>>> We could keep the property so to be able to override the discovery process.
>>>>
>>>> Would anyone volunteer to implement this?
>>>>
>>>> Thanks,
>>>> Sanne
>>>> _______________________________________________
>>>> infinispan-dev mailing list
>>>> infinispan-dev at lists.jboss.org
>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>> _______________________________________________
>>> infinispan-dev mailing list
>>> infinispan-dev at lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>> _______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev

From dan.berindei at gmail.com  Thu Mar  9 08:17:40 2017
From: dan.berindei at gmail.com (Dan Berindei)
Date: Thu, 9 Mar 2017 15:17:40 +0200
Subject: [infinispan-dev] Auto-detection of appropriate
	key2StringMapperClass
In-Reply-To: <CAFm4XO3FqfbT_PincguCjDgSD8aQMJkw_pcOQG63A17H3q=D3g@mail.gmail.com>
References: <CAFm4XO3Mc2rsthTFZuVSRkTGky-iTtSiq-ePqfkYEFdXwNcCqA@mail.gmail.com>
	<CA+nfvwTorPz6ptLwkcYQJ42mhA7416YxxacD8F2za9xt9F_7wA@mail.gmail.com>
	<CAFm4XO17D+fVOb0_BJZ6cU4NeUjBDUuprFwJb+b7WQe4EtLcFA@mail.gmail.com>
	<CA+nfvwTREQqh3dtKgjQ6nD5H4Up7LKN+v66go6GSRPnF0LTmjA@mail.gmail.com>
	<CAFm4XO3FqfbT_PincguCjDgSD8aQMJkw_pcOQG63A17H3q=D3g@mail.gmail.com>
Message-ID: <CA+nfvwSHNZ21tH849Pvcj4pUK+rBETQ9qw7utkGYPd6DQCGoyg@mail.gmail.com>

On Thu, Mar 9, 2017 at 1:09 PM, Sanne Grinovero <sanne at infinispan.org> wrote:
> On 9 March 2017 at 10:22, Dan Berindei <dan.berindei at gmail.com> wrote:
>> On Thu, Mar 9, 2017 at 11:48 AM, Sanne Grinovero <sanne at infinispan.org> wrote:
>>> Hi Dan!
>>>
>>
>> Hey Sanne :)
>>
>>> On 9 March 2017 at 09:38, Dan Berindei <dan.berindei at gmail.com> wrote:
>>>> Every store can have its own Key2StringMapper, so I really don't think
>>>> service discovery is appropriate here.
>>>
>>> I suggested a default, while maintaining the property as an override option.
>>>
>>> It won't make any difference to people in the situation you describe,
>>> but it will avoid trouble for many more users.
>>>
>>> In my experience when one develops a custom type & a custom
>>> Key2StringMapper, you really wouldn't have interest in creating
>>> multiple implementations for the same type. The only case in which I
>>> can see some use for allowing multiple conflicting registrations to
>>> the same type is for upgrade / migration cases..
>>>
>>
>> The problem I see is that each store can only have one
>> Key2StringMapper implementation, so you're going to have conflicts
>> even if your caches use completely different key types. The store
>> doesn't know which key types the application actually uses, so it
>> can't pick the appropriate implementation when there are multiple ones
>> on the classpath.
>
> Good point. I was hoping to allow different services to "contribute"
> their different converters but I realise now that it would introduce a
> risk of ambiguous representations.
> They would need some standard scoping to know to which one to delegate
> to on read operations. I wonder if we could enforce / validate that?
>

We could prefix the string representation with the class and module
name of the mapper ;)

>>>> My favourite solution would be to have a default Key2StringMapper
>>>> implementation that uses Java serialization (not our internal
>>>> marshalling, because that can change between versions). It could even
>>>> do base64 encoding for databases that don't have a VARBINARY/BYTEA
>>>> equivalent.
>>>>
>>
>> Do you see any problem with using serialization everywhere?
>
> I guess we could discuss alternative strategies, but it's an
> orthogonal discussion right? You're proposing a different CacheStore
> implementation, but people would still need a better way to associate
> the adaptors to the types.
>

I don't think it's orthogonal, because having a single key mapper that
supports all key types would make this discussion unnecessary.

> But to answer your question: yes I wouldn't like using Java
> serialization: people need to be able to evolve their domain model.
> When we had to "break" the string format in the Lucene mappers (as I
> added a new field) we managed to do it in an evolutionary way: it
> wasn't too hard to make sure we could read the old string format,
> while with Java serialization this would have been a mess.
>

If your keys are internal it should be possible to use a new key type
and to keep supporting the old type for a while. But yeah, it's a
little more involved than adding support for a new prefix in your key
mapper.

> Also it would be nice to keep the representation as short as possible
> for both efficient processing and storage space efficiency: Java
> Serialization isn't particularly compact.
>

Yeah, I should have just said we should use the same marshalling for
the keys that we use for the values.

I didn't want to bring up our marshalling because we don't have a very
good record for preserving the format of JDK types that get included
in the user types, but Java serialization has its own warts. In the
end, the important point is that what's good for the values must be
good for the keys too.

Cheers
Dan

From paul.ferraro at redhat.com  Thu Mar  9 09:41:08 2017
From: paul.ferraro at redhat.com (Paul Ferraro)
Date: Thu, 9 Mar 2017 09:41:08 -0500
Subject: [infinispan-dev] Auto-detection of appropriate
	key2StringMapperClass
In-Reply-To: <CAFm4XO3Mc2rsthTFZuVSRkTGky-iTtSiq-ePqfkYEFdXwNcCqA@mail.gmail.com>
References: <CAFm4XO3Mc2rsthTFZuVSRkTGky-iTtSiq-ePqfkYEFdXwNcCqA@mail.gmail.com>
Message-ID: <CAF5qqoNUsiCP2r6gFw=CJGf6Szy8bcLCnHEw=tgn4y9YgXq2SA@mail.gmail.com>

Funny that you mention this, because I need to implement this for
WildFly.next to support Infinispan 9, where the binary key store is no
longer an option.  We need to be able to map complex keys for storing
web sessions, Hibernate keys, and @Stateful EJBs to strings.
An auto-discovered mapper is simple enough to integrate into WF since
we allow a given cache-container to be configured with a specific
module, from which marshalling services are auto-discovered.

Paul

On Wed, Mar 8, 2017 at 11:34 AM, Sanne Grinovero <sanne at infinispan.org> wrote:
> It seems I keep forgetting that the "key2StringMapperClass" needs to
> be explicitly set when using the String based JDBC CacheStore, which I
> think is a quite popular store.
>
> http://infinispan.org/docs/dev/user_guide/user_guide.html#storing_the_index_in_a_database
>
> I think we could remove the need to configure this with an appropriate
> service discovery?
>
> We could keep the property so to be able to override the discovery process.
>
> Would anyone volunteer to implement this?
>
> Thanks,
> Sanne
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev

From galder at redhat.com  Thu Mar  9 16:07:57 2017
From: galder at redhat.com (=?utf-8?Q?Galder_Zamarre=C3=B1o?=)
Date: Thu, 9 Mar 2017 16:07:57 -0500
Subject: [infinispan-dev] Calling getCache with a template and defined
	configuration
In-Reply-To: <CA+nfvwQV-0WRMRhwekZtk0omKib6vT1tkHjHjfNPUXMhON8=6A@mail.gmail.com>
References: <CA+YCuUOT0KJNHcg7WNhWkAa_E=NoaSHLXim3+AmR=aMk90aGbw@mail.gmail.com>
	<CA+nfvwTY=JFmspivhHmRyJ+6iWntx_zAEL+L=pqFeK4gqpUj2w@mail.gmail.com>
	<CA+YCuUN_N80u_ybdNGX1kaO6vD6LeK2o7gTXo2QxeXO46-pr1g@mail.gmail.com>
	<CA+YCuUNzEo7zxq8gEfAu5Y3G-_YVOYwasXjThOfQPyQOp6t6xw@mail.gmail.com>
	<8ddeca18-7bf5-b240-4b7a-a155dd0d3c74@redhat.com>
	<CA+nfvwQV-0WRMRhwekZtk0omKib6vT1tkHjHjfNPUXMhON8=6A@mail.gmail.com>
Message-ID: <B0145C0F-7A86-4ECC-AD9D-3FFB40F5539E@redhat.com>

An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/infinispan-dev/attachments/20170309/14e8a348/attachment-0001.html 

From remerson at redhat.com  Fri Mar 10 05:34:23 2017
From: remerson at redhat.com (Ryan Emerson)
Date: Fri, 10 Mar 2017 05:34:23 -0500 (EST)
Subject: [infinispan-dev] JDBC Migrator Blog Post
In-Reply-To: <1660100916.684200.1489142046639.JavaMail.zimbra@redhat.com>
Message-ID: <1758569240.684247.1489142063781.JavaMail.zimbra@redhat.com>

Hi All,

http://blog.infinispan.org/2017/03/jdbc-migrator-or-how-i-learned-to-stop.html

Cheers
Ryab

From galder at redhat.com  Fri Mar 10 06:38:15 2017
From: galder at redhat.com (=?utf-8?Q?Galder_Zamarre=C3=B1o?=)
Date: Fri, 10 Mar 2017 11:38:15 +0000
Subject: [infinispan-dev] GetKeyValueCommand NPE with CR1 in Hibernate
	2LC - ISPN-7029
In-Reply-To: <76d564c8-cd2f-3ef6-f043-e185180f67ed@redhat.com>
References: <B4D2982F-B134-4B07-995F-91D705D09605@redhat.com>
	<b4fe0327-ce54-27b7-7efe-02cb3ea8db18@redhat.com>
	<6F4EF3D8-2509-4209-9CD1-A3D65F195681@redhat.com>
	<b41723a8-1216-43d8-1c0c-76cb12fc63cc@redhat.com>
	<12BF89EA-E18A-4116-B9B8-43B13CAF7BE1@redhat.com>
	<324cb4ad-8e90-a83e-f98a-9b11852bbb7b@redhat.com>
	<76d564c8-cd2f-3ef6-f043-e185180f67ed@redhat.com>
Message-ID: <AFC550E1-E49E-4B5D-A3CE-752B0EFC7E9B@redhat.com>

Neat! Just tried quickly and seems to work on ISPN test... I'll see if it works in 2LC and if so I'll send a PR. 

We can then inspect where else this check might be needed.

Cheers,
--
Galder Zamarre?o
Infinispan, Red Hat

> On 27 Feb 2017, at 14:01, Radim Vansa <rvansa at redhat.com> wrote:
> 
> Hi Galder,
> 
> another solution came upon my mind (actually after finding this SO 
> question [1]): we could check if
> 
> (this.key == key || this.key.equals(key))
> 
> in SingleKeyNonTxInvocation context instead of just
> 
> this.key.equals(key)
> 
> and that could do the trick. Plus, such check maybe needed in couple of 
> other places, haven't tried that.
> 
> Seems that HashMap already does so, so the other types of contexts 
> should work OOTB.
> 
> Radim
> 
> [1] 
> http://stackoverflow.com/questions/13521184/equals-returns-false-yet-object-is-found-in-map
> 
> On 02/20/2017 05:22 PM, Radim Vansa wrote:
>> On 02/20/2017 03:52 PM, Galder Zamarre?o wrote:
>>> I've just verified the problem and the NPE can be reproduced with Infinispan alone.
>>> 
>>> More replies below:
>>> 
>>> --
>>> Galder Zamarre?o
>>> Infinispan, Red Hat
>>> 
>>>> On 16 Feb 2017, at 10:44, Radim Vansa <rvansa at redhat.com> wrote:
>>>> 
>>>> On 02/15/2017 06:07 PM, Galder Zamarre?o wrote:
>>>>> --
>>>>> Galder Zamarre?o
>>>>> Infinispan, Red Hat
>>>>> 
>>>>>> On 15 Feb 2017, at 12:21, Radim Vansa <rvansa at redhat.com> wrote:
>>>>>> 
>>>>>> On 02/15/2017 11:28 AM, Galder Zamarre?o wrote:
>>>>>>> Hey Radim,
>>>>>>> 
>>>>>>> Your changes in ISPN-7029 are causing issues with Hibernate 2LC.
>>>>>>> 
>>>>>>> In particular [2]. Name.equals() always returns false, so it'll never be found in the context. So entry is null.
>>>>>> That's obviously a bug in the 2LC testsuite, isn't it?
>>>>> LOL, is it? You added the class and the javadoc clearly states that this entity defines equals incorrectly. You must have added it for a reason?
>>>>> 
>>>>> https://github.com/hibernate/hibernate-orm/blob/master/hibernate-infinispan/src/test/java/org/hibernate/test/cache/infinispan/functional/entities/Name.java
>>>>> 
>>>>> In any case, an object with an incorrect equals impl should not result in an NPE within Infinispan :|
>>>>> 
>>>>>> Object used as @EmbeddedId needs to have the equals correctly defined. How else would you compare ids? I wonder how could that work in the past.
>>>>> ROFL... it's your baby so you tell me ;)
>>>> Okay, okay - I haven't checked the javadoc, so I just assumed that it's an oversight :)
>>>> 
>>>>>>> Moreover, if EntryLookup.lookupEntry javadoc (and some implementations) can and do return null. Are you expecting that somehow that method will never return null?
>>>>>> With ISPN-7029 in, the entry should be wrapped in the context after EntryWrappingInterceptor if the key is in readCH, otherwise it should be null. In case that xDistributionInterceptor finds out that it needs to work on that value despite not being able to read it (e.g. in case that it's in writeCH during unfinished rebalance), it should wrap NullCacheEntry.getInstance() using EntryFactory. wrapExternalEntry. More info about the logic is in o.i.c.EntryFactory javadoc [3].
>>>>> Not sure I understand what you're trying to imply above... so, is lookupEntry() allowed to return null or not?
>>>> It is allowed to return null, but:
>>>> 
>>>> 1. If the node is an owner according to readCH, the entry must be wrapped into context in EWI (possibly as NullCacheEntry).
>>>> 2. The code can reach the GKVC.perform() iff this node is an owner of given key.
>>>> 
>>>> The problem here is that I've assumed that if the entry was wrapped, it can be fetched. With incorrectly defined equals, as we see here, this does not hold. So we can
>>>> 
>>>> a) check if the entry is null and throw more explanatory exception - more code in perform()
>>>> b) do the lookup after wrapping and throw there - unnecessary map lookup for such annoying problem
>>>> c) ostrich approach
>>>> 
>>>> I think that b) in assert could do, otherwise I'd suggest c)
>>> Hmmmmm, what about not throwing an exception at all?
>>> 
>>> IOW, e.g. in SingleKeyNonTxInvocationContext.lookupEntry(), if the key is not equals to the one cached, but the cached one is NullCacheEntry, couldn't it return that instead of null? What I'm trying to achieve here is that if the equals is not correctly implemented, the get() returns null, rather than throwing an exception.
>>> 
>>> We'd also need to verify whether other context implementations could deal with it.
>>> 
>>> Wouldn't that be better :\ ? I can see the point of throwing an exception (other than NPE of course), but it feels a little abrupt... particularly since seems like previously we've just returned null in such situations.
>> You're right that Cache.get(x) should ideally return null.
>> 
>> InvocationContext.lookupEntry() returning NullCacheEntry and null has a
>> different meaning, though:
>> 
>> * NullCacheEntry means that we are supposed to be able to read that
>> entry (according to readCH) but we haven't found that in DC. This is
>> used for example in CacheLoaderInterceptor, where we skip entries that
>> aren't in the context as opposed to recomputing the hash of the key.
>> * null means that this entry cannot be read on this node because the
>> segment this key belongs to is not in readCH, and therefore the command
>> shouldn't get past distribution interceptor.
>> 
>> I don't mind changing the 'design' as long as it follows some simple
>> rules. I don't see how returning NCE for *any* key could be incorporated
>> into the rules, and how should that behave on multi-key contexts.
>> 
>> In all but distribution mode, during read the key must be always wrapped
>> for read, because there's either no readCH (local) or replicated CH
>> (repl/invalidation). But in invalidation/local mode we can't handle that
>> in distribution interceptor as this is not present.
>> 
>> I would really prefer to see this handled in
>> EntryWrappingInterceptor/EntryWrappingFactory rather than in each
>> command - in #4564 [2] I wanted to narrow the possible situations to be
>> handled in perform(), and other pieces (e.g. return handlers - see
>> dataReadReturnHandler, that will throw NPE as well in TX RR right now).
>> Note that some commands do the check ATM (e.g. functional commands) - I
>> think that those are leftovers and should be gone.
>> 
>> I was about to suggest adding
>> 
>> if (!key.equals(key)) return null;
>> 
>> to all EWI.visitXXX methods as proper implemenations of equals should do the
>> 
>> if (other == this) return true;
>> 
>> anyway as a shortcut, so such check shouldn't affect performance.
>> However, this is not as simple with functional commands which should
>> operate on the null entry (on some node), we have to run these, and you
>> can't even ignore the value, you need some entry in there.
>> 
>> I hate not finding my socks in the drawer where I definitely put those.
>> This just breaks too many things :-(
>> 
>> Btw., it seems that SingleTxContextInvocationContext [1] will fail when
>> the entry is loaded from cache store - this is the place where you could
>> put the entry into invocation context twice. So the use case was
>> somewhat broken before, too.
>> 
>> 
>> [1]
>> https://github.com/infinispan/infinispan/blob/master/core/src/main/java/org/infinispan/context/SingleKeyNonTxInvocationContext.java#L114
>> [2] https://github.com/infinispan/infinispan/pull/4564
>> 
>> 
>>> Cheers,
>>> 
>>>> Radim
>>>> 
>>>>> To be more specific, SingleKeyNonTxInvocationContext.lookupEntry() can return null, so GetKeyValueCommand should be able to handle it? Or should SingleKeyNonTxInvocationContext.lookupEntry return NullCacheEntry.getInstance instead of null?
>>>>> 
>>>>> To provide more specifics, SingleKeyNonTxInvocationContext has NullCacheEntry.getInstance in cacheEntry variable when it's returning null. Should it maybe return NullCacheEntry.getInstance instead of null from lookupEntry() ?
>>>>> 
>>>>> Cheers,
>>>>> 
>>>>>> Radim
>>>>>> 
>>>>>> [3] https://github.com/infinispan/infinispan/blob/master/core/src/main/java/org/infinispan/container/EntryFactory.java
>>>>>> 
>>>>>>> I'll create a JIRA to track all issues arising from Hibernate 2LC in a minute, but wanted to get your thoughts firts.
>>>>>>> 
>>>>>>> Cheers,
>>>>>>> 
>>>>>>> [1] https://issues.jboss.org/browse/ISPN-7029
>>>>>>> [2] https://github.com/hibernate/hibernate-orm/blob/master/hibernate-infinispan/src/test/java/org/hibernate/test/cache/infinispan/CacheKeysFactoryTest.java#L58
>>>>>>> --
>>>>>>> Galder Zamarre?o
>>>>>>> Infinispan, Red Hat
>>>>>>> 
>>>>>> -- 
>>>>>> Radim Vansa <rvansa at redhat.com>
>>>>>> JBoss Performance Team
>>>> -- 
>>>> Radim Vansa <rvansa at redhat.com>
>>>> JBoss Performance Team
>>>> 
>> 
> 
> 
> -- 
> Radim Vansa <rvansa at redhat.com>
> JBoss Performance Team
> 
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev



From galder at redhat.com  Fri Mar 10 07:02:54 2017
From: galder at redhat.com (=?utf-8?Q?Galder_Zamarre=C3=B1o?=)
Date: Fri, 10 Mar 2017 12:02:54 +0000
Subject: [infinispan-dev] GetKeyValueCommand NPE with CR1 in Hibernate
	2LC - ISPN-7029
In-Reply-To: <AFC550E1-E49E-4B5D-A3CE-752B0EFC7E9B@redhat.com>
References: <B4D2982F-B134-4B07-995F-91D705D09605@redhat.com>
	<b4fe0327-ce54-27b7-7efe-02cb3ea8db18@redhat.com>
	<6F4EF3D8-2509-4209-9CD1-A3D65F195681@redhat.com>
	<b41723a8-1216-43d8-1c0c-76cb12fc63cc@redhat.com>
	<12BF89EA-E18A-4116-B9B8-43B13CAF7BE1@redhat.com>
	<324cb4ad-8e90-a83e-f98a-9b11852bbb7b@redhat.com>
	<76d564c8-cd2f-3ef6-f043-e185180f67ed@redhat.com>
	<AFC550E1-E49E-4B5D-A3CE-752B0EFC7E9B@redhat.com>
Message-ID: <9C074A80-BF27-46A2-8B78-E930613994CD@redhat.com>

Looks good on 2LC, so I've added it to this PR containing 2LC related fixes:
https://github.com/infinispan/infinispan/pull/4960

Down to 18 failures on 2LC...

Cheers,
--
Galder Zamarre?o
Infinispan, Red Hat

> On 10 Mar 2017, at 11:38, Galder Zamarre?o <galder at redhat.com> wrote:
> 
> Neat! Just tried quickly and seems to work on ISPN test... I'll see if it works in 2LC and if so I'll send a PR. 
> 
> We can then inspect where else this check might be needed.
> 
> Cheers,
> --
> Galder Zamarre?o
> Infinispan, Red Hat
> 
>> On 27 Feb 2017, at 14:01, Radim Vansa <rvansa at redhat.com> wrote:
>> 
>> Hi Galder,
>> 
>> another solution came upon my mind (actually after finding this SO 
>> question [1]): we could check if
>> 
>> (this.key == key || this.key.equals(key))
>> 
>> in SingleKeyNonTxInvocation context instead of just
>> 
>> this.key.equals(key)
>> 
>> and that could do the trick. Plus, such check maybe needed in couple of 
>> other places, haven't tried that.
>> 
>> Seems that HashMap already does so, so the other types of contexts 
>> should work OOTB.
>> 
>> Radim
>> 
>> [1] 
>> http://stackoverflow.com/questions/13521184/equals-returns-false-yet-object-is-found-in-map
>> 
>> On 02/20/2017 05:22 PM, Radim Vansa wrote:
>>> On 02/20/2017 03:52 PM, Galder Zamarre?o wrote:
>>>> I've just verified the problem and the NPE can be reproduced with Infinispan alone.
>>>> 
>>>> More replies below:
>>>> 
>>>> --
>>>> Galder Zamarre?o
>>>> Infinispan, Red Hat
>>>> 
>>>>> On 16 Feb 2017, at 10:44, Radim Vansa <rvansa at redhat.com> wrote:
>>>>> 
>>>>> On 02/15/2017 06:07 PM, Galder Zamarre?o wrote:
>>>>>> --
>>>>>> Galder Zamarre?o
>>>>>> Infinispan, Red Hat
>>>>>> 
>>>>>>> On 15 Feb 2017, at 12:21, Radim Vansa <rvansa at redhat.com> wrote:
>>>>>>> 
>>>>>>> On 02/15/2017 11:28 AM, Galder Zamarre?o wrote:
>>>>>>>> Hey Radim,
>>>>>>>> 
>>>>>>>> Your changes in ISPN-7029 are causing issues with Hibernate 2LC.
>>>>>>>> 
>>>>>>>> In particular [2]. Name.equals() always returns false, so it'll never be found in the context. So entry is null.
>>>>>>> That's obviously a bug in the 2LC testsuite, isn't it?
>>>>>> LOL, is it? You added the class and the javadoc clearly states that this entity defines equals incorrectly. You must have added it for a reason?
>>>>>> 
>>>>>> https://github.com/hibernate/hibernate-orm/blob/master/hibernate-infinispan/src/test/java/org/hibernate/test/cache/infinispan/functional/entities/Name.java
>>>>>> 
>>>>>> In any case, an object with an incorrect equals impl should not result in an NPE within Infinispan :|
>>>>>> 
>>>>>>> Object used as @EmbeddedId needs to have the equals correctly defined. How else would you compare ids? I wonder how could that work in the past.
>>>>>> ROFL... it's your baby so you tell me ;)
>>>>> Okay, okay - I haven't checked the javadoc, so I just assumed that it's an oversight :)
>>>>> 
>>>>>>>> Moreover, if EntryLookup.lookupEntry javadoc (and some implementations) can and do return null. Are you expecting that somehow that method will never return null?
>>>>>>> With ISPN-7029 in, the entry should be wrapped in the context after EntryWrappingInterceptor if the key is in readCH, otherwise it should be null. In case that xDistributionInterceptor finds out that it needs to work on that value despite not being able to read it (e.g. in case that it's in writeCH during unfinished rebalance), it should wrap NullCacheEntry.getInstance() using EntryFactory. wrapExternalEntry. More info about the logic is in o.i.c.EntryFactory javadoc [3].
>>>>>> Not sure I understand what you're trying to imply above... so, is lookupEntry() allowed to return null or not?
>>>>> It is allowed to return null, but:
>>>>> 
>>>>> 1. If the node is an owner according to readCH, the entry must be wrapped into context in EWI (possibly as NullCacheEntry).
>>>>> 2. The code can reach the GKVC.perform() iff this node is an owner of given key.
>>>>> 
>>>>> The problem here is that I've assumed that if the entry was wrapped, it can be fetched. With incorrectly defined equals, as we see here, this does not hold. So we can
>>>>> 
>>>>> a) check if the entry is null and throw more explanatory exception - more code in perform()
>>>>> b) do the lookup after wrapping and throw there - unnecessary map lookup for such annoying problem
>>>>> c) ostrich approach
>>>>> 
>>>>> I think that b) in assert could do, otherwise I'd suggest c)
>>>> Hmmmmm, what about not throwing an exception at all?
>>>> 
>>>> IOW, e.g. in SingleKeyNonTxInvocationContext.lookupEntry(), if the key is not equals to the one cached, but the cached one is NullCacheEntry, couldn't it return that instead of null? What I'm trying to achieve here is that if the equals is not correctly implemented, the get() returns null, rather than throwing an exception.
>>>> 
>>>> We'd also need to verify whether other context implementations could deal with it.
>>>> 
>>>> Wouldn't that be better :\ ? I can see the point of throwing an exception (other than NPE of course), but it feels a little abrupt... particularly since seems like previously we've just returned null in such situations.
>>> You're right that Cache.get(x) should ideally return null.
>>> 
>>> InvocationContext.lookupEntry() returning NullCacheEntry and null has a
>>> different meaning, though:
>>> 
>>> * NullCacheEntry means that we are supposed to be able to read that
>>> entry (according to readCH) but we haven't found that in DC. This is
>>> used for example in CacheLoaderInterceptor, where we skip entries that
>>> aren't in the context as opposed to recomputing the hash of the key.
>>> * null means that this entry cannot be read on this node because the
>>> segment this key belongs to is not in readCH, and therefore the command
>>> shouldn't get past distribution interceptor.
>>> 
>>> I don't mind changing the 'design' as long as it follows some simple
>>> rules. I don't see how returning NCE for *any* key could be incorporated
>>> into the rules, and how should that behave on multi-key contexts.
>>> 
>>> In all but distribution mode, during read the key must be always wrapped
>>> for read, because there's either no readCH (local) or replicated CH
>>> (repl/invalidation). But in invalidation/local mode we can't handle that
>>> in distribution interceptor as this is not present.
>>> 
>>> I would really prefer to see this handled in
>>> EntryWrappingInterceptor/EntryWrappingFactory rather than in each
>>> command - in #4564 [2] I wanted to narrow the possible situations to be
>>> handled in perform(), and other pieces (e.g. return handlers - see
>>> dataReadReturnHandler, that will throw NPE as well in TX RR right now).
>>> Note that some commands do the check ATM (e.g. functional commands) - I
>>> think that those are leftovers and should be gone.
>>> 
>>> I was about to suggest adding
>>> 
>>> if (!key.equals(key)) return null;
>>> 
>>> to all EWI.visitXXX methods as proper implemenations of equals should do the
>>> 
>>> if (other == this) return true;
>>> 
>>> anyway as a shortcut, so such check shouldn't affect performance.
>>> However, this is not as simple with functional commands which should
>>> operate on the null entry (on some node), we have to run these, and you
>>> can't even ignore the value, you need some entry in there.
>>> 
>>> I hate not finding my socks in the drawer where I definitely put those.
>>> This just breaks too many things :-(
>>> 
>>> Btw., it seems that SingleTxContextInvocationContext [1] will fail when
>>> the entry is loaded from cache store - this is the place where you could
>>> put the entry into invocation context twice. So the use case was
>>> somewhat broken before, too.
>>> 
>>> 
>>> [1]
>>> https://github.com/infinispan/infinispan/blob/master/core/src/main/java/org/infinispan/context/SingleKeyNonTxInvocationContext.java#L114
>>> [2] https://github.com/infinispan/infinispan/pull/4564
>>> 
>>> 
>>>> Cheers,
>>>> 
>>>>> Radim
>>>>> 
>>>>>> To be more specific, SingleKeyNonTxInvocationContext.lookupEntry() can return null, so GetKeyValueCommand should be able to handle it? Or should SingleKeyNonTxInvocationContext.lookupEntry return NullCacheEntry.getInstance instead of null?
>>>>>> 
>>>>>> To provide more specifics, SingleKeyNonTxInvocationContext has NullCacheEntry.getInstance in cacheEntry variable when it's returning null. Should it maybe return NullCacheEntry.getInstance instead of null from lookupEntry() ?
>>>>>> 
>>>>>> Cheers,
>>>>>> 
>>>>>>> Radim
>>>>>>> 
>>>>>>> [3] https://github.com/infinispan/infinispan/blob/master/core/src/main/java/org/infinispan/container/EntryFactory.java
>>>>>>> 
>>>>>>>> I'll create a JIRA to track all issues arising from Hibernate 2LC in a minute, but wanted to get your thoughts firts.
>>>>>>>> 
>>>>>>>> Cheers,
>>>>>>>> 
>>>>>>>> [1] https://issues.jboss.org/browse/ISPN-7029
>>>>>>>> [2] https://github.com/hibernate/hibernate-orm/blob/master/hibernate-infinispan/src/test/java/org/hibernate/test/cache/infinispan/CacheKeysFactoryTest.java#L58
>>>>>>>> --
>>>>>>>> Galder Zamarre?o
>>>>>>>> Infinispan, Red Hat
>>>>>>>> 
>>>>>>> -- 
>>>>>>> Radim Vansa <rvansa at redhat.com>
>>>>>>> JBoss Performance Team
>>>>> -- 
>>>>> Radim Vansa <rvansa at redhat.com>
>>>>> JBoss Performance Team
>>>>> 
>>> 
>> 
>> 
>> -- 
>> Radim Vansa <rvansa at redhat.com>
>> JBoss Performance Team
>> 
>> _______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> 
> 
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev



From sanne at infinispan.org  Mon Mar 13 07:45:47 2017
From: sanne at infinispan.org (Sanne Grinovero)
Date: Mon, 13 Mar 2017 11:45:47 +0000
Subject: [infinispan-dev] Repeatable reads and WSC as default
In-Reply-To: <dde4c7c5-70e7-580c-bae9-c8baba5c188f@redhat.com>
References: <9801e8ae-994a-b23b-e45d-5d60a2ce3181@redhat.com>
	<CA+nfvwR3UAn_=c0H4PYAz0ARf6F-sYAZS+g=waFjHTn048pQ0g@mail.gmail.com>
	<6E978A3B-9052-461E-8621-A1A2848B7654@redhat.com>
	<dde4c7c5-70e7-580c-bae9-c8baba5c188f@redhat.com>
Message-ID: <CAFm4XO0W7t4H9Ew7ydCOt7dC777eBmB0G=dHDVDgPYgOZQZO9w@mail.gmail.com>

Since Infinispan 9 isn't final, and considering it's just flipping a
switch for a default configuration, I don't think it's too late.

This change would only "break" people who don't read the migration
guides, and are running with the default configuration which is
totally broken anyway so we might be doing them a favour..

In addition, I'd highly welcome a warning to be printed when running
with optimistic transactions and WSC turned off. You'd save me a
couple of hours every single year as I keep falling in this trap ;-)

Thanks,
Sanne

On 27 February 2017 at 11:18, Radim Vansa <rvansa at redhat.com> wrote:
> Why is that JIRA closed? I was about to suggest that a complete JIRA
> triage in Alpha -> Beta stage could help, but that wouldn't notice
> closed ones.
>
> R.
>
> On 02/27/2017 11:57 AM, Galder Zamarre?o wrote:
>> We keep missing the train :(
>>
>> Enabling WSC by default is something we've discussed before (JIRA & dev list), e.g.
>>
>> https://issues.jboss.org/browse/ISPN-3655
>>
>> Cheers,
>> --
>> Galder Zamarre?o
>> Infinispan, Red Hat
>>
>>> On 24 Feb 2017, at 15:45, Dan Berindei <dan.berindei at gmail.com> wrote:
>>>
>>> I still think WSC should be enabled by default, but we probably missed
>>> the 9.0 train.
>>>
>>> I see the 8.0 discussion also included batching, i.e. hiding the WSC
>>> options and forcing batching to use optimistic locking w/out WSC
>>> (currently batching allows any locking type, but batch transactions
>>> are isolated from external transactions). That would be at odds with
>>> Sanne's suggestions in the "major version cleaning" thread, because
>>> those would require a batch to join any external transaction.
>>>
>>> Cheers
>>> Dan
>>>
>>>
>>> On Fri, Feb 24, 2017 at 3:53 PM, Radim Vansa <rvansa at redhat.com> wrote:
>>>> Hi all,
>>>>
>>>> I am writing a bit too late in 9.0 cycle, but since I've found couple of
>>>> bugs [1][2] and spaces for improvement [3][4] in WSC implementation, I
>>>> was wondering about confirming the performance effect of fixes. And I am
>>>> wondering if the current defaults for transactional cache, being
>>>> READ_COMMITTED and WSC off are the best options.
>>>>
>>>> During 8.0 cycle Tristan raised a discussion [5] including WSC defaults
>>>> and I think that the general opinion was "make it default", which
>>>> requires RR isolation and simple versioning scheme. But I think that the
>>>> whole thing [6] went a bit forgotten and all that was done is [7].
>>>>
>>>> Do we want to do this, at least using current API? (we most likely don't
>>>> want to refactor ConfigurationBuilder when CR2 is out)
>>>>
>>>> Radim
>>>>
>>>> [1] https://issues.jboss.org/browse/ISPN-7526
>>>> [2] https://issues.jboss.org/browse/ISPN-7527
>>>> [3] https://issues.jboss.org/browse/ISPN-7528
>>>> [4]
>>>> https://github.com/infinispan/infinispan/blob/master/core/src/main/java/org/infinispan/container/entries/ClusteredRepeatableReadEntry.java#L30
>>>> could be a contention point
>>>> [5]
>>>> http://infinispan.markmail.org/message/rasqqkojwdboql7y?q=write+skew+list:org%2Ejboss%2Elists%2Einfinispan-dev+order:date-backward
>>>> [6] https://issues.jboss.org/browse/ISPN-3927
>>>> [7] https://issues.jboss.org/browse/ISPN-7507
>>>>
>>>> --
>>>> Radim Vansa <rvansa at redhat.com>
>>>> JBoss Performance Team
>>>>
>>>> _______________________________________________
>>>> infinispan-dev mailing list
>>>> infinispan-dev at lists.jboss.org
>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>> _______________________________________________
>>> infinispan-dev mailing list
>>> infinispan-dev at lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>
>> _______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
>
> --
> Radim Vansa <rvansa at redhat.com>
> JBoss Performance Team
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev


From ttarrant at redhat.com  Mon Mar 13 07:50:47 2017
From: ttarrant at redhat.com (Tristan Tarrant)
Date: Mon, 13 Mar 2017 12:50:47 +0100
Subject: [infinispan-dev] Repeatable reads and WSC as default
In-Reply-To: <dde4c7c5-70e7-580c-bae9-c8baba5c188f@redhat.com>
References: <9801e8ae-994a-b23b-e45d-5d60a2ce3181@redhat.com>
	<CA+nfvwR3UAn_=c0H4PYAz0ARf6F-sYAZS+g=waFjHTn048pQ0g@mail.gmail.com>
	<6E978A3B-9052-461E-8621-A1A2848B7654@redhat.com>
	<dde4c7c5-70e7-580c-bae9-c8baba5c188f@redhat.com>
Message-ID: <33151391-90ff-288a-9462-e8ecbe4a6c23@redhat.com>

Let's make it default. The 9.0 train definitely hasn't left the station yet.

Tristan

On 27/02/17 12:18, Radim Vansa wrote:
> Why is that JIRA closed? I was about to suggest that a complete JIRA
> triage in Alpha -> Beta stage could help, but that wouldn't notice
> closed ones.
> 
> R.
> 
> On 02/27/2017 11:57 AM, Galder Zamarre?o wrote:
>> We keep missing the train :(
>>
>> Enabling WSC by default is something we've discussed before (JIRA & dev list), e.g.
>>
>> https://issues.jboss.org/browse/ISPN-3655
>>
>> Cheers,
>> --
>> Galder Zamarre?o
>> Infinispan, Red Hat
>>
>>> On 24 Feb 2017, at 15:45, Dan Berindei <dan.berindei at gmail.com> wrote:
>>>
>>> I still think WSC should be enabled by default, but we probably missed
>>> the 9.0 train.
>>>
>>> I see the 8.0 discussion also included batching, i.e. hiding the WSC
>>> options and forcing batching to use optimistic locking w/out WSC
>>> (currently batching allows any locking type, but batch transactions
>>> are isolated from external transactions). That would be at odds with
>>> Sanne's suggestions in the "major version cleaning" thread, because
>>> those would require a batch to join any external transaction.
>>>
>>> Cheers
>>> Dan
>>>
>>>
>>> On Fri, Feb 24, 2017 at 3:53 PM, Radim Vansa <rvansa at redhat.com> wrote:
>>>> Hi all,
>>>>
>>>> I am writing a bit too late in 9.0 cycle, but since I've found couple of
>>>> bugs [1][2] and spaces for improvement [3][4] in WSC implementation, I
>>>> was wondering about confirming the performance effect of fixes. And I am
>>>> wondering if the current defaults for transactional cache, being
>>>> READ_COMMITTED and WSC off are the best options.
>>>>
>>>> During 8.0 cycle Tristan raised a discussion [5] including WSC defaults
>>>> and I think that the general opinion was "make it default", which
>>>> requires RR isolation and simple versioning scheme. But I think that the
>>>> whole thing [6] went a bit forgotten and all that was done is [7].
>>>>
>>>> Do we want to do this, at least using current API? (we most likely don't
>>>> want to refactor ConfigurationBuilder when CR2 is out)
>>>>
>>>> Radim
>>>>
>>>> [1] https://issues.jboss.org/browse/ISPN-7526
>>>> [2] https://issues.jboss.org/browse/ISPN-7527
>>>> [3] https://issues.jboss.org/browse/ISPN-7528
>>>> [4]
>>>> https://github.com/infinispan/infinispan/blob/master/core/src/main/java/org/infinispan/container/entries/ClusteredRepeatableReadEntry.java#L30
>>>> could be a contention point
>>>> [5]
>>>> http://infinispan.markmail.org/message/rasqqkojwdboql7y?q=write+skew+list:org%2Ejboss%2Elists%2Einfinispan-dev+order:date-backward
>>>> [6] https://issues.jboss.org/browse/ISPN-3927
>>>> [7] https://issues.jboss.org/browse/ISPN-7507
>>>>
>>>> --
>>>> Radim Vansa <rvansa at redhat.com>
>>>> JBoss Performance Team
>>>>
>>>> _______________________________________________
>>>> infinispan-dev mailing list
>>>> infinispan-dev at lists.jboss.org
>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>> _______________________________________________
>>> infinispan-dev mailing list
>>> infinispan-dev at lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>
>> _______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> 
> 

-- 
Tristan Tarrant
Infinispan Lead
JBoss, a division of Red Hat

From sanne at infinispan.org  Mon Mar 13 08:11:25 2017
From: sanne at infinispan.org (Sanne Grinovero)
Date: Mon, 13 Mar 2017 12:11:25 +0000
Subject: [infinispan-dev] Repeatable reads and WSC as default
In-Reply-To: <33151391-90ff-288a-9462-e8ecbe4a6c23@redhat.com>
References: <9801e8ae-994a-b23b-e45d-5d60a2ce3181@redhat.com>
	<CA+nfvwR3UAn_=c0H4PYAz0ARf6F-sYAZS+g=waFjHTn048pQ0g@mail.gmail.com>
	<6E978A3B-9052-461E-8621-A1A2848B7654@redhat.com>
	<dde4c7c5-70e7-580c-bae9-c8baba5c188f@redhat.com>
	<33151391-90ff-288a-9462-e8ecbe4a6c23@redhat.com>
Message-ID: <CAFm4XO2OCAenkhZgq+jVxryXJezf=vM10bpWW3kTakMfJuqH=Q@mail.gmail.com>

Thanks!

Sanne

On 13 March 2017 at 11:50, Tristan Tarrant <ttarrant at redhat.com> wrote:
> Let's make it default. The 9.0 train definitely hasn't left the station yet.
>
> Tristan
>
> On 27/02/17 12:18, Radim Vansa wrote:
>> Why is that JIRA closed? I was about to suggest that a complete JIRA
>> triage in Alpha -> Beta stage could help, but that wouldn't notice
>> closed ones.
>>
>> R.
>>
>> On 02/27/2017 11:57 AM, Galder Zamarre?o wrote:
>>> We keep missing the train :(
>>>
>>> Enabling WSC by default is something we've discussed before (JIRA & dev list), e.g.
>>>
>>> https://issues.jboss.org/browse/ISPN-3655
>>>
>>> Cheers,
>>> --
>>> Galder Zamarre?o
>>> Infinispan, Red Hat
>>>
>>>> On 24 Feb 2017, at 15:45, Dan Berindei <dan.berindei at gmail.com> wrote:
>>>>
>>>> I still think WSC should be enabled by default, but we probably missed
>>>> the 9.0 train.
>>>>
>>>> I see the 8.0 discussion also included batching, i.e. hiding the WSC
>>>> options and forcing batching to use optimistic locking w/out WSC
>>>> (currently batching allows any locking type, but batch transactions
>>>> are isolated from external transactions). That would be at odds with
>>>> Sanne's suggestions in the "major version cleaning" thread, because
>>>> those would require a batch to join any external transaction.
>>>>
>>>> Cheers
>>>> Dan
>>>>
>>>>
>>>> On Fri, Feb 24, 2017 at 3:53 PM, Radim Vansa <rvansa at redhat.com> wrote:
>>>>> Hi all,
>>>>>
>>>>> I am writing a bit too late in 9.0 cycle, but since I've found couple of
>>>>> bugs [1][2] and spaces for improvement [3][4] in WSC implementation, I
>>>>> was wondering about confirming the performance effect of fixes. And I am
>>>>> wondering if the current defaults for transactional cache, being
>>>>> READ_COMMITTED and WSC off are the best options.
>>>>>
>>>>> During 8.0 cycle Tristan raised a discussion [5] including WSC defaults
>>>>> and I think that the general opinion was "make it default", which
>>>>> requires RR isolation and simple versioning scheme. But I think that the
>>>>> whole thing [6] went a bit forgotten and all that was done is [7].
>>>>>
>>>>> Do we want to do this, at least using current API? (we most likely don't
>>>>> want to refactor ConfigurationBuilder when CR2 is out)
>>>>>
>>>>> Radim
>>>>>
>>>>> [1] https://issues.jboss.org/browse/ISPN-7526
>>>>> [2] https://issues.jboss.org/browse/ISPN-7527
>>>>> [3] https://issues.jboss.org/browse/ISPN-7528
>>>>> [4]
>>>>> https://github.com/infinispan/infinispan/blob/master/core/src/main/java/org/infinispan/container/entries/ClusteredRepeatableReadEntry.java#L30
>>>>> could be a contention point
>>>>> [5]
>>>>> http://infinispan.markmail.org/message/rasqqkojwdboql7y?q=write+skew+list:org%2Ejboss%2Elists%2Einfinispan-dev+order:date-backward
>>>>> [6] https://issues.jboss.org/browse/ISPN-3927
>>>>> [7] https://issues.jboss.org/browse/ISPN-7507
>>>>>
>>>>> --
>>>>> Radim Vansa <rvansa at redhat.com>
>>>>> JBoss Performance Team
>>>>>
>>>>> _______________________________________________
>>>>> infinispan-dev mailing list
>>>>> infinispan-dev at lists.jboss.org
>>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>>> _______________________________________________
>>>> infinispan-dev mailing list
>>>> infinispan-dev at lists.jboss.org
>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>>
>>> _______________________________________________
>>> infinispan-dev mailing list
>>> infinispan-dev at lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>
>>
>
> --
> Tristan Tarrant
> Infinispan Lead
> JBoss, a division of Red Hat
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev


From remerson at redhat.com  Tue Mar 14 07:26:26 2017
From: remerson at redhat.com (Ryan Emerson)
Date: Tue, 14 Mar 2017 07:26:26 -0400 (EDT)
Subject: [infinispan-dev] Cache Store Marshalling
In-Reply-To: <1289292546.2960092.1489488818067.JavaMail.zimbra@redhat.com>
Message-ID: <4686643.2964817.1489490786403.JavaMail.zimbra@redhat.com>

Hi All,

Currently the CacheWriterInterceptor utilises the internal marshaller for marshalling entries before they are sent to the configured cache stores. This causes several problems [1], most notably that changes to the internal marshaller make stored data incompatible across Infinispan versions. 

I propose that we decouple the internal and store marshaller. To allow 9.x versions to remain compatible, we should default to the internal marshaller (until 10.x at least), but optionally allow users to specify a custom StreamingMarshaller implementation as part of their PersistenceConfiguration. As we already have the protostuff and kryo bridges, users would specify an enum for the marshaller they want to use as well as an optional class string if a custom implementation is required. So for example:

enum StoreMarshaller {
   CUSTOM,
   INTERNAL,
   KRYO
   PROTOSTUFF;
}

new ConfigurationBuilder()
    .persistence()
        .marshaller(StoreMarshaller.CUSTOM)
        .marshallerClass("org.example.Marshaller")
         ...

Finally, Gustavo brought flatbuffers[2] to my attention which could be a good option to provide for users as one of the default StoreMarshaller implementations.

WDYT?

Cheers
Ryan

[1] https://docs.google.com/document/d/1PR0eYgjhqXUR5w03npS7TdOs2KDZjfdMh0au8XAsyJY
[2] https://google.github.io/flatbuffers/

From sanne at infinispan.org  Tue Mar 14 09:08:24 2017
From: sanne at infinispan.org (Sanne Grinovero)
Date: Tue, 14 Mar 2017 13:08:24 +0000
Subject: [infinispan-dev] Infinispan "mitosis"
Message-ID: <CAFm4XO2_79wbm+SkFkzOneb8-3wJsH1LxNs8oOgb1_X-u4sj0w@mail.gmail.com>

Just throwing out an idea I just had while thinking of Hibernate OGM
user needs for data migration.

For people using databases & related frameworks, it's common to have a
staging database which contains not just the staging "schema" but also
data. When legally possible, it's often preferable to have snapshot
from production data.

For example last time I worked with a SQL database, each week I'd take
a production backup and restore it on both our staging environment and
on the developer's instances so that everyone could run tests on it -
without needing access to the real production.

Interestingly, while we don't have an easy tool get a fully consistent
snapshot from a live Infinispan grid, "replicating" should be a
familiar concept here?

Infinispan could have a "mitosis" feature, like cell reproduction, in
which the user connects a pristine grid instance of N nodes and these
N nodes automatically become non-primary owners for the full set of
segments - this could happen via a custom hash which makes them all
backup replicas (no main owners) and then Infinispan would be able to
tell when state transfer is completed, and initiate some some
coordination to sever the link without triggering re-hash on the
original "production" grid.

Incidentally, while this happens the child datagrid would be kept up
to date with in-flight changes, so we could envision either a very
short lock on changes to guarantee a fully consistent snapshot, or not
have any lock at all but minimise the inconsistencies to those which
might happen during the link decoupling.

Maybe this would satisfy also the people who've been asking for the
snapshot feature? I don't think people want a snapshot for the sake of
it, but to replicate the grid..

I realise it's not a 1 day of work and the idea is not fully fleshed
out, but I think this would be a very well received feature.


Thanks,
Sanne

From dan.berindei at gmail.com  Tue Mar 14 09:36:10 2017
From: dan.berindei at gmail.com (Dan Berindei)
Date: Tue, 14 Mar 2017 15:36:10 +0200
Subject: [infinispan-dev] Cache Store Marshalling
In-Reply-To: <4686643.2964817.1489490786403.JavaMail.zimbra@redhat.com>
References: <1289292546.2960092.1489488818067.JavaMail.zimbra@redhat.com>
	<4686643.2964817.1489490786403.JavaMail.zimbra@redhat.com>
Message-ID: <CA+nfvwRFA2d8rbxpYY3cM1BZp5Wymcyj7KQN+UGxkVkAWKkg4A@mail.gmail.com>

+100 to use a separate marshaller for user types, but I think the
persistence configuration is the wrong place for it. Users shouldn't
have to implement an AdvancedExternalizer for efficient replication
between nodes if they already have a marshaller that's efficient
enough for use in a store.

I would have liked the default user type marshaller to use
jboss-marshalling with the all the 8.2.x internal externalizers, in
order to have backwards compatibility for the stores without needing
an upgrade tool. But it seems like the stores change their layout all
the time anyway, so I would settle for some version metadata in the
store and better error messages instead.

I'm not sure about the enum, don't like the feeling that marshallers
not bundled with the core are somehow second-class. I'd rather have a
single marshaller name string attribute, have a NAME constant in each
marshaller class, and register the names on startup, either via
ServiceLoader or via ModuleLifecycle.

There's also the question of whether we should support passing in a
marshaller instance instead of a class name (I think we should, at
least for testing) and whether we should allow custom configuration
attributes/elements for the marshaller in the XML configuration (no
idea).

Cheers
Dan



On Tue, Mar 14, 2017 at 1:26 PM, Ryan Emerson <remerson at redhat.com> wrote:
> Hi All,
>
> Currently the CacheWriterInterceptor utilises the internal marshaller for marshalling entries before they are sent to the configured cache stores. This causes several problems [1], most notably that changes to the internal marshaller make stored data incompatible across Infinispan versions.
>
> I propose that we decouple the internal and store marshaller. To allow 9.x versions to remain compatible, we should default to the internal marshaller (until 10.x at least), but optionally allow users to specify a custom StreamingMarshaller implementation as part of their PersistenceConfiguration. As we already have the protostuff and kryo bridges, users would specify an enum for the marshaller they want to use as well as an optional class string if a custom implementation is required. So for example:
>
> enum StoreMarshaller {
>    CUSTOM,
>    INTERNAL,
>    KRYO
>    PROTOSTUFF;
> }
>
> new ConfigurationBuilder()
>     .persistence()
>         .marshaller(StoreMarshaller.CUSTOM)
>         .marshallerClass("org.example.Marshaller")
>          ...
>
> Finally, Gustavo brought flatbuffers[2] to my attention which could be a good option to provide for users as one of the default StoreMarshaller implementations.
>
> WDYT?
>
> Cheers
> Ryan
>
> [1] https://docs.google.com/document/d/1PR0eYgjhqXUR5w03npS7TdOs2KDZjfdMh0au8XAsyJY
> [2] https://google.github.io/flatbuffers/
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev


From dan.berindei at gmail.com  Tue Mar 14 09:53:55 2017
From: dan.berindei at gmail.com (Dan Berindei)
Date: Tue, 14 Mar 2017 15:53:55 +0200
Subject: [infinispan-dev] Infinispan "mitosis"
In-Reply-To: <CAFm4XO2_79wbm+SkFkzOneb8-3wJsH1LxNs8oOgb1_X-u4sj0w@mail.gmail.com>
References: <CAFm4XO2_79wbm+SkFkzOneb8-3wJsH1LxNs8oOgb1_X-u4sj0w@mail.gmail.com>
Message-ID: <CA+nfvwRAApNvzOx1DO_9W-Wo16Ngn2Rzg1U2-wXWwOYFSpunsg@mail.gmail.com>

Isn't this a lot like x-site state transfer?

Dan

On Tue, Mar 14, 2017 at 3:08 PM, Sanne Grinovero <sanne at infinispan.org> wrote:
> Just throwing out an idea I just had while thinking of Hibernate OGM
> user needs for data migration.
>
> For people using databases & related frameworks, it's common to have a
> staging database which contains not just the staging "schema" but also
> data. When legally possible, it's often preferable to have snapshot
> from production data.
>
> For example last time I worked with a SQL database, each week I'd take
> a production backup and restore it on both our staging environment and
> on the developer's instances so that everyone could run tests on it -
> without needing access to the real production.
>
> Interestingly, while we don't have an easy tool get a fully consistent
> snapshot from a live Infinispan grid, "replicating" should be a
> familiar concept here?
>
> Infinispan could have a "mitosis" feature, like cell reproduction, in
> which the user connects a pristine grid instance of N nodes and these
> N nodes automatically become non-primary owners for the full set of
> segments - this could happen via a custom hash which makes them all
> backup replicas (no main owners) and then Infinispan would be able to
> tell when state transfer is completed, and initiate some some
> coordination to sever the link without triggering re-hash on the
> original "production" grid.
>
> Incidentally, while this happens the child datagrid would be kept up
> to date with in-flight changes, so we could envision either a very
> short lock on changes to guarantee a fully consistent snapshot, or not
> have any lock at all but minimise the inconsistencies to those which
> might happen during the link decoupling.
>
> Maybe this would satisfy also the people who've been asking for the
> snapshot feature? I don't think people want a snapshot for the sake of
> it, but to replicate the grid..
>
> I realise it's not a 1 day of work and the idea is not fully fleshed
> out, but I think this would be a very well received feature.
>
>
> Thanks,
> Sanne
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev

From sanne at infinispan.org  Tue Mar 14 10:06:48 2017
From: sanne at infinispan.org (Sanne Grinovero)
Date: Tue, 14 Mar 2017 14:06:48 +0000
Subject: [infinispan-dev] Infinispan "mitosis"
In-Reply-To: <CA+nfvwRAApNvzOx1DO_9W-Wo16Ngn2Rzg1U2-wXWwOYFSpunsg@mail.gmail.com>
References: <CAFm4XO2_79wbm+SkFkzOneb8-3wJsH1LxNs8oOgb1_X-u4sj0w@mail.gmail.com>
	<CA+nfvwRAApNvzOx1DO_9W-Wo16Ngn2Rzg1U2-wXWwOYFSpunsg@mail.gmail.com>
Message-ID: <CAFm4XO3JNdDGSqLbNNpd-jxuwp_et9KrqVoHKZ1OMXiD1m22wQ@mail.gmail.com>

On 14 March 2017 at 13:53, Dan Berindei <dan.berindei at gmail.com> wrote:
> Isn't this a lot like x-site state transfer?

Indeed! Does it mean we could have such a feature with less work?

Maybe some kind of "x-site for dummies" quickstart would be enough,
but re-framing the use case.

Thanks,
Sanne

>
> Dan
>
> On Tue, Mar 14, 2017 at 3:08 PM, Sanne Grinovero <sanne at infinispan.org> wrote:
>> Just throwing out an idea I just had while thinking of Hibernate OGM
>> user needs for data migration.
>>
>> For people using databases & related frameworks, it's common to have a
>> staging database which contains not just the staging "schema" but also
>> data. When legally possible, it's often preferable to have snapshot
>> from production data.
>>
>> For example last time I worked with a SQL database, each week I'd take
>> a production backup and restore it on both our staging environment and
>> on the developer's instances so that everyone could run tests on it -
>> without needing access to the real production.
>>
>> Interestingly, while we don't have an easy tool get a fully consistent
>> snapshot from a live Infinispan grid, "replicating" should be a
>> familiar concept here?
>>
>> Infinispan could have a "mitosis" feature, like cell reproduction, in
>> which the user connects a pristine grid instance of N nodes and these
>> N nodes automatically become non-primary owners for the full set of
>> segments - this could happen via a custom hash which makes them all
>> backup replicas (no main owners) and then Infinispan would be able to
>> tell when state transfer is completed, and initiate some some
>> coordination to sever the link without triggering re-hash on the
>> original "production" grid.
>>
>> Incidentally, while this happens the child datagrid would be kept up
>> to date with in-flight changes, so we could envision either a very
>> short lock on changes to guarantee a fully consistent snapshot, or not
>> have any lock at all but minimise the inconsistencies to those which
>> might happen during the link decoupling.
>>
>> Maybe this would satisfy also the people who've been asking for the
>> snapshot feature? I don't think people want a snapshot for the sake of
>> it, but to replicate the grid..
>>
>> I realise it's not a 1 day of work and the idea is not fully fleshed
>> out, but I think this would be a very well received feature.
>>
>>
>> Thanks,
>> Sanne
>> _______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev

From gustavo at infinispan.org  Tue Mar 14 10:10:50 2017
From: gustavo at infinispan.org (Gustavo Fernandes)
Date: Tue, 14 Mar 2017 14:10:50 +0000
Subject: [infinispan-dev] Infinispan "mitosis"
In-Reply-To: <CAFm4XO2_79wbm+SkFkzOneb8-3wJsH1LxNs8oOgb1_X-u4sj0w@mail.gmail.com>
References: <CAFm4XO2_79wbm+SkFkzOneb8-3wJsH1LxNs8oOgb1_X-u4sj0w@mail.gmail.com>
Message-ID: <CAH8Ud1EZw=Thqc1gF1udcM5f78dt4gq9eFM-LzFs4NP__gfHrQ@mail.gmail.com>

On Tue, Mar 14, 2017 at 1:08 PM, Sanne Grinovero <sanne at infinispan.org>
wrote:

> Just throwing out an idea I just had while thinking of Hibernate OGM
> user needs for data migration.
>
> For people using databases & related frameworks, it's common to have a
> staging database which contains not just the staging "schema" but also
> data. When legally possible, it's often preferable to have snapshot
> from production data.
>
> For example last time I worked with a SQL database, each week I'd take
> a production backup and restore it on both our staging environment and
> on the developer's instances so that everyone could run tests on it -
> without needing access to the real production.
>
> Interestingly, while we don't have an easy tool get a fully consistent
> snapshot from a live Infinispan grid, "replicating" should be a
> familiar concept here?
>

We do have a way to "copy" data from a grid, which is part of the Rolling
Upgrade process [1], which is
supposed to migrate data from one cluster to another with no downtime for
clients. Both clusters
are independent and can have incompatible Infinispan version, like 8.x and
9.x.

Re-using some parts of the Rolling Upgrade from [1], it'd be possible to
extract a tool that simply grabs data from a
running cluster and save it to another without the clients accessing the
source cluster being aware of it. Although
not technically a snapshot, it fits the use case of pre-production
environment provisioning.

[1]
http://infinispan.org/docs/dev/user_guide/user_guide.html#rolling_upgrades_for_infinispan_servers


>
> Infinispan could have a "mitosis" feature, like cell reproduction, in
> which the user connects a pristine grid instance of N nodes and these
> N nodes automatically become non-primary owners for the full set of
> segments - this could happen via a custom hash which makes them all
> backup replicas (no main owners) and then Infinispan would be able to
> tell when state transfer is completed, and initiate some some
> coordination to sever the link without triggering re-hash on the
> original "production" grid.
>
> Incidentally, while this happens the child datagrid would be kept up
> to date with in-flight changes, so we could envision either a very
> short lock on changes to guarantee a fully consistent snapshot, or not
> have any lock at all but minimise the inconsistencies to those which
> might happen during the link decoupling.
>
> Maybe this would satisfy also the people who've been asking for the
> snapshot feature? I don't think people want a snapshot for the sake of
> it, but to replicate the grid..
>
> I realise it's not a 1 day of work and the idea is not fully fleshed
> out, but I think this would be a very well received feature.
>
>
> Thanks,
> Sanne
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/infinispan-dev/attachments/20170314/51ce8884/attachment.html 

From dan.berindei at gmail.com  Tue Mar 14 10:59:10 2017
From: dan.berindei at gmail.com (Dan Berindei)
Date: Tue, 14 Mar 2017 16:59:10 +0200
Subject: [infinispan-dev] Infinispan "mitosis"
In-Reply-To: <CAH8Ud1EZw=Thqc1gF1udcM5f78dt4gq9eFM-LzFs4NP__gfHrQ@mail.gmail.com>
References: <CAFm4XO2_79wbm+SkFkzOneb8-3wJsH1LxNs8oOgb1_X-u4sj0w@mail.gmail.com>
	<CAH8Ud1EZw=Thqc1gF1udcM5f78dt4gq9eFM-LzFs4NP__gfHrQ@mail.gmail.com>
Message-ID: <CA+nfvwQTMYXXOYfV2hzunk-RWUOcPdthNfA6gvPpXUXA-8iO9g@mail.gmail.com>

Yes, rolling upgrades should be even better, because for x-site you
need the source cluster to have a backup site configured (but not
enabled), and adding that to a running cluster may require a
one-by-one restart of all the nodes.

Dan

On Tue, Mar 14, 2017 at 4:10 PM, Gustavo Fernandes
<gustavo at infinispan.org> wrote:
>
>
> On Tue, Mar 14, 2017 at 1:08 PM, Sanne Grinovero <sanne at infinispan.org>
> wrote:
>>
>> Just throwing out an idea I just had while thinking of Hibernate OGM
>> user needs for data migration.
>>
>> For people using databases & related frameworks, it's common to have a
>> staging database which contains not just the staging "schema" but also
>> data. When legally possible, it's often preferable to have snapshot
>> from production data.
>>
>> For example last time I worked with a SQL database, each week I'd take
>> a production backup and restore it on both our staging environment and
>> on the developer's instances so that everyone could run tests on it -
>> without needing access to the real production.
>>
>> Interestingly, while we don't have an easy tool get a fully consistent
>> snapshot from a live Infinispan grid, "replicating" should be a
>> familiar concept here?
>
>
> We do have a way to "copy" data from a grid, which is part of the Rolling
> Upgrade process [1], which is
> supposed to migrate data from one cluster to another with no downtime for
> clients. Both clusters
> are independent and can have incompatible Infinispan version, like 8.x and
> 9.x.
>
> Re-using some parts of the Rolling Upgrade from [1], it'd be possible to
> extract a tool that simply grabs data from a
> running cluster and save it to another without the clients accessing the
> source cluster being aware of it. Although
> not technically a snapshot, it fits the use case of pre-production
> environment provisioning.
>
> [1]
> http://infinispan.org/docs/dev/user_guide/user_guide.html#rolling_upgrades_for_infinispan_servers
>
>>
>>
>> Infinispan could have a "mitosis" feature, like cell reproduction, in
>> which the user connects a pristine grid instance of N nodes and these
>> N nodes automatically become non-primary owners for the full set of
>> segments - this could happen via a custom hash which makes them all
>> backup replicas (no main owners) and then Infinispan would be able to
>> tell when state transfer is completed, and initiate some some
>> coordination to sever the link without triggering re-hash on the
>> original "production" grid.
>>
>> Incidentally, while this happens the child datagrid would be kept up
>> to date with in-flight changes, so we could envision either a very
>> short lock on changes to guarantee a fully consistent snapshot, or not
>> have any lock at all but minimise the inconsistencies to those which
>> might happen during the link decoupling.
>>
>> Maybe this would satisfy also the people who've been asking for the
>> snapshot feature? I don't think people want a snapshot for the sake of
>> it, but to replicate the grid..
>>
>> I realise it's not a 1 day of work and the idea is not fully fleshed
>> out, but I think this would be a very well received feature.
>>
>>
>> Thanks,
>> Sanne
>> _______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
>
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev

From remerson at redhat.com  Wed Mar 15 05:59:33 2017
From: remerson at redhat.com (Ryan Emerson)
Date: Wed, 15 Mar 2017 05:59:33 -0400 (EDT)
Subject: [infinispan-dev] Cache Store Marshalling
In-Reply-To: <CA+nfvwRFA2d8rbxpYY3cM1BZp5Wymcyj7KQN+UGxkVkAWKkg4A@mail.gmail.com>
References: <1289292546.2960092.1489488818067.JavaMail.zimbra@redhat.com>
	<4686643.2964817.1489490786403.JavaMail.zimbra@redhat.com>
	<CA+nfvwRFA2d8rbxpYY3cM1BZp5Wymcyj7KQN+UGxkVkAWKkg4A@mail.gmail.com>
Message-ID: <236690077.3383786.1489571973479.JavaMail.zimbra@redhat.com>

Comments inline.

----- Original Message -----
> +100 to use a separate marshaller for user types, but I think the
> persistence configuration is the wrong place for it. Users shouldn't
> have to implement an AdvancedExternalizer for efficient replication
> between nodes if they already have a marshaller that's efficient
> enough for use in a store.

When I sent the original email, I was thinking that we would allow users to specify a marshaller that was specific to persistence, in the same way that we allow users to configure a compatibility marshaller. This way users could utilise a serialisation library better suited to their performance needs, i.e. faster deserializaton vs disk space etc. But I don't have a specific use case in mind, so this might be overkill.

Regarding a generic user marshaller, Galder has already created ISPN-7409 [1]. Galder, do you intend to work on this/have a specific release in mind? I noticed in the planning doc that it hasn't been marked for any release. 

[1] https://issues.jboss.org/browse/ISPN-7409

> I would have liked the default user type marshaller to use
> jboss-marshalling with the all the 8.2.x internal externalizers, in
> order to have backwards compatibility for the stores without needing
> an upgrade tool. But it seems like the stores change their layout all
> the time anyway, so I would settle for some version metadata in the
> store and better error messages instead.

+1 to adding some version metadata.

> I'm not sure about the enum, don't like the feeling that marshallers
> not bundled with the core are somehow second-class. I'd rather have a
> single marshaller name string attribute, have a NAME constant in each
> marshaller class, and register the names on startup, either via
> ServiceLoader or via ModuleLifecycle.
> 
> There's also the question of whether we should support passing in a
> marshaller instance instead of a class name (I think we should, at
> least for testing) and whether we should allow custom configuration
> attributes/elements for the marshaller in the XML configuration (no
> idea).

All sounds good.

> Cheers
> Dan
> 
> 
> 
> On Tue, Mar 14, 2017 at 1:26 PM, Ryan Emerson <remerson at redhat.com> wrote:
> > Hi All,
> >
> > Currently the CacheWriterInterceptor utilises the internal marshaller for
> > marshalling entries before they are sent to the configured cache stores.
> > This causes several problems [1], most notably that changes to the
> > internal marshaller make stored data incompatible across Infinispan
> > versions.
> >
> > I propose that we decouple the internal and store marshaller. To allow 9.x
> > versions to remain compatible, we should default to the internal
> > marshaller (until 10.x at least), but optionally allow users to specify a
> > custom StreamingMarshaller implementation as part of their
> > PersistenceConfiguration. As we already have the protostuff and kryo
> > bridges, users would specify an enum for the marshaller they want to use
> > as well as an optional class string if a custom implementation is
> > required. So for example:
> >
> > enum StoreMarshaller {
> >    CUSTOM,
> >    INTERNAL,
> >    KRYO
> >    PROTOSTUFF;
> > }
> >
> > new ConfigurationBuilder()
> >     .persistence()
> >         .marshaller(StoreMarshaller.CUSTOM)
> >         .marshallerClass("org.example.Marshaller")
> >          ...
> >
> > Finally, Gustavo brought flatbuffers[2] to my attention which could be a
> > good option to provide for users as one of the default StoreMarshaller
> > implementations.
> >
> > WDYT?
> >
> > Cheers
> > Ryan
> >
> > [1]
> > https://docs.google.com/document/d/1PR0eYgjhqXUR5w03npS7TdOs2KDZjfdMh0au8XAsyJY
> > [2] https://google.github.io/flatbuffers/
> > _______________________________________________
> > infinispan-dev mailing list
> > infinispan-dev at lists.jboss.org
> > https://lists.jboss.org/mailman/listinfo/infinispan-dev
> 
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> 

From slaskawi at redhat.com  Thu Mar 16 02:15:46 2017
From: slaskawi at redhat.com (Sebastian Laskawiec)
Date: Thu, 16 Mar 2017 06:15:46 +0000
Subject: [infinispan-dev] KUBE_PING 0.9.2 and Infinispan Spring Boot
	Starters 1.0.0.CR1 released
Message-ID: <CAA9R9O7Vq0sWTR_JY-_MsQMDctudVzvxV0GTd_5_Qohtp+f70w@mail.gmail.com>

Hey!

We recently release two components - KUBE_PING [1] and Spring Boot Starters
[2]. Check out our blog posts!

Cheers,
Sebastian

[1] http://blog.infinispan.org/2017/03/kubeping-092-released.html
[2]
http://blog.infinispan.org/2017/03/spring-boot-starter-100cr1-released.html
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/infinispan-dev/attachments/20170316/12e934dc/attachment-0001.html 

From slaskawi at redhat.com  Mon Mar 20 04:48:08 2017
From: slaskawi at redhat.com (Sebastian Laskawiec)
Date: Mon, 20 Mar 2017 08:48:08 +0000
Subject: [infinispan-dev] Docker CPU and memory autoscaling
Message-ID: <CAA9R9O7Wjw3Bk0-m4mbLBJet5Un5rhm5EsPhQSzpydzbmawLqQ@mail.gmail.com>

Dear community!

Recently we introduced a new functionality for our Docker image - CPU and
memory autoscaling.

For more information please have a look at our blog post:
https://goo.gl/yQXvFy

Thanks,
Sebastian
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/infinispan-dev/attachments/20170320/5d2912de/attachment.html 

From galder at redhat.com  Mon Mar 20 06:36:05 2017
From: galder at redhat.com (=?utf-8?Q?Galder_Zamarre=C3=B1o?=)
Date: Mon, 20 Mar 2017 11:36:05 +0100
Subject: [infinispan-dev] Cache Store Marshalling
In-Reply-To: <4686643.2964817.1489490786403.JavaMail.zimbra@redhat.com>
References: <4686643.2964817.1489490786403.JavaMail.zimbra@redhat.com>
Message-ID: <5B049C51-6C8B-463C-A1FE-C9D0CB67E0D7@redhat.com>

Hmmm, we've already discussed moving from the current internal/external marshaller architecture, where even user types are marshalled with internal marshaller, to an internal/user marshaller architecture where user types are only marshalled by the user marshaller. This is mentioned in [1] and the switch to internal/user marshaller is already being tracked in [3].

Your suggestion feels like a subset of what [3] is trying to achieve. In [3], the idea is not only to use the user marshaller for clustering but of course, for persistence.

Cheers,

[3] https://issues.jboss.org/browse/ISPN-7409
--
Galder Zamarre?o
Infinispan, Red Hat

> On 14 Mar 2017, at 12:26, Ryan Emerson <remerson at redhat.com> wrote:
> 
> Hi All,
> 
> Currently the CacheWriterInterceptor utilises the internal marshaller for marshalling entries before they are sent to the configured cache stores. This causes several problems [1], most notably that changes to the internal marshaller make stored data incompatible across Infinispan versions. 
> 
> I propose that we decouple the internal and store marshaller. To allow 9.x versions to remain compatible, we should default to the internal marshaller (until 10.x at least), but optionally allow users to specify a custom StreamingMarshaller implementation as part of their PersistenceConfiguration. As we already have the protostuff and kryo bridges, users would specify an enum for the marshaller they want to use as well as an optional class string if a custom implementation is required. So for example:
> 
> enum StoreMarshaller {
>   CUSTOM,
>   INTERNAL,
>   KRYO
>   PROTOSTUFF;
> }
> 
> new ConfigurationBuilder()
>    .persistence()
>        .marshaller(StoreMarshaller.CUSTOM)
>        .marshallerClass("org.example.Marshaller")
>         ...
> 
> Finally, Gustavo brought flatbuffers[2] to my attention which could be a good option to provide for users as one of the default StoreMarshaller implementations.
> 
> WDYT?
> 
> Cheers
> Ryan
> 
> [1] https://docs.google.com/document/d/1PR0eYgjhqXUR5w03npS7TdOs2KDZjfdMh0au8XAsyJY
> [2] https://google.github.io/flatbuffers/
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev



From galder at redhat.com  Mon Mar 20 06:43:34 2017
From: galder at redhat.com (=?utf-8?Q?Galder_Zamarre=C3=B1o?=)
Date: Mon, 20 Mar 2017 11:43:34 +0100
Subject: [infinispan-dev] Cache Store Marshalling
In-Reply-To: <236690077.3383786.1489571973479.JavaMail.zimbra@redhat.com>
References: <1289292546.2960092.1489488818067.JavaMail.zimbra@redhat.com>
	<4686643.2964817.1489490786403.JavaMail.zimbra@redhat.com>
	<CA+nfvwRFA2d8rbxpYY3cM1BZp5Wymcyj7KQN+UGxkVkAWKkg4A@mail.gmail.com>
	<236690077.3383786.1489571973479.JavaMail.zimbra@redhat.com>
Message-ID: <66DA2BC0-2B46-4A4F-BA4C-55A6723259D4@redhat.com>

Inline:

> On 15 Mar 2017, at 10:59, Ryan Emerson <remerson at redhat.com> wrote:
> 
> Comments inline.
> 
> ----- Original Message -----
>> +100 to use a separate marshaller for user types, but I think the
>> persistence configuration is the wrong place for it. Users shouldn't
>> have to implement an AdvancedExternalizer for efficient replication
>> between nodes if they already have a marshaller that's efficient
>> enough for use in a store.
> 
> When I sent the original email, I was thinking that we would allow users to specify a marshaller that was specific to persistence, in the same way that we allow users to configure a compatibility marshaller. This way users could utilise a serialisation library better suited to their performance needs, i.e. faster deserializaton vs disk space etc. But I don't have a specific use case in mind, so this might be overkill.

Compatibility mode (and compatibility marshaller stuff) will eventually go in favour of transcoding, so I would not consider it a good way of doing things ;)

> 
> Regarding a generic user marshaller, Galder has already created ISPN-7409 [1]. Galder, do you intend to work on this/have a specific release in mind? I noticed in the planning doc that it hasn't been marked for any release. 

No, don't have any release in mind right now and I'm not currently working on it. So, feel free to look into it :)

Cheers,

> 
> [1] https://issues.jboss.org/browse/ISPN-7409
> 
>> I would have liked the default user type marshaller to use
>> jboss-marshalling with the all the 8.2.x internal externalizers, in
>> order to have backwards compatibility for the stores without needing
>> an upgrade tool. But it seems like the stores change their layout all
>> the time anyway, so I would settle for some version metadata in the
>> store and better error messages instead.
> 
> +1 to adding some version metadata.
> 
>> I'm not sure about the enum, don't like the feeling that marshallers
>> not bundled with the core are somehow second-class. I'd rather have a
>> single marshaller name string attribute, have a NAME constant in each
>> marshaller class, and register the names on startup, either via
>> ServiceLoader or via ModuleLifecycle.
>> 
>> There's also the question of whether we should support passing in a
>> marshaller instance instead of a class name (I think we should, at
>> least for testing) and whether we should allow custom configuration
>> attributes/elements for the marshaller in the XML configuration (no
>> idea).
> 
> All sounds good.
> 
>> Cheers
>> Dan
>> 
>> 
>> 
>> On Tue, Mar 14, 2017 at 1:26 PM, Ryan Emerson <remerson at redhat.com> wrote:
>>> Hi All,
>>> 
>>> Currently the CacheWriterInterceptor utilises the internal marshaller for
>>> marshalling entries before they are sent to the configured cache stores.
>>> This causes several problems [1], most notably that changes to the
>>> internal marshaller make stored data incompatible across Infinispan
>>> versions.
>>> 
>>> I propose that we decouple the internal and store marshaller. To allow 9.x
>>> versions to remain compatible, we should default to the internal
>>> marshaller (until 10.x at least), but optionally allow users to specify a
>>> custom StreamingMarshaller implementation as part of their
>>> PersistenceConfiguration. As we already have the protostuff and kryo
>>> bridges, users would specify an enum for the marshaller they want to use
>>> as well as an optional class string if a custom implementation is
>>> required. So for example:
>>> 
>>> enum StoreMarshaller {
>>>   CUSTOM,
>>>   INTERNAL,
>>>   KRYO
>>>   PROTOSTUFF;
>>> }
>>> 
>>> new ConfigurationBuilder()
>>>    .persistence()
>>>        .marshaller(StoreMarshaller.CUSTOM)
>>>        .marshallerClass("org.example.Marshaller")
>>>         ...
>>> 
>>> Finally, Gustavo brought flatbuffers[2] to my attention which could be a
>>> good option to provide for users as one of the default StoreMarshaller
>>> implementations.
>>> 
>>> WDYT?
>>> 
>>> Cheers
>>> Ryan
>>> 
>>> [1]
>>> https://docs.google.com/document/d/1PR0eYgjhqXUR5w03npS7TdOs2KDZjfdMh0au8XAsyJY
>>> [2] https://google.github.io/flatbuffers/
>>> _______________________________________________
>>> infinispan-dev mailing list
>>> infinispan-dev at lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>> 
>> _______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>> 
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev



From rareddy at redhat.com  Mon Mar 20 09:29:18 2017
From: rareddy at redhat.com (Ramesh Reddy)
Date: Mon, 20 Mar 2017 09:29:18 -0400 (EDT)
Subject: [infinispan-dev] Simplest way to check the validity of connection
	to Remote Cache
In-Reply-To: <92895211.4077569.1490016241640.JavaMail.zimbra@redhat.com>
Message-ID: <2098058988.4079799.1490016558856.JavaMail.zimbra@redhat.com>

Hi,

Is there call I can make on the cache API like ping to check the validity of the remote connection? In OpenShift JDV is having issues with keeping the connections fresh to JDG when node count goes to zero and comes back up.

Thank you.

Ramesh..

From ttarrant at redhat.com  Mon Mar 20 09:36:31 2017
From: ttarrant at redhat.com (Tristan Tarrant)
Date: Mon, 20 Mar 2017 14:36:31 +0100
Subject: [infinispan-dev] Simplest way to check the validity of
 connection to Remote Cache
In-Reply-To: <2098058988.4079799.1490016558856.JavaMail.zimbra@redhat.com>
References: <2098058988.4079799.1490016558856.JavaMail.zimbra@redhat.com>
Message-ID: <c9d95a18-7aaa-e02d-b8c0-0ea6722c720f@redhat.com>

How are you configuring the connection pool ?

Tristan

On 20/03/17 14:29, Ramesh Reddy wrote:
> Hi,
> 
> Is there call I can make on the cache API like ping to check the validity of the remote connection? In OpenShift JDV is having issues with keeping the connections fresh to JDG when node count goes to zero and comes back up.
> 
> Thank you.
> 
> Ramesh..
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> 

-- 
Tristan Tarrant
Infinispan Lead
JBoss, a division of Red Hat

From rareddy at redhat.com  Mon Mar 20 09:56:54 2017
From: rareddy at redhat.com (Ramesh Reddy)
Date: Mon, 20 Mar 2017 09:56:54 -0400 (EDT)
Subject: [infinispan-dev] Simplest way to check the validity of
 connection to Remote Cache
In-Reply-To: <c9d95a18-7aaa-e02d-b8c0-0ea6722c720f@redhat.com>
References: <2098058988.4079799.1490016558856.JavaMail.zimbra@redhat.com>
	<c9d95a18-7aaa-e02d-b8c0-0ea6722c720f@redhat.com>
Message-ID: <1100376924.4090968.1490018214201.JavaMail.zimbra@redhat.com>

In Teiid, we wrote a resource adapter around the connection semantics to work in WildFly/EAP.

----- Original Message -----
> How are you configuring the connection pool ?
> 
> Tristan
> 
> On 20/03/17 14:29, Ramesh Reddy wrote:
> > Hi,
> > 
> > Is there call I can make on the cache API like ping to check the validity
> > of the remote connection? In OpenShift JDV is having issues with keeping
> > the connections fresh to JDG when node count goes to zero and comes back
> > up.
> > 
> > Thank you.
> > 
> > Ramesh..
> > _______________________________________________
> > infinispan-dev mailing list
> > infinispan-dev at lists.jboss.org
> > https://lists.jboss.org/mailman/listinfo/infinispan-dev
> > 
> 
> --
> Tristan Tarrant
> Infinispan Lead
> JBoss, a division of Red Hat
> 

From slaskawi at redhat.com  Mon Mar 20 10:16:21 2017
From: slaskawi at redhat.com (Sebastian Laskawiec)
Date: Mon, 20 Mar 2017 14:16:21 +0000
Subject: [infinispan-dev] Simplest way to check the validity of
 connection to Remote Cache
In-Reply-To: <1100376924.4090968.1490018214201.JavaMail.zimbra@redhat.com>
References: <2098058988.4079799.1490016558856.JavaMail.zimbra@redhat.com>
	<c9d95a18-7aaa-e02d-b8c0-0ea6722c720f@redhat.com>
	<1100376924.4090968.1490018214201.JavaMail.zimbra@redhat.com>
Message-ID: <CAA9R9O4T=q0s5cSTdd6iTVA_4vGJSUgP6K8P-pr_1qUtkZOnBA@mail.gmail.com>

Hey Ramesh!

I think I had similar problems to yours when testing Rolling Upgrades in
Kubernetes. I created https://issues.jboss.org/browse/ISPN-7515 to fix it.

The problem is that the RemoteStore's connection pool seems have problems
with initializing/retrying and refreshing connections.

Thanks,
Sebastian

On Mon, Mar 20, 2017 at 3:11 PM Ramesh Reddy <rareddy at redhat.com> wrote:

> In Teiid, we wrote a resource adapter around the connection semantics to
> work in WildFly/EAP.
>
> ----- Original Message -----
> > How are you configuring the connection pool ?
> >
> > Tristan
> >
> > On 20/03/17 14:29, Ramesh Reddy wrote:
> > > Hi,
> > >
> > > Is there call I can make on the cache API like ping to check the
> validity
> > > of the remote connection? In OpenShift JDV is having issues with
> keeping
> > > the connections fresh to JDG when node count goes to zero and comes
> back
> > > up.
> > >
> > > Thank you.
> > >
> > > Ramesh..
> > > _______________________________________________
> > > infinispan-dev mailing list
> > > infinispan-dev at lists.jboss.org
> > > https://lists.jboss.org/mailman/listinfo/infinispan-dev
> > >
> >
> > --
> > Tristan Tarrant
> > Infinispan Lead
> > JBoss, a division of Red Hat
> >
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/infinispan-dev/attachments/20170320/ce84f747/attachment-0001.html 

From ttarrant at redhat.com  Mon Mar 20 13:16:45 2017
From: ttarrant at redhat.com (Tristan Tarrant)
Date: Mon, 20 Mar 2017 18:16:45 +0100
Subject: [infinispan-dev] 9.0.0.CR4
Message-ID: <d74297e8-03f3-7701-72fb-cf8c8e25c1e4@redhat.com>

Guys,

because of the amount of pending changes, I'd like to release 9.0.0.CR4 
on Wednesday 22nd and Final next week.

Opinions ?

Tristan

-- 
Tristan Tarrant
Infinispan Lead
JBoss, a division of Red Hat

From anistor at redhat.com  Mon Mar 20 13:45:13 2017
From: anistor at redhat.com (Adrian Nistor)
Date: Mon, 20 Mar 2017 19:45:13 +0200
Subject: [infinispan-dev] My update
Message-ID: <e32cd7da-12f8-6332-744f-d88b6b4df2ea@redhat.com>

Hi all,

I've been on PTO most of last week and returned to active life only on 
Thursday. I've been hiding in the shadow exercising my writing talent 
(yuck) on documenting query and also cleaned up some jira issues:

ISPN-7300 fixing negative occurence (-) when used with boolean expressions

ISPN-6713 RemoteQueryDslConditionsTest.testIsNullNumericWithProjection1 
no longer fails after Lucene upgrade

ISPN-7002 NPE in DelegatingQuery when deployed in Wildfly 10.1.0.Final

I also spent some time with ISPN-7580, which is a proposal from Teiid 
for modifying Protostream in order to allow dynamic entities, much like 
what OGM needs too. I like the idea and understand the need for this but 
so far the solution is a bit hacky so I'm reluctant to merge that change.

Adrian


From mudokonman at gmail.com  Mon Mar 20 14:16:56 2017
From: mudokonman at gmail.com (William Burns)
Date: Mon, 20 Mar 2017 18:16:56 +0000
Subject: [infinispan-dev] 9.0.0.CR4
In-Reply-To: <d74297e8-03f3-7701-72fb-cf8c8e25c1e4@redhat.com>
References: <d74297e8-03f3-7701-72fb-cf8c8e25c1e4@redhat.com>
Message-ID: <CA+YCuUOZNBTHyda2_f07HbC8TUYu0F11qt3YTzf5G8ctJsrA+g@mail.gmail.com>

Sounds good! Hopefully this will actually be the last CR :D

On Mon, Mar 20, 2017 at 1:27 PM Tristan Tarrant <ttarrant at redhat.com> wrote:

> Guys,
>
> because of the amount of pending changes, I'd like to release 9.0.0.CR4
> on Wednesday 22nd and Final next week.
>
> Opinions ?
>
> Tristan
>
> --
> Tristan Tarrant
> Infinispan Lead
> JBoss, a division of Red Hat
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/infinispan-dev/attachments/20170320/463e1a3c/attachment.html 

From sanne at infinispan.org  Mon Mar 20 15:08:42 2017
From: sanne at infinispan.org (Sanne Grinovero)
Date: Mon, 20 Mar 2017 19:08:42 +0000
Subject: [infinispan-dev] AffinityPartitioner: enable by default please?
Message-ID: <CAFm4XO3OXa0dscuK75eUfxgRVcgHPwm_36SB2WbYKDNPwNsKow@mail.gmail.com>

Hi all,

the explicit approach which we introduced with the AffinityPartitioner
is extremely useful in most use cases for Embedded mode.

I sent a PR on 18 Aug 2016 [1] to make this feature enabled by default
so that people could benefit from it w/o needing to set it explicitly
but it was not merged on the basis that it wouldn't be trivial to
implement the same for Remote caches.

Could we reconsider this please? I agree it's not easy to implement
this same feature for remote, but this feature is not as important to
have in remote mode, while its value is very high in embedded mode.
I think we could agree that this first step is both very useful and
harmless for embedded mode, so let's please merge it?

Thanks,
Sanne

[1] - https://github.com/infinispan/infinispan/pull/4521

From dan.berindei at gmail.com  Mon Mar 20 15:16:34 2017
From: dan.berindei at gmail.com (Dan Berindei)
Date: Mon, 20 Mar 2017 21:16:34 +0200
Subject: [infinispan-dev] 9.0.0.CR4
In-Reply-To: <CA+YCuUOZNBTHyda2_f07HbC8TUYu0F11qt3YTzf5G8ctJsrA+g@mail.gmail.com>
References: <d74297e8-03f3-7701-72fb-cf8c8e25c1e4@redhat.com>
	<CA+YCuUOZNBTHyda2_f07HbC8TUYu0F11qt3YTzf5G8ctJsrA+g@mail.gmail.com>
Message-ID: <CA+nfvwTb7WHnZrtddgcaJcHssvMyzRrHfD58TAbOu2au+fS9nA@mail.gmail.com>

+1

On Mon, Mar 20, 2017 at 8:16 PM, William Burns <mudokonman at gmail.com> wrote:
> Sounds good! Hopefully this will actually be the last CR :D
>
> On Mon, Mar 20, 2017 at 1:27 PM Tristan Tarrant <ttarrant at redhat.com> wrote:
>>
>> Guys,
>>
>> because of the amount of pending changes, I'd like to release 9.0.0.CR4
>> on Wednesday 22nd and Final next week.
>>
>> Opinions ?
>>
>> Tristan
>>
>> --
>> Tristan Tarrant
>> Infinispan Lead
>> JBoss, a division of Red Hat
>> _______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev

From slaskawi at redhat.com  Tue Mar 21 02:37:51 2017
From: slaskawi at redhat.com (Sebastian Laskawiec)
Date: Tue, 21 Mar 2017 06:37:51 +0000
Subject: [infinispan-dev] 9.0.0.CR4
In-Reply-To: <CA+nfvwTb7WHnZrtddgcaJcHssvMyzRrHfD58TAbOu2au+fS9nA@mail.gmail.com>
References: <d74297e8-03f3-7701-72fb-cf8c8e25c1e4@redhat.com>
	<CA+YCuUOZNBTHyda2_f07HbC8TUYu0F11qt3YTzf5G8ctJsrA+g@mail.gmail.com>
	<CA+nfvwTb7WHnZrtddgcaJcHssvMyzRrHfD58TAbOu2au+fS9nA@mail.gmail.com>
Message-ID: <CAA9R9O6_p5aorFsf39Lx9-cQuwW1MHjpzKUAQ92LXut=nxM37g@mail.gmail.com>

Sounds good! Fingers crossed!

On Mon, Mar 20, 2017 at 9:16 PM Dan Berindei <dan.berindei at gmail.com> wrote:

> +1
>
> On Mon, Mar 20, 2017 at 8:16 PM, William Burns <mudokonman at gmail.com>
> wrote:
> > Sounds good! Hopefully this will actually be the last CR :D
> >
> > On Mon, Mar 20, 2017 at 1:27 PM Tristan Tarrant <ttarrant at redhat.com>
> wrote:
> >>
> >> Guys,
> >>
> >> because of the amount of pending changes, I'd like to release 9.0.0.CR4
> >> on Wednesday 22nd and Final next week.
> >>
> >> Opinions ?
> >>
> >> Tristan
> >>
> >> --
> >> Tristan Tarrant
> >> Infinispan Lead
> >> JBoss, a division of Red Hat
> >> _______________________________________________
> >> infinispan-dev mailing list
> >> infinispan-dev at lists.jboss.org
> >> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >
> >
> > _______________________________________________
> > infinispan-dev mailing list
> > infinispan-dev at lists.jboss.org
> > https://lists.jboss.org/mailman/listinfo/infinispan-dev
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/infinispan-dev/attachments/20170321/65e5b2ff/attachment.html 

From remerson at redhat.com  Tue Mar 21 07:00:34 2017
From: remerson at redhat.com (Ryan Emerson)
Date: Tue, 21 Mar 2017 07:00:34 -0400 (EDT)
Subject: [infinispan-dev] Ryan Update
In-Reply-To: <1477129728.5453231.1490093668502.JavaMail.zimbra@redhat.com>
Message-ID: <217995601.5454961.1490094034482.JavaMail.zimbra@redhat.com>

Hi All,

I was on PTO last Friday and this Monday, hence I missed the weekly meeting. 

Last week I mainly worked on implementing partition handling merge policies. 

As well as:
   - Re-working ISPN-7065 Collect clustered memory statistics based upon feedback.
   - Updating JDBC shared store docs ISPN-6798
   - Fixing console log events ISPN-7629
   - Reviewing PRs

This week:
   - Review PRs
   - Continue with Partition Handling merge policies

Ryan

From mudokonman at gmail.com  Tue Mar 21 11:37:17 2017
From: mudokonman at gmail.com (William Burns)
Date: Tue, 21 Mar 2017 15:37:17 +0000
Subject: [infinispan-dev] Stream operations under lock
Message-ID: <CA+YCuUM3h0H+ZDe1ZvK5mOuXnLB9bjjP+CkJg_M8eyGKC+jRHA@mail.gmail.com>

Some users have expressed the need to have some sort of forEach operation
that is performed where the Consumer is called while holding the lock for
the given key and subsequently released after the Consumer operation
completes.

Due to the nature of how streams work with retries and performing the
operation on the primary owner, this works out quite well with forEach to
be done in an efficient way.

The problem is that this only really works well with non tx and pessimistic
tx. This obviously leaves out optimistic tx, which at first I was a little
worried about. But after thinking about it more, this prelocking and
optimistic tx don't really fit that well together anyways. So I am thinking
whenever this operation is performed it would throw an exception not
letting the user use this feature in optimistic transactions.

Another question is what does the API for this look like. I was debating
between 3 options myself:

1. AdvancedCache.forEachWithLock(BiConsumer<Cache, CacheEntry<K, V>>
consumer)

This require the least amount of changes, however the user can't customize
certain parameters that CacheStream currently provides (listed below - big
one being filterKeys).

2. CacheStream.forEachWithLock(BiConsumer<Cache, CacheEntry<K, V>> consumer)

This method would only be allowed to be invoked on the Stream if no other
intermediate operations were invoked, otherwise an exception would be
thrown. This still gives us access to all of the CacheStream methods that
aren't on the Stream interface (ie. sequentialDistribution,
parallelDistribution, parallel, sequential, filterKeys, filterKeySegments,
distributedBatchSize, disableRehashAware, timeout).

3. LockedStream<CacheEntry<K, V>> AdvancedCache.lockedStream()

This requires the most changes, however the API would be the most explicit.
In this case the LockedStream would only have the methods on it that are
able to be invoked as noted above and forEach.

I personally feel that #3 might be the cleanest, but obviously requires
adding more classes. Let me know what you guys think and if you think the
optimistic exclusion is acceptable.

Thanks,

 - Will
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/infinispan-dev/attachments/20170321/bc2d3ee1/attachment-0001.html 

From dan.berindei at gmail.com  Tue Mar 21 12:16:21 2017
From: dan.berindei at gmail.com (Dan Berindei)
Date: Tue, 21 Mar 2017 18:16:21 +0200
Subject: [infinispan-dev] Stream operations under lock
In-Reply-To: <CA+YCuUM3h0H+ZDe1ZvK5mOuXnLB9bjjP+CkJg_M8eyGKC+jRHA@mail.gmail.com>
References: <CA+YCuUM3h0H+ZDe1ZvK5mOuXnLB9bjjP+CkJg_M8eyGKC+jRHA@mail.gmail.com>
Message-ID: <CA+nfvwTaTFf_Ba4FkLAhejjQwC=p9YeiDuTrUaGj1krr2xgobw@mail.gmail.com>

I'm leaning towards option 1.

Are you thinking about also allowing the consumer to modify the entry,
like JCache's EntryProcessors? For a consumer that can only modify the
current entry, we could even "emulate" locking in an optimistic cache
by catching the WriteSkewException and running the consumer again.

I wouldn't allow this to be mixed with other operations in a stream,
because then you may have to run filters/mappers/sorting while holding
the lock as well.

Cheers
Dan


On Tue, Mar 21, 2017 at 5:37 PM, William Burns <mudokonman at gmail.com> wrote:
> Some users have expressed the need to have some sort of forEach operation
> that is performed where the Consumer is called while holding the lock for
> the given key and subsequently released after the Consumer operation
> completes.
>
> Due to the nature of how streams work with retries and performing the
> operation on the primary owner, this works out quite well with forEach to be
> done in an efficient way.
>
> The problem is that this only really works well with non tx and pessimistic
> tx. This obviously leaves out optimistic tx, which at first I was a little
> worried about. But after thinking about it more, this prelocking and
> optimistic tx don't really fit that well together anyways. So I am thinking
> whenever this operation is performed it would throw an exception not letting
> the user use this feature in optimistic transactions.
>
> Another question is what does the API for this look like. I was debating
> between 3 options myself:
>
> 1. AdvancedCache.forEachWithLock(BiConsumer<Cache, CacheEntry<K, V>>
> consumer)
>
> This require the least amount of changes, however the user can't customize
> certain parameters that CacheStream currently provides (listed below - big
> one being filterKeys).
>
> 2. CacheStream.forEachWithLock(BiConsumer<Cache, CacheEntry<K, V>> consumer)
>
> This method would only be allowed to be invoked on the Stream if no other
> intermediate operations were invoked, otherwise an exception would be
> thrown. This still gives us access to all of the CacheStream methods that
> aren't on the Stream interface (ie. sequentialDistribution,
> parallelDistribution, parallel, sequential, filterKeys, filterKeySegments,
> distributedBatchSize, disableRehashAware, timeout).
>
> 3. LockedStream<CacheEntry<K, V>> AdvancedCache.lockedStream()
>
> This requires the most changes, however the API would be the most explicit.
> In this case the LockedStream would only have the methods on it that are
> able to be invoked as noted above and forEach.
>
> I personally feel that #3 might be the cleanest, but obviously requires
> adding more classes. Let me know what you guys think and if you think the
> optimistic exclusion is acceptable.
>
> Thanks,
>
>  - Will
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev

From rvansa at redhat.com  Tue Mar 21 12:51:46 2017
From: rvansa at redhat.com (Radim Vansa)
Date: Tue, 21 Mar 2017 17:51:46 +0100
Subject: [infinispan-dev] Stream operations under lock
In-Reply-To: <CA+YCuUM3h0H+ZDe1ZvK5mOuXnLB9bjjP+CkJg_M8eyGKC+jRHA@mail.gmail.com>
References: <CA+YCuUM3h0H+ZDe1ZvK5mOuXnLB9bjjP+CkJg_M8eyGKC+jRHA@mail.gmail.com>
Message-ID: <d7e8a644-cb77-3e2a-f17b-79ceb70b2d85@redhat.com>

On 03/21/2017 04:37 PM, William Burns wrote:
> Some users have expressed the need to have some sort of forEach 
> operation that is performed where the Consumer is called while holding 
> the lock for the given key and subsequently released after the 
> Consumer operation completes.

Seconding Dan's question - is that intended to be able to modify the 
entry? In my opinion, sending a function that will work on the 
ReadWriteEntryView directly to the node is the only reasonable request. 
I wouldn't like to see blocking operations in there.

>
> Due to the nature of how streams work with retries and performing the 
> operation on the primary owner, this works out quite well with forEach 
> to be done in an efficient way.
>
> The problem is that this only really works well with non tx and 
> pessimistic tx. This obviously leaves out optimistic tx, which at 
> first I was a little worried about. But after thinking about it more, 
> this prelocking and optimistic tx don't really fit that well together 
> anyways. So I am thinking whenever this operation is performed it 
> would throw an exception not letting the user use this feature in 
> optimistic transactions.

How exactly reading streams interacts with transactions? Does it wrap 
read entries into context? This would be a scalability issue.

I agree that "locking" should not be exposed with optimistic transactions.

With pessimistic transactions, how do you expect to handle locking 
order? For regular operations, user is responsible for setting up some 
locking order in order to not get a deadlock. With pessimistic 
transaction, it's the cache itself who will order the calls. Also, if 
you lock anything that is read, you just end up locking everything (or, 
getting a deadlock). If you don't it's the same as issuing the lock and 
reading again (to check the locked value) - but you'd do that internally 
anyway. Therefore, I don't feel well about pessimistic transactions neither.

>
> Another question is what does the API for this look like. I was 
> debating between 3 options myself:
>
> 1. AdvancedCache.forEachWithLock(BiConsumer<Cache, CacheEntry<K, V>> 
> consumer)
>
> This require the least amount of changes, however the user can't 
> customize certain parameters that CacheStream currently provides 
> (listed below - big one being filterKeys).
>
> 2. CacheStream.forEachWithLock(BiConsumer<Cache, CacheEntry<K, V>> 
> consumer)
>
> This method would only be allowed to be invoked on the Stream if no 
> other intermediate operations were invoked, otherwise an exception 
> would be thrown. This still gives us access to all of the CacheStream 
> methods that aren't on the Stream interface (ie. 
> sequentialDistribution, parallelDistribution, parallel, sequential, 
> filterKeys, filterKeySegments, distributedBatchSize, 
> disableRehashAware, timeout).

For both options, I don't like Cache being passed around. You should 
modify the CacheEntry (or some kind of view) directly.

Radim

>
> 3. LockedStream<CacheEntry<K, V>> AdvancedCache.lockedStream()
>
> This requires the most changes, however the API would be the most 
> explicit. In this case the LockedStream would only have the methods on 
> it that are able to be invoked as noted above and forEach.
>
> I personally feel that #3 might be the cleanest, but obviously 
> requires adding more classes. Let me know what you guys think and if 
> you think the optimistic exclusion is acceptable.
>
> Thanks,
>
>  - Will
>
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev


-- 
Radim Vansa <rvansa at redhat.com>
JBoss Performance Team


From mudokonman at gmail.com  Tue Mar 21 13:31:03 2017
From: mudokonman at gmail.com (William Burns)
Date: Tue, 21 Mar 2017 17:31:03 +0000
Subject: [infinispan-dev] Stream operations under lock
In-Reply-To: <CA+nfvwTaTFf_Ba4FkLAhejjQwC=p9YeiDuTrUaGj1krr2xgobw@mail.gmail.com>
References: <CA+YCuUM3h0H+ZDe1ZvK5mOuXnLB9bjjP+CkJg_M8eyGKC+jRHA@mail.gmail.com>
	<CA+nfvwTaTFf_Ba4FkLAhejjQwC=p9YeiDuTrUaGj1krr2xgobw@mail.gmail.com>
Message-ID: <CA+YCuUNbfWhq_bKtHZaxySohKgRcyZnmWKHv-XWOCz7wCW35XQ@mail.gmail.com>

On Tue, Mar 21, 2017 at 12:17 PM Dan Berindei <dan.berindei at gmail.com>
wrote:

> I'm leaning towards option 1.
>

This is actually what I have implemented right now. The big problem I find
is the lack of configuring the operation. For example I know that users
will require being able to pass in a set of keys. The more I think about it
a Predicate would also be preferred. This could be handled by having
another override of the method that takes this Predicate or Set of keys. My
main concern was whether we thought users may want to tweak other settings
such as timeout, parallel or sequential operation per node or cluster and
also batch size possibly to reduce chances of more than once operations.
This ends up exploding the overrides, which is why I was hoping to use
something like the Stream interface to handle each as a different method
invocation. This is what brought me to think of #2, and subsequently #3
after I thought of excluding all those methods.


>
> Are you thinking about also allowing the consumer to modify the entry,
> like JCache's EntryProcessors? For a consumer that can only modify the
> current entry, we could even "emulate" locking in an optimistic cache
> by catching the WriteSkewException and running the consumer again.
>

That is one possible case (updating cache entry). To be honest the user
could want it for other reasons that I am not privy to.

The catching of WriteSkewException is interesting, but I don't think it is
acceptable. The problem is that the value could change between which could
in turn change what the user wants to write or operation they perform and
there is no way to inform them.


>
> I wouldn't allow this to be mixed with other operations in a stream,
> because then you may have to run filters/mappers/sorting while holding
> the lock as well.
>

That was my idea as well and why I was preventing those in all three APIs.
Although as I mentioned above allowing filter should be fine too. It also
makes the process of locking and unlocking the entry quite cumbersome if we
allowed other operations.


>
> Cheers
> Dan
>
>
> On Tue, Mar 21, 2017 at 5:37 PM, William Burns <mudokonman at gmail.com>
> wrote:
> > Some users have expressed the need to have some sort of forEach operation
> > that is performed where the Consumer is called while holding the lock for
> > the given key and subsequently released after the Consumer operation
> > completes.
> >
> > Due to the nature of how streams work with retries and performing the
> > operation on the primary owner, this works out quite well with forEach
> to be
> > done in an efficient way.
> >
> > The problem is that this only really works well with non tx and
> pessimistic
> > tx. This obviously leaves out optimistic tx, which at first I was a
> little
> > worried about. But after thinking about it more, this prelocking and
> > optimistic tx don't really fit that well together anyways. So I am
> thinking
> > whenever this operation is performed it would throw an exception not
> letting
> > the user use this feature in optimistic transactions.
> >
> > Another question is what does the API for this look like. I was debating
> > between 3 options myself:
> >
> > 1. AdvancedCache.forEachWithLock(BiConsumer<Cache, CacheEntry<K, V>>
> > consumer)
> >
> > This require the least amount of changes, however the user can't
> customize
> > certain parameters that CacheStream currently provides (listed below -
> big
> > one being filterKeys).
> >
> > 2. CacheStream.forEachWithLock(BiConsumer<Cache, CacheEntry<K, V>>
> consumer)
> >
> > This method would only be allowed to be invoked on the Stream if no other
> > intermediate operations were invoked, otherwise an exception would be
> > thrown. This still gives us access to all of the CacheStream methods that
> > aren't on the Stream interface (ie. sequentialDistribution,
> > parallelDistribution, parallel, sequential, filterKeys,
> filterKeySegments,
> > distributedBatchSize, disableRehashAware, timeout).
> >
> > 3. LockedStream<CacheEntry<K, V>> AdvancedCache.lockedStream()
> >
> > This requires the most changes, however the API would be the most
> explicit.
> > In this case the LockedStream would only have the methods on it that are
> > able to be invoked as noted above and forEach.
> >
> > I personally feel that #3 might be the cleanest, but obviously requires
> > adding more classes. Let me know what you guys think and if you think the
> > optimistic exclusion is acceptable.
> >
> > Thanks,
> >
> >  - Will
> >
> > _______________________________________________
> > infinispan-dev mailing list
> > infinispan-dev at lists.jboss.org
> > https://lists.jboss.org/mailman/listinfo/infinispan-dev
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/infinispan-dev/attachments/20170321/3d8dbdfa/attachment.html 

From mudokonman at gmail.com  Tue Mar 21 13:42:35 2017
From: mudokonman at gmail.com (William Burns)
Date: Tue, 21 Mar 2017 17:42:35 +0000
Subject: [infinispan-dev] Stream operations under lock
In-Reply-To: <d7e8a644-cb77-3e2a-f17b-79ceb70b2d85@redhat.com>
References: <CA+YCuUM3h0H+ZDe1ZvK5mOuXnLB9bjjP+CkJg_M8eyGKC+jRHA@mail.gmail.com>
	<d7e8a644-cb77-3e2a-f17b-79ceb70b2d85@redhat.com>
Message-ID: <CA+YCuUOMttxZqxSpFzvPu_Q0sFS_W0P9Wh062UmSWGWewmgxaA@mail.gmail.com>

On Tue, Mar 21, 2017 at 12:53 PM Radim Vansa <rvansa at redhat.com> wrote:

> On 03/21/2017 04:37 PM, William Burns wrote:
> > Some users have expressed the need to have some sort of forEach
> > operation that is performed where the Consumer is called while holding
> > the lock for the given key and subsequently released after the
> > Consumer operation completes.
>
> Seconding Dan's question - is that intended to be able to modify the
> entry? In my opinion, sending a function that will work on the
> ReadWriteEntryView directly to the node is the only reasonable request.
> I wouldn't like to see blocking operations in there.
>

Hrmm the user can use the FunctionalMap interface for this then it seems? I
wonder if this should just be the going in API. I will need to discuss with
Galder the semantics of the evalAll/evalMany methods.


>
> >
> > Due to the nature of how streams work with retries and performing the
> > operation on the primary owner, this works out quite well with forEach
> > to be done in an efficient way.
> >
> > The problem is that this only really works well with non tx and
> > pessimistic tx. This obviously leaves out optimistic tx, which at
> > first I was a little worried about. But after thinking about it more,
> > this prelocking and optimistic tx don't really fit that well together
> > anyways. So I am thinking whenever this operation is performed it
> > would throw an exception not letting the user use this feature in
> > optimistic transactions.
>
> How exactly reading streams interacts with transactions? Does it wrap
> read entries into context? This would be a scalability issue.
>

It doesn't wrap read entries into the context for that exact reason. It
does however use existing entries in the context to override ones in
memory/store.


>
> I agree that "locking" should not be exposed with optimistic transactions.
>

Yeah I can't find a good way to do this really and it seems to be opposite
of what optimistic transactions are.


>
> With pessimistic transactions, how do you expect to handle locking
> order? For regular operations, user is responsible for setting up some
> locking order in order to not get a deadlock. With pessimistic
> transaction, it's the cache itself who will order the calls. Also, if
> you lock anything that is read, you just end up locking everything (or,
> getting a deadlock). If you don't it's the same as issuing the lock and
> reading again (to check the locked value) - but you'd do that internally
> anyway. Therefore, I don't feel well about pessimistic transactions
> neither.
>

The lock is done per key only for each invocation. There is no ordering as
only one is obtained at a time before it goes to the next. If the user then
acquires a lock for another key while in the Consumer this could cause a
deadlock if the inverse occurs on a different thread/node, but this is on
the user. It is the same as it is today really, except we do the read lock
for them before invoking their Consumer.


>
> >
> > Another question is what does the API for this look like. I was
> > debating between 3 options myself:
> >
> > 1. AdvancedCache.forEachWithLock(BiConsumer<Cache, CacheEntry<K, V>>
> > consumer)
> >
> > This require the least amount of changes, however the user can't
> > customize certain parameters that CacheStream currently provides
> > (listed below - big one being filterKeys).
> >
> > 2. CacheStream.forEachWithLock(BiConsumer<Cache, CacheEntry<K, V>>
> > consumer)
> >
> > This method would only be allowed to be invoked on the Stream if no
> > other intermediate operations were invoked, otherwise an exception
> > would be thrown. This still gives us access to all of the CacheStream
> > methods that aren't on the Stream interface (ie.
> > sequentialDistribution, parallelDistribution, parallel, sequential,
> > filterKeys, filterKeySegments, distributedBatchSize,
> > disableRehashAware, timeout).
>
> For both options, I don't like Cache being passed around. You should
> modify the CacheEntry (or some kind of view) directly.
>

I don't know for sure if that is sufficient for the user. Sometimes they
may modify another Cache given the value in this one for example, which
they could access from the CacheManager of that Cache. Maybe Tristan knows
more about some use cases.


>
> Radim
>
> >
> > 3. LockedStream<CacheEntry<K, V>> AdvancedCache.lockedStream()
> >
> > This requires the most changes, however the API would be the most
> > explicit. In this case the LockedStream would only have the methods on
> > it that are able to be invoked as noted above and forEach.
> >
> > I personally feel that #3 might be the cleanest, but obviously
> > requires adding more classes. Let me know what you guys think and if
> > you think the optimistic exclusion is acceptable.
> >
> > Thanks,
> >
> >  - Will
> >
> >
> > _______________________________________________
> > infinispan-dev mailing list
> > infinispan-dev at lists.jboss.org
> > https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
>
> --
> Radim Vansa <rvansa at redhat.com>
> JBoss Performance Team
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/infinispan-dev/attachments/20170321/c4ddaacc/attachment-0001.html 

From mudokonman at gmail.com  Tue Mar 21 13:50:58 2017
From: mudokonman at gmail.com (William Burns)
Date: Tue, 21 Mar 2017 17:50:58 +0000
Subject: [infinispan-dev] Stream operations under lock
In-Reply-To: <CA+YCuUOMttxZqxSpFzvPu_Q0sFS_W0P9Wh062UmSWGWewmgxaA@mail.gmail.com>
References: <CA+YCuUM3h0H+ZDe1ZvK5mOuXnLB9bjjP+CkJg_M8eyGKC+jRHA@mail.gmail.com>
	<d7e8a644-cb77-3e2a-f17b-79ceb70b2d85@redhat.com>
	<CA+YCuUOMttxZqxSpFzvPu_Q0sFS_W0P9Wh062UmSWGWewmgxaA@mail.gmail.com>
Message-ID: <CA+YCuUML6s77x9JNk6m1o=c526HK6porbaymNoFK=nmj7Ny24w@mail.gmail.com>

On Tue, Mar 21, 2017 at 1:42 PM William Burns <mudokonman at gmail.com> wrote:

> On Tue, Mar 21, 2017 at 12:53 PM Radim Vansa <rvansa at redhat.com> wrote:
>
> On 03/21/2017 04:37 PM, William Burns wrote:
> > Some users have expressed the need to have some sort of forEach
> > operation that is performed where the Consumer is called while holding
> > the lock for the given key and subsequently released after the
> > Consumer operation completes.
>
> Seconding Dan's question - is that intended to be able to modify the
> entry? In my opinion, sending a function that will work on the
> ReadWriteEntryView directly to the node is the only reasonable request.
> I wouldn't like to see blocking operations in there.
>
>
> Hrmm the user can use the FunctionalMap interface for this then it seems?
> I wonder if this should just be the going in API. I will need to discuss
> with Galder the semantics of the evalAll/evalMany methods.
>

Actually looking at evalAll it seems it doesn't scale as it keeps all
entries in memory at once, so this is only for caches with a limited amount
of entries.


>
>
>
> >
> > Due to the nature of how streams work with retries and performing the
> > operation on the primary owner, this works out quite well with forEach
> > to be done in an efficient way.
> >
> > The problem is that this only really works well with non tx and
> > pessimistic tx. This obviously leaves out optimistic tx, which at
> > first I was a little worried about. But after thinking about it more,
> > this prelocking and optimistic tx don't really fit that well together
> > anyways. So I am thinking whenever this operation is performed it
> > would throw an exception not letting the user use this feature in
> > optimistic transactions.
>
> How exactly reading streams interacts with transactions? Does it wrap
> read entries into context? This would be a scalability issue.
>
>
> It doesn't wrap read entries into the context for that exact reason. It
> does however use existing entries in the context to override ones in
> memory/store.
>
>
>
> I agree that "locking" should not be exposed with optimistic transactions.
>
>
> Yeah I can't find a good way to do this really and it seems to be opposite
> of what optimistic transactions are.
>
>
>
> With pessimistic transactions, how do you expect to handle locking
> order? For regular operations, user is responsible for setting up some
> locking order in order to not get a deadlock. With pessimistic
> transaction, it's the cache itself who will order the calls. Also, if
> you lock anything that is read, you just end up locking everything (or,
> getting a deadlock). If you don't it's the same as issuing the lock and
> reading again (to check the locked value) - but you'd do that internally
> anyway. Therefore, I don't feel well about pessimistic transactions
> neither.
>
>
> The lock is done per key only for each invocation. There is no ordering as
> only one is obtained at a time before it goes to the next. If the user then
> acquires a lock for another key while in the Consumer this could cause a
> deadlock if the inverse occurs on a different thread/node, but this is on
> the user. It is the same as it is today really, except we do the read lock
> for them before invoking their Consumer.
>
>
>
> >
> > Another question is what does the API for this look like. I was
> > debating between 3 options myself:
> >
> > 1. AdvancedCache.forEachWithLock(BiConsumer<Cache, CacheEntry<K, V>>
> > consumer)
> >
> > This require the least amount of changes, however the user can't
> > customize certain parameters that CacheStream currently provides
> > (listed below - big one being filterKeys).
> >
> > 2. CacheStream.forEachWithLock(BiConsumer<Cache, CacheEntry<K, V>>
> > consumer)
> >
> > This method would only be allowed to be invoked on the Stream if no
> > other intermediate operations were invoked, otherwise an exception
> > would be thrown. This still gives us access to all of the CacheStream
> > methods that aren't on the Stream interface (ie.
> > sequentialDistribution, parallelDistribution, parallel, sequential,
> > filterKeys, filterKeySegments, distributedBatchSize,
> > disableRehashAware, timeout).
>
> For both options, I don't like Cache being passed around. You should
> modify the CacheEntry (or some kind of view) directly.
>
>
> I don't know for sure if that is sufficient for the user. Sometimes they
> may modify another Cache given the value in this one for example, which
> they could access from the CacheManager of that Cache. Maybe Tristan knows
> more about some use cases.
>
>
>
> Radim
>
> >
> > 3. LockedStream<CacheEntry<K, V>> AdvancedCache.lockedStream()
> >
> > This requires the most changes, however the API would be the most
> > explicit. In this case the LockedStream would only have the methods on
> > it that are able to be invoked as noted above and forEach.
> >
> > I personally feel that #3 might be the cleanest, but obviously
> > requires adding more classes. Let me know what you guys think and if
> > you think the optimistic exclusion is acceptable.
> >
> > Thanks,
> >
> >  - Will
> >
> >
> > _______________________________________________
> > infinispan-dev mailing list
> > infinispan-dev at lists.jboss.org
> > https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
>
> --
> Radim Vansa <rvansa at redhat.com>
> JBoss Performance Team
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/infinispan-dev/attachments/20170321/8f9c34d3/attachment.html 

From ttarrant at redhat.com  Tue Mar 21 15:46:18 2017
From: ttarrant at redhat.com (Tristan Tarrant)
Date: Tue, 21 Mar 2017 20:46:18 +0100
Subject: [infinispan-dev] Stream operations under lock
In-Reply-To: <CA+YCuUM3h0H+ZDe1ZvK5mOuXnLB9bjjP+CkJg_M8eyGKC+jRHA@mail.gmail.com>
References: <CA+YCuUM3h0H+ZDe1ZvK5mOuXnLB9bjjP+CkJg_M8eyGKC+jRHA@mail.gmail.com>
Message-ID: <e2976041-56b6-2ae6-c826-9f3cee0cf07f@redhat.com>



On 21/03/17 16:37, William Burns wrote:
> Some users have expressed the need to have some sort of forEach 
> operation that is performed where the Consumer is called while holding 
> the lock for the given key and subsequently released after the Consumer 
> operation completes.
> 
> Due to the nature of how streams work with retries and performing the 
> operation on the primary owner, this works out quite well with forEach 
> 1. AdvancedCache.forEachWithLock(BiConsumer<Cache, CacheEntry<K, V>> 
> consumer)
> 
> This require the least amount of changes, however the user can't 
> customize certain parameters that CacheStream currently provides (listed 
> below - big one being filterKeys).
> 
> 2. CacheStream.forEachWithLock(BiConsumer<Cache, CacheEntry<K, V>> consumer)
> 
> This method would only be allowed to be invoked on the Stream if no 
> other intermediate operations were invoked, otherwise an exception would 
> be thrown. This still gives us access to all of the CacheStream methods 
> that aren't on the Stream interface (ie. sequentialDistribution, 
> parallelDistribution, parallel, sequential, filterKeys, 
> filterKeySegments, distributedBatchSize, disableRehashAware, timeout).
> 
> 3. LockedStream<CacheEntry<K, V>> AdvancedCache.lockedStream()
> 
> This requires the most changes, however the API would be the most 
> explicit. In this case the LockedStream would only have the methods on 
> it that are able to be invoked as noted above and forEach.
> 
> I personally feel that #3 might be the cleanest, but obviously requires 
> adding more classes. Let me know what you guys think and if you think 
> the optimistic exclusion is acceptable.

I prefer option 3, because I don't like UnsupportedOperationExceptions 
lurking until you hit runtime. Can you quantify the amount of extra 
effort here ?

Tristan

-- 
Tristan Tarrant
Infinispan Lead
JBoss, a division of Red Hat

From mudokonman at gmail.com  Tue Mar 21 16:43:58 2017
From: mudokonman at gmail.com (William Burns)
Date: Tue, 21 Mar 2017 20:43:58 +0000
Subject: [infinispan-dev] Stream operations under lock
In-Reply-To: <e2976041-56b6-2ae6-c826-9f3cee0cf07f@redhat.com>
References: <CA+YCuUM3h0H+ZDe1ZvK5mOuXnLB9bjjP+CkJg_M8eyGKC+jRHA@mail.gmail.com>
	<e2976041-56b6-2ae6-c826-9f3cee0cf07f@redhat.com>
Message-ID: <CA+YCuUM5UssG0KUY7RRgLBKhyNw2mTosW1jMvittm_ZENDGB5w@mail.gmail.com>

On Tue, Mar 21, 2017 at 4:28 PM Tristan Tarrant <ttarrant at redhat.com> wrote:

>
>
> On 21/03/17 16:37, William Burns wrote:
> > Some users have expressed the need to have some sort of forEach
> > operation that is performed where the Consumer is called while holding
> > the lock for the given key and subsequently released after the Consumer
> > operation completes.
> >
> > Due to the nature of how streams work with retries and performing the
> > operation on the primary owner, this works out quite well with forEach
> > 1. AdvancedCache.forEachWithLock(BiConsumer<Cache, CacheEntry<K, V>>
> > consumer)
> >
> > This require the least amount of changes, however the user can't
> > customize certain parameters that CacheStream currently provides (listed
> > below - big one being filterKeys).
> >
> > 2. CacheStream.forEachWithLock(BiConsumer<Cache, CacheEntry<K, V>>
> consumer)
> >
> > This method would only be allowed to be invoked on the Stream if no
> > other intermediate operations were invoked, otherwise an exception would
> > be thrown. This still gives us access to all of the CacheStream methods
> > that aren't on the Stream interface (ie. sequentialDistribution,
> > parallelDistribution, parallel, sequential, filterKeys,
> > filterKeySegments, distributedBatchSize, disableRehashAware, timeout).
> >
> > 3. LockedStream<CacheEntry<K, V>> AdvancedCache.lockedStream()
> >
> > This requires the most changes, however the API would be the most
> > explicit. In this case the LockedStream would only have the methods on
> > it that are able to be invoked as noted above and forEach.
> >
> > I personally feel that #3 might be the cleanest, but obviously requires
> > adding more classes. Let me know what you guys think and if you think
> > the optimistic exclusion is acceptable.
>
> I prefer option 3, because I don't like UnsupportedOperationExceptions
> lurking until you hit runtime. Can you quantify the amount of extra
> effort here ?
>

I would probably add a new interface that defines the common methods
between CacheStream and LockedStream (or whatever we call it). Then the
actual LockedStream class should be a self contained class. Thinking about
it more it shouldn't require more than a couple hundred extra lines of
code. The hard part was just getting the locking to work with non tx and
pessimistic tx, which is common to any impl.

Also just to note you would still get an UnsupportedOperationException when
running in optimistic tx when invoking the method.


>
> Tristan
>
> --
> Tristan Tarrant
> Infinispan Lead
> JBoss, a division of Red Hat
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/infinispan-dev/attachments/20170321/0d762b4e/attachment-0001.html 

From rvansa at redhat.com  Wed Mar 22 05:51:47 2017
From: rvansa at redhat.com (Radim Vansa)
Date: Wed, 22 Mar 2017 10:51:47 +0100
Subject: [infinispan-dev] Stream operations under lock
In-Reply-To: <CA+YCuUML6s77x9JNk6m1o=c526HK6porbaymNoFK=nmj7Ny24w@mail.gmail.com>
References: <CA+YCuUM3h0H+ZDe1ZvK5mOuXnLB9bjjP+CkJg_M8eyGKC+jRHA@mail.gmail.com>
	<d7e8a644-cb77-3e2a-f17b-79ceb70b2d85@redhat.com>
	<CA+YCuUOMttxZqxSpFzvPu_Q0sFS_W0P9Wh062UmSWGWewmgxaA@mail.gmail.com>
	<CA+YCuUML6s77x9JNk6m1o=c526HK6porbaymNoFK=nmj7Ny24w@mail.gmail.com>
Message-ID: <914f7208-ae02-7d9b-cf0b-6f5feb332609@redhat.com>

On 03/21/2017 06:50 PM, William Burns wrote:
>
>
> On Tue, Mar 21, 2017 at 1:42 PM William Burns <mudokonman at gmail.com 
> <mailto:mudokonman at gmail.com>> wrote:
>
>     On Tue, Mar 21, 2017 at 12:53 PM Radim Vansa <rvansa at redhat.com
>     <mailto:rvansa at redhat.com>> wrote:
>
>         On 03/21/2017 04:37 PM, William Burns wrote:
>         > Some users have expressed the need to have some sort of forEach
>         > operation that is performed where the Consumer is called
>         while holding
>         > the lock for the given key and subsequently released after the
>         > Consumer operation completes.
>
>         Seconding Dan's question - is that intended to be able to
>         modify the
>         entry? In my opinion, sending a function that will work on the
>         ReadWriteEntryView directly to the node is the only reasonable
>         request.
>         I wouldn't like to see blocking operations in there.
>
>
>     Hrmm the user can use the FunctionalMap interface for this then it
>     seems? I wonder if this should just be the going in API. I will
>     need to discuss with Galder the semantics of the evalAll/evalMany
>     methods.
>
>
> Actually looking at evalAll it seems it doesn't scale as it keeps all 
> entries in memory at once, so this is only for caches with a limited 
> amount of entries.

Don't look into the implementation; I think Galder has focused more on 
the API side than having optimal implementation. IMO there's no reason 
evalAll should load all the entries into memory in non-transactional mode.

>
>         >
>         > Due to the nature of how streams work with retries and
>         performing the
>         > operation on the primary owner, this works out quite well
>         with forEach
>         > to be done in an efficient way.
>         >
>         > The problem is that this only really works well with non tx and
>         > pessimistic tx. This obviously leaves out optimistic tx,
>         which at
>         > first I was a little worried about. But after thinking about
>         it more,
>         > this prelocking and optimistic tx don't really fit that well
>         together
>         > anyways. So I am thinking whenever this operation is
>         performed it
>         > would throw an exception not letting the user use this
>         feature in
>         > optimistic transactions.
>
>         How exactly reading streams interacts with transactions? Does
>         it wrap
>         read entries into context? This would be a scalability issue.
>
>
>     It doesn't wrap read entries into the context for that exact
>     reason. It does however use existing entries in the context to
>     override ones in memory/store.
>

Uuh, so you end up with a copy of the cache in single invocation 
context, without any means to flush it. I think that we need add 
InvocationContext.current().forget(key) API (throwing exception if the 
entry was modified) or something like that, even for the regular 
streams. Maybe an override for filter methods, too, because you want to 
pass a nice predicate, but you can't just forget all filtered out entries.

>
>         I agree that "locking" should not be exposed with optimistic
>         transactions.
>
>
>     Yeah I can't find a good way to do this really and it seems to be
>     opposite of what optimistic transactions are.
>
>
>         With pessimistic transactions, how do you expect to handle locking
>         order? For regular operations, user is responsible for setting
>         up some
>         locking order in order to not get a deadlock. With pessimistic
>         transaction, it's the cache itself who will order the calls.
>         Also, if
>         you lock anything that is read, you just end up locking
>         everything (or,
>         getting a deadlock). If you don't it's the same as issuing the
>         lock and
>         reading again (to check the locked value) - but you'd do that
>         internally
>         anyway. Therefore, I don't feel well about pessimistic
>         transactions neither.
>
>
>     The lock is done per key only for each invocation. There is no
>     ordering as only one is obtained at a time before it goes to the
>     next. If the user then acquires a lock for another key while in
>     the Consumer this could cause a deadlock if the inverse occurs on
>     a different thread/node, but this is on the user. It is the same
>     as it is today really, except we do the read lock for them before
>     invoking their Consumer.
>

In pessimistic mode, you should not release a lock before the end of the 
transaction.

>
>         >
>         > Another question is what does the API for this look like. I was
>         > debating between 3 options myself:
>         >
>         > 1. AdvancedCache.forEachWithLock(BiConsumer<Cache,
>         CacheEntry<K, V>>
>         > consumer)
>         >
>         > This require the least amount of changes, however the user can't
>         > customize certain parameters that CacheStream currently provides
>         > (listed below - big one being filterKeys).
>         >
>         > 2. CacheStream.forEachWithLock(BiConsumer<Cache,
>         CacheEntry<K, V>>
>         > consumer)
>         >
>         > This method would only be allowed to be invoked on the
>         Stream if no
>         > other intermediate operations were invoked, otherwise an
>         exception
>         > would be thrown. This still gives us access to all of the
>         CacheStream
>         > methods that aren't on the Stream interface (ie.
>         > sequentialDistribution, parallelDistribution, parallel,
>         sequential,
>         > filterKeys, filterKeySegments, distributedBatchSize,
>         > disableRehashAware, timeout).
>
>         For both options, I don't like Cache being passed around. You
>         should
>         modify the CacheEntry (or some kind of view) directly.
>
>
>     I don't know for sure if that is sufficient for the user.
>     Sometimes they may modify another Cache given the value in this
>     one for example, which they could access from the CacheManager of
>     that Cache. Maybe Tristan knows more about some use cases.
>

Rather than guessing what could the user need, the Consumer could be CDI 
enabled.

>
>         Radim
>
>         >
>         > 3. LockedStream<CacheEntry<K, V>> AdvancedCache.lockedStream()
>         >
>         > This requires the most changes, however the API would be the
>         most
>         > explicit. In this case the LockedStream would only have the
>         methods on
>         > it that are able to be invoked as noted above and forEach.
>         >
>         > I personally feel that #3 might be the cleanest, but obviously
>         > requires adding more classes. Let me know what you guys
>         think and if
>         > you think the optimistic exclusion is acceptable.
>         >
>         > Thanks,
>         >
>         >  - Will
>         >
>         >
>         > _______________________________________________
>         > infinispan-dev mailing list
>         > infinispan-dev at lists.jboss.org
>         <mailto:infinispan-dev at lists.jboss.org>
>         > https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
>
>         --
>         Radim Vansa <rvansa at redhat.com <mailto:rvansa at redhat.com>>
>         JBoss Performance Team
>
>         _______________________________________________
>         infinispan-dev mailing list
>         infinispan-dev at lists.jboss.org
>         <mailto:infinispan-dev at lists.jboss.org>
>         https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
>
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev


-- 
Radim Vansa <rvansa at redhat.com>
JBoss Performance Team


From karesti at redhat.com  Wed Mar 22 15:46:05 2017
From: karesti at redhat.com (Katia Aresti)
Date: Wed, 22 Mar 2017 20:46:05 +0100
Subject: [infinispan-dev]  ConcurrentMap new methods implementation
Message-ID: <CAKypdvk6w24dukHXwckBoeCNxGiu+41L4nbXuMZrOu4iJXU77Q@mail.gmail.com>

Hi all,

Since Java 8, some new methods are available in the ConcurrentMap
interface: merge, compute, computeIfAbsent, computeIfPresent, forEach,
replaceAll. ConcurrentMap interface provides a default implementation.

I'm working on https://issues.jboss.org/browse/ISPN-5728 in order to
provide the infinispan specific implementation. The issue here is that to
make it work, these lambdas must be Serializables, so actual code using
these methods and not passing serializables lambdas will break.

I see two possibilities here, please fell free to suggest any other idea.

1) Override the default implementation and specify on the release that all
the lambdas must be serializables from now on ... ?
2)  Leave the implementation of the default methods as they are and provide
new methods implemented the infinispan way :

V compute(K key, V compute(K key,

                  SerializableBiFunction<? super K, ? super V, ?
extends V> remappingFunction)


What do you think ?

-- Katia
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/infinispan-dev/attachments/20170322/6610e59b/attachment.html 

From ttarrant at redhat.com  Wed Mar 22 17:34:48 2017
From: ttarrant at redhat.com (Tristan Tarrant)
Date: Wed, 22 Mar 2017 22:34:48 +0100
Subject: [infinispan-dev] Infinispan 9.0.0.CR4
Message-ID: <ea8ee025-9bb0-e3de-e344-8be60c0da4e6@redhat.com>

One more candidate release before Final:

http://blog.infinispan.org/2017/03/infinispan-900cr4.html

Tristan
-- 
Tristan Tarrant
Infinispan Lead
JBoss, a division of Red Hat

From rvansa at redhat.com  Thu Mar 23 04:42:58 2017
From: rvansa at redhat.com (Radim Vansa)
Date: Thu, 23 Mar 2017 09:42:58 +0100
Subject: [infinispan-dev] ConcurrentMap new methods implementation
In-Reply-To: <CAKypdvk6w24dukHXwckBoeCNxGiu+41L4nbXuMZrOu4iJXU77Q@mail.gmail.com>
References: <CAKypdvk6w24dukHXwckBoeCNxGiu+41L4nbXuMZrOu4iJXU77Q@mail.gmail.com>
Message-ID: <14e1f874-7ee4-a49f-3c24-de17d61ec89f@redhat.com>

On 03/22/2017 08:46 PM, Katia Aresti wrote:
> Hi all,
>
> Since Java 8, some new methods are available in the ConcurrentMap 
> interface: merge, compute, computeIfAbsent, computeIfPresent, forEach, 
> replaceAll. ConcurrentMap interface provides a default implementation.
>
> I'm working on https://issues.jboss.org/browse/ISPN-5728 
> <https://issues.jboss.org/browse/ISPN-5728> in order to provide the 
> infinispan specific implementation. The issue here is that to make it 
> work, these lambdas must be Serializables, so actual code using these 
> methods and not passing serializables lambdas will break.
>
> I see two possibilities here, please fell free to suggest any other idea.
>
> 1) Override the default implementation and specify on the release that 
> all the lambdas must be serializables from now on ... ?
> 2)  Leave the implementation of the default methods as they are and 
> provide new methods implemented the infinispan way :

To be consistent with Streams API, we should assume that the lambda is 
serializable (and fail as gracefully as we can if it's not and should 
be) and provide the SerializableX overload method to make the lambdas 
serializable when the lambda is used directly on Cache. I wouldn't leave 
the default implementation, because:

1) it's prone to user error, particularly during refactoring. Switching 
from functional to default implementation would introduce a regression 
in his app's performance
2) For local caches or local-mode operations, the lambda doesn't need to 
be serializable

Radim

> V compute(K key, Vcompute(K key,
>                    SerializableBiFunction<?super K, ?super V, ?extends V> remappingFunction)
>
> What do you think ?
>
> -- Katia
>
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev


-- 
Radim Vansa <rvansa at redhat.com>
JBoss Performance Team


From mudokonman at gmail.com  Thu Mar 23 09:26:41 2017
From: mudokonman at gmail.com (William Burns)
Date: Thu, 23 Mar 2017 13:26:41 +0000
Subject: [infinispan-dev] Deprecate collection methods on DataContainer
	interface
Message-ID: <CA+YCuUOR1WNxoXzgRZX+aQmE=eFdzMO=beCLSGG+L34VA8+ZXg@mail.gmail.com>

Recently while looking at the DataContainer interface and implementation it
was noticed that the entrySet, values and keySet methods that return
collections are very inconsistent. The big issue is whether these methods
return expired entries or not and if they are backing or copies etc. It was
suggested [1] to deprecate these. I am all for this. I was looking at the
references to these methods and they all can be replace with using the
proper iterator and forEachRemaining instead.

I just wanted to confirm if everyone is alright with me deprecating the
collection methods on DataContainer interface for 9.0 Final.

[1]
https://github.com/infinispan/infinispan/pull/5011#pullrequestreview-28599145

Thanks,

 - Will
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/infinispan-dev/attachments/20170323/db9268f7/attachment.html 

From mudokonman at gmail.com  Thu Mar 23 09:41:09 2017
From: mudokonman at gmail.com (William Burns)
Date: Thu, 23 Mar 2017 13:41:09 +0000
Subject: [infinispan-dev] Stream operations under lock
In-Reply-To: <914f7208-ae02-7d9b-cf0b-6f5feb332609@redhat.com>
References: <CA+YCuUM3h0H+ZDe1ZvK5mOuXnLB9bjjP+CkJg_M8eyGKC+jRHA@mail.gmail.com>
	<d7e8a644-cb77-3e2a-f17b-79ceb70b2d85@redhat.com>
	<CA+YCuUOMttxZqxSpFzvPu_Q0sFS_W0P9Wh062UmSWGWewmgxaA@mail.gmail.com>
	<CA+YCuUML6s77x9JNk6m1o=c526HK6porbaymNoFK=nmj7Ny24w@mail.gmail.com>
	<914f7208-ae02-7d9b-cf0b-6f5feb332609@redhat.com>
Message-ID: <CA+YCuUN8pY0BOgHazmcPyZpFMqOgvDtjL_A+jexeiJp1mvJgBg@mail.gmail.com>

On Wed, Mar 22, 2017 at 5:51 AM Radim Vansa <rvansa at redhat.com> wrote:

> On 03/21/2017 06:50 PM, William Burns wrote:
> >
> >
> > On Tue, Mar 21, 2017 at 1:42 PM William Burns <mudokonman at gmail.com
> > <mailto:mudokonman at gmail.com>> wrote:
> >
> >     On Tue, Mar 21, 2017 at 12:53 PM Radim Vansa <rvansa at redhat.com
> >     <mailto:rvansa at redhat.com>> wrote:
> >
> >         On 03/21/2017 04:37 PM, William Burns wrote:
> >         > Some users have expressed the need to have some sort of forEach
> >         > operation that is performed where the Consumer is called
> >         while holding
> >         > the lock for the given key and subsequently released after the
> >         > Consumer operation completes.
> >
> >         Seconding Dan's question - is that intended to be able to
> >         modify the
> >         entry? In my opinion, sending a function that will work on the
> >         ReadWriteEntryView directly to the node is the only reasonable
> >         request.
> >         I wouldn't like to see blocking operations in there.
> >
> >
> >     Hrmm the user can use the FunctionalMap interface for this then it
> >     seems? I wonder if this should just be the going in API. I will
> >     need to discuss with Galder the semantics of the evalAll/evalMany
> >     methods.
> >
> >
> > Actually looking at evalAll it seems it doesn't scale as it keeps all
> > entries in memory at once, so this is only for caches with a limited
> > amount of entries.
>
> Don't look into the implementation; I think Galder has focused more on
> the API side than having optimal implementation. IMO there's no reason
> evalAll should load all the entries into memory in non-transactional mode.
>
>
I agree that it shouldn't, but there is no guarantee this will be ready any
time soon.


> >
> >         >
> >         > Due to the nature of how streams work with retries and
> >         performing the
> >         > operation on the primary owner, this works out quite well
> >         with forEach
> >         > to be done in an efficient way.
> >         >
> >         > The problem is that this only really works well with non tx and
> >         > pessimistic tx. This obviously leaves out optimistic tx,
> >         which at
> >         > first I was a little worried about. But after thinking about
> >         it more,
> >         > this prelocking and optimistic tx don't really fit that well
> >         together
> >         > anyways. So I am thinking whenever this operation is
> >         performed it
> >         > would throw an exception not letting the user use this
> >         feature in
> >         > optimistic transactions.
> >
> >         How exactly reading streams interacts with transactions? Does
> >         it wrap
> >         read entries into context? This would be a scalability issue.
> >
> >
> >     It doesn't wrap read entries into the context for that exact
> >     reason. It does however use existing entries in the context to
> >     override ones in memory/store.
> >
>
> Uuh, so you end up with a copy of the cache in single invocation
> context, without any means to flush it. I think that we need add
>

Maybe I worded it poorly. Streams don't wrap any entries at all. All it
does is read from the current context if there any, it then reads from the
data container (skipping entries it read from the context) and then finally
reading from the store if present.

Although the more I think about it using Stream with lock this may be a non
issue, read below.


> InvocationContext.current().forget(key) API (throwing exception if the
> entry was modified) or something like that, even for the regular
> streams. Maybe an override for filter methods, too, because you want to
> pass a nice predicate, but you can't just forget all filtered out entries.
>
> >
> >         I agree that "locking" should not be exposed with optimistic
> >         transactions.
> >
> >
> >     Yeah I can't find a good way to do this really and it seems to be
> >     opposite of what optimistic transactions are.
> >
> >
> >         With pessimistic transactions, how do you expect to handle
> locking
> >         order? For regular operations, user is responsible for setting
> >         up some
> >         locking order in order to not get a deadlock. With pessimistic
> >         transaction, it's the cache itself who will order the calls.
> >         Also, if
> >         you lock anything that is read, you just end up locking
> >         everything (or,
> >         getting a deadlock). If you don't it's the same as issuing the
> >         lock and
> >         reading again (to check the locked value) - but you'd do that
> >         internally
> >         anyway. Therefore, I don't feel well about pessimistic
> >         transactions neither.
> >
> >
> >     The lock is done per key only for each invocation. There is no
> >     ordering as only one is obtained at a time before it goes to the
> >     next. If the user then acquires a lock for another key while in
> >     the Consumer this could cause a deadlock if the inverse occurs on
> >     a different thread/node, but this is on the user. It is the same
> >     as it is today really, except we do the read lock for them before
> >     invoking their Consumer.
> >
>
> In pessimistic mode, you should not release a lock before the end of the
> transaction.
>

I agree. I didn't discuss finer details, but what I have right now doesn't
work with an ongoing pessimistic transaction. And to be honest I am not
sure if this can work with an ongoing transaction. And even if it did it
would be horrendously horrendously slow since each remote node is
performing the Consumer but it would need to read from the originators
context (or have it copied over). Unless someone with more transaction
knowledge knows of a better way I personally feel using stream with lock
should run in its own dedicated transaction.


>
> >
> >         >
> >         > Another question is what does the API for this look like. I was
> >         > debating between 3 options myself:
> >         >
> >         > 1. AdvancedCache.forEachWithLock(BiConsumer<Cache,
> >         CacheEntry<K, V>>
> >         > consumer)
> >         >
> >         > This require the least amount of changes, however the user
> can't
> >         > customize certain parameters that CacheStream currently
> provides
> >         > (listed below - big one being filterKeys).
> >         >
> >         > 2. CacheStream.forEachWithLock(BiConsumer<Cache,
> >         CacheEntry<K, V>>
> >         > consumer)
> >         >
> >         > This method would only be allowed to be invoked on the
> >         Stream if no
> >         > other intermediate operations were invoked, otherwise an
> >         exception
> >         > would be thrown. This still gives us access to all of the
> >         CacheStream
> >         > methods that aren't on the Stream interface (ie.
> >         > sequentialDistribution, parallelDistribution, parallel,
> >         sequential,
> >         > filterKeys, filterKeySegments, distributedBatchSize,
> >         > disableRehashAware, timeout).
> >
> >         For both options, I don't like Cache being passed around. You
> >         should
> >         modify the CacheEntry (or some kind of view) directly.
> >
> >
> >     I don't know for sure if that is sufficient for the user.
> >     Sometimes they may modify another Cache given the value in this
> >     one for example, which they could access from the CacheManager of
> >     that Cache. Maybe Tristan knows more about some use cases.
> >
>
> Rather than guessing what could the user need, the Consumer could be CDI
> enabled.
>

I am not the biggest CDI fan, especially since the benefit of this is you
can use lambdas (that automatically become Serializable) and make very
concise code. But others can chime in here too.


>
> >
> >         Radim
> >
> >         >
> >         > 3. LockedStream<CacheEntry<K, V>> AdvancedCache.lockedStream()
> >         >
> >         > This requires the most changes, however the API would be the
> >         most
> >         > explicit. In this case the LockedStream would only have the
> >         methods on
> >         > it that are able to be invoked as noted above and forEach.
> >         >
> >         > I personally feel that #3 might be the cleanest, but obviously
> >         > requires adding more classes. Let me know what you guys
> >         think and if
> >         > you think the optimistic exclusion is acceptable.
> >         >
> >         > Thanks,
> >         >
> >         >  - Will
> >         >
> >         >
> >         > _______________________________________________
> >         > infinispan-dev mailing list
> >         > infinispan-dev at lists.jboss.org
> >         <mailto:infinispan-dev at lists.jboss.org>
> >         > https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >
> >
> >         --
> >         Radim Vansa <rvansa at redhat.com <mailto:rvansa at redhat.com>>
> >         JBoss Performance Team
> >
> >         _______________________________________________
> >         infinispan-dev mailing list
> >         infinispan-dev at lists.jboss.org
> >         <mailto:infinispan-dev at lists.jboss.org>
> >         https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >
> >
> >
> > _______________________________________________
> > infinispan-dev mailing list
> > infinispan-dev at lists.jboss.org
> > https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
>
> --
> Radim Vansa <rvansa at redhat.com>
> JBoss Performance Team
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/infinispan-dev/attachments/20170323/64096167/attachment-0001.html 

From dan.berindei at gmail.com  Thu Mar 23 09:52:44 2017
From: dan.berindei at gmail.com (Dan Berindei)
Date: Thu, 23 Mar 2017 15:52:44 +0200
Subject: [infinispan-dev] ConcurrentMap new methods implementation
In-Reply-To: <14e1f874-7ee4-a49f-3c24-de17d61ec89f@redhat.com>
References: <CAKypdvk6w24dukHXwckBoeCNxGiu+41L4nbXuMZrOu4iJXU77Q@mail.gmail.com>
	<14e1f874-7ee4-a49f-3c24-de17d61ec89f@redhat.com>
Message-ID: <CA+nfvwQ2KLC=7X219bw5brj5qkXNXaitzibXVYezXuJGd_F=+g@mail.gmail.com>

Ideally I'd like to properly support non-serializable lambdas, too, by
using conditional write operations and retrying if unsuccessful. If
that turns out to be too tricky, we could also support
non-serializable lambdas for local caches only, and throw an exception
if the cache is clustered.

I'm not sure if we should overload the method (like the streams API
does) or create new methods though, because Java 9 seems to have a
problem picking the appropriate overload:

http://ci.infinispan.org/viewLog.html?buildId=52816&tab=buildResultsDiv&buildTypeId=Infinispan_MasterHotspotJdk9

Cheers
Dan


On Thu, Mar 23, 2017 at 10:42 AM, Radim Vansa <rvansa at redhat.com> wrote:
> On 03/22/2017 08:46 PM, Katia Aresti wrote:
>> Hi all,
>>
>> Since Java 8, some new methods are available in the ConcurrentMap
>> interface: merge, compute, computeIfAbsent, computeIfPresent, forEach,
>> replaceAll. ConcurrentMap interface provides a default implementation.
>>
>> I'm working on https://issues.jboss.org/browse/ISPN-5728
>> <https://issues.jboss.org/browse/ISPN-5728> in order to provide the
>> infinispan specific implementation. The issue here is that to make it
>> work, these lambdas must be Serializables, so actual code using these
>> methods and not passing serializables lambdas will break.
>>
>> I see two possibilities here, please fell free to suggest any other idea.
>>
>> 1) Override the default implementation and specify on the release that
>> all the lambdas must be serializables from now on ... ?
>> 2)  Leave the implementation of the default methods as they are and
>> provide new methods implemented the infinispan way :
>
> To be consistent with Streams API, we should assume that the lambda is
> serializable (and fail as gracefully as we can if it's not and should
> be) and provide the SerializableX overload method to make the lambdas
> serializable when the lambda is used directly on Cache. I wouldn't leave
> the default implementation, because:
>
> 1) it's prone to user error, particularly during refactoring. Switching
> from functional to default implementation would introduce a regression
> in his app's performance
> 2) For local caches or local-mode operations, the lambda doesn't need to
> be serializable
>
> Radim
>
>> V compute(K key, Vcompute(K key,
>>                    SerializableBiFunction<?super K, ?super V, ?extends V> remappingFunction)
>>
>> What do you think ?
>>
>> -- Katia
>>
>>
>> _______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
>
> --
> Radim Vansa <rvansa at redhat.com>
> JBoss Performance Team
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev

From dan.berindei at gmail.com  Thu Mar 23 10:33:24 2017
From: dan.berindei at gmail.com (Dan Berindei)
Date: Thu, 23 Mar 2017 16:33:24 +0200
Subject: [infinispan-dev] Stream operations under lock
In-Reply-To: <914f7208-ae02-7d9b-cf0b-6f5feb332609@redhat.com>
References: <CA+YCuUM3h0H+ZDe1ZvK5mOuXnLB9bjjP+CkJg_M8eyGKC+jRHA@mail.gmail.com>
	<d7e8a644-cb77-3e2a-f17b-79ceb70b2d85@redhat.com>
	<CA+YCuUOMttxZqxSpFzvPu_Q0sFS_W0P9Wh062UmSWGWewmgxaA@mail.gmail.com>
	<CA+YCuUML6s77x9JNk6m1o=c526HK6porbaymNoFK=nmj7Ny24w@mail.gmail.com>
	<914f7208-ae02-7d9b-cf0b-6f5feb332609@redhat.com>
Message-ID: <CA+nfvwRsihnU93juiboxiwEAaDQXTKPvg5ofUj=Ypa5JvkQiWw@mail.gmail.com>

On Wed, Mar 22, 2017 at 11:51 AM, Radim Vansa <rvansa at redhat.com> wrote:
> On 03/21/2017 06:50 PM, William Burns wrote:
>>
>>
>> On Tue, Mar 21, 2017 at 1:42 PM William Burns <mudokonman at gmail.com
>> <mailto:mudokonman at gmail.com>> wrote:
>>
>>     On Tue, Mar 21, 2017 at 12:53 PM Radim Vansa <rvansa at redhat.com
>>     <mailto:rvansa at redhat.com>> wrote:
>>
>>         On 03/21/2017 04:37 PM, William Burns wrote:
>>         > Some users have expressed the need to have some sort of forEach
>>         > operation that is performed where the Consumer is called
>>         while holding
>>         > the lock for the given key and subsequently released after the
>>         > Consumer operation completes.
>>
>>         Seconding Dan's question - is that intended to be able to
>>         modify the
>>         entry? In my opinion, sending a function that will work on the
>>         ReadWriteEntryView directly to the node is the only reasonable
>>         request.
>>         I wouldn't like to see blocking operations in there.
>>
>>
>>     Hrmm the user can use the FunctionalMap interface for this then it
>>     seems? I wonder if this should just be the going in API. I will
>>     need to discuss with Galder the semantics of the evalAll/evalMany
>>     methods.
>>
>>
>> Actually looking at evalAll it seems it doesn't scale as it keeps all
>> entries in memory at once, so this is only for caches with a limited
>> amount of entries.
>
> Don't look into the implementation; I think Galder has focused more on
> the API side than having optimal implementation. IMO there's no reason
> evalAll should load all the entries into memory in non-transactional mode.
>

I'm pretty sure we do need to load all the entries in order to provide
REPEATABLE_READ isolation.

>>
>>         >
>>         > Due to the nature of how streams work with retries and
>>         performing the
>>         > operation on the primary owner, this works out quite well
>>         with forEach
>>         > to be done in an efficient way.
>>         >
>>         > The problem is that this only really works well with non tx and
>>         > pessimistic tx. This obviously leaves out optimistic tx,
>>         which at
>>         > first I was a little worried about. But after thinking about
>>         it more,
>>         > this prelocking and optimistic tx don't really fit that well
>>         together
>>         > anyways. So I am thinking whenever this operation is
>>         performed it
>>         > would throw an exception not letting the user use this
>>         feature in
>>         > optimistic transactions.
>>
>>         How exactly reading streams interacts with transactions? Does
>>         it wrap
>>         read entries into context? This would be a scalability issue.
>>
>>
>>     It doesn't wrap read entries into the context for that exact
>>     reason. It does however use existing entries in the context to
>>     override ones in memory/store.
>>
>
> Uuh, so you end up with a copy of the cache in single invocation
> context, without any means to flush it. I think that we need add
> InvocationContext.current().forget(key) API (throwing exception if the
> entry was modified) or something like that, even for the regular
> streams. Maybe an override for filter methods, too, because you want to
> pass a nice predicate, but you can't just forget all filtered out entries.
>

I think Will said he *doesn't* want to wrap the entries read by the consumer :)

IMO there's no "good" way to provide repeatable read isolation for a
transaction that reads all the keys in the cache, so this API should
create a separate transaction for each entry. I wouldn't try to make
the consumers see the current transaction's modifications if started
from a transaction either, I'd throw an exception if started from a
transaction instead.

>>
>>         I agree that "locking" should not be exposed with optimistic
>>         transactions.
>>
>>
>>     Yeah I can't find a good way to do this really and it seems to be
>>     opposite of what optimistic transactions are.
>>

Ok, the name forEachWithLock doesn't really fit with optimistic
locking, but I think with a more neutral name it could work for
optimistic caches as well.

>>
>>         With pessimistic transactions, how do you expect to handle locking
>>         order? For regular operations, user is responsible for setting
>>         up some
>>         locking order in order to not get a deadlock. With pessimistic
>>         transaction, it's the cache itself who will order the calls.
>>         Also, if
>>         you lock anything that is read, you just end up locking
>>         everything (or,
>>         getting a deadlock). If you don't it's the same as issuing the
>>         lock and
>>         reading again (to check the locked value) - but you'd do that
>>         internally
>>         anyway. Therefore, I don't feel well about pessimistic
>>         transactions neither.
>>
>>
>>     The lock is done per key only for each invocation. There is no
>>     ordering as only one is obtained at a time before it goes to the
>>     next. If the user then acquires a lock for another key while in
>>     the Consumer this could cause a deadlock if the inverse occurs on
>>     a different thread/node, but this is on the user. It is the same
>>     as it is today really, except we do the read lock for them before
>>     invoking their Consumer.
>>
>
> In pessimistic mode, you should not release a lock before the end of the
> transaction.
>

Exactly. Each consumer needs to have its own transaction, otherwise
the transaction's lockedKeys collection would have to grow to include
all the keys in the cache.

>>
>>         >
>>         > Another question is what does the API for this look like. I was
>>         > debating between 3 options myself:
>>         >
>>         > 1. AdvancedCache.forEachWithLock(BiConsumer<Cache,
>>         CacheEntry<K, V>>
>>         > consumer)
>>         >
>>         > This require the least amount of changes, however the user can't
>>         > customize certain parameters that CacheStream currently provides
>>         > (listed below - big one being filterKeys).
>>         >
>>         > 2. CacheStream.forEachWithLock(BiConsumer<Cache,
>>         CacheEntry<K, V>>
>>         > consumer)
>>         >
>>         > This method would only be allowed to be invoked on the
>>         Stream if no
>>         > other intermediate operations were invoked, otherwise an
>>         exception
>>         > would be thrown. This still gives us access to all of the
>>         CacheStream
>>         > methods that aren't on the Stream interface (ie.
>>         > sequentialDistribution, parallelDistribution, parallel,
>>         sequential,
>>         > filterKeys, filterKeySegments, distributedBatchSize,
>>         > disableRehashAware, timeout).
>>
>>         For both options, I don't like Cache being passed around. You
>>         should
>>         modify the CacheEntry (or some kind of view) directly.
>>
>>
>>     I don't know for sure if that is sufficient for the user.
>>     Sometimes they may modify another Cache given the value in this
>>     one for example, which they could access from the CacheManager of
>>     that Cache. Maybe Tristan knows more about some use cases.
>>
>
> Rather than guessing what could the user need, the Consumer could be CDI
> enabled.
>

If the user actually needs to work with more than one entry at a time,
I think it would be much cleaner for him to use regular forEach() and
start an explicit transaction in the consumer.

>>
>>         Radim
>>
>>         >
>>         > 3. LockedStream<CacheEntry<K, V>> AdvancedCache.lockedStream()
>>         >
>>         > This requires the most changes, however the API would be the
>>         most
>>         > explicit. In this case the LockedStream would only have the
>>         methods on
>>         > it that are able to be invoked as noted above and forEach.
>>         >
>>         > I personally feel that #3 might be the cleanest, but obviously
>>         > requires adding more classes. Let me know what you guys
>>         think and if
>>         > you think the optimistic exclusion is acceptable.
>>         >
>>         > Thanks,
>>         >
>>         >  - Will
>>         >

From dan.berindei at gmail.com  Thu Mar 23 10:43:42 2017
From: dan.berindei at gmail.com (Dan Berindei)
Date: Thu, 23 Mar 2017 16:43:42 +0200
Subject: [infinispan-dev] Deprecate collection methods on DataContainer
	interface
In-Reply-To: <CA+YCuUOR1WNxoXzgRZX+aQmE=eFdzMO=beCLSGG+L34VA8+ZXg@mail.gmail.com>
References: <CA+YCuUOR1WNxoXzgRZX+aQmE=eFdzMO=beCLSGG+L34VA8+ZXg@mail.gmail.com>
Message-ID: <CA+nfvwQ=5_WxnYC68XxMv-6Vq+PxjeoQKr2LkgVxCxo5ww+Hug@mail.gmail.com>

You have my vote :)

Dan


On Thu, Mar 23, 2017 at 3:26 PM, William Burns <mudokonman at gmail.com> wrote:
> Recently while looking at the DataContainer interface and implementation it
> was noticed that the entrySet, values and keySet methods that return
> collections are very inconsistent. The big issue is whether these methods
> return expired entries or not and if they are backing or copies etc. It was
> suggested [1] to deprecate these. I am all for this. I was looking at the
> references to these methods and they all can be replace with using the
> proper iterator and forEachRemaining instead.
>
> I just wanted to confirm if everyone is alright with me deprecating the
> collection methods on DataContainer interface for 9.0 Final.
>
> [1]
> https://github.com/infinispan/infinispan/pull/5011#pullrequestreview-28599145
>
> Thanks,
>
>  - Will
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev

From mudokonman at gmail.com  Thu Mar 23 11:41:13 2017
From: mudokonman at gmail.com (William Burns)
Date: Thu, 23 Mar 2017 15:41:13 +0000
Subject: [infinispan-dev] ConcurrentMap new methods implementation
In-Reply-To: <CA+nfvwQ2KLC=7X219bw5brj5qkXNXaitzibXVYezXuJGd_F=+g@mail.gmail.com>
References: <CAKypdvk6w24dukHXwckBoeCNxGiu+41L4nbXuMZrOu4iJXU77Q@mail.gmail.com>
	<14e1f874-7ee4-a49f-3c24-de17d61ec89f@redhat.com>
	<CA+nfvwQ2KLC=7X219bw5brj5qkXNXaitzibXVYezXuJGd_F=+g@mail.gmail.com>
Message-ID: <CA+YCuUOENyab3JwrMJ105x=Tk_-_WW=_VmLOYDMX_n5W2zAtAA@mail.gmail.com>

On Thu, Mar 23, 2017 at 11:10 AM Dan Berindei <dan.berindei at gmail.com>
wrote:

> Ideally I'd like to properly support non-serializable lambdas, too, by
> using conditional write operations and retrying if unsuccessful. If
> that turns out to be too tricky, we could also support
> non-serializable lambdas for local caches only, and throw an exception
> if the cache is clustered.
>

The thing is the user could always do this as well. They would basically do
the replace themselves and retry just like pre Java 8. To me it seems the
default way these methods behave should be the most performant for the
underlying implementation. I would say even in the local method we should
perform the lambda under lock.

However I was thinking it might be nice to have a way to choose which, but
I don't know if adding more overrides/almost same name methods is the
answer. I wonder if we add a new FLAG maybe? Prefer non serializable or
something?


>
> I'm not sure if we should overload the method (like the streams API
> does) or create new methods though, because Java 9 seems to have a
> problem picking the appropriate overload:
>
>
> http://ci.infinispan.org/viewLog.html?buildId=52816&tab=buildResultsDiv&buildTypeId=Infinispan_MasterHotspotJdk9


This seems like a bug in the compiler to me. Unless they have changed the
spec definition they detailed in Java 8
https://docs.oracle.com/javase/specs/jls/se8/html/jls-15.html#jls-15.12.2.5


>
>
> Cheers
> Dan
>
>
> On Thu, Mar 23, 2017 at 10:42 AM, Radim Vansa <rvansa at redhat.com> wrote:
> > On 03/22/2017 08:46 PM, Katia Aresti wrote:
> >> Hi all,
> >>
> >> Since Java 8, some new methods are available in the ConcurrentMap
> >> interface: merge, compute, computeIfAbsent, computeIfPresent, forEach,
> >> replaceAll. ConcurrentMap interface provides a default implementation.
> >>
> >> I'm working on https://issues.jboss.org/browse/ISPN-5728
> >> <https://issues.jboss.org/browse/ISPN-5728> in order to provide the
> >> infinispan specific implementation. The issue here is that to make it
> >> work, these lambdas must be Serializables, so actual code using these
> >> methods and not passing serializables lambdas will break.
> >>
> >> I see two possibilities here, please fell free to suggest any other
> idea.
> >>
> >> 1) Override the default implementation and specify on the release that
> >> all the lambdas must be serializables from now on ... ?
> >> 2)  Leave the implementation of the default methods as they are and
> >> provide new methods implemented the infinispan way :
> >
> > To be consistent with Streams API, we should assume that the lambda is
> > serializable (and fail as gracefully as we can if it's not and should
> > be) and provide the SerializableX overload method to make the lambdas
> > serializable when the lambda is used directly on Cache. I wouldn't leave
> > the default implementation, because:
> >
> > 1) it's prone to user error, particularly during refactoring. Switching
> > from functional to default implementation would introduce a regression
> > in his app's performance
> > 2) For local caches or local-mode operations, the lambda doesn't need to
> > be serializable
> >
> > Radim
> >
> >> V compute(K key, Vcompute(K key,
> >>                    SerializableBiFunction<?super K, ?super V, ?extends
> V> remappingFunction)
> >>
> >> What do you think ?
> >>
> >> -- Katia
> >>
> >>
> >> _______________________________________________
> >> infinispan-dev mailing list
> >> infinispan-dev at lists.jboss.org
> >> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >
> >
> > --
> > Radim Vansa <rvansa at redhat.com>
> > JBoss Performance Team
> >
> > _______________________________________________
> > infinispan-dev mailing list
> > infinispan-dev at lists.jboss.org
> > https://lists.jboss.org/mailman/listinfo/infinispan-dev
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/infinispan-dev/attachments/20170323/f18738de/attachment-0001.html 

From rvansa at redhat.com  Thu Mar 23 11:50:43 2017
From: rvansa at redhat.com (Radim Vansa)
Date: Thu, 23 Mar 2017 16:50:43 +0100
Subject: [infinispan-dev] ConcurrentMap new methods implementation
In-Reply-To: <CA+nfvwQ2KLC=7X219bw5brj5qkXNXaitzibXVYezXuJGd_F=+g@mail.gmail.com>
References: <CAKypdvk6w24dukHXwckBoeCNxGiu+41L4nbXuMZrOu4iJXU77Q@mail.gmail.com>
	<14e1f874-7ee4-a49f-3c24-de17d61ec89f@redhat.com>
	<CA+nfvwQ2KLC=7X219bw5brj5qkXNXaitzibXVYezXuJGd_F=+g@mail.gmail.com>
Message-ID: <55a7d6c1-ef81-e013-1977-0aa408eafc0a@redhat.com>

On 03/23/2017 02:52 PM, Dan Berindei wrote:
> Ideally I'd like to properly support non-serializable lambdas, too, by
> using conditional write operations and retrying if unsuccessful. If
> that turns out to be too tricky, we could also support
> non-serializable lambdas for local caches only, and throw an exception
> if the cache is clustered.

At least we should voice a warning log if the operation requires to be 
handled using conditional ops.

>
> I'm not sure if we should overload the method (like the streams API
> does) or create new methods though, because Java 9 seems to have a
> problem picking the appropriate overload:
>
> http://ci.infinispan.org/viewLog.html?buildId=52816&tab=buildResultsDiv&buildTypeId=Infinispan_MasterHotspotJdk9

:-/ Have you checked spec if this is a bug or if we have used 
unsupported behaviour? Anyway, until we figure out how to sort this out, 
we should keep the Streams API and Cache API in sync.
I think that having this as compilation issue in user code at least 
forces user to define the lambda as serializable, it's not necessarily a 
bad thing (though annoying).

>
> Cheers
> Dan
>
>
> On Thu, Mar 23, 2017 at 10:42 AM, Radim Vansa <rvansa at redhat.com> wrote:
>> On 03/22/2017 08:46 PM, Katia Aresti wrote:
>>> Hi all,
>>>
>>> Since Java 8, some new methods are available in the ConcurrentMap
>>> interface: merge, compute, computeIfAbsent, computeIfPresent, forEach,
>>> replaceAll. ConcurrentMap interface provides a default implementation.
>>>
>>> I'm working on https://issues.jboss.org/browse/ISPN-5728
>>> <https://issues.jboss.org/browse/ISPN-5728> in order to provide the
>>> infinispan specific implementation. The issue here is that to make it
>>> work, these lambdas must be Serializables, so actual code using these
>>> methods and not passing serializables lambdas will break.
>>>
>>> I see two possibilities here, please fell free to suggest any other idea.
>>>
>>> 1) Override the default implementation and specify on the release that
>>> all the lambdas must be serializables from now on ... ?
>>> 2)  Leave the implementation of the default methods as they are and
>>> provide new methods implemented the infinispan way :
>> To be consistent with Streams API, we should assume that the lambda is
>> serializable (and fail as gracefully as we can if it's not and should
>> be) and provide the SerializableX overload method to make the lambdas
>> serializable when the lambda is used directly on Cache. I wouldn't leave
>> the default implementation, because:
>>
>> 1) it's prone to user error, particularly during refactoring. Switching
>> from functional to default implementation would introduce a regression
>> in his app's performance
>> 2) For local caches or local-mode operations, the lambda doesn't need to
>> be serializable
>>
>> Radim
>>
>>> V compute(K key, Vcompute(K key,
>>>                     SerializableBiFunction<?super K, ?super V, ?extends V> remappingFunction)
>>>
>>> What do you think ?
>>>
>>> -- Katia
>>>
>>>
>>> _______________________________________________
>>> infinispan-dev mailing list
>>> infinispan-dev at lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>
>> --
>> Radim Vansa <rvansa at redhat.com>
>> JBoss Performance Team
>>
>> _______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev


-- 
Radim Vansa <rvansa at redhat.com>
JBoss Performance Team


From galder at redhat.com  Thu Mar 23 09:40:00 2017
From: galder at redhat.com (=?utf-8?Q?Galder_Zamarre=C3=B1o?=)
Date: Thu, 23 Mar 2017 14:40:00 +0100
Subject: [infinispan-dev] NoSuchMethodError in Spark simple tutorial
Message-ID: <4FB5C903-3A8C-4D94-9E1E-C1D2512C1ECF@redhat.com>

Hey guys,

The Spark Java simple tutorial [1] tutorial does not work as is due to some dependencies mix up.

If you run that class, with the dependencies defined in the pom.xml, you get:

java.lang.NoSuchMethodError: org.infinispan.client.hotrod.logging.Log.tracef(Ljava/lang/Throwable;Ljava/lang/String;II)V

The dependencies are defined as:

<dependency>
  <groupId>org.infinispan</groupId>
  <artifactId>infinispan-spark_2.11</artifactId>
  <version>0.4</version>
</dependency>
...

Looking at the dependency tree, I see this:

[INFO] +- org.infinispan:infinispan-spark_2.11:jar:0.4:compile
[INFO] |  +- org.infinispan:infinispan-client-hotrod:jar:9.0.0.Alpha4:compile
...
[INFO] |  +- org.infinispan.protostream:protostream:jar:3.0.5.Final:compile
[INFO] |  |  +- org.jboss.logging:jboss-logging:jar:3.1.4.GA:compile

That logging jar seems an old one, it should be 3.3.x. I worked around this by doing:

<dependency>
  <groupId>org.infinispan</groupId>
  <artifactId>infinispan-spark_2.11</artifactId>
  <version>${version.spark-connector}</version>
  <exclusions>
    <exclusion>
      <groupId>org.jboss.logging</groupId>
      <artifactId>jboss-logging</artifactId>
    </exclusion>
  </exclusions>
</dependency>
<dependency>
  <groupId>org.jboss.logging</groupId>
  <artifactId>jboss-logging</artifactId>
  <version>${version.jboss.logging}</version>
</dependency>

@Adrian Are the dependencies of the latest protostream versions in line with the infinispan hot rod client ones?

@Gustavo, Once there's a client version which depends on a protostream version that fixes this (if there's not one already...), can you release a 0.5 alpha/beta/cr version?

Cheers,

[1] http://infinispan.org/tutorials/simple/spark/
--
Galder Zamarre?o
Infinispan, Red Hat



From gustavo at infinispan.org  Fri Mar 24 04:31:58 2017
From: gustavo at infinispan.org (Gustavo Fernandes)
Date: Fri, 24 Mar 2017 08:31:58 +0000
Subject: [infinispan-dev] NoSuchMethodError in Spark simple tutorial
In-Reply-To: <4FB5C903-3A8C-4D94-9E1E-C1D2512C1ECF@redhat.com>
References: <4FB5C903-3A8C-4D94-9E1E-C1D2512C1ECF@redhat.com>
Message-ID: <CAH8Ud1FQHQMxM132CvjHv41OLkAEHKA7fGUbr8jVz32iz9EQeA@mail.gmail.com>

This was fixed [1]

Regarding the tutorial, I could run it "as is":

git clone https://github.com/infinispan/infinispan-simple-tutorials.git
cd infinispan-simple-tutorials
mvn clean install
cd spark
mvn exec:exec


How did you reproduced the issue?


[1] https://issues.jboss.org/browse/ISPRK-33


Thanks,
Gustavo

On Thu, Mar 23, 2017 at 1:40 PM, Galder Zamarre?o <galder at redhat.com> wrote:

> Hey guys,
>
> The Spark Java simple tutorial [1] tutorial does not work as is due to
> some dependencies mix up.
>
> If you run that class, with the dependencies defined in the pom.xml, you
> get:
>
> java.lang.NoSuchMethodError: org.infinispan.client.hotrod.
> logging.Log.tracef(Ljava/lang/Throwable;Ljava/lang/String;II)V
>
> The dependencies are defined as:
>
> <dependency>
>   <groupId>org.infinispan</groupId>
>   <artifactId>infinispan-spark_2.11</artifactId>
>   <version>0.4</version>
> </dependency>
> ...
>
> Looking at the dependency tree, I see this:
>
> [INFO] +- org.infinispan:infinispan-spark_2.11:jar:0.4:compile
> [INFO] |  +- org.infinispan:infinispan-client-hotrod:jar:9.0.0.
> Alpha4:compile
> ...
> [INFO] |  +- org.infinispan.protostream:protostream:jar:3.0.5.Final:
> compile
> [INFO] |  |  +- org.jboss.logging:jboss-logging:jar:3.1.4.GA:compile
>
> That logging jar seems an old one, it should be 3.3.x. I worked around
> this by doing:
>
> <dependency>
>   <groupId>org.infinispan</groupId>
>   <artifactId>infinispan-spark_2.11</artifactId>
>   <version>${version.spark-connector}</version>
>   <exclusions>
>     <exclusion>
>       <groupId>org.jboss.logging</groupId>
>       <artifactId>jboss-logging</artifactId>
>     </exclusion>
>   </exclusions>
> </dependency>
> <dependency>
>   <groupId>org.jboss.logging</groupId>
>   <artifactId>jboss-logging</artifactId>
>   <version>${version.jboss.logging}</version>
> </dependency>
>
> @Adrian Are the dependencies of the latest protostream versions in line
> with the infinispan hot rod client ones?
>
> @Gustavo, Once there's a client version which depends on a protostream
> version that fixes this (if there's not one already...), can you release a
> 0.5 alpha/beta/cr version?
>
> Cheers,
>
> [1] http://infinispan.org/tutorials/simple/spark/
> --
> Galder Zamarre?o
> Infinispan, Red Hat
>
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/infinispan-dev/attachments/20170324/3e658022/attachment.html 

From ttarrant at redhat.com  Fri Mar 24 06:07:17 2017
From: ttarrant at redhat.com (Tristan Tarrant)
Date: Fri, 24 Mar 2017 11:07:17 +0100
Subject: [infinispan-dev] Deprecate collection methods on DataContainer
 interface
In-Reply-To: <CA+nfvwQ=5_WxnYC68XxMv-6Vq+PxjeoQKr2LkgVxCxo5ww+Hug@mail.gmail.com>
References: <CA+YCuUOR1WNxoXzgRZX+aQmE=eFdzMO=beCLSGG+L34VA8+ZXg@mail.gmail.com>
	<CA+nfvwQ=5_WxnYC68XxMv-6Vq+PxjeoQKr2LkgVxCxo5ww+Hug@mail.gmail.com>
Message-ID: <9c383e1c-3900-b949-575b-2cf49c550445@redhat.com>

+1

On 23/03/17 15:43, Dan Berindei wrote:
> You have my vote :)
> 
> Dan
> 
> 
> On Thu, Mar 23, 2017 at 3:26 PM, William Burns <mudokonman at gmail.com> wrote:
>> Recently while looking at the DataContainer interface and implementation it
>> was noticed that the entrySet, values and keySet methods that return
>> collections are very inconsistent. The big issue is whether these methods
>> return expired entries or not and if they are backing or copies etc. It was
>> suggested [1] to deprecate these. I am all for this. I was looking at the
>> references to these methods and they all can be replace with using the
>> proper iterator and forEachRemaining instead.
>>
>> I just wanted to confirm if everyone is alright with me deprecating the
>> collection methods on DataContainer interface for 9.0 Final.
>>
>> [1]
>> https://github.com/infinispan/infinispan/pull/5011#pullrequestreview-28599145
>>
>> Thanks,
>>
>>   - Will
>>
>> _______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> 

-- 
Tristan Tarrant
Infinispan Lead
JBoss, a division of Red Hat

From rory.odonnell at oracle.com  Fri Mar 24 06:46:58 2017
From: rory.odonnell at oracle.com (Rory O'Donnell)
Date: Fri, 24 Mar 2017 10:46:58 +0000
Subject: [infinispan-dev] JDK 9 EA Build 162 is available on java.net
Message-ID: <f7fe3220-6df1-ff71-37b4-dcb9c1cad84d@oracle.com>

Hi Galder,

*JDK 9 Early Access* b162 <https://jdk9.java.net/download/>  is 
available on java.net, summary of  changes are listed here 
<http://download.java.net/java/jdk9/changes/jdk-9+162.html>.

There is one fix for a bug reported by Open Source projects since the 
last availability email  :

  * b161 - JDK 8176265 Method overload resolution on a covariant base
    type doesn't work in 9

Other change that maybe of interest:

  * b162 - JDK 8176503 security-libs Disable SHA-1 TLS Server Certificates


*Better tools for adjusting to strong encapsulation -* please read Mark 
Reinhold's email on this topic [1]
*
* *Quality Outreach Report for March 2017 *is available [2], many thanks 
for your continued support
and welcome to the new projects!

***Schedule - **JDK 9 Rampdown Phase 2*: Proposal accepted [3].
The overall goal of this process is to ensure that we fix just the bugs 
that must be fixed in order to ensure a successful release.

Oracle's JRE and JDK Cryptographic Roadmap has been updated since last 
availability email [4]

Rgds,Rory

[1] http://mail.openjdk.java.net/pipermail/jigsaw-dev/2017-March/011763.html
[2] 
https://wiki.openjdk.java.net/display/quality/Quality+Outreach+report+March+2017
[3] http://mail.openjdk.java.net/pipermail/jdk9-dev/2017-March/005689.html
[4] https://www.java.com/en/jre-jdk-cryptoroadmap.html

-- 
Rgds,Rory O'Donnell
Quality Engineering Manager
Oracle EMEA , Dublin, Ireland

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/infinispan-dev/attachments/20170324/b1097e30/attachment-0001.html 

From rvansa at redhat.com  Mon Mar 27 04:58:02 2017
From: rvansa at redhat.com (Radim Vansa)
Date: Mon, 27 Mar 2017 10:58:02 +0200
Subject: [infinispan-dev] TestingUtil.shortTimeoutMillis
Message-ID: <3c77bc70-f732-2a46-2a01-0a97ed2f8277@redhat.com>

Hi,

I see that short timeout (500 ms) is an often cause of random failures 
in CI. The original issue (ISPN-4772) raised this from 100 to 500; could 
we increase this on CI even further, to, say, 3 seconds? It's 
configurable via a sys property

infinispan.test.shortTimeoutMillis

Radim

-- 
Radim Vansa <rvansa at redhat.com>
JBoss Performance Team


From slaskawi at redhat.com  Mon Mar 27 04:59:59 2017
From: slaskawi at redhat.com (Sebastian Laskawiec)
Date: Mon, 27 Mar 2017 08:59:59 +0000
Subject: [infinispan-dev] TestingUtil.shortTimeoutMillis
In-Reply-To: <3c77bc70-f732-2a46-2a01-0a97ed2f8277@redhat.com>
References: <3c77bc70-f732-2a46-2a01-0a97ed2f8277@redhat.com>
Message-ID: <CAA9R9O536c93v05=jM9yDwbe=yZWebnynpw-Uo+ps5YzG9xNSA@mail.gmail.com>

Sounds good...

Sadly our build infrastructure is not very fast...

On Mon, Mar 27, 2017 at 10:58 AM Radim Vansa <rvansa at redhat.com> wrote:

> Hi,
>
> I see that short timeout (500 ms) is an often cause of random failures
> in CI. The original issue (ISPN-4772) raised this from 100 to 500; could
> we increase this on CI even further, to, say, 3 seconds? It's
> configurable via a sys property
>
> infinispan.test.shortTimeoutMillis
>
> Radim
>
> --
> Radim Vansa <rvansa at redhat.com>
> JBoss Performance Team
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/infinispan-dev/attachments/20170327/9e010d9f/attachment.html 

From dan.berindei at gmail.com  Mon Mar 27 05:25:14 2017
From: dan.berindei at gmail.com (Dan Berindei)
Date: Mon, 27 Mar 2017 12:25:14 +0300
Subject: [infinispan-dev] TestingUtil.shortTimeoutMillis
In-Reply-To: <CAA9R9O536c93v05=jM9yDwbe=yZWebnynpw-Uo+ps5YzG9xNSA@mail.gmail.com>
References: <3c77bc70-f732-2a46-2a01-0a97ed2f8277@redhat.com>
	<CAA9R9O536c93v05=jM9yDwbe=yZWebnynpw-Uo+ps5YzG9xNSA@mail.gmail.com>
Message-ID: <CA+nfvwQhd6yHDwHFKMM-nqAswccKjAg0V-jMfXXBZQ7H+dqHkQ@mail.gmail.com>

Radim, are you seeing this in the regular builds too, or just in the
TRACE build?

Dan


On Mon, Mar 27, 2017 at 11:59 AM, Sebastian Laskawiec
<slaskawi at redhat.com> wrote:
> Sounds good...
>
> Sadly our build infrastructure is not very fast...
>
> On Mon, Mar 27, 2017 at 10:58 AM Radim Vansa <rvansa at redhat.com> wrote:
>>
>> Hi,
>>
>> I see that short timeout (500 ms) is an often cause of random failures
>> in CI. The original issue (ISPN-4772) raised this from 100 to 500; could
>> we increase this on CI even further, to, say, 3 seconds? It's
>> configurable via a sys property
>>
>> infinispan.test.shortTimeoutMillis
>>
>> Radim
>>
>> --
>> Radim Vansa <rvansa at redhat.com>
>> JBoss Performance Team
>>
>> _______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev

From slaskawi at redhat.com  Mon Mar 27 05:33:13 2017
From: slaskawi at redhat.com (Sebastian Laskawiec)
Date: Mon, 27 Mar 2017 09:33:13 +0000
Subject: [infinispan-dev] Branching proposal
Message-ID: <CAA9R9O6X3kOu3h4=_2z6RZKExYC30B0JiJMvEYQ3mu9HGq20Rw@mail.gmail.com>

Hey!

We are about to start working on 9.1.x and 9.2.y branches so I would like
to propose alternative merging strategy.

Our current workflow looks like this:

X - new commit
X` - cherry pick to maintenance branch
--+-------------------+-------X----- master
  |                    \------X`---- 9.2.x
  \---------------------------X``--- 9.1.x

Each commit needs to be reviewed in master branch and backported to the
maintenance branches. From maintenance perspective this is a bit painful,
since in above example we need to get 3 times through PR queue. Also it's
worth to mention that X is not X` nor X``. Cherry-picking creates a copy of
a commit. This makes some useful tricks (like git tag --contains <sha1>) a
bit harder to use. Finally, this approach allows the codebase to diverge
from maintenance branches very fast (someone might just forget to backport
some of the refactoring stuff).

The proposal:

X, Y - new commits
/ - merge commits
--+---------+------/----/--- master
  |          \----/---Y/---- 9.2.x
  \-------------X/---------- 9.1.x

With the proposal, a developer should always implement a given feature in
the lowest possible maintenance branch. Then we will run a set of merges
from 9.1.x into 9.2.x and finally into master. The biggest advantage of
this approach is that given functionality (identified by a commit) will
have the same SHA1 for all branches. This will allow all tools like
(mentioned before) `git tag --contains <sha1>` to work. There are also some
further implications of this approach:

   - Merging commits should be performed very often (even automatically in
   the night (if merged without any problems)).
   - After releasing each maintenance release, someone will need to do a
   merge with strategy `ours` (`git merge -s ours upstream/9.2.x`). This way
   we will not have to solve version conflicts in poms.
   - Since there is no nice way to rebase a merge commit, they should be
   pushed directly into the master branch (without review, without CI). After
   the merge, HEAD will change and CI will automatically pick the build.
   Remember, merges should be done very often. So I assume there won't be any
   problems most of the times.
   - Finally, with this approach the code diverges slight slower (at least
   from my experience). Mainly because we don't need to remember to
   cherry-pick individual commits. They are automatically "taken" by a merge.

>From my past experience, this strategy works pretty nice and can be almost
fully automated. It significantly lowers the maintenance pain around
cherry-picks. However there is nothing for free, and we would need to get
used to pushing merged directly into master (which is fine to me but some
of you might not like it).

Thanks,
Sebastian
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/infinispan-dev/attachments/20170327/f05f5456/attachment.html 

From rvansa at redhat.com  Mon Mar 27 06:33:23 2017
From: rvansa at redhat.com (Radim Vansa)
Date: Mon, 27 Mar 2017 12:33:23 +0200
Subject: [infinispan-dev] Branching proposal
In-Reply-To: <CAA9R9O6X3kOu3h4=_2z6RZKExYC30B0JiJMvEYQ3mu9HGq20Rw@mail.gmail.com>
References: <CAA9R9O6X3kOu3h4=_2z6RZKExYC30B0JiJMvEYQ3mu9HGq20Rw@mail.gmail.com>
Message-ID: <f9406b3d-af44-1627-7037-15ff080d6e3f@redhat.com>

If you can't merge a commit (based on 9.0.x) to master clearly, do you 
need to file another PR anyway? Then the lag to get some code to master 
increases a lot. I am not sure how useful is git tag --contains <sha1> 
if you cannot be sure that you'll find all occurrences due to this kind 
of issues.

R.

On 03/27/2017 11:33 AM, Sebastian Laskawiec wrote:
> Hey!
>
> We are about to start working on 9.1.x and 9.2.y branches so I would 
> like to propose alternative merging strategy.
>
> Our current workflow looks like this:
>
> X - new commit
> X` - cherry pick to maintenance branch
> --+-------------------+-------X----- master
>   |                    \------X`---- 9.2.x
>   \---------------------------X``--- 9.1.x
>
> Each commit needs to be reviewed in master branch and backported to 
> the maintenance branches. From maintenance perspective this is a bit 
> painful, since in above example we need to get 3 times through PR 
> queue. Also it's worth to mention that X is not X` nor X``. 
> Cherry-picking creates a copy of a commit. This makes some useful 
> tricks (like git tag --contains <sha1>) a bit harder to use. Finally, 
> this approach allows the codebase to diverge from maintenance branches 
> very fast (someone might just forget to backport some of the 
> refactoring stuff).
>
> The proposal:
>
> X, Y - new commits
> / - merge commits
> --+---------+------/----/--- master
>   |          \----/---Y/---- 9.2.x
>   \-------------X/---------- 9.1.x
>
> With the proposal, a developer should always implement a given feature 
> in the lowest possible maintenance branch. Then we will run a set of 
> merges from 9.1.x into 9.2.x and finally into master. The biggest 
> advantage of this approach is that given functionality (identified by 
> a commit) will have the same SHA1 for all branches. This will allow 
> all tools like (mentioned before) `git tag --contains <sha1>` to work. 
> There are also some further implications of this approach:
>
>   * Merging commits should be performed very often (even automatically
>     in the night (if merged without any problems)).
>   * After releasing each maintenance release, someone will need to do
>     a merge with strategy `ours` (`git merge -s ours upstream/9.2.x`).
>     This way we will not have to solve version conflicts in poms.
>   * Since there is no nice way to rebase a merge commit, they should
>     be pushed directly into the master branch (without review, without
>     CI). After the merge, HEAD will change and CI will
>     automatically pick the build. Remember, merges should be done very
>     often. So I assume there won't be any problems most of the times.
>   * Finally, with this approach the code diverges slight slower (at
>     least from my experience). Mainly because we don't need to
>     remember to cherry-pick individual commits. They are automatically
>     "taken" by a merge.
>
> From my past experience, this strategy works pretty nice and can be 
> almost fully automated. It significantly lowers the maintenance pain 
> around cherry-picks. However there is nothing for free, and we would 
> need to get used to pushing merged directly into master (which is fine 
> to me but some of you might not like it).
>
> Thanks,
> Sebastian
>
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev


-- 
Radim Vansa <rvansa at redhat.com>
JBoss Performance Team


From rvansa at redhat.com  Mon Mar 27 06:35:36 2017
From: rvansa at redhat.com (Radim Vansa)
Date: Mon, 27 Mar 2017 12:35:36 +0200
Subject: [infinispan-dev] TestingUtil.shortTimeoutMillis
In-Reply-To: <CA+nfvwQhd6yHDwHFKMM-nqAswccKjAg0V-jMfXXBZQ7H+dqHkQ@mail.gmail.com>
References: <3c77bc70-f732-2a46-2a01-0a97ed2f8277@redhat.com>
	<CAA9R9O536c93v05=jM9yDwbe=yZWebnynpw-Uo+ps5YzG9xNSA@mail.gmail.com>
	<CA+nfvwQhd6yHDwHFKMM-nqAswccKjAg0V-jMfXXBZQ7H+dqHkQ@mail.gmail.com>
Message-ID: <fb1cab12-2d39-a017-5aef-4ab1a3e94700@redhat.com>

In regular ones [1] - see SyncLockingTest

Btw., how do I enable TRACE build in CI?


[1] http://ci.infinispan.org/viewLog.html?buildId=53604&buildTypeId=bt9

On 03/27/2017 11:25 AM, Dan Berindei wrote:
> Radim, are you seeing this in the regular builds too, or just in the
> TRACE build?
>
> Dan
>
>
> On Mon, Mar 27, 2017 at 11:59 AM, Sebastian Laskawiec
> <slaskawi at redhat.com> wrote:
>> Sounds good...
>>
>> Sadly our build infrastructure is not very fast...
>>
>> On Mon, Mar 27, 2017 at 10:58 AM Radim Vansa <rvansa at redhat.com> wrote:
>>> Hi,
>>>
>>> I see that short timeout (500 ms) is an often cause of random failures
>>> in CI. The original issue (ISPN-4772) raised this from 100 to 500; could
>>> we increase this on CI even further, to, say, 3 seconds? It's
>>> configurable via a sys property
>>>
>>> infinispan.test.shortTimeoutMillis
>>>
>>> Radim
>>>
>>> --
>>> Radim Vansa <rvansa at redhat.com>
>>> JBoss Performance Team
>>>
>>> _______________________________________________
>>> infinispan-dev mailing list
>>> infinispan-dev at lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>
>> _______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev


-- 
Radim Vansa <rvansa at redhat.com>
JBoss Performance Team


From slaskawi at redhat.com  Mon Mar 27 06:45:44 2017
From: slaskawi at redhat.com (Sebastian Laskawiec)
Date: Mon, 27 Mar 2017 10:45:44 +0000
Subject: [infinispan-dev] Branching proposal
In-Reply-To: <f9406b3d-af44-1627-7037-15ff080d6e3f@redhat.com>
References: <CAA9R9O6X3kOu3h4=_2z6RZKExYC30B0JiJMvEYQ3mu9HGq20Rw@mail.gmail.com>
	<f9406b3d-af44-1627-7037-15ff080d6e3f@redhat.com>
Message-ID: <CAA9R9O6WYL_YSZT1A8qhBqoiubZNSB+8D+n-s=2+P9wzvHxBig@mail.gmail.com>

>From my past experience, if a commit caused a conflict when merging, we
always asked the author to fix it and do the merge.

After a while it became a habit that each dev who submitted a code that
could result in conflicts, did all the merges.

On Mon, Mar 27, 2017 at 12:37 PM Radim Vansa <rvansa at redhat.com> wrote:

> If you can't merge a commit (based on 9.0.x) to master clearly, do you
> need to file another PR anyway? Then the lag to get some code to master
> increases a lot. I am not sure how useful is git tag --contains <sha1>
> if you cannot be sure that you'll find all occurrences due to this kind
> of issues.
>
> R.
>
> On 03/27/2017 11:33 AM, Sebastian Laskawiec wrote:
> > Hey!
> >
> > We are about to start working on 9.1.x and 9.2.y branches so I would
> > like to propose alternative merging strategy.
> >
> > Our current workflow looks like this:
> >
> > X - new commit
> > X` - cherry pick to maintenance branch
> > --+-------------------+-------X----- master
> >   |                    \------X`---- 9.2.x
> >   \---------------------------X``--- 9.1.x
> >
> > Each commit needs to be reviewed in master branch and backported to
> > the maintenance branches. From maintenance perspective this is a bit
> > painful, since in above example we need to get 3 times through PR
> > queue. Also it's worth to mention that X is not X` nor X``.
> > Cherry-picking creates a copy of a commit. This makes some useful
> > tricks (like git tag --contains <sha1>) a bit harder to use. Finally,
> > this approach allows the codebase to diverge from maintenance branches
> > very fast (someone might just forget to backport some of the
> > refactoring stuff).
> >
> > The proposal:
> >
> > X, Y - new commits
> > / - merge commits
> > --+---------+------/----/--- master
> >   |          \----/---Y/---- 9.2.x
> >   \-------------X/---------- 9.1.x
> >
> > With the proposal, a developer should always implement a given feature
> > in the lowest possible maintenance branch. Then we will run a set of
> > merges from 9.1.x into 9.2.x and finally into master. The biggest
> > advantage of this approach is that given functionality (identified by
> > a commit) will have the same SHA1 for all branches. This will allow
> > all tools like (mentioned before) `git tag --contains <sha1>` to work.
> > There are also some further implications of this approach:
> >
> >   * Merging commits should be performed very often (even automatically
> >     in the night (if merged without any problems)).
> >   * After releasing each maintenance release, someone will need to do
> >     a merge with strategy `ours` (`git merge -s ours upstream/9.2.x`).
> >     This way we will not have to solve version conflicts in poms.
> >   * Since there is no nice way to rebase a merge commit, they should
> >     be pushed directly into the master branch (without review, without
> >     CI). After the merge, HEAD will change and CI will
> >     automatically pick the build. Remember, merges should be done very
> >     often. So I assume there won't be any problems most of the times.
> >   * Finally, with this approach the code diverges slight slower (at
> >     least from my experience). Mainly because we don't need to
> >     remember to cherry-pick individual commits. They are automatically
> >     "taken" by a merge.
> >
> > From my past experience, this strategy works pretty nice and can be
> > almost fully automated. It significantly lowers the maintenance pain
> > around cherry-picks. However there is nothing for free, and we would
> > need to get used to pushing merged directly into master (which is fine
> > to me but some of you might not like it).
> >
> > Thanks,
> > Sebastian
> >
> >
> > _______________________________________________
> > infinispan-dev mailing list
> > infinispan-dev at lists.jboss.org
> > https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
>
> --
> Radim Vansa <rvansa at redhat.com>
> JBoss Performance Team
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/infinispan-dev/attachments/20170327/4d121ae8/attachment.html 

From rvansa at redhat.com  Mon Mar 27 06:59:02 2017
From: rvansa at redhat.com (Radim Vansa)
Date: Mon, 27 Mar 2017 12:59:02 +0200
Subject: [infinispan-dev] Branching proposal
In-Reply-To: <CAA9R9O6WYL_YSZT1A8qhBqoiubZNSB+8D+n-s=2+P9wzvHxBig@mail.gmail.com>
References: <CAA9R9O6X3kOu3h4=_2z6RZKExYC30B0JiJMvEYQ3mu9HGq20Rw@mail.gmail.com>
	<f9406b3d-af44-1627-7037-15ff080d6e3f@redhat.com>
	<CAA9R9O6WYL_YSZT1A8qhBqoiubZNSB+8D+n-s=2+P9wzvHxBig@mail.gmail.com>
Message-ID: <27b9795d-8908-8a41-e964-6a1ec9ee6f15@redhat.com>

On 03/27/2017 12:45 PM, Sebastian Laskawiec wrote:
> From my past experience, if a commit caused a conflict when merging, 
> we always asked the author to fix it and do the merge.

I don't understand. The PR should be filed against 9.0.x, there're no 
conflicts. Merging the same against master results in conflicts - where 
should I resolve those?

Another q: I decide to file a PR against 9.1, because I don't think it 
should be applied to 9.0. I get a review, but then someone explains that 
it should get to 9.0 as well. I can't change a target branch in GitHub's 
PR: should I close the PR with nice history of comments (some of them 
not addressed yet) and open another PR?

R.

>
> After a while it became a habit that each dev who submitted a code 
> that could result in conflicts, did all the merges.
>
> On Mon, Mar 27, 2017 at 12:37 PM Radim Vansa <rvansa at redhat.com 
> <mailto:rvansa at redhat.com>> wrote:
>
>     If you can't merge a commit (based on 9.0.x) to master clearly, do you
>     need to file another PR anyway? Then the lag to get some code to
>     master
>     increases a lot. I am not sure how useful is git tag --contains <sha1>
>     if you cannot be sure that you'll find all occurrences due to this
>     kind
>     of issues.
>
>     R.
>
>     On 03/27/2017 11:33 AM, Sebastian Laskawiec wrote:
>     > Hey!
>     >
>     > We are about to start working on 9.1.x and 9.2.y branches so I would
>     > like to propose alternative merging strategy.
>     >
>     > Our current workflow looks like this:
>     >
>     > X - new commit
>     > X` - cherry pick to maintenance branch
>     > --+-------------------+-------X----- master
>     >   |                    \------X`---- 9.2.x
>     >   \---------------------------X``--- 9.1.x
>     >
>     > Each commit needs to be reviewed in master branch and backported to
>     > the maintenance branches. From maintenance perspective this is a bit
>     > painful, since in above example we need to get 3 times through PR
>     > queue. Also it's worth to mention that X is not X` nor X``.
>     > Cherry-picking creates a copy of a commit. This makes some useful
>     > tricks (like git tag --contains <sha1>) a bit harder to use.
>     Finally,
>     > this approach allows the codebase to diverge from maintenance
>     branches
>     > very fast (someone might just forget to backport some of the
>     > refactoring stuff).
>     >
>     > The proposal:
>     >
>     > X, Y - new commits
>     > / - merge commits
>     > --+---------+------/----/--- master
>     >   |          \----/---Y/---- 9.2.x
>     >   \-------------X/---------- 9.1.x
>     >
>     > With the proposal, a developer should always implement a given
>     feature
>     > in the lowest possible maintenance branch. Then we will run a set of
>     > merges from 9.1.x into 9.2.x and finally into master. The biggest
>     > advantage of this approach is that given functionality
>     (identified by
>     > a commit) will have the same SHA1 for all branches. This will allow
>     > all tools like (mentioned before) `git tag --contains <sha1>` to
>     work.
>     > There are also some further implications of this approach:
>     >
>     >   * Merging commits should be performed very often (even
>     automatically
>     >     in the night (if merged without any problems)).
>     >   * After releasing each maintenance release, someone will need
>     to do
>     >     a merge with strategy `ours` (`git merge -s ours
>     upstream/9.2.x`).
>     >     This way we will not have to solve version conflicts in poms.
>     >   * Since there is no nice way to rebase a merge commit, they should
>     >     be pushed directly into the master branch (without review,
>     without
>     >     CI). After the merge, HEAD will change and CI will
>     >     automatically pick the build. Remember, merges should be
>     done very
>     >     often. So I assume there won't be any problems most of the
>     times.
>     >   * Finally, with this approach the code diverges slight slower (at
>     >     least from my experience). Mainly because we don't need to
>     >     remember to cherry-pick individual commits. They are
>     automatically
>     >     "taken" by a merge.
>     >
>     > From my past experience, this strategy works pretty nice and can be
>     > almost fully automated. It significantly lowers the maintenance pain
>     > around cherry-picks. However there is nothing for free, and we would
>     > need to get used to pushing merged directly into master (which
>     is fine
>     > to me but some of you might not like it).
>     >
>     > Thanks,
>     > Sebastian
>     >
>     >
>     > _______________________________________________
>     > infinispan-dev mailing list
>     > infinispan-dev at lists.jboss.org
>     <mailto:infinispan-dev at lists.jboss.org>
>     > https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
>
>     --
>     Radim Vansa <rvansa at redhat.com <mailto:rvansa at redhat.com>>
>     JBoss Performance Team
>
>     _______________________________________________
>     infinispan-dev mailing list
>     infinispan-dev at lists.jboss.org <mailto:infinispan-dev at lists.jboss.org>
>     https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
>
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev


-- 
Radim Vansa <rvansa at redhat.com>
JBoss Performance Team


From sanne at infinispan.org  Mon Mar 27 07:04:32 2017
From: sanne at infinispan.org (Sanne Grinovero)
Date: Mon, 27 Mar 2017 12:04:32 +0100
Subject: [infinispan-dev] Branching proposal
In-Reply-To: <CAA9R9O6WYL_YSZT1A8qhBqoiubZNSB+8D+n-s=2+P9wzvHxBig@mail.gmail.com>
References: <CAA9R9O6X3kOu3h4=_2z6RZKExYC30B0JiJMvEYQ3mu9HGq20Rw@mail.gmail.com>
	<f9406b3d-af44-1627-7037-15ff080d6e3f@redhat.com>
	<CAA9R9O6WYL_YSZT1A8qhBqoiubZNSB+8D+n-s=2+P9wzvHxBig@mail.gmail.com>
Message-ID: <CAFm4XO3AwHv3u_D2t42Fu-DkwAnea=3zkLnfrFh-fMB436Cayw@mail.gmail.com>

You need to make sure you optimise for engineers coding stuff which
works on master, not maintenance.

Isn't it a bit naive to expect people to work on the "last maintained branch" ?

Your proposal expect that each and every patch author:
  a) knows for sure it's not going to be backported further to even
older branches
  b) you're not investigating issues on master (as investigation often
implies making code changes already)

Both of these are not very practical as I generally don't expect to
backport each and every fix, and it's going to be someone else to
recommend/approve some backports *after* the problem has been
understood in depth.
You even mention "forgetting to backport refactoring" ? Hell yes you
should never backport refactorings..

What I'm reading here is that you're all doing way too many backports.
Be more conservative about them?

In my experience with recent contributions to Infinispan - either
because the testsuite is slow, or because it takes us too long to
figure out there has been a regression, I sent a PR to master and two
maintenance branches but I had to repeat the exercise two more times
to add follow up fixes on the same subject: if I had waited on
backports it would have taken me a third of the time.

Thanks,
Sanne




On 27 March 2017 at 11:45, Sebastian Laskawiec <slaskawi at redhat.com> wrote:
> From my past experience, if a commit caused a conflict when merging, we
> always asked the author to fix it and do the merge.
>
> After a while it became a habit that each dev who submitted a code that
> could result in conflicts, did all the merges.
>
> On Mon, Mar 27, 2017 at 12:37 PM Radim Vansa <rvansa at redhat.com> wrote:
>>
>> If you can't merge a commit (based on 9.0.x) to master clearly, do you
>> need to file another PR anyway? Then the lag to get some code to master
>> increases a lot. I am not sure how useful is git tag --contains <sha1>
>> if you cannot be sure that you'll find all occurrences due to this kind
>> of issues.
>>
>> R.
>>
>> On 03/27/2017 11:33 AM, Sebastian Laskawiec wrote:
>> > Hey!
>> >
>> > We are about to start working on 9.1.x and 9.2.y branches so I would
>> > like to propose alternative merging strategy.
>> >
>> > Our current workflow looks like this:
>> >
>> > X - new commit
>> > X` - cherry pick to maintenance branch
>> > --+-------------------+-------X----- master
>> >   |                    \------X`---- 9.2.x
>> >   \---------------------------X``--- 9.1.x
>> >
>> > Each commit needs to be reviewed in master branch and backported to
>> > the maintenance branches. From maintenance perspective this is a bit
>> > painful, since in above example we need to get 3 times through PR
>> > queue. Also it's worth to mention that X is not X` nor X``.
>> > Cherry-picking creates a copy of a commit. This makes some useful
>> > tricks (like git tag --contains <sha1>) a bit harder to use. Finally,
>> > this approach allows the codebase to diverge from maintenance branches
>> > very fast (someone might just forget to backport some of the
>> > refactoring stuff).
>> >
>> > The proposal:
>> >
>> > X, Y - new commits
>> > / - merge commits
>> > --+---------+------/----/--- master
>> >   |          \----/---Y/---- 9.2.x
>> >   \-------------X/---------- 9.1.x
>> >
>> > With the proposal, a developer should always implement a given feature
>> > in the lowest possible maintenance branch. Then we will run a set of
>> > merges from 9.1.x into 9.2.x and finally into master. The biggest
>> > advantage of this approach is that given functionality (identified by
>> > a commit) will have the same SHA1 for all branches. This will allow
>> > all tools like (mentioned before) `git tag --contains <sha1>` to work.
>> > There are also some further implications of this approach:
>> >
>> >   * Merging commits should be performed very often (even automatically
>> >     in the night (if merged without any problems)).
>> >   * After releasing each maintenance release, someone will need to do
>> >     a merge with strategy `ours` (`git merge -s ours upstream/9.2.x`).
>> >     This way we will not have to solve version conflicts in poms.
>> >   * Since there is no nice way to rebase a merge commit, they should
>> >     be pushed directly into the master branch (without review, without
>> >     CI). After the merge, HEAD will change and CI will
>> >     automatically pick the build. Remember, merges should be done very
>> >     often. So I assume there won't be any problems most of the times.
>> >   * Finally, with this approach the code diverges slight slower (at
>> >     least from my experience). Mainly because we don't need to
>> >     remember to cherry-pick individual commits. They are automatically
>> >     "taken" by a merge.
>> >
>> > From my past experience, this strategy works pretty nice and can be
>> > almost fully automated. It significantly lowers the maintenance pain
>> > around cherry-picks. However there is nothing for free, and we would
>> > need to get used to pushing merged directly into master (which is fine
>> > to me but some of you might not like it).
>> >
>> > Thanks,
>> > Sebastian
>> >
>> >
>> > _______________________________________________
>> > infinispan-dev mailing list
>> > infinispan-dev at lists.jboss.org
>> > https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>
>>
>> --
>> Radim Vansa <rvansa at redhat.com>
>> JBoss Performance Team
>>
>> _______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev

From slaskawi at redhat.com  Mon Mar 27 07:16:21 2017
From: slaskawi at redhat.com (Sebastian Laskawiec)
Date: Mon, 27 Mar 2017 11:16:21 +0000
Subject: [infinispan-dev] Branching proposal
In-Reply-To: <27b9795d-8908-8a41-e964-6a1ec9ee6f15@redhat.com>
References: <CAA9R9O6X3kOu3h4=_2z6RZKExYC30B0JiJMvEYQ3mu9HGq20Rw@mail.gmail.com>
	<f9406b3d-af44-1627-7037-15ff080d6e3f@redhat.com>
	<CAA9R9O6WYL_YSZT1A8qhBqoiubZNSB+8D+n-s=2+P9wzvHxBig@mail.gmail.com>
	<27b9795d-8908-8a41-e964-6a1ec9ee6f15@redhat.com>
Message-ID: <CAA9R9O4FnbHw96umL_d_SZovXXLyJ+rO7u26QNXyn7FOa4LVDw@mail.gmail.com>

On Mon, Mar 27, 2017 at 12:59 PM Radim Vansa <rvansa at redhat.com> wrote:

> On 03/27/2017 12:45 PM, Sebastian Laskawiec wrote:
> > From my past experience, if a commit caused a conflict when merging,
> > we always asked the author to fix it and do the merge.
>
> I don't understand. The PR should be filed against 9.0.x, there're no
> conflicts. Merging the same against master results in conflicts - where
> should I resolve those?
>

I think we mean 9.1.x (the oldest maintenance branch). In that case you
should merge 0.9.1 into 0.9.2. Than another merge 0.9.2 into master.

Once the conflict occurs, a developer who does the merge should simply
resolve it.


>
> Another q: I decide to file a PR against 9.1, because I don't think it
> should be applied to 9.0. I get a review, but then someone explains that
> it should get to 9.0 as well. I can't change a target branch in GitHub's
> PR: should I close the PR with nice history of comments (some of them
> not addressed yet) and open another PR?
>

You can do both (if depends on your intuition which is better). The nice
thing about git merge is that it won't throw any error if a change is
already present on target branch. So it is possible and legal to
cherry-pick stuff as well.


>
> R.
>
> >
> > After a while it became a habit that each dev who submitted a code
> > that could result in conflicts, did all the merges.
> >
> > On Mon, Mar 27, 2017 at 12:37 PM Radim Vansa <rvansa at redhat.com
> > <mailto:rvansa at redhat.com>> wrote:
> >
> >     If you can't merge a commit (based on 9.0.x) to master clearly, do
> you
> >     need to file another PR anyway? Then the lag to get some code to
> >     master
> >     increases a lot. I am not sure how useful is git tag --contains
> <sha1>
> >     if you cannot be sure that you'll find all occurrences due to this
> >     kind
> >     of issues.
> >
> >     R.
> >
> >     On 03/27/2017 11:33 AM, Sebastian Laskawiec wrote:
> >     > Hey!
> >     >
> >     > We are about to start working on 9.1.x and 9.2.y branches so I
> would
> >     > like to propose alternative merging strategy.
> >     >
> >     > Our current workflow looks like this:
> >     >
> >     > X - new commit
> >     > X` - cherry pick to maintenance branch
> >     > --+-------------------+-------X----- master
> >     >   |                    \------X`---- 9.2.x
> >     >   \---------------------------X``--- 9.1.x
> >     >
> >     > Each commit needs to be reviewed in master branch and backported to
> >     > the maintenance branches. From maintenance perspective this is a
> bit
> >     > painful, since in above example we need to get 3 times through PR
> >     > queue. Also it's worth to mention that X is not X` nor X``.
> >     > Cherry-picking creates a copy of a commit. This makes some useful
> >     > tricks (like git tag --contains <sha1>) a bit harder to use.
> >     Finally,
> >     > this approach allows the codebase to diverge from maintenance
> >     branches
> >     > very fast (someone might just forget to backport some of the
> >     > refactoring stuff).
> >     >
> >     > The proposal:
> >     >
> >     > X, Y - new commits
> >     > / - merge commits
> >     > --+---------+------/----/--- master
> >     >   |          \----/---Y/---- 9.2.x
> >     >   \-------------X/---------- 9.1.x
> >     >
> >     > With the proposal, a developer should always implement a given
> >     feature
> >     > in the lowest possible maintenance branch. Then we will run a set
> of
> >     > merges from 9.1.x into 9.2.x and finally into master. The biggest
> >     > advantage of this approach is that given functionality
> >     (identified by
> >     > a commit) will have the same SHA1 for all branches. This will allow
> >     > all tools like (mentioned before) `git tag --contains <sha1>` to
> >     work.
> >     > There are also some further implications of this approach:
> >     >
> >     >   * Merging commits should be performed very often (even
> >     automatically
> >     >     in the night (if merged without any problems)).
> >     >   * After releasing each maintenance release, someone will need
> >     to do
> >     >     a merge with strategy `ours` (`git merge -s ours
> >     upstream/9.2.x`).
> >     >     This way we will not have to solve version conflicts in poms.
> >     >   * Since there is no nice way to rebase a merge commit, they
> should
> >     >     be pushed directly into the master branch (without review,
> >     without
> >     >     CI). After the merge, HEAD will change and CI will
> >     >     automatically pick the build. Remember, merges should be
> >     done very
> >     >     often. So I assume there won't be any problems most of the
> >     times.
> >     >   * Finally, with this approach the code diverges slight slower (at
> >     >     least from my experience). Mainly because we don't need to
> >     >     remember to cherry-pick individual commits. They are
> >     automatically
> >     >     "taken" by a merge.
> >     >
> >     > From my past experience, this strategy works pretty nice and can be
> >     > almost fully automated. It significantly lowers the maintenance
> pain
> >     > around cherry-picks. However there is nothing for free, and we
> would
> >     > need to get used to pushing merged directly into master (which
> >     is fine
> >     > to me but some of you might not like it).
> >     >
> >     > Thanks,
> >     > Sebastian
> >     >
> >     >
> >     > _______________________________________________
> >     > infinispan-dev mailing list
> >     > infinispan-dev at lists.jboss.org
> >     <mailto:infinispan-dev at lists.jboss.org>
> >     > https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >
> >
> >     --
> >     Radim Vansa <rvansa at redhat.com <mailto:rvansa at redhat.com>>
> >     JBoss Performance Team
> >
> >     _______________________________________________
> >     infinispan-dev mailing list
> >     infinispan-dev at lists.jboss.org <mailto:
> infinispan-dev at lists.jboss.org>
> >     https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >
> >
> >
> > _______________________________________________
> > infinispan-dev mailing list
> > infinispan-dev at lists.jboss.org
> > https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
>
> --
> Radim Vansa <rvansa at redhat.com>
> JBoss Performance Team
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/infinispan-dev/attachments/20170327/f5952e26/attachment.html 

From slaskawi at redhat.com  Mon Mar 27 07:51:19 2017
From: slaskawi at redhat.com (Sebastian Laskawiec)
Date: Mon, 27 Mar 2017 11:51:19 +0000
Subject: [infinispan-dev] Branching proposal
In-Reply-To: <CAFm4XO3AwHv3u_D2t42Fu-DkwAnea=3zkLnfrFh-fMB436Cayw@mail.gmail.com>
References: <CAA9R9O6X3kOu3h4=_2z6RZKExYC30B0JiJMvEYQ3mu9HGq20Rw@mail.gmail.com>
	<f9406b3d-af44-1627-7037-15ff080d6e3f@redhat.com>
	<CAA9R9O6WYL_YSZT1A8qhBqoiubZNSB+8D+n-s=2+P9wzvHxBig@mail.gmail.com>
	<CAFm4XO3AwHv3u_D2t42Fu-DkwAnea=3zkLnfrFh-fMB436Cayw@mail.gmail.com>
Message-ID: <CAA9R9O5jeXrTBOanQnGO8Ca8y2sG6ZfbPppO-_4PjdtwC0Tj=g@mail.gmail.com>

On Mon, Mar 27, 2017 at 1:05 PM Sanne Grinovero <sanne at infinispan.org>
wrote:

> You need to make sure you optimise for engineers coding stuff which
> works on master, not maintenance.
>

Well it depends on your strategy. Note that with my proposal you can always
do that (optimize features for development not for maintenance).... just
put your PR on master branch. That's all...

The proposal is all about making maintenance easier. Development stays the
same.


> Isn't it a bit naive to expect people to work on the "last maintained
> branch" ?
>

I don't think so. In that past I worked on a project with 3 maintenance
branches plus a development branch. It worked. That's why I'm proposing
this here.


> Your proposal expect that each and every patch author:
>   a) knows for sure it's not going to be backported further to even
> older branches
>

As I explained this to Radim, occasional cherry-picking is totally fine.
During the merge git will look at the commit, say... oh this commit is
already there and move on.


>   b) you're not investigating issues on master (as investigation often
> implies making code changes already)
>

Again it depends. If you do not care about maintenance branches, you can
still code at master branch.

I also disagree with you here. We should always fix errors where they occur
and not only in maintenance branch. If the failure is on 9.1.x branch, then
the fix should go there. Not to master.


> Both of these are not very practical as I generally don't expect to
> backport each and every fix, and it's going to be someone else to
> recommend/approve some backports *after* the problem has been
> understood in depth.
>

In that kind of workflow my proposal wouldn't work well. I'm also a bit
curious, how often do we need such an approval?


> You even mention "forgetting to backport refactoring" ? Hell yes you
> should never backport refactorings..


It depends. How about our checkstyle changes? After some time I guess I
should have backported them to make cherry-picking easier.


> What I'm reading here is that you're all doing way too many backports.
> Be more conservative about them?
>

That's a very well stated question. My proposal makes the maintenance
easier but developers need to be a bit more responsible (they need to know
in which branch to implement a given fix and remember about doing merges).
On the other hand if we prefer to encourage our users to move to more
recent version quicker, than it probably doesn't make much sense.


> In my experience with recent contributions to Infinispan - either
> because the testsuite is slow, or because it takes us too long to
> figure out there has been a regression, I sent a PR to master and two
> maintenance branches but I had to repeat the exercise two more times
> to add follow up fixes on the same subject: if I had waited on
> backports it would have taken me a third of the time.
>

Note that my proposal partially addresses this concern. Individual PRs
wouldn't need to go through CI loop but they would be merged in batches. On
the other hand, even an easy merge might break the build.


>
> Thanks,
> Sanne
>
>
>
>
> On 27 March 2017 at 11:45, Sebastian Laskawiec <slaskawi at redhat.com>
> wrote:
> > From my past experience, if a commit caused a conflict when merging, we
> > always asked the author to fix it and do the merge.
> >
> > After a while it became a habit that each dev who submitted a code that
> > could result in conflicts, did all the merges.
> >
> > On Mon, Mar 27, 2017 at 12:37 PM Radim Vansa <rvansa at redhat.com> wrote:
> >>
> >> If you can't merge a commit (based on 9.0.x) to master clearly, do you
> >> need to file another PR anyway? Then the lag to get some code to master
> >> increases a lot. I am not sure how useful is git tag --contains <sha1>
> >> if you cannot be sure that you'll find all occurrences due to this kind
> >> of issues.
> >>
> >> R.
> >>
> >> On 03/27/2017 11:33 AM, Sebastian Laskawiec wrote:
> >> > Hey!
> >> >
> >> > We are about to start working on 9.1.x and 9.2.y branches so I would
> >> > like to propose alternative merging strategy.
> >> >
> >> > Our current workflow looks like this:
> >> >
> >> > X - new commit
> >> > X` - cherry pick to maintenance branch
> >> > --+-------------------+-------X----- master
> >> >   |                    \------X`---- 9.2.x
> >> >   \---------------------------X``--- 9.1.x
> >> >
> >> > Each commit needs to be reviewed in master branch and backported to
> >> > the maintenance branches. From maintenance perspective this is a bit
> >> > painful, since in above example we need to get 3 times through PR
> >> > queue. Also it's worth to mention that X is not X` nor X``.
> >> > Cherry-picking creates a copy of a commit. This makes some useful
> >> > tricks (like git tag --contains <sha1>) a bit harder to use. Finally,
> >> > this approach allows the codebase to diverge from maintenance branches
> >> > very fast (someone might just forget to backport some of the
> >> > refactoring stuff).
> >> >
> >> > The proposal:
> >> >
> >> > X, Y - new commits
> >> > / - merge commits
> >> > --+---------+------/----/--- master
> >> >   |          \----/---Y/---- 9.2.x
> >> >   \-------------X/---------- 9.1.x
> >> >
> >> > With the proposal, a developer should always implement a given feature
> >> > in the lowest possible maintenance branch. Then we will run a set of
> >> > merges from 9.1.x into 9.2.x and finally into master. The biggest
> >> > advantage of this approach is that given functionality (identified by
> >> > a commit) will have the same SHA1 for all branches. This will allow
> >> > all tools like (mentioned before) `git tag --contains <sha1>` to work.
> >> > There are also some further implications of this approach:
> >> >
> >> >   * Merging commits should be performed very often (even automatically
> >> >     in the night (if merged without any problems)).
> >> >   * After releasing each maintenance release, someone will need to do
> >> >     a merge with strategy `ours` (`git merge -s ours upstream/9.2.x`).
> >> >     This way we will not have to solve version conflicts in poms.
> >> >   * Since there is no nice way to rebase a merge commit, they should
> >> >     be pushed directly into the master branch (without review, without
> >> >     CI). After the merge, HEAD will change and CI will
> >> >     automatically pick the build. Remember, merges should be done very
> >> >     often. So I assume there won't be any problems most of the times.
> >> >   * Finally, with this approach the code diverges slight slower (at
> >> >     least from my experience). Mainly because we don't need to
> >> >     remember to cherry-pick individual commits. They are automatically
> >> >     "taken" by a merge.
> >> >
> >> > From my past experience, this strategy works pretty nice and can be
> >> > almost fully automated. It significantly lowers the maintenance pain
> >> > around cherry-picks. However there is nothing for free, and we would
> >> > need to get used to pushing merged directly into master (which is fine
> >> > to me but some of you might not like it).
> >> >
> >> > Thanks,
> >> > Sebastian
> >> >
> >> >
> >> > _______________________________________________
> >> > infinispan-dev mailing list
> >> > infinispan-dev at lists.jboss.org
> >> > https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >>
> >>
> >> --
> >> Radim Vansa <rvansa at redhat.com>
> >> JBoss Performance Team
> >>
> >> _______________________________________________
> >> infinispan-dev mailing list
> >> infinispan-dev at lists.jboss.org
> >> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >
> >
> > _______________________________________________
> > infinispan-dev mailing list
> > infinispan-dev at lists.jboss.org
> > https://lists.jboss.org/mailman/listinfo/infinispan-dev
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/infinispan-dev/attachments/20170327/ec1531a8/attachment-0001.html 

From dan.berindei at gmail.com  Mon Mar 27 08:36:35 2017
From: dan.berindei at gmail.com (Dan Berindei)
Date: Mon, 27 Mar 2017 15:36:35 +0300
Subject: [infinispan-dev] Branching proposal
In-Reply-To: <f9406b3d-af44-1627-7037-15ff080d6e3f@redhat.com>
References: <CAA9R9O6X3kOu3h4=_2z6RZKExYC30B0JiJMvEYQ3mu9HGq20Rw@mail.gmail.com>
	<f9406b3d-af44-1627-7037-15ff080d6e3f@redhat.com>
Message-ID: <CA+nfvwRjtzdO=gS_E5PqhtA5qe=3pEyG9y0jXi316vzs=cOQUA@mail.gmail.com>

I use something like this to check what tags contain a particular fix:

git tag --contains $(git log --grep <JIRA> -1 --format="%h" master)

True, it's a bit longer, but it stays in the bash/zsh history :)

Cheers
Dan


On Mon, Mar 27, 2017 at 1:33 PM, Radim Vansa <rvansa at redhat.com> wrote:
> If you can't merge a commit (based on 9.0.x) to master clearly, do you
> need to file another PR anyway? Then the lag to get some code to master
> increases a lot. I am not sure how useful is git tag --contains <sha1>
> if you cannot be sure that you'll find all occurrences due to this kind
> of issues.
>
> R.
>
> On 03/27/2017 11:33 AM, Sebastian Laskawiec wrote:
>> Hey!
>>
>> We are about to start working on 9.1.x and 9.2.y branches so I would
>> like to propose alternative merging strategy.
>>
>> Our current workflow looks like this:
>>
>> X - new commit
>> X` - cherry pick to maintenance branch
>> --+-------------------+-------X----- master
>>   |                    \------X`---- 9.2.x
>>   \---------------------------X``--- 9.1.x
>>
>> Each commit needs to be reviewed in master branch and backported to
>> the maintenance branches. From maintenance perspective this is a bit
>> painful, since in above example we need to get 3 times through PR
>> queue. Also it's worth to mention that X is not X` nor X``.
>> Cherry-picking creates a copy of a commit. This makes some useful
>> tricks (like git tag --contains <sha1>) a bit harder to use. Finally,
>> this approach allows the codebase to diverge from maintenance branches
>> very fast (someone might just forget to backport some of the
>> refactoring stuff).
>>
>> The proposal:
>>
>> X, Y - new commits
>> / - merge commits
>> --+---------+------/----/--- master
>>   |          \----/---Y/---- 9.2.x
>>   \-------------X/---------- 9.1.x
>>
>> With the proposal, a developer should always implement a given feature
>> in the lowest possible maintenance branch. Then we will run a set of
>> merges from 9.1.x into 9.2.x and finally into master. The biggest
>> advantage of this approach is that given functionality (identified by
>> a commit) will have the same SHA1 for all branches. This will allow
>> all tools like (mentioned before) `git tag --contains <sha1>` to work.
>> There are also some further implications of this approach:
>>
>>   * Merging commits should be performed very often (even automatically
>>     in the night (if merged without any problems)).
>>   * After releasing each maintenance release, someone will need to do
>>     a merge with strategy `ours` (`git merge -s ours upstream/9.2.x`).
>>     This way we will not have to solve version conflicts in poms.
>>   * Since there is no nice way to rebase a merge commit, they should
>>     be pushed directly into the master branch (without review, without
>>     CI). After the merge, HEAD will change and CI will
>>     automatically pick the build. Remember, merges should be done very
>>     often. So I assume there won't be any problems most of the times.
>>   * Finally, with this approach the code diverges slight slower (at
>>     least from my experience). Mainly because we don't need to
>>     remember to cherry-pick individual commits. They are automatically
>>     "taken" by a merge.
>>
>> From my past experience, this strategy works pretty nice and can be
>> almost fully automated. It significantly lowers the maintenance pain
>> around cherry-picks. However there is nothing for free, and we would
>> need to get used to pushing merged directly into master (which is fine
>> to me but some of you might not like it).
>>
>> Thanks,
>> Sebastian
>>
>>
>> _______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
>
> --
> Radim Vansa <rvansa at redhat.com>
> JBoss Performance Team
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev

From rvansa at redhat.com  Mon Mar 27 08:41:00 2017
From: rvansa at redhat.com (Radim Vansa)
Date: Mon, 27 Mar 2017 14:41:00 +0200
Subject: [infinispan-dev] Branching proposal
In-Reply-To: <CAA9R9O4FnbHw96umL_d_SZovXXLyJ+rO7u26QNXyn7FOa4LVDw@mail.gmail.com>
References: <CAA9R9O6X3kOu3h4=_2z6RZKExYC30B0JiJMvEYQ3mu9HGq20Rw@mail.gmail.com>
	<f9406b3d-af44-1627-7037-15ff080d6e3f@redhat.com>
	<CAA9R9O6WYL_YSZT1A8qhBqoiubZNSB+8D+n-s=2+P9wzvHxBig@mail.gmail.com>
	<27b9795d-8908-8a41-e964-6a1ec9ee6f15@redhat.com>
	<CAA9R9O4FnbHw96umL_d_SZovXXLyJ+rO7u26QNXyn7FOa4LVDw@mail.gmail.com>
Message-ID: <eacb158f-5be6-099e-3ad3-93b28abf853b@redhat.com>

On 03/27/2017 01:16 PM, Sebastian Laskawiec wrote:
>
>
> On Mon, Mar 27, 2017 at 12:59 PM Radim Vansa <rvansa at redhat.com 
> <mailto:rvansa at redhat.com>> wrote:
>
>     On 03/27/2017 12:45 PM, Sebastian Laskawiec wrote:
>     > From my past experience, if a commit caused a conflict when merging,
>     > we always asked the author to fix it and do the merge.
>
>     I don't understand. The PR should be filed against 9.0.x, there're no
>     conflicts. Merging the same against master results in conflicts -
>     where
>     should I resolve those?
>
>
> I think we mean 9.1.x (the oldest maintenance branch). In that case 
> you should merge 0.9.1 into 0.9.2. Than another merge 0.9.2 into master.

I am lost - we won't have 9.0.x as maintenance branch starting with 
9.0.0.Final and 9.1.x == master moving towards 9.1.0.Final? What's 0.9.1 
and 0.9.2?

>
> Once the conflict occurs, a developer who does the merge should simply 
> resolve it.

You haven't explained *where*. I am a contributor, I don't "merge" 
stuff, I file PRs. So, I'll do a PR against [maintanance branch] with 
commits that can't be merged against [master] because these have 
conflicts. This is one PR (against [maintenance branch]). Should I issue 
another PR? With backports I wait until it gets into master (and usually 
somebody requests the backport), how do I make sure that it gets into 
master ASAP?

>
>     Another q: I decide to file a PR against 9.1, because I don't think it
>     should be applied to 9.0. I get a review, but then someone
>     explains that
>     it should get to 9.0 as well. I can't change a target branch in
>     GitHub's
>     PR: should I close the PR with nice history of comments (some of them
>     not addressed yet) and open another PR?
>
>
> You can do both (if depends on your intuition which is better). The 
> nice thing about git merge is that it won't throw any error if a 
> change is already present on target branch. So it is possible and 
> legal to cherry-pick stuff as well.
>
>
>     R.
>
>     >
>     > After a while it became a habit that each dev who submitted a code
>     > that could result in conflicts, did all the merges.
>     >
>     > On Mon, Mar 27, 2017 at 12:37 PM Radim Vansa <rvansa at redhat.com
>     <mailto:rvansa at redhat.com>
>     > <mailto:rvansa at redhat.com <mailto:rvansa at redhat.com>>> wrote:
>     >
>     >     If you can't merge a commit (based on 9.0.x) to master
>     clearly, do you
>     >     need to file another PR anyway? Then the lag to get some code to
>     >     master
>     >     increases a lot. I am not sure how useful is git tag
>     --contains <sha1>
>     >     if you cannot be sure that you'll find all occurrences due
>     to this
>     >     kind
>     >     of issues.
>     >
>     >     R.
>     >
>     >     On 03/27/2017 11:33 AM, Sebastian Laskawiec wrote:
>     >     > Hey!
>     >     >
>     >     > We are about to start working on 9.1.x and 9.2.y branches
>     so I would
>     >     > like to propose alternative merging strategy.
>     >     >
>     >     > Our current workflow looks like this:
>     >     >
>     >     > X - new commit
>     >     > X` - cherry pick to maintenance branch
>     >     > --+-------------------+-------X----- master
>     >     >   |                    \------X`---- 9.2.x
>     >     >   \---------------------------X``--- 9.1.x
>     >     >
>     >     > Each commit needs to be reviewed in master branch and
>     backported to
>     >     > the maintenance branches. From maintenance perspective
>     this is a bit
>     >     > painful, since in above example we need to get 3 times
>     through PR
>     >     > queue. Also it's worth to mention that X is not X` nor X``.
>     >     > Cherry-picking creates a copy of a commit. This makes some
>     useful
>     >     > tricks (like git tag --contains <sha1>) a bit harder to use.
>     >     Finally,
>     >     > this approach allows the codebase to diverge from maintenance
>     >     branches
>     >     > very fast (someone might just forget to backport some of the
>     >     > refactoring stuff).
>     >     >
>     >     > The proposal:
>     >     >
>     >     > X, Y - new commits
>     >     > / - merge commits
>     >     > --+---------+------/----/--- master
>     >     >   |          \----/---Y/---- 9.2.x
>     >     >   \-------------X/---------- 9.1.x
>     >     >
>     >     > With the proposal, a developer should always implement a given
>     >     feature
>     >     > in the lowest possible maintenance branch. Then we will
>     run a set of
>     >     > merges from 9.1.x into 9.2.x and finally into master. The
>     biggest
>     >     > advantage of this approach is that given functionality
>     >     (identified by
>     >     > a commit) will have the same SHA1 for all branches. This
>     will allow
>     >     > all tools like (mentioned before) `git tag --contains
>     <sha1>` to
>     >     work.
>     >     > There are also some further implications of this approach:
>     >     >
>     >     >   * Merging commits should be performed very often (even
>     >     automatically
>     >     >     in the night (if merged without any problems)).
>     >     >   * After releasing each maintenance release, someone will
>     need
>     >     to do
>     >     >     a merge with strategy `ours` (`git merge -s ours
>     >     upstream/9.2.x`).
>     >     >     This way we will not have to solve version conflicts
>     in poms.
>     >     >   * Since there is no nice way to rebase a merge commit,
>     they should
>     >     >     be pushed directly into the master branch (without review,
>     >     without
>     >     >     CI). After the merge, HEAD will change and CI will
>     >     >     automatically pick the build. Remember, merges should be
>     >     done very
>     >     >     often. So I assume there won't be any problems most of the
>     >     times.
>     >     >   * Finally, with this approach the code diverges slight
>     slower (at
>     >     >     least from my experience). Mainly because we don't need to
>     >     >     remember to cherry-pick individual commits. They are
>     >     automatically
>     >     >     "taken" by a merge.
>     >     >
>     >     > From my past experience, this strategy works pretty nice
>     and can be
>     >     > almost fully automated. It significantly lowers the
>     maintenance pain
>     >     > around cherry-picks. However there is nothing for free,
>     and we would
>     >     > need to get used to pushing merged directly into master (which
>     >     is fine
>     >     > to me but some of you might not like it).
>     >     >
>     >     > Thanks,
>     >     > Sebastian
>     >     >
>     >     >
>     >     > _______________________________________________
>     >     > infinispan-dev mailing list
>     >     > infinispan-dev at lists.jboss.org
>     <mailto:infinispan-dev at lists.jboss.org>
>     >     <mailto:infinispan-dev at lists.jboss.org
>     <mailto:infinispan-dev at lists.jboss.org>>
>     >     > https://lists.jboss.org/mailman/listinfo/infinispan-dev
>     >
>     >
>     >     --
>     >     Radim Vansa <rvansa at redhat.com <mailto:rvansa at redhat.com>
>     <mailto:rvansa at redhat.com <mailto:rvansa at redhat.com>>>
>     >     JBoss Performance Team
>     >
>     >     _______________________________________________
>     >     infinispan-dev mailing list
>     > infinispan-dev at lists.jboss.org
>     <mailto:infinispan-dev at lists.jboss.org>
>     <mailto:infinispan-dev at lists.jboss.org
>     <mailto:infinispan-dev at lists.jboss.org>>
>     > https://lists.jboss.org/mailman/listinfo/infinispan-dev
>     >
>     >
>     >
>     > _______________________________________________
>     > infinispan-dev mailing list
>     > infinispan-dev at lists.jboss.org
>     <mailto:infinispan-dev at lists.jboss.org>
>     > https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
>
>     --
>     Radim Vansa <rvansa at redhat.com <mailto:rvansa at redhat.com>>
>     JBoss Performance Team
>
>     _______________________________________________
>     infinispan-dev mailing list
>     infinispan-dev at lists.jboss.org <mailto:infinispan-dev at lists.jboss.org>
>     https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
>
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev


-- 
Radim Vansa <rvansa at redhat.com>
JBoss Performance Team


From galder at redhat.com  Mon Mar 27 10:09:33 2017
From: galder at redhat.com (=?utf-8?Q?Galder_Zamarre=C3=B1o?=)
Date: Mon, 27 Mar 2017 08:09:33 -0600
Subject: [infinispan-dev] NoSuchMethodError in Spark simple tutorial
In-Reply-To: <CAH8Ud1FQHQMxM132CvjHv41OLkAEHKA7fGUbr8jVz32iz9EQeA@mail.gmail.com>
References: <4FB5C903-3A8C-4D94-9E1E-C1D2512C1ECF@redhat.com>
	<CAH8Ud1FQHQMxM132CvjHv41OLkAEHKA7fGUbr8jVz32iz9EQeA@mail.gmail.com>
Message-ID: <26FBFDF8-E82F-43B4-92B7-67B5D68B559B@redhat.com>

Ok, I think I know what I was doing wrong. I was trying to use the spark tutorial on its own, without the parent dependencies. As a result of that, the Infinispan Client being resolved was Alpha4.

In the simple tutorials, the parent pom defines an Infinispan dependency that's newer and the dependency resolution ends up picking up the newer version and hence it works.

I don't think this should be the case, the spark tutorial shouldn't be relying on this kind of mixed dependencies resolution.  That's why I think there should have been a Spark release that fixes the Alpha4 problem.

On top of that, I find tutorials should be runnable on their own, without parent dependencies. This might make tutorial poms bigger or more verbose but I think it helps with users when trying to run each individually outside of the simple tutorials repo.

Cheers,
--
Galder Zamarre?o
Infinispan, Red Hat

> On 24 Mar 2017, at 02:31, Gustavo Fernandes <gustavo at infinispan.org> wrote:
> 
> This was fixed [1]
> 
> Regarding the tutorial, I could run it "as is":
> 
> git clone https://github.com/infinispan/infinispan-simple-tutorials.git
> cd infinispan-simple-tutorials
> mvn clean install
> cd spark
> mvn exec:exec
> 
> 
> How did you reproduced the issue?
> 
> 
> [1] https://issues.jboss.org/browse/ISPRK-33
> 
> 
> Thanks,
> Gustavo
> 
> On Thu, Mar 23, 2017 at 1:40 PM, Galder Zamarre?o <galder at redhat.com> wrote:
> Hey guys,
> 
> The Spark Java simple tutorial [1] tutorial does not work as is due to some dependencies mix up.
> 
> If you run that class, with the dependencies defined in the pom.xml, you get:
> 
> java.lang.NoSuchMethodError: org.infinispan.client.hotrod.logging.Log.tracef(Ljava/lang/Throwable;Ljava/lang/String;II)V
> 
> The dependencies are defined as:
> 
> <dependency>
>   <groupId>org.infinispan</groupId>
>   <artifactId>infinispan-spark_2.11</artifactId>
>   <version>0.4</version>
> </dependency>
> ...
> 
> Looking at the dependency tree, I see this:
> 
> [INFO] +- org.infinispan:infinispan-spark_2.11:jar:0.4:compile
> [INFO] |  +- org.infinispan:infinispan-client-hotrod:jar:9.0.0.Alpha4:compile
> ...
> [INFO] |  +- org.infinispan.protostream:protostream:jar:3.0.5.Final:compile
> [INFO] |  |  +- org.jboss.logging:jboss-logging:jar:3.1.4.GA:compile
> 
> That logging jar seems an old one, it should be 3.3.x. I worked around this by doing:
> 
> <dependency>
>   <groupId>org.infinispan</groupId>
>   <artifactId>infinispan-spark_2.11</artifactId>
>   <version>${version.spark-connector}</version>
>   <exclusions>
>     <exclusion>
>       <groupId>org.jboss.logging</groupId>
>       <artifactId>jboss-logging</artifactId>
>     </exclusion>
>   </exclusions>
> </dependency>
> <dependency>
>   <groupId>org.jboss.logging</groupId>
>   <artifactId>jboss-logging</artifactId>
>   <version>${version.jboss.logging}</version>
> </dependency>
> 
> @Adrian Are the dependencies of the latest protostream versions in line with the infinispan hot rod client ones?
> 
> @Gustavo, Once there's a client version which depends on a protostream version that fixes this (if there's not one already...), can you release a 0.5 alpha/beta/cr version?
> 
> Cheers,
> 
> [1] http://infinispan.org/tutorials/simple/spark/
> --
> Galder Zamarre?o
> Infinispan, Red Hat
> 
> 
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> 
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev



From dan.berindei at gmail.com  Mon Mar 27 10:26:34 2017
From: dan.berindei at gmail.com (Dan Berindei)
Date: Mon, 27 Mar 2017 17:26:34 +0300
Subject: [infinispan-dev] Branching proposal
In-Reply-To: <CAA9R9O5jeXrTBOanQnGO8Ca8y2sG6ZfbPppO-_4PjdtwC0Tj=g@mail.gmail.com>
References: <CAA9R9O6X3kOu3h4=_2z6RZKExYC30B0JiJMvEYQ3mu9HGq20Rw@mail.gmail.com>
	<f9406b3d-af44-1627-7037-15ff080d6e3f@redhat.com>
	<CAA9R9O6WYL_YSZT1A8qhBqoiubZNSB+8D+n-s=2+P9wzvHxBig@mail.gmail.com>
	<CAFm4XO3AwHv3u_D2t42Fu-DkwAnea=3zkLnfrFh-fMB436Cayw@mail.gmail.com>
	<CAA9R9O5jeXrTBOanQnGO8Ca8y2sG6ZfbPppO-_4PjdtwC0Tj=g@mail.gmail.com>
Message-ID: <CA+nfvwQ3Lt+HB+4iHqrMC9dAGZ1MSrZcLPZXwBk1qXgZsqgC7g@mail.gmail.com>

On Mon, Mar 27, 2017 at 2:51 PM, Sebastian Laskawiec
<slaskawi at redhat.com> wrote:
>
>
> On Mon, Mar 27, 2017 at 1:05 PM Sanne Grinovero <sanne at infinispan.org>
> wrote:
>>
>> You need to make sure you optimise for engineers coding stuff which
>> works on master, not maintenance.
>
>
> Well it depends on your strategy. Note that with my proposal you can always
> do that (optimize features for development not for maintenance).... just put
> your PR on master branch. That's all...
>

The problem is that my master-only changes will also make your
maintenance changes harder to merge into master. Would you be willing
to cherry-pick others' commits into a maintenance branch just to make
your pull request apply cleanly to both branches?

> The proposal is all about making maintenance easier. Development stays the
> same.
>
>>
>> Isn't it a bit naive to expect people to work on the "last maintained
>> branch" ?
>
>
> I don't think so. In that past I worked on a project with 3 maintenance
> branches plus a development branch. It worked. That's why I'm proposing this
> here.
>
>>
>> Your proposal expect that each and every patch author:
>>   a) knows for sure it's not going to be backported further to even
>> older branches
>
>
> As I explained this to Radim, occasional cherry-picking is totally fine.
> During the merge git will look at the commit, say... oh this commit is
> already there and move on.
>

Unless the commit was already modified because of a conflict...

>>
>>   b) you're not investigating issues on master (as investigation often
>> implies making code changes already)
>
>
> Again it depends. If you do not care about maintenance branches, you can
> still code at master branch.
>
> I also disagree with you here. We should always fix errors where they occur
> and not only in maintenance branch. If the failure is on 9.1.x branch, then
> the fix should go there. Not to master.
>

I agree with Sanne. If there's already a 9.2.x branch, I really don't
want to look into all the previous releases to see if it also
reproduces there. I may not even check if it's reproducible on 9.2.x,
if I know the 10.0.x release is coming "soon".

>>
>> Both of these are not very practical as I generally don't expect to
>> backport each and every fix, and it's going to be someone else to
>> recommend/approve some backports *after* the problem has been
>> understood in depth.
>
>
> In that kind of workflow my proposal wouldn't work well. I'm also a bit
> curious, how often do we need such an approval?
>

At least in my case, I'd call it "prodding" instead of "approval", and
it happens fairly often.

>>
>> You even mention "forgetting to backport refactoring" ? Hell yes you
>> should never backport refactorings..
>
>
> It depends. How about our checkstyle changes? After some time I guess I
> should have backported them to make cherry-picking easier.
>
>>
>> What I'm reading here is that you're all doing way too many backports.
>> Be more conservative about them?
>
>
> That's a very well stated question. My proposal makes the maintenance easier
> but developers need to be a bit more responsible (they need to know in which
> branch to implement a given fix and remember about doing merges). On the
> other hand if we prefer to encourage our users to move to more recent
> version quicker, than it probably doesn't make much sense.
>

IMO we should never have more than one maintenance branch in which we
do regular backports. We may have to do the occasional backport into
older branches, but that should be the exception, not the rule.

>>
>> In my experience with recent contributions to Infinispan - either
>> because the testsuite is slow, or because it takes us too long to
>> figure out there has been a regression, I sent a PR to master and two
>> maintenance branches but I had to repeat the exercise two more times
>> to add follow up fixes on the same subject: if I had waited on
>> backports it would have taken me a third of the time.
>
>
> Note that my proposal partially addresses this concern. Individual PRs
> wouldn't need to go through CI loop but they would be merged in batches. On
> the other hand, even an easy merge might break the build.
>

-1000 to blind merges, the master build is broken enough as it is.

Cheers
Dan

From galder at redhat.com  Mon Mar 27 18:40:47 2017
From: galder at redhat.com (=?utf-8?Q?Galder_Zamarre=C3=B1o?=)
Date: Tue, 28 Mar 2017 00:40:47 +0200
Subject: [infinispan-dev] Stream operations under lock
In-Reply-To: <CA+nfvwTaTFf_Ba4FkLAhejjQwC=p9YeiDuTrUaGj1krr2xgobw@mail.gmail.com>
References: <CA+YCuUM3h0H+ZDe1ZvK5mOuXnLB9bjjP+CkJg_M8eyGKC+jRHA@mail.gmail.com>
	<CA+nfvwTaTFf_Ba4FkLAhejjQwC=p9YeiDuTrUaGj1krr2xgobw@mail.gmail.com>
Message-ID: <28A6D5BF-6878-4778-A48A-489D91A2BBCB@redhat.com>


--
Galder Zamarre?o
Infinispan, Red Hat

> On 21 Mar 2017, at 17:16, Dan Berindei <dan.berindei at gmail.com> wrote:
> 
> I'm leaning towards option 1.
> 
> Are you thinking about also allowing the consumer to modify the entry,
> like JCache's EntryProcessors? For a consumer that can only modify the
> current entry, we could even "emulate" locking in an optimistic cache
> by catching the WriteSkewException and running the consumer again.
> 
> I wouldn't allow this to be mixed with other operations in a stream,
> because then you may have to run filters/mappers/sorting while holding
> the lock as well.

^ Would forEach w/ lock still run for all entries in originator? If so, not being able to filter could be a pain. IOW, you'd be forcing all entries to be shipped to a node and user to do its own filtering. Not ideal :\


> 
> Cheers
> Dan
> 
> 
> On Tue, Mar 21, 2017 at 5:37 PM, William Burns <mudokonman at gmail.com> wrote:
>> Some users have expressed the need to have some sort of forEach operation
>> that is performed where the Consumer is called while holding the lock for
>> the given key and subsequently released after the Consumer operation
>> completes.
>> 
>> Due to the nature of how streams work with retries and performing the
>> operation on the primary owner, this works out quite well with forEach to be
>> done in an efficient way.
>> 
>> The problem is that this only really works well with non tx and pessimistic
>> tx. This obviously leaves out optimistic tx, which at first I was a little
>> worried about. But after thinking about it more, this prelocking and
>> optimistic tx don't really fit that well together anyways. So I am thinking
>> whenever this operation is performed it would throw an exception not letting
>> the user use this feature in optimistic transactions.
>> 
>> Another question is what does the API for this look like. I was debating
>> between 3 options myself:
>> 
>> 1. AdvancedCache.forEachWithLock(BiConsumer<Cache, CacheEntry<K, V>>
>> consumer)
>> 
>> This require the least amount of changes, however the user can't customize
>> certain parameters that CacheStream currently provides (listed below - big
>> one being filterKeys).
>> 
>> 2. CacheStream.forEachWithLock(BiConsumer<Cache, CacheEntry<K, V>> consumer)
>> 
>> This method would only be allowed to be invoked on the Stream if no other
>> intermediate operations were invoked, otherwise an exception would be
>> thrown. This still gives us access to all of the CacheStream methods that
>> aren't on the Stream interface (ie. sequentialDistribution,
>> parallelDistribution, parallel, sequential, filterKeys, filterKeySegments,
>> distributedBatchSize, disableRehashAware, timeout).
>> 
>> 3. LockedStream<CacheEntry<K, V>> AdvancedCache.lockedStream()
>> 
>> This requires the most changes, however the API would be the most explicit.
>> In this case the LockedStream would only have the methods on it that are
>> able to be invoked as noted above and forEach.
>> 
>> I personally feel that #3 might be the cleanest, but obviously requires
>> adding more classes. Let me know what you guys think and if you think the
>> optimistic exclusion is acceptable.
>> 
>> Thanks,
>> 
>> - Will
>> 
>> _______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev



From dan.berindei at gmail.com  Tue Mar 28 07:19:51 2017
From: dan.berindei at gmail.com (Dan Berindei)
Date: Tue, 28 Mar 2017 14:19:51 +0300
Subject: [infinispan-dev] TestingUtil.shortTimeoutMillis
In-Reply-To: <fb1cab12-2d39-a017-5aef-4ab1a3e94700@redhat.com>
References: <3c77bc70-f732-2a46-2a01-0a97ed2f8277@redhat.com>
	<CAA9R9O536c93v05=jM9yDwbe=yZWebnynpw-Uo+ps5YzG9xNSA@mail.gmail.com>
	<CA+nfvwQhd6yHDwHFKMM-nqAswccKjAg0V-jMfXXBZQ7H+dqHkQ@mail.gmail.com>
	<fb1cab12-2d39-a017-5aef-4ab1a3e94700@redhat.com>
Message-ID: <CA+nfvwQLkZL2-13Le82DGKx5cMBWFsdEe=F0K5rBirWeP9qTog@mail.gmail.com>

On Mon, Mar 27, 2017 at 1:35 PM, Radim Vansa <rvansa at redhat.com> wrote:
> In regular ones [1] - see SyncLockingTest
>

shortTimeoutMillis() doesn't really fit here, it's intended for tests
that expect the operation to time out. I probably used it only because
the cache was already configured with 500ms.

OTOH there's no contention here, except for the fact that
TxCompletionNotification is async, so 500ms should be plenty even on
our puny CI agents...

> Btw., how do I enable TRACE build in CI?
>

I meant this build:

http://ci.infinispan.org/viewType.html?buildTypeId=Infinispan_MasterHotspotJdk7trac

You can also enable trace logging in custom builds (the Run... button)
by changing the maven_opts.log.debug parameter to include
log4j2-trace.xml instead of log4j2.xml, it's just that there's no
filtering step so the logs will be quite big.

>
> [1] http://ci.infinispan.org/viewLog.html?buildId=53604&buildTypeId=bt9
>
> On 03/27/2017 11:25 AM, Dan Berindei wrote:
>> Radim, are you seeing this in the regular builds too, or just in the
>> TRACE build?
>>
>> Dan
>>
>>
>> On Mon, Mar 27, 2017 at 11:59 AM, Sebastian Laskawiec
>> <slaskawi at redhat.com> wrote:
>>> Sounds good...
>>>
>>> Sadly our build infrastructure is not very fast...
>>>
>>> On Mon, Mar 27, 2017 at 10:58 AM Radim Vansa <rvansa at redhat.com> wrote:
>>>> Hi,
>>>>
>>>> I see that short timeout (500 ms) is an often cause of random failures
>>>> in CI. The original issue (ISPN-4772) raised this from 100 to 500; could
>>>> we increase this on CI even further, to, say, 3 seconds? It's
>>>> configurable via a sys property
>>>>
>>>> infinispan.test.shortTimeoutMillis
>>>>
>>>> Radim
>>>>
>>>> --
>>>> Radim Vansa <rvansa at redhat.com>
>>>> JBoss Performance Team
>>>>
>>>> _______________________________________________
>>>> infinispan-dev mailing list
>>>> infinispan-dev at lists.jboss.org
>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>>
>>> _______________________________________________
>>> infinispan-dev mailing list
>>> infinispan-dev at lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>> _______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
>
> --
> Radim Vansa <rvansa at redhat.com>
> JBoss Performance Team
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev

From galder at redhat.com  Mon Mar 27 18:50:42 2017
From: galder at redhat.com (=?utf-8?Q?Galder_Zamarre=C3=B1o?=)
Date: Tue, 28 Mar 2017 00:50:42 +0200
Subject: [infinispan-dev] Stream operations under lock
In-Reply-To: <914f7208-ae02-7d9b-cf0b-6f5feb332609@redhat.com>
References: <CA+YCuUM3h0H+ZDe1ZvK5mOuXnLB9bjjP+CkJg_M8eyGKC+jRHA@mail.gmail.com>
	<d7e8a644-cb77-3e2a-f17b-79ceb70b2d85@redhat.com>
	<CA+YCuUOMttxZqxSpFzvPu_Q0sFS_W0P9Wh062UmSWGWewmgxaA@mail.gmail.com>
	<CA+YCuUML6s77x9JNk6m1o=c526HK6porbaymNoFK=nmj7Ny24w@mail.gmail.com>
	<914f7208-ae02-7d9b-cf0b-6f5feb332609@redhat.com>
Message-ID: <2F47C5D6-6C46-4A66-9B66-201E07665A3F@redhat.com>


--
Galder Zamarre?o
Infinispan, Red Hat

> On 22 Mar 2017, at 10:51, Radim Vansa <rvansa at redhat.com> wrote:
> 
> On 03/21/2017 06:50 PM, William Burns wrote:
>> 
>> 
>> On Tue, Mar 21, 2017 at 1:42 PM William Burns <mudokonman at gmail.com 
>> <mailto:mudokonman at gmail.com>> wrote:
>> 
>>    On Tue, Mar 21, 2017 at 12:53 PM Radim Vansa <rvansa at redhat.com
>>    <mailto:rvansa at redhat.com>> wrote:
>> 
>>        On 03/21/2017 04:37 PM, William Burns wrote:
>>> Some users have expressed the need to have some sort of forEach
>>> operation that is performed where the Consumer is called
>>        while holding
>>> the lock for the given key and subsequently released after the
>>> Consumer operation completes.
>> 
>>        Seconding Dan's question - is that intended to be able to
>>        modify the
>>        entry? In my opinion, sending a function that will work on the
>>        ReadWriteEntryView directly to the node is the only reasonable
>>        request.
>>        I wouldn't like to see blocking operations in there.
>> 
>> 
>>    Hrmm the user can use the FunctionalMap interface for this then it
>>    seems? I wonder if this should just be the going in API. I will
>>    need to discuss with Galder the semantics of the evalAll/evalMany
>>    methods.
>> 
>> 
>> Actually looking at evalAll it seems it doesn't scale as it keeps all 
>> entries in memory at once, so this is only for caches with a limited 
>> amount of entries.
> 
> Don't look into the implementation; I think Galder has focused more on 
> the API side than having optimal implementation.

That's why it's marked experimental ;p

> IMO there's no reason 
> evalAll should load all the entries into memory in non-transactional mode.
> 
>> 
>>> 
>>> Due to the nature of how streams work with retries and
>>        performing the
>>> operation on the primary owner, this works out quite well
>>        with forEach
>>> to be done in an efficient way.
>>> 
>>> The problem is that this only really works well with non tx and
>>> pessimistic tx. This obviously leaves out optimistic tx,
>>        which at
>>> first I was a little worried about. But after thinking about
>>        it more,
>>> this prelocking and optimistic tx don't really fit that well
>>        together
>>> anyways. So I am thinking whenever this operation is
>>        performed it
>>> would throw an exception not letting the user use this
>>        feature in
>>> optimistic transactions.
>> 
>>        How exactly reading streams interacts with transactions? Does
>>        it wrap
>>        read entries into context? This would be a scalability issue.
>> 
>> 
>>    It doesn't wrap read entries into the context for that exact
>>    reason. It does however use existing entries in the context to
>>    override ones in memory/store.
>> 
> 
> Uuh, so you end up with a copy of the cache in single invocation 
> context, without any means to flush it. I think that we need add 
> InvocationContext.current().forget(key) API (throwing exception if the 
> entry was modified) or something like that, even for the regular 
> streams. Maybe an override for filter methods, too, because you want to 
> pass a nice predicate, but you can't just forget all filtered out entries.
> 
>> 
>>        I agree that "locking" should not be exposed with optimistic
>>        transactions.
>> 
>> 
>>    Yeah I can't find a good way to do this really and it seems to be
>>    opposite of what optimistic transactions are.
>> 
>> 
>>        With pessimistic transactions, how do you expect to handle locking
>>        order? For regular operations, user is responsible for setting
>>        up some
>>        locking order in order to not get a deadlock. With pessimistic
>>        transaction, it's the cache itself who will order the calls.
>>        Also, if
>>        you lock anything that is read, you just end up locking
>>        everything (or,
>>        getting a deadlock). If you don't it's the same as issuing the
>>        lock and
>>        reading again (to check the locked value) - but you'd do that
>>        internally
>>        anyway. Therefore, I don't feel well about pessimistic
>>        transactions neither.
>> 
>> 
>>    The lock is done per key only for each invocation. There is no
>>    ordering as only one is obtained at a time before it goes to the
>>    next. If the user then acquires a lock for another key while in
>>    the Consumer this could cause a deadlock if the inverse occurs on
>>    a different thread/node, but this is on the user. It is the same
>>    as it is today really, except we do the read lock for them before
>>    invoking their Consumer.
>> 
> 
> In pessimistic mode, you should not release a lock before the end of the 
> transaction.
> 
>> 
>>> 
>>> Another question is what does the API for this look like. I was
>>> debating between 3 options myself:
>>> 
>>> 1. AdvancedCache.forEachWithLock(BiConsumer<Cache,
>>        CacheEntry<K, V>>
>>> consumer)
>>> 
>>> This require the least amount of changes, however the user can't
>>> customize certain parameters that CacheStream currently provides
>>> (listed below - big one being filterKeys).
>>> 
>>> 2. CacheStream.forEachWithLock(BiConsumer<Cache,
>>        CacheEntry<K, V>>
>>> consumer)
>>> 
>>> This method would only be allowed to be invoked on the
>>        Stream if no
>>> other intermediate operations were invoked, otherwise an
>>        exception
>>> would be thrown. This still gives us access to all of the
>>        CacheStream
>>> methods that aren't on the Stream interface (ie.
>>> sequentialDistribution, parallelDistribution, parallel,
>>        sequential,
>>> filterKeys, filterKeySegments, distributedBatchSize,
>>> disableRehashAware, timeout).
>> 
>>        For both options, I don't like Cache being passed around. You
>>        should
>>        modify the CacheEntry (or some kind of view) directly.
>> 
>> 
>>    I don't know for sure if that is sufficient for the user.
>>    Sometimes they may modify another Cache given the value in this
>>    one for example, which they could access from the CacheManager of
>>    that Cache. Maybe Tristan knows more about some use cases.
>> 
> 
> Rather than guessing what could the user need, the Consumer could be CDI 
> enabled.
> 
>> 
>>        Radim
>> 
>>> 
>>> 3. LockedStream<CacheEntry<K, V>> AdvancedCache.lockedStream()
>>> 
>>> This requires the most changes, however the API would be the
>>        most
>>> explicit. In this case the LockedStream would only have the
>>        methods on
>>> it that are able to be invoked as noted above and forEach.
>>> 
>>> I personally feel that #3 might be the cleanest, but obviously
>>> requires adding more classes. Let me know what you guys
>>        think and if
>>> you think the optimistic exclusion is acceptable.
>>> 
>>> Thanks,
>>> 
>>> - Will
>>> 
>>> 
>>> _______________________________________________
>>> infinispan-dev mailing list
>>> infinispan-dev at lists.jboss.org
>>        <mailto:infinispan-dev at lists.jboss.org>
>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>> 
>> 
>>        --
>>        Radim Vansa <rvansa at redhat.com <mailto:rvansa at redhat.com>>
>>        JBoss Performance Team
>> 
>>        _______________________________________________
>>        infinispan-dev mailing list
>>        infinispan-dev at lists.jboss.org
>>        <mailto:infinispan-dev at lists.jboss.org>
>>        https://lists.jboss.org/mailman/listinfo/infinispan-dev
>> 
>> 
>> 
>> _______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> 
> 
> -- 
> Radim Vansa <rvansa at redhat.com>
> JBoss Performance Team
> 
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev



From galder at redhat.com  Mon Mar 27 19:15:43 2017
From: galder at redhat.com (=?utf-8?Q?Galder_Zamarre=C3=B1o?=)
Date: Tue, 28 Mar 2017 01:15:43 +0200
Subject: [infinispan-dev] Branching proposal
In-Reply-To: <CAA9R9O6X3kOu3h4=_2z6RZKExYC30B0JiJMvEYQ3mu9HGq20Rw@mail.gmail.com>
References: <CAA9R9O6X3kOu3h4=_2z6RZKExYC30B0JiJMvEYQ3mu9HGq20Rw@mail.gmail.com>
Message-ID: <A8D365BF-8BF1-4C60-893B-F65901C2217C@redhat.com>

Why are we working in 9.1.x, 9.2.x and master in paralell? We normally work on master and maybe one more maintenance branch.

Except for occasional tricky backports (e.g. Radim's work) the rest has been pretty straightforward for me. Also, the number of backports I work on is low in general.

Cheers,
--
Galder Zamarre?o
Infinispan, Red Hat

> On 27 Mar 2017, at 11:33, Sebastian Laskawiec <slaskawi at redhat.com> wrote:
> 
> Hey!
> 
> We are about to start working on 9.1.x and 9.2.y branches so I would like to propose alternative merging strategy.
> 
> Our current workflow looks like this:
> 
> X - new commit
> X` - cherry pick to maintenance branch
> --+-------------------+-------X----- master
>   |                    \------X`---- 9.2.x
>   \---------------------------X``--- 9.1.x
> 
> Each commit needs to be reviewed in master branch and backported to the maintenance branches. From maintenance perspective this is a bit painful, since in above example we need to get 3 times through PR queue. Also it's worth to mention that X is not X` nor X``. Cherry-picking creates a copy of a commit. This makes some useful tricks (like git tag --contains <sha1>) a bit harder to use. Finally, this approach allows the codebase to diverge from maintenance branches very fast (someone might just forget to backport some of the refactoring stuff).
> 
> The proposal:
> 
> X, Y - new commits
> / - merge commits
> --+---------+------/----/--- master
>   |          \----/---Y/---- 9.2.x
>   \-------------X/---------- 9.1.x
> 
> With the proposal, a developer should always implement a given feature in the lowest possible maintenance branch. Then we will run a set of merges from 9.1.x into 9.2.x and finally into master. The biggest advantage of this approach is that given functionality (identified by a commit) will have the same SHA1 for all branches. This will allow all tools like (mentioned before) `git tag --contains <sha1>` to work. There are also some further implications of this approach:
> 	? Merging commits should be performed very often (even automatically in the night (if merged without any problems)).
> 	? After releasing each maintenance release, someone will need to do a merge with strategy `ours` (`git merge -s ours upstream/9.2.x`). This way we will not have to solve version conflicts in poms.
> 	? Since there is no nice way to rebase a merge commit, they should be pushed directly into the master branch (without review, without CI). After the merge, HEAD will change and CI will automatically pick the build. Remember, merges should be done very often. So I assume there won't be any problems most of the times.
> 	? Finally, with this approach the code diverges slight slower (at least from my experience). Mainly because we don't need to remember to cherry-pick individual commits. They are automatically "taken" by a merge.
> From my past experience, this strategy works pretty nice and can be almost fully automated. It significantly lowers the maintenance pain around cherry-picks. However there is nothing for free, and we would need to get used to pushing merged directly into master (which is fine to me but some of you might not like it).
> 
> Thanks,
> Sebastian
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev



From galder at redhat.com  Mon Mar 27 19:16:49 2017
From: galder at redhat.com (=?utf-8?Q?Galder_Zamarre=C3=B1o?=)
Date: Tue, 28 Mar 2017 01:16:49 +0200
Subject: [infinispan-dev] Branching proposal
In-Reply-To: <CA+nfvwRjtzdO=gS_E5PqhtA5qe=3pEyG9y0jXi316vzs=cOQUA@mail.gmail.com>
References: <CAA9R9O6X3kOu3h4=_2z6RZKExYC30B0JiJMvEYQ3mu9HGq20Rw@mail.gmail.com>
	<f9406b3d-af44-1627-7037-15ff080d6e3f@redhat.com>
	<CA+nfvwRjtzdO=gS_E5PqhtA5qe=3pEyG9y0jXi316vzs=cOQUA@mail.gmail.com>
Message-ID: <29071401-E777-473A-84CB-BCC0908DF1E1@redhat.com>

Nice one-liner. The fact that we always put the JIRA id helps.

Cheers,
--
Galder Zamarre?o
Infinispan, Red Hat

> On 27 Mar 2017, at 14:36, Dan Berindei <dan.berindei at gmail.com> wrote:
> 
> I use something like this to check what tags contain a particular fix:
> 
> git tag --contains $(git log --grep <JIRA> -1 --format="%h" master)
> 
> True, it's a bit longer, but it stays in the bash/zsh history :)
> 
> Cheers
> Dan
> 
> 
> On Mon, Mar 27, 2017 at 1:33 PM, Radim Vansa <rvansa at redhat.com> wrote:
>> If you can't merge a commit (based on 9.0.x) to master clearly, do you
>> need to file another PR anyway? Then the lag to get some code to master
>> increases a lot. I am not sure how useful is git tag --contains <sha1>
>> if you cannot be sure that you'll find all occurrences due to this kind
>> of issues.
>> 
>> R.
>> 
>> On 03/27/2017 11:33 AM, Sebastian Laskawiec wrote:
>>> Hey!
>>> 
>>> We are about to start working on 9.1.x and 9.2.y branches so I would
>>> like to propose alternative merging strategy.
>>> 
>>> Our current workflow looks like this:
>>> 
>>> X - new commit
>>> X` - cherry pick to maintenance branch
>>> --+-------------------+-------X----- master
>>>  |                    \------X`---- 9.2.x
>>>  \---------------------------X``--- 9.1.x
>>> 
>>> Each commit needs to be reviewed in master branch and backported to
>>> the maintenance branches. From maintenance perspective this is a bit
>>> painful, since in above example we need to get 3 times through PR
>>> queue. Also it's worth to mention that X is not X` nor X``.
>>> Cherry-picking creates a copy of a commit. This makes some useful
>>> tricks (like git tag --contains <sha1>) a bit harder to use. Finally,
>>> this approach allows the codebase to diverge from maintenance branches
>>> very fast (someone might just forget to backport some of the
>>> refactoring stuff).
>>> 
>>> The proposal:
>>> 
>>> X, Y - new commits
>>> / - merge commits
>>> --+---------+------/----/--- master
>>>  |          \----/---Y/---- 9.2.x
>>>  \-------------X/---------- 9.1.x
>>> 
>>> With the proposal, a developer should always implement a given feature
>>> in the lowest possible maintenance branch. Then we will run a set of
>>> merges from 9.1.x into 9.2.x and finally into master. The biggest
>>> advantage of this approach is that given functionality (identified by
>>> a commit) will have the same SHA1 for all branches. This will allow
>>> all tools like (mentioned before) `git tag --contains <sha1>` to work.
>>> There are also some further implications of this approach:
>>> 
>>>  * Merging commits should be performed very often (even automatically
>>>    in the night (if merged without any problems)).
>>>  * After releasing each maintenance release, someone will need to do
>>>    a merge with strategy `ours` (`git merge -s ours upstream/9.2.x`).
>>>    This way we will not have to solve version conflicts in poms.
>>>  * Since there is no nice way to rebase a merge commit, they should
>>>    be pushed directly into the master branch (without review, without
>>>    CI). After the merge, HEAD will change and CI will
>>>    automatically pick the build. Remember, merges should be done very
>>>    often. So I assume there won't be any problems most of the times.
>>>  * Finally, with this approach the code diverges slight slower (at
>>>    least from my experience). Mainly because we don't need to
>>>    remember to cherry-pick individual commits. They are automatically
>>>    "taken" by a merge.
>>> 
>>> From my past experience, this strategy works pretty nice and can be
>>> almost fully automated. It significantly lowers the maintenance pain
>>> around cherry-picks. However there is nothing for free, and we would
>>> need to get used to pushing merged directly into master (which is fine
>>> to me but some of you might not like it).
>>> 
>>> Thanks,
>>> Sebastian
>>> 
>>> 
>>> _______________________________________________
>>> infinispan-dev mailing list
>>> infinispan-dev at lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>> 
>> 
>> --
>> Radim Vansa <rvansa at redhat.com>
>> JBoss Performance Team
>> 
>> _______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev



From galder at redhat.com  Mon Mar 27 19:24:41 2017
From: galder at redhat.com (=?utf-8?Q?Galder_Zamarre=C3=B1o?=)
Date: Tue, 28 Mar 2017 01:24:41 +0200
Subject: [infinispan-dev] Stream operations under lock
In-Reply-To: <CA+YCuUML6s77x9JNk6m1o=c526HK6porbaymNoFK=nmj7Ny24w@mail.gmail.com>
References: <CA+YCuUM3h0H+ZDe1ZvK5mOuXnLB9bjjP+CkJg_M8eyGKC+jRHA@mail.gmail.com>
	<d7e8a644-cb77-3e2a-f17b-79ceb70b2d85@redhat.com>
	<CA+YCuUOMttxZqxSpFzvPu_Q0sFS_W0P9Wh062UmSWGWewmgxaA@mail.gmail.com>
	<CA+YCuUML6s77x9JNk6m1o=c526HK6porbaymNoFK=nmj7Ny24w@mail.gmail.com>
Message-ID: <3FB487C2-DC28-45D5-A160-9C1585EC598A@redhat.com>

An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/infinispan-dev/attachments/20170328/511ce95f/attachment.html 

From mudokonman at gmail.com  Tue Mar 28 09:36:39 2017
From: mudokonman at gmail.com (William Burns)
Date: Tue, 28 Mar 2017 13:36:39 +0000
Subject: [infinispan-dev] Branching proposal
In-Reply-To: <A8D365BF-8BF1-4C60-893B-F65901C2217C@redhat.com>
References: <CAA9R9O6X3kOu3h4=_2z6RZKExYC30B0JiJMvEYQ3mu9HGq20Rw@mail.gmail.com>
	<A8D365BF-8BF1-4C60-893B-F65901C2217C@redhat.com>
Message-ID: <CA+YCuUOcM9xtdGzi-pQP-xm87kFhiZnq6+pTy4vKBgkp=+aB4g@mail.gmail.com>

On Tue, Mar 28, 2017 at 9:27 AM Galder Zamarre?o <galder at redhat.com> wrote:

> Why are we working in 9.1.x, 9.2.x and master in paralell? We normally
> work on master and maybe one more maintenance branch.
>
> Except for occasional tricky backports (e.g. Radim's work) the rest has
> been pretty straightforward for me. Also, the number of backports I work on
> is low in general.
>

+1 For me the hard part is just remembering it needs to be backported. And
as Sanne mentioned refactorings shouldn't really be backported and these
are the types of the things that cause the most conflicts. And to be honest
on some backports I might not pull every change since the real fix may have
been quite small.


>
> Cheers,
> --
> Galder Zamarre?o
> Infinispan, Red Hat
>
> > On 27 Mar 2017, at 11:33, Sebastian Laskawiec <slaskawi at redhat.com>
> wrote:
> >
> > Hey!
> >
> > We are about to start working on 9.1.x and 9.2.y branches so I would
> like to propose alternative merging strategy.
> >
> > Our current workflow looks like this:
> >
> > X - new commit
> > X` - cherry pick to maintenance branch
> > --+-------------------+-------X----- master
> >   |                    \------X`---- 9.2.x
> >   \---------------------------X``--- 9.1.x
> >
> > Each commit needs to be reviewed in master branch and backported to the
> maintenance branches. From maintenance perspective this is a bit painful,
> since in above example we need to get 3 times through PR queue. Also it's
> worth to mention that X is not X` nor X``. Cherry-picking creates a copy of
> a commit. This makes some useful tricks (like git tag --contains <sha1>) a
> bit harder to use. Finally, this approach allows the codebase to diverge
> from maintenance branches very fast (someone might just forget to backport
> some of the refactoring stuff).
> >
> > The proposal:
> >
> > X, Y - new commits
> > / - merge commits
> > --+---------+------/----/--- master
> >   |          \----/---Y/---- 9.2.x
> >   \-------------X/---------- 9.1.x
> >
> > With the proposal, a developer should always implement a given feature
> in the lowest possible maintenance branch. Then we will run a set of merges
> from 9.1.x into 9.2.x and finally into master. The biggest advantage of
> this approach is that given functionality (identified by a commit) will
> have the same SHA1 for all branches. This will allow all tools like
> (mentioned before) `git tag --contains <sha1>` to work. There are also some
> further implications of this approach:
> >       ? Merging commits should be performed very often (even
> automatically in the night (if merged without any problems)).
> >       ? After releasing each maintenance release, someone will need to
> do a merge with strategy `ours` (`git merge -s ours upstream/9.2.x`). This
> way we will not have to solve version conflicts in poms.
> >       ? Since there is no nice way to rebase a merge commit, they should
> be pushed directly into the master branch (without review, without CI).
> After the merge, HEAD will change and CI will automatically pick the build.
> Remember, merges should be done very often. So I assume there won't be any
> problems most of the times.
> >       ? Finally, with this approach the code diverges slight slower (at
> least from my experience). Mainly because we don't need to remember to
> cherry-pick individual commits. They are automatically "taken" by a merge.
> > From my past experience, this strategy works pretty nice and can be
> almost fully automated. It significantly lowers the maintenance pain around
> cherry-picks. However there is nothing for free, and we would need to get
> used to pushing merged directly into master (which is fine to me but some
> of you might not like it).
> >
> > Thanks,
> > Sebastian
> > _______________________________________________
> > infinispan-dev mailing list
> > infinispan-dev at lists.jboss.org
> > https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/infinispan-dev/attachments/20170328/017f8229/attachment.html 

From mudokonman at gmail.com  Tue Mar 28 09:42:11 2017
From: mudokonman at gmail.com (William Burns)
Date: Tue, 28 Mar 2017 13:42:11 +0000
Subject: [infinispan-dev] Stream operations under lock
In-Reply-To: <3FB487C2-DC28-45D5-A160-9C1585EC598A@redhat.com>
References: <CA+YCuUM3h0H+ZDe1ZvK5mOuXnLB9bjjP+CkJg_M8eyGKC+jRHA@mail.gmail.com>
	<d7e8a644-cb77-3e2a-f17b-79ceb70b2d85@redhat.com>
	<CA+YCuUOMttxZqxSpFzvPu_Q0sFS_W0P9Wh062UmSWGWewmgxaA@mail.gmail.com>
	<CA+YCuUML6s77x9JNk6m1o=c526HK6porbaymNoFK=nmj7Ny24w@mail.gmail.com>
	<3FB487C2-DC28-45D5-A160-9C1585EC598A@redhat.com>
Message-ID: <CA+YCuUPvwHVSJpfj_zhCLShCgEZT=LvyFAeoaiWYCwcxAfvPZg@mail.gmail.com>

On Tue, Mar 28, 2017 at 9:27 AM Galder Zamarre?o <galder at redhat.com> wrote:

>
> --
> Galder Zamarre?o
> Infinispan, Red Hat
>
> On 21 Mar 2017, at 18:50, William Burns <mudokonman at gmail.com> wrote:
>
>
>
> On Tue, Mar 21, 2017 at 1:42 PM William Burns <mudokonman at gmail.com>
> wrote:
> On Tue, Mar 21, 2017 at 12:53 PM Radim Vansa <rvansa at redhat.com> wrote:
> On 03/21/2017 04:37 PM, William Burns wrote:
>
> Some users have expressed the need to have some sort of forEach
> operation that is performed where the Consumer is called while holding
> the lock for the given key and subsequently released after the
> Consumer operation completes.
>
>
> Seconding Dan's question - is that intended to be able to modify the
> entry? In my opinion, sending a function that will work on the
> ReadWriteEntryView directly to the node is the only reasonable request.
> I wouldn't like to see blocking operations in there.
>
> Hrmm the user can use the FunctionalMap interface for this then it seems?
> I wonder if this should just be the going in API. I will need to discuss
> with Galder the semantics of the evalAll/evalMany methods.
>
> Actually looking at evalAll it seems it doesn't scale as it keeps all
> entries in memory at once, so this is only for caches with a limited amount
> of entries.
>
>
> ^ I might be wrong but didn't forEach work this way? I probably looked at
> that when trying to implement evalAll
>

It actually is very similar to the distributed iterator. Basically it
performs distributedBatchSize number of consumer calls then tells the
originator it finished those keys sending them back and continues on. This
way at most you can have distributedBatchSize * numNodes worth of more than
once calls per state transfer. In practice this will be significantly lower
though as you would only have a subset and you have to lose complete
ownership of said key.

While keys are sent across the wire, values are never returned with
forEach. And key references are released when a segment is completed so it
should have a subset of keys in memory.


>
>
>
>
>
>
> Due to the nature of how streams work with retries and performing the
> operation on the primary owner, this works out quite well with forEach
> to be done in an efficient way.
>
> The problem is that this only really works well with non tx and
> pessimistic tx. This obviously leaves out optimistic tx, which at
> first I was a little worried about. But after thinking about it more,
> this prelocking and optimistic tx don't really fit that well together
> anyways. So I am thinking whenever this operation is performed it
> would throw an exception not letting the user use this feature in
> optimistic transactions.
>
>
> How exactly reading streams interacts with transactions? Does it wrap
> read entries into context? This would be a scalability issue.
>
> It doesn't wrap read entries into the context for that exact reason. It
> does however use existing entries in the context to override ones in
> memory/store.
>
>
> I agree that "locking" should not be exposed with optimistic transactions.
>
> Yeah I can't find a good way to do this really and it seems to be opposite
> of what optimistic transactions are.
>
>
> With pessimistic transactions, how do you expect to handle locking
> order? For regular operations, user is responsible for setting up some
> locking order in order to not get a deadlock. With pessimistic
> transaction, it's the cache itself who will order the calls. Also, if
> you lock anything that is read, you just end up locking everything (or,
> getting a deadlock). If you don't it's the same as issuing the lock and
> reading again (to check the locked value) - but you'd do that internally
> anyway. Therefore, I don't feel well about pessimistic transactions
> neither.
>
> The lock is done per key only for each invocation. There is no ordering as
> only one is obtained at a time before it goes to the next. If the user then
> acquires a lock for another key while in the Consumer this could cause a
> deadlock if the inverse occurs on a different thread/node, but this is on
> the user. It is the same as it is today really, except we do the read lock
> for them before invoking their Consumer.
>
>
>
> Another question is what does the API for this look like. I was
> debating between 3 options myself:
>
> 1. AdvancedCache.forEachWithLock(BiConsumer<Cache, CacheEntry<K, V>>
> consumer)
>
> This require the least amount of changes, however the user can't
> customize certain parameters that CacheStream currently provides
> (listed below - big one being filterKeys).
>
> 2. CacheStream.forEachWithLock(BiConsumer<Cache, CacheEntry<K, V>>
> consumer)
>
> This method would only be allowed to be invoked on the Stream if no
> other intermediate operations were invoked, otherwise an exception
> would be thrown. This still gives us access to all of the CacheStream
> methods that aren't on the Stream interface (ie.
> sequentialDistribution, parallelDistribution, parallel, sequential,
> filterKeys, filterKeySegments, distributedBatchSize,
> disableRehashAware, timeout).
>
>
> For both options, I don't like Cache being passed around. You should
> modify the CacheEntry (or some kind of view) directly.
>
> I don't know for sure if that is sufficient for the user. Sometimes they
> may modify another Cache given the value in this one for example, which
> they could access from the CacheManager of that Cache. Maybe Tristan knows
> more about some use cases.
>
>
> Radim
>
>
> 3. LockedStream<CacheEntry<K, V>> AdvancedCache.lockedStream()
>
> This requires the most changes, however the API would be the most
> explicit. In this case the LockedStream would only have the methods on
> it that are able to be invoked as noted above and forEach.
>
> I personally feel that #3 might be the cleanest, but obviously
> requires adding more classes. Let me know what you guys think and if
> you think the optimistic exclusion is acceptable.
>
> Thanks,
>
> - Will
>
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
>
>
> --
> Radim Vansa <rvansa at redhat.com>
> JBoss Performance Team
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/infinispan-dev/attachments/20170328/989cbc78/attachment-0001.html 

From mudokonman at gmail.com  Tue Mar 28 09:49:46 2017
From: mudokonman at gmail.com (William Burns)
Date: Tue, 28 Mar 2017 13:49:46 +0000
Subject: [infinispan-dev] Stream operations under lock
In-Reply-To: <28A6D5BF-6878-4778-A48A-489D91A2BBCB@redhat.com>
References: <CA+YCuUM3h0H+ZDe1ZvK5mOuXnLB9bjjP+CkJg_M8eyGKC+jRHA@mail.gmail.com>
	<CA+nfvwTaTFf_Ba4FkLAhejjQwC=p9YeiDuTrUaGj1krr2xgobw@mail.gmail.com>
	<28A6D5BF-6878-4778-A48A-489D91A2BBCB@redhat.com>
Message-ID: <CA+YCuUMm6XUWh+5wQcYaYfqqA6UKqA9KSwb6TNdZpFJeg_G+cA@mail.gmail.com>

On Mon, Mar 27, 2017 at 9:02 PM Galder Zamarre?o <galder at redhat.com> wrote:

>
> --
> Galder Zamarre?o
> Infinispan, Red Hat
>
> > On 21 Mar 2017, at 17:16, Dan Berindei <dan.berindei at gmail.com> wrote:
> >
> > I'm leaning towards option 1.
> >
> > Are you thinking about also allowing the consumer to modify the entry,
> > like JCache's EntryProcessors? For a consumer that can only modify the
> > current entry, we could even "emulate" locking in an optimistic cache
> > by catching the WriteSkewException and running the consumer again.
> >
> > I wouldn't allow this to be mixed with other operations in a stream,
> > because then you may have to run filters/mappers/sorting while holding
> > the lock as well.
>
> ^ Would forEach w/ lock still run for all entries in originator? If so,
> not being able to filter could be a pain. IOW, you'd be forcing all entries
> to be shipped to a node and user to do its own filtering. Not ideal :\
>

No the primary owner would run the operation per entry. I was thinking we
would have 2 levels of filtering in my proposal above.

We would have the first one which is using filterKeys on the CacheStream
method. This requires serializing keys sent to each node (only primary
owned keys are sent). While it has the cost of serialization it makes up
for by having constant time lookups (no iterating memory/stores) for the
keys as it creates a stream using Cache.get to populate it.

The second was to support the filter method on the Stream API which would
allow for a Predicate so you don't have to serialize keys. In this case you
wouldn't want to include keys in this Predicate as all keys would be
serialized to all nodes and then you still have to iterate and check the
entire data container/store.

You could actually do both as well. So if you only want a subset of known
keys where their values match a Predicate this can be done too.

cache.lockedStream().filterKeys(keys).filter(predicate).forEach();


>
>
> >
> > Cheers
> > Dan
> >
> >
> > On Tue, Mar 21, 2017 at 5:37 PM, William Burns <mudokonman at gmail.com>
> wrote:
> >> Some users have expressed the need to have some sort of forEach
> operation
> >> that is performed where the Consumer is called while holding the lock
> for
> >> the given key and subsequently released after the Consumer operation
> >> completes.
> >>
> >> Due to the nature of how streams work with retries and performing the
> >> operation on the primary owner, this works out quite well with forEach
> to be
> >> done in an efficient way.
> >>
> >> The problem is that this only really works well with non tx and
> pessimistic
> >> tx. This obviously leaves out optimistic tx, which at first I was a
> little
> >> worried about. But after thinking about it more, this prelocking and
> >> optimistic tx don't really fit that well together anyways. So I am
> thinking
> >> whenever this operation is performed it would throw an exception not
> letting
> >> the user use this feature in optimistic transactions.
> >>
> >> Another question is what does the API for this look like. I was debating
> >> between 3 options myself:
> >>
> >> 1. AdvancedCache.forEachWithLock(BiConsumer<Cache, CacheEntry<K, V>>
> >> consumer)
> >>
> >> This require the least amount of changes, however the user can't
> customize
> >> certain parameters that CacheStream currently provides (listed below -
> big
> >> one being filterKeys).
> >>
> >> 2. CacheStream.forEachWithLock(BiConsumer<Cache, CacheEntry<K, V>>
> consumer)
> >>
> >> This method would only be allowed to be invoked on the Stream if no
> other
> >> intermediate operations were invoked, otherwise an exception would be
> >> thrown. This still gives us access to all of the CacheStream methods
> that
> >> aren't on the Stream interface (ie. sequentialDistribution,
> >> parallelDistribution, parallel, sequential, filterKeys,
> filterKeySegments,
> >> distributedBatchSize, disableRehashAware, timeout).
> >>
> >> 3. LockedStream<CacheEntry<K, V>> AdvancedCache.lockedStream()
> >>
> >> This requires the most changes, however the API would be the most
> explicit.
> >> In this case the LockedStream would only have the methods on it that are
> >> able to be invoked as noted above and forEach.
> >>
> >> I personally feel that #3 might be the cleanest, but obviously requires
> >> adding more classes. Let me know what you guys think and if you think
> the
> >> optimistic exclusion is acceptable.
> >>
> >> Thanks,
> >>
> >> - Will
> >>
> >> _______________________________________________
> >> infinispan-dev mailing list
> >> infinispan-dev at lists.jboss.org
> >> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> > _______________________________________________
> > infinispan-dev mailing list
> > infinispan-dev at lists.jboss.org
> > https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/infinispan-dev/attachments/20170328/8dd3c405/attachment.html 

From sanne at infinispan.org  Tue Mar 28 10:55:29 2017
From: sanne at infinispan.org (Sanne Grinovero)
Date: Tue, 28 Mar 2017 15:55:29 +0100
Subject: [infinispan-dev] Stream operations under lock
In-Reply-To: <CA+YCuUMm6XUWh+5wQcYaYfqqA6UKqA9KSwb6TNdZpFJeg_G+cA@mail.gmail.com>
References: <CA+YCuUM3h0H+ZDe1ZvK5mOuXnLB9bjjP+CkJg_M8eyGKC+jRHA@mail.gmail.com>
	<CA+nfvwTaTFf_Ba4FkLAhejjQwC=p9YeiDuTrUaGj1krr2xgobw@mail.gmail.com>
	<28A6D5BF-6878-4778-A48A-489D91A2BBCB@redhat.com>
	<CA+YCuUMm6XUWh+5wQcYaYfqqA6UKqA9KSwb6TNdZpFJeg_G+cA@mail.gmail.com>
Message-ID: <CAFm4XO39HKQF==fo7H31BgZhwSnw6qp2_-UM9LQVX65KOcoz+w@mail.gmail.com>

Hi Will,

I'm confused about the premise; when you state

" the Consumer is called while holding the lock for the given key and
subsequently released after the Consumer operation completes."

What are the transaction boundaries?
I see two options, please correct me so I understand:

A)

transaction.begin();
cache.lockedStream().filterKeys(keys).filter(predicate).forEach(
someOperation );
transaction.commit(); <-- release all locks

B)

cache.lockedStream().filterKeys(keys).filter(predicate).forEach(
    transaction.begin();
    //to stuff on entry
    /transaction.commit(); <-- release this single entry's lock
);


I think it's important to clarify this as I suspect that #A is not
implementable within reasonable guarantees, while in the case of B# I
see no use for optimistic locking *from a user's perspective*.

Also: what kind of guarantees do you expect about having the operation
being applied on some entry which is strictly *still* a subset of the
keys as defined by the filter predicates?
I take it you'd want to acquire the locks during the filtering process?

That would require the move the transaction boundary to the scenario
A# which seems undesirable.
Technically if I were to need something like this I guess I'd expect
to have a user experience akin to B# but have Infinispan essentially
use optimistic locking (and auto-skip) on entries which are mutated
and fall out of the filter predicate during the lock attempt.

Essentially I suspect that we'd not want to implement different
versions of this depending on the transaction mode, but figure out the
use case and implement a one and only transaction mode which suites
such use cases. So for example we'd simply not offer a mode which
requires to copy the whole grid into the current TX context.

Thanks,
Sanne



On 28 March 2017 at 14:49, William Burns <mudokonman at gmail.com> wrote:
>
>
> On Mon, Mar 27, 2017 at 9:02 PM Galder Zamarre?o <galder at redhat.com> wrote:
>>
>>
>> --
>> Galder Zamarre?o
>> Infinispan, Red Hat
>>
>> > On 21 Mar 2017, at 17:16, Dan Berindei <dan.berindei at gmail.com> wrote:
>> >
>> > I'm leaning towards option 1.
>> >
>> > Are you thinking about also allowing the consumer to modify the entry,
>> > like JCache's EntryProcessors? For a consumer that can only modify the
>> > current entry, we could even "emulate" locking in an optimistic cache
>> > by catching the WriteSkewException and running the consumer again.
>> >
>> > I wouldn't allow this to be mixed with other operations in a stream,
>> > because then you may have to run filters/mappers/sorting while holding
>> > the lock as well.
>>
>> ^ Would forEach w/ lock still run for all entries in originator? If so,
>> not being able to filter could be a pain. IOW, you'd be forcing all entries
>> to be shipped to a node and user to do its own filtering. Not ideal :\
>
>
> No the primary owner would run the operation per entry. I was thinking we
> would have 2 levels of filtering in my proposal above.
>
> We would have the first one which is using filterKeys on the CacheStream
> method. This requires serializing keys sent to each node (only primary owned
> keys are sent). While it has the cost of serialization it makes up for by
> having constant time lookups (no iterating memory/stores) for the keys as it
> creates a stream using Cache.get to populate it.
>
> The second was to support the filter method on the Stream API which would
> allow for a Predicate so you don't have to serialize keys. In this case you
> wouldn't want to include keys in this Predicate as all keys would be
> serialized to all nodes and then you still have to iterate and check the
> entire data container/store.
>
> You could actually do both as well. So if you only want a subset of known
> keys where their values match a Predicate this can be done too.
>
> cache.lockedStream().filterKeys(keys).filter(predicate).forEach();
>
>>
>>
>>
>> >
>> > Cheers
>> > Dan
>> >
>> >
>> > On Tue, Mar 21, 2017 at 5:37 PM, William Burns <mudokonman at gmail.com>
>> > wrote:
>> >> Some users have expressed the need to have some sort of forEach
>> >> operation
>> >> that is performed where the Consumer is called while holding the lock
>> >> for
>> >> the given key and subsequently released after the Consumer operation
>> >> completes.
>> >>
>> >> Due to the nature of how streams work with retries and performing the
>> >> operation on the primary owner, this works out quite well with forEach
>> >> to be
>> >> done in an efficient way.
>> >>
>> >> The problem is that this only really works well with non tx and
>> >> pessimistic
>> >> tx. This obviously leaves out optimistic tx, which at first I was a
>> >> little
>> >> worried about. But after thinking about it more, this prelocking and
>> >> optimistic tx don't really fit that well together anyways. So I am
>> >> thinking
>> >> whenever this operation is performed it would throw an exception not
>> >> letting
>> >> the user use this feature in optimistic transactions.
>> >>
>> >> Another question is what does the API for this look like. I was
>> >> debating
>> >> between 3 options myself:
>> >>
>> >> 1. AdvancedCache.forEachWithLock(BiConsumer<Cache, CacheEntry<K, V>>
>> >> consumer)
>> >>
>> >> This require the least amount of changes, however the user can't
>> >> customize
>> >> certain parameters that CacheStream currently provides (listed below -
>> >> big
>> >> one being filterKeys).
>> >>
>> >> 2. CacheStream.forEachWithLock(BiConsumer<Cache, CacheEntry<K, V>>
>> >> consumer)
>> >>
>> >> This method would only be allowed to be invoked on the Stream if no
>> >> other
>> >> intermediate operations were invoked, otherwise an exception would be
>> >> thrown. This still gives us access to all of the CacheStream methods
>> >> that
>> >> aren't on the Stream interface (ie. sequentialDistribution,
>> >> parallelDistribution, parallel, sequential, filterKeys,
>> >> filterKeySegments,
>> >> distributedBatchSize, disableRehashAware, timeout).
>> >>
>> >> 3. LockedStream<CacheEntry<K, V>> AdvancedCache.lockedStream()
>> >>
>> >> This requires the most changes, however the API would be the most
>> >> explicit.
>> >> In this case the LockedStream would only have the methods on it that
>> >> are
>> >> able to be invoked as noted above and forEach.
>> >>
>> >> I personally feel that #3 might be the cleanest, but obviously requires
>> >> adding more classes. Let me know what you guys think and if you think
>> >> the
>> >> optimistic exclusion is acceptable.
>> >>
>> >> Thanks,
>> >>
>> >> - Will
>> >>
>> >> _______________________________________________
>> >> infinispan-dev mailing list
>> >> infinispan-dev at lists.jboss.org
>> >> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>> > _______________________________________________
>> > infinispan-dev mailing list
>> > infinispan-dev at lists.jboss.org
>> > https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>
>>
>> _______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev


From mudokonman at gmail.com  Tue Mar 28 11:44:05 2017
From: mudokonman at gmail.com (William Burns)
Date: Tue, 28 Mar 2017 15:44:05 +0000
Subject: [infinispan-dev] Stream operations under lock
In-Reply-To: <CAFm4XO39HKQF==fo7H31BgZhwSnw6qp2_-UM9LQVX65KOcoz+w@mail.gmail.com>
References: <CA+YCuUM3h0H+ZDe1ZvK5mOuXnLB9bjjP+CkJg_M8eyGKC+jRHA@mail.gmail.com>
	<CA+nfvwTaTFf_Ba4FkLAhejjQwC=p9YeiDuTrUaGj1krr2xgobw@mail.gmail.com>
	<28A6D5BF-6878-4778-A48A-489D91A2BBCB@redhat.com>
	<CA+YCuUMm6XUWh+5wQcYaYfqqA6UKqA9KSwb6TNdZpFJeg_G+cA@mail.gmail.com>
	<CAFm4XO39HKQF==fo7H31BgZhwSnw6qp2_-UM9LQVX65KOcoz+w@mail.gmail.com>
Message-ID: <CA+YCuUM87dvVkV0gGnnKVzpAPHyQd8fJkH_3iRF0ccyvUz9WPA@mail.gmail.com>

On Tue, Mar 28, 2017 at 11:24 AM Sanne Grinovero <sanne at infinispan.org>
wrote:

> Hi Will,
>
> I'm confused about the premise; when you state
>
> " the Consumer is called while holding the lock for the given key and
> subsequently released after the Consumer operation completes."
>
> What are the transaction boundaries?
> I see two options, please correct me so I understand:
>
> A)
>
> transaction.begin();
> cache.lockedStream().filterKeys(keys).filter(predicate).forEach(
> someOperation );
> transaction.commit(); <-- release all locks
>
> B)
>
> cache.lockedStream().filterKeys(keys).filter(predicate).forEach(
>     transaction.begin();
>     //to stuff on entry
>     /transaction.commit(); <-- release this single entry's lock
> );
>

The user code doesn't really affect this, it is done internally by
Infinispan. We would acquire the lock on the user's behalf in the stream
operation then call the user's accept method. Then after they return we
would unlock the lock. The way I have it implemented at the moment in my
PoC (which could change) is I don't even start a transaction. It just locks
the key and then if the user were to invoke an operation that requires a
transaction it adds the lock owner to their tx context at that point so
they have ownership of the key.

Also to note that a commit or rollback in the Consumer currently doesn't
release the lock on the key. Although this could be discussed to be
possibly changed.


>
>
> I think it's important to clarify this as I suspect that #A is not
> implementable within reasonable guarantees, while in the case of B# I
> see no use for optimistic locking *from a user's perspective*.
>

Exactly my thoughts regarding optimistic. I don't think #A is even feasible
given constraints of having a distributed transaction like this.


>
> Also: what kind of guarantees do you expect about having the operation
> being applied on some entry which is strictly *still* a subset of the
> keys as defined by the filter predicates?
> I take it you'd want to acquire the locks during the filtering process?
>

That is a good question. I hadn't thought about the details but what I had
implemented was we have to first read the entry, lock the key, reread the
entry (in case of concurrent update) and then finally call their Predicate.
Another reason the filterKeys is much more efficient :) Note this read,
locking and reread is done even without a Predicate supplied. And actually
I can also optimize filterKeys to not do the initial read since we already
have the key.


>
> That would require the move the transaction boundary to the scenario
> A# which seems undesirable.
> Technically if I were to need something like this I guess I'd expect
> to have a user experience akin to B# but have Infinispan essentially
> use optimistic locking (and auto-skip) on entries which are mutated
> and fall out of the filter predicate during the lock attempt.
>
> Essentially I suspect that we'd not want to implement different
> versions of this depending on the transaction mode, but figure out the
> use case and implement a one and only transaction mode which suites
> such use cases. So for example we'd simply not offer a mode which
> requires to copy the whole grid into the current TX context.
>

This was never the intent and in my follow up emails I came to what seems
like the same conclusion that basically this can't be done with the user
controlling the transaction and it doesn't really make sense in an
optimistic transaction (since you are already at that node, you are already
doing everything pessimistically).


>
> Thanks,
> Sanne
>
>
>
> On 28 March 2017 at 14:49, William Burns <mudokonman at gmail.com> wrote:
> >
> >
> > On Mon, Mar 27, 2017 at 9:02 PM Galder Zamarre?o <galder at redhat.com>
> wrote:
> >>
> >>
> >> --
> >> Galder Zamarre?o
> >> Infinispan, Red Hat
> >>
> >> > On 21 Mar 2017, at 17:16, Dan Berindei <dan.berindei at gmail.com>
> wrote:
> >> >
> >> > I'm leaning towards option 1.
> >> >
> >> > Are you thinking about also allowing the consumer to modify the entry,
> >> > like JCache's EntryProcessors? For a consumer that can only modify the
> >> > current entry, we could even "emulate" locking in an optimistic cache
> >> > by catching the WriteSkewException and running the consumer again.
> >> >
> >> > I wouldn't allow this to be mixed with other operations in a stream,
> >> > because then you may have to run filters/mappers/sorting while holding
> >> > the lock as well.
> >>
> >> ^ Would forEach w/ lock still run for all entries in originator? If so,
> >> not being able to filter could be a pain. IOW, you'd be forcing all
> entries
> >> to be shipped to a node and user to do its own filtering. Not ideal :\
> >
> >
> > No the primary owner would run the operation per entry. I was thinking we
> > would have 2 levels of filtering in my proposal above.
> >
> > We would have the first one which is using filterKeys on the CacheStream
> > method. This requires serializing keys sent to each node (only primary
> owned
> > keys are sent). While it has the cost of serialization it makes up for by
> > having constant time lookups (no iterating memory/stores) for the keys
> as it
> > creates a stream using Cache.get to populate it.
> >
> > The second was to support the filter method on the Stream API which would
> > allow for a Predicate so you don't have to serialize keys. In this case
> you
> > wouldn't want to include keys in this Predicate as all keys would be
> > serialized to all nodes and then you still have to iterate and check the
> > entire data container/store.
> >
> > You could actually do both as well. So if you only want a subset of known
> > keys where their values match a Predicate this can be done too.
> >
> > cache.lockedStream().filterKeys(keys).filter(predicate).forEach();
> >
> >>
> >>
> >>
> >> >
> >> > Cheers
> >> > Dan
> >> >
> >> >
> >> > On Tue, Mar 21, 2017 at 5:37 PM, William Burns <mudokonman at gmail.com>
> >> > wrote:
> >> >> Some users have expressed the need to have some sort of forEach
> >> >> operation
> >> >> that is performed where the Consumer is called while holding the lock
> >> >> for
> >> >> the given key and subsequently released after the Consumer operation
> >> >> completes.
> >> >>
> >> >> Due to the nature of how streams work with retries and performing the
> >> >> operation on the primary owner, this works out quite well with
> forEach
> >> >> to be
> >> >> done in an efficient way.
> >> >>
> >> >> The problem is that this only really works well with non tx and
> >> >> pessimistic
> >> >> tx. This obviously leaves out optimistic tx, which at first I was a
> >> >> little
> >> >> worried about. But after thinking about it more, this prelocking and
> >> >> optimistic tx don't really fit that well together anyways. So I am
> >> >> thinking
> >> >> whenever this operation is performed it would throw an exception not
> >> >> letting
> >> >> the user use this feature in optimistic transactions.
> >> >>
> >> >> Another question is what does the API for this look like. I was
> >> >> debating
> >> >> between 3 options myself:
> >> >>
> >> >> 1. AdvancedCache.forEachWithLock(BiConsumer<Cache, CacheEntry<K, V>>
> >> >> consumer)
> >> >>
> >> >> This require the least amount of changes, however the user can't
> >> >> customize
> >> >> certain parameters that CacheStream currently provides (listed below
> -
> >> >> big
> >> >> one being filterKeys).
> >> >>
> >> >> 2. CacheStream.forEachWithLock(BiConsumer<Cache, CacheEntry<K, V>>
> >> >> consumer)
> >> >>
> >> >> This method would only be allowed to be invoked on the Stream if no
> >> >> other
> >> >> intermediate operations were invoked, otherwise an exception would be
> >> >> thrown. This still gives us access to all of the CacheStream methods
> >> >> that
> >> >> aren't on the Stream interface (ie. sequentialDistribution,
> >> >> parallelDistribution, parallel, sequential, filterKeys,
> >> >> filterKeySegments,
> >> >> distributedBatchSize, disableRehashAware, timeout).
> >> >>
> >> >> 3. LockedStream<CacheEntry<K, V>> AdvancedCache.lockedStream()
> >> >>
> >> >> This requires the most changes, however the API would be the most
> >> >> explicit.
> >> >> In this case the LockedStream would only have the methods on it that
> >> >> are
> >> >> able to be invoked as noted above and forEach.
> >> >>
> >> >> I personally feel that #3 might be the cleanest, but obviously
> requires
> >> >> adding more classes. Let me know what you guys think and if you think
> >> >> the
> >> >> optimistic exclusion is acceptable.
> >> >>
> >> >> Thanks,
> >> >>
> >> >> - Will
> >> >>
> >> >> _______________________________________________
> >> >> infinispan-dev mailing list
> >> >> infinispan-dev at lists.jboss.org
> >> >> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >> > _______________________________________________
> >> > infinispan-dev mailing list
> >> > infinispan-dev at lists.jboss.org
> >> > https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >>
> >>
> >> _______________________________________________
> >> infinispan-dev mailing list
> >> infinispan-dev at lists.jboss.org
> >> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >
> >
> > _______________________________________________
> > infinispan-dev mailing list
> > infinispan-dev at lists.jboss.org
> > https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/infinispan-dev/attachments/20170328/91706d60/attachment-0001.html 

From dan.berindei at gmail.com  Tue Mar 28 12:44:40 2017
From: dan.berindei at gmail.com (Dan Berindei)
Date: Tue, 28 Mar 2017 19:44:40 +0300
Subject: [infinispan-dev] Stream operations under lock
In-Reply-To: <CA+YCuUM87dvVkV0gGnnKVzpAPHyQd8fJkH_3iRF0ccyvUz9WPA@mail.gmail.com>
References: <CA+YCuUM3h0H+ZDe1ZvK5mOuXnLB9bjjP+CkJg_M8eyGKC+jRHA@mail.gmail.com>
	<CA+nfvwTaTFf_Ba4FkLAhejjQwC=p9YeiDuTrUaGj1krr2xgobw@mail.gmail.com>
	<28A6D5BF-6878-4778-A48A-489D91A2BBCB@redhat.com>
	<CA+YCuUMm6XUWh+5wQcYaYfqqA6UKqA9KSwb6TNdZpFJeg_G+cA@mail.gmail.com>
	<CAFm4XO39HKQF==fo7H31BgZhwSnw6qp2_-UM9LQVX65KOcoz+w@mail.gmail.com>
	<CA+YCuUM87dvVkV0gGnnKVzpAPHyQd8fJkH_3iRF0ccyvUz9WPA@mail.gmail.com>
Message-ID: <CA+nfvwSvHGusN7-YL+ZWBig7id5GDxFu1ThN3CD_z=LGn6x9Zw@mail.gmail.com>

On Tue, Mar 28, 2017 at 6:44 PM, William Burns <mudokonman at gmail.com> wrote:
>
>
> On Tue, Mar 28, 2017 at 11:24 AM Sanne Grinovero <sanne at infinispan.org>
> wrote:
>>
>> Hi Will,
>>
>> I'm confused about the premise; when you state
>>
>> " the Consumer is called while holding the lock for the given key and
>> subsequently released after the Consumer operation completes."
>>
>> What are the transaction boundaries?
>> I see two options, please correct me so I understand:
>>
>> A)
>>
>> transaction.begin();
>> cache.lockedStream().filterKeys(keys).filter(predicate).forEach(
>> someOperation );
>> transaction.commit(); <-- release all locks
>>
>> B)
>>
>> cache.lockedStream().filterKeys(keys).filter(predicate).forEach(
>>     transaction.begin();
>>     //to stuff on entry
>>     /transaction.commit(); <-- release this single entry's lock
>> );
>
>
> The user code doesn't really affect this, it is done internally by
> Infinispan. We would acquire the lock on the user's behalf in the stream
> operation then call the user's accept method. Then after they return we
> would unlock the lock. The way I have it implemented at the moment in my PoC
> (which could change) is I don't even start a transaction. It just locks the
> key and then if the user were to invoke an operation that requires a
> transaction it adds the lock owner to their tx context at that point so they
> have ownership of the key.
>
> Also to note that a commit or rollback in the Consumer currently doesn't
> release the lock on the key. Although this could be discussed to be possibly
> changed.
>

With a transactional cache I was assuming you manage the transaction
yourself... if the user has to call
transactionManager.begin()/commit()/rollback() anyway, why not use a
regular stream?

>>
>>
>>
>> I think it's important to clarify this as I suspect that #A is not
>> implementable within reasonable guarantees, while in the case of B# I
>> see no use for optimistic locking *from a user's perspective*.
>
>
> Exactly my thoughts regarding optimistic. I don't think #A is even feasible
> given constraints of having a distributed transaction like this.
>

Totally agree that #A can't work with any kind of transaction configuration.

As to optimistic locking, I would like having "feature parity" between
pessimistic and optimistic caches as much as possible, but I agree
locking eagerly and retrying the consumer on WriteSkewException are a
bit too different to fit under the same API.

>>
>>
>> Also: what kind of guarantees do you expect about having the operation
>> being applied on some entry which is strictly *still* a subset of the
>> keys as defined by the filter predicates?
>> I take it you'd want to acquire the locks during the filtering process?
>
>
> That is a good question. I hadn't thought about the details but what I had
> implemented was we have to first read the entry, lock the key, reread the
> entry (in case of concurrent update) and then finally call their Predicate.
> Another reason the filterKeys is much more efficient :) Note this read,
> locking and reread is done even without a Predicate supplied. And actually I
> can also optimize filterKeys to not do the initial read since we already
> have the key.
>

Would this be more efficient than the consumer reading the key with
FORCE_WRITE_LOCK and deciding what to do based on the value?

>>
>>
>> That would require the move the transaction boundary to the scenario
>> A# which seems undesirable.
>> Technically if I were to need something like this I guess I'd expect
>> to have a user experience akin to B# but have Infinispan essentially
>> use optimistic locking (and auto-skip) on entries which are mutated
>> and fall out of the filter predicate during the lock attempt.
>>
>> Essentially I suspect that we'd not want to implement different
>> versions of this depending on the transaction mode, but figure out the
>> use case and implement a one and only transaction mode which suites
>> such use cases. So for example we'd simply not offer a mode which
>> requires to copy the whole grid into the current TX context.
>
>
> This was never the intent and in my follow up emails I came to what seems
> like the same conclusion that basically this can't be done with the user
> controlling the transaction and it doesn't really make sense in an
> optimistic transaction (since you are already at that node, you are already
> doing everything pessimistically).
>

Even local caches can use optimistic locking :)

>>
>>
>> Thanks,
>> Sanne
>>
>>
>>
>> On 28 March 2017 at 14:49, William Burns <mudokonman at gmail.com> wrote:
>> >
>> >
>> > On Mon, Mar 27, 2017 at 9:02 PM Galder Zamarre?o <galder at redhat.com>
>> > wrote:
>> >>
>> >>
>> >> --
>> >> Galder Zamarre?o
>> >> Infinispan, Red Hat
>> >>
>> >> > On 21 Mar 2017, at 17:16, Dan Berindei <dan.berindei at gmail.com>
>> >> > wrote:
>> >> >
>> >> > I'm leaning towards option 1.
>> >> >
>> >> > Are you thinking about also allowing the consumer to modify the
>> >> > entry,
>> >> > like JCache's EntryProcessors? For a consumer that can only modify
>> >> > the
>> >> > current entry, we could even "emulate" locking in an optimistic cache
>> >> > by catching the WriteSkewException and running the consumer again.
>> >> >
>> >> > I wouldn't allow this to be mixed with other operations in a stream,
>> >> > because then you may have to run filters/mappers/sorting while
>> >> > holding
>> >> > the lock as well.
>> >>
>> >> ^ Would forEach w/ lock still run for all entries in originator? If so,
>> >> not being able to filter could be a pain. IOW, you'd be forcing all
>> >> entries
>> >> to be shipped to a node and user to do its own filtering. Not ideal :\
>> >
>> >
>> > No the primary owner would run the operation per entry. I was thinking
>> > we
>> > would have 2 levels of filtering in my proposal above.
>> >
>> > We would have the first one which is using filterKeys on the CacheStream
>> > method. This requires serializing keys sent to each node (only primary
>> > owned
>> > keys are sent). While it has the cost of serialization it makes up for
>> > by
>> > having constant time lookups (no iterating memory/stores) for the keys
>> > as it
>> > creates a stream using Cache.get to populate it.
>> >
>> > The second was to support the filter method on the Stream API which
>> > would
>> > allow for a Predicate so you don't have to serialize keys. In this case
>> > you
>> > wouldn't want to include keys in this Predicate as all keys would be
>> > serialized to all nodes and then you still have to iterate and check the
>> > entire data container/store.
>> >
>> > You could actually do both as well. So if you only want a subset of
>> > known
>> > keys where their values match a Predicate this can be done too.
>> >
>> > cache.lockedStream().filterKeys(keys).filter(predicate).forEach();
>> >
>> >>
>> >>
>> >>
>> >> >
>> >> > Cheers
>> >> > Dan
>> >> >
>> >> >
>> >> > On Tue, Mar 21, 2017 at 5:37 PM, William Burns <mudokonman at gmail.com>
>> >> > wrote:
>> >> >> Some users have expressed the need to have some sort of forEach
>> >> >> operation
>> >> >> that is performed where the Consumer is called while holding the
>> >> >> lock
>> >> >> for
>> >> >> the given key and subsequently released after the Consumer operation
>> >> >> completes.
>> >> >>
>> >> >> Due to the nature of how streams work with retries and performing
>> >> >> the
>> >> >> operation on the primary owner, this works out quite well with
>> >> >> forEach
>> >> >> to be
>> >> >> done in an efficient way.
>> >> >>
>> >> >> The problem is that this only really works well with non tx and
>> >> >> pessimistic
>> >> >> tx. This obviously leaves out optimistic tx, which at first I was a
>> >> >> little
>> >> >> worried about. But after thinking about it more, this prelocking and
>> >> >> optimistic tx don't really fit that well together anyways. So I am
>> >> >> thinking
>> >> >> whenever this operation is performed it would throw an exception not
>> >> >> letting
>> >> >> the user use this feature in optimistic transactions.
>> >> >>
>> >> >> Another question is what does the API for this look like. I was
>> >> >> debating
>> >> >> between 3 options myself:
>> >> >>
>> >> >> 1. AdvancedCache.forEachWithLock(BiConsumer<Cache, CacheEntry<K, V>>
>> >> >> consumer)
>> >> >>
>> >> >> This require the least amount of changes, however the user can't
>> >> >> customize
>> >> >> certain parameters that CacheStream currently provides (listed below
>> >> >> -
>> >> >> big
>> >> >> one being filterKeys).
>> >> >>
>> >> >> 2. CacheStream.forEachWithLock(BiConsumer<Cache, CacheEntry<K, V>>
>> >> >> consumer)
>> >> >>
>> >> >> This method would only be allowed to be invoked on the Stream if no
>> >> >> other
>> >> >> intermediate operations were invoked, otherwise an exception would
>> >> >> be
>> >> >> thrown. This still gives us access to all of the CacheStream methods
>> >> >> that
>> >> >> aren't on the Stream interface (ie. sequentialDistribution,
>> >> >> parallelDistribution, parallel, sequential, filterKeys,
>> >> >> filterKeySegments,
>> >> >> distributedBatchSize, disableRehashAware, timeout).
>> >> >>
>> >> >> 3. LockedStream<CacheEntry<K, V>> AdvancedCache.lockedStream()
>> >> >>
>> >> >> This requires the most changes, however the API would be the most
>> >> >> explicit.
>> >> >> In this case the LockedStream would only have the methods on it that
>> >> >> are
>> >> >> able to be invoked as noted above and forEach.
>> >> >>
>> >> >> I personally feel that #3 might be the cleanest, but obviously
>> >> >> requires
>> >> >> adding more classes. Let me know what you guys think and if you
>> >> >> think
>> >> >> the
>> >> >> optimistic exclusion is acceptable.
>> >> >>
>> >> >> Thanks,
>> >> >>
>> >> >> - Will
>> >> >>
>> >> >> _______________________________________________
>> >> >> infinispan-dev mailing list
>> >> >> infinispan-dev at lists.jboss.org
>> >> >> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>> >> > _______________________________________________
>> >> > infinispan-dev mailing list
>> >> > infinispan-dev at lists.jboss.org
>> >> > https://lists.jboss.org/mailman/listinfo/infinispan-dev
>> >>
>> >>
>> >> _______________________________________________
>> >> infinispan-dev mailing list
>> >> infinispan-dev at lists.jboss.org
>> >> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>> >
>> >
>> > _______________________________________________
>> > infinispan-dev mailing list
>> > infinispan-dev at lists.jboss.org
>> > https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>
>> _______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev


From mudokonman at gmail.com  Tue Mar 28 13:54:17 2017
From: mudokonman at gmail.com (William Burns)
Date: Tue, 28 Mar 2017 17:54:17 +0000
Subject: [infinispan-dev] Stream operations under lock
In-Reply-To: <CA+nfvwSvHGusN7-YL+ZWBig7id5GDxFu1ThN3CD_z=LGn6x9Zw@mail.gmail.com>
References: <CA+YCuUM3h0H+ZDe1ZvK5mOuXnLB9bjjP+CkJg_M8eyGKC+jRHA@mail.gmail.com>
	<CA+nfvwTaTFf_Ba4FkLAhejjQwC=p9YeiDuTrUaGj1krr2xgobw@mail.gmail.com>
	<28A6D5BF-6878-4778-A48A-489D91A2BBCB@redhat.com>
	<CA+YCuUMm6XUWh+5wQcYaYfqqA6UKqA9KSwb6TNdZpFJeg_G+cA@mail.gmail.com>
	<CAFm4XO39HKQF==fo7H31BgZhwSnw6qp2_-UM9LQVX65KOcoz+w@mail.gmail.com>
	<CA+YCuUM87dvVkV0gGnnKVzpAPHyQd8fJkH_3iRF0ccyvUz9WPA@mail.gmail.com>
	<CA+nfvwSvHGusN7-YL+ZWBig7id5GDxFu1ThN3CD_z=LGn6x9Zw@mail.gmail.com>
Message-ID: <CA+YCuUPQr7YB9bNuU308RPEhf9E4GF57+mF5QWS6sDJpaHDKnQ@mail.gmail.com>

On Tue, Mar 28, 2017 at 12:52 PM Dan Berindei <dan.berindei at gmail.com>
wrote:

> On Tue, Mar 28, 2017 at 6:44 PM, William Burns <mudokonman at gmail.com>
> wrote:
> >
> >
> > On Tue, Mar 28, 2017 at 11:24 AM Sanne Grinovero <sanne at infinispan.org>
> > wrote:
> >>
> >> Hi Will,
> >>
> >> I'm confused about the premise; when you state
> >>
> >> " the Consumer is called while holding the lock for the given key and
> >> subsequently released after the Consumer operation completes."
> >>
> >> What are the transaction boundaries?
> >> I see two options, please correct me so I understand:
> >>
> >> A)
> >>
> >> transaction.begin();
> >> cache.lockedStream().filterKeys(keys).filter(predicate).forEach(
> >> someOperation );
> >> transaction.commit(); <-- release all locks
> >>
> >> B)
> >>
> >> cache.lockedStream().filterKeys(keys).filter(predicate).forEach(
> >>     transaction.begin();
> >>     //to stuff on entry
> >>     /transaction.commit(); <-- release this single entry's lock
> >> );
> >
> >
> > The user code doesn't really affect this, it is done internally by
> > Infinispan. We would acquire the lock on the user's behalf in the stream
> > operation then call the user's accept method. Then after they return we
> > would unlock the lock. The way I have it implemented at the moment in my
> PoC
> > (which could change) is I don't even start a transaction. It just locks
> the
> > key and then if the user were to invoke an operation that requires a
> > transaction it adds the lock owner to their tx context at that point so
> they
> > have ownership of the key.
> >
> > Also to note that a commit or rollback in the Consumer currently doesn't
> > release the lock on the key. Although this could be discussed to be
> possibly
> > changed.
> >
>
> With a transactional cache I was assuming you manage the transaction
> yourself... if the user has to call
> transactionManager.begin()/commit()/rollback() anyway, why not use a
> regular stream?
>

The user would be using an implicit transaction or explicit. Like I said
this is up for discussion. The main reason I am staying away from managing
the transaction is that the user can mess with the transaction as well
which would possibly release the lock. It is much simpler if all I am doing
is managing the lock. And if the user doesn't require a transaction in
Infinispan we didn't waste time starting one and releasing one.


>
> >>
> >>
> >>
> >> I think it's important to clarify this as I suspect that #A is not
> >> implementable within reasonable guarantees, while in the case of B# I
> >> see no use for optimistic locking *from a user's perspective*.
> >
> >
> > Exactly my thoughts regarding optimistic. I don't think #A is even
> feasible
> > given constraints of having a distributed transaction like this.
> >
>
> Totally agree that #A can't work with any kind of transaction
> configuration.
>
> As to optimistic locking, I would like having "feature parity" between
> pessimistic and optimistic caches as much as possible, but I agree
> locking eagerly and retrying the consumer on WriteSkewException are a
> bit too different to fit under the same API.
>

Yeah I would definitely like to have it as well, but I just can't see how
it fits in. This is despite the implementation detail that it is quite
difficult to get it working currently :D


>
> >>
> >>
> >> Also: what kind of guarantees do you expect about having the operation
> >> being applied on some entry which is strictly *still* a subset of the
> >> keys as defined by the filter predicates?
> >> I take it you'd want to acquire the locks during the filtering process?
> >
> >
> > That is a good question. I hadn't thought about the details but what I
> had
> > implemented was we have to first read the entry, lock the key, reread the
> > entry (in case of concurrent update) and then finally call their
> Predicate.
> > Another reason the filterKeys is much more efficient :) Note this read,
> > locking and reread is done even without a Predicate supplied. And
> actually I
> > can also optimize filterKeys to not do the initial read since we already
> > have the key.
> >
>
> Would this be more efficient than the consumer reading the key with
> FORCE_WRITE_LOCK and deciding what to do based on the value?
>
>
The problem with this is you have to go remote to lock the key, return the
value then do something for every key (not to mention pulling those keys
using an iterator). Very costly! The main benefit of the stream with lock
is that you are performing everything on the primary owner of the data with
the lock already acquired. The only piece sent remotely is the consumer and
some internal classes, very light weight and you have all the benefits of
data locality.


> >>
> >>
> >> That would require the move the transaction boundary to the scenario
> >> A# which seems undesirable.
> >> Technically if I were to need something like this I guess I'd expect
> >> to have a user experience akin to B# but have Infinispan essentially
> >> use optimistic locking (and auto-skip) on entries which are mutated
> >> and fall out of the filter predicate during the lock attempt.
> >>
> >> Essentially I suspect that we'd not want to implement different
> >> versions of this depending on the transaction mode, but figure out the
> >> use case and implement a one and only transaction mode which suites
> >> such use cases. So for example we'd simply not offer a mode which
> >> requires to copy the whole grid into the current TX context.
> >
> >
> > This was never the intent and in my follow up emails I came to what seems
> > like the same conclusion that basically this can't be done with the user
> > controlling the transaction and it doesn't really make sense in an
> > optimistic transaction (since you are already at that node, you are
> already
> > doing everything pessimistically).
> >
>
> Even local caches can use optimistic locking :)
>

Yes I know :) I was just framing it in the notion of remote. If anyone can
think of a nice way of using this with optimistic transactions I would all
be for it. But optimistic transactions just doesn't make any sense to me
when you are locking a key eagerly for someone to do something with it
(definition of pessimistic transaction).


>
> >>
> >>
> >> Thanks,
> >> Sanne
> >>
> >>
> >>
> >> On 28 March 2017 at 14:49, William Burns <mudokonman at gmail.com> wrote:
> >> >
> >> >
> >> > On Mon, Mar 27, 2017 at 9:02 PM Galder Zamarre?o <galder at redhat.com>
> >> > wrote:
> >> >>
> >> >>
> >> >> --
> >> >> Galder Zamarre?o
> >> >> Infinispan, Red Hat
> >> >>
> >> >> > On 21 Mar 2017, at 17:16, Dan Berindei <dan.berindei at gmail.com>
> >> >> > wrote:
> >> >> >
> >> >> > I'm leaning towards option 1.
> >> >> >
> >> >> > Are you thinking about also allowing the consumer to modify the
> >> >> > entry,
> >> >> > like JCache's EntryProcessors? For a consumer that can only modify
> >> >> > the
> >> >> > current entry, we could even "emulate" locking in an optimistic
> cache
> >> >> > by catching the WriteSkewException and running the consumer again.
> >> >> >
> >> >> > I wouldn't allow this to be mixed with other operations in a
> stream,
> >> >> > because then you may have to run filters/mappers/sorting while
> >> >> > holding
> >> >> > the lock as well.
> >> >>
> >> >> ^ Would forEach w/ lock still run for all entries in originator? If
> so,
> >> >> not being able to filter could be a pain. IOW, you'd be forcing all
> >> >> entries
> >> >> to be shipped to a node and user to do its own filtering. Not ideal
> :\
> >> >
> >> >
> >> > No the primary owner would run the operation per entry. I was thinking
> >> > we
> >> > would have 2 levels of filtering in my proposal above.
> >> >
> >> > We would have the first one which is using filterKeys on the
> CacheStream
> >> > method. This requires serializing keys sent to each node (only primary
> >> > owned
> >> > keys are sent). While it has the cost of serialization it makes up for
> >> > by
> >> > having constant time lookups (no iterating memory/stores) for the keys
> >> > as it
> >> > creates a stream using Cache.get to populate it.
> >> >
> >> > The second was to support the filter method on the Stream API which
> >> > would
> >> > allow for a Predicate so you don't have to serialize keys. In this
> case
> >> > you
> >> > wouldn't want to include keys in this Predicate as all keys would be
> >> > serialized to all nodes and then you still have to iterate and check
> the
> >> > entire data container/store.
> >> >
> >> > You could actually do both as well. So if you only want a subset of
> >> > known
> >> > keys where their values match a Predicate this can be done too.
> >> >
> >> > cache.lockedStream().filterKeys(keys).filter(predicate).forEach();
> >> >
> >> >>
> >> >>
> >> >>
> >> >> >
> >> >> > Cheers
> >> >> > Dan
> >> >> >
> >> >> >
> >> >> > On Tue, Mar 21, 2017 at 5:37 PM, William Burns <
> mudokonman at gmail.com>
> >> >> > wrote:
> >> >> >> Some users have expressed the need to have some sort of forEach
> >> >> >> operation
> >> >> >> that is performed where the Consumer is called while holding the
> >> >> >> lock
> >> >> >> for
> >> >> >> the given key and subsequently released after the Consumer
> operation
> >> >> >> completes.
> >> >> >>
> >> >> >> Due to the nature of how streams work with retries and performing
> >> >> >> the
> >> >> >> operation on the primary owner, this works out quite well with
> >> >> >> forEach
> >> >> >> to be
> >> >> >> done in an efficient way.
> >> >> >>
> >> >> >> The problem is that this only really works well with non tx and
> >> >> >> pessimistic
> >> >> >> tx. This obviously leaves out optimistic tx, which at first I was
> a
> >> >> >> little
> >> >> >> worried about. But after thinking about it more, this prelocking
> and
> >> >> >> optimistic tx don't really fit that well together anyways. So I am
> >> >> >> thinking
> >> >> >> whenever this operation is performed it would throw an exception
> not
> >> >> >> letting
> >> >> >> the user use this feature in optimistic transactions.
> >> >> >>
> >> >> >> Another question is what does the API for this look like. I was
> >> >> >> debating
> >> >> >> between 3 options myself:
> >> >> >>
> >> >> >> 1. AdvancedCache.forEachWithLock(BiConsumer<Cache, CacheEntry<K,
> V>>
> >> >> >> consumer)
> >> >> >>
> >> >> >> This require the least amount of changes, however the user can't
> >> >> >> customize
> >> >> >> certain parameters that CacheStream currently provides (listed
> below
> >> >> >> -
> >> >> >> big
> >> >> >> one being filterKeys).
> >> >> >>
> >> >> >> 2. CacheStream.forEachWithLock(BiConsumer<Cache, CacheEntry<K, V>>
> >> >> >> consumer)
> >> >> >>
> >> >> >> This method would only be allowed to be invoked on the Stream if
> no
> >> >> >> other
> >> >> >> intermediate operations were invoked, otherwise an exception would
> >> >> >> be
> >> >> >> thrown. This still gives us access to all of the CacheStream
> methods
> >> >> >> that
> >> >> >> aren't on the Stream interface (ie. sequentialDistribution,
> >> >> >> parallelDistribution, parallel, sequential, filterKeys,
> >> >> >> filterKeySegments,
> >> >> >> distributedBatchSize, disableRehashAware, timeout).
> >> >> >>
> >> >> >> 3. LockedStream<CacheEntry<K, V>> AdvancedCache.lockedStream()
> >> >> >>
> >> >> >> This requires the most changes, however the API would be the most
> >> >> >> explicit.
> >> >> >> In this case the LockedStream would only have the methods on it
> that
> >> >> >> are
> >> >> >> able to be invoked as noted above and forEach.
> >> >> >>
> >> >> >> I personally feel that #3 might be the cleanest, but obviously
> >> >> >> requires
> >> >> >> adding more classes. Let me know what you guys think and if you
> >> >> >> think
> >> >> >> the
> >> >> >> optimistic exclusion is acceptable.
> >> >> >>
> >> >> >> Thanks,
> >> >> >>
> >> >> >> - Will
> >> >> >>
> >> >> >> _______________________________________________
> >> >> >> infinispan-dev mailing list
> >> >> >> infinispan-dev at lists.jboss.org
> >> >> >> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >> >> > _______________________________________________
> >> >> > infinispan-dev mailing list
> >> >> > infinispan-dev at lists.jboss.org
> >> >> > https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >> >>
> >> >>
> >> >> _______________________________________________
> >> >> infinispan-dev mailing list
> >> >> infinispan-dev at lists.jboss.org
> >> >> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >> >
> >> >
> >> > _______________________________________________
> >> > infinispan-dev mailing list
> >> > infinispan-dev at lists.jboss.org
> >> > https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >>
> >> _______________________________________________
> >> infinispan-dev mailing list
> >> infinispan-dev at lists.jboss.org
> >> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >
> >
> > _______________________________________________
> > infinispan-dev mailing list
> > infinispan-dev at lists.jboss.org
> > https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/infinispan-dev/attachments/20170328/9e56c695/attachment-0001.html 

From dan.berindei at gmail.com  Wed Mar 29 04:04:50 2017
From: dan.berindei at gmail.com (Dan Berindei)
Date: Wed, 29 Mar 2017 11:04:50 +0300
Subject: [infinispan-dev] Stream operations under lock
In-Reply-To: <CA+YCuUPQr7YB9bNuU308RPEhf9E4GF57+mF5QWS6sDJpaHDKnQ@mail.gmail.com>
References: <CA+YCuUM3h0H+ZDe1ZvK5mOuXnLB9bjjP+CkJg_M8eyGKC+jRHA@mail.gmail.com>
	<CA+nfvwTaTFf_Ba4FkLAhejjQwC=p9YeiDuTrUaGj1krr2xgobw@mail.gmail.com>
	<28A6D5BF-6878-4778-A48A-489D91A2BBCB@redhat.com>
	<CA+YCuUMm6XUWh+5wQcYaYfqqA6UKqA9KSwb6TNdZpFJeg_G+cA@mail.gmail.com>
	<CAFm4XO39HKQF==fo7H31BgZhwSnw6qp2_-UM9LQVX65KOcoz+w@mail.gmail.com>
	<CA+YCuUM87dvVkV0gGnnKVzpAPHyQd8fJkH_3iRF0ccyvUz9WPA@mail.gmail.com>
	<CA+nfvwSvHGusN7-YL+ZWBig7id5GDxFu1ThN3CD_z=LGn6x9Zw@mail.gmail.com>
	<CA+YCuUPQr7YB9bNuU308RPEhf9E4GF57+mF5QWS6sDJpaHDKnQ@mail.gmail.com>
Message-ID: <CA+nfvwTFriN-FCBrxgrowttKrWd+fEYhrq1_QWbB8AHe869rFQ@mail.gmail.com>

On Tue, Mar 28, 2017 at 8:54 PM, William Burns <mudokonman at gmail.com> wrote:
>
>
> On Tue, Mar 28, 2017 at 12:52 PM Dan Berindei <dan.berindei at gmail.com>
> wrote:
>>
>> On Tue, Mar 28, 2017 at 6:44 PM, William Burns <mudokonman at gmail.com>
>> wrote:
>> >
>> >
>> > On Tue, Mar 28, 2017 at 11:24 AM Sanne Grinovero <sanne at infinispan.org>
>> > wrote:
>> >>
>> >> Hi Will,
>> >>
>> >> I'm confused about the premise; when you state
>> >>
>> >> " the Consumer is called while holding the lock for the given key and
>> >> subsequently released after the Consumer operation completes."
>> >>
>> >> What are the transaction boundaries?
>> >> I see two options, please correct me so I understand:
>> >>
>> >> A)
>> >>
>> >> transaction.begin();
>> >> cache.lockedStream().filterKeys(keys).filter(predicate).forEach(
>> >> someOperation );
>> >> transaction.commit(); <-- release all locks
>> >>
>> >> B)
>> >>
>> >> cache.lockedStream().filterKeys(keys).filter(predicate).forEach(
>> >>     transaction.begin();
>> >>     //to stuff on entry
>> >>     /transaction.commit(); <-- release this single entry's lock
>> >> );
>> >
>> >
>> > The user code doesn't really affect this, it is done internally by
>> > Infinispan. We would acquire the lock on the user's behalf in the stream
>> > operation then call the user's accept method. Then after they return we
>> > would unlock the lock. The way I have it implemented at the moment in my
>> > PoC
>> > (which could change) is I don't even start a transaction. It just locks
>> > the
>> > key and then if the user were to invoke an operation that requires a
>> > transaction it adds the lock owner to their tx context at that point so
>> > they
>> > have ownership of the key.
>> >
>> > Also to note that a commit or rollback in the Consumer currently doesn't
>> > release the lock on the key. Although this could be discussed to be
>> > possibly
>> > changed.
>> >
>>
>> With a transactional cache I was assuming you manage the transaction
>> yourself... if the user has to call
>> transactionManager.begin()/commit()/rollback() anyway, why not use a
>> regular stream?
>
>
> The user would be using an implicit transaction or explicit. Like I said
> this is up for discussion. The main reason I am staying away from managing
> the transaction is that the user can mess with the transaction as well which
> would possibly release the lock. It is much simpler if all I am doing is
> managing the lock. And if the user doesn't require a transaction in
> Infinispan we didn't waste time starting one and releasing one.
>

Every write to a transactional cache will start an implicit
transaction, so I don't think we really have the option of not
starting a transaction.

>>
>>
>> >>
>> >>
>> >>
>> >> I think it's important to clarify this as I suspect that #A is not
>> >> implementable within reasonable guarantees, while in the case of B# I
>> >> see no use for optimistic locking *from a user's perspective*.
>> >
>> >
>> > Exactly my thoughts regarding optimistic. I don't think #A is even
>> > feasible
>> > given constraints of having a distributed transaction like this.
>> >
>>
>> Totally agree that #A can't work with any kind of transaction
>> configuration.
>>
>> As to optimistic locking, I would like having "feature parity" between
>> pessimistic and optimistic caches as much as possible, but I agree
>> locking eagerly and retrying the consumer on WriteSkewException are a
>> bit too different to fit under the same API.
>
>
> Yeah I would definitely like to have it as well, but I just can't see how it
> fits in. This is despite the implementation detail that it is quite
> difficult to get it working currently :D
>
>>
>>
>> >>
>> >>
>> >> Also: what kind of guarantees do you expect about having the operation
>> >> being applied on some entry which is strictly *still* a subset of the
>> >> keys as defined by the filter predicates?
>> >> I take it you'd want to acquire the locks during the filtering process?
>> >
>> >
>> > That is a good question. I hadn't thought about the details but what I
>> > had
>> > implemented was we have to first read the entry, lock the key, reread
>> > the
>> > entry (in case of concurrent update) and then finally call their
>> > Predicate.
>> > Another reason the filterKeys is much more efficient :) Note this read,
>> > locking and reread is done even without a Predicate supplied. And
>> > actually I
>> > can also optimize filterKeys to not do the initial read since we already
>> > have the key.
>> >
>>
>> Would this be more efficient than the consumer reading the key with
>> FORCE_WRITE_LOCK and deciding what to do based on the value?
>>
>
> The problem with this is you have to go remote to lock the key, return the
> value then do something for every key (not to mention pulling those keys
> using an iterator). Very costly! The main benefit of the stream with lock is
> that you are performing everything on the primary owner of the data with the
> lock already acquired. The only piece sent remotely is the consumer and some
> internal classes, very light weight and you have all the benefits of data
> locality.
>

I meant using cache.withFlags(FORCE_WRITE_LOCK).get(key) in the
Consumer itself, and assuming it runs on the primary owner.

>>
>> >>
>> >>
>> >> That would require the move the transaction boundary to the scenario
>> >> A# which seems undesirable.
>> >> Technically if I were to need something like this I guess I'd expect
>> >> to have a user experience akin to B# but have Infinispan essentially
>> >> use optimistic locking (and auto-skip) on entries which are mutated
>> >> and fall out of the filter predicate during the lock attempt.
>> >>
>> >> Essentially I suspect that we'd not want to implement different
>> >> versions of this depending on the transaction mode, but figure out the
>> >> use case and implement a one and only transaction mode which suites
>> >> such use cases. So for example we'd simply not offer a mode which
>> >> requires to copy the whole grid into the current TX context.
>> >
>> >
>> > This was never the intent and in my follow up emails I came to what
>> > seems
>> > like the same conclusion that basically this can't be done with the user
>> > controlling the transaction and it doesn't really make sense in an
>> > optimistic transaction (since you are already at that node, you are
>> > already
>> > doing everything pessimistically).
>> >
>>
>> Even local caches can use optimistic locking :)
>
>
> Yes I know :) I was just framing it in the notion of remote. If anyone can
> think of a nice way of using this with optimistic transactions I would all
> be for it. But optimistic transactions just doesn't make any sense to me
> when you are locking a key eagerly for someone to do something with it
> (definition of pessimistic transaction).
>

Maybe I didn't explain properly...

I meant in optimistic caches we wouldn't acquire the lock at all. The
stream would just start an optimistic transaction, run the consumer,
and try to commit. If the prepare fails because of a
WriteSkewException, start from the beginning.

>>
>>
>> >>
>> >>
>> >> Thanks,
>> >> Sanne
>> >>
>> >>
>> >>
>> >> On 28 March 2017 at 14:49, William Burns <mudokonman at gmail.com> wrote:
>> >> >
>> >> >
>> >> > On Mon, Mar 27, 2017 at 9:02 PM Galder Zamarre?o <galder at redhat.com>
>> >> > wrote:
>> >> >>
>> >> >>
>> >> >> --
>> >> >> Galder Zamarre?o
>> >> >> Infinispan, Red Hat
>> >> >>
>> >> >> > On 21 Mar 2017, at 17:16, Dan Berindei <dan.berindei at gmail.com>
>> >> >> > wrote:
>> >> >> >
>> >> >> > I'm leaning towards option 1.
>> >> >> >
>> >> >> > Are you thinking about also allowing the consumer to modify the
>> >> >> > entry,
>> >> >> > like JCache's EntryProcessors? For a consumer that can only modify
>> >> >> > the
>> >> >> > current entry, we could even "emulate" locking in an optimistic
>> >> >> > cache
>> >> >> > by catching the WriteSkewException and running the consumer again.
>> >> >> >
>> >> >> > I wouldn't allow this to be mixed with other operations in a
>> >> >> > stream,
>> >> >> > because then you may have to run filters/mappers/sorting while
>> >> >> > holding
>> >> >> > the lock as well.
>> >> >>
>> >> >> ^ Would forEach w/ lock still run for all entries in originator? If
>> >> >> so,
>> >> >> not being able to filter could be a pain. IOW, you'd be forcing all
>> >> >> entries
>> >> >> to be shipped to a node and user to do its own filtering. Not ideal
>> >> >> :\
>> >> >
>> >> >
>> >> > No the primary owner would run the operation per entry. I was
>> >> > thinking
>> >> > we
>> >> > would have 2 levels of filtering in my proposal above.
>> >> >
>> >> > We would have the first one which is using filterKeys on the
>> >> > CacheStream
>> >> > method. This requires serializing keys sent to each node (only
>> >> > primary
>> >> > owned
>> >> > keys are sent). While it has the cost of serialization it makes up
>> >> > for
>> >> > by
>> >> > having constant time lookups (no iterating memory/stores) for the
>> >> > keys
>> >> > as it
>> >> > creates a stream using Cache.get to populate it.
>> >> >
>> >> > The second was to support the filter method on the Stream API which
>> >> > would
>> >> > allow for a Predicate so you don't have to serialize keys. In this
>> >> > case
>> >> > you
>> >> > wouldn't want to include keys in this Predicate as all keys would be
>> >> > serialized to all nodes and then you still have to iterate and check
>> >> > the
>> >> > entire data container/store.
>> >> >
>> >> > You could actually do both as well. So if you only want a subset of
>> >> > known
>> >> > keys where their values match a Predicate this can be done too.
>> >> >
>> >> > cache.lockedStream().filterKeys(keys).filter(predicate).forEach();
>> >> >
>> >> >>
>> >> >>
>> >> >>
>> >> >> >
>> >> >> > Cheers
>> >> >> > Dan
>> >> >> >
>> >> >> >
>> >> >> > On Tue, Mar 21, 2017 at 5:37 PM, William Burns
>> >> >> > <mudokonman at gmail.com>
>> >> >> > wrote:
>> >> >> >> Some users have expressed the need to have some sort of forEach
>> >> >> >> operation
>> >> >> >> that is performed where the Consumer is called while holding the
>> >> >> >> lock
>> >> >> >> for
>> >> >> >> the given key and subsequently released after the Consumer
>> >> >> >> operation
>> >> >> >> completes.
>> >> >> >>
>> >> >> >> Due to the nature of how streams work with retries and performing
>> >> >> >> the
>> >> >> >> operation on the primary owner, this works out quite well with
>> >> >> >> forEach
>> >> >> >> to be
>> >> >> >> done in an efficient way.
>> >> >> >>
>> >> >> >> The problem is that this only really works well with non tx and
>> >> >> >> pessimistic
>> >> >> >> tx. This obviously leaves out optimistic tx, which at first I was
>> >> >> >> a
>> >> >> >> little
>> >> >> >> worried about. But after thinking about it more, this prelocking
>> >> >> >> and
>> >> >> >> optimistic tx don't really fit that well together anyways. So I
>> >> >> >> am
>> >> >> >> thinking
>> >> >> >> whenever this operation is performed it would throw an exception
>> >> >> >> not
>> >> >> >> letting
>> >> >> >> the user use this feature in optimistic transactions.
>> >> >> >>
>> >> >> >> Another question is what does the API for this look like. I was
>> >> >> >> debating
>> >> >> >> between 3 options myself:
>> >> >> >>
>> >> >> >> 1. AdvancedCache.forEachWithLock(BiConsumer<Cache, CacheEntry<K,
>> >> >> >> V>>
>> >> >> >> consumer)
>> >> >> >>
>> >> >> >> This require the least amount of changes, however the user can't
>> >> >> >> customize
>> >> >> >> certain parameters that CacheStream currently provides (listed
>> >> >> >> below
>> >> >> >> -
>> >> >> >> big
>> >> >> >> one being filterKeys).
>> >> >> >>
>> >> >> >> 2. CacheStream.forEachWithLock(BiConsumer<Cache, CacheEntry<K,
>> >> >> >> V>>
>> >> >> >> consumer)
>> >> >> >>
>> >> >> >> This method would only be allowed to be invoked on the Stream if
>> >> >> >> no
>> >> >> >> other
>> >> >> >> intermediate operations were invoked, otherwise an exception
>> >> >> >> would
>> >> >> >> be
>> >> >> >> thrown. This still gives us access to all of the CacheStream
>> >> >> >> methods
>> >> >> >> that
>> >> >> >> aren't on the Stream interface (ie. sequentialDistribution,
>> >> >> >> parallelDistribution, parallel, sequential, filterKeys,
>> >> >> >> filterKeySegments,
>> >> >> >> distributedBatchSize, disableRehashAware, timeout).
>> >> >> >>
>> >> >> >> 3. LockedStream<CacheEntry<K, V>> AdvancedCache.lockedStream()
>> >> >> >>
>> >> >> >> This requires the most changes, however the API would be the most
>> >> >> >> explicit.
>> >> >> >> In this case the LockedStream would only have the methods on it
>> >> >> >> that
>> >> >> >> are
>> >> >> >> able to be invoked as noted above and forEach.
>> >> >> >>
>> >> >> >> I personally feel that #3 might be the cleanest, but obviously
>> >> >> >> requires
>> >> >> >> adding more classes. Let me know what you guys think and if you
>> >> >> >> think
>> >> >> >> the
>> >> >> >> optimistic exclusion is acceptable.
>> >> >> >>
>> >> >> >> Thanks,
>> >> >> >>
>> >> >> >> - Will
>> >> >> >>
>> >> >> >> _______________________________________________
>> >> >> >> infinispan-dev mailing list
>> >> >> >> infinispan-dev at lists.jboss.org
>> >> >> >> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>> >> >> > _______________________________________________
>> >> >> > infinispan-dev mailing list
>> >> >> > infinispan-dev at lists.jboss.org
>> >> >> > https://lists.jboss.org/mailman/listinfo/infinispan-dev
>> >> >>
>> >> >>
>> >> >> _______________________________________________
>> >> >> infinispan-dev mailing list
>> >> >> infinispan-dev at lists.jboss.org
>> >> >> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>> >> >
>> >> >
>> >> > _______________________________________________
>> >> > infinispan-dev mailing list
>> >> > infinispan-dev at lists.jboss.org
>> >> > https://lists.jboss.org/mailman/listinfo/infinispan-dev
>> >>
>> >> _______________________________________________
>> >> infinispan-dev mailing list
>> >> infinispan-dev at lists.jboss.org
>> >> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>> >
>> >
>> > _______________________________________________
>> > infinispan-dev mailing list
>> > infinispan-dev at lists.jboss.org
>> > https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>
>> _______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev


From mudokonman at gmail.com  Wed Mar 29 09:24:38 2017
From: mudokonman at gmail.com (William Burns)
Date: Wed, 29 Mar 2017 13:24:38 +0000
Subject: [infinispan-dev] Stream operations under lock
In-Reply-To: <CA+nfvwTFriN-FCBrxgrowttKrWd+fEYhrq1_QWbB8AHe869rFQ@mail.gmail.com>
References: <CA+YCuUM3h0H+ZDe1ZvK5mOuXnLB9bjjP+CkJg_M8eyGKC+jRHA@mail.gmail.com>
	<CA+nfvwTaTFf_Ba4FkLAhejjQwC=p9YeiDuTrUaGj1krr2xgobw@mail.gmail.com>
	<28A6D5BF-6878-4778-A48A-489D91A2BBCB@redhat.com>
	<CA+YCuUMm6XUWh+5wQcYaYfqqA6UKqA9KSwb6TNdZpFJeg_G+cA@mail.gmail.com>
	<CAFm4XO39HKQF==fo7H31BgZhwSnw6qp2_-UM9LQVX65KOcoz+w@mail.gmail.com>
	<CA+YCuUM87dvVkV0gGnnKVzpAPHyQd8fJkH_3iRF0ccyvUz9WPA@mail.gmail.com>
	<CA+nfvwSvHGusN7-YL+ZWBig7id5GDxFu1ThN3CD_z=LGn6x9Zw@mail.gmail.com>
	<CA+YCuUPQr7YB9bNuU308RPEhf9E4GF57+mF5QWS6sDJpaHDKnQ@mail.gmail.com>
	<CA+nfvwTFriN-FCBrxgrowttKrWd+fEYhrq1_QWbB8AHe869rFQ@mail.gmail.com>
Message-ID: <CA+YCuUM3g4sDdyTN3Wg0LNUwZrFEVpvrJbKyeuU9RkoKfEfj+g@mail.gmail.com>

On Wed, Mar 29, 2017 at 4:05 AM Dan Berindei <dan.berindei at gmail.com> wrote:

> On Tue, Mar 28, 2017 at 8:54 PM, William Burns <mudokonman at gmail.com>
> wrote:
> >
> >
> > On Tue, Mar 28, 2017 at 12:52 PM Dan Berindei <dan.berindei at gmail.com>
> > wrote:
> >>
> >> On Tue, Mar 28, 2017 at 6:44 PM, William Burns <mudokonman at gmail.com>
> >> wrote:
> >> >
> >> >
> >> > On Tue, Mar 28, 2017 at 11:24 AM Sanne Grinovero <
> sanne at infinispan.org>
> >> > wrote:
> >> >>
> >> >> Hi Will,
> >> >>
> >> >> I'm confused about the premise; when you state
> >> >>
> >> >> " the Consumer is called while holding the lock for the given key and
> >> >> subsequently released after the Consumer operation completes."
> >> >>
> >> >> What are the transaction boundaries?
> >> >> I see two options, please correct me so I understand:
> >> >>
> >> >> A)
> >> >>
> >> >> transaction.begin();
> >> >> cache.lockedStream().filterKeys(keys).filter(predicate).forEach(
> >> >> someOperation );
> >> >> transaction.commit(); <-- release all locks
> >> >>
> >> >> B)
> >> >>
> >> >> cache.lockedStream().filterKeys(keys).filter(predicate).forEach(
> >> >>     transaction.begin();
> >> >>     //to stuff on entry
> >> >>     /transaction.commit(); <-- release this single entry's lock
> >> >> );
> >> >
> >> >
> >> > The user code doesn't really affect this, it is done internally by
> >> > Infinispan. We would acquire the lock on the user's behalf in the
> stream
> >> > operation then call the user's accept method. Then after they return
> we
> >> > would unlock the lock. The way I have it implemented at the moment in
> my
> >> > PoC
> >> > (which could change) is I don't even start a transaction. It just
> locks
> >> > the
> >> > key and then if the user were to invoke an operation that requires a
> >> > transaction it adds the lock owner to their tx context at that point
> so
> >> > they
> >> > have ownership of the key.
> >> >
> >> > Also to note that a commit or rollback in the Consumer currently
> doesn't
> >> > release the lock on the key. Although this could be discussed to be
> >> > possibly
> >> > changed.
> >> >
> >>
> >> With a transactional cache I was assuming you manage the transaction
> >> yourself... if the user has to call
> >> transactionManager.begin()/commit()/rollback() anyway, why not use a
> >> regular stream?
> >
> >
> > The user would be using an implicit transaction or explicit. Like I said
> > this is up for discussion. The main reason I am staying away from
> managing
> > the transaction is that the user can mess with the transaction as well
> which
> > would possibly release the lock. It is much simpler if all I am doing is
> > managing the lock. And if the user doesn't require a transaction in
> > Infinispan we didn't waste time starting one and releasing one.
> >
>
> Every write to a transactional cache will start an implicit
> transaction, so I don't think we really have the option of not
> starting a transaction.
>

Yes a write to it will, but the stream stuff doesn't write to it, it just
locks. So if the user doesn't have to actually write to the cache, they may
want to just update a 3rd party system for example.


>
> >>
> >>
> >> >>
> >> >>
> >> >>
> >> >> I think it's important to clarify this as I suspect that #A is not
> >> >> implementable within reasonable guarantees, while in the case of B# I
> >> >> see no use for optimistic locking *from a user's perspective*.
> >> >
> >> >
> >> > Exactly my thoughts regarding optimistic. I don't think #A is even
> >> > feasible
> >> > given constraints of having a distributed transaction like this.
> >> >
> >>
> >> Totally agree that #A can't work with any kind of transaction
> >> configuration.
> >>
> >> As to optimistic locking, I would like having "feature parity" between
> >> pessimistic and optimistic caches as much as possible, but I agree
> >> locking eagerly and retrying the consumer on WriteSkewException are a
> >> bit too different to fit under the same API.
> >
> >
> > Yeah I would definitely like to have it as well, but I just can't see
> how it
> > fits in. This is despite the implementation detail that it is quite
> > difficult to get it working currently :D
> >
> >>
> >>
> >> >>
> >> >>
> >> >> Also: what kind of guarantees do you expect about having the
> operation
> >> >> being applied on some entry which is strictly *still* a subset of the
> >> >> keys as defined by the filter predicates?
> >> >> I take it you'd want to acquire the locks during the filtering
> process?
> >> >
> >> >
> >> > That is a good question. I hadn't thought about the details but what I
> >> > had
> >> > implemented was we have to first read the entry, lock the key, reread
> >> > the
> >> > entry (in case of concurrent update) and then finally call their
> >> > Predicate.
> >> > Another reason the filterKeys is much more efficient :) Note this
> read,
> >> > locking and reread is done even without a Predicate supplied. And
> >> > actually I
> >> > can also optimize filterKeys to not do the initial read since we
> already
> >> > have the key.
> >> >
> >>
> >> Would this be more efficient than the consumer reading the key with
> >> FORCE_WRITE_LOCK and deciding what to do based on the value?
> >>
> >
> > The problem with this is you have to go remote to lock the key, return
> the
> > value then do something for every key (not to mention pulling those keys
> > using an iterator). Very costly! The main benefit of the stream with
> lock is
> > that you are performing everything on the primary owner of the data with
> the
> > lock already acquired. The only piece sent remotely is the consumer and
> some
> > internal classes, very light weight and you have all the benefits of data
> > locality.
> >
>
> I meant using cache.withFlags(FORCE_WRITE_LOCK).get(key) in the
> Consumer itself, and assuming it runs on the primary owner.
>

Ah yes you could for pessimistic, but not optimistic or nontx :( In this
case it would be very similar.


>
> >>
> >> >>
> >> >>
> >> >> That would require the move the transaction boundary to the scenario
> >> >> A# which seems undesirable.
> >> >> Technically if I were to need something like this I guess I'd expect
> >> >> to have a user experience akin to B# but have Infinispan essentially
> >> >> use optimistic locking (and auto-skip) on entries which are mutated
> >> >> and fall out of the filter predicate during the lock attempt.
> >> >>
> >> >> Essentially I suspect that we'd not want to implement different
> >> >> versions of this depending on the transaction mode, but figure out
> the
> >> >> use case and implement a one and only transaction mode which suites
> >> >> such use cases. So for example we'd simply not offer a mode which
> >> >> requires to copy the whole grid into the current TX context.
> >> >
> >> >
> >> > This was never the intent and in my follow up emails I came to what
> >> > seems
> >> > like the same conclusion that basically this can't be done with the
> user
> >> > controlling the transaction and it doesn't really make sense in an
> >> > optimistic transaction (since you are already at that node, you are
> >> > already
> >> > doing everything pessimistically).
> >> >
> >>
> >> Even local caches can use optimistic locking :)
> >
> >
> > Yes I know :) I was just framing it in the notion of remote. If anyone
> can
> > think of a nice way of using this with optimistic transactions I would
> all
> > be for it. But optimistic transactions just doesn't make any sense to me
> > when you are locking a key eagerly for someone to do something with it
> > (definition of pessimistic transaction).
> >
>
> Maybe I didn't explain properly...
>
> I meant in optimistic caches we wouldn't acquire the lock at all. The
> stream would just start an optimistic transaction, run the consumer,
> and try to commit. If the prepare fails because of a
> WriteSkewException, start from the beginning.
>

This assumes the user will be updating the cache, I am not sure we can
always assume that (in regards to starting a transaction).

But this is a good point (I misunderstood earlier), it should be pretty
easy to just catch WriteSkewException and retry the given Consumer with the
new entry (if it exists still).


>
> >>
> >>
> >> >>
> >> >>
> >> >> Thanks,
> >> >> Sanne
> >> >>
> >> >>
> >> >>
> >> >> On 28 March 2017 at 14:49, William Burns <mudokonman at gmail.com>
> wrote:
> >> >> >
> >> >> >
> >> >> > On Mon, Mar 27, 2017 at 9:02 PM Galder Zamarre?o <
> galder at redhat.com>
> >> >> > wrote:
> >> >> >>
> >> >> >>
> >> >> >> --
> >> >> >> Galder Zamarre?o
> >> >> >> Infinispan, Red Hat
> >> >> >>
> >> >> >> > On 21 Mar 2017, at 17:16, Dan Berindei <dan.berindei at gmail.com>
> >> >> >> > wrote:
> >> >> >> >
> >> >> >> > I'm leaning towards option 1.
> >> >> >> >
> >> >> >> > Are you thinking about also allowing the consumer to modify the
> >> >> >> > entry,
> >> >> >> > like JCache's EntryProcessors? For a consumer that can only
> modify
> >> >> >> > the
> >> >> >> > current entry, we could even "emulate" locking in an optimistic
> >> >> >> > cache
> >> >> >> > by catching the WriteSkewException and running the consumer
> again.
> >> >> >> >
> >> >> >> > I wouldn't allow this to be mixed with other operations in a
> >> >> >> > stream,
> >> >> >> > because then you may have to run filters/mappers/sorting while
> >> >> >> > holding
> >> >> >> > the lock as well.
> >> >> >>
> >> >> >> ^ Would forEach w/ lock still run for all entries in originator?
> If
> >> >> >> so,
> >> >> >> not being able to filter could be a pain. IOW, you'd be forcing
> all
> >> >> >> entries
> >> >> >> to be shipped to a node and user to do its own filtering. Not
> ideal
> >> >> >> :\
> >> >> >
> >> >> >
> >> >> > No the primary owner would run the operation per entry. I was
> >> >> > thinking
> >> >> > we
> >> >> > would have 2 levels of filtering in my proposal above.
> >> >> >
> >> >> > We would have the first one which is using filterKeys on the
> >> >> > CacheStream
> >> >> > method. This requires serializing keys sent to each node (only
> >> >> > primary
> >> >> > owned
> >> >> > keys are sent). While it has the cost of serialization it makes up
> >> >> > for
> >> >> > by
> >> >> > having constant time lookups (no iterating memory/stores) for the
> >> >> > keys
> >> >> > as it
> >> >> > creates a stream using Cache.get to populate it.
> >> >> >
> >> >> > The second was to support the filter method on the Stream API which
> >> >> > would
> >> >> > allow for a Predicate so you don't have to serialize keys. In this
> >> >> > case
> >> >> > you
> >> >> > wouldn't want to include keys in this Predicate as all keys would
> be
> >> >> > serialized to all nodes and then you still have to iterate and
> check
> >> >> > the
> >> >> > entire data container/store.
> >> >> >
> >> >> > You could actually do both as well. So if you only want a subset of
> >> >> > known
> >> >> > keys where their values match a Predicate this can be done too.
> >> >> >
> >> >> > cache.lockedStream().filterKeys(keys).filter(predicate).forEach();
> >> >> >
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >> >
> >> >> >> > Cheers
> >> >> >> > Dan
> >> >> >> >
> >> >> >> >
> >> >> >> > On Tue, Mar 21, 2017 at 5:37 PM, William Burns
> >> >> >> > <mudokonman at gmail.com>
> >> >> >> > wrote:
> >> >> >> >> Some users have expressed the need to have some sort of forEach
> >> >> >> >> operation
> >> >> >> >> that is performed where the Consumer is called while holding
> the
> >> >> >> >> lock
> >> >> >> >> for
> >> >> >> >> the given key and subsequently released after the Consumer
> >> >> >> >> operation
> >> >> >> >> completes.
> >> >> >> >>
> >> >> >> >> Due to the nature of how streams work with retries and
> performing
> >> >> >> >> the
> >> >> >> >> operation on the primary owner, this works out quite well with
> >> >> >> >> forEach
> >> >> >> >> to be
> >> >> >> >> done in an efficient way.
> >> >> >> >>
> >> >> >> >> The problem is that this only really works well with non tx and
> >> >> >> >> pessimistic
> >> >> >> >> tx. This obviously leaves out optimistic tx, which at first I
> was
> >> >> >> >> a
> >> >> >> >> little
> >> >> >> >> worried about. But after thinking about it more, this
> prelocking
> >> >> >> >> and
> >> >> >> >> optimistic tx don't really fit that well together anyways. So I
> >> >> >> >> am
> >> >> >> >> thinking
> >> >> >> >> whenever this operation is performed it would throw an
> exception
> >> >> >> >> not
> >> >> >> >> letting
> >> >> >> >> the user use this feature in optimistic transactions.
> >> >> >> >>
> >> >> >> >> Another question is what does the API for this look like. I was
> >> >> >> >> debating
> >> >> >> >> between 3 options myself:
> >> >> >> >>
> >> >> >> >> 1. AdvancedCache.forEachWithLock(BiConsumer<Cache,
> CacheEntry<K,
> >> >> >> >> V>>
> >> >> >> >> consumer)
> >> >> >> >>
> >> >> >> >> This require the least amount of changes, however the user
> can't
> >> >> >> >> customize
> >> >> >> >> certain parameters that CacheStream currently provides (listed
> >> >> >> >> below
> >> >> >> >> -
> >> >> >> >> big
> >> >> >> >> one being filterKeys).
> >> >> >> >>
> >> >> >> >> 2. CacheStream.forEachWithLock(BiConsumer<Cache, CacheEntry<K,
> >> >> >> >> V>>
> >> >> >> >> consumer)
> >> >> >> >>
> >> >> >> >> This method would only be allowed to be invoked on the Stream
> if
> >> >> >> >> no
> >> >> >> >> other
> >> >> >> >> intermediate operations were invoked, otherwise an exception
> >> >> >> >> would
> >> >> >> >> be
> >> >> >> >> thrown. This still gives us access to all of the CacheStream
> >> >> >> >> methods
> >> >> >> >> that
> >> >> >> >> aren't on the Stream interface (ie. sequentialDistribution,
> >> >> >> >> parallelDistribution, parallel, sequential, filterKeys,
> >> >> >> >> filterKeySegments,
> >> >> >> >> distributedBatchSize, disableRehashAware, timeout).
> >> >> >> >>
> >> >> >> >> 3. LockedStream<CacheEntry<K, V>> AdvancedCache.lockedStream()
> >> >> >> >>
> >> >> >> >> This requires the most changes, however the API would be the
> most
> >> >> >> >> explicit.
> >> >> >> >> In this case the LockedStream would only have the methods on it
> >> >> >> >> that
> >> >> >> >> are
> >> >> >> >> able to be invoked as noted above and forEach.
> >> >> >> >>
> >> >> >> >> I personally feel that #3 might be the cleanest, but obviously
> >> >> >> >> requires
> >> >> >> >> adding more classes. Let me know what you guys think and if you
> >> >> >> >> think
> >> >> >> >> the
> >> >> >> >> optimistic exclusion is acceptable.
> >> >> >> >>
> >> >> >> >> Thanks,
> >> >> >> >>
> >> >> >> >> - Will
> >> >> >> >>
> >> >> >> >> _______________________________________________
> >> >> >> >> infinispan-dev mailing list
> >> >> >> >> infinispan-dev at lists.jboss.org
> >> >> >> >> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >> >> >> > _______________________________________________
> >> >> >> > infinispan-dev mailing list
> >> >> >> > infinispan-dev at lists.jboss.org
> >> >> >> > https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >> >> >>
> >> >> >>
> >> >> >> _______________________________________________
> >> >> >> infinispan-dev mailing list
> >> >> >> infinispan-dev at lists.jboss.org
> >> >> >> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >> >> >
> >> >> >
> >> >> > _______________________________________________
> >> >> > infinispan-dev mailing list
> >> >> > infinispan-dev at lists.jboss.org
> >> >> > https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >> >>
> >> >> _______________________________________________
> >> >> infinispan-dev mailing list
> >> >> infinispan-dev at lists.jboss.org
> >> >> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >> >
> >> >
> >> > _______________________________________________
> >> > infinispan-dev mailing list
> >> > infinispan-dev at lists.jboss.org
> >> > https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >>
> >> _______________________________________________
> >> infinispan-dev mailing list
> >> infinispan-dev at lists.jboss.org
> >> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >
> >
> > _______________________________________________
> > infinispan-dev mailing list
> > infinispan-dev at lists.jboss.org
> > https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/infinispan-dev/attachments/20170329/55d6a3af/attachment-0001.html 

From dan.berindei at gmail.com  Wed Mar 29 11:37:35 2017
From: dan.berindei at gmail.com (Dan Berindei)
Date: Wed, 29 Mar 2017 18:37:35 +0300
Subject: [infinispan-dev] Stream operations under lock
In-Reply-To: <CA+YCuUM3g4sDdyTN3Wg0LNUwZrFEVpvrJbKyeuU9RkoKfEfj+g@mail.gmail.com>
References: <CA+YCuUM3h0H+ZDe1ZvK5mOuXnLB9bjjP+CkJg_M8eyGKC+jRHA@mail.gmail.com>
	<CA+nfvwTaTFf_Ba4FkLAhejjQwC=p9YeiDuTrUaGj1krr2xgobw@mail.gmail.com>
	<28A6D5BF-6878-4778-A48A-489D91A2BBCB@redhat.com>
	<CA+YCuUMm6XUWh+5wQcYaYfqqA6UKqA9KSwb6TNdZpFJeg_G+cA@mail.gmail.com>
	<CAFm4XO39HKQF==fo7H31BgZhwSnw6qp2_-UM9LQVX65KOcoz+w@mail.gmail.com>
	<CA+YCuUM87dvVkV0gGnnKVzpAPHyQd8fJkH_3iRF0ccyvUz9WPA@mail.gmail.com>
	<CA+nfvwSvHGusN7-YL+ZWBig7id5GDxFu1ThN3CD_z=LGn6x9Zw@mail.gmail.com>
	<CA+YCuUPQr7YB9bNuU308RPEhf9E4GF57+mF5QWS6sDJpaHDKnQ@mail.gmail.com>
	<CA+nfvwTFriN-FCBrxgrowttKrWd+fEYhrq1_QWbB8AHe869rFQ@mail.gmail.com>
	<CA+YCuUM3g4sDdyTN3Wg0LNUwZrFEVpvrJbKyeuU9RkoKfEfj+g@mail.gmail.com>
Message-ID: <CA+nfvwTBKaUyvoYqcaYc-bW6ozVi2s020Z9wxkYKRBUfj40XWw@mail.gmail.com>

On Wed, Mar 29, 2017 at 4:24 PM, William Burns <mudokonman at gmail.com> wrote:
>
>
> On Wed, Mar 29, 2017 at 4:05 AM Dan Berindei <dan.berindei at gmail.com> wrote:
>>
>> On Tue, Mar 28, 2017 at 8:54 PM, William Burns <mudokonman at gmail.com>
>> wrote:
>> >
>> >
>> > On Tue, Mar 28, 2017 at 12:52 PM Dan Berindei <dan.berindei at gmail.com>
>> > wrote:
>> >>
>> >> On Tue, Mar 28, 2017 at 6:44 PM, William Burns <mudokonman at gmail.com>
>> >> wrote:
>> >> >
>> >> >
>> >> > On Tue, Mar 28, 2017 at 11:24 AM Sanne Grinovero
>> >> > <sanne at infinispan.org>
>> >> > wrote:
>> >> >>
>> >> >> Hi Will,
>> >> >>
>> >> >> I'm confused about the premise; when you state
>> >> >>
>> >> >> " the Consumer is called while holding the lock for the given key
>> >> >> and
>> >> >> subsequently released after the Consumer operation completes."
>> >> >>
>> >> >> What are the transaction boundaries?
>> >> >> I see two options, please correct me so I understand:
>> >> >>
>> >> >> A)
>> >> >>
>> >> >> transaction.begin();
>> >> >> cache.lockedStream().filterKeys(keys).filter(predicate).forEach(
>> >> >> someOperation );
>> >> >> transaction.commit(); <-- release all locks
>> >> >>
>> >> >> B)
>> >> >>
>> >> >> cache.lockedStream().filterKeys(keys).filter(predicate).forEach(
>> >> >>     transaction.begin();
>> >> >>     //to stuff on entry
>> >> >>     /transaction.commit(); <-- release this single entry's lock
>> >> >> );
>> >> >
>> >> >
>> >> > The user code doesn't really affect this, it is done internally by
>> >> > Infinispan. We would acquire the lock on the user's behalf in the
>> >> > stream
>> >> > operation then call the user's accept method. Then after they return
>> >> > we
>> >> > would unlock the lock. The way I have it implemented at the moment in
>> >> > my
>> >> > PoC
>> >> > (which could change) is I don't even start a transaction. It just
>> >> > locks
>> >> > the
>> >> > key and then if the user were to invoke an operation that requires a
>> >> > transaction it adds the lock owner to their tx context at that point
>> >> > so
>> >> > they
>> >> > have ownership of the key.
>> >> >
>> >> > Also to note that a commit or rollback in the Consumer currently
>> >> > doesn't
>> >> > release the lock on the key. Although this could be discussed to be
>> >> > possibly
>> >> > changed.
>> >> >
>> >>
>> >> With a transactional cache I was assuming you manage the transaction
>> >> yourself... if the user has to call
>> >> transactionManager.begin()/commit()/rollback() anyway, why not use a
>> >> regular stream?
>> >
>> >
>> > The user would be using an implicit transaction or explicit. Like I said
>> > this is up for discussion. The main reason I am staying away from
>> > managing
>> > the transaction is that the user can mess with the transaction as well
>> > which
>> > would possibly release the lock. It is much simpler if all I am doing is
>> > managing the lock. And if the user doesn't require a transaction in
>> > Infinispan we didn't waste time starting one and releasing one.
>> >
>>
>> Every write to a transactional cache will start an implicit
>> transaction, so I don't think we really have the option of not
>> starting a transaction.
>
>
> Yes a write to it will, but the stream stuff doesn't write to it, it just
> locks. So if the user doesn't have to actually write to the cache, they may
> want to just update a 3rd party system for example.
>

Yeah, my main use case was "processing" all/most of the entries in the
cache in some way. I must confess I didn't even think about using
Infinispan locks to synchronize access to external systems, I was
thinking only what would happen when the Consumer would access
multiple caches :)

OTOH a read-only transaction doesn't have to be very expensive either.
We already have an optimization in PessimisticLockingInterceptor to
skip sending the LockControlCommand to backup owners [1]. This applies
only for LockControlCommands generated implicitly by writes, but we
could modify TxDistributionInterceptor.visitLockControlCommand() to
skip the RPC for explicit AdvancedCache.lock() calls as well. If there
are no writes and no remote locks, the prepare and commit should be
local as well.

There will still be some overhead from interacting with the
transaction manager, and I can understand wanting to avoid that. I'm
not yet convinced it's worth writing a new API and potentially
confusing lock release behaviour because of it...

[1]: https://github.com/infinispan/infinispan/blob/9fb8752d847a85e168d785237f196967ace313f3/core/src/main/java/org/infinispan/interceptors/locking/PessimisticLockingInterceptor.java#L364-L376

>>
>>
>> >>
>> >>
>> >> >>
>> >> >>
>> >> >>
>> >> >> I think it's important to clarify this as I suspect that #A is not
>> >> >> implementable within reasonable guarantees, while in the case of B#
>> >> >> I
>> >> >> see no use for optimistic locking *from a user's perspective*.
>> >> >
>> >> >
>> >> > Exactly my thoughts regarding optimistic. I don't think #A is even
>> >> > feasible
>> >> > given constraints of having a distributed transaction like this.
>> >> >
>> >>
>> >> Totally agree that #A can't work with any kind of transaction
>> >> configuration.
>> >>
>> >> As to optimistic locking, I would like having "feature parity" between
>> >> pessimistic and optimistic caches as much as possible, but I agree
>> >> locking eagerly and retrying the consumer on WriteSkewException are a
>> >> bit too different to fit under the same API.
>> >
>> >
>> > Yeah I would definitely like to have it as well, but I just can't see
>> > how it
>> > fits in. This is despite the implementation detail that it is quite
>> > difficult to get it working currently :D
>> >
>> >>
>> >>
>> >> >>
>> >> >>
>> >> >> Also: what kind of guarantees do you expect about having the
>> >> >> operation
>> >> >> being applied on some entry which is strictly *still* a subset of
>> >> >> the
>> >> >> keys as defined by the filter predicates?
>> >> >> I take it you'd want to acquire the locks during the filtering
>> >> >> process?
>> >> >
>> >> >
>> >> > That is a good question. I hadn't thought about the details but what
>> >> > I
>> >> > had
>> >> > implemented was we have to first read the entry, lock the key, reread
>> >> > the
>> >> > entry (in case of concurrent update) and then finally call their
>> >> > Predicate.
>> >> > Another reason the filterKeys is much more efficient :) Note this
>> >> > read,
>> >> > locking and reread is done even without a Predicate supplied. And
>> >> > actually I
>> >> > can also optimize filterKeys to not do the initial read since we
>> >> > already
>> >> > have the key.
>> >> >
>> >>
>> >> Would this be more efficient than the consumer reading the key with
>> >> FORCE_WRITE_LOCK and deciding what to do based on the value?
>> >>
>> >
>> > The problem with this is you have to go remote to lock the key, return
>> > the
>> > value then do something for every key (not to mention pulling those keys
>> > using an iterator). Very costly! The main benefit of the stream with
>> > lock is
>> > that you are performing everything on the primary owner of the data with
>> > the
>> > lock already acquired. The only piece sent remotely is the consumer and
>> > some
>> > internal classes, very light weight and you have all the benefits of
>> > data
>> > locality.
>> >
>>
>> I meant using cache.withFlags(FORCE_WRITE_LOCK).get(key) in the
>> Consumer itself, and assuming it runs on the primary owner.
>
>
> Ah yes you could for pessimistic, but not optimistic or nontx :( In this
> case it would be very similar.
>
>>
>>
>> >>
>> >> >>
>> >> >>
>> >> >> That would require the move the transaction boundary to the scenario
>> >> >> A# which seems undesirable.
>> >> >> Technically if I were to need something like this I guess I'd expect
>> >> >> to have a user experience akin to B# but have Infinispan essentially
>> >> >> use optimistic locking (and auto-skip) on entries which are mutated
>> >> >> and fall out of the filter predicate during the lock attempt.
>> >> >>
>> >> >> Essentially I suspect that we'd not want to implement different
>> >> >> versions of this depending on the transaction mode, but figure out
>> >> >> the
>> >> >> use case and implement a one and only transaction mode which suites
>> >> >> such use cases. So for example we'd simply not offer a mode which
>> >> >> requires to copy the whole grid into the current TX context.
>> >> >
>> >> >
>> >> > This was never the intent and in my follow up emails I came to what
>> >> > seems
>> >> > like the same conclusion that basically this can't be done with the
>> >> > user
>> >> > controlling the transaction and it doesn't really make sense in an
>> >> > optimistic transaction (since you are already at that node, you are
>> >> > already
>> >> > doing everything pessimistically).
>> >> >
>> >>
>> >> Even local caches can use optimistic locking :)
>> >
>> >
>> > Yes I know :) I was just framing it in the notion of remote. If anyone
>> > can
>> > think of a nice way of using this with optimistic transactions I would
>> > all
>> > be for it. But optimistic transactions just doesn't make any sense to me
>> > when you are locking a key eagerly for someone to do something with it
>> > (definition of pessimistic transaction).
>> >
>>
>> Maybe I didn't explain properly...
>>
>> I meant in optimistic caches we wouldn't acquire the lock at all. The
>> stream would just start an optimistic transaction, run the consumer,
>> and try to commit. If the prepare fails because of a
>> WriteSkewException, start from the beginning.
>
>
> This assumes the user will be updating the cache, I am not sure we can
> always assume that (in regards to starting a transaction).
>
> But this is a good point (I misunderstood earlier), it should be pretty easy
> to just catch WriteSkewException and retry the given Consumer with the new
> entry (if it exists still).
>
>>
>>
>> >>
>> >>
>> >> >>
>> >> >>
>> >> >> Thanks,
>> >> >> Sanne
>> >> >>
>> >> >>
>> >> >>
>> >> >> On 28 March 2017 at 14:49, William Burns <mudokonman at gmail.com>
>> >> >> wrote:
>> >> >> >
>> >> >> >
>> >> >> > On Mon, Mar 27, 2017 at 9:02 PM Galder Zamarre?o
>> >> >> > <galder at redhat.com>
>> >> >> > wrote:
>> >> >> >>
>> >> >> >>
>> >> >> >> --
>> >> >> >> Galder Zamarre?o
>> >> >> >> Infinispan, Red Hat
>> >> >> >>
>> >> >> >> > On 21 Mar 2017, at 17:16, Dan Berindei <dan.berindei at gmail.com>
>> >> >> >> > wrote:
>> >> >> >> >
>> >> >> >> > I'm leaning towards option 1.
>> >> >> >> >
>> >> >> >> > Are you thinking about also allowing the consumer to modify the
>> >> >> >> > entry,
>> >> >> >> > like JCache's EntryProcessors? For a consumer that can only
>> >> >> >> > modify
>> >> >> >> > the
>> >> >> >> > current entry, we could even "emulate" locking in an optimistic
>> >> >> >> > cache
>> >> >> >> > by catching the WriteSkewException and running the consumer
>> >> >> >> > again.
>> >> >> >> >
>> >> >> >> > I wouldn't allow this to be mixed with other operations in a
>> >> >> >> > stream,
>> >> >> >> > because then you may have to run filters/mappers/sorting while
>> >> >> >> > holding
>> >> >> >> > the lock as well.
>> >> >> >>
>> >> >> >> ^ Would forEach w/ lock still run for all entries in originator?
>> >> >> >> If
>> >> >> >> so,
>> >> >> >> not being able to filter could be a pain. IOW, you'd be forcing
>> >> >> >> all
>> >> >> >> entries
>> >> >> >> to be shipped to a node and user to do its own filtering. Not
>> >> >> >> ideal
>> >> >> >> :\
>> >> >> >
>> >> >> >
>> >> >> > No the primary owner would run the operation per entry. I was
>> >> >> > thinking
>> >> >> > we
>> >> >> > would have 2 levels of filtering in my proposal above.
>> >> >> >
>> >> >> > We would have the first one which is using filterKeys on the
>> >> >> > CacheStream
>> >> >> > method. This requires serializing keys sent to each node (only
>> >> >> > primary
>> >> >> > owned
>> >> >> > keys are sent). While it has the cost of serialization it makes up
>> >> >> > for
>> >> >> > by
>> >> >> > having constant time lookups (no iterating memory/stores) for the
>> >> >> > keys
>> >> >> > as it
>> >> >> > creates a stream using Cache.get to populate it.
>> >> >> >
>> >> >> > The second was to support the filter method on the Stream API
>> >> >> > which
>> >> >> > would
>> >> >> > allow for a Predicate so you don't have to serialize keys. In this
>> >> >> > case
>> >> >> > you
>> >> >> > wouldn't want to include keys in this Predicate as all keys would
>> >> >> > be
>> >> >> > serialized to all nodes and then you still have to iterate and
>> >> >> > check
>> >> >> > the
>> >> >> > entire data container/store.
>> >> >> >
>> >> >> > You could actually do both as well. So if you only want a subset
>> >> >> > of
>> >> >> > known
>> >> >> > keys where their values match a Predicate this can be done too.
>> >> >> >
>> >> >> > cache.lockedStream().filterKeys(keys).filter(predicate).forEach();
>> >> >> >
>> >> >> >>
>> >> >> >>
>> >> >> >>
>> >> >> >> >
>> >> >> >> > Cheers
>> >> >> >> > Dan
>> >> >> >> >
>> >> >> >> >
>> >> >> >> > On Tue, Mar 21, 2017 at 5:37 PM, William Burns
>> >> >> >> > <mudokonman at gmail.com>
>> >> >> >> > wrote:
>> >> >> >> >> Some users have expressed the need to have some sort of
>> >> >> >> >> forEach
>> >> >> >> >> operation
>> >> >> >> >> that is performed where the Consumer is called while holding
>> >> >> >> >> the
>> >> >> >> >> lock
>> >> >> >> >> for
>> >> >> >> >> the given key and subsequently released after the Consumer
>> >> >> >> >> operation
>> >> >> >> >> completes.
>> >> >> >> >>
>> >> >> >> >> Due to the nature of how streams work with retries and
>> >> >> >> >> performing
>> >> >> >> >> the
>> >> >> >> >> operation on the primary owner, this works out quite well with
>> >> >> >> >> forEach
>> >> >> >> >> to be
>> >> >> >> >> done in an efficient way.
>> >> >> >> >>
>> >> >> >> >> The problem is that this only really works well with non tx
>> >> >> >> >> and
>> >> >> >> >> pessimistic
>> >> >> >> >> tx. This obviously leaves out optimistic tx, which at first I
>> >> >> >> >> was
>> >> >> >> >> a
>> >> >> >> >> little
>> >> >> >> >> worried about. But after thinking about it more, this
>> >> >> >> >> prelocking
>> >> >> >> >> and
>> >> >> >> >> optimistic tx don't really fit that well together anyways. So
>> >> >> >> >> I
>> >> >> >> >> am
>> >> >> >> >> thinking
>> >> >> >> >> whenever this operation is performed it would throw an
>> >> >> >> >> exception
>> >> >> >> >> not
>> >> >> >> >> letting
>> >> >> >> >> the user use this feature in optimistic transactions.
>> >> >> >> >>
>> >> >> >> >> Another question is what does the API for this look like. I
>> >> >> >> >> was
>> >> >> >> >> debating
>> >> >> >> >> between 3 options myself:
>> >> >> >> >>
>> >> >> >> >> 1. AdvancedCache.forEachWithLock(BiConsumer<Cache,
>> >> >> >> >> CacheEntry<K,
>> >> >> >> >> V>>
>> >> >> >> >> consumer)
>> >> >> >> >>
>> >> >> >> >> This require the least amount of changes, however the user
>> >> >> >> >> can't
>> >> >> >> >> customize
>> >> >> >> >> certain parameters that CacheStream currently provides (listed
>> >> >> >> >> below
>> >> >> >> >> -
>> >> >> >> >> big
>> >> >> >> >> one being filterKeys).
>> >> >> >> >>
>> >> >> >> >> 2. CacheStream.forEachWithLock(BiConsumer<Cache, CacheEntry<K,
>> >> >> >> >> V>>
>> >> >> >> >> consumer)
>> >> >> >> >>
>> >> >> >> >> This method would only be allowed to be invoked on the Stream
>> >> >> >> >> if
>> >> >> >> >> no
>> >> >> >> >> other
>> >> >> >> >> intermediate operations were invoked, otherwise an exception
>> >> >> >> >> would
>> >> >> >> >> be
>> >> >> >> >> thrown. This still gives us access to all of the CacheStream
>> >> >> >> >> methods
>> >> >> >> >> that
>> >> >> >> >> aren't on the Stream interface (ie. sequentialDistribution,
>> >> >> >> >> parallelDistribution, parallel, sequential, filterKeys,
>> >> >> >> >> filterKeySegments,
>> >> >> >> >> distributedBatchSize, disableRehashAware, timeout).
>> >> >> >> >>
>> >> >> >> >> 3. LockedStream<CacheEntry<K, V>> AdvancedCache.lockedStream()
>> >> >> >> >>
>> >> >> >> >> This requires the most changes, however the API would be the
>> >> >> >> >> most
>> >> >> >> >> explicit.
>> >> >> >> >> In this case the LockedStream would only have the methods on
>> >> >> >> >> it
>> >> >> >> >> that
>> >> >> >> >> are
>> >> >> >> >> able to be invoked as noted above and forEach.
>> >> >> >> >>
>> >> >> >> >> I personally feel that #3 might be the cleanest, but obviously
>> >> >> >> >> requires
>> >> >> >> >> adding more classes. Let me know what you guys think and if
>> >> >> >> >> you
>> >> >> >> >> think
>> >> >> >> >> the
>> >> >> >> >> optimistic exclusion is acceptable.
>> >> >> >> >>
>> >> >> >> >> Thanks,
>> >> >> >> >>
>> >> >> >> >> - Will
>> >> >> >> >>
>> >> >> >> >> _______________________________________________
>> >> >> >> >> infinispan-dev mailing list
>> >> >> >> >> infinispan-dev at lists.jboss.org
>> >> >> >> >> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>> >> >> >> > _______________________________________________
>> >> >> >> > infinispan-dev mailing list
>> >> >> >> > infinispan-dev at lists.jboss.org
>> >> >> >> > https://lists.jboss.org/mailman/listinfo/infinispan-dev
>> >> >> >>
>> >> >> >>
>> >> >> >> _______________________________________________
>> >> >> >> infinispan-dev mailing list
>> >> >> >> infinispan-dev at lists.jboss.org
>> >> >> >> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>> >> >> >
>> >> >> >
>> >> >> > _______________________________________________
>> >> >> > infinispan-dev mailing list
>> >> >> > infinispan-dev at lists.jboss.org
>> >> >> > https://lists.jboss.org/mailman/listinfo/infinispan-dev
>> >> >>
>> >> >> _______________________________________________
>> >> >> infinispan-dev mailing list
>> >> >> infinispan-dev at lists.jboss.org
>> >> >> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>> >> >
>> >> >
>> >> > _______________________________________________
>> >> > infinispan-dev mailing list
>> >> > infinispan-dev at lists.jboss.org
>> >> > https://lists.jboss.org/mailman/listinfo/infinispan-dev
>> >>
>> >> _______________________________________________
>> >> infinispan-dev mailing list
>> >> infinispan-dev at lists.jboss.org
>> >> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>> >
>> >
>> > _______________________________________________
>> > infinispan-dev mailing list
>> > infinispan-dev at lists.jboss.org
>> > https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>
>> _______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev


From slaskawi at redhat.com  Thu Mar 30 05:45:57 2017
From: slaskawi at redhat.com (Sebastian Laskawiec)
Date: Thu, 30 Mar 2017 09:45:57 +0000
Subject: [infinispan-dev] Branching proposal
In-Reply-To: <CA+YCuUOcM9xtdGzi-pQP-xm87kFhiZnq6+pTy4vKBgkp=+aB4g@mail.gmail.com>
References: <CAA9R9O6X3kOu3h4=_2z6RZKExYC30B0JiJMvEYQ3mu9HGq20Rw@mail.gmail.com>
	<A8D365BF-8BF1-4C60-893B-F65901C2217C@redhat.com>
	<CA+YCuUOcM9xtdGzi-pQP-xm87kFhiZnq6+pTy4vKBgkp=+aB4g@mail.gmail.com>
Message-ID: <CAA9R9O6Kebhor=VuapMVRh6eHkhdpHtWdqH7bh4i1FaR=D+wPQ@mail.gmail.com>

Ok, I think I can try to sum this proposal and discussion up:

   - The proposal doesn't fit well into our workflow.
   - We are more focused on developing in master branch and do occasional
   (and sometimes partial) backports.
   - We use at most one maintenance branch, so there's no big deal.
   - Dan's one-liner (`git tag --contains $(git log --grep <JIRA> -1
   --format="%h" master)`) very nicely replaces `git tag --contains <SHA1>`
   but as Galder mentioned, we should always use JIRAs.

Thanks a lot guys for all the input!

Cheers,
Sebastian

On Tue, Mar 28, 2017 at 3:56 PM William Burns <mudokonman at gmail.com> wrote:

> On Tue, Mar 28, 2017 at 9:27 AM Galder Zamarre?o <galder at redhat.com>
> wrote:
>
> Why are we working in 9.1.x, 9.2.x and master in paralell? We normally
> work on master and maybe one more maintenance branch.
>
> Except for occasional tricky backports (e.g. Radim's work) the rest has
> been pretty straightforward for me. Also, the number of backports I work on
> is low in general.
>
>
> +1 For me the hard part is just remembering it needs to be backported. And
> as Sanne mentioned refactorings shouldn't really be backported and these
> are the types of the things that cause the most conflicts. And to be honest
> on some backports I might not pull every change since the real fix may have
> been quite small.
>
>
>
> Cheers,
> --
> Galder Zamarre?o
> Infinispan, Red Hat
>
> > On 27 Mar 2017, at 11:33, Sebastian Laskawiec <slaskawi at redhat.com>
> wrote:
> >
> > Hey!
> >
> > We are about to start working on 9.1.x and 9.2.y branches so I would
> like to propose alternative merging strategy.
> >
> > Our current workflow looks like this:
> >
> > X - new commit
> > X` - cherry pick to maintenance branch
> > --+-------------------+-------X----- master
> >   |                    \------X`---- 9.2.x
> >   \---------------------------X``--- 9.1.x
> >
> > Each commit needs to be reviewed in master branch and backported to the
> maintenance branches. From maintenance perspective this is a bit painful,
> since in above example we need to get 3 times through PR queue. Also it's
> worth to mention that X is not X` nor X``. Cherry-picking creates a copy of
> a commit. This makes some useful tricks (like git tag --contains <sha1>) a
> bit harder to use. Finally, this approach allows the codebase to diverge
> from maintenance branches very fast (someone might just forget to backport
> some of the refactoring stuff).
> >
> > The proposal:
> >
> > X, Y - new commits
> > / - merge commits
> > --+---------+------/----/--- master
> >   |          \----/---Y/---- 9.2.x
> >   \-------------X/---------- 9.1.x
> >
> > With the proposal, a developer should always implement a given feature
> in the lowest possible maintenance branch. Then we will run a set of merges
> from 9.1.x into 9.2.x and finally into master. The biggest advantage of
> this approach is that given functionality (identified by a commit) will
> have the same SHA1 for all branches. This will allow all tools like
> (mentioned before) `git tag --contains <sha1>` to work. There are also some
> further implications of this approach:
> >       ? Merging commits should be performed very often (even
> automatically in the night (if merged without any problems)).
> >       ? After releasing each maintenance release, someone will need to
> do a merge with strategy `ours` (`git merge -s ours upstream/9.2.x`). This
> way we will not have to solve version conflicts in poms.
> >       ? Since there is no nice way to rebase a merge commit, they should
> be pushed directly into the master branch (without review, without CI).
> After the merge, HEAD will change and CI will automatically pick the build.
> Remember, merges should be done very often. So I assume there won't be any
> problems most of the times.
> >       ? Finally, with this approach the code diverges slight slower (at
> least from my experience). Mainly because we don't need to remember to
> cherry-pick individual commits. They are automatically "taken" by a merge.
> > From my past experience, this strategy works pretty nice and can be
> almost fully automated. It significantly lowers the maintenance pain around
> cherry-picks. However there is nothing for free, and we would need to get
> used to pushing merged directly into master (which is fine to me but some
> of you might not like it).
> >
> > Thanks,
> > Sebastian
> > _______________________________________________
> > infinispan-dev mailing list
> > infinispan-dev at lists.jboss.org
> > https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/infinispan-dev/attachments/20170330/7962c5ee/attachment-0001.html 

From ttarrant at redhat.com  Thu Mar 30 08:25:53 2017
From: ttarrant at redhat.com (Tristan Tarrant)
Date: Thu, 30 Mar 2017 14:25:53 +0200
Subject: [infinispan-dev] Hot Rod secured by default
Message-ID: <f3cee023-96a1-3c7d-a3ce-b575cd5b0f7e@redhat.com>

Dear all,

after a mini chat on IRC, I wanted to bring this to everybody's attention.

We should make the Hot Rod endpoint require authentication in the 
out-of-the-box configuration.
The proposal is to enable the PLAIN (or, preferably, DIGEST) SASL 
mechanism against the ApplicationRealm and require users to run the 
add-user script.
This would achieve two goals:
- secure out-of-the-box configuration, which is always a good idea
- access to the "protected" schema and script caches which is prevented 
when not on loopback on non-authenticated endpoints.

Tristan
-- 
Tristan Tarrant
Infinispan Lead
JBoss, a division of Red Hat

From ttarrant at redhat.com  Thu Mar 30 08:39:07 2017
From: ttarrant at redhat.com (Tristan Tarrant)
Date: Thu, 30 Mar 2017 14:39:07 +0200
Subject: [infinispan-dev] Hot Rod secured by default
In-Reply-To: <f3cee023-96a1-3c7d-a3ce-b575cd5b0f7e@redhat.com>
References: <f3cee023-96a1-3c7d-a3ce-b575cd5b0f7e@redhat.com>
Message-ID: <a50ae5ea-cef6-0af1-e33a-85242b5dcb26@redhat.com>

Let me add another item:

combined with Sebastian's PR [1] we could also turn on encryption by 
default using a self-signed certificate. We would also need to have an 
easy option (i.e. a boolean, false by default) on the Hot Rod clients to 
trust all certs.

This means that a Hot Rod client would need to be configured as follows:

ConfigurationBuilder clientBuilder = new ConfigurationBuilder();
clientBuilder
   .security()
     .authentication()
       .username("user").realm("realm").password("password")
     .ssl()
       .enable().trustAll(true);

without having to manually set up callback handlers or trustmanagers.

I don't think this would affect the user experience too much.

Tristan

[1] https://github.com/infinispan/infinispan/pull/5036

On 30/03/2017 14:25, Tristan Tarrant wrote:
> Dear all,
> 
> after a mini chat on IRC, I wanted to bring this to everybody's attention.
> 
> We should make the Hot Rod endpoint require authentication in the 
> out-of-the-box configuration.
> The proposal is to enable the PLAIN (or, preferably, DIGEST) SASL 
> mechanism against the ApplicationRealm and require users to run the 
> add-user script.
> This would achieve two goals:
> - secure out-of-the-box configuration, which is always a good idea
> - access to the "protected" schema and script caches which is prevented 
> when not on loopback on non-authenticated endpoints.
> 
> Tristan

-- 
Tristan Tarrant
Infinispan Lead
JBoss, a division of Red Hat

From gustavo at infinispan.org  Thu Mar 30 08:48:20 2017
From: gustavo at infinispan.org (Gustavo Fernandes)
Date: Thu, 30 Mar 2017 13:48:20 +0100
Subject: [infinispan-dev] Hot Rod secured by default
In-Reply-To: <f3cee023-96a1-3c7d-a3ce-b575cd5b0f7e@redhat.com>
References: <f3cee023-96a1-3c7d-a3ce-b575cd5b0f7e@redhat.com>
Message-ID: <CAH8Ud1GiyphzhNAsFL5LTAtLid65UbJEnri=d57gR9vpNq+kbA@mail.gmail.com>

+1

On Thu, Mar 30, 2017 at 1:25 PM, Tristan Tarrant <ttarrant at redhat.com>
wrote:

> Dear all,
>
> after a mini chat on IRC, I wanted to bring this to everybody's attention.
>
> We should make the Hot Rod endpoint require authentication in the
> out-of-the-box configuration.
> The proposal is to enable the PLAIN (or, preferably, DIGEST) SASL
> mechanism against the ApplicationRealm and require users to run the
> add-user script.
> This would achieve two goals:
> - secure out-of-the-box configuration, which is always a good idea
> - access to the "protected" schema and script caches which is prevented
> when not on loopback on non-authenticated endpoints.
>
> Tristan
> --
> Tristan Tarrant
> Infinispan Lead
> JBoss, a division of Red Hat
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/infinispan-dev/attachments/20170330/f34ced1a/attachment.html 

From galder at redhat.com  Thu Mar 30 08:51:17 2017
From: galder at redhat.com (=?utf-8?Q?Galder_Zamarre=C3=B1o?=)
Date: Thu, 30 Mar 2017 14:51:17 +0200
Subject: [infinispan-dev] Executing server tasks that contain POJOs
Message-ID: <7383B4E7-B386-41CE-A62E-8DFBD439C48D@redhat.com>

Hi all,

For a demo I'm giving next week, I'd like to show how to use distributed streams via a remote server task. All server tasks that we have in testsuite rely on primitives but in my case I wanted to use POJOs. 

To do that, I needed to get compatibility mode working in such way that those POJOs could be unmarshalled for the server task. Since in another demo I'm showing Protostream based POJOs, I thought I'd try to use that as mechanism to unmarshall POJOs server side.

We have a test for such scenario [1], but the reality (running on a proper server) is anything that simple. Here's a list of things I've found out while creating a WordCount example that relies on a POJO:

1. Out of the box, it's impossible to set compatibility marshaller to org.infinispan.query.remote.CompatibilityProtoStreamMarshaller [1] because "org.infinispan.main" classloader can't access that class. I worked around that by tweaking the module.xml to have an optional dependency to "org.infinispan.remote-query.server" module.

2. After doing that, I had to register the protofile and associated classes remotely in the server. Again, there's no out of the box mechanism for that, so I created a remote server task that would do that [3].

3. Finally, with all that in place, I was able to complete the WordCount test [4] with a final caveat: the return of the word count, and words protofile registration, tasks return objects that are not marshalled by the compatibility marshaller, so I had to make sure that the remote cache manager used for those tasks uses the default marshaller.

Clearly we need to improve on this, and we have plans to address these issues (with new upcoming transcoding capabilities), but I thought it'd be worth mentioning the problems found in case anyone else encounters them before transcoding is in place.

Cheers,

[1] https://github.com/galderz/datagrid-patterns/blob/master/server-config/domain/domain.xml#L139
[2] https://github.com/galderz/datagrid-patterns/blob/master/server-config/org.infinispan.main_module.xml#L18
[3] https://github.com/galderz/datagrid-patterns/blob/master/analytics-stream/tasks-server/src/main/java/test/WordsProtoTask.java
[4] https://github.com/galderz/datagrid-patterns/blob/master/analytics-stream/tasks-client/src/test/java/test/WordCountTest.java
--
Galder Zamarre?o
Infinispan, Red Hat



From slaskawi at redhat.com  Thu Mar 30 08:54:06 2017
From: slaskawi at redhat.com (Sebastian Laskawiec)
Date: Thu, 30 Mar 2017 12:54:06 +0000
Subject: [infinispan-dev] Hot Rod secured by default
In-Reply-To: <f3cee023-96a1-3c7d-a3ce-b575cd5b0f7e@redhat.com>
References: <f3cee023-96a1-3c7d-a3ce-b575cd5b0f7e@redhat.com>
Message-ID: <CAA9R9O7S7frPE-sS60Z2E6jV_+OdL18+dvMDFJwQsd-FQfQt7w@mail.gmail.com>

I agree the security out of the box is good. But at the same time we don't
want to make Infinispan harder to use for new developers. Out of the box
configuration should be "good enough" to start hacking.

I would propose to make all the endpoints unprotected (with authentication
disabled) on localhost/loopback and protected when calling from the outside
world.

On Thu, Mar 30, 2017 at 2:39 PM Tristan Tarrant <ttarrant at redhat.com> wrote:

> Dear all,
>
> after a mini chat on IRC, I wanted to bring this to everybody's attention.
>
> We should make the Hot Rod endpoint require authentication in the
> out-of-the-box configuration.
> The proposal is to enable the PLAIN (or, preferably, DIGEST) SASL
> mechanism against the ApplicationRealm and require users to run the
> add-user script.
> This would achieve two goals:
> - secure out-of-the-box configuration, which is always a good idea
> - access to the "protected" schema and script caches which is prevented
> when not on loopback on non-authenticated endpoints.
>
> Tristan
> --
> Tristan Tarrant
> Infinispan Lead
> JBoss, a division of Red Hat
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/infinispan-dev/attachments/20170330/8f1e51ae/attachment.html 

From slaskawi at redhat.com  Thu Mar 30 09:01:40 2017
From: slaskawi at redhat.com (Sebastian Laskawiec)
Date: Thu, 30 Mar 2017 13:01:40 +0000
Subject: [infinispan-dev] Single Endpoint design
Message-ID: <CAA9R9O6c8SHB3E9PboofxRuj1LcA5yuvz3EEAUeX5=2Gdr-J0g@mail.gmail.com>

Hey!

My plan is to start working on a Single Point support for Infinispan Server
very soon and I prepared a design:
https://github.com/infinispan/infinispan/pull/5041

As you can see I did not use our Wiki (as we used to) because it doesn't
support inline comments (which is pretty bad in my opinion). I would like
to propose to keep all the designs along with our source code. This
approach has been successfully used by the Kubernetes [1] folks (although
they migrated designs into the new Community repository [2] recently). I
think it might be a good idea to do something similar.

Feedback on both items is more than welcome.

Thanks,
Sebastian

[1] https://github.com/kubernetes/kubernetes/tree/master/docs/proposals
[2]
https://github.com/kubernetes/community/tree/master/contributors/design-proposals
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/infinispan-dev/attachments/20170330/0c5d3dd8/attachment.html 

From gustavo at infinispan.org  Thu Mar 30 09:12:13 2017
From: gustavo at infinispan.org (Gustavo Fernandes)
Date: Thu, 30 Mar 2017 14:12:13 +0100
Subject: [infinispan-dev] Hot Rod secured by default
In-Reply-To: <a50ae5ea-cef6-0af1-e33a-85242b5dcb26@redhat.com>
References: <f3cee023-96a1-3c7d-a3ce-b575cd5b0f7e@redhat.com>
	<a50ae5ea-cef6-0af1-e33a-85242b5dcb26@redhat.com>
Message-ID: <CAH8Ud1FUcyju1KLtQDJbTK-QS2z6nmc=KPXU7Wmu67x1ng7TXw@mail.gmail.com>

-1 to SSL by default

On Thu, Mar 30, 2017 at 1:39 PM, Tristan Tarrant <ttarrant at redhat.com>
wrote:

> Let me add another item:
>
> combined with Sebastian's PR [1] we could also turn on encryption by
> default using a self-signed certificate. We would also need to have an
> easy option (i.e. a boolean, false by default) on the Hot Rod clients to
> trust all certs.
>
> This means that a Hot Rod client would need to be configured as follows:
>
> ConfigurationBuilder clientBuilder = new ConfigurationBuilder();
> clientBuilder
>    .security()
>      .authentication()
>        .username("user").realm("realm").password("password")
>      .ssl()
>        .enable().trustAll(true);
>
> without having to manually set up callback handlers or trustmanagers.
>
> I don't think this would affect the user experience too much.
>
> Tristan
>
> [1] https://github.com/infinispan/infinispan/pull/5036
>
> On 30/03/2017 14:25, Tristan Tarrant wrote:
> > Dear all,
> >
> > after a mini chat on IRC, I wanted to bring this to everybody's
> attention.
> >
> > We should make the Hot Rod endpoint require authentication in the
> > out-of-the-box configuration.
> > The proposal is to enable the PLAIN (or, preferably, DIGEST) SASL
> > mechanism against the ApplicationRealm and require users to run the
> > add-user script.
> > This would achieve two goals:
> > - secure out-of-the-box configuration, which is always a good idea
> > - access to the "protected" schema and script caches which is prevented
> > when not on loopback on non-authenticated endpoints.
> >
> > Tristan
>
> --
> Tristan Tarrant
> Infinispan Lead
> JBoss, a division of Red Hat
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/infinispan-dev/attachments/20170330/6a3b9b8d/attachment-0001.html 

From ttarrant at redhat.com  Thu Mar 30 09:29:18 2017
From: ttarrant at redhat.com (Tristan Tarrant)
Date: Thu, 30 Mar 2017 15:29:18 +0200
Subject: [infinispan-dev] Hot Rod secured by default
In-Reply-To: <CAA9R9O7S7frPE-sS60Z2E6jV_+OdL18+dvMDFJwQsd-FQfQt7w@mail.gmail.com>
References: <f3cee023-96a1-3c7d-a3ce-b575cd5b0f7e@redhat.com>
	<CAA9R9O7S7frPE-sS60Z2E6jV_+OdL18+dvMDFJwQsd-FQfQt7w@mail.gmail.com>
Message-ID: <df6c29dc-4d48-1dc7-6045-2a14e0dd01d8@redhat.com>

While the "unsecure" over loopback is quite tempting, I would prefer to 
have homogeneous behaviour with the possibility to disable security 
altogether for quick demos.
Otherwise a developer would need to code differently for the local use 
case than for the remote one, causing more confusion.

Tristan

On 30/03/2017 14:54, Sebastian Laskawiec wrote:
> I agree the security out of the box is good. But at the same time we 
> don't want to make Infinispan harder to use for new developers. Out of 
> the box configuration should be "good enough" to start hacking.
> 
> I would propose to make all the endpoints unprotected (with 
> authentication disabled) on localhost/loopback and protected when 
> calling from the outside world.
> 
> On Thu, Mar 30, 2017 at 2:39 PM Tristan Tarrant <ttarrant at redhat.com 
> <mailto:ttarrant at redhat.com>> wrote:
> 
>     Dear all,
> 
>     after a mini chat on IRC, I wanted to bring this to everybody's
>     attention.
> 
>     We should make the Hot Rod endpoint require authentication in the
>     out-of-the-box configuration.
>     The proposal is to enable the PLAIN (or, preferably, DIGEST) SASL
>     mechanism against the ApplicationRealm and require users to run the
>     add-user script.
>     This would achieve two goals:
>     - secure out-of-the-box configuration, which is always a good idea
>     - access to the "protected" schema and script caches which is prevented
>     when not on loopback on non-authenticated endpoints.
> 
>     Tristan
>     --
>     Tristan Tarrant
>     Infinispan Lead
>     JBoss, a division of Red Hat
>     _______________________________________________
>     infinispan-dev mailing list
>     infinispan-dev at lists.jboss.org <mailto:infinispan-dev at lists.jboss.org>
>     https://lists.jboss.org/mailman/listinfo/infinispan-dev
> 
> 
> 
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> 

-- 
Tristan Tarrant
Infinispan Lead
JBoss, a division of Red Hat

From rareddy at redhat.com  Thu Mar 30 10:30:16 2017
From: rareddy at redhat.com (Ramesh Reddy)
Date: Thu, 30 Mar 2017 10:30:16 -0400 (EDT)
Subject: [infinispan-dev] Executing server tasks that contain POJOs
In-Reply-To: <7383B4E7-B386-41CE-A62E-8DFBD439C48D@redhat.com>
References: <7383B4E7-B386-41CE-A62E-8DFBD439C48D@redhat.com>
Message-ID: <1054043886.8626849.1490884216683.JavaMail.zimbra@redhat.com>

Glader,

FWIW, I am rewriting Teiid translator to Infinispan, where I needed a portable marshaller that is simply based on .proto file as you describe issues with your step #2. The use of predefined custom java marshaller is not viable in my usecase, as I want to dynamically convert Relational Tables defined in the Teiid into POJO's in Infinispan, and ability to query them. I have written a such marshaller for my usecase you can see testcase at [1]. Basically I capture the metadata from .proto file and use that information in encoding/decoding the protostream, *most* of the needed code is already there in the ProtoStream libraries. 

BTW, with your Task example you gave me another idea how I can even further enhance the this integration layer in terms of updating multiple POJOs in single call :)

Ramesh..

[1] https://github.com/rareddy/infinispan/blob/master/translator-infinispan-hotrod/src/test/java/org/teiid/translator/infinispan/hotrod/TestTeiidTableMarsheller.java


----- Original Message -----
> Hi all,
> 
> For a demo I'm giving next week, I'd like to show how to use distributed
> streams via a remote server task. All server tasks that we have in testsuite
> rely on primitives but in my case I wanted to use POJOs.
> 
> To do that, I needed to get compatibility mode working in such way that those
> POJOs could be unmarshalled for the server task. Since in another demo I'm
> showing Protostream based POJOs, I thought I'd try to use that as mechanism
> to unmarshall POJOs server side.
> 
> We have a test for such scenario [1], but the reality (running on a proper
> server) is anything that simple. Here's a list of things I've found out
> while creating a WordCount example that relies on a POJO:
> 
> 1. Out of the box, it's impossible to set compatibility marshaller to
> org.infinispan.query.remote.CompatibilityProtoStreamMarshaller [1] because
> "org.infinispan.main" classloader can't access that class. I worked around
> that by tweaking the module.xml to have an optional dependency to
> "org.infinispan.remote-query.server" module.
> 
> 2. After doing that, I had to register the protofile and associated classes
> remotely in the server. Again, there's no out of the box mechanism for that,
> so I created a remote server task that would do that [3].
> 
> 3. Finally, with all that in place, I was able to complete the WordCount test
> [4] with a final caveat: the return of the word count, and words protofile
> registration, tasks return objects that are not marshalled by the
> compatibility marshaller, so I had to make sure that the remote cache
> manager used for those tasks uses the default marshaller.
> 
> Clearly we need to improve on this, and we have plans to address these issues
> (with new upcoming transcoding capabilities), but I thought it'd be worth
> mentioning the problems found in case anyone else encounters them before
> transcoding is in place.
> 
> Cheers,
> 
> [1]
> https://github.com/galderz/datagrid-patterns/blob/master/server-config/domain/domain.xml#L139
> [2]
> https://github.com/galderz/datagrid-patterns/blob/master/server-config/org.infinispan.main_module.xml#L18
> [3]
> https://github.com/galderz/datagrid-patterns/blob/master/analytics-stream/tasks-server/src/main/java/test/WordsProtoTask.java
> [4]
> https://github.com/galderz/datagrid-patterns/blob/master/analytics-stream/tasks-client/src/test/java/test/WordCountTest.java
> --
> Galder Zamarre?o
> Infinispan, Red Hat
> 
> 
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev


From galder at redhat.com  Thu Mar 30 11:12:52 2017
From: galder at redhat.com (=?utf-8?Q?Galder_Zamarre=C3=B1o?=)
Date: Thu, 30 Mar 2017 17:12:52 +0200
Subject: [infinispan-dev] Executing server tasks that contain POJOs
In-Reply-To: <1054043886.8626849.1490884216683.JavaMail.zimbra@redhat.com>
References: <7383B4E7-B386-41CE-A62E-8DFBD439C48D@redhat.com>
	<1054043886.8626849.1490884216683.JavaMail.zimbra@redhat.com>
Message-ID: <F15D147B-4848-4BA8-B94F-E052A805E037@redhat.com>

Hey Ramesh,

I don't know your use case very well, so allow me to ask you some qs:

1. When does your marshaller come into play? At the compatibility layer? Or is it used as a client marshaller?

1a. If it's at the compatibility layer, why can't you use CompatibilityProtoStreamMarshaller?

Your description below makes it sound like your marshaller does similar work to CompatibilityProtoStreamMarshaller, hence the questions and see whether it could fit your use case.

Cheers,
--
Galder Zamarre?o
Infinispan, Red Hat

> On 30 Mar 2017, at 16:30, Ramesh Reddy <rareddy at redhat.com> wrote:
> 
> Glader,
> 
> FWIW, I am rewriting Teiid translator to Infinispan, where I needed a portable marshaller that is simply based on .proto file as you describe issues with your step #2. The use of predefined custom java marshaller is not viable in my usecase, as I want to dynamically convert Relational Tables defined in the Teiid into POJO's in Infinispan, and ability to query them. I have written a such marshaller for my usecase you can see testcase at [1]. Basically I capture the metadata from .proto file and use that information in encoding/decoding the protostream, *most* of the needed code is already there in the ProtoStream libraries. 
> 
> BTW, with your Task example you gave me another idea how I can even further enhance the this integration layer in terms of updating multiple POJOs in single call :)
> 
> Ramesh..
> 
> [1] https://github.com/rareddy/infinispan/blob/master/translator-infinispan-hotrod/src/test/java/org/teiid/translator/infinispan/hotrod/TestTeiidTableMarsheller.java
> 
> 
> ----- Original Message -----
>> Hi all,
>> 
>> For a demo I'm giving next week, I'd like to show how to use distributed
>> streams via a remote server task. All server tasks that we have in testsuite
>> rely on primitives but in my case I wanted to use POJOs.
>> 
>> To do that, I needed to get compatibility mode working in such way that those
>> POJOs could be unmarshalled for the server task. Since in another demo I'm
>> showing Protostream based POJOs, I thought I'd try to use that as mechanism
>> to unmarshall POJOs server side.
>> 
>> We have a test for such scenario [1], but the reality (running on a proper
>> server) is anything that simple. Here's a list of things I've found out
>> while creating a WordCount example that relies on a POJO:
>> 
>> 1. Out of the box, it's impossible to set compatibility marshaller to
>> org.infinispan.query.remote.CompatibilityProtoStreamMarshaller [1] because
>> "org.infinispan.main" classloader can't access that class. I worked around
>> that by tweaking the module.xml to have an optional dependency to
>> "org.infinispan.remote-query.server" module.
>> 
>> 2. After doing that, I had to register the protofile and associated classes
>> remotely in the server. Again, there's no out of the box mechanism for that,
>> so I created a remote server task that would do that [3].
>> 
>> 3. Finally, with all that in place, I was able to complete the WordCount test
>> [4] with a final caveat: the return of the word count, and words protofile
>> registration, tasks return objects that are not marshalled by the
>> compatibility marshaller, so I had to make sure that the remote cache
>> manager used for those tasks uses the default marshaller.
>> 
>> Clearly we need to improve on this, and we have plans to address these issues
>> (with new upcoming transcoding capabilities), but I thought it'd be worth
>> mentioning the problems found in case anyone else encounters them before
>> transcoding is in place.
>> 
>> Cheers,
>> 
>> [1]
>> https://github.com/galderz/datagrid-patterns/blob/master/server-config/domain/domain.xml#L139
>> [2]
>> https://github.com/galderz/datagrid-patterns/blob/master/server-config/org.infinispan.main_module.xml#L18
>> [3]
>> https://github.com/galderz/datagrid-patterns/blob/master/analytics-stream/tasks-server/src/main/java/test/WordsProtoTask.java
>> [4]
>> https://github.com/galderz/datagrid-patterns/blob/master/analytics-stream/tasks-client/src/test/java/test/WordCountTest.java
>> --
>> Galder Zamarre?o
>> Infinispan, Red Hat
>> 
>> 
>> _______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> 
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev



From gustavo at infinispan.org  Thu Mar 30 11:15:53 2017
From: gustavo at infinispan.org (Gustavo Fernandes)
Date: Thu, 30 Mar 2017 16:15:53 +0100
Subject: [infinispan-dev] Executing server tasks that contain POJOs
In-Reply-To: <7383B4E7-B386-41CE-A62E-8DFBD439C48D@redhat.com>
References: <7383B4E7-B386-41CE-A62E-8DFBD439C48D@redhat.com>
Message-ID: <CAH8Ud1FxCfqxC6zDwRDWGWhZ=ZbVuqd1uVTw349TsLKegBLUUQ@mail.gmail.com>

On Thu, Mar 30, 2017 at 1:51 PM, Galder Zamarre?o <galder at redhat.com> wrote:

> Hi all,
>
> For a demo I'm giving next week, I'd like to show how to use distributed
> streams via a remote server task. All server tasks that we have in
> testsuite rely on primitives but in my case I wanted to use POJOs.
>
> To do that, I needed to get compatibility mode working in such way that
> those POJOs could be unmarshalled for the server task. Since in another
> demo I'm showing Protostream based POJOs, I thought I'd try to use that as
> mechanism to unmarshall POJOs server side.
>
> We have a test for such scenario [1], but the reality (running on a proper
> server) is anything that simple. Here's a list of things I've found out
> while creating a WordCount example that relies on a POJO:
>
> 1. Out of the box, it's impossible to set compatibility marshaller to
> org.infinispan.query.remote.CompatibilityProtoStreamMarshaller [1]
> because "org.infinispan.main" classloader can't access that class. I worked
> around that by tweaking the module.xml to have an optional dependency to
> "org.infinispan.remote-query.server" module.
>
> 2. After doing that, I had to register the protofile and associated
> classes remotely in the server. Again, there's no out of the box mechanism
> for that, so I created a remote server task that would do that [3].
>


AFAICT, you should be able to do that doing a PUT in the Protobuf_Metadata
cache, which entails having auth enabled. This cache should be REPL_SYNC,
so no need to run a server task.



>
> 3. Finally, with all that in place, I was able to complete the WordCount
> test [4] with a final caveat: the return of the word count, and words
> protofile registration, tasks return objects that are not marshalled by the
> compatibility marshaller, so I had to make sure that the remote cache
> manager used for those tasks uses the default marshaller.
>
> Clearly we need to improve on this, and we have plans to address these
> issues (with new upcoming transcoding capabilities), but I thought it'd be
> worth mentioning the problems found in case anyone else encounters them
> before transcoding is in place.
>
> Cheers,
>
> [1] https://github.com/galderz/datagrid-patterns/blob/master/
> server-config/domain/domain.xml#L139
> [2] https://github.com/galderz/datagrid-patterns/blob/master/
> server-config/org.infinispan.main_module.xml#L18
> [3] https://github.com/galderz/datagrid-patterns/blob/master/
> analytics-stream/tasks-server/src/main/java/test/WordsProtoTask.java
> [4] https://github.com/galderz/datagrid-patterns/blob/master/
> analytics-stream/tasks-client/src/test/java/test/WordCountTest.java
> --
> Galder Zamarre?o
> Infinispan, Red Hat
>
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/infinispan-dev/attachments/20170330/a77523c6/attachment.html 

From dereed at redhat.com  Thu Mar 30 11:31:44 2017
From: dereed at redhat.com (Dennis Reed)
Date: Thu, 30 Mar 2017 11:31:44 -0400
Subject: [infinispan-dev] Hot Rod secured by default
In-Reply-To: <df6c29dc-4d48-1dc7-6045-2a14e0dd01d8@redhat.com>
References: <f3cee023-96a1-3c7d-a3ce-b575cd5b0f7e@redhat.com>
	<CAA9R9O7S7frPE-sS60Z2E6jV_+OdL18+dvMDFJwQsd-FQfQt7w@mail.gmail.com>
	<df6c29dc-4d48-1dc7-6045-2a14e0dd01d8@redhat.com>
Message-ID: <06b88c80-433f-3745-2112-1424bcd9c342@redhat.com>

+1 to authentication and encryption by default.
  This is 2017, that's how *everything* should be configured.

-1 to making it easy to trust all certs.  That negates the point of
using encryption in the first place and should really never be done.

If it's too hard to configure the correct way that we think it would
turn users away, that's a usability problem that needs to be fixed.

-Dennis


On 03/30/2017 09:29 AM, Tristan Tarrant wrote:
> While the "unsecure" over loopback is quite tempting, I would prefer to 
> have homogeneous behaviour with the possibility to disable security 
> altogether for quick demos.
> Otherwise a developer would need to code differently for the local use 
> case than for the remote one, causing more confusion.
> 
> Tristan
> 
> On 30/03/2017 14:54, Sebastian Laskawiec wrote:
>> I agree the security out of the box is good. But at the same time we 
>> don't want to make Infinispan harder to use for new developers. Out of 
>> the box configuration should be "good enough" to start hacking.
>>
>> I would propose to make all the endpoints unprotected (with 
>> authentication disabled) on localhost/loopback and protected when 
>> calling from the outside world.
>>
>> On Thu, Mar 30, 2017 at 2:39 PM Tristan Tarrant <ttarrant at redhat.com 
>> <mailto:ttarrant at redhat.com>> wrote:
>>
>>     Dear all,
>>
>>     after a mini chat on IRC, I wanted to bring this to everybody's
>>     attention.
>>
>>     We should make the Hot Rod endpoint require authentication in the
>>     out-of-the-box configuration.
>>     The proposal is to enable the PLAIN (or, preferably, DIGEST) SASL
>>     mechanism against the ApplicationRealm and require users to run the
>>     add-user script.
>>     This would achieve two goals:
>>     - secure out-of-the-box configuration, which is always a good idea
>>     - access to the "protected" schema and script caches which is prevented
>>     when not on loopback on non-authenticated endpoints.
>>
>>     Tristan
>>     --
>>     Tristan Tarrant
>>     Infinispan Lead
>>     JBoss, a division of Red Hat
>>     _______________________________________________
>>     infinispan-dev mailing list
>>     infinispan-dev at lists.jboss.org <mailto:infinispan-dev at lists.jboss.org>
>>     https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>
>>
>>
>> _______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>
> 

From rareddy at redhat.com  Thu Mar 30 11:38:12 2017
From: rareddy at redhat.com (Ramesh Reddy)
Date: Thu, 30 Mar 2017 11:38:12 -0400 (EDT)
Subject: [infinispan-dev] Executing server tasks that contain POJOs
In-Reply-To: <F15D147B-4848-4BA8-B94F-E052A805E037@redhat.com>
References: <7383B4E7-B386-41CE-A62E-8DFBD439C48D@redhat.com>
	<1054043886.8626849.1490884216683.JavaMail.zimbra@redhat.com>
	<F15D147B-4848-4BA8-B94F-E052A805E037@redhat.com>
Message-ID: <1761511125.8654937.1490888292278.JavaMail.zimbra@redhat.com>

Yes, it is at the client layer. When I write the server tasks to I will look into CompatibilityProtoStreamMarshaller.


----- Original Message -----
> Hey Ramesh,
> 
> I don't know your use case very well, so allow me to ask you some qs:
> 
> 1. When does your marshaller come into play? At the compatibility layer? Or
> is it used as a client marshaller?
> 
> 1a. If it's at the compatibility layer, why can't you use
> CompatibilityProtoStreamMarshaller?
> 
> Your description below makes it sound like your marshaller does similar work
> to CompatibilityProtoStreamMarshaller, hence the questions and see whether
> it could fit your use case.
> 
> Cheers,
> --
> Galder Zamarre?o
> Infinispan, Red Hat
> 
> > On 30 Mar 2017, at 16:30, Ramesh Reddy <rareddy at redhat.com> wrote:
> > 
> > Glader,
> > 
> > FWIW, I am rewriting Teiid translator to Infinispan, where I needed a
> > portable marshaller that is simply based on .proto file as you describe
> > issues with your step #2. The use of predefined custom java marshaller is
> > not viable in my usecase, as I want to dynamically convert Relational
> > Tables defined in the Teiid into POJO's in Infinispan, and ability to
> > query them. I have written a such marshaller for my usecase you can see
> > testcase at [1]. Basically I capture the metadata from .proto file and use
> > that information in encoding/decoding the protostream, *most* of the
> > needed code is already there in the ProtoStream libraries.
> > 
> > BTW, with your Task example you gave me another idea how I can even further
> > enhance the this integration layer in terms of updating multiple POJOs in
> > single call :)
> > 
> > Ramesh..
> > 
> > [1]
> > https://github.com/rareddy/infinispan/blob/master/translator-infinispan-hotrod/src/test/java/org/teiid/translator/infinispan/hotrod/TestTeiidTableMarsheller.java
> > 
> > 
> > ----- Original Message -----
> >> Hi all,
> >> 
> >> For a demo I'm giving next week, I'd like to show how to use distributed
> >> streams via a remote server task. All server tasks that we have in
> >> testsuite
> >> rely on primitives but in my case I wanted to use POJOs.
> >> 
> >> To do that, I needed to get compatibility mode working in such way that
> >> those
> >> POJOs could be unmarshalled for the server task. Since in another demo I'm
> >> showing Protostream based POJOs, I thought I'd try to use that as
> >> mechanism
> >> to unmarshall POJOs server side.
> >> 
> >> We have a test for such scenario [1], but the reality (running on a proper
> >> server) is anything that simple. Here's a list of things I've found out
> >> while creating a WordCount example that relies on a POJO:
> >> 
> >> 1. Out of the box, it's impossible to set compatibility marshaller to
> >> org.infinispan.query.remote.CompatibilityProtoStreamMarshaller [1] because
> >> "org.infinispan.main" classloader can't access that class. I worked around
> >> that by tweaking the module.xml to have an optional dependency to
> >> "org.infinispan.remote-query.server" module.
> >> 
> >> 2. After doing that, I had to register the protofile and associated
> >> classes
> >> remotely in the server. Again, there's no out of the box mechanism for
> >> that,
> >> so I created a remote server task that would do that [3].
> >> 
> >> 3. Finally, with all that in place, I was able to complete the WordCount
> >> test
> >> [4] with a final caveat: the return of the word count, and words protofile
> >> registration, tasks return objects that are not marshalled by the
> >> compatibility marshaller, so I had to make sure that the remote cache
> >> manager used for those tasks uses the default marshaller.
> >> 
> >> Clearly we need to improve on this, and we have plans to address these
> >> issues
> >> (with new upcoming transcoding capabilities), but I thought it'd be worth
> >> mentioning the problems found in case anyone else encounters them before
> >> transcoding is in place.
> >> 
> >> Cheers,
> >> 
> >> [1]
> >> https://github.com/galderz/datagrid-patterns/blob/master/server-config/domain/domain.xml#L139
> >> [2]
> >> https://github.com/galderz/datagrid-patterns/blob/master/server-config/org.infinispan.main_module.xml#L18
> >> [3]
> >> https://github.com/galderz/datagrid-patterns/blob/master/analytics-stream/tasks-server/src/main/java/test/WordsProtoTask.java
> >> [4]
> >> https://github.com/galderz/datagrid-patterns/blob/master/analytics-stream/tasks-client/src/test/java/test/WordCountTest.java
> >> --
> >> Galder Zamarre?o
> >> Infinispan, Red Hat
> >> 
> >> 
> >> _______________________________________________
> >> infinispan-dev mailing list
> >> infinispan-dev at lists.jboss.org
> >> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> > 
> > _______________________________________________
> > infinispan-dev mailing list
> > infinispan-dev at lists.jboss.org
> > https://lists.jboss.org/mailman/listinfo/infinispan-dev
> 
> 
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev


From rvansa at redhat.com  Thu Mar 30 11:47:58 2017
From: rvansa at redhat.com (Radim Vansa)
Date: Thu, 30 Mar 2017 17:47:58 +0200
Subject: [infinispan-dev] Strategy to adopting Optional in APIs
Message-ID: <7ecd428c-6c32-2907-f0ef-721a1e6c4a5f@redhat.com>

Hi,

I was wondering what's the common attitude towards using Optional in 
APIs, and what naming pattern should we use. As an example, I dislike 
calling

if (entry.getMetadata() != null && entry.getMetadata().version() != null) {
     foo.use(entry.getMetadata().version())
}

where I could just do

entry.metadata().flatMap(Metadata::optionalVersion).ifPresent(foo::use)

Here I have proposed metadata() method returning Optional<Metadata> 
(regular getter method is called getMetadata()) and annoying 
optionalVersion() as version() is the regular getter.

Shall we adopt some common stance (use/don't use/use at developer's 
discretion) and naming conventions? Is it acceptable to start adding

default Optional<Foo> foo() { Optional.ofNullable(getFoo()); }

whenever we feel the urge to chain Optionals?

Radim

-- 
Radim Vansa <rvansa at redhat.com>
JBoss Performance Team


From galder at redhat.com  Thu Mar 30 11:49:24 2017
From: galder at redhat.com (=?utf-8?Q?Galder_Zamarre=C3=B1o?=)
Date: Thu, 30 Mar 2017 17:49:24 +0200
Subject: [infinispan-dev] Executing server tasks that contain POJOs
In-Reply-To: <CAH8Ud1FxCfqxC6zDwRDWGWhZ=ZbVuqd1uVTw349TsLKegBLUUQ@mail.gmail.com>
References: <7383B4E7-B386-41CE-A62E-8DFBD439C48D@redhat.com>
	<CAH8Ud1FxCfqxC6zDwRDWGWhZ=ZbVuqd1uVTw349TsLKegBLUUQ@mail.gmail.com>
Message-ID: <9E4183B3-7447-409A-BA79-45A04BBB986A@redhat.com>


--
Galder Zamarre?o
Infinispan, Red Hat

> On 30 Mar 2017, at 17:15, Gustavo Fernandes <gustavo at infinispan.org> wrote:
> 
> 
> 
> On Thu, Mar 30, 2017 at 1:51 PM, Galder Zamarre?o <galder at redhat.com> wrote:
> Hi all,
> 
> For a demo I'm giving next week, I'd like to show how to use distributed streams via a remote server task. All server tasks that we have in testsuite rely on primitives but in my case I wanted to use POJOs.
> 
> To do that, I needed to get compatibility mode working in such way that those POJOs could be unmarshalled for the server task. Since in another demo I'm showing Protostream based POJOs, I thought I'd try to use that as mechanism to unmarshall POJOs server side.
> 
> We have a test for such scenario [1], but the reality (running on a proper server) is anything that simple. Here's a list of things I've found out while creating a WordCount example that relies on a POJO:
> 
> 1. Out of the box, it's impossible to set compatibility marshaller to org.infinispan.query.remote.CompatibilityProtoStreamMarshaller [1] because "org.infinispan.main" classloader can't access that class. I worked around that by tweaking the module.xml to have an optional dependency to "org.infinispan.remote-query.server" module.
> 
> 2. After doing that, I had to register the protofile and associated classes remotely in the server. Again, there's no out of the box mechanism for that, so I created a remote server task that would do that [3].
> 
> 
> AFAICT, you should be able to do that doing a PUT in the Protobuf_Metadata cache, which entails having auth enabled. This cache should be REPL_SYNC, so no need to run a server task.

Good point but not so sure it completely removes the need for the task. The task does two things:

1. Call ProtobufMetadataManager.registerProtofile, which as you say could be swapped with a cache.put on the metadata cache.

2. Call ProtobufMetadataManager.registerMarshaller. This goes deep into updating SerializationContextImpl, which seems independent of any replicated cache.

In fact, I had originally set up the task to execute in only in one node, but when I did that I found that marshallers were not registered in all nodes, so I had to execute the task in all nodes.

I guess the task could be just limited to only executing 2.) in all nodes (and store the protofile contents by accessing the cache remotely), but I can't see how I can avoid the task altogether right now.

> 
>  
> 
> 3. Finally, with all that in place, I was able to complete the WordCount test [4] with a final caveat: the return of the word count, and words protofile registration, tasks return objects that are not marshalled by the compatibility marshaller, so I had to make sure that the remote cache manager used for those tasks uses the default marshaller.
> 
> Clearly we need to improve on this, and we have plans to address these issues (with new upcoming transcoding capabilities), but I thought it'd be worth mentioning the problems found in case anyone else encounters them before transcoding is in place.
> 
> Cheers,
> 
> [1] https://github.com/galderz/datagrid-patterns/blob/master/server-config/domain/domain.xml#L139
> [2] https://github.com/galderz/datagrid-patterns/blob/master/server-config/org.infinispan.main_module.xml#L18
> [3] https://github.com/galderz/datagrid-patterns/blob/master/analytics-stream/tasks-server/src/main/java/test/WordsProtoTask.java
> [4] https://github.com/galderz/datagrid-patterns/blob/master/analytics-stream/tasks-client/src/test/java/test/WordCountTest.java
> --
> Galder Zamarre?o
> Infinispan, Red Hat
> 
> 
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> 
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev



From gustavo at infinispan.org  Thu Mar 30 11:57:28 2017
From: gustavo at infinispan.org (Gustavo Fernandes)
Date: Thu, 30 Mar 2017 16:57:28 +0100
Subject: [infinispan-dev] Hot Rod secured by default
In-Reply-To: <06b88c80-433f-3745-2112-1424bcd9c342@redhat.com>
References: <f3cee023-96a1-3c7d-a3ce-b575cd5b0f7e@redhat.com>
	<CAA9R9O7S7frPE-sS60Z2E6jV_+OdL18+dvMDFJwQsd-FQfQt7w@mail.gmail.com>
	<df6c29dc-4d48-1dc7-6045-2a14e0dd01d8@redhat.com>
	<06b88c80-433f-3745-2112-1424bcd9c342@redhat.com>
Message-ID: <CAH8Ud1FbCEHJmBUHcveQszhhTHpYeTUCQ=b0UKdXM-tcO5DjPQ@mail.gmail.com>

On Thu, Mar 30, 2017 at 4:31 PM, Dennis Reed <dereed at redhat.com> wrote:

> +1 to authentication and encryption by default.
>   This is 2017, that's how *everything* should be configured.
>

Agree, and SSL can always be turned on if needed.
But enabling it by default and forcing upfront handling of certificates,
keystores and trustores is overkill IMO.


>
> -1 to making it easy to trust all certs.  That negates the point of
> using encryption in the first place and should really never be done.
>
> If it's too hard to configure the correct way that we think it would
> turn users away, that's a usability problem that needs to be fixed.
>
> -Dennis
>
>
> On 03/30/2017 09:29 AM, Tristan Tarrant wrote:
> > While the "unsecure" over loopback is quite tempting, I would prefer to
> > have homogeneous behaviour with the possibility to disable security
> > altogether for quick demos.
> > Otherwise a developer would need to code differently for the local use
> > case than for the remote one, causing more confusion.
> >
> > Tristan
> >
> > On 30/03/2017 14:54, Sebastian Laskawiec wrote:
> >> I agree the security out of the box is good. But at the same time we
> >> don't want to make Infinispan harder to use for new developers. Out of
> >> the box configuration should be "good enough" to start hacking.
> >>
> >> I would propose to make all the endpoints unprotected (with
> >> authentication disabled) on localhost/loopback and protected when
> >> calling from the outside world.
> >>
> >> On Thu, Mar 30, 2017 at 2:39 PM Tristan Tarrant <ttarrant at redhat.com
> >> <mailto:ttarrant at redhat.com>> wrote:
> >>
> >>     Dear all,
> >>
> >>     after a mini chat on IRC, I wanted to bring this to everybody's
> >>     attention.
> >>
> >>     We should make the Hot Rod endpoint require authentication in the
> >>     out-of-the-box configuration.
> >>     The proposal is to enable the PLAIN (or, preferably, DIGEST) SASL
> >>     mechanism against the ApplicationRealm and require users to run the
> >>     add-user script.
> >>     This would achieve two goals:
> >>     - secure out-of-the-box configuration, which is always a good idea
> >>     - access to the "protected" schema and script caches which is
> prevented
> >>     when not on loopback on non-authenticated endpoints.
> >>
> >>     Tristan
> >>     --
> >>     Tristan Tarrant
> >>     Infinispan Lead
> >>     JBoss, a division of Red Hat
> >>     _______________________________________________
> >>     infinispan-dev mailing list
> >>     infinispan-dev at lists.jboss.org <mailto:infinispan-dev at lists.
> jboss.org>
> >>     https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >>
> >>
> >>
> >> _______________________________________________
> >> infinispan-dev mailing list
> >> infinispan-dev at lists.jboss.org
> >> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >>
> >
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/infinispan-dev/attachments/20170330/0926893c/attachment.html 

From ttarrant at redhat.com  Thu Mar 30 12:00:32 2017
From: ttarrant at redhat.com (Tristan Tarrant)
Date: Thu, 30 Mar 2017 18:00:32 +0200
Subject: [infinispan-dev] Hot Rod secured by default
In-Reply-To: <06b88c80-433f-3745-2112-1424bcd9c342@redhat.com>
References: <f3cee023-96a1-3c7d-a3ce-b575cd5b0f7e@redhat.com>
	<CAA9R9O7S7frPE-sS60Z2E6jV_+OdL18+dvMDFJwQsd-FQfQt7w@mail.gmail.com>
	<df6c29dc-4d48-1dc7-6045-2a14e0dd01d8@redhat.com>
	<06b88c80-433f-3745-2112-1424bcd9c342@redhat.com>
Message-ID: <f8af916f-4b0d-a2db-4a30-341729c9f6bf@redhat.com>



On 30/03/2017 17:31, Dennis Reed wrote:
> +1 to authentication and encryption by default.
>    This is 2017, that's how *everything* should be configured.
> 
> -1 to making it easy to trust all certs.  That negates the point of
> using encryption in the first place and should really never be done.
> 
> If it's too hard to configure the correct way that we think it would
> turn users away, that's a usability problem that needs to be fixed.
Well, none of the databases I know of require you to set up client side 
truststores, so that is already a usability hurdle.

Tristan

-- 
Tristan Tarrant
Infinispan Lead
JBoss, a division of Red Hat

From gustavo at infinispan.org  Thu Mar 30 12:01:10 2017
From: gustavo at infinispan.org (Gustavo Fernandes)
Date: Thu, 30 Mar 2017 17:01:10 +0100
Subject: [infinispan-dev] Executing server tasks that contain POJOs
In-Reply-To: <9E4183B3-7447-409A-BA79-45A04BBB986A@redhat.com>
References: <7383B4E7-B386-41CE-A62E-8DFBD439C48D@redhat.com>
	<CAH8Ud1FxCfqxC6zDwRDWGWhZ=ZbVuqd1uVTw349TsLKegBLUUQ@mail.gmail.com>
	<9E4183B3-7447-409A-BA79-45A04BBB986A@redhat.com>
Message-ID: <CAH8Ud1FSUUURed0a3ZcGKt-_3h=2jyDCnhYfnS8mS2yO=g82SA@mail.gmail.com>

On Thu, Mar 30, 2017 at 4:49 PM, Galder Zamarre?o <galder at redhat.com> wrote:

>
> --
> Galder Zamarre?o
> Infinispan, Red Hat
>
> > On 30 Mar 2017, at 17:15, Gustavo Fernandes <gustavo at infinispan.org>
> wrote:
> >
> >
> >
> > On Thu, Mar 30, 2017 at 1:51 PM, Galder Zamarre?o <galder at redhat.com>
> wrote:
> > Hi all,
> >
> > For a demo I'm giving next week, I'd like to show how to use distributed
> streams via a remote server task. All server tasks that we have in
> testsuite rely on primitives but in my case I wanted to use POJOs.
> >
> > To do that, I needed to get compatibility mode working in such way that
> those POJOs could be unmarshalled for the server task. Since in another
> demo I'm showing Protostream based POJOs, I thought I'd try to use that as
> mechanism to unmarshall POJOs server side.
> >
> > We have a test for such scenario [1], but the reality (running on a
> proper server) is anything that simple. Here's a list of things I've found
> out while creating a WordCount example that relies on a POJO:
> >
> > 1. Out of the box, it's impossible to set compatibility marshaller to
> org.infinispan.query.remote.CompatibilityProtoStreamMarshaller [1]
> because "org.infinispan.main" classloader can't access that class. I worked
> around that by tweaking the module.xml to have an optional dependency to
> "org.infinispan.remote-query.server" module.
> >
> > 2. After doing that, I had to register the protofile and associated
> classes remotely in the server. Again, there's no out of the box mechanism
> for that, so I created a remote server task that would do that [3].
> >
> >
> > AFAICT, you should be able to do that doing a PUT in the
> Protobuf_Metadata cache, which entails having auth enabled. This cache
> should be REPL_SYNC, so no need to run a server task.
>
> Good point but not so sure it completely removes the need for the task.
> The task does two things:
>
> 1. Call ProtobufMetadataManager.registerProtofile, which as you say could
> be swapped with a cache.put on the metadata cache.
>
> 2. Call ProtobufMetadataManager.registerMarshaller. This goes deep into
> updating SerializationContextImpl, which seems independent of any
> replicated cache.
>


AFAICT (again), there is an internal listener or interceptor that upon
metadata cache change, will update the SerCtx on all nodes.

Gustavo


>
> In fact, I had originally set up the task to execute in only in one node,
> but when I did that I found that marshallers were not registered in all
> nodes, so I had to execute the task in all nodes.
>
> I guess the task could be just limited to only executing 2.) in all nodes
> (and store the protofile contents by accessing the cache remotely), but I
> can't see how I can avoid the task altogether right now.
>
> >
> >
> >
> > 3. Finally, with all that in place, I was able to complete the WordCount
> test [4] with a final caveat: the return of the word count, and words
> protofile registration, tasks return objects that are not marshalled by the
> compatibility marshaller, so I had to make sure that the remote cache
> manager used for those tasks uses the default marshaller.
> >
> > Clearly we need to improve on this, and we have plans to address these
> issues (with new upcoming transcoding capabilities), but I thought it'd be
> worth mentioning the problems found in case anyone else encounters them
> before transcoding is in place.
> >
> > Cheers,
> >
> > [1] https://github.com/galderz/datagrid-patterns/blob/master/
> server-config/domain/domain.xml#L139
> > [2] https://github.com/galderz/datagrid-patterns/blob/master/
> server-config/org.infinispan.main_module.xml#L18
> > [3] https://github.com/galderz/datagrid-patterns/blob/master/
> analytics-stream/tasks-server/src/main/java/test/WordsProtoTask.java
> > [4] https://github.com/galderz/datagrid-patterns/blob/master/
> analytics-stream/tasks-client/src/test/java/test/WordCountTest.java
> > --
> > Galder Zamarre?o
> > Infinispan, Red Hat
> >
> >
> > _______________________________________________
> > infinispan-dev mailing list
> > infinispan-dev at lists.jboss.org
> > https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >
> > _______________________________________________
> > infinispan-dev mailing list
> > infinispan-dev at lists.jboss.org
> > https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/infinispan-dev/attachments/20170330/1f3abdda/attachment.html 

From gustavo at infinispan.org  Thu Mar 30 12:03:27 2017
From: gustavo at infinispan.org (Gustavo Fernandes)
Date: Thu, 30 Mar 2017 17:03:27 +0100
Subject: [infinispan-dev] Executing server tasks that contain POJOs
In-Reply-To: <CAH8Ud1FSUUURed0a3ZcGKt-_3h=2jyDCnhYfnS8mS2yO=g82SA@mail.gmail.com>
References: <7383B4E7-B386-41CE-A62E-8DFBD439C48D@redhat.com>
	<CAH8Ud1FxCfqxC6zDwRDWGWhZ=ZbVuqd1uVTw349TsLKegBLUUQ@mail.gmail.com>
	<9E4183B3-7447-409A-BA79-45A04BBB986A@redhat.com>
	<CAH8Ud1FSUUURed0a3ZcGKt-_3h=2jyDCnhYfnS8mS2yO=g82SA@mail.gmail.com>
Message-ID: <CAH8Ud1HtOWdOVk1Hn4TafinSZ+MED7rvTat2z4pg6irei6AY0Q@mail.gmail.com>

On Thu, Mar 30, 2017 at 5:01 PM, Gustavo Fernandes <gustavo at infinispan.org>
wrote:

> On Thu, Mar 30, 2017 at 4:49 PM, Galder Zamarre?o <galder at redhat.com>
> wrote:
>
>>
>> --
>> Galder Zamarre?o
>> Infinispan, Red Hat
>>
>> > On 30 Mar 2017, at 17:15, Gustavo Fernandes <gustavo at infinispan.org>
>> wrote:
>> >
>> >
>> >
>> > On Thu, Mar 30, 2017 at 1:51 PM, Galder Zamarre?o <galder at redhat.com>
>> wrote:
>> > Hi all,
>> >
>> > For a demo I'm giving next week, I'd like to show how to use
>> distributed streams via a remote server task. All server tasks that we have
>> in testsuite rely on primitives but in my case I wanted to use POJOs.
>> >
>> > To do that, I needed to get compatibility mode working in such way that
>> those POJOs could be unmarshalled for the server task. Since in another
>> demo I'm showing Protostream based POJOs, I thought I'd try to use that as
>> mechanism to unmarshall POJOs server side.
>> >
>> > We have a test for such scenario [1], but the reality (running on a
>> proper server) is anything that simple. Here's a list of things I've found
>> out while creating a WordCount example that relies on a POJO:
>> >
>> > 1. Out of the box, it's impossible to set compatibility marshaller to
>> org.infinispan.query.remote.CompatibilityProtoStreamMarshaller [1]
>> because "org.infinispan.main" classloader can't access that class. I worked
>> around that by tweaking the module.xml to have an optional dependency to
>> "org.infinispan.remote-query.server" module.
>> >
>> > 2. After doing that, I had to register the protofile and associated
>> classes remotely in the server. Again, there's no out of the box mechanism
>> for that, so I created a remote server task that would do that [3].
>> >
>> >
>> > AFAICT, you should be able to do that doing a PUT in the
>> Protobuf_Metadata cache, which entails having auth enabled. This cache
>> should be REPL_SYNC, so no need to run a server task.
>>
>> Good point but not so sure it completely removes the need for the task.
>> The task does two things:
>>
>> 1. Call ProtobufMetadataManager.registerProtofile, which as you say
>> could be swapped with a cache.put on the metadata cache.
>>
>> 2. Call ProtobufMetadataManager.registerMarshaller. This goes deep into
>> updating SerializationContextImpl, which seems independent of any
>> replicated cache.
>>
>
>
> AFAICT (again), there is an internal listener or interceptor that upon
> metadata cache change, will update the SerCtx on all nodes.
>
> Gustavo
>


Ops, misread that, you are registering marshallers *in the server*, so yes,
you probably need a task for that :)



>
>
>>
>> In fact, I had originally set up the task to execute in only in one node,
>> but when I did that I found that marshallers were not registered in all
>> nodes, so I had to execute the task in all nodes.
>>
>> I guess the task could be just limited to only executing 2.) in all nodes
>> (and store the protofile contents by accessing the cache remotely), but I
>> can't see how I can avoid the task altogether right now.
>>
>> >
>> >
>> >
>> > 3. Finally, with all that in place, I was able to complete the
>> WordCount test [4] with a final caveat: the return of the word count, and
>> words protofile registration, tasks return objects that are not marshalled
>> by the compatibility marshaller, so I had to make sure that the remote
>> cache manager used for those tasks uses the default marshaller.
>> >
>> > Clearly we need to improve on this, and we have plans to address these
>> issues (with new upcoming transcoding capabilities), but I thought it'd be
>> worth mentioning the problems found in case anyone else encounters them
>> before transcoding is in place.
>> >
>> > Cheers,
>> >
>> > [1] https://github.com/galderz/datagrid-patterns/blob/master/ser
>> ver-config/domain/domain.xml#L139
>> > [2] https://github.com/galderz/datagrid-patterns/blob/master/ser
>> ver-config/org.infinispan.main_module.xml#L18
>> > [3] https://github.com/galderz/datagrid-patterns/blob/master/ana
>> lytics-stream/tasks-server/src/main/java/test/WordsProtoTask.java
>> > [4] https://github.com/galderz/datagrid-patterns/blob/master/ana
>> lytics-stream/tasks-client/src/test/java/test/WordCountTest.java
>> > --
>> > Galder Zamarre?o
>> > Infinispan, Red Hat
>> >
>> >
>> > _______________________________________________
>> > infinispan-dev mailing list
>> > infinispan-dev at lists.jboss.org
>> > https://lists.jboss.org/mailman/listinfo/infinispan-dev
>> >
>> > _______________________________________________
>> > infinispan-dev mailing list
>> > infinispan-dev at lists.jboss.org
>> > https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>
>>
>> _______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/infinispan-dev/attachments/20170330/cef1cee5/attachment-0001.html 

From rvansa at redhat.com  Thu Mar 30 12:04:44 2017
From: rvansa at redhat.com (Radim Vansa)
Date: Thu, 30 Mar 2017 18:04:44 +0200
Subject: [infinispan-dev] Hot Rod secured by default
In-Reply-To: <a50ae5ea-cef6-0af1-e33a-85242b5dcb26@redhat.com>
References: <f3cee023-96a1-3c7d-a3ce-b575cd5b0f7e@redhat.com>
	<a50ae5ea-cef6-0af1-e33a-85242b5dcb26@redhat.com>
Message-ID: <e02c9780-f8cf-76f3-e7ea-f17e773d3bc0@redhat.com>

I don't see an example/quickstart using security on the tutorials page 
[1]. Could you provide one that would show what all would the user have 
to configure in order to do HotRod hello world?

While secure OOTB is good, I like performant OOTB. And it seems to me 
that "I need security" is something they'll probably think about when 
deploying (if they need that), and will look it up, I wouldn't like to 
see "turn off security" in the perf tuning guide. So I'd say -1 to SSL 
by default.

-1 to PLAIN authentication without encryption, that's not a safe 
default. [2]

Radim

[1] http://infinispan.org/tutorials/
[2] http://www.prokofiev.ru/prikol/pics/p12/winfirewall.jpg

On 03/30/2017 02:39 PM, Tristan Tarrant wrote:
> Let me add another item:
>
> combined with Sebastian's PR [1] we could also turn on encryption by
> default using a self-signed certificate. We would also need to have an
> easy option (i.e. a boolean, false by default) on the Hot Rod clients to
> trust all certs.
>
> This means that a Hot Rod client would need to be configured as follows:
>
> ConfigurationBuilder clientBuilder = new ConfigurationBuilder();
> clientBuilder
>     .security()
>       .authentication()
>         .username("user").realm("realm").password("password")
>       .ssl()
>         .enable().trustAll(true);
>
> without having to manually set up callback handlers or trustmanagers.
>
> I don't think this would affect the user experience too much.
>
> Tristan
>
> [1] https://github.com/infinispan/infinispan/pull/5036
>
> On 30/03/2017 14:25, Tristan Tarrant wrote:
>> Dear all,
>>
>> after a mini chat on IRC, I wanted to bring this to everybody's attention.
>>
>> We should make the Hot Rod endpoint require authentication in the
>> out-of-the-box configuration.
>> The proposal is to enable the PLAIN (or, preferably, DIGEST) SASL
>> mechanism against the ApplicationRealm and require users to run the
>> add-user script.
>> This would achieve two goals:
>> - secure out-of-the-box configuration, which is always a good idea
>> - access to the "protected" schema and script caches which is prevented
>> when not on loopback on non-authenticated endpoints.
>>
>> Tristan


-- 
Radim Vansa <rvansa at redhat.com>
JBoss Performance Team


From wfink at redhat.com  Thu Mar 30 12:07:25 2017
From: wfink at redhat.com (Wolf Fink)
Date: Thu, 30 Mar 2017 18:07:25 +0200
Subject: [infinispan-dev] Hot Rod secured by default
In-Reply-To: <06b88c80-433f-3745-2112-1424bcd9c342@redhat.com>
References: <f3cee023-96a1-3c7d-a3ce-b575cd5b0f7e@redhat.com>
	<CAA9R9O7S7frPE-sS60Z2E6jV_+OdL18+dvMDFJwQsd-FQfQt7w@mail.gmail.com>
	<df6c29dc-4d48-1dc7-6045-2a14e0dd01d8@redhat.com>
	<06b88c80-433f-3745-2112-1424bcd9c342@redhat.com>
Message-ID: <CAKwmrdPrJgy_KPyZNHnK40mGrCqG05DyntvC_3R=TKK2P6-AoQ@mail.gmail.com>

+1 to make the default secure.

-1 SSL by default as it makes it slower and I think not most will use it

-1 easy trust all certs, That sounds to me we close one door and make it
possible to open another one


What if we add an example configuration unsecured which can be simple
copied for examples and to start.


On Thu, Mar 30, 2017 at 5:31 PM, Dennis Reed <dereed at redhat.com> wrote:

> +1 to authentication and encryption by default.
>   This is 2017, that's how *everything* should be configured.
>
> -1 to making it easy to trust all certs.  That negates the point of
> using encryption in the first place and should really never be done.
>
> If it's too hard to configure the correct way that we think it would
> turn users away, that's a usability problem that needs to be fixed.
>
> -Dennis
>
>
> On 03/30/2017 09:29 AM, Tristan Tarrant wrote:
> > While the "unsecure" over loopback is quite tempting, I would prefer to
> > have homogeneous behaviour with the possibility to disable security
> > altogether for quick demos.
> > Otherwise a developer would need to code differently for the local use
> > case than for the remote one, causing more confusion.
> >
> > Tristan
> >
> > On 30/03/2017 14:54, Sebastian Laskawiec wrote:
> >> I agree the security out of the box is good. But at the same time we
> >> don't want to make Infinispan harder to use for new developers. Out of
> >> the box configuration should be "good enough" to start hacking.
> >>
> >> I would propose to make all the endpoints unprotected (with
> >> authentication disabled) on localhost/loopback and protected when
> >> calling from the outside world.
> >>
> >> On Thu, Mar 30, 2017 at 2:39 PM Tristan Tarrant <ttarrant at redhat.com
> >> <mailto:ttarrant at redhat.com>> wrote:
> >>
> >>     Dear all,
> >>
> >>     after a mini chat on IRC, I wanted to bring this to everybody's
> >>     attention.
> >>
> >>     We should make the Hot Rod endpoint require authentication in the
> >>     out-of-the-box configuration.
> >>     The proposal is to enable the PLAIN (or, preferably, DIGEST) SASL
> >>     mechanism against the ApplicationRealm and require users to run the
> >>     add-user script.
> >>     This would achieve two goals:
> >>     - secure out-of-the-box configuration, which is always a good idea
> >>     - access to the "protected" schema and script caches which is
> prevented
> >>     when not on loopback on non-authenticated endpoints.
> >>
> >>     Tristan
> >>     --
> >>     Tristan Tarrant
> >>     Infinispan Lead
> >>     JBoss, a division of Red Hat
> >>     _______________________________________________
> >>     infinispan-dev mailing list
> >>     infinispan-dev at lists.jboss.org <mailto:infinispan-dev at lists.
> jboss.org>
> >>     https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >>
> >>
> >>
> >> _______________________________________________
> >> infinispan-dev mailing list
> >> infinispan-dev at lists.jboss.org
> >> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >>
> >
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/infinispan-dev/attachments/20170330/d3f37f92/attachment.html 

From dan.berindei at gmail.com  Thu Mar 30 12:33:59 2017
From: dan.berindei at gmail.com (Dan Berindei)
Date: Thu, 30 Mar 2017 19:33:59 +0300
Subject: [infinispan-dev] Executing server tasks that contain POJOs
In-Reply-To: <7383B4E7-B386-41CE-A62E-8DFBD439C48D@redhat.com>
References: <7383B4E7-B386-41CE-A62E-8DFBD439C48D@redhat.com>
Message-ID: <CA+nfvwR4ctf2QyV1LR-R7KFxzQA+zLO8zQe_2juyeZs5CwheSQ@mail.gmail.com>

On Thu, Mar 30, 2017 at 3:51 PM, Galder Zamarre?o <galder at redhat.com> wrote:
> Hi all,
>
> For a demo I'm giving next week, I'd like to show how to use distributed streams via a remote server task. All server tasks that we have in testsuite rely on primitives but in my case I wanted to use POJOs.
>
> To do that, I needed to get compatibility mode working in such way that those POJOs could be unmarshalled for the server task. Since in another demo I'm showing Protostream based POJOs, I thought I'd try to use that as mechanism to unmarshall POJOs server side.
>
> We have a test for such scenario [1], but the reality (running on a proper server) is anything that simple. Here's a list of things I've found out while creating a WordCount example that relies on a POJO:
>
> 1. Out of the box, it's impossible to set compatibility marshaller to org.infinispan.query.remote.CompatibilityProtoStreamMarshaller [1] because "org.infinispan.main" classloader can't access that class. I worked around that by tweaking the module.xml to have an optional dependency to "org.infinispan.remote-query.server" module.
>

I know Sanne also wanted to add one of the query modules as an
optional dependency to the core module for a similar reason, but it
seems really hacky.

Could the server create the GlobalConfigurationBuilder with a
classloader that has access to the query module instead?

Alternatively, I know ModularClassResolver prefixes class names with
the slot and module name, and can load a class from any module. Maybe
we could also allow a slot:module:class format everywhere the
configuration currently accepts a class name?

> 2. After doing that, I had to register the protofile and associated classes remotely in the server. Again, there's no out of the box mechanism for that, so I created a remote server task that would do that [3].
>
> 3. Finally, with all that in place, I was able to complete the WordCount test [4] with a final caveat: the return of the word count, and words protofile registration, tasks return objects that are not marshalled by the compatibility marshaller, so I had to make sure that the remote cache manager used for those tasks uses the default marshaller.
>
> Clearly we need to improve on this, and we have plans to address these issues (with new upcoming transcoding capabilities), but I thought it'd be worth mentioning the problems found in case anyone else encounters them before transcoding is in place.
>
> Cheers,
>
> [1] https://github.com/galderz/datagrid-patterns/blob/master/server-config/domain/domain.xml#L139
> [2] https://github.com/galderz/datagrid-patterns/blob/master/server-config/org.infinispan.main_module.xml#L18
> [3] https://github.com/galderz/datagrid-patterns/blob/master/analytics-stream/tasks-server/src/main/java/test/WordsProtoTask.java
> [4] https://github.com/galderz/datagrid-patterns/blob/master/analytics-stream/tasks-client/src/test/java/test/WordCountTest.java
> --
> Galder Zamarre?o
> Infinispan, Red Hat
>
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev


From dan.berindei at gmail.com  Thu Mar 30 14:38:50 2017
From: dan.berindei at gmail.com (Dan Berindei)
Date: Thu, 30 Mar 2017 21:38:50 +0300
Subject: [infinispan-dev] Hot Rod secured by default
In-Reply-To: <CAKwmrdPrJgy_KPyZNHnK40mGrCqG05DyntvC_3R=TKK2P6-AoQ@mail.gmail.com>
References: <f3cee023-96a1-3c7d-a3ce-b575cd5b0f7e@redhat.com>
	<CAA9R9O7S7frPE-sS60Z2E6jV_+OdL18+dvMDFJwQsd-FQfQt7w@mail.gmail.com>
	<df6c29dc-4d48-1dc7-6045-2a14e0dd01d8@redhat.com>
	<06b88c80-433f-3745-2112-1424bcd9c342@redhat.com>
	<CAKwmrdPrJgy_KPyZNHnK40mGrCqG05DyntvC_3R=TKK2P6-AoQ@mail.gmail.com>
Message-ID: <CA+nfvwQjSy+oC0gwoEccEMs_E0k0ksD5JsF7rYb2Jdxm3g4mxA@mail.gmail.com>

I agree with Radim, PLAIN authentication without encryption makes it
too easy to sniff the password from another machine.

I have no idea how expensive SSL encryption is in WildFly, but I think
all recent processors have specialized instructions for helping with
encryption, so it may not be that bad.

Even with encryption, if the client trusts all certs, it may be
possible for an attacker to insert itself in the middle and decode
everything -- depending on network topology and what kind of access
the attacker already has. I think it only makes sense to trust all
certs if we also implement something like HPKP [1], to make it more
like ssh.

[1]: https://en.wikipedia.org/wiki/HTTP_Public_Key_Pinning

Cheers
Dan



On Thu, Mar 30, 2017 at 7:07 PM, Wolf Fink <wfink at redhat.com> wrote:
> +1 to make the default secure.
>
> -1 SSL by default as it makes it slower and I think not most will use it
>
> -1 easy trust all certs, That sounds to me we close one door and make it
> possible to open another one
>
>
> What if we add an example configuration unsecured which can be simple copied
> for examples and to start.
>
>
> On Thu, Mar 30, 2017 at 5:31 PM, Dennis Reed <dereed at redhat.com> wrote:
>>
>> +1 to authentication and encryption by default.
>>   This is 2017, that's how *everything* should be configured.
>>
>> -1 to making it easy to trust all certs.  That negates the point of
>> using encryption in the first place and should really never be done.
>>
>> If it's too hard to configure the correct way that we think it would
>> turn users away, that's a usability problem that needs to be fixed.
>>
>> -Dennis
>>
>>
>> On 03/30/2017 09:29 AM, Tristan Tarrant wrote:
>> > While the "unsecure" over loopback is quite tempting, I would prefer to
>> > have homogeneous behaviour with the possibility to disable security
>> > altogether for quick demos.
>> > Otherwise a developer would need to code differently for the local use
>> > case than for the remote one, causing more confusion.
>> >
>> > Tristan
>> >
>> > On 30/03/2017 14:54, Sebastian Laskawiec wrote:
>> >> I agree the security out of the box is good. But at the same time we
>> >> don't want to make Infinispan harder to use for new developers. Out of
>> >> the box configuration should be "good enough" to start hacking.
>> >>
>> >> I would propose to make all the endpoints unprotected (with
>> >> authentication disabled) on localhost/loopback and protected when
>> >> calling from the outside world.
>> >>
>> >> On Thu, Mar 30, 2017 at 2:39 PM Tristan Tarrant <ttarrant at redhat.com
>> >> <mailto:ttarrant at redhat.com>> wrote:
>> >>
>> >>     Dear all,
>> >>
>> >>     after a mini chat on IRC, I wanted to bring this to everybody's
>> >>     attention.
>> >>
>> >>     We should make the Hot Rod endpoint require authentication in the
>> >>     out-of-the-box configuration.
>> >>     The proposal is to enable the PLAIN (or, preferably, DIGEST) SASL
>> >>     mechanism against the ApplicationRealm and require users to run the
>> >>     add-user script.
>> >>     This would achieve two goals:
>> >>     - secure out-of-the-box configuration, which is always a good idea
>> >>     - access to the "protected" schema and script caches which is
>> >> prevented
>> >>     when not on loopback on non-authenticated endpoints.
>> >>
>> >>     Tristan
>> >>     --
>> >>     Tristan Tarrant
>> >>     Infinispan Lead
>> >>     JBoss, a division of Red Hat
>> >>     _______________________________________________
>> >>     infinispan-dev mailing list
>> >>     infinispan-dev at lists.jboss.org
>> >> <mailto:infinispan-dev at lists.jboss.org>
>> >>     https://lists.jboss.org/mailman/listinfo/infinispan-dev
>> >>
>> >>
>> >>
>> >> _______________________________________________
>> >> infinispan-dev mailing list
>> >> infinispan-dev at lists.jboss.org
>> >> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>> >>
>> >
>> _______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
>
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev

From sanne at infinispan.org  Thu Mar 30 17:08:19 2017
From: sanne at infinispan.org (Sanne Grinovero)
Date: Thu, 30 Mar 2017 22:08:19 +0100
Subject: [infinispan-dev] Strategy to adopting Optional in APIs
In-Reply-To: <7ecd428c-6c32-2907-f0ef-721a1e6c4a5f@redhat.com>
References: <7ecd428c-6c32-2907-f0ef-721a1e6c4a5f@redhat.com>
Message-ID: <CAFm4XO3Y5Dmtb_7prAa37g8jCRnSmNCcUVUs6bN6cpz-jAKHEw@mail.gmail.com>

I'm for "at discretion" and "avoid if not really needed" : not cool to
allocate objects for no reason.

On 30 Mar 2017 16:57, "Radim Vansa" <rvansa at redhat.com> wrote:

> Hi,
>
> I was wondering what's the common attitude towards using Optional in
> APIs, and what naming pattern should we use. As an example, I dislike
> calling
>
> if (entry.getMetadata() != null && entry.getMetadata().version() != null) {
>      foo.use(entry.getMetadata().version())
> }
>
> where I could just do
>
> entry.metadata().flatMap(Metadata::optionalVersion).ifPresent(foo::use)
>
> Here I have proposed metadata() method returning Optional<Metadata>
> (regular getter method is called getMetadata()) and annoying
> optionalVersion() as version() is the regular getter.
>
> Shall we adopt some common stance (use/don't use/use at developer's
> discretion) and naming conventions? Is it acceptable to start adding
>
> default Optional<Foo> foo() { Optional.ofNullable(getFoo()); }
>
> whenever we feel the urge to chain Optionals?
>
> Radim
>
> --
> Radim Vansa <rvansa at redhat.com>
> JBoss Performance Team
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/infinispan-dev/attachments/20170330/154e15cd/attachment-0001.html 

From rvansa at redhat.com  Fri Mar 31 03:57:48 2017
From: rvansa at redhat.com (Radim Vansa)
Date: Fri, 31 Mar 2017 09:57:48 +0200
Subject: [infinispan-dev] Strategy to adopting Optional in APIs
In-Reply-To: <CAFm4XO3Y5Dmtb_7prAa37g8jCRnSmNCcUVUs6bN6cpz-jAKHEw@mail.gmail.com>
References: <7ecd428c-6c32-2907-f0ef-721a1e6c4a5f@redhat.com>
	<CAFm4XO3Y5Dmtb_7prAa37g8jCRnSmNCcUVUs6bN6cpz-jAKHEw@mail.gmail.com>
Message-ID: <63ed80a2-7a88-f9ef-5c7b-345433ffc6c2@redhat.com>

I secretly hope that all these allocations would be inlined and 
eliminated. If we find out that it really allocates the objects (from 
JFR's allocation stats), it's a reason to rewrite that piece of code to 
the dull optionless version.
TBH I am rather afraid that the JVM will allocate the consumer which 
will need some captured variables. Maybe I trust C2 compiler too much, 
believing that if the handler isn't too big, it will generate similar 
instructions with nicer source code :-/

R.


On 03/30/2017 11:08 PM, Sanne Grinovero wrote:
> I'm for "at discretion" and "avoid if not really needed" : not cool to 
> allocate objects for no reason.
>
> On 30 Mar 2017 16:57, "Radim Vansa" <rvansa at redhat.com 
> <mailto:rvansa at redhat.com>> wrote:
>
>     Hi,
>
>     I was wondering what's the common attitude towards using Optional in
>     APIs, and what naming pattern should we use. As an example, I dislike
>     calling
>
>     if (entry.getMetadata() != null && entry.getMetadata().version()
>     != null) {
>          foo.use(entry.getMetadata().version())
>     }
>
>     where I could just do
>
>     entry.metadata().flatMap(Metadata::optionalVersion).ifPresent(foo::use)
>
>     Here I have proposed metadata() method returning Optional<Metadata>
>     (regular getter method is called getMetadata()) and annoying
>     optionalVersion() as version() is the regular getter.
>
>     Shall we adopt some common stance (use/don't use/use at developer's
>     discretion) and naming conventions? Is it acceptable to start adding
>
>     default Optional<Foo> foo() { Optional.ofNullable(getFoo()); }
>
>     whenever we feel the urge to chain Optionals?
>
>     Radim
>
>     --
>     Radim Vansa <rvansa at redhat.com <mailto:rvansa at redhat.com>>
>     JBoss Performance Team
>
>     _______________________________________________
>     infinispan-dev mailing list
>     infinispan-dev at lists.jboss.org <mailto:infinispan-dev at lists.jboss.org>
>     https://lists.jboss.org/mailman/listinfo/infinispan-dev
>     <https://lists.jboss.org/mailman/listinfo/infinispan-dev>
>
>
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev


-- 
Radim Vansa <rvansa at redhat.com>
JBoss Performance Team


From ttarrant at redhat.com  Fri Mar 31 03:59:42 2017
From: ttarrant at redhat.com (Tristan Tarrant)
Date: Fri, 31 Mar 2017 09:59:42 +0200
Subject: [infinispan-dev] Strategy to adopting Optional in APIs
In-Reply-To: <63ed80a2-7a88-f9ef-5c7b-345433ffc6c2@redhat.com>
References: <7ecd428c-6c32-2907-f0ef-721a1e6c4a5f@redhat.com>
	<CAFm4XO3Y5Dmtb_7prAa37g8jCRnSmNCcUVUs6bN6cpz-jAKHEw@mail.gmail.com>
	<63ed80a2-7a88-f9ef-5c7b-345433ffc6c2@redhat.com>
Message-ID: <3a302757-662a-68e9-23ea-4345a6bbb787@redhat.com>

I was about to say the same: in the typical use case of returning an 
optional and using it immediately it would probably end up on the stack 
anyway...

Tristan

On 31/03/2017 09:57, Radim Vansa wrote:
> I secretly hope that all these allocations would be inlined and
> eliminated. If we find out that it really allocates the objects (from
> JFR's allocation stats), it's a reason to rewrite that piece of code to
> the dull optionless version.
> TBH I am rather afraid that the JVM will allocate the consumer which
> will need some captured variables. Maybe I trust C2 compiler too much,
> believing that if the handler isn't too big, it will generate similar
> instructions with nicer source code :-/
> 
> R.
> 
> 
> On 03/30/2017 11:08 PM, Sanne Grinovero wrote:
>> I'm for "at discretion" and "avoid if not really needed" : not cool to
>> allocate objects for no reason.
>>
>> On 30 Mar 2017 16:57, "Radim Vansa" <rvansa at redhat.com
>> <mailto:rvansa at redhat.com>> wrote:
>>
>>      Hi,
>>
>>      I was wondering what's the common attitude towards using Optional in
>>      APIs, and what naming pattern should we use. As an example, I dislike
>>      calling
>>
>>      if (entry.getMetadata() != null && entry.getMetadata().version()
>>      != null) {
>>           foo.use(entry.getMetadata().version())
>>      }
>>
>>      where I could just do
>>
>>      entry.metadata().flatMap(Metadata::optionalVersion).ifPresent(foo::use)
>>
>>      Here I have proposed metadata() method returning Optional<Metadata>
>>      (regular getter method is called getMetadata()) and annoying
>>      optionalVersion() as version() is the regular getter.
>>
>>      Shall we adopt some common stance (use/don't use/use at developer's
>>      discretion) and naming conventions? Is it acceptable to start adding
>>
>>      default Optional<Foo> foo() { Optional.ofNullable(getFoo()); }
>>
>>      whenever we feel the urge to chain Optionals?
>>
>>      Radim
>>
>>      --
>>      Radim Vansa <rvansa at redhat.com <mailto:rvansa at redhat.com>>
>>      JBoss Performance Team
>>
>>      _______________________________________________
>>      infinispan-dev mailing list
>>      infinispan-dev at lists.jboss.org <mailto:infinispan-dev at lists.jboss.org>
>>      https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>      <https://lists.jboss.org/mailman/listinfo/infinispan-dev>
>>
>>
>>
>> _______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> 
> 

-- 
Tristan Tarrant
Infinispan Lead
JBoss, a division of Red Hat

From ttarrant at redhat.com  Fri Mar 31 05:09:26 2017
From: ttarrant at redhat.com (Tristan Tarrant)
Date: Fri, 31 Mar 2017 11:09:26 +0200
Subject: [infinispan-dev] Infinispan 9.0 Final
Message-ID: <6319cb13-0359-aa9f-8d92-cb094c822e50@redhat.com>

Dear all,

we are proud to announce Infinispan 9.0 Final.
This release includes many new features and improvements:

- much improved performance in all scenarios
- off-heap data container, to avoid GC pauses
- Ickle, a new query language based on JP-QL with full-text capabilities
- multi-tenancy with SNI support for the server
- vastly improved cloud and container integrations

Read more about it in our announcement [1]
As usual you can find all the downloads, documentation and community 
links on our website: http://infinispan.org

Enjoy !

The Infinispan Team

[1] http://blog.infinispan.org/2017/03/infinispan-9.html

-- 
Tristan Tarrant
Infinispan Lead
JBoss, a division of Red Hat

From gustavo at infinispan.org  Fri Mar 31 05:58:36 2017
From: gustavo at infinispan.org (Gustavo Fernandes)
Date: Fri, 31 Mar 2017 10:58:36 +0100
Subject: [infinispan-dev] Single Endpoint design
In-Reply-To: <CAA9R9O6c8SHB3E9PboofxRuj1LcA5yuvz3EEAUeX5=2Gdr-J0g@mail.gmail.com>
References: <CAA9R9O6c8SHB3E9PboofxRuj1LcA5yuvz3EEAUeX5=2Gdr-J0g@mail.gmail.com>
Message-ID: <CAH8Ud1Fi4DnxT_c-ERy3zpt5QxkeERV0Bc+O0D=jRvSs4YS47A@mail.gmail.com>

Hi Sebastian,

If I understood it correctly, all the Hot Rod clients will be changed from
using:

- Binary over TCP, circa 40 bytes header, no hops to contact the server, no
protocol negotiation, no encryption (default)

to

- HTTP/2 with SSL, protocol upgrade negotiation, and a hop (router) to
connect to the server.


Any idea of how significant would be this extra overhead introduced?


Thanks,
Gustavo


On Thu, Mar 30, 2017 at 2:01 PM, Sebastian Laskawiec <slaskawi at redhat.com>
wrote:

> Hey!
>
> My plan is to start working on a Single Point support for Infinispan
> Server very soon and I prepared a design: https://github.com/
> infinispan/infinispan/pull/5041
>
> As you can see I did not use our Wiki (as we used to) because it doesn't
> support inline comments (which is pretty bad in my opinion). I would like
> to propose to keep all the designs along with our source code. This
> approach has been successfully used by the Kubernetes [1] folks (although
> they migrated designs into the new Community repository [2] recently). I
> think it might be a good idea to do something similar.
>
> Feedback on both items is more than welcome.
>
> Thanks,
> Sebastian
>
> [1] https://github.com/kubernetes/kubernetes/tree/master/docs/proposals
> [2] https://github.com/kubernetes/community/tree/
> master/contributors/design-proposals
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/infinispan-dev/attachments/20170331/6df7ceb2/attachment.html 

From ttarrant at redhat.com  Fri Mar 31 06:02:01 2017
From: ttarrant at redhat.com (Tristan Tarrant)
Date: Fri, 31 Mar 2017 12:02:01 +0200
Subject: [infinispan-dev] Single Endpoint design
In-Reply-To: <CAH8Ud1Fi4DnxT_c-ERy3zpt5QxkeERV0Bc+O0D=jRvSs4YS47A@mail.gmail.com>
References: <CAA9R9O6c8SHB3E9PboofxRuj1LcA5yuvz3EEAUeX5=2Gdr-J0g@mail.gmail.com>
	<CAH8Ud1Fi4DnxT_c-ERy3zpt5QxkeERV0Bc+O0D=jRvSs4YS47A@mail.gmail.com>
Message-ID: <1bcc751c-a246-bcb4-3c43-8bfe0ed252a7@redhat.com>

You understood incorrectly.
The only change to the Hot Rod clients is that, if they get a 400 error 
from a HR PING request, they will initiate an upgrade to Hot Rod and 
then proceed with the usual Hot Rod protocol after that.

Tristan

On 31/03/2017 11:58, Gustavo Fernandes wrote:
> Hi Sebastian,
> 
> If I understood it correctly, all the Hot Rod clients will be changed 
> from using:
> 
> - Binary over TCP, circa 40 bytes header, no hops to contact the server, 
> no protocol negotiation, no encryption (default)
> 
> to
> 
> - HTTP/2 with SSL, protocol upgrade negotiation, and a hop (router) to 
> connect to the server.
> 
> 
> Any idea of how significant would be this extra overhead introduced?
> 
> 
> Thanks,
> Gustavo
> 
> 
> On Thu, Mar 30, 2017 at 2:01 PM, Sebastian Laskawiec 
> <slaskawi at redhat.com <mailto:slaskawi at redhat.com>> wrote:
> 
>     Hey!
> 
>     My plan is to start working on a Single Point support for Infinispan
>     Server very soon and I prepared a design:
>     https://github.com/infinispan/infinispan/pull/5041
>     <https://github.com/infinispan/infinispan/pull/5041>
> 
>     As you can see I did not use our Wiki (as we used to) because it
>     doesn't support inline comments (which is pretty bad in my opinion).
>     I would like to propose to keep all the designs along with our
>     source code. This approach has been successfully used by the
>     Kubernetes [1] folks (although they migrated designs into the new
>     Community repository [2] recently). I think it might be a good idea
>     to do something similar.
> 
>     Feedback on both items is more than welcome.
> 
>     Thanks,
>     Sebastian
> 
>     [1]
>     https://github.com/kubernetes/kubernetes/tree/master/docs/proposals
>     <https://github.com/kubernetes/kubernetes/tree/master/docs/proposals>
>     [2]
>     https://github.com/kubernetes/community/tree/master/contributors/design-proposals
>     <https://github.com/kubernetes/community/tree/master/contributors/design-proposals>
> 
>     _______________________________________________
>     infinispan-dev mailing list
>     infinispan-dev at lists.jboss.org <mailto:infinispan-dev at lists.jboss.org>
>     https://lists.jboss.org/mailman/listinfo/infinispan-dev
>     <https://lists.jboss.org/mailman/listinfo/infinispan-dev>
> 
> 
> 
> 
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> 

-- 
Tristan Tarrant
Infinispan Lead
JBoss, a division of Red Hat

From sanne at infinispan.org  Fri Mar 31 06:27:36 2017
From: sanne at infinispan.org (Sanne Grinovero)
Date: Fri, 31 Mar 2017 11:27:36 +0100
Subject: [infinispan-dev] Infinispan 9.0 Final
In-Reply-To: <6319cb13-0359-aa9f-8d92-cb094c822e50@redhat.com>
References: <6319cb13-0359-aa9f-8d92-cb094c822e50@redhat.com>
Message-ID: <CAFm4XO3G0icpJQGgL6Xk0yGDw-gM+JNkzgMreT7Z=rFMGAa1Mg@mail.gmail.com>

That looks awesome. Congratulations everyone!

Now I'll start filing new issues from my wishlist for Hibernate OGM ;-)

Thanks,
Sanne


On 31 March 2017 at 10:09, Tristan Tarrant <ttarrant at redhat.com> wrote:
> Dear all,
>
> we are proud to announce Infinispan 9.0 Final.
> This release includes many new features and improvements:
>
> - much improved performance in all scenarios
> - off-heap data container, to avoid GC pauses
> - Ickle, a new query language based on JP-QL with full-text capabilities
> - multi-tenancy with SNI support for the server
> - vastly improved cloud and container integrations
>
> Read more about it in our announcement [1]
> As usual you can find all the downloads, documentation and community
> links on our website: http://infinispan.org
>
> Enjoy !
>
> The Infinispan Team
>
> [1] http://blog.infinispan.org/2017/03/infinispan-9.html
>
> --
> Tristan Tarrant
> Infinispan Lead
> JBoss, a division of Red Hat
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev

From gustavo at infinispan.org  Fri Mar 31 07:57:56 2017
From: gustavo at infinispan.org (Gustavo Fernandes)
Date: Fri, 31 Mar 2017 12:57:56 +0100
Subject: [infinispan-dev] Single Endpoint design
In-Reply-To: <1bcc751c-a246-bcb4-3c43-8bfe0ed252a7@redhat.com>
References: <CAA9R9O6c8SHB3E9PboofxRuj1LcA5yuvz3EEAUeX5=2Gdr-J0g@mail.gmail.com>
	<CAH8Ud1Fi4DnxT_c-ERy3zpt5QxkeERV0Bc+O0D=jRvSs4YS47A@mail.gmail.com>
	<1bcc751c-a246-bcb4-3c43-8bfe0ed252a7@redhat.com>
Message-ID: <CAH8Ud1GH=Gky-kNEOx55eicaUOGi-ehQ6T0pXcocx4=G5C+f+w@mail.gmail.com>

On Fri, Mar 31, 2017 at 11:02 AM, Tristan Tarrant <ttarrant at redhat.com>
wrote:

> You understood incorrectly.
> The only change to the Hot Rod clients is that, if they get a 400 error
> from a HR PING request, they will initiate an upgrade to Hot Rod and
> then proceed with the usual Hot Rod protocol after that.
>
>
Thanks for the clarification. Still, after the HR protocol is negotiated,
communication will go
through a router, thus adding an extra hop?

Gustavo

Tristan
>
> On 31/03/2017 11:58, Gustavo Fernandes wrote:
> > Hi Sebastian,
> >
> > If I understood it correctly, all the Hot Rod clients will be changed
> > from using:
> >
> > - Binary over TCP, circa 40 bytes header, no hops to contact the server,
> > no protocol negotiation, no encryption (default)
> >
> > to
> >
> > - HTTP/2 with SSL, protocol upgrade negotiation, and a hop (router) to
> > connect to the server.
> >
> >
> > Any idea of how significant would be this extra overhead introduced?
> >
> >
> > Thanks,
> > Gustavo
> >
> >
> > On Thu, Mar 30, 2017 at 2:01 PM, Sebastian Laskawiec
> > <slaskawi at redhat.com <mailto:slaskawi at redhat.com>> wrote:
> >
> >     Hey!
> >
> >     My plan is to start working on a Single Point support for Infinispan
> >     Server very soon and I prepared a design:
> >     https://github.com/infinispan/infinispan/pull/5041
> >     <https://github.com/infinispan/infinispan/pull/5041>
> >
> >     As you can see I did not use our Wiki (as we used to) because it
> >     doesn't support inline comments (which is pretty bad in my opinion).
> >     I would like to propose to keep all the designs along with our
> >     source code. This approach has been successfully used by the
> >     Kubernetes [1] folks (although they migrated designs into the new
> >     Community repository [2] recently). I think it might be a good idea
> >     to do something similar.
> >
> >     Feedback on both items is more than welcome.
> >
> >     Thanks,
> >     Sebastian
> >
> >     [1]
> >     https://github.com/kubernetes/kubernetes/tree/master/docs/proposals
> >     <https://github.com/kubernetes/kubernetes/tree/master/docs/proposals
> >
> >     [2]
> >     https://github.com/kubernetes/community/tree/
> master/contributors/design-proposals
> >     <https://github.com/kubernetes/community/tree/master/
> contributors/design-proposals>
> >
> >     _______________________________________________
> >     infinispan-dev mailing list
> >     infinispan-dev at lists.jboss.org <mailto:infinispan-dev at lists.j
> boss.org>
> >     https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >     <https://lists.jboss.org/mailman/listinfo/infinispan-dev>
> >
> >
> >
> >
> > _______________________________________________
> > infinispan-dev mailing list
> > infinispan-dev at lists.jboss.org
> > https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >
>
> --
> Tristan Tarrant
> Infinispan Lead
> JBoss, a division of Red Hat
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/infinispan-dev/attachments/20170331/c2476fd3/attachment.html 

From ttarrant at redhat.com  Fri Mar 31 08:10:02 2017
From: ttarrant at redhat.com (Tristan Tarrant)
Date: Fri, 31 Mar 2017 14:10:02 +0200
Subject: [infinispan-dev] Single Endpoint design
In-Reply-To: <CAH8Ud1GH=Gky-kNEOx55eicaUOGi-ehQ6T0pXcocx4=G5C+f+w@mail.gmail.com>
References: <CAA9R9O6c8SHB3E9PboofxRuj1LcA5yuvz3EEAUeX5=2Gdr-J0g@mail.gmail.com>
	<CAH8Ud1Fi4DnxT_c-ERy3zpt5QxkeERV0Bc+O0D=jRvSs4YS47A@mail.gmail.com>
	<1bcc751c-a246-bcb4-3c43-8bfe0ed252a7@redhat.com>
	<CAH8Ud1GH=Gky-kNEOx55eicaUOGi-ehQ6T0pXcocx4=G5C+f+w@mail.gmail.com>
Message-ID: <3ae53674-5362-139b-518e-717642a39bcd@redhat.com>

No, once the connection is established, I believe the netty pipeline can 
be trimmed to the necessary elements.

Tristan

On 31/03/2017 13:57, Gustavo Fernandes wrote:
> On Fri, Mar 31, 2017 at 11:02 AM, Tristan Tarrant <ttarrant at redhat.com 
> <mailto:ttarrant at redhat.com>> wrote:
> 
>     You understood incorrectly.
>     The only change to the Hot Rod clients is that, if they get a 400 error
>     from a HR PING request, they will initiate an upgrade to Hot Rod and
>     then proceed with the usual Hot Rod protocol after that.
> 
> 
> Thanks for the clarification. Still, after the HR protocol is 
> negotiated, communication will go
> through a router, thus adding an extra hop?
> 
> Gustavo
> 
>     Tristan
> 
>     On 31/03/2017 11:58, Gustavo Fernandes wrote:
>     > Hi Sebastian,
>     >
>     > If I understood it correctly, all the Hot Rod clients will be changed
>     > from using:
>     >
>     > - Binary over TCP, circa 40 bytes header, no hops to contact the server,
>     > no protocol negotiation, no encryption (default)
>     >
>     > to
>     >
>     > - HTTP/2 with SSL, protocol upgrade negotiation, and a hop (router) to
>     > connect to the server.
>     >
>     >
>     > Any idea of how significant would be this extra overhead introduced?
>     >
>     >
>     > Thanks,
>     > Gustavo
>     >
>     >
>     > On Thu, Mar 30, 2017 at 2:01 PM, Sebastian Laskawiec
>      > <slaskawi at redhat.com <mailto:slaskawi at redhat.com>
>     <mailto:slaskawi at redhat.com <mailto:slaskawi at redhat.com>>> wrote:
>      >
>      >     Hey!
>      >
>      >     My plan is to start working on a Single Point support for
>     Infinispan
>      >     Server very soon and I prepared a design:
>      > https://github.com/infinispan/infinispan/pull/5041
>     <https://github.com/infinispan/infinispan/pull/5041>
>      >     <https://github.com/infinispan/infinispan/pull/5041
>     <https://github.com/infinispan/infinispan/pull/5041>>
>      >
>      >     As you can see I did not use our Wiki (as we used to) because it
>      >     doesn't support inline comments (which is pretty bad in my
>     opinion).
>      >     I would like to propose to keep all the designs along with our
>      >     source code. This approach has been successfully used by the
>      >     Kubernetes [1] folks (although they migrated designs into the new
>      >     Community repository [2] recently). I think it might be a
>     good idea
>      >     to do something similar.
>      >
>      >     Feedback on both items is more than welcome.
>      >
>      >     Thanks,
>      >     Sebastian
>      >
>      >     [1]
>      >
>     https://github.com/kubernetes/kubernetes/tree/master/docs/proposals
>     <https://github.com/kubernetes/kubernetes/tree/master/docs/proposals>
>      >   
>       <https://github.com/kubernetes/kubernetes/tree/master/docs/proposals <https://github.com/kubernetes/kubernetes/tree/master/docs/proposals>>
>      >     [2]
>      >
>     https://github.com/kubernetes/community/tree/master/contributors/design-proposals
>     <https://github.com/kubernetes/community/tree/master/contributors/design-proposals>
>      >   
>       <https://github.com/kubernetes/community/tree/master/contributors/design-proposals <https://github.com/kubernetes/community/tree/master/contributors/design-proposals>>
>      >
>      >     _______________________________________________
>      >     infinispan-dev mailing list
>      > infinispan-dev at lists.jboss.org
>     <mailto:infinispan-dev at lists.jboss.org>
>     <mailto:infinispan-dev at lists.jboss.org
>     <mailto:infinispan-dev at lists.jboss.org>>
>      > https://lists.jboss.org/mailman/listinfo/infinispan-dev
>     <https://lists.jboss.org/mailman/listinfo/infinispan-dev>
>      >     <https://lists.jboss.org/mailman/listinfo/infinispan-dev
>     <https://lists.jboss.org/mailman/listinfo/infinispan-dev>>
>     >
>     >
>     >
>     >
>     > _______________________________________________
>     > infinispan-dev mailing list
>     > infinispan-dev at lists.jboss.org <mailto:infinispan-dev at lists.jboss.org>
>     > https://lists.jboss.org/mailman/listinfo/infinispan-dev
>     <https://lists.jboss.org/mailman/listinfo/infinispan-dev>
>     >
> 
>     --
>     Tristan Tarrant
>     Infinispan Lead
>     JBoss, a division of Red Hat
>     _______________________________________________
>     infinispan-dev mailing list
>     infinispan-dev at lists.jboss.org <mailto:infinispan-dev at lists.jboss.org>
>     https://lists.jboss.org/mailman/listinfo/infinispan-dev
>     <https://lists.jboss.org/mailman/listinfo/infinispan-dev>
> 
> 
> 
> 
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> 

-- 
Tristan Tarrant
Infinispan Lead
JBoss, a division of Red Hat

From ttarrant at redhat.com  Fri Mar 31 08:25:15 2017
From: ttarrant at redhat.com (Tristan Tarrant)
Date: Fri, 31 Mar 2017 14:25:15 +0200
Subject: [infinispan-dev] Infinispan Designs repository
Message-ID: <67964d52-724f-f88d-da2a-926ce5d81649@redhat.com>

As was pointed out by Sebastian, GitHub's wiki doesn't really take 
advantage of the wonderful review tools that are instead available for 
traditional repos.
Instead of creating noise in the main infinispan repo, I have setup a 
dedicated repo for design ideas. Let's fill it up !

https://github.com/infinispan/infinispan-designs

Tristan

-- 
Tristan Tarrant
Infinispan Lead
JBoss, a division of Red Hat

From emmanuel at hibernate.org  Fri Mar 31 09:20:37 2017
From: emmanuel at hibernate.org (Emmanuel Bernard)
Date: Fri, 31 Mar 2017 15:20:37 +0200
Subject: [infinispan-dev] Infinispan Designs repository
In-Reply-To: <67964d52-724f-f88d-da2a-926ce5d81649@redhat.com>
References: <67964d52-724f-f88d-da2a-926ce5d81649@redhat.com>
Message-ID: <35250E9A-7A8C-4BFE-81E3-20CEEAE543F6@hibernate.org>

Should we (as in someone specific that is not me) migrate all the GitHub proposal / design pages from the Wiki?

> On 31 Mar 2017, at 14:25, Tristan Tarrant <ttarrant at redhat.com> wrote:
> 
> As was pointed out by Sebastian, GitHub's wiki doesn't really take 
> advantage of the wonderful review tools that are instead available for 
> traditional repos.
> Instead of creating noise in the main infinispan repo, I have setup a 
> dedicated repo for design ideas. Let's fill it up !
> 
> https://github.com/infinispan/infinispan-designs
> 
> Tristan
> 
> -- 
> Tristan Tarrant
> Infinispan Lead
> JBoss, a division of Red Hat
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev



From dan.berindei at gmail.com  Fri Mar 31 09:40:44 2017
From: dan.berindei at gmail.com (Dan Berindei)
Date: Fri, 31 Mar 2017 16:40:44 +0300
Subject: [infinispan-dev] Strategy to adopting Optional in APIs
In-Reply-To: <3a302757-662a-68e9-23ea-4345a6bbb787@redhat.com>
References: <7ecd428c-6c32-2907-f0ef-721a1e6c4a5f@redhat.com>
	<CAFm4XO3Y5Dmtb_7prAa37g8jCRnSmNCcUVUs6bN6cpz-jAKHEw@mail.gmail.com>
	<63ed80a2-7a88-f9ef-5c7b-345433ffc6c2@redhat.com>
	<3a302757-662a-68e9-23ea-4345a6bbb787@redhat.com>
Message-ID: <CA+nfvwTR7yoQkFpy8Yg73u0gBCCeBnmA5_CCuq1r5F6iJUoyaw@mail.gmail.com>

Maybe yes, maybe not... in my experience escape analysis is very
fickle, e.g. it's very easy for a method to become big enough after
inlining that escape analysis is not performed at all.

Dan

On Fri, Mar 31, 2017 at 10:59 AM, Tristan Tarrant <ttarrant at redhat.com> wrote:
> I was about to say the same: in the typical use case of returning an
> optional and using it immediately it would probably end up on the stack
> anyway...
>
> Tristan
>
> On 31/03/2017 09:57, Radim Vansa wrote:
>> I secretly hope that all these allocations would be inlined and
>> eliminated. If we find out that it really allocates the objects (from
>> JFR's allocation stats), it's a reason to rewrite that piece of code to
>> the dull optionless version.
>> TBH I am rather afraid that the JVM will allocate the consumer which
>> will need some captured variables. Maybe I trust C2 compiler too much,
>> believing that if the handler isn't too big, it will generate similar
>> instructions with nicer source code :-/
>>
>> R.
>>
>>
>> On 03/30/2017 11:08 PM, Sanne Grinovero wrote:
>>> I'm for "at discretion" and "avoid if not really needed" : not cool to
>>> allocate objects for no reason.
>>>
>>> On 30 Mar 2017 16:57, "Radim Vansa" <rvansa at redhat.com
>>> <mailto:rvansa at redhat.com>> wrote:
>>>
>>>      Hi,
>>>
>>>      I was wondering what's the common attitude towards using Optional in
>>>      APIs, and what naming pattern should we use. As an example, I dislike
>>>      calling
>>>
>>>      if (entry.getMetadata() != null && entry.getMetadata().version()
>>>      != null) {
>>>           foo.use(entry.getMetadata().version())
>>>      }
>>>
>>>      where I could just do
>>>
>>>      entry.metadata().flatMap(Metadata::optionalVersion).ifPresent(foo::use)
>>>
>>>      Here I have proposed metadata() method returning Optional<Metadata>
>>>      (regular getter method is called getMetadata()) and annoying
>>>      optionalVersion() as version() is the regular getter.
>>>
>>>      Shall we adopt some common stance (use/don't use/use at developer's
>>>      discretion) and naming conventions? Is it acceptable to start adding
>>>
>>>      default Optional<Foo> foo() { Optional.ofNullable(getFoo()); }
>>>
>>>      whenever we feel the urge to chain Optionals?
>>>
>>>      Radim
>>>
>>>      --
>>>      Radim Vansa <rvansa at redhat.com <mailto:rvansa at redhat.com>>
>>>      JBoss Performance Team
>>>
>>>      _______________________________________________
>>>      infinispan-dev mailing list
>>>      infinispan-dev at lists.jboss.org <mailto:infinispan-dev at lists.jboss.org>
>>>      https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>>      <https://lists.jboss.org/mailman/listinfo/infinispan-dev>
>>>
>>>
>>>
>>> _______________________________________________
>>> infinispan-dev mailing list
>>> infinispan-dev at lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>
>>
>
> --
> Tristan Tarrant
> Infinispan Lead
> JBoss, a division of Red Hat
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev

From slaskawi at redhat.com  Fri Mar 31 09:55:15 2017
From: slaskawi at redhat.com (Sebastian Laskawiec)
Date: Fri, 31 Mar 2017 13:55:15 +0000
Subject: [infinispan-dev] Hot Rod secured by default
In-Reply-To: <CA+nfvwQjSy+oC0gwoEccEMs_E0k0ksD5JsF7rYb2Jdxm3g4mxA@mail.gmail.com>
References: <f3cee023-96a1-3c7d-a3ce-b575cd5b0f7e@redhat.com>
	<CAA9R9O7S7frPE-sS60Z2E6jV_+OdL18+dvMDFJwQsd-FQfQt7w@mail.gmail.com>
	<df6c29dc-4d48-1dc7-6045-2a14e0dd01d8@redhat.com>
	<06b88c80-433f-3745-2112-1424bcd9c342@redhat.com>
	<CAKwmrdPrJgy_KPyZNHnK40mGrCqG05DyntvC_3R=TKK2P6-AoQ@mail.gmail.com>
	<CA+nfvwQjSy+oC0gwoEccEMs_E0k0ksD5JsF7rYb2Jdxm3g4mxA@mail.gmail.com>
Message-ID: <CAA9R9O5eYa2GXuk+bmQcDYt7RVsNnKeJMadXM99+hTMhjGwueA@mail.gmail.com>

Unfortunately TLS still slows down stuff (a lot). When I was doing tests
for the multi-tenancy router (which is based on TLS/SNI), my average
results were like this:

Use-case Type Avg Error
initConnectionAndPerform10KPuts SingleServerNoSsl 1034.817 14.424
initConnectionAndPerform10KPuts SingleServerWithSsl 1567.553 24.872
initConnectionAndPerform10KPuts TwoServersWithSslSni 1563.229 34.05
initConnectionOnly SingleServerNoSsl *3.389* 0.198
initConnectionOnly SingleServerWithSsl *14.086* 0.794
initConnectionOnly TwoServersWithSslSni *14.722* 0.684
perform10KPuts SingleServerNoSsl *4.602* 0.585
perform10KPuts SingleServerWithSsl *16.583* 0.198
perform10KPuts TwoServersWithSslSni *17.02* 0.794

This is nothing new, but initializing Hot Rod connection took was ~4 times
slower and putting 10K random strings (UUIDs) was also ~4 times slower. But
what's worth to mention, there is no significant difference between TLS and
TLS+SNI.

As far as I know, it is possible to install specialized hardware to deal
with encryption in data centers. It is called SSL Acceleration [1]. However
I'm not aware of any special processor instructions that can help you with
that. But the implementations are getting better and better, so who knows...

But getting back to the original question, I think the problem we are
trying to solve (correct me if I'm wrong) is to prevent unauthorized folks
to put their hands on a victims data (either pushing something
malicious/corrupted to the cache or obtaining something from the cache).
Another problem is transmission security - encryption. If we want our new
devs to be secured out of the box, I think we should do both - use TLS
(without trusting all certificated) and authentication. This makes
Infinispan harder to use of course. So the other extremum is to turn both
things off.

I voted for the latter, making Infinispan super easy to use. But you guys
convinced me that we should care about the security in this case too, so I
would use PLAIN authentication + TLS. I would also love to see one magic
switch, for example `./bin/standalone.sh --dev-mode`, which would turn all
security off.

Thanks,
Sebastian

[1] https://en.wikipedia.org/wiki/SSL_acceleration


On Thu, Mar 30, 2017 at 9:22 PM Dan Berindei <dan.berindei at gmail.com> wrote:

> I agree with Radim, PLAIN authentication without encryption makes it
> too easy to sniff the password from another machine.
>
> I have no idea how expensive SSL encryption is in WildFly, but I think
> all recent processors have specialized instructions for helping with
> encryption, so it may not be that bad.
>
> Even with encryption, if the client trusts all certs, it may be
> possible for an attacker to insert itself in the middle and decode
> everything -- depending on network topology and what kind of access
> the attacker already has. I think it only makes sense to trust all
> certs if we also implement something like HPKP [1], to make it more
> like ssh.
>
> [1]: https://en.wikipedia.org/wiki/HTTP_Public_Key_Pinning
>
> Cheers
> Dan
>
>
>
> On Thu, Mar 30, 2017 at 7:07 PM, Wolf Fink <wfink at redhat.com> wrote:
> > +1 to make the default secure.
> >
> > -1 SSL by default as it makes it slower and I think not most will use it
> >
> > -1 easy trust all certs, That sounds to me we close one door and make it
> > possible to open another one
> >
> >
> > What if we add an example configuration unsecured which can be simple
> copied
> > for examples and to start.
> >
> >
> > On Thu, Mar 30, 2017 at 5:31 PM, Dennis Reed <dereed at redhat.com> wrote:
> >>
> >> +1 to authentication and encryption by default.
> >>   This is 2017, that's how *everything* should be configured.
> >>
> >> -1 to making it easy to trust all certs.  That negates the point of
> >> using encryption in the first place and should really never be done.
> >>
> >> If it's too hard to configure the correct way that we think it would
> >> turn users away, that's a usability problem that needs to be fixed.
> >>
> >> -Dennis
> >>
> >>
> >> On 03/30/2017 09:29 AM, Tristan Tarrant wrote:
> >> > While the "unsecure" over loopback is quite tempting, I would prefer
> to
> >> > have homogeneous behaviour with the possibility to disable security
> >> > altogether for quick demos.
> >> > Otherwise a developer would need to code differently for the local use
> >> > case than for the remote one, causing more confusion.
> >> >
> >> > Tristan
> >> >
> >> > On 30/03/2017 14:54, Sebastian Laskawiec wrote:
> >> >> I agree the security out of the box is good. But at the same time we
> >> >> don't want to make Infinispan harder to use for new developers. Out
> of
> >> >> the box configuration should be "good enough" to start hacking.
> >> >>
> >> >> I would propose to make all the endpoints unprotected (with
> >> >> authentication disabled) on localhost/loopback and protected when
> >> >> calling from the outside world.
> >> >>
> >> >> On Thu, Mar 30, 2017 at 2:39 PM Tristan Tarrant <ttarrant at redhat.com
> >> >> <mailto:ttarrant at redhat.com>> wrote:
> >> >>
> >> >>     Dear all,
> >> >>
> >> >>     after a mini chat on IRC, I wanted to bring this to everybody's
> >> >>     attention.
> >> >>
> >> >>     We should make the Hot Rod endpoint require authentication in the
> >> >>     out-of-the-box configuration.
> >> >>     The proposal is to enable the PLAIN (or, preferably, DIGEST) SASL
> >> >>     mechanism against the ApplicationRealm and require users to run
> the
> >> >>     add-user script.
> >> >>     This would achieve two goals:
> >> >>     - secure out-of-the-box configuration, which is always a good
> idea
> >> >>     - access to the "protected" schema and script caches which is
> >> >> prevented
> >> >>     when not on loopback on non-authenticated endpoints.
> >> >>
> >> >>     Tristan
> >> >>     --
> >> >>     Tristan Tarrant
> >> >>     Infinispan Lead
> >> >>     JBoss, a division of Red Hat
> >> >>     _______________________________________________
> >> >>     infinispan-dev mailing list
> >> >>     infinispan-dev at lists.jboss.org
> >> >> <mailto:infinispan-dev at lists.jboss.org>
> >> >>     https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >> >>
> >> >>
> >> >>
> >> >> _______________________________________________
> >> >> infinispan-dev mailing list
> >> >> infinispan-dev at lists.jboss.org
> >> >> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >> >>
> >> >
> >> _______________________________________________
> >> infinispan-dev mailing list
> >> infinispan-dev at lists.jboss.org
> >> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >
> >
> >
> > _______________________________________________
> > infinispan-dev mailing list
> > infinispan-dev at lists.jboss.org
> > https://lists.jboss.org/mailman/listinfo/infinispan-dev
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/infinispan-dev/attachments/20170331/c00313ba/attachment.html 

From slaskawi at redhat.com  Fri Mar 31 09:58:54 2017
From: slaskawi at redhat.com (Sebastian Laskawiec)
Date: Fri, 31 Mar 2017 13:58:54 +0000
Subject: [infinispan-dev] Strategy to adopting Optional in APIs
In-Reply-To: <3a302757-662a-68e9-23ea-4345a6bbb787@redhat.com>
References: <7ecd428c-6c32-2907-f0ef-721a1e6c4a5f@redhat.com>
	<CAFm4XO3Y5Dmtb_7prAa37g8jCRnSmNCcUVUs6bN6cpz-jAKHEw@mail.gmail.com>
	<63ed80a2-7a88-f9ef-5c7b-345433ffc6c2@redhat.com>
	<3a302757-662a-68e9-23ea-4345a6bbb787@redhat.com>
Message-ID: <CAA9R9O55DCo89=HY2epCxvvxhZAOya_AsV=2=TjWh4kf1FpqkQ@mail.gmail.com>

I like the idea of using Optionals. But I think we would need to enhance
all our APIs to use this (to be consistent).

Maybe you could come up with a small proposition of API changes? It might
not work correctly (we can just mock them) but I would love to see some
examples and "feel" the semantics.

On Fri, Mar 31, 2017 at 10:00 AM Tristan Tarrant <ttarrant at redhat.com>
wrote:

> I was about to say the same: in the typical use case of returning an
> optional and using it immediately it would probably end up on the stack
> anyway...
>
> Tristan
>
> On 31/03/2017 09:57, Radim Vansa wrote:
> > I secretly hope that all these allocations would be inlined and
> > eliminated. If we find out that it really allocates the objects (from
> > JFR's allocation stats), it's a reason to rewrite that piece of code to
> > the dull optionless version.
> > TBH I am rather afraid that the JVM will allocate the consumer which
> > will need some captured variables. Maybe I trust C2 compiler too much,
> > believing that if the handler isn't too big, it will generate similar
> > instructions with nicer source code :-/
> >
> > R.
> >
> >
> > On 03/30/2017 11:08 PM, Sanne Grinovero wrote:
> >> I'm for "at discretion" and "avoid if not really needed" : not cool to
> >> allocate objects for no reason.
> >>
> >> On 30 Mar 2017 16:57, "Radim Vansa" <rvansa at redhat.com
> >> <mailto:rvansa at redhat.com>> wrote:
> >>
> >>      Hi,
> >>
> >>      I was wondering what's the common attitude towards using Optional
> in
> >>      APIs, and what naming pattern should we use. As an example, I
> dislike
> >>      calling
> >>
> >>      if (entry.getMetadata() != null && entry.getMetadata().version()
> >>      != null) {
> >>           foo.use(entry.getMetadata().version())
> >>      }
> >>
> >>      where I could just do
> >>
> >>
> entry.metadata().flatMap(Metadata::optionalVersion).ifPresent(foo::use)
> >>
> >>      Here I have proposed metadata() method returning Optional<Metadata>
> >>      (regular getter method is called getMetadata()) and annoying
> >>      optionalVersion() as version() is the regular getter.
> >>
> >>      Shall we adopt some common stance (use/don't use/use at developer's
> >>      discretion) and naming conventions? Is it acceptable to start
> adding
> >>
> >>      default Optional<Foo> foo() { Optional.ofNullable(getFoo()); }
> >>
> >>      whenever we feel the urge to chain Optionals?
> >>
> >>      Radim
> >>
> >>      --
> >>      Radim Vansa <rvansa at redhat.com <mailto:rvansa at redhat.com>>
> >>      JBoss Performance Team
> >>
> >>      _______________________________________________
> >>      infinispan-dev mailing list
> >>      infinispan-dev at lists.jboss.org <mailto:
> infinispan-dev at lists.jboss.org>
> >>      https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >>      <https://lists.jboss.org/mailman/listinfo/infinispan-dev>
> >>
> >>
> >>
> >> _______________________________________________
> >> infinispan-dev mailing list
> >> infinispan-dev at lists.jboss.org
> >> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >
> >
>
> --
> Tristan Tarrant
> Infinispan Lead
> JBoss, a division of Red Hat
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/infinispan-dev/attachments/20170331/65d78a81/attachment-0001.html 

From slaskawi at redhat.com  Fri Mar 31 10:02:00 2017
From: slaskawi at redhat.com (Sebastian Laskawiec)
Date: Fri, 31 Mar 2017 14:02:00 +0000
Subject: [infinispan-dev] Infinispan Designs repository
In-Reply-To: <35250E9A-7A8C-4BFE-81E3-20CEEAE543F6@hibernate.org>
References: <67964d52-724f-f88d-da2a-926ce5d81649@redhat.com>
	<35250E9A-7A8C-4BFE-81E3-20CEEAE543F6@hibernate.org>
Message-ID: <CAA9R9O4ByACDBaEKXxQozX+1wZmOZmk+Bt-HSaX2heDqcYU8ew@mail.gmail.com>

+1. I would also replace migrated documents with links to the new repo.

On Fri, Mar 31, 2017 at 3:52 PM Emmanuel Bernard <emmanuel at hibernate.org>
wrote:

> Should we (as in someone specific that is not me) migrate all the GitHub
> proposal / design pages from the Wiki?
>
> > On 31 Mar 2017, at 14:25, Tristan Tarrant <ttarrant at redhat.com> wrote:
> >
> > As was pointed out by Sebastian, GitHub's wiki doesn't really take
> > advantage of the wonderful review tools that are instead available for
> > traditional repos.
> > Instead of creating noise in the main infinispan repo, I have setup a
> > dedicated repo for design ideas. Let's fill it up !
> >
> > https://github.com/infinispan/infinispan-designs
> >
> > Tristan
> >
> > --
> > Tristan Tarrant
> > Infinispan Lead
> > JBoss, a division of Red Hat
> > _______________________________________________
> > infinispan-dev mailing list
> > infinispan-dev at lists.jboss.org
> > https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/infinispan-dev/attachments/20170331/7b1e51c9/attachment.html 

From ttarrant at redhat.com  Fri Mar 31 10:06:54 2017
From: ttarrant at redhat.com (Tristan Tarrant)
Date: Fri, 31 Mar 2017 16:06:54 +0200
Subject: [infinispan-dev] Infinispan Designs repository
In-Reply-To: <35250E9A-7A8C-4BFE-81E3-20CEEAE543F6@hibernate.org>
References: <67964d52-724f-f88d-da2a-926ce5d81649@redhat.com>
	<35250E9A-7A8C-4BFE-81E3-20CEEAE543F6@hibernate.org>
Message-ID: <952eec7f-21fb-3ad6-a347-97e39470e9f7@redhat.com>

Already done sir.

Tristan

On 31/03/2017 15:20, Emmanuel Bernard wrote:
> Should we (as in someone specific that is not me) migrate all the GitHub proposal / design pages from the Wiki?
> 
>> On 31 Mar 2017, at 14:25, Tristan Tarrant <ttarrant at redhat.com> wrote:
>>
>> As was pointed out by Sebastian, GitHub's wiki doesn't really take
>> advantage of the wonderful review tools that are instead available for
>> traditional repos.
>> Instead of creating noise in the main infinispan repo, I have setup a
>> dedicated repo for design ideas. Let's fill it up !
>>
>> https://github.com/infinispan/infinispan-designs
>>
>> Tristan
>>
>> -- 
>> Tristan Tarrant
>> Infinispan Lead
>> JBoss, a division of Red Hat
>> _______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> 
> 
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> 

-- 
Tristan Tarrant
Infinispan Lead
JBoss, a division of Red Hat

From slaskawi at redhat.com  Fri Mar 31 10:20:03 2017
From: slaskawi at redhat.com (Sebastian Laskawiec)
Date: Fri, 31 Mar 2017 14:20:03 +0000
Subject: [infinispan-dev] Single Endpoint design
In-Reply-To: <3ae53674-5362-139b-518e-717642a39bcd@redhat.com>
References: <CAA9R9O6c8SHB3E9PboofxRuj1LcA5yuvz3EEAUeX5=2Gdr-J0g@mail.gmail.com>
	<CAH8Ud1Fi4DnxT_c-ERy3zpt5QxkeERV0Bc+O0D=jRvSs4YS47A@mail.gmail.com>
	<1bcc751c-a246-bcb4-3c43-8bfe0ed252a7@redhat.com>
	<CAH8Ud1GH=Gky-kNEOx55eicaUOGi-ehQ6T0pXcocx4=G5C+f+w@mail.gmail.com>
	<3ae53674-5362-139b-518e-717642a39bcd@redhat.com>
Message-ID: <CAA9R9O4hNT_r7rpfyJ=dwSLAxMkiFm6ZEbXw3RFmrFVceSPg+A@mail.gmail.com>

With TLS+ALPN the use case is very simple. The protocol is negotiated
during the handshake. Then the existing TCP connection is reused and the
client starts to send Hot Rod binary bits through the wire. Sadly, TLS adds
some significant overhead to the transmission.

With HTTP/1.1 Upgrade it's a bit more complicated. The Upgrade request
might issued by the client as well as by the server. Each of the parties
can ignore the request if it doesn't recognize it (or doesn't support
protocols proposed by the other side). The more I think about this, the
more I lean towards to server initiated upgrade requests. This way, only
clients which supports the upgrade will react to it. It still needs some
more thought...

Of course you are right, some additional code will need to be added to the
clients to support switching protocols. The good news is that you won't be
forced to use the Single Endpoint. You may want to expose our old, good Hot
Rod Endpoint and forget about all this switching complexity.

Regarding to network hops - no, there is no additional network hops. The
router component simply identifies the incoming request and plugs proper
Netty Handler stack to it. Here's an example how it is done with
multi-tenancy based on SNI [1].

[1]
https://github.com/infinispan/infinispan/blob/master/server/router/src/main/java/org/infinispan/server/router/router/impl/hotrod/handlers/SniRouteHandler.java#L48-L61

On Fri, Mar 31, 2017 at 2:22 PM Tristan Tarrant <ttarrant at redhat.com> wrote:

> No, once the connection is established, I believe the netty pipeline can
> be trimmed to the necessary elements.
>
> Tristan
>
> On 31/03/2017 13:57, Gustavo Fernandes wrote:
> > On Fri, Mar 31, 2017 at 11:02 AM, Tristan Tarrant <ttarrant at redhat.com
> > <mailto:ttarrant at redhat.com>> wrote:
> >
> >     You understood incorrectly.
> >     The only change to the Hot Rod clients is that, if they get a 400
> error
> >     from a HR PING request, they will initiate an upgrade to Hot Rod and
> >     then proceed with the usual Hot Rod protocol after that.
> >
> >
> > Thanks for the clarification. Still, after the HR protocol is
> > negotiated, communication will go
> > through a router, thus adding an extra hop?
> >
> > Gustavo
> >
> >     Tristan
> >
> >     On 31/03/2017 11:58, Gustavo Fernandes wrote:
> >     > Hi Sebastian,
> >     >
> >     > If I understood it correctly, all the Hot Rod clients will be
> changed
> >     > from using:
> >     >
> >     > - Binary over TCP, circa 40 bytes header, no hops to contact the
> server,
> >     > no protocol negotiation, no encryption (default)
> >     >
> >     > to
> >     >
> >     > - HTTP/2 with SSL, protocol upgrade negotiation, and a hop
> (router) to
> >     > connect to the server.
> >     >
> >     >
> >     > Any idea of how significant would be this extra overhead
> introduced?
> >     >
> >     >
> >     > Thanks,
> >     > Gustavo
> >     >
> >     >
> >     > On Thu, Mar 30, 2017 at 2:01 PM, Sebastian Laskawiec
> >      > <slaskawi at redhat.com <mailto:slaskawi at redhat.com>
> >     <mailto:slaskawi at redhat.com <mailto:slaskawi at redhat.com>>> wrote:
> >      >
> >      >     Hey!
> >      >
> >      >     My plan is to start working on a Single Point support for
> >     Infinispan
> >      >     Server very soon and I prepared a design:
> >      > https://github.com/infinispan/infinispan/pull/5041
> >     <https://github.com/infinispan/infinispan/pull/5041>
> >      >     <https://github.com/infinispan/infinispan/pull/5041
> >     <https://github.com/infinispan/infinispan/pull/5041>>
> >      >
> >      >     As you can see I did not use our Wiki (as we used to) because
> it
> >      >     doesn't support inline comments (which is pretty bad in my
> >     opinion).
> >      >     I would like to propose to keep all the designs along with our
> >      >     source code. This approach has been successfully used by the
> >      >     Kubernetes [1] folks (although they migrated designs into the
> new
> >      >     Community repository [2] recently). I think it might be a
> >     good idea
> >      >     to do something similar.
> >      >
> >      >     Feedback on both items is more than welcome.
> >      >
> >      >     Thanks,
> >      >     Sebastian
> >      >
> >      >     [1]
> >      >
> >     https://github.com/kubernetes/kubernetes/tree/master/docs/proposals
> >     <https://github.com/kubernetes/kubernetes/tree/master/docs/proposals
> >
> >      >
> >       <
> https://github.com/kubernetes/kubernetes/tree/master/docs/proposals <
> https://github.com/kubernetes/kubernetes/tree/master/docs/proposals>>
> >      >     [2]
> >      >
> >
> https://github.com/kubernetes/community/tree/master/contributors/design-proposals
> >     <
> https://github.com/kubernetes/community/tree/master/contributors/design-proposals
> >
> >      >
> >       <
> https://github.com/kubernetes/community/tree/master/contributors/design-proposals
> <
> https://github.com/kubernetes/community/tree/master/contributors/design-proposals
> >>
> >      >
> >      >     _______________________________________________
> >      >     infinispan-dev mailing list
> >      > infinispan-dev at lists.jboss.org
> >     <mailto:infinispan-dev at lists.jboss.org>
> >     <mailto:infinispan-dev at lists.jboss.org
> >     <mailto:infinispan-dev at lists.jboss.org>>
> >      > https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >     <https://lists.jboss.org/mailman/listinfo/infinispan-dev>
> >      >     <https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >     <https://lists.jboss.org/mailman/listinfo/infinispan-dev>>
> >     >
> >     >
> >     >
> >     >
> >     > _______________________________________________
> >     > infinispan-dev mailing list
> >     > infinispan-dev at lists.jboss.org <mailto:
> infinispan-dev at lists.jboss.org>
> >     > https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >     <https://lists.jboss.org/mailman/listinfo/infinispan-dev>
> >     >
> >
> >     --
> >     Tristan Tarrant
> >     Infinispan Lead
> >     JBoss, a division of Red Hat
> >     _______________________________________________
> >     infinispan-dev mailing list
> >     infinispan-dev at lists.jboss.org <mailto:
> infinispan-dev at lists.jboss.org>
> >     https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >     <https://lists.jboss.org/mailman/listinfo/infinispan-dev>
> >
> >
> >
> >
> > _______________________________________________
> > infinispan-dev mailing list
> > infinispan-dev at lists.jboss.org
> > https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >
>
> --
> Tristan Tarrant
> Infinispan Lead
> JBoss, a division of Red Hat
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/infinispan-dev/attachments/20170331/6eae9376/attachment-0001.html 

From ttarrant at redhat.com  Fri Mar 31 11:17:31 2017
From: ttarrant at redhat.com (Tristan Tarrant)
Date: Fri, 31 Mar 2017 17:17:31 +0200
Subject: [infinispan-dev] Hot Rod secured by default
In-Reply-To: <CAA9R9O5eYa2GXuk+bmQcDYt7RVsNnKeJMadXM99+hTMhjGwueA@mail.gmail.com>
References: <f3cee023-96a1-3c7d-a3ce-b575cd5b0f7e@redhat.com>
	<CAA9R9O7S7frPE-sS60Z2E6jV_+OdL18+dvMDFJwQsd-FQfQt7w@mail.gmail.com>
	<df6c29dc-4d48-1dc7-6045-2a14e0dd01d8@redhat.com>
	<06b88c80-433f-3745-2112-1424bcd9c342@redhat.com>
	<CAKwmrdPrJgy_KPyZNHnK40mGrCqG05DyntvC_3R=TKK2P6-AoQ@mail.gmail.com>
	<CA+nfvwQjSy+oC0gwoEccEMs_E0k0ksD5JsF7rYb2Jdxm3g4mxA@mail.gmail.com>
	<CAA9R9O5eYa2GXuk+bmQcDYt7RVsNnKeJMadXM99+hTMhjGwueA@mail.gmail.com>
Message-ID: <dec3db85-5b52-33bd-e9c7-a5a65cee3edc@redhat.com>

You want to use OpenSSL with Netty:

http://netty.io/wiki/requirements-for-4.x.html#wiki-h4-4

Tristan

On 31/03/2017 15:55, Sebastian Laskawiec wrote:
> Unfortunately TLS still slows down stuff (a lot). When I was doing tests 
> for the multi-tenancy router (which is based on TLS/SNI), my average 
> results were like this:
> 
> Use-caseTypeAvgError
> initConnectionAndPerform10KPutsSingleServerNoSsl1034.81714.424
> initConnectionAndPerform10KPutsSingleServerWithSsl1567.55324.872
> initConnectionAndPerform10KPutsTwoServersWithSslSni1563.22934.05
> initConnectionOnlySingleServerNoSsl*3.389*0.198
> initConnectionOnlySingleServerWithSsl*14.086*0.794
> initConnectionOnlyTwoServersWithSslSni*14.722*0.684
> perform10KPutsSingleServerNoSsl*4.602*0.585
> perform10KPutsSingleServerWithSsl*16.583*0.198
> perform10KPutsTwoServersWithSslSni*17.02*0.794
> 
> This is nothing new, but initializing Hot Rod connection took was ~4 
> times slower and putting 10K random strings (UUIDs) was also ~4 times 
> slower. But what's worth to mention, there is no significant difference 
> between TLS and TLS+SNI.
> 
> As far as I know, it is possible to install specialized hardware to deal 
> with encryption in data centers. It is called SSL Acceleration [1]. 
> However I'm not aware of any special processor instructions that can 
> help you with that. But the implementations are getting better and 
> better, so who knows...
> 
> But getting back to the original question, I think the problem we are 
> trying to solve (correct me if I'm wrong) is to prevent unauthorized 
> folks to put their hands on a victims data (either pushing something 
> malicious/corrupted to the cache or obtaining something from the cache). 
> Another problem is transmission security - encryption. If we want our 
> new devs to be secured out of the box, I think we should do both - use 
> TLS (without trusting all certificated) and authentication. This makes 
> Infinispan harder to use of course. So the other extremum is to turn 
> both things off.
> 
> I voted for the latter, making Infinispan super easy to use. But you 
> guys convinced me that we should care about the security in this case 
> too, so I would use PLAIN authentication + TLS. I would also love to see 
> one magic switch, for example `./bin/standalone.sh --dev-mode`, which 
> would turn all security off.
> 
> Thanks,
> Sebastian
> 
> [1] https://en.wikipedia.org/wiki/SSL_acceleration
> 
> 
> On Thu, Mar 30, 2017 at 9:22 PM Dan Berindei <dan.berindei at gmail.com 
> <mailto:dan.berindei at gmail.com>> wrote:
> 
>     I agree with Radim, PLAIN authentication without encryption makes it
>     too easy to sniff the password from another machine.
> 
>     I have no idea how expensive SSL encryption is in WildFly, but I think
>     all recent processors have specialized instructions for helping with
>     encryption, so it may not be that bad.
> 
>     Even with encryption, if the client trusts all certs, it may be
>     possible for an attacker to insert itself in the middle and decode
>     everything -- depending on network topology and what kind of access
>     the attacker already has. I think it only makes sense to trust all
>     certs if we also implement something like HPKP [1], to make it more
>     like ssh.
> 
>     [1]: https://en.wikipedia.org/wiki/HTTP_Public_Key_Pinning
> 
>     Cheers
>     Dan
> 
> 
> 
>     On Thu, Mar 30, 2017 at 7:07 PM, Wolf Fink <wfink at redhat.com
>     <mailto:wfink at redhat.com>> wrote:
>      > +1 to make the default secure.
>      >
>      > -1 SSL by default as it makes it slower and I think not most will
>     use it
>      >
>      > -1 easy trust all certs, That sounds to me we close one door and
>     make it
>      > possible to open another one
>      >
>      >
>      > What if we add an example configuration unsecured which can be
>     simple copied
>      > for examples and to start.
>      >
>      >
>      > On Thu, Mar 30, 2017 at 5:31 PM, Dennis Reed <dereed at redhat.com
>     <mailto:dereed at redhat.com>> wrote:
>      >>
>      >> +1 to authentication and encryption by default.
>      >>   This is 2017, that's how *everything* should be configured.
>      >>
>      >> -1 to making it easy to trust all certs.  That negates the point of
>      >> using encryption in the first place and should really never be done.
>      >>
>      >> If it's too hard to configure the correct way that we think it would
>      >> turn users away, that's a usability problem that needs to be fixed.
>      >>
>      >> -Dennis
>      >>
>      >>
>      >> On 03/30/2017 09:29 AM, Tristan Tarrant wrote:
>      >> > While the "unsecure" over loopback is quite tempting, I would
>     prefer to
>      >> > have homogeneous behaviour with the possibility to disable
>     security
>      >> > altogether for quick demos.
>      >> > Otherwise a developer would need to code differently for the
>     local use
>      >> > case than for the remote one, causing more confusion.
>      >> >
>      >> > Tristan
>      >> >
>      >> > On 30/03/2017 14:54, Sebastian Laskawiec wrote:
>      >> >> I agree the security out of the box is good. But at the same
>     time we
>      >> >> don't want to make Infinispan harder to use for new
>     developers. Out of
>      >> >> the box configuration should be "good enough" to start hacking.
>      >> >>
>      >> >> I would propose to make all the endpoints unprotected (with
>      >> >> authentication disabled) on localhost/loopback and protected when
>      >> >> calling from the outside world.
>      >> >>
>      >> >> On Thu, Mar 30, 2017 at 2:39 PM Tristan Tarrant
>     <ttarrant at redhat.com <mailto:ttarrant at redhat.com>
>      >> >> <mailto:ttarrant at redhat.com <mailto:ttarrant at redhat.com>>> wrote:
>      >> >>
>      >> >>     Dear all,
>      >> >>
>      >> >>     after a mini chat on IRC, I wanted to bring this to
>     everybody's
>      >> >>     attention.
>      >> >>
>      >> >>     We should make the Hot Rod endpoint require
>     authentication in the
>      >> >>     out-of-the-box configuration.
>      >> >>     The proposal is to enable the PLAIN (or, preferably,
>     DIGEST) SASL
>      >> >>     mechanism against the ApplicationRealm and require users
>     to run the
>      >> >>     add-user script.
>      >> >>     This would achieve two goals:
>      >> >>     - secure out-of-the-box configuration, which is always a
>     good idea
>      >> >>     - access to the "protected" schema and script caches which is
>      >> >> prevented
>      >> >>     when not on loopback on non-authenticated endpoints.
>      >> >>
>      >> >>     Tristan
>      >> >>     --
>      >> >>     Tristan Tarrant
>      >> >>     Infinispan Lead
>      >> >>     JBoss, a division of Red Hat
>      >> >>     _______________________________________________
>      >> >>     infinispan-dev mailing list
>      >> >> infinispan-dev at lists.jboss.org
>     <mailto:infinispan-dev at lists.jboss.org>
>      >> >> <mailto:infinispan-dev at lists.jboss.org
>     <mailto:infinispan-dev at lists.jboss.org>>
>      >> >> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>      >> >>
>      >> >>
>      >> >>
>      >> >> _______________________________________________
>      >> >> infinispan-dev mailing list
>      >> >> infinispan-dev at lists.jboss.org
>     <mailto:infinispan-dev at lists.jboss.org>
>      >> >> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>      >> >>
>      >> >
>      >> _______________________________________________
>      >> infinispan-dev mailing list
>      >> infinispan-dev at lists.jboss.org
>     <mailto:infinispan-dev at lists.jboss.org>
>      >> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>      >
>      >
>      >
>      > _______________________________________________
>      > infinispan-dev mailing list
>      > infinispan-dev at lists.jboss.org
>     <mailto:infinispan-dev at lists.jboss.org>
>      > https://lists.jboss.org/mailman/listinfo/infinispan-dev
>     _______________________________________________
>     infinispan-dev mailing list
>     infinispan-dev at lists.jboss.org <mailto:infinispan-dev at lists.jboss.org>
>     https://lists.jboss.org/mailman/listinfo/infinispan-dev
> 
> 
> 
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> 

-- 
Tristan Tarrant
Infinispan Lead
JBoss, a division of Red Hat