7:14 a.m.
Hi,
in my current project, it’s not wanted to use Infinispan as cache in a cluster.
However, I have to deal with the user session and token information.
And as I can remember, in early versions of Keycloak was an option, to store this
information via JPA or MongoDB instead of Infinispan.
Also, I saw there is a User Sessions SPI, and also a User Cache SPI and Realm Cache SPI.
If I implement those SPIs, can I get rid of Infinispan replication in a cluster?
And are there some examples or good starting points? (documentation?)
Regards,
- Niko
8:30 a.m.
Yes, you can replace Infinispan... No, we will not support you :) We
had to reduce the scope of Keycloak. Same reason why we only support
running the server on Wildfly/EAP now. Its just too much extra work.
On 12/11/2015 8:14 AM, Niko Köbler wrote:
Hi,
in my current project, it’s not wanted to use Infinispan as cache in a cluster.
However, I have to deal with the user session and token information.
And as I can remember, in early versions of Keycloak was an option, to store this
information via JPA or MongoDB instead of Infinispan.
Also, I saw there is a User Sessions SPI, and also a User Cache SPI and Realm Cache SPI.
If I implement those SPIs, can I get rid of Infinispan replication in a cluster?
And are there some examples or good starting points? (documentation?)
Regards,
- Niko
_______________________________________________
keycloak-user mailing list
keycloak-user(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/keycloak-user
--
Bill Burke
JBoss, a division of Red Hat
http://bill.burkecentral.com
7:55 p.m.
I highly suggest, from production experience, that you stick with
Infinispan.
On Fri, Dec 11, 2015 at 1:56 PM Bill Burke <bburke(a)redhat.com> wrote:
Yes, you can replace Infinispan... No, we will not support you :)
We
had to reduce the scope of Keycloak. Same reason why we only support
running the server on Wildfly/EAP now. Its just too much extra work.
On 12/11/2015 8:14 AM, Niko Köbler wrote:
> Hi,
>
> in my current project, it’s not wanted to use Infinispan as cache in a
cluster.
> However, I have to deal with the user session and token information.
> And as I can remember, in early versions of Keycloak was an option, to
store this information via JPA or MongoDB instead of Infinispan.
> Also, I saw there is a User Sessions SPI, and also a User Cache SPI and
Realm Cache SPI.
> If I implement those SPIs, can I get rid of Infinispan replication in a
cluster?
> And are there some examples or good starting points? (documentation?)
>
> Regards,
> - Niko
>
>
> _______________________________________________
> keycloak-user mailing list
> keycloak-user(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/keycloak-user
>
--
Bill Burke
JBoss, a division of Red Hat
http://bill.burkecentral.com
_______________________________________________
keycloak-user mailing list
keycloak-user(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/keycloak-user
9:17 a.m.
Ok, I understand.
But then I suggest, if I was right with my assumption about the SPIs, that you should
remove the lines from the documentation. Also, there seems to be some relicts of classes
in the code (if I’m not completely wrong).
Am 12.12.2015 um 02:55 schrieb Scott Rossillo
<srossillo(a)smartling.com>:
I highly suggest, from production experience, that you stick with Infinispan.
On Fri, Dec 11, 2015 at 1:56 PM Bill Burke <bburke(a)redhat.com
<mailto:bburke@redhat.com>> wrote:
Yes, you can replace Infinispan... No, we will not support you :) We
had to reduce the scope of Keycloak. Same reason why we only support
running the server on Wildfly/EAP now. Its just too much extra work.
On 12/11/2015 8:14 AM, Niko Köbler wrote:
> Hi,
>
> in my current project, it’s not wanted to use Infinispan as cache in a cluster.
> However, I have to deal with the user session and token information.
> And as I can remember, in early versions of Keycloak was an option, to store this
information via JPA or MongoDB instead of Infinispan.
> Also, I saw there is a User Sessions SPI, and also a User Cache SPI and Realm Cache
SPI.
> If I implement those SPIs, can I get rid of Infinispan replication in a cluster?
> And are there some examples or good starting points? (documentation?)
>
> Regards,
> - Niko
>
>
> _______________________________________________
> keycloak-user mailing list
> keycloak-user(a)lists.jboss.org <mailto:keycloak-user@lists.jboss.org>
> https://lists.jboss.org/mailman/listinfo/keycloak-user
<https://lists.jboss.org/mailman/listinfo/keycloak-user>
>
--
Bill Burke
JBoss, a division of Red Hat
http://bill.burkecentral.com <http://bill.burkecentral.com/>
_______________________________________________
keycloak-user mailing list
keycloak-user(a)lists.jboss.org <mailto:keycloak-user@lists.jboss.org>
https://lists.jboss.org/mailman/listinfo/keycloak-user
<https://lists.jboss.org/mailman/listinfo/keycloak-user>_______________...
keycloak-user mailing list
keycloak-user(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/keycloak-user
1:50 a.m.
On 12/12/15 16:17, Niko Köbler wrote:
Ok, I understand.
But then I suggest, if I was right with my assumption about the SPIs,
that you should remove the lines from the documentation.
Could you please create
JIRA pointing to incorrect part in the
documentation?
Also, there seems to be some relicts of classes in the code (if I’m
not completely wrong).
Yes, in code we still have MemUserSessionProvider, which is
userSession
implementation based on pure memory. Did you mean this? This is used
just for backwards compatibility in EAP 6.4 (because infinispan local
mode doesn't work correctly here and doesn't support all the stuff we
need) and will be removed.
Btv. what's your motivation to not use infinispan? If you afraid of
cluster communication, you don't need to worry much about it, because if
you run single keycloak through standalone.xml, the infinispan
automatically works in LOCAL mode and there is no any cluster
communication at all.
Or do you want persistent userSession/clientSessions, which will survive
server restart? We already have userSessionPersister SPI, which is used
to persist just "offline" userSessions (those used for retrieve offline
token) but possibly we will extend it with the optional possibility to
persist all user sessions.
Marek
> Am 12.12.2015 um 02:55 schrieb Scott Rossillo
> <srossillo(a)smartling.com <mailto:srossillo@smartling.com>>:
>
> I highly suggest, from production experience, that you stick with
> Infinispan.
> On Fri, Dec 11, 2015 at 1:56 PM Bill Burke <bburke(a)redhat.com
> <mailto:bburke@redhat.com>> wrote:
>
> Yes, you can replace Infinispan... No, we will not support you :) We
> had to reduce the scope of Keycloak. Same reason why we only support
> running the server on Wildfly/EAP now. Its just too much extra work.
>
> On 12/11/2015 8:14 AM, Niko Köbler wrote:
> > Hi,
> >
> > in my current project, it’s not wanted to use Infinispan as
> cache in a cluster.
> > However, I have to deal with the user session and token
> information.
> > And as I can remember, in early versions of Keycloak was an
> option, to store this information via JPA or MongoDB instead of
> Infinispan.
> > Also, I saw there is a User Sessions SPI, and also a User Cache
> SPI and Realm Cache SPI.
> > If I implement those SPIs, can I get rid of Infinispan
> replication in a cluster?
> > And are there some examples or good starting points?
> (documentation?)
> >
> > Regards,
> > - Niko
> >
> >
> > _______________________________________________
> > keycloak-user mailing list
> > keycloak-user(a)lists.jboss.org
> <mailto:keycloak-user@lists.jboss.org>
> > https://lists.jboss.org/mailman/listinfo/keycloak-user
> >
>
> --
> Bill Burke
> JBoss, a division of Red Hat
> http://bill.burkecentral.com <http://bill.burkecentral.com/>
> _______________________________________________
> keycloak-user mailing list
> keycloak-user(a)lists.jboss.org <mailto:keycloak-user@lists.jboss.org>
> https://lists.jboss.org/mailman/listinfo/keycloak-user
>
> _______________________________________________
> keycloak-user mailing list
> keycloak-user(a)lists.jboss.org <mailto:keycloak-user@lists.jboss.org>
> https://lists.jboss.org/mailman/listinfo/keycloak-user
_______________________________________________
keycloak-user mailing list
keycloak-user(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/keycloak-user
4:01 a.m.
Hi Marek,
Am 14.12.2015 um 08:50 schrieb Marek Posolda
<mposolda(a)redhat.com>:
Btv. what's your motivation to not use infinispan? If you afraid of cluster
communication, you don't need to worry much about it, because if you run single
keycloak through standalone.xml, the infinispan automatically works in LOCAL mode and
there is no any cluster communication at all.
My current customer is running his apps in AWS. As known, multicast is not available in
cloud infrastructures. Wildfly/Infinispan Cluster works pretty well with multicast w/o
having to know too much about JGroups config. S3_PING seams to be a viable way to get a
cluster running in AWS.
But additionally, my customer doesn’t have any (deep) knowledge about JBoss
infrastructures and so I’m looking for a way to be able to run Keycloak in a cluster in
AWS without the need to build up deeper knowlegde of JGroups config, for example in
getting rid of Infinispan. But I do understand all the concerns in doing this.
I still have to test S3_PING, if it works as easy as multicast. If yes, we can use it, if
no… I don’t know yet. But this gets offtopic for Keycloak mailinglist, it’s more related
to pure Wildfly/Infinispan.
- Niko
8:58 a.m.
On 12/14/2015 5:01 AM, Niko Köbler wrote:
Hi Marek,
> Am 14.12.2015 um 08:50 schrieb Marek Posolda <mposolda(a)redhat.com
> <mailto:mposolda@redhat.com>>:
>
> Btv. what's your motivation to not use infinispan? If you afraid of
> cluster communication, you don't need to worry much about it, because
> if you run single keycloak through standalone.xml, the infinispan
> automatically works in LOCAL mode and there is no any cluster
> communication at all.
My current customer is running his apps in AWS. As known, multicast is
not available in cloud infrastructures. Wildfly/Infinispan Cluster works
pretty well with multicast w/o having to know too much about JGroups
config. S3_PING seams to be a viable way to get a cluster running in AWS.
But additionally, my customer doesn’t have any (deep) knowledge about
JBoss infrastructures and so I’m looking for a way to be able to run
Keycloak in a cluster in AWS without the need to build up deeper
knowlegde of JGroups config, for example in getting rid of Infinispan.
But I do understand all the concerns in doing this.
I still have to test S3_PING, if it works as easy as multicast. If yes,
we can use it, if no… I don’t know yet. But this gets offtopic for
Keycloak mailinglist, it’s more related to pure Wildfly/Infinispan.
seems to me it would be much easier to get Infinispan working on AWS
than to write and maintain an entire new caching mechanism and hope we
don't refactor the cache SPI.
--
Bill Burke
JBoss, a division of Red Hat
http://bill.burkecentral.com
9:55 a.m.
On 14/12/15 15:58, Bill Burke wrote:
On 12/14/2015 5:01 AM, Niko Köbler wrote:
> Hi Marek,
>
>> Am 14.12.2015 um 08:50 schrieb Marek Posolda <mposolda(a)redhat.com
>> <mailto:mposolda@redhat.com>>:
>>
>> Btv. what's your motivation to not use infinispan? If you afraid of
>> cluster communication, you don't need to worry much about it, because
>> if you run single keycloak through standalone.xml, the infinispan
>> automatically works in LOCAL mode and there is no any cluster
>> communication at all.
> My current customer is running his apps in AWS. As known, multicast is
> not available in cloud infrastructures. Wildfly/Infinispan Cluster works
> pretty well with multicast w/o having to know too much about JGroups
> config. S3_PING seams to be a viable way to get a cluster running in AWS.
> But additionally, my customer doesn’t have any (deep) knowledge about
> JBoss infrastructures and so I’m looking for a way to be able to run
> Keycloak in a cluster in AWS without the need to build up deeper
> knowlegde of JGroups config, for example in getting rid of Infinispan.
> But I do understand all the concerns in doing this.
> I still have to test S3_PING, if it works as easy as multicast. If yes,
> we can use it, if no… I don’t know yet. But this gets offtopic for
> Keycloak mailinglist, it’s more related to pure Wildfly/Infinispan.
>
seems to me it would be much easier to get Infinispan working on AWS
than to write and maintain an entire new caching mechanism and hope we
don't refactor the cache SPI.
+1
I am sure infinispan/JGroups has possibility to run in non-multicast
environment. You may just need to figure how exactly to configure it. So
I agree that this issue is more related to Wildfly/Infinispan itself
than to Keycloak.
You may need to use jgroups protocols like TCP instead of default UDP
and maybe TCPPING (this requires to manually list all your cluster
nodes. But still, it's much better option IMO than rewriting UserSession
SPI)
Marek
9:59 a.m.
On 14/12/15 16:55, Marek Posolda wrote:
On 14/12/15 15:58, Bill Burke wrote:
> On 12/14/2015 5:01 AM, Niko Köbler wrote:
>> Hi Marek,
>>
>>> Am 14.12.2015 um 08:50 schrieb Marek Posolda <mposolda(a)redhat.com
>>> <mailto:mposolda@redhat.com>>:
>>>
>>> Btv. what's your motivation to not use infinispan? If you afraid of
>>> cluster communication, you don't need to worry much about it, because
>>> if you run single keycloak through standalone.xml, the infinispan
>>> automatically works in LOCAL mode and there is no any cluster
>>> communication at all.
>> My current customer is running his apps in AWS. As known, multicast is
>> not available in cloud infrastructures. Wildfly/Infinispan Cluster works
>> pretty well with multicast w/o having to know too much about JGroups
>> config. S3_PING seams to be a viable way to get a cluster running in AWS.
>> But additionally, my customer doesn’t have any (deep) knowledge about
>> JBoss infrastructures and so I’m looking for a way to be able to run
>> Keycloak in a cluster in AWS without the need to build up deeper
>> knowlegde of JGroups config, for example in getting rid of Infinispan.
>> But I do understand all the concerns in doing this.
>> I still have to test S3_PING, if it works as easy as multicast. If yes,
>> we can use it, if no… I don’t know yet. But this gets offtopic for
>> Keycloak mailinglist, it’s more related to pure Wildfly/Infinispan.
>>
> seems to me it would be much easier to get Infinispan working on AWS
> than to write and maintain an entire new caching mechanism and hope we
> don't refactor the cache SPI.
>
>
+1
I am sure infinispan/JGroups has possibility to run in non-multicast
environment. You may just need to figure how exactly to configure it. So
I agree that this issue is more related to Wildfly/Infinispan itself
than to Keycloak.
You may need to use jgroups protocols like TCP instead of default UDP
and maybe TCPPING (this requires to manually list all your cluster
nodes. But still, it's much better option IMO than rewriting UserSession
SPI)
Btv. if TCPPING or S3_PING is an issue, there is also AWS_PING
http://www.jgroups.org/manual-3.x/html/protlist.html#d0e5100 , but it's
not official part of jgroups.
Marek
Marek
_______________________________________________
keycloak-user mailing list
keycloak-user(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/keycloak-user
3:24 p.m.
AWS was why we didn’t use Infinispan to begin with. That and it’s even more complicated
when you deploy using Amazon’s Docker service (ECS) or Beanstalk.
It’s too bad Infinispan / JGroups are beasts when the out of the box configuration can’t
be used. I’m planning to document this as we fix but I’d avoid S3_PING and use JDBC_PING.
You already need JDBC for the Keycloak DB, unless you’re using Mongo and it’s easier to
test locally.
TCPPING will bite you on AWS if Amazon decides to replace one of your instances (which it
does occasionally w/ECS or Beanstalk).
Best,
Scott
Scott Rossillo
Smartling | Senior Software Engineer
srossillo(a)smartling.com
<http://www.sigstr.com/>
On Dec 14, 2015, at 10:59 AM, Marek Posolda
<mposolda(a)redhat.com> wrote:
On 14/12/15 16:55, Marek Posolda wrote:
> On 14/12/15 15:58, Bill Burke wrote:
>> On 12/14/2015 5:01 AM, Niko Köbler wrote:
>>> Hi Marek,
>>>
>>>> Am 14.12.2015 um 08:50 schrieb Marek Posolda <mposolda(a)redhat.com
>>>> <mailto:mposolda@redhat.com>>:
>>>>
>>>> Btv. what's your motivation to not use infinispan? If you afraid of
>>>> cluster communication, you don't need to worry much about it,
because
>>>> if you run single keycloak through standalone.xml, the infinispan
>>>> automatically works in LOCAL mode and there is no any cluster
>>>> communication at all.
>>> My current customer is running his apps in AWS. As known, multicast is
>>> not available in cloud infrastructures. Wildfly/Infinispan Cluster works
>>> pretty well with multicast w/o having to know too much about JGroups
>>> config. S3_PING seams to be a viable way to get a cluster running in AWS.
>>> But additionally, my customer doesn’t have any (deep) knowledge about
>>> JBoss infrastructures and so I’m looking for a way to be able to run
>>> Keycloak in a cluster in AWS without the need to build up deeper
>>> knowlegde of JGroups config, for example in getting rid of Infinispan.
>>> But I do understand all the concerns in doing this.
>>> I still have to test S3_PING, if it works as easy as multicast. If yes,
>>> we can use it, if no… I don’t know yet. But this gets offtopic for
>>> Keycloak mailinglist, it’s more related to pure Wildfly/Infinispan.
>>>
>> seems to me it would be much easier to get Infinispan working on AWS
>> than to write and maintain an entire new caching mechanism and hope we
>> don't refactor the cache SPI.
>>
>>
> +1
>
> I am sure infinispan/JGroups has possibility to run in non-multicast
> environment. You may just need to figure how exactly to configure it. So
> I agree that this issue is more related to Wildfly/Infinispan itself
> than to Keycloak.
>
> You may need to use jgroups protocols like TCP instead of default UDP
> and maybe TCPPING (this requires to manually list all your cluster
> nodes. But still, it's much better option IMO than rewriting UserSession
> SPI)
Btv. if TCPPING or S3_PING is an issue, there is also AWS_PING
http://www.jgroups.org/manual-3.x/html/protlist.html#d0e5100 , but it's
not official part of jgroups.
Marek
>
> Marek
> _______________________________________________
> keycloak-user mailing list
> keycloak-user(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/keycloak-user
_______________________________________________
keycloak-user mailing list
keycloak-user(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/keycloak-user
4:32 p.m.
CCing Alan Field from RH Infinispan team and forwarding his question:
I'd like to know which configuration files you are using and why is is
harder to use with Amazon’s Docker service (ECS) or Beanstalk. I'd also be
interested in how big a cluster you are using in AWS.
On 14/12/15 22:24, Scott Rossillo wrote:
AWS was why we didn’t use Infinispan to begin with. That and it’s
even more complicated when you deploy using Amazon’s Docker service
(ECS) or Beanstalk.
It’s too bad Infinispan / JGroups are beasts when the out of the box
configuration can’t be used. I’m planning to document this as we fix
but I’d avoid S3_PING and use JDBC_PING. You already need JDBC for the
Keycloak DB, unless you’re using Mongo and it’s easier to test locally.
TCPPING will bite you on AWS if Amazon decides to replace one of your
instances (which it does occasionally w/ECS or Beanstalk).
Best,
Scott
Scott Rossillo
Smartling | Senior Software Engineer
srossillo(a)smartling.com <mailto:srossillo@smartling.com>
Powered by Sigstr <http://www.sigstr.com/>
> On Dec 14, 2015, at 10:59 AM, Marek Posolda <mposolda(a)redhat.com
> <mailto:mposolda@redhat.com>> wrote:
>
> On 14/12/15 16:55, Marek Posolda wrote:
>> On 14/12/15 15:58, Bill Burke wrote:
>>> On 12/14/2015 5:01 AM, Niko Köbler wrote:
>>>> Hi Marek,
>>>>
>>>>> Am 14.12.2015 um 08:50 schrieb Marek Posolda <mposolda(a)redhat.com
>>>>> <mailto:mposolda@redhat.com>
>>>>> <mailto:mposolda@redhat.com>>:
>>>>>
>>>>> Btv. what's your motivation to not use infinispan? If you afraid
of
>>>>> cluster communication, you don't need to worry much about it,
because
>>>>> if you run single keycloak through standalone.xml, the infinispan
>>>>> automatically works in LOCAL mode and there is no any cluster
>>>>> communication at all.
>>>> My current customer is running his apps in AWS. As known, multicast is
>>>> not available in cloud infrastructures. Wildfly/Infinispan Cluster
>>>> works
>>>> pretty well with multicast w/o having to know too much about JGroups
>>>> config. S3_PING seams to be a viable way to get a cluster running
>>>> in AWS.
>>>> But additionally, my customer doesn’t have any (deep) knowledge about
>>>> JBoss infrastructures and so I’m looking for a way to be able to run
>>>> Keycloak in a cluster in AWS without the need to build up deeper
>>>> knowlegde of JGroups config, for example in getting rid of Infinispan.
>>>> But I do understand all the concerns in doing this.
>>>> I still have to test S3_PING, if it works as easy as multicast. If
>>>> yes,
>>>> we can use it, if no… I don’t know yet. But this gets offtopic for
>>>> Keycloak mailinglist, it’s more related to pure Wildfly/Infinispan.
>>>>
>>> seems to me it would be much easier to get Infinispan working on AWS
>>> than to write and maintain an entire new caching mechanism and hope we
>>> don't refactor the cache SPI.
>>>
>>>
>> +1
>>
>> I am sure infinispan/JGroups has possibility to run in non-multicast
>> environment. You may just need to figure how exactly to configure it. So
>> I agree that this issue is more related to Wildfly/Infinispan itself
>> than to Keycloak.
>>
>> You may need to use jgroups protocols like TCP instead of default UDP
>> and maybe TCPPING (this requires to manually list all your cluster
>> nodes. But still, it's much better option IMO than rewriting UserSession
>> SPI)
> Btv. if TCPPING or S3_PING is an issue, there is also AWS_PING
> http://www.jgroups.org/manual-3.x/html/protlist.html#d0e5100 , but it's
> not official part of jgroups.
>
> Marek
>>
>> Marek
>> _______________________________________________
>> keycloak-user mailing list
>> keycloak-user(a)lists.jboss.org <mailto:keycloak-user@lists.jboss.org>
>> https://lists.jboss.org/mailman/listinfo/keycloak-user
>
> _______________________________________________
> keycloak-user mailing list
> keycloak-user(a)lists.jboss.org <mailto:keycloak-user@lists.jboss.org>
> https://lists.jboss.org/mailman/listinfo/keycloak-user
5:31 p.m.
There are two issues:
1. Infinispan relies on JGroups, which is difficult to configure correctly with the
various ping techniques that aren’t UDP multicast. I can elaborate on each one that we
tested but it’s just generally complex to get right. That’s not to say it’s impossible or
the biggest reason this is complicated on ECS or _insert container service here_, see #2
for that.
2. It is difficult to do discovery correctly with JGroups and Docker. Non-privileged
Docker instances - the default and recommend type - do not implicitly know their host’s
IP. This causes IP mismatches between what JGroups thinks the machine’s IP is and what it
actually is when connecting to hosts on different machines. This is the main issue and
it’s not the fault of JGroups per se, but there’s no simple work around.
Take for example a simple 2 node cluster:
Node 1 comes up on the docker0 interface of host A with the IP address 172.16.0.4. The
host A IP is 10.10.0.100.
Node 2 comes up on the docker0 interface of host B with the IP address 172.16.0.8. The
host B IP is 10.10.0.108.
The 172.16 network is not routable between hosts (by design). Docker does port forwarding
for ports we wish to expose to this works fine for HTTP/HTTPS but not the cluster
traffic.
So Node 1 will advertise itself as having IP 172.16.0.4 while Node 2 advertises
172.16.0.8. The two cannot talk to each other by default. However, using the hard coded
IPs and TCP PING, we can set external_addr on Node 1 to 10.10.0.100 and external_addr on
Node 2 to 10.10.0.108 and set initial_hosts to 10.10.0.100, 10.10.0.108. This will cause
the nodes to discover each other. However, they will not form a cluster. The nodes will
reject the handshake thinking they’re not actually 10.10.0.100 or 10.10.0.108
respectively.
I’d like to discuss further and I can share where we’ve gotten so far with workarounds to
this but it may be better to get into the weeds on another list.
Let me know what you think.
Best,
Scott
Scott Rossillo
Smartling | Senior Software Engineer
srossillo(a)smartling.com
<http://www.sigstr.com/>
On Dec 14, 2015, at 5:32 PM, Marek Posolda
<mposolda(a)redhat.com> wrote:
CCing Alan Field from RH Infinispan team and forwarding his question:
I'd like to know which configuration files you are using and why is is
harder to use with Amazon’s Docker service (ECS) or Beanstalk. I'd also be
interested in how big a cluster you are using in AWS.
On 14/12/15 22:24, Scott Rossillo wrote:
> AWS was why we didn’t use Infinispan to begin with. That and it’s even more
complicated when you deploy using Amazon’s Docker service (ECS) or Beanstalk.
>
> It’s too bad Infinispan / JGroups are beasts when the out of the box configuration
can’t be used. I’m planning to document this as we fix but I’d avoid S3_PING and use
JDBC_PING. You already need JDBC for the Keycloak DB, unless you’re using Mongo and it’s
easier to test locally.
>
> TCPPING will bite you on AWS if Amazon decides to replace one of your instances
(which it does occasionally w/ECS or Beanstalk).
>
> Best,
> Scott
>
> Scott Rossillo
> Smartling | Senior Software Engineer
> srossillo(a)smartling.com <mailto:srossillo@smartling.com>
>
> <http://www.sigstr.com/>
>> On Dec 14, 2015, at 10:59 AM, Marek Posolda <mposolda(a)redhat.com
<mailto:mposolda@redhat.com>> wrote:
>>
>> On 14/12/15 16:55, Marek Posolda wrote:
>>> On 14/12/15 15:58, Bill Burke wrote:
>>>> On 12/14/2015 5:01 AM, Niko Köbler wrote:
>>>>> Hi Marek,
>>>>>
>>>>>> Am 14.12.2015 um 08:50 schrieb Marek Posolda <
<mailto:mposolda@redhat.com>mposolda@redhat.com <mailto:mposolda@redhat.com>
>>>>>> <mailto:mposolda@redhat.com
<mailto:mposolda@redhat.com>>>:
>>>>>>
>>>>>> Btv. what's your motivation to not use infinispan? If you
afraid of
>>>>>> cluster communication, you don't need to worry much about it,
because
>>>>>> if you run single keycloak through standalone.xml, the
infinispan
>>>>>> automatically works in LOCAL mode and there is no any cluster
>>>>>> communication at all.
>>>>> My current customer is running his apps in AWS. As known, multicast
is
>>>>> not available in cloud infrastructures. Wildfly/Infinispan Cluster
works
>>>>> pretty well with multicast w/o having to know too much about JGroups
>>>>> config. S3_PING seams to be a viable way to get a cluster running in
AWS.
>>>>> But additionally, my customer doesn’t have any (deep) knowledge
about
>>>>> JBoss infrastructures and so I’m looking for a way to be able to run
>>>>> Keycloak in a cluster in AWS without the need to build up deeper
>>>>> knowlegde of JGroups config, for example in getting rid of
Infinispan.
>>>>> But I do understand all the concerns in doing this.
>>>>> I still have to test S3_PING, if it works as easy as multicast. If
yes,
>>>>> we can use it, if no… I don’t know yet. But this gets offtopic for
>>>>> Keycloak mailinglist, it’s more related to pure Wildfly/Infinispan.
>>>>>
>>>> seems to me it would be much easier to get Infinispan working on AWS
>>>> than to write and maintain an entire new caching mechanism and hope we
>>>> don't refactor the cache SPI.
>>>>
>>>>
>>> +1
>>>
>>> I am sure infinispan/JGroups has possibility to run in non-multicast
>>> environment. You may just need to figure how exactly to configure it. So
>>> I agree that this issue is more related to Wildfly/Infinispan itself
>>> than to Keycloak.
>>>
>>> You may need to use jgroups protocols like TCP instead of default UDP
>>> and maybe TCPPING (this requires to manually list all your cluster
>>> nodes. But still, it's much better option IMO than rewriting UserSession
>>> SPI)
>> Btv. if TCPPING or S3_PING is an issue, there is also AWS_PING
>> http://www.jgroups.org/manual-3.x/html/protlist.html#d0e5100
<http://www.jgroups.org/manual-3.x/html/protlist.html#d0e5100> , but it's
>> not official part of jgroups.
>>
>> Marek
>>>
>>> Marek
>>> _______________________________________________
>>> keycloak-user mailing list
>>> keycloak-user(a)lists.jboss.org <mailto:keycloak-user@lists.jboss.org>
>>> https://lists.jboss.org/mailman/listinfo/keycloak-user
<https://lists.jboss.org/mailman/listinfo/keycloak-user>
>>
>> _______________________________________________
>> keycloak-user mailing list
>> keycloak-user(a)lists.jboss.org <mailto:keycloak-user@lists.jboss.org>
>> https://lists.jboss.org/mailman/listinfo/keycloak-user
<https://lists.jboss.org/mailman/listinfo/keycloak-user>
>
8:47 a.m.
I've also been working on setting up clustered Keycloak on Docker containers in EC2
and would be interested in any potential solutions for this configuration.
Alternatively I've set up on EC2 without Docker with S3_PING. I'd be interested in
hearing about the issues with this configuration.
From: Scott Rossillo
<srossillo@smartling.com<mailto:srossillo@smartling.com>>
Date: Mon, 14 Dec 2015 18:31:30 -0500
To: Marek Posolda <mposolda@redhat.com<mailto:mposolda@redhat.com>>,
<afield@redhat.com<mailto:afield@redhat.com>>
Cc: keycloak-user
<keycloak-user@lists.jboss.org<mailto:keycloak-user@lists.jboss.org>>
Subject: Re: [keycloak-user] Replace use of Infinispan with User Sessions SPI ?
There are two issues:
1. Infinispan relies on JGroups, which is difficult to configure correctly with the
various ping techniques that aren’t UDP multicast. I can elaborate on each one that we
tested but it’s just generally complex to get right. That’s not to say it’s impossible or
the biggest reason this is complicated on ECS or _insert container service here_, see #2
for that.
2. It is difficult to do discovery correctly with JGroups and Docker. Non-privileged
Docker instances - the default and recommend type - do not implicitly know their host’s
IP. This causes IP mismatches between what JGroups thinks the machine’s IP is and what it
actually is when connecting to hosts on different machines. This is the main issue and
it’s not the fault of JGroups per se, but there’s no simple work around.
Take for example a simple 2 node cluster:
Node 1 comes up on the docker0 interface of host A with the IP address 172.16.0.4. The
host A IP is 10.10.0.100.
Node 2 comes up on the docker0 interface of host B with the IP address 172.16.0.8. The
host B IP is 10.10.0.108.
The 172.16 network is not routable between hosts (by design). Docker does port forwarding
for ports we wish to expose to this works fine for HTTP/HTTPS but not the cluster
traffic.
So Node 1 will advertise itself as having IP 172.16.0.4 while Node 2 advertises
172.16.0.8. The two cannot talk to each other by default. However, using the hard coded
IPs and TCP PING, we can set external_addr on Node 1 to 10.10.0.100 and external_addr on
Node 2 to 10.10.0.108 and set initial_hosts to 10.10.0.100, 10.10.0.108. This will cause
the nodes to discover each other. However, they will not form a cluster. The nodes will
reject the handshake thinking they’re not actually 10.10.0.100 or 10.10.0.108
respectively.
I’d like to discuss further and I can share where we’ve gotten so far with workarounds to
this but it may be better to get into the weeds on another list.
Let me know what you think.
Best,
Scott
Scott Rossillo
Smartling | Senior Software Engineer
srossillo@smartling.com<mailto:srossillo@smartling.com>
[Powered by Sigstr]<http://www.sigstr.com/>
On Dec 14, 2015, at 5:32 PM, Marek Posolda
<mposolda@redhat.com<mailto:mposolda@redhat.com>> wrote:
CCing Alan Field from RH Infinispan team and forwarding his question:
I'd like to know which configuration files you are using and why is is
harder to use with Amazon’s Docker service (ECS) or Beanstalk. I'd also be
interested in how big a cluster you are using in AWS.
On 14/12/15 22:24, Scott Rossillo wrote:
AWS was why we didn’t use Infinispan to begin with. That and it’s even more complicated
when you deploy using Amazon’s Docker service (ECS) or Beanstalk.
It’s too bad Infinispan / JGroups are beasts when the out of the box configuration can’t
be used. I’m planning to document this as we fix but I’d avoid S3_PING and use JDBC_PING.
You already need JDBC for the Keycloak DB, unless you’re using Mongo and it’s easier to
test locally.
TCPPING will bite you on AWS if Amazon decides to replace one of your instances (which it
does occasionally w/ECS or Beanstalk).
Best,
Scott
Scott Rossillo
Smartling | Senior Software Engineer
srossillo@smartling.com<mailto:srossillo@smartling.com>
[Powered by Sigstr]<http://www.sigstr.com/>
On Dec 14, 2015, at 10:59 AM, Marek Posolda
<mposolda@redhat.com<mailto:mposolda@redhat.com>> wrote:
On 14/12/15 16:55, Marek Posolda wrote:
On 14/12/15 15:58, Bill Burke wrote:
On 12/14/2015 5:01 AM, Niko Köbler wrote:
Hi Marek,
Am 14.12.2015 um 08:50 schrieb Marek Posolda
<<mailto:mposolda@redhat.com>mposolda@redhat.com<mailto:mposolda@redhat.com>
<mailto:mposolda@redhat.com>>:
Btv. what's your motivation to not use infinispan? If you afraid of
cluster communication, you don't need to worry much about it, because
if you run single keycloak through standalone.xml, the infinispan
automatically works in LOCAL mode and there is no any cluster
communication at all.
My current customer is running his apps in AWS. As known, multicast is
not available in cloud infrastructures. Wildfly/Infinispan Cluster works
pretty well with multicast w/o having to know too much about JGroups
config. S3_PING seams to be a viable way to get a cluster running in AWS.
But additionally, my customer doesn’t have any (deep) knowledge about
JBoss infrastructures and so I’m looking for a way to be able to run
Keycloak in a cluster in AWS without the need to build up deeper
knowlegde of JGroups config, for example in getting rid of Infinispan.
But I do understand all the concerns in doing this.
I still have to test S3_PING, if it works as easy as multicast. If yes,
we can use it, if no… I don’t know yet. But this gets offtopic for
Keycloak mailinglist, it’s more related to pure Wildfly/Infinispan.
seems to me it would be much easier to get Infinispan working on AWS
than to write and maintain an entire new caching mechanism and hope we
don't refactor the cache SPI.
+1
I am sure infinispan/JGroups has possibility to run in non-multicast
environment. You may just need to figure how exactly to configure it. So
I agree that this issue is more related to Wildfly/Infinispan itself
than to Keycloak.
You may need to use jgroups protocols like TCP instead of default UDP
and maybe TCPPING (this requires to manually list all your cluster
nodes. But still, it's much better option IMO than rewriting UserSession
SPI)
Btv. if TCPPING or S3_PING is an issue, there is also AWS_PING
http://www.jgroups.org/manual-3.x/html/protlist.html#d0e5100 , but it's
not official part of jgroups.
Marek
Marek
_______________________________________________
keycloak-user mailing list
keycloak-user@lists.jboss.org<mailto:keycloak-user@lists.jboss.org>
https://lists.jboss.org/mailman/listinfo/keycloak-user
_______________________________________________
keycloak-user mailing list
keycloak-user@lists.jboss.org<mailto:keycloak-user@lists.jboss.org>
https://lists.jboss.org/mailman/listinfo/keycloak-user
_______________________________________________ keycloak-user mailing list
keycloak-user@lists.jboss.org<mailto:keycloak-user@lists.jboss.org>
https://lists.jboss.org/mailman/listinfo/keycloak-user
12:53 p.m.
We will go for the first run with EC2 and S3_PING, but w/o Docker.
If we/you/whoever will find a proper solution (possibly on the jgroups mailinglist), we
will test this.
Seams that everybody is aware of the Docker/Cloud/Multicast issues, but no-one has a
proper solution, only workarounds. :(
Am 15.12.2015 um 15:47 schrieb Paul Blair
<pblair(a)clearme.com>:
I've also been working on setting up clustered Keycloak on Docker containers in EC2
and would be interested in any potential solutions for this configuration.
Alternatively I've set up on EC2 without Docker with S3_PING. I'd be interested
in hearing about the issues with this configuration.
From: Scott Rossillo <srossillo(a)smartling.com
<mailto:srossillo@smartling.com>>
Date: Mon, 14 Dec 2015 18:31:30 -0500
To: Marek Posolda <mposolda(a)redhat.com <mailto:mposolda@redhat.com>>,
<afield(a)redhat.com <mailto:afield@redhat.com>>
Cc: keycloak-user <keycloak-user(a)lists.jboss.org
<mailto:keycloak-user@lists.jboss.org>>
Subject: Re: [keycloak-user] Replace use of Infinispan with User Sessions SPI ?
There are two issues:
1. Infinispan relies on JGroups, which is difficult to configure correctly with the
various ping techniques that aren’t UDP multicast. I can elaborate on each one that we
tested but it’s just generally complex to get right. That’s not to say it’s impossible or
the biggest reason this is complicated on ECS or _insert container service here_, see #2
for that.
2. It is difficult to do discovery correctly with JGroups and Docker. Non-privileged
Docker instances - the default and recommend type - do not implicitly know their host’s
IP. This causes IP mismatches between what JGroups thinks the machine’s IP is and what it
actually is when connecting to hosts on different machines. This is the main issue and
it’s not the fault of JGroups per se, but there’s no simple work around.
Take for example a simple 2 node cluster:
Node 1 comes up on the docker0 interface of host A with the IP address 172.16.0.4. The
host A IP is 10.10.0.100.
Node 2 comes up on the docker0 interface of host B with the IP address 172.16.0.8. The
host B IP is 10.10.0.108.
The 172.16 network is not routable between hosts (by design). Docker does port forwarding
for ports we wish to expose to this works fine for HTTP/HTTPS but not the cluster
traffic.
So Node 1 will advertise itself as having IP 172.16.0.4 while Node 2 advertises
172.16.0.8. The two cannot talk to each other by default. However, using the hard coded
IPs and TCP PING, we can set external_addr on Node 1 to 10.10.0.100 and external_addr on
Node 2 to 10.10.0.108 and set initial_hosts to 10.10.0.100, 10.10.0.108. This will cause
the nodes to discover each other. However, they will not form a cluster. The nodes will
reject the handshake thinking they’re not actually 10.10.0.100 or 10.10.0.108
respectively.
I’d like to discuss further and I can share where we’ve gotten so far with workarounds to
this but it may be better to get into the weeds on another list.
Let me know what you think.
Best,
Scott
Scott Rossillo
Smartling | Senior Software Engineer
srossillo(a)smartling.com <mailto:srossillo@smartling.com>
<http://www.sigstr.com/>
> On Dec 14, 2015, at 5:32 PM, Marek Posolda <mposolda(a)redhat.com
<mailto:mposolda@redhat.com>> wrote:
>
> CCing Alan Field from RH Infinispan team and forwarding his question:
> I'd like to know which configuration files you are using and why is is
> harder to use with Amazon’s Docker service (ECS) or Beanstalk. I'd also be
> interested in how big a cluster you are using in AWS.
>
>
>
> On 14/12/15 22:24, Scott Rossillo wrote:
>> AWS was why we didn’t use Infinispan to begin with. That and it’s even more
complicated when you deploy using Amazon’s Docker service (ECS) or Beanstalk.
>>
>> It’s too bad Infinispan / JGroups are beasts when the out of the box
configuration can’t be used. I’m planning to document this as we fix but I’d avoid S3_PING
and use JDBC_PING. You already need JDBC for the Keycloak DB, unless you’re using Mongo
and it’s easier to test locally.
>>
>> TCPPING will bite you on AWS if Amazon decides to replace one of your instances
(which it does occasionally w/ECS or Beanstalk).
>>
>> Best,
>> Scott
>>
>> Scott Rossillo
>> Smartling | Senior Software Engineer
>> srossillo(a)smartling.com <mailto:srossillo@smartling.com>
>>
>> <http://www.sigstr.com/>
>>> On Dec 14, 2015, at 10:59 AM, Marek Posolda <mposolda(a)redhat.com
<mailto:mposolda@redhat.com>> wrote:
>>>
>>> On 14/12/15 16:55, Marek Posolda wrote:
>>>> On 14/12/15 15:58, Bill Burke wrote:
>>>>> On 12/14/2015 5:01 AM, Niko Köbler wrote:
>>>>>> Hi Marek,
>>>>>>
>>>>>>> Am 14.12.2015 um 08:50 schrieb Marek Posolda <
<mailto:mposolda@redhat.com>mposolda@redhat.com <mailto:mposolda@redhat.com>
>>>>>>> <mailto:mposolda@redhat.com
<mailto:mposolda@redhat.com>>>:
>>>>>>>
>>>>>>> Btv. what's your motivation to not use infinispan? If you
afraid of
>>>>>>> cluster communication, you don't need to worry much about
it, because
>>>>>>> if you run single keycloak through standalone.xml, the
infinispan
>>>>>>> automatically works in LOCAL mode and there is no any
cluster
>>>>>>> communication at all.
>>>>>> My current customer is running his apps in AWS. As known,
multicast is
>>>>>> not available in cloud infrastructures. Wildfly/Infinispan
Cluster works
>>>>>> pretty well with multicast w/o having to know too much about
JGroups
>>>>>> config. S3_PING seams to be a viable way to get a cluster running
in AWS.
>>>>>> But additionally, my customer doesn’t have any (deep) knowledge
about
>>>>>> JBoss infrastructures and so I’m looking for a way to be able to
run
>>>>>> Keycloak in a cluster in AWS without the need to build up deeper
>>>>>> knowlegde of JGroups config, for example in getting rid of
Infinispan.
>>>>>> But I do understand all the concerns in doing this.
>>>>>> I still have to test S3_PING, if it works as easy as multicast.
If yes,
>>>>>> we can use it, if no… I don’t know yet. But this gets offtopic
for
>>>>>> Keycloak mailinglist, it’s more related to pure
Wildfly/Infinispan.
>>>>>>
>>>>> seems to me it would be much easier to get Infinispan working on AWS
>>>>> than to write and maintain an entire new caching mechanism and hope
we
>>>>> don't refactor the cache SPI.
>>>>>
>>>>>
>>>> +1
>>>>
>>>> I am sure infinispan/JGroups has possibility to run in non-multicast
>>>> environment. You may just need to figure how exactly to configure it. So
>>>> I agree that this issue is more related to Wildfly/Infinispan itself
>>>> than to Keycloak.
>>>>
>>>> You may need to use jgroups protocols like TCP instead of default UDP
>>>> and maybe TCPPING (this requires to manually list all your cluster
>>>> nodes. But still, it's much better option IMO than rewriting
UserSession
>>>> SPI)
>>> Btv. if TCPPING or S3_PING is an issue, there is also AWS_PING
>>> http://www.jgroups.org/manual-3.x/html/protlist.html#d0e5100
<http://www.jgroups.org/manual-3.x/html/protlist.html#d0e5100> , but it's
>>> not official part of jgroups.
>>>
>>> Marek
>>>>
>>>> Marek
>>>> _______________________________________________
>>>> keycloak-user mailing list
>>>> keycloak-user(a)lists.jboss.org
<mailto:keycloak-user@lists.jboss.org>
>>>> https://lists.jboss.org/mailman/listinfo/keycloak-user
<https://lists.jboss.org/mailman/listinfo/keycloak-user>
>>>
>>> _______________________________________________
>>> keycloak-user mailing list
>>> keycloak-user(a)lists.jboss.org <mailto:keycloak-user@lists.jboss.org>
>>> https://lists.jboss.org/mailman/listinfo/keycloak-user
<https://lists.jboss.org/mailman/listinfo/keycloak-user>
>>
>
_______________________________________________ keycloak-user mailing list
keycloak-user(a)lists.jboss.org
<mailto:keycloak-user@lists.jboss.org>https://lists.jboss.org/mailman/listinfo/keycloak-user
<https://lists.jboss.org/mailman/listinfo/keycloak-user>_______________...
keycloak-user mailing list
keycloak-user(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/keycloak-user
1:13 p.m.
New subject: Replace use of Infinispan with User Sessions SPI ?
Just to be clear, I have successfully tested Infinispan library and server mode clusters
on EC2 using S3_PING, TCP, and the internal EC2 IP addresses. None of the cloud providers
support multicast. The Docker case is a little different though, because of the issues
with getting access to the IP address.
Thanks,
Alan
----- Original Message -----
From: "Niko Köbler" <niko(a)n-k.de>
To: "Paul Blair" <pblair(a)clearme.com>
Cc: "keycloak-user" <keycloak-user(a)lists.jboss.org>
Sent: Tuesday, December 15, 2015 1:53:18 PM
Subject: Re: [keycloak-user] Replace use of Infinispan with User Sessions SPI
?
We will go for the first run with EC2 and S3_PING, but w/o Docker.
If we/you/whoever will find a proper solution (possibly on the jgroups
mailinglist), we will test this.
Seams that everybody is aware of the Docker/Cloud/Multicast issues,
but
no-one has a proper solution, only workarounds. :(
> Am 15.12.2015 um 15:47 schrieb Paul Blair <
pblair(a)clearme.com >:
> I've also been working on setting up clustered Keycloak on
Docker
> containers
> in EC2 and would be interested in any potential solutions for this
> configuration.
> Alternatively I've set up on EC2 without Docker with
S3_PING. I'd be
> interested in hearing about the issues with this configuration.
> From: Scott Rossillo < srossillo(a)smartling.com >
> Date: Mon, 14 Dec 2015 18:31:30 -0500
> To: Marek Posolda < mposolda(a)redhat.com >, < afield(a)redhat.com >
> Cc: keycloak-user < keycloak-user(a)lists.jboss.org >
> Subject: Re: [keycloak-user] Replace use of Infinispan with User Sessions
> SPI
> ?
> There are two issues:
> 1. Infinispan relies on JGroups, which is difficult to configure
correctly
> with the various ping techniques that aren’t UDP multicast. I can elaborate
> on each one that we tested but it’s just generally complex to get right.
> That’s not to say it’s impossible or the biggest reason this is complicated
> on ECS or _insert container service here_, see #2 for that.
> 2. It is difficult to do discovery correctly with JGroups and
Docker.
> Non-privileged Docker instances - the default and recommend type - do not
> implicitly know their host’s IP. This causes IP mismatches between what
> JGroups thinks the machine’s IP is and what it actually is when connecting
> to hosts on different machines. This is the main issue and it’s not the
> fault of JGroups per se, but there’s no simple work around.
> Take for example a simple 2 node cluster:
> Node 1 comes up on the docker0 interface of host A with the IP
address
> 172.16.0.4. The host A IP is 10.10.0.100.
> Node 2 comes up on the docker0 interface of host B with the IP address
> 172.16.0.8. The host B IP is 10.10.0.108.
> The 172.16 network is not routable between hosts (by design).
Docker does
> port forwarding for ports we wish to expose to this works fine for
> HTTP/HTTPS but not the cluster traffic.
> So Node 1 will advertise itself as having IP 172.16.0.4 while
Node 2
> advertises 172.16.0.8. The two cannot talk to each other by default.
> However, using the hard coded IPs and TCP PING, we can set external_addr on
> Node 1 to 10.10.0.100 and external_addr on Node 2 to 10.10.0.108 and set
> initial_hosts to 10.10.0.100, 10.10.0.108. This will cause the nodes to
> discover each other. However, they will not form a cluster. The nodes will
> reject the handshake thinking they’re not actually 10.10.0.100 or
> 10.10.0.108 respectively.
> I’d like to discuss further and I can share where we’ve gotten
so far with
> workarounds to this but it may be better to get into the weeds on another
> list.
> Let me know what you think.
> Best,
> Scott
> Scott Rossillo
> Smartling | Senior Software Engineer
> srossillo(a)smartling.com
> > On Dec 14, 2015, at 5:32 PM, Marek Posolda <
mposolda(a)redhat.com > wrote:
>
> > CCing Alan Field from RH Infinispan team and forwarding his
question:
>
> > I'd like to know which configuration files you are using and why is is
>
> > harder to use with Amazon’s Docker service (ECS) or Beanstalk. I'd also
> > be
>
> > interested in how big a cluster you are using in AWS.
>
> > On 14/12/15 22:24, Scott Rossillo wrote:
>
> > > AWS was why we didn’t use Infinispan to begin with.
That and it’s even
> > > more
> > > complicated when you deploy using Amazon’s Docker service (ECS) or
> > > Beanstalk.
> >
>
> > > It’s too bad Infinispan / JGroups are beasts when the
out of the box
> > > configuration can’t be used. I’m planning to document this as we fix
> > > but
> > > I’d
> > > avoid S3_PING and use JDBC_PING. You already need JDBC for the Keycloak
> > > DB,
> > > unless you’re using Mongo and it’s easier to test locally.
> >
>
> > > TCPPING will bite you on AWS if Amazon decides to
replace one of your
> > > instances (which it does occasionally w/ECS or Beanstalk).
> >
>
> > > Best,
> >
>
> > > Scott
> >
>
> > > Scott Rossillo
> >
>
> > > Smartling | Senior Software Engineer
> >
>
> > > srossillo(a)smartling.com
> >
>
> > > > On Dec 14, 2015, at 10:59 AM, Marek Posolda <
mposolda(a)redhat.com >
> > > > wrote:
> > >
> >
>
> > > > On 14/12/15 16:55, Marek Posolda wrote:
> > >
> >
>
> > > > > On 14/12/15 15:58, Bill Burke wrote:
> > > >
> > >
> >
>
> > > > > > On 12/14/2015 5:01 AM, Niko Köbler
wrote:
> > > > >
> > > >
> > >
> >
>
> > > > > > > Hi Marek,
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > Am 14.12.2015 um 08:50
schrieb Marek Posolda <
> > > > > > > > mposolda(a)redhat.com
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > < mailto:mposolda@redhat.com >>:
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > Btv. what's your
motivation to not use infinispan? If you
> > > > > > > > afraid
> > > > > > > > of
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > cluster communication, you don't need to
worry much about it,
> > > > > > > > because
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > if you run single keycloak through
standalone.xml, the
> > > > > > > > infinispan
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > automatically works in LOCAL mode and there is no
any cluster
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > communication at all.
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > My current customer is running his
apps in AWS. As known,
> > > > > > > multicast
> > > > > > > is
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > not available in cloud infrastructures.
Wildfly/Infinispan
> > > > > > > Cluster
> > > > > > > works
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > pretty well with multicast w/o having to know too much
about
> > > > > > > JGroups
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > config. S3_PING seams to be a viable way to get a
cluster
> > > > > > > running
> > > > > > > in
> > > > > > > AWS.
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > But additionally, my customer doesn’t have any (deep)
knowledge
> > > > > > > about
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > JBoss infrastructures and so I’m looking for a way to
be able
> > > > > > > to
> > > > > > > run
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > Keycloak in a cluster in AWS without the need to build
up
> > > > > > > deeper
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > knowlegde of JGroups config, for example in getting
rid of
> > > > > > > Infinispan.
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > But I do understand all the concerns in doing this.
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > I still have to test S3_PING, if it works as easy as
multicast.
> > > > > > > If
> > > > > > > yes,
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > we can use it, if no… I don’t know yet. But this gets
offtopic
> > > > > > > for
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > Keycloak mailinglist, it’s more related to pure
> > > > > > > Wildfly/Infinispan.
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > seems to me it would be much easier to
get Infinispan working on
> > > > > > AWS
> > > > >
> > > >
> > >
> >
>
> > > > > > than to write and maintain an entire new caching mechanism
and
> > > > > > hope
> > > > > > we
> > > > >
> > > >
> > >
> >
>
> > > > > > don't refactor the cache SPI.
> > > > >
> > > >
> > >
> >
>
> > > > > +1
> > > >
> > >
> >
>
> > > > > I am sure infinispan/JGroups has possibility
to run in
> > > > > non-multicast
> > > >
> > >
> >
>
> > > > > environment. You may just need to figure how exactly to
configure
> > > > > it.
> > > > > So
> > > >
> > >
> >
>
> > > > > I agree that this issue is more related to Wildfly/Infinispan
> > > > > itself
> > > >
> > >
> >
>
> > > > > than to Keycloak.
> > > >
> > >
> >
>
> > > > > You may need to use jgroups protocols like
TCP instead of default
> > > > > UDP
> > > >
> > >
> >
>
> > > > > and maybe TCPPING (this requires to manually list all your
cluster
> > > >
> > >
> >
>
> > > > > nodes. But still, it's much better option IMO than
rewriting
> > > > > UserSession
> > > >
> > >
> >
>
> > > > > SPI)
> > > >
> > >
> >
>
> > > > Btv. if TCPPING or S3_PING is an issue, there is
also AWS_PING
> > >
> >
>
> > > > http://www.jgroups.org/manual-3.x/html/protlist.html#d0e5100 , but
> > > > it's
> > >
> >
>
> > > > not official part of jgroups.
> > >
> >
>
> > > > Marek
> > >
> >
>
> > > > > Marek
> > > >
> > >
> >
>
> > > > > _______________________________________________
> > > >
> > >
> >
>
> > > > > keycloak-user mailing list
> > > >
> > >
> >
>
> > > > > keycloak-user(a)lists.jboss.org
> > > >
> > >
> >
>
> > > > > https://lists.jboss.org/mailman/listinfo/keycloak-user
> > > >
> > >
> >
>
> > > > _______________________________________________
> > >
> >
>
> > > > keycloak-user mailing list
> > >
> >
>
> > > > keycloak-user(a)lists.jboss.org
> > >
> >
>
> > > > https://lists.jboss.org/mailman/listinfo/keycloak-user
> > >
> >
>
> _______________________________________________ keycloak-user
mailing list
> keycloak-user(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/keycloak-user
> _______________________________________________
> keycloak-user mailing list
> keycloak-user(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/keycloak-user
_______________________________________________
keycloak-user mailing list
keycloak-user(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/keycloak-user
1:19 p.m.
New subject: Replace use of Infinispan with User Sessions SPI ?
Hi Alan,
Thanks for the informative email. The steps you outlined are similar to what I’ve tested
with ECS. The gossip router is definitely a no-go for production since it’s a single
point of failure.
I am testing this down at the JGroups level right now and got it working with ECS. There
were two issues. On TCP you have to specify the external_addr to match the EC2 host
otherwise the nodes won’t form a cluster. Secondly, FD_SOCK attempts to connect back on a
random port. With Docker instances, this fails. Using a known client_bind_port works
well.
Here’s the code I’m testing with: https://github.com/foo4u/aws-infinispan-poc
<https://github.com/foo4u/aws-infinispan-poc>
Most interesting are probably:
https://github.com/foo4u/aws-infinispan-poc/blob/master/ecs-jgroups-poc/e...
<https://github.com/foo4u/aws-infinispan-poc/blob/master/ecs-jgroups-poc/e...
https://github.com/foo4u/aws-infinispan-poc/blob/master/ecs-jgroups-poc/s...
<https://github.com/foo4u/aws-infinispan-poc/blob/master/ecs-jgroups-poc/s...
With this set up the nodes on different machines communicate without issue. I still have
to add in something other than TCP_PING, but that wasn’t the main issue. Will use
JDBC_PING most likely. Not a fan of S3 for coordination. Plus I already need an RDBMS for
Keycloak.
Scott Rossillo
Smartling | Senior Software Engineer
srossillo(a)smartling.com
<https://app.sigstr.com/uc/55e5d41c6533390d03580000>
<http://www.sigstr.com/>
On Dec 15, 2015, at 2:13 PM, Alan Field <afield(a)redhat.com>
wrote:
Just to be clear, I have successfully tested Infinispan library and server mode clusters
on EC2 using S3_PING, TCP, and the internal EC2 IP addresses. None of the cloud providers
support multicast. The Docker case is a little different though, because of the issues
with getting access to the IP address.
Thanks,
Alan
From: "Niko Köbler" <niko(a)n-k.de>
To: "Paul Blair" <pblair(a)clearme.com>
Cc: "keycloak-user" <keycloak-user(a)lists.jboss.org>
Sent: Tuesday, December 15, 2015 1:53:18 PM
Subject: Re: [keycloak-user] Replace use of Infinispan with User Sessions SPI ?
We will go for the first run with EC2 and S3_PING, but w/o Docker.
If we/you/whoever will find a proper solution (possibly on the jgroups mailinglist), we
will test this.
Seams that everybody is aware of the Docker/Cloud/Multicast issues, but no-one has a
proper solution, only workarounds. :(
Am 15.12.2015 um 15:47 schrieb Paul Blair <pblair(a)clearme.com
<mailto:pblair@clearme.com>>:
I've also been working on setting up clustered Keycloak on Docker containers in EC2
and would be interested in any potential solutions for this configuration.
Alternatively I've set up on EC2 without Docker with S3_PING. I'd be interested
in hearing about the issues with this configuration.
From: Scott Rossillo <srossillo(a)smartling.com
<mailto:srossillo@smartling.com>>
Date: Mon, 14 Dec 2015 18:31:30 -0500
To: Marek Posolda <mposolda(a)redhat.com <mailto:mposolda@redhat.com>>,
<afield(a)redhat.com <mailto:afield@redhat.com>>
Cc: keycloak-user <keycloak-user(a)lists.jboss.org
<mailto:keycloak-user@lists.jboss.org>>
Subject: Re: [keycloak-user] Replace use of Infinispan with User Sessions SPI ?
There are two issues:
1. Infinispan relies on JGroups, which is difficult to configure correctly with the
various ping techniques that aren’t UDP multicast. I can elaborate on each one that we
tested but it’s just generally complex to get right. That’s not to say it’s impossible or
the biggest reason this is complicated on ECS or _insert container service here_, see #2
for that.
2. It is difficult to do discovery correctly with JGroups and Docker. Non-privileged
Docker instances - the default and recommend type - do not implicitly know their host’s
IP. This causes IP mismatches between what JGroups thinks the machine’s IP is and what it
actually is when connecting to hosts on different machines. This is the main issue and
it’s not the fault of JGroups per se, but there’s no simple work around.
Take for example a simple 2 node cluster:
Node 1 comes up on the docker0 interface of host A with the IP address 172.16.0.4. The
host A IP is 10.10.0.100.
Node 2 comes up on the docker0 interface of host B with the IP address 172.16.0.8. The
host B IP is 10.10.0.108.
The 172.16 network is not routable between hosts (by design). Docker does port forwarding
for ports we wish to expose to this works fine for HTTP/HTTPS but not the cluster
traffic.
So Node 1 will advertise itself as having IP 172.16.0.4 while Node 2 advertises
172.16.0.8. The two cannot talk to each other by default. However, using the hard coded
IPs and TCP PING, we can set external_addr on Node 1 to 10.10.0.100 and external_addr on
Node 2 to 10.10.0.108 and set initial_hosts to 10.10.0.100, 10.10.0.108. This will cause
the nodes to discover each other. However, they will not form a cluster. The nodes will
reject the handshake thinking they’re not actually 10.10.0.100 or 10.10.0.108
respectively.
I’d like to discuss further and I can share where we’ve gotten so far with workarounds to
this but it may be better to get into the weeds on another list.
Let me know what you think.
Best,
Scott
Scott Rossillo
Smartling | Senior Software Engineer
srossillo(a)smartling.com <mailto:srossillo@smartling.com>
<http://www.sigstr.com/>
On Dec 14, 2015, at 5:32 PM, Marek Posolda <mposolda(a)redhat.com
<mailto:mposolda@redhat.com>> wrote:
CCing Alan Field from RH Infinispan team and forwarding his question:
I'd like to know which configuration files you are using and why is is
harder to use with Amazon’s Docker service (ECS) or Beanstalk. I'd also be
interested in how big a cluster you are using in AWS.
On 14/12/15 22:24, Scott Rossillo wrote:
AWS was why we didn’t use Infinispan to begin with. That and it’s even more complicated
when you deploy using Amazon’s Docker service (ECS) or Beanstalk.
It’s too bad Infinispan / JGroups are beasts when the out of the box configuration can’t
be used. I’m planning to document this as we fix but I’d avoid S3_PING and use JDBC_PING.
You already need JDBC for the Keycloak DB, unless you’re using Mongo and it’s easier to
test locally.
TCPPING will bite you on AWS if Amazon decides to replace one of your instances (which it
does occasionally w/ECS or Beanstalk).
Best,
Scott
Scott Rossillo
Smartling | Senior Software Engineer
srossillo(a)smartling.com <mailto:srossillo@smartling.com>
<http://www.sigstr.com/>
On Dec 14, 2015, at 10:59 AM, Marek Posolda <mposolda(a)redhat.com
<mailto:mposolda@redhat.com>> wrote:
On 14/12/15 16:55, Marek Posolda wrote:
On 14/12/15 15:58, Bill Burke wrote:
On 12/14/2015 5:01 AM, Niko Köbler wrote:
Hi Marek,
Am 14.12.2015 um 08:50 schrieb Marek Posolda <
<mailto:mposolda@redhat.com>mposolda@redhat.com <mailto:mposolda@redhat.com>
<mailto:mposolda@redhat.com <mailto:mposolda@redhat.com>>>:
Btv. what's your motivation to not use infinispan? If you afraid of
cluster communication, you don't need to worry much about it, because
if you run single keycloak through standalone.xml, the infinispan
automatically works in LOCAL mode and there is no any cluster
communication at all.
My current customer is running his apps in AWS. As known, multicast is
not available in cloud infrastructures. Wildfly/Infinispan Cluster works
pretty well with multicast w/o having to know too much about JGroups
config. S3_PING seams to be a viable way to get a cluster running in AWS.
But additionally, my customer doesn’t have any (deep) knowledge about
JBoss infrastructures and so I’m looking for a way to be able to run
Keycloak in a cluster in AWS without the need to build up deeper
knowlegde of JGroups config, for example in getting rid of Infinispan.
But I do understand all the concerns in doing this.
I still have to test S3_PING, if it works as easy as multicast. If yes,
we can use it, if no… I don’t know yet. But this gets offtopic for
Keycloak mailinglist, it’s more related to pure Wildfly/Infinispan.
seems to me it would be much easier to get Infinispan working on AWS
than to write and maintain an entire new caching mechanism and hope we
don't refactor the cache SPI.
+1
I am sure infinispan/JGroups has possibility to run in non-multicast
environment. You may just need to figure how exactly to configure it. So
I agree that this issue is more related to Wildfly/Infinispan itself
than to Keycloak.
You may need to use jgroups protocols like TCP instead of default UDP
and maybe TCPPING (this requires to manually list all your cluster
nodes. But still, it's much better option IMO than rewriting UserSession
SPI)
Btv. if TCPPING or S3_PING is an issue, there is also AWS_PING
http://www.jgroups.org/manual-3.x/html/protlist.html#d0e5100
<http://www.jgroups.org/manual-3.x/html/protlist.html#d0e5100> , but it's
not official part of jgroups.
Marek
Marek
_______________________________________________
keycloak-user mailing list
keycloak-user(a)lists.jboss.org <mailto:keycloak-user@lists.jboss.org>
https://lists.jboss.org/mailman/listinfo/keycloak-user
<https://lists.jboss.org/mailman/listinfo/keycloak-user>
_______________________________________________
keycloak-user mailing list
keycloak-user(a)lists.jboss.org <mailto:keycloak-user@lists.jboss.org>
https://lists.jboss.org/mailman/listinfo/keycloak-user
<https://lists.jboss.org/mailman/listinfo/keycloak-user>
_______________________________________________ keycloak-user mailing list
keycloak-user(a)lists.jboss.org
<mailto:keycloak-user@lists.jboss.org>https://lists.jboss.org/mailman/listinfo/keycloak-user
<https://lists.jboss.org/mailman/listinfo/keycloak-user>_______________...
keycloak-user mailing list
keycloak-user(a)lists.jboss.org <mailto:keycloak-user@lists.jboss.org>
https://lists.jboss.org/mailman/listinfo/keycloak-user
_______________________________________________
keycloak-user mailing list
keycloak-user(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/keycloak-user
_______________________________________________
keycloak-user mailing list
keycloak-user(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/keycloak-user
2:33 p.m.
New subject: Replace use of Infinispan with User Sessions SPI ?
Hey Scott,
Thanks for following up and showing me your code. I have some questions inline for you:
----- Original Message -----
From: "Scott Rossillo" <srossillo(a)smartling.com>
To: "Alan Field" <afield(a)redhat.com>
Cc: "Niko Köbler" <niko(a)n-k.de>, "keycloak-user"
<keycloak-user(a)lists.jboss.org>
Sent: Wednesday, December 16, 2015 2:19:27 PM
Subject: Re: [keycloak-user] Replace use of Infinispan with User Sessions SPI
?
Hi Alan,
Thanks for the informative email. The steps you outlined are similar
to what
I’ve tested with ECS. The gossip router is definitely a no-go for production
since it’s a single point of failure.
It is possible to use the TUNNEL with multiple gossip routers to avoid this, but I
understand not wanting to have to setup and maintain the extra gossip router processes.
I am testing this down at the JGroups level right now and got it
working with
ECS. There were two issues. On TCP you have to specify the external_addr to
match the EC2 host otherwise the nodes won’t form a cluster. Secondly,
FD_SOCK attempts to connect back on a random port. With Docker instances,
this fails. Using a known client_bind_port works well.
Which IP address from your example is retrieved with this command:
EXTERNAL_HOST_IP= $( curl http://169.254.169.254/latest/meta-data/local-ipv4 ) "
Is it the 172.16.0.4 address or the 10.10.0.100 address? When I use this command in EC2, I
get the internal IP address for the instance, but not the public IP address. In your
example, that would be the 172.16.0.4 address. Also which address is used for the
bind_addr when you use -Djgroups.bind_addr=global?
Here’s the code I’m testing with:
https://github.com/foo4u/aws-infinispan-poc
Most interesting are probably:
How are you setting the JGROUPS_INITIAL_HOSTS environment variable?
With this set up the nodes on different machines communicate without
issue. I
still have to add in something other than TCP_PING, but that wasn’t the main
issue. Will use JDBC_PING most likely. Not a fan of S3 for coordination.
Plus I already need an RDBMS for Keycloak.
For my curiosity, can you tell me more about why you don't want to use S3_PING? Is it
the cost or something else? Just wondering and JDBC_PING should work fine.
Thanks,
Alan
Scott Rossillo
Smartling | Senior Software Engineer
srossillo(a)smartling.com
> On Dec 15, 2015, at 2:13 PM, Alan Field < afield(a)redhat.com
> wrote:
> Just to be clear, I have successfully tested Infinispan library
and server
> mode clusters on EC2 using S3_PING, TCP, and the internal EC2 IP addresses.
> None of the cloud providers support multicast. The Docker case is a little
> different though, because of the issues with getting access to the IP
> address.
> Thanks,
> Alan
> ----- Original Message -----
> > From: "Niko Köbler" < niko(a)n-k.de >
>
> > To: "Paul Blair" < pblair(a)clearme.com >
>
> > Cc: "keycloak-user" < keycloak-user(a)lists.jboss.org >
>
> > Sent: Tuesday, December 15, 2015 1:53:18 PM
>
> > Subject: Re: [keycloak-user] Replace use of Infinispan with User Sessions
> > SPI
> > ?
>
> > We will go for the first run with EC2 and S3_PING, but w/o
Docker.
>
> > If we/you/whoever will find a proper solution (possibly on the jgroups
> > mailinglist), we will test this.
>
> > Seams that everybody is aware of the Docker/Cloud/Multicast
issues, but
> > no-one has a proper solution, only workarounds. :(
>
> > > Am 15.12.2015 um 15:47 schrieb Paul Blair <
pblair(a)clearme.com >:
> >
>
> > > I've also been working on setting up clustered
Keycloak on Docker
> > > containers
> > > in EC2 and would be interested in any potential solutions for this
> > > configuration.
> >
>
> > > Alternatively I've set up on EC2 without Docker
with S3_PING. I'd be
> > > interested in hearing about the issues with this configuration.
> >
>
> > > From: Scott Rossillo < srossillo(a)smartling.com
>
> >
>
> > > Date: Mon, 14 Dec 2015 18:31:30 -0500
> >
>
> > > To: Marek Posolda < mposolda(a)redhat.com >, < afield(a)redhat.com
>
> >
>
> > > Cc: keycloak-user < keycloak-user(a)lists.jboss.org >
> >
>
> > > Subject: Re: [keycloak-user] Replace use of Infinispan with User
> > > Sessions
> > > SPI
> > > ?
> >
>
> > > There are two issues:
> >
>
> > > 1. Infinispan relies on JGroups, which is difficult to
configure
> > > correctly
> > > with the various ping techniques that aren’t UDP multicast. I can
> > > elaborate
> > > on each one that we tested but it’s just generally complex to get
> > > right.
> > > That’s not to say it’s impossible or the biggest reason this is
> > > complicated
> > > on ECS or _insert container service here_, see #2 for that.
> >
>
> > > 2. It is difficult to do discovery correctly with
JGroups and Docker.
> > > Non-privileged Docker instances - the default and recommend type - do
> > > not
> > > implicitly know their host’s IP. This causes IP mismatches between what
> > > JGroups thinks the machine’s IP is and what it actually is when
> > > connecting
> > > to hosts on different machines. This is the main issue and it’s not the
> > > fault of JGroups per se, but there’s no simple work around.
> >
>
> > > Take for example a simple 2 node cluster:
> >
>
> > > Node 1 comes up on the docker0 interface of host A
with the IP address
> > > 172.16.0.4. The host A IP is 10.10.0.100.
> >
>
> > > Node 2 comes up on the docker0 interface of host B with the IP address
> > > 172.16.0.8. The host B IP is 10.10.0.108.
> >
>
> > > The 172.16 network is not routable between hosts (by
design). Docker
> > > does
> > > port forwarding for ports we wish to expose to this works fine for
> > > HTTP/HTTPS but not the cluster traffic.
> >
>
> > > So Node 1 will advertise itself as having IP
172.16.0.4 while Node 2
> > > advertises 172.16.0.8. The two cannot talk to each other by default.
> > > However, using the hard coded IPs and TCP PING, we can set
> > > external_addr
> > > on
> > > Node 1 to 10.10.0.100 and external_addr on Node 2 to 10.10.0.108 and
> > > set
> > > initial_hosts to 10.10.0.100, 10.10.0.108. This will cause the nodes to
> > > discover each other. However, they will not form a cluster. The nodes
> > > will
> > > reject the handshake thinking they’re not actually 10.10.0.100 or
> > > 10.10.0.108 respectively.
> >
>
> > > I’d like to discuss further and I can share where
we’ve gotten so far
> > > with
> > > workarounds to this but it may be better to get into the weeds on
> > > another
> > > list.
> >
>
> > > Let me know what you think.
> >
>
> > > Best,
> >
>
> > > Scott
> >
>
> > > Scott Rossillo
> >
>
> > > Smartling | Senior Software Engineer
> >
>
> > > srossillo(a)smartling.com
> >
>
> > > > On Dec 14, 2015, at 5:32 PM, Marek Posolda <
mposolda(a)redhat.com >
> > > > wrote:
> > >
> >
>
> > > > CCing Alan Field from RH Infinispan team and
forwarding his question:
> > >
> >
>
> > > > I'd like to know which configuration files you are using and why
is
> > > > is
> > >
> >
>
> > > > harder to use with Amazon’s Docker service (ECS) or Beanstalk.
I'd
> > > > also
> > > > be
> > >
> >
>
> > > > interested in how big a cluster you are using in AWS.
> > >
> >
>
> > > > On 14/12/15 22:24, Scott Rossillo wrote:
> > >
> >
>
> > > > > AWS was why we didn’t use Infinispan to
begin with. That and it’s
> > > > > even
> > > > > more
> > > > > complicated when you deploy using Amazon’s Docker service (ECS)
or
> > > > > Beanstalk.
> > > >
> > >
> >
>
> > > > > It’s too bad Infinispan / JGroups are beasts
when the out of the
> > > > > box
> > > > > configuration can’t be used. I’m planning to document this as
we
> > > > > fix
> > > > > but
> > > > > I’d
> > > > > avoid S3_PING and use JDBC_PING. You already need JDBC for the
> > > > > Keycloak
> > > > > DB,
> > > > > unless you’re using Mongo and it’s easier to test locally.
> > > >
> > >
> >
>
> > > > > TCPPING will bite you on AWS if Amazon
decides to replace one of
> > > > > your
> > > > > instances (which it does occasionally w/ECS or Beanstalk).
> > > >
> > >
> >
>
> > > > > Best,
> > > >
> > >
> >
>
> > > > > Scott
> > > >
> > >
> >
>
> > > > > Scott Rossillo
> > > >
> > >
> >
>
> > > > > Smartling | Senior Software Engineer
> > > >
> > >
> >
>
> > > > > srossillo(a)smartling.com
> > > >
> > >
> >
>
> > > > > > On Dec 14, 2015, at 10:59 AM, Marek
Posolda < mposolda(a)redhat.com
> > > > > > >
> > > > > > wrote:
> > > > >
> > > >
> > >
> >
>
> > > > > > On 14/12/15 16:55, Marek Posolda
wrote:
> > > > >
> > > >
> > >
> >
>
> > > > > > > On 14/12/15 15:58, Bill Burke
wrote:
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > On 12/14/2015 5:01 AM, Niko
Köbler wrote:
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > > Hi Marek,
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > > > Am 14.12.2015 um
08:50 schrieb Marek Posolda <
> > > > > > > > > > mposolda(a)redhat.com
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > > > < mailto:mposolda@redhat.com
>>:
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > > > Btv. what's
your motivation to not use infinispan? If you
> > > > > > > > > > afraid
> > > > > > > > > > of
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > > > cluster communication, you don't
need to worry much about
> > > > > > > > > > it,
> > > > > > > > > > because
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > > > if you run single keycloak through
standalone.xml, the
> > > > > > > > > > infinispan
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > > > automatically works in LOCAL mode and
there is no any
> > > > > > > > > > cluster
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > > > communication at all.
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > > My current customer is
running his apps in AWS. As known,
> > > > > > > > > multicast
> > > > > > > > > is
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > > not available in cloud infrastructures.
Wildfly/Infinispan
> > > > > > > > > Cluster
> > > > > > > > > works
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > > pretty well with multicast w/o having to
know too much
> > > > > > > > > about
> > > > > > > > > JGroups
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > > config. S3_PING seams to be a viable way to
get a cluster
> > > > > > > > > running
> > > > > > > > > in
> > > > > > > > > AWS.
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > > But additionally, my customer doesn’t have
any (deep)
> > > > > > > > > knowledge
> > > > > > > > > about
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > > JBoss infrastructures and so I’m looking for
a way to be
> > > > > > > > > able
> > > > > > > > > to
> > > > > > > > > run
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > > Keycloak in a cluster in AWS without the
need to build up
> > > > > > > > > deeper
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > > knowlegde of JGroups config, for example in
getting rid of
> > > > > > > > > Infinispan.
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > > But I do understand all the concerns in
doing this.
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > > I still have to test S3_PING, if it works as
easy as
> > > > > > > > > multicast.
> > > > > > > > > If
> > > > > > > > > yes,
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > > we can use it, if no… I don’t know yet. But
this gets
> > > > > > > > > offtopic
> > > > > > > > > for
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > > Keycloak mailinglist, it’s more related to
pure
> > > > > > > > > Wildfly/Infinispan.
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > seems to me it would be much
easier to get Infinispan working
> > > > > > > > on
> > > > > > > > AWS
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > than to write and maintain an entire new caching
mechanism
> > > > > > > > and
> > > > > > > > hope
> > > > > > > > we
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > don't refactor the cache SPI.
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > +1
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > I am sure infinispan/JGroups has
possibility to run in
> > > > > > > non-multicast
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > environment. You may just need to figure how exactly
to
> > > > > > > configure
> > > > > > > it.
> > > > > > > So
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > I agree that this issue is more related to
Wildfly/Infinispan
> > > > > > > itself
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > than to Keycloak.
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > You may need to use jgroups
protocols like TCP instead of
> > > > > > > default
> > > > > > > UDP
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > and maybe TCPPING (this requires to manually list all
your
> > > > > > > cluster
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > nodes. But still, it's much better option IMO than
rewriting
> > > > > > > UserSession
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > SPI)
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > Btv. if TCPPING or S3_PING is an issue,
there is also AWS_PING
> > > > >
> > > >
> > >
> >
>
> > > > > >
http://www.jgroups.org/manual-3.x/html/protlist.html#d0e5100 ,
> > > > > > but
> > > > > > it's
> > > > >
> > > >
> > >
> >
>
> > > > > > not official part of jgroups.
> > > > >
> > > >
> > >
> >
>
> > > > > > Marek
> > > > >
> > > >
> > >
> >
>
> > > > > > > Marek
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > _______________________________________________
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > keycloak-user mailing list
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > keycloak-user(a)lists.jboss.org
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > >
https://lists.jboss.org/mailman/listinfo/keycloak-user
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > >
_______________________________________________
> > > > >
> > > >
> > >
> >
>
> > > > > > keycloak-user mailing list
> > > > >
> > > >
> > >
> >
>
> > > > > > keycloak-user(a)lists.jboss.org
> > > > >
> > > >
> > >
> >
>
> > > > > > https://lists.jboss.org/mailman/listinfo/keycloak-user
> > > > >
> > > >
> > >
> >
>
> > > _______________________________________________
keycloak-user mailing
> > > list
> > > keycloak-user(a)lists.jboss.org
> > > https://lists.jboss.org/mailman/listinfo/keycloak-user
> >
>
> > > _______________________________________________
> >
>
> > > keycloak-user mailing list
> >
>
> > > keycloak-user(a)lists.jboss.org
> >
>
> > > https://lists.jboss.org/mailman/listinfo/keycloak-user
> >
>
> > _______________________________________________
>
> > keycloak-user mailing list
>
> > keycloak-user(a)lists.jboss.org
>
> > https://lists.jboss.org/mailman/listinfo/keycloak-user
>
> _______________________________________________
> keycloak-user mailing list
> keycloak-user(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/keycloak-user
2:45 p.m.
Hi Alan,
It is possible to use the TUNNEL with multiple gossip routers to
avoid this, but I understand not wanting to have to setup and maintain the extra gossip
router processes.
True, it’s mainly about maintaining extra components.
Which IP address from your example is retrieved with this command:
EXTERNAL_HOST_IP=$(curl http://169.254.169.254/latest/meta-data/local-ipv4)”
I get the Amazon EC2 instance’s internal IP. This is what I want. There’s another endpoint
for public but I don’t want to use it. What’s good about this is when called from inside a
Docker container, I manage to get the actual internal IP for the EC2 instance.
How are you setting the JGROUPS_INITIAL_HOSTS environment variable?
Since this was a test with just 2 known hosts, I injected them as a Docker environment
variable with two fixed IPs. Once we switch to JDBC_PING, this will be removed.
For my curiosity, can you tell me more about why you don't want
to use S3_PING? Is it the cost or something else? Just wondering and JDBC_PING should work
fine.
S3_PING, like Gossip Router adds an external dependency on another service. S3 has had
consistency issues 3 times in 2015 (at least in US East). I don’t want to rely another
component when I already need the database to be up. Less components, less chance of
failure. Also, there are ton of variables to set with S3 and it requires preliminary work.
I want something that scales well from dev to QA to prod. JDBC_PING has a
datasource_jndi_name property. I can just reuse the data source I set up for Keycloak.
I hope I got all your questions.
Best,
Scott
Scott Rossillo
Smartling | Senior Software Engineer
srossillo(a)smartling.com
<http://www.sigstr.com/>
> On Dec 16, 2015, at 3:33 PM, Alan Field <afield(a)redhat.com> wrote:
>
> Hey Scott,
>
> Thanks for following up and showing me your code. I have some questions inline for
you:
>
> From: "Scott Rossillo" <srossillo(a)smartling.com>
> To: "Alan Field" <afield(a)redhat.com>
> Cc: "Niko Köbler" <niko(a)n-k.de>, "keycloak-user"
<keycloak-user(a)lists.jboss.org>
> Sent: Wednesday, December 16, 2015 2:19:27 PM
> Subject: Re: [keycloak-user] Replace use of Infinispan with User Sessions SPI
?
>
> Hi Alan,
>
> Thanks for the informative email. The steps you outlined are similar to what I’ve
tested with ECS. The gossip router is definitely a no-go for production since it’s a
single point of failure.
It is possible to use the TUNNEL with multiple gossip routers to
avoid this, but I understand not wanting to have to setup and maintain the extra gossip
router processes.
>
> I am testing this down at the JGroups level right now and got it working with ECS.
There were two issues. On TCP you have to specify the external_addr to match the EC2 host
otherwise the nodes won’t form a cluster. Secondly, FD_SOCK attempts to connect back on a
random port. With Docker instances, this fails. Using a known client_bind_port works
well.
> Which IP address from your example is retrieved with this command:
>
> EXTERNAL_HOST_IP=$(curl http://169.254.169.254/latest/meta-data/local-ipv4
<http://169.254.169.254/latest/meta-data/local-ipv4>)"
>
> Is it the 172.16.0.4 address or the 10.10.0.100 address? When I use this command in
EC2, I get the internal IP address for the instance, but not the public IP address. In
your example, that would be the 172.16.0.4 address. Also which address is used for the
bind_addr when you use -Djgroups.bind_addr=global?
>
> Here’s the code I’m testing with: https://github.com/foo4u/aws-infinispan-poc
<https://github.com/foo4u/aws-infinispan-poc>
>
> Most interesting are probably:
>
> https://github.com/foo4u/aws-infinispan-poc/blob/master/ecs-jgroups-poc/e...
<https://github.com/foo4u/aws-infinispan-poc/blob/master/ecs-jgroups-poc/e...
How are you setting the JGROUPS_INITIAL_HOSTS environment variable?
>
https://github.com/foo4u/aws-infinispan-poc/blob/master/ecs-jgroups-poc/s...
<https://github.com/foo4u/aws-infinispan-poc/blob/master/ecs-jgroups-poc/s...
>
> With this set up the nodes on different machines communicate without issue. I still
have to add in something other than TCP_PING, but that wasn’t the main issue. Will use
JDBC_PING most likely. Not a fan of S3 for coordination. Plus I already need an RDBMS for
Keycloak.
For my curiosity, can you tell me more about why you don't want
to use S3_PING? Is it the cost or something else? Just wondering and JDBC_PING should work
fine.
>
> Thanks,
> Alan
>
> Scott Rossillo
> Smartling | Senior Software Engineer
> srossillo(a)smartling.com <mailto:srossillo@smartling.com>
>
> <https://app.sigstr.com/uc/55e5d41c6533390d03580000>
> <http://www.sigstr.com/>
> On Dec 15, 2015, at 2:13 PM, Alan Field <afield(a)redhat.com
<mailto:afield@redhat.com>> wrote:
>
> Just to be clear, I have successfully tested Infinispan library and server mode
clusters on EC2 using S3_PING, TCP, and the internal EC2 IP addresses. None of the cloud
providers support multicast. The Docker case is a little different though, because of the
issues with getting access to the IP address.
>
> Thanks,
> Alan
>
> From: "Niko Köbler" <niko(a)n-k.de <mailto:niko@n-k.de>>
> To: "Paul Blair" <pblair(a)clearme.com
<mailto:pblair@clearme.com>>
> Cc: "keycloak-user" <keycloak-user(a)lists.jboss.org
<mailto:keycloak-user@lists.jboss.org>>
> Sent: Tuesday, December 15, 2015 1:53:18 PM
> Subject: Re: [keycloak-user] Replace use of Infinispan with User Sessions SPI
?
>
> We will go for the first run with EC2 and S3_PING, but w/o Docker.
> If we/you/whoever will find a proper solution (possibly on the jgroups mailinglist),
we will test this.
>
> Seams that everybody is aware of the Docker/Cloud/Multicast issues, but no-one has a
proper solution, only workarounds. :(
>
>
>
> Am 15.12.2015 um 15:47 schrieb Paul Blair <pblair(a)clearme.com
<mailto:pblair@clearme.com>>:
>
> I've also been working on setting up clustered Keycloak on Docker containers in
EC2 and would be interested in any potential solutions for this configuration.
>
> Alternatively I've set up on EC2 without Docker with S3_PING. I'd be
interested in hearing about the issues with this configuration.
>
> From: Scott Rossillo <srossillo(a)smartling.com
<mailto:srossillo@smartling.com>>
> Date: Mon, 14 Dec 2015 18:31:30 -0500
> To: Marek Posolda <mposolda(a)redhat.com <mailto:mposolda@redhat.com>>,
<afield(a)redhat.com <mailto:afield@redhat.com>>
> Cc: keycloak-user <keycloak-user(a)lists.jboss.org
<mailto:keycloak-user@lists.jboss.org>>
> Subject: Re: [keycloak-user] Replace use of Infinispan with User Sessions SPI ?
>
> There are two issues:
>
> 1. Infinispan relies on JGroups, which is difficult to configure correctly with the
various ping techniques that aren’t UDP multicast. I can elaborate on each one that we
tested but it’s just generally complex to get right. That’s not to say it’s impossible or
the biggest reason this is complicated on ECS or _insert container service here_, see #2
for that.
>
> 2. It is difficult to do discovery correctly with JGroups and Docker. Non-privileged
Docker instances - the default and recommend type - do not implicitly know their host’s
IP. This causes IP mismatches between what JGroups thinks the machine’s IP is and what it
actually is when connecting to hosts on different machines. This is the main issue and
it’s not the fault of JGroups per se, but there’s no simple work around.
>
> Take for example a simple 2 node cluster:
>
> Node 1 comes up on the docker0 interface of host A with the IP address 172.16.0.4.
The host A IP is 10.10.0.100.
> Node 2 comes up on the docker0 interface of host B with the IP address 172.16.0.8.
The host B IP is 10.10.0.108.
>
> The 172.16 network is not routable between hosts (by design). Docker does port
forwarding for ports we wish to expose to this works fine for HTTP/HTTPS but not the
cluster traffic.
>
> So Node 1 will advertise itself as having IP 172.16.0.4 while Node 2 advertises
172.16.0.8. The two cannot talk to each other by default. However, using the hard coded
IPs and TCP PING, we can set external_addr on Node 1 to 10.10.0.100 and external_addr on
Node 2 to 10.10.0.108 and set initial_hosts to 10.10.0.100, 10.10.0.108. This will cause
the nodes to discover each other. However, they will not form a cluster. The nodes will
reject the handshake thinking they’re not actually 10.10.0.100 or 10.10.0.108
respectively.
>
> I’d like to discuss further and I can share where we’ve gotten so far with
workarounds to this but it may be better to get into the weeds on another list.
>
> Let me know what you think.
>
> Best,
> Scott
>
> Scott Rossillo
> Smartling | Senior Software Engineer
> srossillo(a)smartling.com <mailto:srossillo@smartling.com>
>
> <http://www.sigstr.com/>
> On Dec 14, 2015, at 5:32 PM, Marek Posolda <mposolda(a)redhat.com
<mailto:mposolda@redhat.com>> wrote:
>
> CCing Alan Field from RH Infinispan team and forwarding his question:
> I'd like to know which configuration files you are using and why is is
> harder to use with Amazon’s Docker service (ECS) or Beanstalk. I'd also be
> interested in how big a cluster you are using in AWS.
>
>
>
> On 14/12/15 22:24, Scott Rossillo wrote:
> AWS was why we didn’t use Infinispan to begin with. That and it’s even more
complicated when you deploy using Amazon’s Docker service (ECS) or Beanstalk.
>
> It’s too bad Infinispan / JGroups are beasts when the out of the box configuration
can’t be used. I’m planning to document this as we fix but I’d avoid S3_PING and use
JDBC_PING. You already need JDBC for the Keycloak DB, unless you’re using Mongo and it’s
easier to test locally.
>
> TCPPING will bite you on AWS if Amazon decides to replace one of your instances
(which it does occasionally w/ECS or Beanstalk).
>
> Best,
> Scott
>
> Scott Rossillo
> Smartling | Senior Software Engineer
> srossillo(a)smartling.com <mailto:srossillo@smartling.com>
>
> <http://www.sigstr.com/>
> On Dec 14, 2015, at 10:59 AM, Marek Posolda <mposolda(a)redhat.com
<mailto:mposolda@redhat.com>> wrote:
>
> On 14/12/15 16:55, Marek Posolda wrote:
> On 14/12/15 15:58, Bill Burke wrote:
> On 12/14/2015 5:01 AM, Niko Köbler wrote:
> Hi Marek,
>
> Am 14.12.2015 um 08:50 schrieb Marek Posolda <mposolda(a)redhat.com
<mailto:mposolda@redhat.com>
> <mailto:mposolda@redhat.com <mailto:mposolda@redhat.com>>>:
>
> Btv. what's your motivation to not use infinispan? If you afraid of
> cluster communication, you don't need to worry much about it, because
> if you run single keycloak through standalone.xml, the infinispan
> automatically works in LOCAL mode and there is no any cluster
> communication at all.
> My current customer is running his apps in AWS. As known, multicast is
> not available in cloud infrastructures. Wildfly/Infinispan Cluster works
> pretty well with multicast w/o having to know too much about JGroups
> config. S3_PING seams to be a viable way to get a cluster running in AWS.
> But additionally, my customer doesn’t have any (deep) knowledge about
> JBoss infrastructures and so I’m looking for a way to be able to run
> Keycloak in a cluster in AWS without the need to build up deeper
> knowlegde of JGroups config, for example in getting rid of Infinispan.
> But I do understand all the concerns in doing this.
> I still have to test S3_PING, if it works as easy as multicast. If yes,
> we can use it, if no… I don’t know yet. But this gets offtopic for
> Keycloak mailinglist, it’s more related to pure Wildfly/Infinispan.
>
> seems to me it would be much easier to get Infinispan working on AWS
> than to write and maintain an entire new caching mechanism and hope we
> don't refactor the cache SPI.
>
>
> +1
>
> I am sure infinispan/JGroups has possibility to run in non-multicast
> environment. You may just need to figure how exactly to configure it. So
> I agree that this issue is more related to Wildfly/Infinispan itself
> than to Keycloak.
>
> You may need to use jgroups protocols like TCP instead of default UDP
> and maybe TCPPING (this requires to manually list all your cluster
> nodes. But still, it's much better option IMO than rewriting UserSession
> SPI)
> Btv. if TCPPING or S3_PING is an issue, there is also AWS_PING
> http://www.jgroups.org/manual-3.x/html/protlist.html#d0e5100
<http://www.jgroups.org/manual-3.x/html/protlist.html#d0e5100> , but it's
> not official part of jgroups.
>
> Marek
>
> Marek
> _______________________________________________
> keycloak-user mailing list
> keycloak-user(a)lists.jboss.org <mailto:keycloak-user@lists.jboss.org>
> https://lists.jboss.org/mailman/listinfo/keycloak-user
<https://lists.jboss.org/mailman/listinfo/keycloak-user>
>
> _______________________________________________
> keycloak-user mailing list
> keycloak-user(a)lists.jboss.org <mailto:keycloak-user@lists.jboss.org>
> https://lists.jboss.org/mailman/listinfo/keycloak-user
<https://lists.jboss.org/mailman/listinfo/keycloak-user>
>
>
>
> _______________________________________________ keycloak-user mailing list
keycloak-user(a)lists.jboss.org <mailto:keycloak-user@lists.jboss.org>
https://lists.jboss.org/mailman/listinfo/keycloak-user
<https://lists.jboss.org/mailman/listinfo/keycloak-user>_______________...
> keycloak-user mailing list
> keycloak-user(a)lists.jboss.org <mailto:keycloak-user@lists.jboss.org>
> https://lists.jboss.org/mailman/listinfo/keycloak-user
<https://lists.jboss.org/mailman/listinfo/keycloak-user>
>
> _______________________________________________
> keycloak-user mailing list
> keycloak-user(a)lists.jboss.org <mailto:keycloak-user@lists.jboss.org>
> https://lists.jboss.org/mailman/listinfo/keycloak-user
>
> _______________________________________________
> keycloak-user mailing list
> keycloak-user(a)lists.jboss.org <mailto:keycloak-user@lists.jboss.org>
> https://lists.jboss.org/mailman/listinfo/keycloak-user
3:11 p.m.
Hey Scott,
Thanks, I think you answered all of my questions, but I'm confused by something you
said in your first email:
"
The 172.16 network is not routable between hosts (by design). Docker does port forwarding
for ports we wish to expose to this works fine for HTTP/HTTPS but not the cluster traffic.
So Node 1 will advertise itself as having IP 172.16.0.4 while Node 2 advertises
172.16.0.8. The two cannot talk to each other by default.
"
My understanding is that the 172.16 addresses are the Amazon EC2 instance’s internal IP,
so I'm confused why this didn't work for you before. Is the difference that you
were setting jgroups.bind_addr to this address and now you are setting it to global and
setting external_addr to the instance’s internal IP? Just trying to understand what the
problem was and how you fixed it!
Thanks again,
Alan
----- Original Message -----
From: "Scott Rossillo" <srossillo(a)smartling.com>
To: "Alan Field" <afield(a)redhat.com>
Cc: "Niko Köbler" <niko(a)n-k.de>, "keycloak-user"
<keycloak-user(a)lists.jboss.org>
Sent: Wednesday, December 16, 2015 3:45:40 PM
Subject: Re: [keycloak-user] Replace use of Infinispan with User Sessions SPI
?
Hi Alan,
> It is possible to use the TUNNEL with multiple gossip routers to
avoid
> this, but I understand not wanting to have to setup and maintain the extra
> gossip router processes.
True, it’s mainly about maintaining extra components.
> Which IP address from your example is retrieved with this
command:
> EXTERNAL_HOST_IP=$(curl http://169.254.169.254/latest/meta-data/local-ipv4
> )”
I get the Amazon EC2 instance’s internal IP. This is what I want.
There’s
another endpoint for public but I don’t want to use it. What’s good about
this is when called from inside a Docker container, I manage to get the
actual internal IP for the EC2 instance.
> How are you setting the JGROUPS_INITIAL_HOSTS environment
variable?
Since this was a test with just 2 known hosts, I injected them as a
Docker
environment variable with two fixed IPs. Once we switch to JDBC_PING, this
will be removed.
> For my curiosity, can you tell me more about why you don't
want to use
> S3_PING? Is it the cost or something else? Just wondering and JDBC_PING
> should work fine.
S3_PING, like Gossip Router adds an external dependency on another
service.
S3 has had consistency issues 3 times in 2015 (at least in US East). I don’t
want to rely another component when I already need the database to be up.
Less components, less chance of failure. Also, there are ton of variables to
set with S3 and it requires preliminary work. I want something that scales
well from dev to QA to prod. JDBC_PING has a datasource_jndi_name property.
I can just reuse the data source I set up for Keycloak.
I hope I got all your questions.
Best,
Scott
Scott Rossillo
Smartling | Senior Software Engineer
srossillo(a)smartling.com
> On Dec 16, 2015, at 3:33 PM, Alan Field < afield(a)redhat.com
> wrote:
> Hey Scott,
> Thanks for following up and showing me your code. I have some
questions
> inline for you:
> ----- Original Message -----
> > From: "Scott Rossillo" <
srossillo(a)smartling.com >
>
> > To: "Alan Field" < afield(a)redhat.com >
>
> > Cc: "Niko Köbler" < niko(a)n-k.de >, "keycloak-user"
<
> > keycloak-user(a)lists.jboss.org >
>
> > Sent: Wednesday, December 16, 2015 2:19:27 PM
>
> > Subject: Re: [keycloak-user] Replace use of Infinispan with User Sessions
> > SPI
> > ?
>
> > Hi Alan,
> >
>
> > Thanks for the informative email. The steps you outlined
are similar to
> > what
> > I’ve tested with ECS. The gossip router is definitely a no-go for
> > production
> > since it’s a single point of failure.
>
> It is possible to use the TUNNEL with multiple gossip routers to
avoid
> this,
> but I understand not wanting to have to setup and maintain the extra gossip
> router processes.
> > I am testing this down at the JGroups level right now and
got it working
> > with
> > ECS. There were two issues. On TCP you have to specify the external_addr
> > to
> > match the EC2 host otherwise the nodes won’t form a cluster. Secondly,
> > FD_SOCK attempts to connect back on a random port. With Docker instances,
> > this fails. Using a known client_bind_port works well.
>
> Which IP address from your example is retrieved with this
command:
> EXTERNAL_HOST_IP= $( curl
> http://169.254.169.254/latest/meta-data/local-ipv4
> ) "
> Is it the 172.16.0.4 address or the 10.10.0.100 address? When I
use this
> command in EC2, I get the internal IP address for the instance, but not the
> public IP address. In your example, that would be the 172.16.0.4 address.
> Also which address is used for the bind_addr when you use
> -Djgroups.bind_addr=global?
> > Here’s the code I’m testing with:
> > https://github.com/foo4u/aws-infinispan-poc
>
> > Most interesting are probably:
>
> How are you setting the JGROUPS_INITIAL_HOSTS environment
variable?
>
> > With this set up the nodes on different machines
communicate without
> > issue.
> > I
> > still have to add in something other than TCP_PING, but that wasn’t the
> > main
> > issue. Will use JDBC_PING most likely. Not a fan of S3 for coordination.
> > Plus I already need an RDBMS for Keycloak.
>
> For my curiosity, can you tell me more about why you don't
want to use
> S3_PING? Is it the cost or something else? Just wondering and JDBC_PING
> should work fine.
>
> Thanks,
> Alan
> > Scott Rossillo
>
> > Smartling | Senior Software Engineer
>
> > srossillo(a)smartling.com
>
> > > On Dec 15, 2015, at 2:13 PM, Alan Field <
afield(a)redhat.com > wrote:
> >
>
> > > Just to be clear, I have successfully tested
Infinispan library and
> > > server
> > > mode clusters on EC2 using S3_PING, TCP, and the internal EC2 IP
> > > addresses.
> > > None of the cloud providers support multicast. The Docker case is a
> > > little
> > > different though, because of the issues with getting access to the IP
> > > address.
> >
>
> > > Thanks,
> >
>
> > > Alan
> >
>
> > > ----- Original Message -----
> >
>
> > > > From: "Niko Köbler" < niko(a)n-k.de
>
> > >
> >
>
> > > > To: "Paul Blair" < pblair(a)clearme.com >
> > >
> >
>
> > > > Cc: "keycloak-user" < keycloak-user(a)lists.jboss.org
>
> > >
> >
>
> > > > Sent: Tuesday, December 15, 2015 1:53:18 PM
> > >
> >
>
> > > > Subject: Re: [keycloak-user] Replace use of Infinispan with User
> > > > Sessions
> > > > SPI
> > > > ?
> > >
> >
>
> > > > We will go for the first run with EC2 and
S3_PING, but w/o Docker.
> > >
> >
>
> > > > If we/you/whoever will find a proper solution (possibly on the
> > > > jgroups
> > > > mailinglist), we will test this.
> > >
> >
>
> > > > Seams that everybody is aware of the
Docker/Cloud/Multicast issues,
> > > > but
> > > > no-one has a proper solution, only workarounds. :(
> > >
> >
>
> > > > > Am 15.12.2015 um 15:47 schrieb Paul Blair
< pblair(a)clearme.com >:
> > > >
> > >
> >
>
> > > > > I've also been working on setting up
clustered Keycloak on Docker
> > > > > containers
> > > > > in EC2 and would be interested in any potential solutions for
this
> > > > > configuration.
> > > >
> > >
> >
>
> > > > > Alternatively I've set up on EC2 without
Docker with S3_PING. I'd
> > > > > be
> > > > > interested in hearing about the issues with this configuration.
> > > >
> > >
> >
>
> > > > > From: Scott Rossillo <
srossillo(a)smartling.com >
> > > >
> > >
> >
>
> > > > > Date: Mon, 14 Dec 2015 18:31:30 -0500
> > > >
> > >
> >
>
> > > > > To: Marek Posolda < mposolda(a)redhat.com >, <
afield(a)redhat.com >
> > > >
> > >
> >
>
> > > > > Cc: keycloak-user < keycloak-user(a)lists.jboss.org >
> > > >
> > >
> >
>
> > > > > Subject: Re: [keycloak-user] Replace use of Infinispan with
User
> > > > > Sessions
> > > > > SPI
> > > > > ?
> > > >
> > >
> >
>
> > > > > There are two issues:
> > > >
> > >
> >
>
> > > > > 1. Infinispan relies on JGroups, which is
difficult to configure
> > > > > correctly
> > > > > with the various ping techniques that aren’t UDP multicast. I
can
> > > > > elaborate
> > > > > on each one that we tested but it’s just generally complex to
get
> > > > > right.
> > > > > That’s not to say it’s impossible or the biggest reason this is
> > > > > complicated
> > > > > on ECS or _insert container service here_, see #2 for that.
> > > >
> > >
> >
>
> > > > > 2. It is difficult to do discovery correctly
with JGroups and
> > > > > Docker.
> > > > > Non-privileged Docker instances - the default and recommend type
-
> > > > > do
> > > > > not
> > > > > implicitly know their host’s IP. This causes IP mismatches
between
> > > > > what
> > > > > JGroups thinks the machine’s IP is and what it actually is when
> > > > > connecting
> > > > > to hosts on different machines. This is the main issue and it’s
not
> > > > > the
> > > > > fault of JGroups per se, but there’s no simple work around.
> > > >
> > >
> >
>
> > > > > Take for example a simple 2 node cluster:
> > > >
> > >
> >
>
> > > > > Node 1 comes up on the docker0 interface of
host A with the IP
> > > > > address
> > > > > 172.16.0.4. The host A IP is 10.10.0.100.
> > > >
> > >
> >
>
> > > > > Node 2 comes up on the docker0 interface of host B with the IP
> > > > > address
> > > > > 172.16.0.8. The host B IP is 10.10.0.108.
> > > >
> > >
> >
>
> > > > > The 172.16 network is not routable between
hosts (by design).
> > > > > Docker
> > > > > does
> > > > > port forwarding for ports we wish to expose to this works fine
for
> > > > > HTTP/HTTPS but not the cluster traffic.
> > > >
> > >
> >
>
> > > > > So Node 1 will advertise itself as having IP
172.16.0.4 while Node
> > > > > 2
> > > > > advertises 172.16.0.8. The two cannot talk to each other by
> > > > > default.
> > > > > However, using the hard coded IPs and TCP PING, we can set
> > > > > external_addr
> > > > > on
> > > > > Node 1 to 10.10.0.100 and external_addr on Node 2 to
10.10.0.108
> > > > > and
> > > > > set
> > > > > initial_hosts to 10.10.0.100, 10.10.0.108. This will cause the
> > > > > nodes
> > > > > to
> > > > > discover each other. However, they will not form a cluster. The
> > > > > nodes
> > > > > will
> > > > > reject the handshake thinking they’re not actually 10.10.0.100
or
> > > > > 10.10.0.108 respectively.
> > > >
> > >
> >
>
> > > > > I’d like to discuss further and I can share
where we’ve gotten so
> > > > > far
> > > > > with
> > > > > workarounds to this but it may be better to get into the weeds
on
> > > > > another
> > > > > list.
> > > >
> > >
> >
>
> > > > > Let me know what you think.
> > > >
> > >
> >
>
> > > > > Best,
> > > >
> > >
> >
>
> > > > > Scott
> > > >
> > >
> >
>
> > > > > Scott Rossillo
> > > >
> > >
> >
>
> > > > > Smartling | Senior Software Engineer
> > > >
> > >
> >
>
> > > > > srossillo(a)smartling.com
> > > >
> > >
> >
>
> > > > > > On Dec 14, 2015, at 5:32 PM, Marek
Posolda < mposolda(a)redhat.com
> > > > > > >
> > > > > > wrote:
> > > > >
> > > >
> > >
> >
>
> > > > > > CCing Alan Field from RH Infinispan
team and forwarding his
> > > > > > question:
> > > > >
> > > >
> > >
> >
>
> > > > > > I'd like to know which configuration files you are
using and why
> > > > > > is
> > > > > > is
> > > > >
> > > >
> > >
> >
>
> > > > > > harder to use with Amazon’s Docker service (ECS) or
Beanstalk.
> > > > > > I'd
> > > > > > also
> > > > > > be
> > > > >
> > > >
> > >
> >
>
> > > > > > interested in how big a cluster you are using in AWS.
> > > > >
> > > >
> > >
> >
>
> > > > > > On 14/12/15 22:24, Scott Rossillo
wrote:
> > > > >
> > > >
> > >
> >
>
> > > > > > > AWS was why we didn’t use
Infinispan to begin with. That and
> > > > > > > it’s
> > > > > > > even
> > > > > > > more
> > > > > > > complicated when you deploy using Amazon’s Docker
service (ECS)
> > > > > > > or
> > > > > > > Beanstalk.
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > It’s too bad Infinispan / JGroups
are beasts when the out of
> > > > > > > the
> > > > > > > box
> > > > > > > configuration can’t be used. I’m planning to document
this as
> > > > > > > we
> > > > > > > fix
> > > > > > > but
> > > > > > > I’d
> > > > > > > avoid S3_PING and use JDBC_PING. You already need JDBC
for the
> > > > > > > Keycloak
> > > > > > > DB,
> > > > > > > unless you’re using Mongo and it’s easier to test
locally.
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > TCPPING will bite you on AWS if
Amazon decides to replace one
> > > > > > > of
> > > > > > > your
> > > > > > > instances (which it does occasionally w/ECS or
Beanstalk).
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > Best,
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > Scott
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > Scott Rossillo
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > Smartling | Senior Software Engineer
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > srossillo(a)smartling.com
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > On Dec 14, 2015, at 10:59 AM,
Marek Posolda <
> > > > > > > > mposolda(a)redhat.com
> > > > > > > > >
> > > > > > > > wrote:
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > On 14/12/15 16:55, Marek
Posolda wrote:
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > > On 14/12/15 15:58, Bill
Burke wrote:
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > > > On 12/14/2015 5:01
AM, Niko Köbler wrote:
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > > > > Hi Marek,
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > > > > > Am
14.12.2015 um 08:50 schrieb Marek Posolda <
> > > > > > > > > > > > mposolda(a)redhat.com
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > > > > > <
mailto:mposolda@redhat.com >>:
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > > > > > Btv.
what's your motivation to not use infinispan? If
> > > > > > > > > > > > you
> > > > > > > > > > > > afraid
> > > > > > > > > > > > of
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > > > > > cluster communication, you
don't need to worry much
> > > > > > > > > > > > about
> > > > > > > > > > > > it,
> > > > > > > > > > > > because
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > > > > > if you run single keycloak
through standalone.xml,
> > > > > > > > > > > > the
> > > > > > > > > > > > infinispan
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > > > > > automatically works in LOCAL
mode and there is no any
> > > > > > > > > > > > cluster
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > > > > > communication at all.
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > > > > My current
customer is running his apps in AWS. As
> > > > > > > > > > > known,
> > > > > > > > > > > multicast
> > > > > > > > > > > is
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > > > > not available in cloud
infrastructures.
> > > > > > > > > > > Wildfly/Infinispan
> > > > > > > > > > > Cluster
> > > > > > > > > > > works
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > > > > pretty well with multicast w/o
having to know too much
> > > > > > > > > > > about
> > > > > > > > > > > JGroups
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > > > > config. S3_PING seams to be a
viable way to get a
> > > > > > > > > > > cluster
> > > > > > > > > > > running
> > > > > > > > > > > in
> > > > > > > > > > > AWS.
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > > > > But additionally, my customer
doesn’t have any (deep)
> > > > > > > > > > > knowledge
> > > > > > > > > > > about
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > > > > JBoss infrastructures and so I’m
looking for a way to
> > > > > > > > > > > be
> > > > > > > > > > > able
> > > > > > > > > > > to
> > > > > > > > > > > run
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > > > > Keycloak in a cluster in AWS
without the need to build
> > > > > > > > > > > up
> > > > > > > > > > > deeper
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > > > > knowlegde of JGroups config, for
example in getting rid
> > > > > > > > > > > of
> > > > > > > > > > > Infinispan.
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > > > > But I do understand all the
concerns in doing this.
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > > > > I still have to test S3_PING, if
it works as easy as
> > > > > > > > > > > multicast.
> > > > > > > > > > > If
> > > > > > > > > > > yes,
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > > > > we can use it, if no… I don’t know
yet. But this gets
> > > > > > > > > > > offtopic
> > > > > > > > > > > for
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > > > > Keycloak mailinglist, it’s more
related to pure
> > > > > > > > > > > Wildfly/Infinispan.
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > > > seems to me it
would be much easier to get Infinispan
> > > > > > > > > > working
> > > > > > > > > > on
> > > > > > > > > > AWS
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > > > than to write and maintain an entire
new caching
> > > > > > > > > > mechanism
> > > > > > > > > > and
> > > > > > > > > > hope
> > > > > > > > > > we
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > > > don't refactor the cache SPI.
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > > +1
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > > I am sure
infinispan/JGroups has possibility to run in
> > > > > > > > > non-multicast
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > > environment. You may just need to figure how
exactly to
> > > > > > > > > configure
> > > > > > > > > it.
> > > > > > > > > So
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > > I agree that this issue is more related to
> > > > > > > > > Wildfly/Infinispan
> > > > > > > > > itself
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > > than to Keycloak.
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > > You may need to use
jgroups protocols like TCP instead of
> > > > > > > > > default
> > > > > > > > > UDP
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > > and maybe TCPPING (this requires to manually
list all your
> > > > > > > > > cluster
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > > nodes. But still, it's much better
option IMO than
> > > > > > > > > rewriting
> > > > > > > > > UserSession
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > > SPI)
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > Btv. if TCPPING or S3_PING is
an issue, there is also
> > > > > > > > AWS_PING
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > >
http://www.jgroups.org/manual-3.x/html/protlist.html#d0e5100
> > > > > > > > ,
> > > > > > > > but
> > > > > > > > it's
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > not official part of jgroups.
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > Marek
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > > Marek
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > >
_______________________________________________
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > > keycloak-user mailing list
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > > keycloak-user(a)lists.jboss.org
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > >
https://lists.jboss.org/mailman/listinfo/keycloak-user
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > >
_______________________________________________
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > keycloak-user mailing list
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > keycloak-user(a)lists.jboss.org
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > >
https://lists.jboss.org/mailman/listinfo/keycloak-user
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > >
_______________________________________________ keycloak-user
> > > > > mailing
> > > > > list
> > > > > keycloak-user(a)lists.jboss.org
> > > > > https://lists.jboss.org/mailman/listinfo/keycloak-user
> > > >
> > >
> >
>
> > > > > _______________________________________________
> > > >
> > >
> >
>
> > > > > keycloak-user mailing list
> > > >
> > >
> >
>
> > > > > keycloak-user(a)lists.jboss.org
> > > >
> > >
> >
>
> > > > > https://lists.jboss.org/mailman/listinfo/keycloak-user
> > > >
> > >
> >
>
> > > > _______________________________________________
> > >
> >
>
> > > > keycloak-user mailing list
> > >
> >
>
> > > > keycloak-user(a)lists.jboss.org
> > >
> >
>
> > > > https://lists.jboss.org/mailman/listinfo/keycloak-user
> > >
> >
>
> > > _______________________________________________
> >
>
> > > keycloak-user mailing list
> >
>
> > > keycloak-user(a)lists.jboss.org
> >
>
> > > https://lists.jboss.org/mailman/listinfo/keycloak-user
> >
>
3:17 p.m.
Ah, sorry, my originally contrived example wasn’t using Amazon but just my local Docker
machine IP.
In the case of my ECS tests, 172.16.0.0/16 is the Docker network’s IP, which is local to
the machine / EC2 instance. Using ECS, my VPC has an IP range of 172.31.0.0/16, so the
bind_addr has to be on this network. On my small cluster, that’s either 172.31.44.109 or
172.31.45.191.
Does that clear it up?
Scott Rossillo
Smartling | Senior Software Engineer
srossillo(a)smartling.com
<http://www.sigstr.com/>
On Dec 16, 2015, at 4:11 PM, Alan Field <afield(a)redhat.com>
wrote:
Hey Scott,
Thanks, I think you answered all of my questions, but I'm confused by something you
said in your first email:
"
The 172.16 network is not routable between hosts (by design). Docker does port forwarding
for ports we wish to expose to this works fine for HTTP/HTTPS but not the cluster
traffic.
So Node 1 will advertise itself as having IP 172.16.0.4 while Node 2 advertises
172.16.0.8. The two cannot talk to each other by default.
"
My understanding is that the 172.16 addresses are the Amazon EC2 instance’s internal IP,
so I'm confused why this didn't work for you before. Is the difference that you
were setting jgroups.bind_addr to this address and now you are setting it to global and
setting external_addr to the instance’s internal IP? Just trying to understand what the
problem was and how you fixed it!
Thanks again,
Alan
From: "Scott Rossillo" <srossillo(a)smartling.com>
To: "Alan Field" <afield(a)redhat.com>
Cc: "Niko Köbler" <niko(a)n-k.de>, "keycloak-user"
<keycloak-user(a)lists.jboss.org>
Sent: Wednesday, December 16, 2015 3:45:40 PM
Subject: Re: [keycloak-user] Replace use of Infinispan with User Sessions SPI ?
Hi Alan,
> It is possible to use the TUNNEL with multiple gossip routers to avoid this, but I
understand not wanting to have to setup and maintain the extra gossip router processes.
True, it’s mainly about maintaining extra components.
> Which IP address from your example is retrieved with this command:
> EXTERNAL_HOST_IP=$(curl http://169.254.169.254/latest/meta-data/local-ipv4
<http://169.254.169.254/latest/meta-data/local-ipv4>)”
I get the Amazon EC2 instance’s internal IP. This is what I want. There’s another
endpoint for public but I don’t want to use it. What’s good about this is when called from
inside a Docker container, I manage to get the actual internal IP for the EC2 instance.
> How are you setting the JGROUPS_INITIAL_HOSTS environment variable?
Since this was a test with just 2 known hosts, I injected them as a Docker environment
variable with two fixed IPs. Once we switch to JDBC_PING, this will be removed.
> For my curiosity, can you tell me more about why you don't want to use S3_PING?
Is it the cost or something else? Just wondering and JDBC_PING should work fine.
S3_PING, like Gossip Router adds an external dependency on another service. S3 has had
consistency issues 3 times in 2015 (at least in US East). I don’t want to rely another
component when I already need the database to be up. Less components, less chance of
failure. Also, there are ton of variables to set with S3 and it requires preliminary work.
I want something that scales well from dev to QA to prod. JDBC_PING has a
datasource_jndi_name property. I can just reuse the data source I set up for Keycloak.
I hope I got all your questions.
Best,
Scott
Scott Rossillo
Smartling | Senior Software Engineer
srossillo(a)smartling.com <mailto:srossillo@smartling.com>
<http://www.sigstr.com/>
On Dec 16, 2015, at 3:33 PM, Alan Field <afield(a)redhat.com
<mailto:afield@redhat.com>> wrote:
Hey Scott,
Thanks for following up and showing me your code. I have some questions inline for you:
From: "Scott Rossillo" <srossillo(a)smartling.com
<mailto:srossillo@smartling.com>>
To: "Alan Field" <afield(a)redhat.com <mailto:afield@redhat.com>>
Cc: "Niko Köbler" <niko(a)n-k.de <mailto:niko@n-k.de>>,
"keycloak-user" <keycloak-user(a)lists.jboss.org
<mailto:keycloak-user@lists.jboss.org>>
Sent: Wednesday, December 16, 2015 2:19:27 PM
Subject: Re: [keycloak-user] Replace use of Infinispan with User Sessions SPI ?
Hi Alan,
Thanks for the informative email. The steps you outlined are similar to what I’ve tested
with ECS. The gossip router is definitely a no-go for production since it’s a single
point of failure.
It is possible to use the TUNNEL with multiple gossip routers to avoid this, but I
understand not wanting to have to setup and maintain the extra gossip router processes.
I am testing this down at the JGroups level right now and got it working with ECS. There
were two issues. On TCP you have to specify the external_addr to match the EC2 host
otherwise the nodes won’t form a cluster. Secondly, FD_SOCK attempts to connect back on a
random port. With Docker instances, this fails. Using a known client_bind_port works
well.
Which IP address from your example is retrieved with this command:
EXTERNAL_HOST_IP=$(curl http://169.254.169.254/latest/meta-data/local-ipv4
<http://169.254.169.254/latest/meta-data/local-ipv4>)"
Is it the 172.16.0.4 address or the 10.10.0.100 address? When I use this command in EC2,
I get the internal IP address for the instance, but not the public IP address. In your
example, that would be the 172.16.0.4 address. Also which address is used for the
bind_addr when you use -Djgroups.bind_addr=global?
Here’s the code I’m testing with: https://github.com/foo4u/aws-infinispan-poc
<https://github.com/foo4u/aws-infinispan-poc>
Most interesting are probably:
https://github.com/foo4u/aws-infinispan-poc/blob/master/ecs-jgroups-poc/e...
<https://github.com/foo4u/aws-infinispan-poc/blob/master/ecs-jgroups-poc/e...
How are you setting the JGROUPS_INITIAL_HOSTS environment variable?
https://github.com/foo4u/aws-infinispan-poc/blob/master/ecs-jgroups-poc/s...
<https://github.com/foo4u/aws-infinispan-poc/blob/master/ecs-jgroups-poc/s...
With this set up the nodes on different machines communicate without issue. I still have
to add in something other than TCP_PING, but that wasn’t the main issue. Will use
JDBC_PING most likely. Not a fan of S3 for coordination. Plus I already need an RDBMS for
Keycloak.
For my curiosity, can you tell me more about why you don't want to use S3_PING? Is it
the cost or something else? Just wondering and JDBC_PING should work fine.
Thanks,
Alan
Scott Rossillo
Smartling | Senior Software Engineer
srossillo(a)smartling.com <mailto:srossillo@smartling.com>
<https://app.sigstr.com/uc/55e5d41c6533390d03580000>
<http://www.sigstr.com/>
On Dec 15, 2015, at 2:13 PM, Alan Field <afield(a)redhat.com
<mailto:afield@redhat.com>> wrote:
Just to be clear, I have successfully tested Infinispan library and server mode clusters
on EC2 using S3_PING, TCP, and the internal EC2 IP addresses. None of the cloud providers
support multicast. The Docker case is a little different though, because of the issues
with getting access to the IP address.
Thanks,
Alan
From: "Niko Köbler" <niko(a)n-k.de <mailto:niko@n-k.de>>
To: "Paul Blair" <pblair(a)clearme.com <mailto:pblair@clearme.com>>
Cc: "keycloak-user" <keycloak-user(a)lists.jboss.org
<mailto:keycloak-user@lists.jboss.org>>
Sent: Tuesday, December 15, 2015 1:53:18 PM
Subject: Re: [keycloak-user] Replace use of Infinispan with User Sessions SPI ?
We will go for the first run with EC2 and S3_PING, but w/o Docker.
If we/you/whoever will find a proper solution (possibly on the jgroups mailinglist), we
will test this.
Seams that everybody is aware of the Docker/Cloud/Multicast issues, but no-one has a
proper solution, only workarounds. :(
Am 15.12.2015 um 15:47 schrieb Paul Blair <pblair(a)clearme.com
<mailto:pblair@clearme.com>>:
I've also been working on setting up clustered Keycloak on Docker containers in EC2
and would be interested in any potential solutions for this configuration.
Alternatively I've set up on EC2 without Docker with S3_PING. I'd be interested
in hearing about the issues with this configuration.
From: Scott Rossillo <srossillo(a)smartling.com
<mailto:srossillo@smartling.com>>
Date: Mon, 14 Dec 2015 18:31:30 -0500
To: Marek Posolda <mposolda(a)redhat.com <mailto:mposolda@redhat.com>>,
<afield(a)redhat.com <mailto:afield@redhat.com>>
Cc: keycloak-user <keycloak-user(a)lists.jboss.org
<mailto:keycloak-user@lists.jboss.org>>
Subject: Re: [keycloak-user] Replace use of Infinispan with User Sessions SPI ?
There are two issues:
1. Infinispan relies on JGroups, which is difficult to configure correctly with the
various ping techniques that aren’t UDP multicast. I can elaborate on each one that we
tested but it’s just generally complex to get right. That’s not to say it’s impossible or
the biggest reason this is complicated on ECS or _insert container service here_, see #2
for that.
2. It is difficult to do discovery correctly with JGroups and Docker. Non-privileged
Docker instances - the default and recommend type - do not implicitly know their host’s
IP. This causes IP mismatches between what JGroups thinks the machine’s IP is and what it
actually is when connecting to hosts on different machines. This is the main issue and
it’s not the fault of JGroups per se, but there’s no simple work around.
Take for example a simple 2 node cluster:
Node 1 comes up on the docker0 interface of host A with the IP address 172.16.0.4. The
host A IP is 10.10.0.100.
Node 2 comes up on the docker0 interface of host B with the IP address 172.16.0.8. The
host B IP is 10.10.0.108.
The 172.16 network is not routable between hosts (by design). Docker does port forwarding
for ports we wish to expose to this works fine for HTTP/HTTPS but not the cluster
traffic.
So Node 1 will advertise itself as having IP 172.16.0.4 while Node 2 advertises
172.16.0.8. The two cannot talk to each other by default. However, using the hard coded
IPs and TCP PING, we can set external_addr on Node 1 to 10.10.0.100 and external_addr on
Node 2 to 10.10.0.108 and set initial_hosts to 10.10.0.100, 10.10.0.108. This will cause
the nodes to discover each other. However, they will not form a cluster. The nodes will
reject the handshake thinking they’re not actually 10.10.0.100 or 10.10.0.108
respectively.
I’d like to discuss further and I can share where we’ve gotten so far with workarounds to
this but it may be better to get into the weeds on another list.
Let me know what you think.
Best,
Scott
Scott Rossillo
Smartling | Senior Software Engineer
srossillo(a)smartling.com <mailto:srossillo@smartling.com>
<http://www.sigstr.com/>
On Dec 14, 2015, at 5:32 PM, Marek Posolda <mposolda(a)redhat.com
<mailto:mposolda@redhat.com>> wrote:
CCing Alan Field from RH Infinispan team and forwarding his question:
I'd like to know which configuration files you are using and why is is
harder to use with Amazon’s Docker service (ECS) or Beanstalk. I'd also be
interested in how big a cluster you are using in AWS.
On 14/12/15 22:24, Scott Rossillo wrote:
AWS was why we didn’t use Infinispan to begin with. That and it’s even more complicated
when you deploy using Amazon’s Docker service (ECS) or Beanstalk.
It’s too bad Infinispan / JGroups are beasts when the out of the box configuration can’t
be used. I’m planning to document this as we fix but I’d avoid S3_PING and use JDBC_PING.
You already need JDBC for the Keycloak DB, unless you’re using Mongo and it’s easier to
test locally.
TCPPING will bite you on AWS if Amazon decides to replace one of your instances (which it
does occasionally w/ECS or Beanstalk).
Best,
Scott
Scott Rossillo
Smartling | Senior Software Engineer
srossillo(a)smartling.com <mailto:srossillo@smartling.com>
<http://www.sigstr.com/>
On Dec 14, 2015, at 10:59 AM, Marek Posolda <mposolda(a)redhat.com
<mailto:mposolda@redhat.com>> wrote:
On 14/12/15 16:55, Marek Posolda wrote:
On 14/12/15 15:58, Bill Burke wrote:
On 12/14/2015 5:01 AM, Niko Köbler wrote:
Hi Marek,
Am 14.12.2015 um 08:50 schrieb Marek Posolda <mposolda(a)redhat.com
<mailto:mposolda@redhat.com>
<mailto:mposolda@redhat.com <mailto:mposolda@redhat.com>>>:
Btv. what's your motivation to not use infinispan? If you afraid of
cluster communication, you don't need to worry much about it, because
if you run single keycloak through standalone.xml, the infinispan
automatically works in LOCAL mode and there is no any cluster
communication at all.
My current customer is running his apps in AWS. As known, multicast is
not available in cloud infrastructures. Wildfly/Infinispan Cluster works
pretty well with multicast w/o having to know too much about JGroups
config. S3_PING seams to be a viable way to get a cluster running in AWS.
But additionally, my customer doesn’t have any (deep) knowledge about
JBoss infrastructures and so I’m looking for a way to be able to run
Keycloak in a cluster in AWS without the need to build up deeper
knowlegde of JGroups config, for example in getting rid of Infinispan.
But I do understand all the concerns in doing this.
I still have to test S3_PING, if it works as easy as multicast. If yes,
we can use it, if no… I don’t know yet. But this gets offtopic for
Keycloak mailinglist, it’s more related to pure Wildfly/Infinispan.
seems to me it would be much easier to get Infinispan working on AWS
than to write and maintain an entire new caching mechanism and hope we
don't refactor the cache SPI.
+1
I am sure infinispan/JGroups has possibility to run in non-multicast
environment. You may just need to figure how exactly to configure it. So
I agree that this issue is more related to Wildfly/Infinispan itself
than to Keycloak.
You may need to use jgroups protocols like TCP instead of default UDP
and maybe TCPPING (this requires to manually list all your cluster
nodes. But still, it's much better option IMO than rewriting UserSession
SPI)
Btv. if TCPPING or S3_PING is an issue, there is also AWS_PING
http://www.jgroups.org/manual-3.x/html/protlist.html#d0e5100
<http://www.jgroups.org/manual-3.x/html/protlist.html#d0e5100> , but it's
not official part of jgroups.
Marek
Marek
_______________________________________________
keycloak-user mailing list
keycloak-user(a)lists.jboss.org <mailto:keycloak-user@lists.jboss.org>
https://lists.jboss.org/mailman/listinfo/keycloak-user
<https://lists.jboss.org/mailman/listinfo/keycloak-user>
_______________________________________________
keycloak-user mailing list
keycloak-user(a)lists.jboss.org <mailto:keycloak-user@lists.jboss.org>
https://lists.jboss.org/mailman/listinfo/keycloak-user
<https://lists.jboss.org/mailman/listinfo/keycloak-user>
_______________________________________________ keycloak-user mailing list
keycloak-user(a)lists.jboss.org <mailto:keycloak-user@lists.jboss.org>
https://lists.jboss.org/mailman/listinfo/keycloak-user
<https://lists.jboss.org/mailman/listinfo/keycloak-user>_______________...
keycloak-user mailing list
keycloak-user(a)lists.jboss.org <mailto:keycloak-user@lists.jboss.org>
https://lists.jboss.org/mailman/listinfo/keycloak-user
<https://lists.jboss.org/mailman/listinfo/keycloak-user>
_______________________________________________
keycloak-user mailing list
keycloak-user(a)lists.jboss.org <mailto:keycloak-user@lists.jboss.org>
https://lists.jboss.org/mailman/listinfo/keycloak-user
<https://lists.jboss.org/mailman/listinfo/keycloak-user>
_______________________________________________
keycloak-user mailing list
keycloak-user(a)lists.jboss.org <mailto:keycloak-user@lists.jboss.org>
https://lists.jboss.org/mailman/listinfo/keycloak-user
<https://lists.jboss.org/mailman/listinfo/keycloak-user>
3:21 p.m.
Almost...
I guess if the EC2 instance IP works for the bind address, why do you need to set
external_addr?
Thanks for bearing with me on this! :-)
Alan
----- Original Message -----
From: "Scott Rossillo" <srossillo(a)smartling.com>
To: "Alan Field" <afield(a)redhat.com>
Cc: "Niko Köbler" <niko(a)n-k.de>, "keycloak-user"
<keycloak-user(a)lists.jboss.org>
Sent: Wednesday, December 16, 2015 4:17:29 PM
Subject: Re: [keycloak-user] Replace use of Infinispan with User Sessions SPI
?
Ah, sorry, my originally contrived example wasn’t using Amazon but
just my
local Docker machine IP.
In the case of my ECS tests, 172.16.0.0/16 is the Docker network’s
IP, which
is local to the machine / EC2 instance. Using ECS, my VPC has an IP range of
172.31.0.0/16, so the bind_addr has to be on this network. On my small
cluster, that’s either 172.31.44.109 or 172.31.45.191.
Does that clear it up?
Scott Rossillo
Smartling | Senior Software Engineer
srossillo(a)smartling.com
> On Dec 16, 2015, at 4:11 PM, Alan Field < afield(a)redhat.com
> wrote:
> Hey Scott,
> Thanks, I think you answered all of my questions, but I'm
confused by
> something you said in your first email:
> "
> The 172.16 network is not routable between hosts (by design). Docker does
> port forwarding for ports we wish to expose to this works fine for
> HTTP/HTTPS but not the cluster traffic.
> So Node 1 will advertise itself as having IP 172.16.0.4 while
Node 2
> advertises 172.16.0.8. The two cannot talk to each other by default.
> "
> My understanding is that the 172.16 addresses are the Amazon EC2
instance’s
> internal IP, so I'm confused why this didn't work for you before. Is the
> difference that you were setting jgroups.bind_addr to this address and now
> you are setting it to global and setting external_addr to the instance’s
> internal IP? Just trying to understand what the problem was and how you
> fixed it!
> Thanks again,
> Alan
> ----- Original Message -----
> > From: "Scott Rossillo" <
srossillo(a)smartling.com >
>
> > To: "Alan Field" < afield(a)redhat.com >
>
> > Cc: "Niko Köbler" < niko(a)n-k.de >, "keycloak-user"
<
> > keycloak-user(a)lists.jboss.org >
>
> > Sent: Wednesday, December 16, 2015 3:45:40 PM
>
> > Subject: Re: [keycloak-user] Replace use of Infinispan with User Sessions
> > SPI
> > ?
>
> > Hi Alan,
>
> > > It is possible to use the TUNNEL with multiple gossip
routers to avoid
> > > this, but I understand not wanting to have to setup and maintain the
> > > extra
> > > gossip router processes.
>
> > True, it’s mainly about maintaining extra components.
>
> > > Which IP address from your example is retrieved with
this command:
>
> > > EXTERNAL_HOST_IP=$(curl
> > > http://169.254.169.254/latest/meta-data/local-ipv4
> > > )”
>
> > I get the Amazon EC2 instance’s internal IP. This is what I
want. There’s
> > another endpoint for public but I don’t want to use it. What’s good about
> > this is when called from inside a Docker container, I manage to get the
> > actual internal IP for the EC2 instance.
>
> > > How are you setting the JGROUPS_INITIAL_HOSTS
environment variable?
>
> > Since this was a test with just 2 known hosts, I injected
them as a
> > Docker
> > environment variable with two fixed IPs. Once we switch to JDBC_PING,
> > this
> > will be removed.
>
> > > For my curiosity, can you tell me more about why you
don't want to use
> > > S3_PING? Is it the cost or something else? Just wondering and JDBC_PING
> > > should work fine.
>
> > S3_PING, like Gossip Router adds an external dependency on
another
> > service.
> > S3 has had consistency issues 3 times in 2015 (at least in US East). I
> > don’t
> > want to rely another component when I already need the database to be up.
> > Less components, less chance of failure. Also, there are ton of variables
> > to
> > set with S3 and it requires preliminary work. I want something that
> > scales
> > well from dev to QA to prod. JDBC_PING has a datasource_jndi_name
> > property.
> > I can just reuse the data source I set up for Keycloak.
>
> > I hope I got all your questions.
>
> > Best,
>
> > Scott
>
> > Scott Rossillo
>
> > Smartling | Senior Software Engineer
>
> > srossillo(a)smartling.com
>
> > > On Dec 16, 2015, at 3:33 PM, Alan Field <
afield(a)redhat.com > wrote:
> >
>
> > > Hey Scott,
> >
>
> > > Thanks for following up and showing me your code. I
have some questions
> > > inline for you:
> >
>
> > > ----- Original Message -----
> >
>
> > > > From: "Scott Rossillo" <
srossillo(a)smartling.com >
> > >
> >
>
> > > > To: "Alan Field" < afield(a)redhat.com >
> > >
> >
>
> > > > Cc: "Niko Köbler" < niko(a)n-k.de >,
"keycloak-user" <
> > > > keycloak-user(a)lists.jboss.org >
> > >
> >
>
> > > > Sent: Wednesday, December 16, 2015 2:19:27 PM
> > >
> >
>
> > > > Subject: Re: [keycloak-user] Replace use of Infinispan with User
> > > > Sessions
> > > > SPI
> > > > ?
> > >
> >
>
> > > > Hi Alan,
> > >
> >
>
> > > > Thanks for the informative email. The steps you
outlined are similar
> > > > to
> > > > what
> > > > I’ve tested with ECS. The gossip router is definitely a no-go for
> > > > production
> > > > since it’s a single point of failure.
> > >
> >
>
> > > It is possible to use the TUNNEL with multiple gossip
routers to avoid
> > > this,
> > > but I understand not wanting to have to setup and maintain the extra
> > > gossip
> > > router processes.
> >
>
> > > > I am testing this down at the JGroups level right
now and got it
> > > > working
> > > > with
> > > > ECS. There were two issues. On TCP you have to specify the
> > > > external_addr
> > > > to
> > > > match the EC2 host otherwise the nodes won’t form a cluster.
> > > > Secondly,
> > > > FD_SOCK attempts to connect back on a random port. With Docker
> > > > instances,
> > > > this fails. Using a known client_bind_port works well.
> > >
> >
>
> > > Which IP address from your example is retrieved with
this command:
> >
>
> > > EXTERNAL_HOST_IP= $( curl
> > > http://169.254.169.254/latest/meta-data/local-ipv4
> > > ) "
> >
>
> > > Is it the 172.16.0.4 address or the 10.10.0.100
address? When I use
> > > this
> > > command in EC2, I get the internal IP address for the instance, but not
> > > the
> > > public IP address. In your example, that would be the 172.16.0.4
> > > address.
> > > Also which address is used for the bind_addr when you use
> > > -Djgroups.bind_addr=global?
> >
>
> > > > Here’s the code I’m testing with:
> > > > https://github.com/foo4u/aws-infinispan-poc
> > >
> >
>
> > > > Most interesting are probably:
> > >
> >
>
> > > >
https://github.com/foo4u/aws-infinispan-poc/blob/master/ecs-jgroups-poc/e...
> > >
> >
>
> > > How are you setting the JGROUPS_INITIAL_HOSTS
environment variable?
> >
>
> > > >
https://github.com/foo4u/aws-infinispan-poc/blob/master/ecs-jgroups-poc/s...
> > >
> >
>
> > > > With this set up the nodes on different machines
communicate without
> > > > issue.
> > > > I
> > > > still have to add in something other than TCP_PING, but that wasn’t
> > > > the
> > > > main
> > > > issue. Will use JDBC_PING most likely. Not a fan of S3 for
> > > > coordination.
> > > > Plus I already need an RDBMS for Keycloak.
> > >
> >
>
> > > For my curiosity, can you tell me more about why you
don't want to use
> > > S3_PING? Is it the cost or something else? Just wondering and JDBC_PING
> > > should work fine.
> >
>
> > > Thanks,
> >
>
> > > Alan
> >
>
> > > > Scott Rossillo
> > >
> >
>
> > > > Smartling | Senior Software Engineer
> > >
> >
>
> > > > srossillo(a)smartling.com
> > >
> >
>
> > > > > On Dec 15, 2015, at 2:13 PM, Alan Field <
afield(a)redhat.com >
> > > > > wrote:
> > > >
> > >
> >
>
> > > > > Just to be clear, I have successfully tested
Infinispan library and
> > > > > server
> > > > > mode clusters on EC2 using S3_PING, TCP, and the internal EC2
IP
> > > > > addresses.
> > > > > None of the cloud providers support multicast. The Docker case
is a
> > > > > little
> > > > > different though, because of the issues with getting access to
the
> > > > > IP
> > > > > address.
> > > >
> > >
> >
>
> > > > > Thanks,
> > > >
> > >
> >
>
> > > > > Alan
> > > >
> > >
> >
>
> > > > > ----- Original Message -----
> > > >
> > >
> >
>
> > > > > > From: "Niko Köbler" <
niko(a)n-k.de >
> > > > >
> > > >
> > >
> >
>
> > > > > > To: "Paul Blair" < pblair(a)clearme.com >
> > > > >
> > > >
> > >
> >
>
> > > > > > Cc: "keycloak-user" <
keycloak-user(a)lists.jboss.org >
> > > > >
> > > >
> > >
> >
>
> > > > > > Sent: Tuesday, December 15, 2015 1:53:18 PM
> > > > >
> > > >
> > >
> >
>
> > > > > > Subject: Re: [keycloak-user] Replace use of Infinispan with
User
> > > > > > Sessions
> > > > > > SPI
> > > > > > ?
> > > > >
> > > >
> > >
> >
>
> > > > > > We will go for the first run with EC2
and S3_PING, but w/o
> > > > > > Docker.
> > > > >
> > > >
> > >
> >
>
> > > > > > If we/you/whoever will find a proper solution (possibly on
the
> > > > > > jgroups
> > > > > > mailinglist), we will test this.
> > > > >
> > > >
> > >
> >
>
> > > > > > Seams that everybody is aware of the
Docker/Cloud/Multicast
> > > > > > issues,
> > > > > > but
> > > > > > no-one has a proper solution, only workarounds. :(
> > > > >
> > > >
> > >
> >
>
> > > > > > > Am 15.12.2015 um 15:47 schrieb
Paul Blair < pblair(a)clearme.com
> > > > > > > >:
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > I've also been working on
setting up clustered Keycloak on
> > > > > > > Docker
> > > > > > > containers
> > > > > > > in EC2 and would be interested in any potential
solutions for
> > > > > > > this
> > > > > > > configuration.
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > Alternatively I've set up on
EC2 without Docker with S3_PING.
> > > > > > > I'd
> > > > > > > be
> > > > > > > interested in hearing about the issues with this
configuration.
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > From: Scott Rossillo <
srossillo(a)smartling.com >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > Date: Mon, 14 Dec 2015 18:31:30 -0500
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > To: Marek Posolda < mposolda(a)redhat.com >, <
afield(a)redhat.com
> > > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > Cc: keycloak-user < keycloak-user(a)lists.jboss.org
>
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > Subject: Re: [keycloak-user] Replace use of Infinispan
with
> > > > > > > User
> > > > > > > Sessions
> > > > > > > SPI
> > > > > > > ?
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > There are two issues:
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > 1. Infinispan relies on JGroups,
which is difficult to
> > > > > > > configure
> > > > > > > correctly
> > > > > > > with the various ping techniques that aren’t UDP
multicast. I
> > > > > > > can
> > > > > > > elaborate
> > > > > > > on each one that we tested but it’s just generally
complex to
> > > > > > > get
> > > > > > > right.
> > > > > > > That’s not to say it’s impossible or the biggest
reason this is
> > > > > > > complicated
> > > > > > > on ECS or _insert container service here_, see #2 for
that.
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > 2. It is difficult to do discovery
correctly with JGroups and
> > > > > > > Docker.
> > > > > > > Non-privileged Docker instances - the default and
recommend
> > > > > > > type
> > > > > > > -
> > > > > > > do
> > > > > > > not
> > > > > > > implicitly know their host’s IP. This causes IP
mismatches
> > > > > > > between
> > > > > > > what
> > > > > > > JGroups thinks the machine’s IP is and what it
actually is when
> > > > > > > connecting
> > > > > > > to hosts on different machines. This is the main issue
and it’s
> > > > > > > not
> > > > > > > the
> > > > > > > fault of JGroups per se, but there’s no simple work
around.
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > Take for example a simple 2 node
cluster:
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > Node 1 comes up on the docker0
interface of host A with the IP
> > > > > > > address
> > > > > > > 172.16.0.4. The host A IP is 10.10.0.100.
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > Node 2 comes up on the docker0 interface of host B
with the IP
> > > > > > > address
> > > > > > > 172.16.0.8. The host B IP is 10.10.0.108.
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > The 172.16 network is not routable
between hosts (by design).
> > > > > > > Docker
> > > > > > > does
> > > > > > > port forwarding for ports we wish to expose to this
works fine
> > > > > > > for
> > > > > > > HTTP/HTTPS but not the cluster traffic.
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > So Node 1 will advertise itself as
having IP 172.16.0.4 while
> > > > > > > Node
> > > > > > > 2
> > > > > > > advertises 172.16.0.8. The two cannot talk to each
other by
> > > > > > > default.
> > > > > > > However, using the hard coded IPs and TCP PING, we can
set
> > > > > > > external_addr
> > > > > > > on
> > > > > > > Node 1 to 10.10.0.100 and external_addr on Node 2 to
> > > > > > > 10.10.0.108
> > > > > > > and
> > > > > > > set
> > > > > > > initial_hosts to 10.10.0.100, 10.10.0.108. This will
cause the
> > > > > > > nodes
> > > > > > > to
> > > > > > > discover each other. However, they will not form a
cluster. The
> > > > > > > nodes
> > > > > > > will
> > > > > > > reject the handshake thinking they’re not actually
10.10.0.100
> > > > > > > or
> > > > > > > 10.10.0.108 respectively.
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > I’d like to discuss further and I
can share where we’ve gotten
> > > > > > > so
> > > > > > > far
> > > > > > > with
> > > > > > > workarounds to this but it may be better to get into
the weeds
> > > > > > > on
> > > > > > > another
> > > > > > > list.
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > Let me know what you think.
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > Best,
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > Scott
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > Scott Rossillo
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > Smartling | Senior Software Engineer
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > srossillo(a)smartling.com
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > On Dec 14, 2015, at 5:32 PM,
Marek Posolda <
> > > > > > > > mposolda(a)redhat.com
> > > > > > > > >
> > > > > > > > wrote:
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > CCing Alan Field from RH
Infinispan team and forwarding his
> > > > > > > > question:
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > I'd like to know which configuration files
you are using and
> > > > > > > > why
> > > > > > > > is
> > > > > > > > is
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > harder to use with Amazon’s Docker service (ECS)
or
> > > > > > > > Beanstalk.
> > > > > > > > I'd
> > > > > > > > also
> > > > > > > > be
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > interested in how big a cluster you are using in
AWS.
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > On 14/12/15 22:24, Scott
Rossillo wrote:
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > > AWS was why we didn’t
use Infinispan to begin with. That
> > > > > > > > > and
> > > > > > > > > it’s
> > > > > > > > > even
> > > > > > > > > more
> > > > > > > > > complicated when you deploy using Amazon’s
Docker service
> > > > > > > > > (ECS)
> > > > > > > > > or
> > > > > > > > > Beanstalk.
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > > It’s too bad Infinispan
/ JGroups are beasts when the out
> > > > > > > > > of
> > > > > > > > > the
> > > > > > > > > box
> > > > > > > > > configuration can’t be used. I’m planning to
document this
> > > > > > > > > as
> > > > > > > > > we
> > > > > > > > > fix
> > > > > > > > > but
> > > > > > > > > I’d
> > > > > > > > > avoid S3_PING and use JDBC_PING. You already
need JDBC for
> > > > > > > > > the
> > > > > > > > > Keycloak
> > > > > > > > > DB,
> > > > > > > > > unless you’re using Mongo and it’s easier to
test locally.
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > > TCPPING will bite you on
AWS if Amazon decides to replace
> > > > > > > > > one
> > > > > > > > > of
> > > > > > > > > your
> > > > > > > > > instances (which it does occasionally w/ECS
or Beanstalk).
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > > Best,
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > > Scott
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > > Scott Rossillo
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > > Smartling | Senior Software Engineer
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > > srossillo(a)smartling.com
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > > > On Dec 14, 2015, at
10:59 AM, Marek Posolda <
> > > > > > > > > > mposolda(a)redhat.com
> > > > > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > > > On 14/12/15 16:55,
Marek Posolda wrote:
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > > > > On 14/12/15
15:58, Bill Burke wrote:
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > > > > > On
12/14/2015 5:01 AM, Niko Köbler wrote:
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > > > > > > Hi
Marek,
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > > > > > > >
Am 14.12.2015 um 08:50 schrieb Marek Posolda <
> > > > > > > > > > > > > >
mposolda(a)redhat.com
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > > > > > > > <
mailto:mposolda@redhat.com >>:
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > > > > > > >
Btv. what's your motivation to not use
> > > > > > > > > > > > > > infinispan?
> > > > > > > > > > > > > > If
> > > > > > > > > > > > > > you
> > > > > > > > > > > > > > afraid
> > > > > > > > > > > > > > of
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > > > > > > > cluster
communication, you don't need to worry
> > > > > > > > > > > > > > much
> > > > > > > > > > > > > > about
> > > > > > > > > > > > > > it,
> > > > > > > > > > > > > > because
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > > > > > > > if you run single
keycloak through
> > > > > > > > > > > > > > standalone.xml,
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > infinispan
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > > > > > > > automatically works
in LOCAL mode and there is no
> > > > > > > > > > > > > > any
> > > > > > > > > > > > > > cluster
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > > > > > > > communication at
all.
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > > > > > > My
current customer is running his apps in AWS. As
> > > > > > > > > > > > > known,
> > > > > > > > > > > > > multicast
> > > > > > > > > > > > > is
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > > > > > > not available in cloud
infrastructures.
> > > > > > > > > > > > > Wildfly/Infinispan
> > > > > > > > > > > > > Cluster
> > > > > > > > > > > > > works
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > > > > > > pretty well with
multicast w/o having to know too
> > > > > > > > > > > > > much
> > > > > > > > > > > > > about
> > > > > > > > > > > > > JGroups
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > > > > > > config. S3_PING seams to
be a viable way to get a
> > > > > > > > > > > > > cluster
> > > > > > > > > > > > > running
> > > > > > > > > > > > > in
> > > > > > > > > > > > > AWS.
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > > > > > > But additionally, my
customer doesn’t have any
> > > > > > > > > > > > > (deep)
> > > > > > > > > > > > > knowledge
> > > > > > > > > > > > > about
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > > > > > > JBoss infrastructures
and so I’m looking for a way
> > > > > > > > > > > > > to
> > > > > > > > > > > > > be
> > > > > > > > > > > > > able
> > > > > > > > > > > > > to
> > > > > > > > > > > > > run
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > > > > > > Keycloak in a cluster in
AWS without the need to
> > > > > > > > > > > > > build
> > > > > > > > > > > > > up
> > > > > > > > > > > > > deeper
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > > > > > > knowlegde of JGroups
config, for example in getting
> > > > > > > > > > > > > rid
> > > > > > > > > > > > > of
> > > > > > > > > > > > > Infinispan.
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > > > > > > But I do understand all
the concerns in doing this.
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > > > > > > I still have to test
S3_PING, if it works as easy
> > > > > > > > > > > > > as
> > > > > > > > > > > > > multicast.
> > > > > > > > > > > > > If
> > > > > > > > > > > > > yes,
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > > > > > > we can use it, if no… I
don’t know yet. But this
> > > > > > > > > > > > > gets
> > > > > > > > > > > > > offtopic
> > > > > > > > > > > > > for
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > > > > > > Keycloak mailinglist,
it’s more related to pure
> > > > > > > > > > > > > Wildfly/Infinispan.
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > > > > > seems to
me it would be much easier to get Infinispan
> > > > > > > > > > > > working
> > > > > > > > > > > > on
> > > > > > > > > > > > AWS
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > > > > > than to write and maintain an
entire new caching
> > > > > > > > > > > > mechanism
> > > > > > > > > > > > and
> > > > > > > > > > > > hope
> > > > > > > > > > > > we
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > > > > > don't refactor the cache
SPI.
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > > > > +1
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > > > > I am sure
infinispan/JGroups has possibility to run in
> > > > > > > > > > > non-multicast
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > > > > environment. You may just need to
figure how exactly to
> > > > > > > > > > > configure
> > > > > > > > > > > it.
> > > > > > > > > > > So
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > > > > I agree that this issue is more
related to
> > > > > > > > > > > Wildfly/Infinispan
> > > > > > > > > > > itself
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > > > > than to Keycloak.
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > > > > You may need
to use jgroups protocols like TCP instead
> > > > > > > > > > > of
> > > > > > > > > > > default
> > > > > > > > > > > UDP
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > > > > and maybe TCPPING (this requires
to manually list all
> > > > > > > > > > > your
> > > > > > > > > > > cluster
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > > > > nodes. But still, it's much
better option IMO than
> > > > > > > > > > > rewriting
> > > > > > > > > > > UserSession
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > > > > SPI)
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > > > Btv. if TCPPING or
S3_PING is an issue, there is also
> > > > > > > > > > AWS_PING
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > > >
http://www.jgroups.org/manual-3.x/html/protlist.html#d0e5100
> > > > > > > > > > ,
> > > > > > > > > > but
> > > > > > > > > > it's
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > > > not official part of jgroups.
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > > > Marek
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > > > > Marek
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > > > >
_______________________________________________
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > > > > keycloak-user mailing list
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > > > > keycloak-user(a)lists.jboss.org
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > > > >
https://lists.jboss.org/mailman/listinfo/keycloak-user
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > > >
_______________________________________________
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > > > keycloak-user mailing list
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > > > keycloak-user(a)lists.jboss.org
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > > >
https://lists.jboss.org/mailman/listinfo/keycloak-user
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > >
_______________________________________________ keycloak-user
> > > > > > > mailing
> > > > > > > list
> > > > > > > keycloak-user(a)lists.jboss.org
> > > > > > >
https://lists.jboss.org/mailman/listinfo/keycloak-user
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > _______________________________________________
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > keycloak-user mailing list
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > keycloak-user(a)lists.jboss.org
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > >
https://lists.jboss.org/mailman/listinfo/keycloak-user
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > >
_______________________________________________
> > > > >
> > > >
> > >
> >
>
> > > > > > keycloak-user mailing list
> > > > >
> > > >
> > >
> >
>
> > > > > > keycloak-user(a)lists.jboss.org
> > > > >
> > > >
> > >
> >
>
> > > > > > https://lists.jboss.org/mailman/listinfo/keycloak-user
> > > > >
> > > >
> > >
> >
>
> > > > > _______________________________________________
> > > >
> > >
> >
>
> > > > > keycloak-user mailing list
> > > >
> > >
> >
>
> > > > > keycloak-user(a)lists.jboss.org
> > > >
> > >
> >
>
> > > > > https://lists.jboss.org/mailman/listinfo/keycloak-user
> > > >
> > >
> >
>
4:11 p.m.
I actually set the jgroups.bind_addr to global. I need the EC2 instance's address for
jgroups.external.addr, see:
https://github.com/foo4u/aws-infinispan-poc/blob/master/ecs-jgroups-poc/e...
<https://github.com/foo4u/aws-infinispan-poc/blob/master/ecs-jgroups-poc/e...
Scott Rossillo
Smartling | Senior Software Engineer
srossillo(a)smartling.com
<http://www.sigstr.com/>
On Dec 16, 2015, at 4:21 PM, Alan Field <afield(a)redhat.com>
wrote:
Almost...
I guess if the EC2 instance IP works for the bind address, why do you need to set
external_addr?
Thanks for bearing with me on this! :-)
Alan
From: "Scott Rossillo" <srossillo(a)smartling.com>
To: "Alan Field" <afield(a)redhat.com>
Cc: "Niko Köbler" <niko(a)n-k.de>, "keycloak-user"
<keycloak-user(a)lists.jboss.org>
Sent: Wednesday, December 16, 2015 4:17:29 PM
Subject: Re: [keycloak-user] Replace use of Infinispan with User Sessions SPI ?
Ah, sorry, my originally contrived example wasn’t using Amazon but just my local Docker
machine IP.
In the case of my ECS tests, 172.16.0.0/16 is the Docker network’s IP, which is local to
the machine / EC2 instance. Using ECS, my VPC has an IP range of 172.31.0.0/16, so the
bind_addr has to be on this network. On my small cluster, that’s either 172.31.44.109 or
172.31.45.191.
Does that clear it up?
Scott Rossillo
Smartling | Senior Software Engineer
srossillo(a)smartling.com <mailto:srossillo@smartling.com>
<http://www.sigstr.com/>
On Dec 16, 2015, at 4:11 PM, Alan Field <afield(a)redhat.com
<mailto:afield@redhat.com>> wrote:
Hey Scott,
Thanks, I think you answered all of my questions, but I'm confused by something you
said in your first email:
"
The 172.16 network is not routable between hosts (by design). Docker does port forwarding
for ports we wish to expose to this works fine for HTTP/HTTPS but not the cluster
traffic.
So Node 1 will advertise itself as having IP 172.16.0.4 while Node 2 advertises
172.16.0.8. The two cannot talk to each other by default.
"
My understanding is that the 172.16 addresses are the Amazon EC2 instance’s internal IP,
so I'm confused why this didn't work for you before. Is the difference that you
were setting jgroups.bind_addr to this address and now you are setting it to global and
setting external_addr to the instance’s internal IP? Just trying to understand what the
problem was and how you fixed it!
Thanks again,
Alan
From: "Scott Rossillo" <srossillo(a)smartling.com
<mailto:srossillo@smartling.com>>
To: "Alan Field" <afield(a)redhat.com <mailto:afield@redhat.com>>
Cc: "Niko Köbler" <niko(a)n-k.de <mailto:niko@n-k.de>>,
"keycloak-user" <keycloak-user(a)lists.jboss.org
<mailto:keycloak-user@lists.jboss.org>>
Sent: Wednesday, December 16, 2015 3:45:40 PM
Subject: Re: [keycloak-user] Replace use of Infinispan with User Sessions SPI ?
Hi Alan,
> It is possible to use the TUNNEL with multiple gossip routers to avoid this, but I
understand not wanting to have to setup and maintain the extra gossip router processes.
True, it’s mainly about maintaining extra components.
> Which IP address from your example is retrieved with this command:
> EXTERNAL_HOST_IP=$(curl http://169.254.169.254/latest/meta-data/local-ipv4
<http://169.254.169.254/latest/meta-data/local-ipv4>)”
I get the Amazon EC2 instance’s internal IP. This is what I want. There’s another
endpoint for public but I don’t want to use it. What’s good about this is when called from
inside a Docker container, I manage to get the actual internal IP for the EC2 instance.
> How are you setting the JGROUPS_INITIAL_HOSTS environment variable?
Since this was a test with just 2 known hosts, I injected them as a Docker environment
variable with two fixed IPs. Once we switch to JDBC_PING, this will be removed.
> For my curiosity, can you tell me more about why you don't want to use S3_PING?
Is it the cost or something else? Just wondering and JDBC_PING should work fine.
S3_PING, like Gossip Router adds an external dependency on another service. S3 has had
consistency issues 3 times in 2015 (at least in US East). I don’t want to rely another
component when I already need the database to be up. Less components, less chance of
failure. Also, there are ton of variables to set with S3 and it requires preliminary work.
I want something that scales well from dev to QA to prod. JDBC_PING has a
datasource_jndi_name property. I can just reuse the data source I set up for Keycloak.
I hope I got all your questions.
Best,
Scott
Scott Rossillo
Smartling | Senior Software Engineer
srossillo(a)smartling.com <mailto:srossillo@smartling.com>
<http://www.sigstr.com/>
On Dec 16, 2015, at 3:33 PM, Alan Field <afield(a)redhat.com
<mailto:afield@redhat.com>> wrote:
Hey Scott,
Thanks for following up and showing me your code. I have some questions inline for you:
From: "Scott Rossillo" <srossillo(a)smartling.com
<mailto:srossillo@smartling.com>>
To: "Alan Field" <afield(a)redhat.com <mailto:afield@redhat.com>>
Cc: "Niko Köbler" <niko(a)n-k.de <mailto:niko@n-k.de>>,
"keycloak-user" <keycloak-user(a)lists.jboss.org
<mailto:keycloak-user@lists.jboss.org>>
Sent: Wednesday, December 16, 2015 2:19:27 PM
Subject: Re: [keycloak-user] Replace use of Infinispan with User Sessions SPI ?
Hi Alan,
Thanks for the informative email. The steps you outlined are similar to what I’ve tested
with ECS. The gossip router is definitely a no-go for production since it’s a single
point of failure.
It is possible to use the TUNNEL with multiple gossip routers to avoid this, but I
understand not wanting to have to setup and maintain the extra gossip router processes.
I am testing this down at the JGroups level right now and got it working with ECS. There
were two issues. On TCP you have to specify the external_addr to match the EC2 host
otherwise the nodes won’t form a cluster. Secondly, FD_SOCK attempts to connect back on a
random port. With Docker instances, this fails. Using a known client_bind_port works
well.
Which IP address from your example is retrieved with this command:
EXTERNAL_HOST_IP=$(curl http://169.254.169.254/latest/meta-data/local-ipv4
<http://169.254.169.254/latest/meta-data/local-ipv4>)"
Is it the 172.16.0.4 address or the 10.10.0.100 address? When I use this command in EC2,
I get the internal IP address for the instance, but not the public IP address. In your
example, that would be the 172.16.0.4 address. Also which address is used for the
bind_addr when you use -Djgroups.bind_addr=global?
Here’s the code I’m testing with: https://github.com/foo4u/aws-infinispan-poc
<https://github.com/foo4u/aws-infinispan-poc>
Most interesting are probably:
https://github.com/foo4u/aws-infinispan-poc/blob/master/ecs-jgroups-poc/e...
<https://github.com/foo4u/aws-infinispan-poc/blob/master/ecs-jgroups-poc/e...
How are you setting the JGROUPS_INITIAL_HOSTS environment variable?
https://github.com/foo4u/aws-infinispan-poc/blob/master/ecs-jgroups-poc/s...
<https://github.com/foo4u/aws-infinispan-poc/blob/master/ecs-jgroups-poc/s...
With this set up the nodes on different machines communicate without issue. I still have
to add in something other than TCP_PING, but that wasn’t the main issue. Will use
JDBC_PING most likely. Not a fan of S3 for coordination. Plus I already need an RDBMS for
Keycloak.
For my curiosity, can you tell me more about why you don't want to use S3_PING? Is it
the cost or something else? Just wondering and JDBC_PING should work fine.
Thanks,
Alan
Scott Rossillo
Smartling | Senior Software Engineer
srossillo(a)smartling.com <mailto:srossillo@smartling.com>
<https://app.sigstr.com/uc/55e5d41c6533390d03580000>
<http://www.sigstr.com/>
On Dec 15, 2015, at 2:13 PM, Alan Field <afield(a)redhat.com
<mailto:afield@redhat.com>> wrote:
Just to be clear, I have successfully tested Infinispan library and server mode clusters
on EC2 using S3_PING, TCP, and the internal EC2 IP addresses. None of the cloud providers
support multicast. The Docker case is a little different though, because of the issues
with getting access to the IP address.
Thanks,
Alan
From: "Niko Köbler" <niko(a)n-k.de <mailto:niko@n-k.de>>
To: "Paul Blair" <pblair(a)clearme.com <mailto:pblair@clearme.com>>
Cc: "keycloak-user" <keycloak-user(a)lists.jboss.org
<mailto:keycloak-user@lists.jboss.org>>
Sent: Tuesday, December 15, 2015 1:53:18 PM
Subject: Re: [keycloak-user] Replace use of Infinispan with User Sessions SPI ?
We will go for the first run with EC2 and S3_PING, but w/o Docker.
If we/you/whoever will find a proper solution (possibly on the jgroups mailinglist), we
will test this.
Seams that everybody is aware of the Docker/Cloud/Multicast issues, but no-one has a
proper solution, only workarounds. :(
Am 15.12.2015 um 15:47 schrieb Paul Blair <pblair(a)clearme.com
<mailto:pblair@clearme.com>>:
I've also been working on setting up clustered Keycloak on Docker containers in EC2
and would be interested in any potential solutions for this configuration.
Alternatively I've set up on EC2 without Docker with S3_PING. I'd be interested
in hearing about the issues with this configuration.
From: Scott Rossillo <srossillo(a)smartling.com
<mailto:srossillo@smartling.com>>
Date: Mon, 14 Dec 2015 18:31:30 -0500
To: Marek Posolda <mposolda(a)redhat.com <mailto:mposolda@redhat.com>>,
<afield(a)redhat.com <mailto:afield@redhat.com>>
Cc: keycloak-user <keycloak-user(a)lists.jboss.org
<mailto:keycloak-user@lists.jboss.org>>
Subject: Re: [keycloak-user] Replace use of Infinispan with User Sessions SPI ?
There are two issues:
1. Infinispan relies on JGroups, which is difficult to configure correctly with the
various ping techniques that aren’t UDP multicast. I can elaborate on each one that we
tested but it’s just generally complex to get right. That’s not to say it’s impossible or
the biggest reason this is complicated on ECS or _insert container service here_, see #2
for that.
2. It is difficult to do discovery correctly with JGroups and Docker. Non-privileged
Docker instances - the default and recommend type - do not implicitly know their host’s
IP. This causes IP mismatches between what JGroups thinks the machine’s IP is and what it
actually is when connecting to hosts on different machines. This is the main issue and
it’s not the fault of JGroups per se, but there’s no simple work around.
Take for example a simple 2 node cluster:
Node 1 comes up on the docker0 interface of host A with the IP address 172.16.0.4. The
host A IP is 10.10.0.100.
Node 2 comes up on the docker0 interface of host B with the IP address 172.16.0.8. The
host B IP is 10.10.0.108.
The 172.16 network is not routable between hosts (by design). Docker does port forwarding
for ports we wish to expose to this works fine for HTTP/HTTPS but not the cluster
traffic.
So Node 1 will advertise itself as having IP 172.16.0.4 while Node 2 advertises
172.16.0.8. The two cannot talk to each other by default. However, using the hard coded
IPs and TCP PING, we can set external_addr on Node 1 to 10.10.0.100 and external_addr on
Node 2 to 10.10.0.108 and set initial_hosts to 10.10.0.100, 10.10.0.108. This will cause
the nodes to discover each other. However, they will not form a cluster. The nodes will
reject the handshake thinking they’re not actually 10.10.0.100 or 10.10.0.108
respectively.
I’d like to discuss further and I can share where we’ve gotten so far with workarounds to
this but it may be better to get into the weeds on another list.
Let me know what you think.
Best,
Scott
Scott Rossillo
Smartling | Senior Software Engineer
srossillo(a)smartling.com <mailto:srossillo@smartling.com>
<http://www.sigstr.com/>
On Dec 14, 2015, at 5:32 PM, Marek Posolda <mposolda(a)redhat.com
<mailto:mposolda@redhat.com>> wrote:
CCing Alan Field from RH Infinispan team and forwarding his question:
I'd like to know which configuration files you are using and why is is
harder to use with Amazon’s Docker service (ECS) or Beanstalk. I'd also be
interested in how big a cluster you are using in AWS.
On 14/12/15 22:24, Scott Rossillo wrote:
AWS was why we didn’t use Infinispan to begin with. That and it’s even more complicated
when you deploy using Amazon’s Docker service (ECS) or Beanstalk.
It’s too bad Infinispan / JGroups are beasts when the out of the box configuration can’t
be used. I’m planning to document this as we fix but I’d avoid S3_PING and use JDBC_PING.
You already need JDBC for the Keycloak DB, unless you’re using Mongo and it’s easier to
test locally.
TCPPING will bite you on AWS if Amazon decides to replace one of your instances (which it
does occasionally w/ECS or Beanstalk).
Best,
Scott
Scott Rossillo
Smartling | Senior Software Engineer
srossillo(a)smartling.com <mailto:srossillo@smartling.com>
<http://www.sigstr.com/>
On Dec 14, 2015, at 10:59 AM, Marek Posolda <mposolda(a)redhat.com
<mailto:mposolda@redhat.com>> wrote:
On 14/12/15 16:55, Marek Posolda wrote:
On 14/12/15 15:58, Bill Burke wrote:
On 12/14/2015 5:01 AM, Niko Köbler wrote:
Hi Marek,
Am 14.12.2015 um 08:50 schrieb Marek Posolda <mposolda(a)redhat.com
<mailto:mposolda@redhat.com>
<mailto:mposolda@redhat.com <mailto:mposolda@redhat.com>>>:
Btv. what's your motivation to not use infinispan? If you afraid of
cluster communication, you don't need to worry much about it, because
if you run single keycloak through standalone.xml, the infinispan
automatically works in LOCAL mode and there is no any cluster
communication at all.
My current customer is running his apps in AWS. As known, multicast is
not available in cloud infrastructures. Wildfly/Infinispan Cluster works
pretty well with multicast w/o having to know too much about JGroups
config. S3_PING seams to be a viable way to get a cluster running in AWS.
But additionally, my customer doesn’t have any (deep) knowledge about
JBoss infrastructures and so I’m looking for a way to be able to run
Keycloak in a cluster in AWS without the need to build up deeper
knowlegde of JGroups config, for example in getting rid of Infinispan.
But I do understand all the concerns in doing this.
I still have to test S3_PING, if it works as easy as multicast. If yes,
we can use it, if no… I don’t know yet. But this gets offtopic for
Keycloak mailinglist, it’s more related to pure Wildfly/Infinispan.
seems to me it would be much easier to get Infinispan working on AWS
than to write and maintain an entire new caching mechanism and hope we
don't refactor the cache SPI.
+1
I am sure infinispan/JGroups has possibility to run in non-multicast
environment. You may just need to figure how exactly to configure it. So
I agree that this issue is more related to Wildfly/Infinispan itself
than to Keycloak.
You may need to use jgroups protocols like TCP instead of default UDP
and maybe TCPPING (this requires to manually list all your cluster
nodes. But still, it's much better option IMO than rewriting UserSession
SPI)
Btv. if TCPPING or S3_PING is an issue, there is also AWS_PING
http://www.jgroups.org/manual-3.x/html/protlist.html#d0e5100
<http://www.jgroups.org/manual-3.x/html/protlist.html#d0e5100> , but it's
not official part of jgroups.
Marek
Marek
_______________________________________________
keycloak-user mailing list
keycloak-user(a)lists.jboss.org <mailto:keycloak-user@lists.jboss.org>
https://lists.jboss.org/mailman/listinfo/keycloak-user
<https://lists.jboss.org/mailman/listinfo/keycloak-user>
_______________________________________________
keycloak-user mailing list
keycloak-user(a)lists.jboss.org <mailto:keycloak-user@lists.jboss.org>
https://lists.jboss.org/mailman/listinfo/keycloak-user
<https://lists.jboss.org/mailman/listinfo/keycloak-user>
_______________________________________________ keycloak-user mailing list
keycloak-user(a)lists.jboss.org <mailto:keycloak-user@lists.jboss.org>
https://lists.jboss.org/mailman/listinfo/keycloak-user
<https://lists.jboss.org/mailman/listinfo/keycloak-user>_______________...
keycloak-user mailing list
keycloak-user(a)lists.jboss.org <mailto:keycloak-user@lists.jboss.org>
https://lists.jboss.org/mailman/listinfo/keycloak-user
<https://lists.jboss.org/mailman/listinfo/keycloak-user>
_______________________________________________
keycloak-user mailing list
keycloak-user(a)lists.jboss.org <mailto:keycloak-user@lists.jboss.org>
https://lists.jboss.org/mailman/listinfo/keycloak-user
<https://lists.jboss.org/mailman/listinfo/keycloak-user>
_______________________________________________
keycloak-user mailing list
keycloak-user(a)lists.jboss.org <mailto:keycloak-user@lists.jboss.org>
https://lists.jboss.org/mailman/listinfo/keycloak-user
<https://lists.jboss.org/mailman/listinfo/keycloak-user>
1:56 a.m.
If anyone (Scott?) wants to contribute how to configure KC clustering on
EC2 to our documentation that'd be great :)
On 16 December 2015 at 23:11, Scott Rossillo <srossillo(a)smartling.com>
wrote:
I actually set the jgroups.bind_addr to global. I need the EC2 instance's
address for jgroups.external.addr, see:
https://github.com/foo4u/aws-infinispan-poc/blob/master/ecs-jgroups-poc/e...
Scott Rossillo
Smartling | Senior Software Engineer
srossillo(a)smartling.com
[image: Powered by Sigstr] <http://www.sigstr.com/>
On Dec 16, 2015, at 4:21 PM, Alan Field <afield(a)redhat.com> wrote:
Almost...
I guess if the EC2 instance IP works for the bind address, why do you need
to set external_addr?
Thanks for bearing with me on this! :-)
Alan
------------------------------
*From: *"Scott Rossillo" <srossillo(a)smartling.com>
*To: *"Alan Field" <afield(a)redhat.com>
*Cc: *"Niko Köbler" <niko(a)n-k.de>, "keycloak-user" <
keycloak-user(a)lists.jboss.org>
*Sent: *Wednesday, December 16, 2015 4:17:29 PM
*Subject: *Re: [keycloak-user] Replace use of Infinispan with User
Sessions SPI ?
Ah, sorry, my originally contrived example wasn’t using Amazon but just my
local Docker machine IP.
In the case of my ECS tests, 172.16.0.0/16 is the Docker network’s IP,
which is local to the machine / EC2 instance. Using ECS, my VPC has an IP
range of 172.31.0.0/16, so the bind_addr has to be on this network. On my
small cluster, that’s either 172.31.44.109 or 172.31.45.191.
Does that clear it up?
Scott Rossillo
Smartling | Senior Software Engineer
srossillo(a)smartling.com
[image: Powered by Sigstr] <http://www.sigstr.com/>
On Dec 16, 2015, at 4:11 PM, Alan Field <afield(a)redhat.com> wrote:
Hey Scott,
Thanks, I think you answered all of my questions, but I'm confused by
something you said in your first email:
"
The 172.16 network is not routable between hosts (by design). Docker does
port forwarding for ports we wish to expose to this works fine for
HTTP/HTTPS but not the cluster traffic.
So Node 1 will advertise itself as having IP 172.16.0.4 while Node 2
advertises 172.16.0.8. The two cannot talk to each other by default.
"
My understanding is that the 172.16 addresses are the Amazon EC2
instance’s internal IP, so I'm confused why this didn't work for you
before. Is the difference that you were setting jgroups.bind_addr to this
address and now you are setting it to global and setting external_addr to
the instance’s internal IP? Just trying to understand what the problem was
and how you fixed it!
Thanks again,
Alan
------------------------------
*From: *"Scott Rossillo" <srossillo(a)smartling.com>
*To: *"Alan Field" <afield(a)redhat.com>
*Cc: *"Niko Köbler" <niko(a)n-k.de>, "keycloak-user" <
keycloak-user(a)lists.jboss.org>
*Sent: *Wednesday, December 16, 2015 3:45:40 PM
*Subject: *Re: [keycloak-user] Replace use of Infinispan with User
Sessions SPI ?
Hi Alan,
> It is possible to use the TUNNEL with multiple gossip routers to avoid
this, but I understand not wanting to have to setup and maintain the extra
gossip router processes.
True, it’s mainly about maintaining extra components.
> Which IP address from your example is retrieved with this command:
> EXTERNAL_HOST_IP=$(curl
http://169.254.169.254/latest/meta-data/local-ipv4)”
I get the Amazon EC2 instance’s internal IP. This is what I want. There’s
another endpoint for public but I don’t want to use it. What’s good about
this is when called from inside a Docker container, I manage to get the
actual internal IP for the EC2 instance.
> How are you setting the JGROUPS_INITIAL_HOSTS environment variable?
Since this was a test with just 2 known hosts, I injected them as a Docker
environment variable with two fixed IPs. Once we switch to JDBC_PING, this
will be removed.
> For my curiosity, can you tell me more about why you don't want to use
S3_PING? Is it the cost or something else? Just wondering and JDBC_PING
should work fine.
S3_PING, like Gossip Router adds an external dependency on another
service. S3 has had consistency issues 3 times in 2015 (at least in US
East). I don’t want to rely another component when I already need the
database to be up. Less components, less chance of failure. Also, there are
ton of variables to set with S3 and it requires preliminary work. I want
something that scales well from dev to QA to prod. JDBC_PING has a
datasource_jndi_name property. I can just reuse the data source I set up
for Keycloak.
I hope I got all your questions.
Best,
Scott
Scott Rossillo
Smartling | Senior Software Engineer
srossillo(a)smartling.com
[image: Powered by Sigstr] <http://www.sigstr.com/>
On Dec 16, 2015, at 3:33 PM, Alan Field <afield(a)redhat.com> wrote:
Hey Scott,
Thanks for following up and showing me your code. I have some questions
inline for you:
------------------------------
*From: *"Scott Rossillo" <srossillo(a)smartling.com>
*To: *"Alan Field" <afield(a)redhat.com>
*Cc: *"Niko Köbler" <niko(a)n-k.de>, "keycloak-user" <
keycloak-user(a)lists.jboss.org>
*Sent: *Wednesday, December 16, 2015 2:19:27 PM
*Subject: *Re: [keycloak-user] Replace use of Infinispan with User
Sessions SPI ?
Hi Alan,
Thanks for the informative email. The steps you outlined are similar to
what I’ve tested with ECS. The gossip router is definitely a no-go for
production since it’s a single point of failure.
It is possible to use the TUNNEL with multiple gossip routers to avoid
this, but I understand not wanting to have to setup and maintain the extra
gossip router processes.
I am testing this down at the JGroups level right now and got it working
with ECS. There were two issues. On TCP you have to specify
the external_addr to match the EC2 host otherwise the nodes won’t form a
cluster. Secondly, FD_SOCK attempts to connect back on a random port. With
Docker instances, this fails. Using a known client_bind_port works well.
Which IP address from your example is retrieved with this command:
EXTERNAL_HOST_IP=$(curl http://169.254.169.254/latest/meta-data/local-ipv4
)"
Is it the 172.16.0.4 address or the 10.10.0.100 address? When I use this
command in EC2, I get the internal IP address for the instance, but not the
public IP address. In your example, that would be the 172.16.0.4 address.
Also which address is used for the bind_addr when you use
-Djgroups.bind_addr=global?
Here’s the code I’m testing with:
https://github.com/foo4u/aws-infinispan-poc
Most interesting are probably:
https://github.com/foo4u/aws-infinispan-poc/blob/master/ecs-jgroups-poc/e...
How are you setting the JGROUPS_INITIAL_HOSTS environment variable?
https://github.com/foo4u/aws-infinispan-poc/blob/master/ecs-jgroups-poc/s...
With this set up the nodes on different machines communicate without
issue. I still have to add in something other than TCP_PING, but that
wasn’t the main issue. Will use JDBC_PING most likely. Not a fan of S3 for
coordination. Plus I already need an RDBMS for Keycloak.
For my curiosity, can you tell me more about why you don't want to use
S3_PING? Is it the cost or something else? Just wondering and JDBC_PING
should work fine.
Thanks,
Alan
Scott Rossillo
Smartling | Senior Software Engineer
srossillo(a)smartling.com
[image: Latest News + Events]
<https://app.sigstr.com/uc/55e5d41c6533390d03580000>
[image: Powered by Sigstr] <http://www.sigstr.com/>
On Dec 15, 2015, at 2:13 PM, Alan Field <afield(a)redhat.com> wrote:
Just to be clear, I have successfully tested Infinispan library and server
mode clusters on EC2 using S3_PING, TCP, and the internal EC2 IP addresses.
None of the cloud providers support multicast. The Docker case is a little
different though, because of the issues with getting access to the IP
address.
Thanks,
Alan
------------------------------
*From: *"Niko Köbler" <niko(a)n-k.de>
*To: *"Paul Blair" <pblair(a)clearme.com>
*Cc: *"keycloak-user" <keycloak-user(a)lists.jboss.org>
*Sent: *Tuesday, December 15, 2015 1:53:18 PM
*Subject: *Re: [keycloak-user] Replace use of Infinispan with User
Sessions SPI ?
We will go for the first run with EC2 and S3_PING, but w/o Docker.
If we/you/whoever will find a proper solution (possibly on the jgroups
mailinglist), we will test this.
Seams that everybody is aware of the Docker/Cloud/Multicast issues, but
no-one has a proper solution, only workarounds. :(
Am 15.12.2015 um 15:47 schrieb Paul Blair <pblair(a)clearme.com>:
I've also been working on setting up clustered Keycloak on Docker
containers in EC2 and would be interested in any potential solutions for
this configuration.
Alternatively I've set up on EC2 without Docker with S3_PING. I'd be
interested in hearing about the issues with this configuration.
From: Scott Rossillo <srossillo(a)smartling.com>
Date: Mon, 14 Dec 2015 18:31:30 -0500
To: Marek Posolda <mposolda(a)redhat.com>, <afield(a)redhat.com>
Cc: keycloak-user <keycloak-user(a)lists.jboss.org>
Subject: Re: [keycloak-user] Replace use of Infinispan with User Sessions
SPI ?
There are two issues:
1. Infinispan relies on JGroups, which is difficult to configure correctly
with the various ping techniques that aren’t UDP multicast. I can elaborate
on each one that we tested but it’s just generally complex to get right.
That’s not to say it’s impossible or the biggest reason this is complicated
on ECS or _insert container service here_, see #2 for that.
2. It is difficult to do discovery correctly with JGroups and Docker.
Non-privileged Docker instances - the default and recommend type - do not
implicitly know their host’s IP. This causes IP mismatches between what
JGroups thinks the machine’s IP is and what it actually is when connecting
to hosts on different machines. This is the main issue and it’s not the
fault of JGroups per se, but there’s no simple work around.
Take for example a simple 2 node cluster:
Node 1 comes up on the docker0 interface of host A with the IP address
172.16.0.4. The host A IP is 10.10.0.100.
Node 2 comes up on the docker0 interface of host B with the IP address
172.16.0.8. The host B IP is 10.10.0.108.
The 172.16 network is not routable between hosts (by design). Docker does
port forwarding for ports we wish to expose to this works fine for
HTTP/HTTPS but not the cluster traffic.
So Node 1 will advertise itself as having IP 172.16.0.4 while Node 2
advertises 172.16.0.8. The two cannot talk to each other by default.
However, using the hard coded IPs and TCP PING, we can set external_addr on
Node 1 to 10.10.0.100 and external_addr on Node 2 to 10.10.0.108 and set
initial_hosts to 10.10.0.100, 10.10.0.108. This will cause the nodes to
discover each other. However, they will not form a cluster. The nodes will
reject the handshake thinking they’re not actually 10.10.0.100 or
10.10.0.108 respectively.
I’d like to discuss further and I can share where we’ve gotten so far with
workarounds to this but it may be better to get into the weeds on another
list.
Let me know what you think.
Best,
Scott
Scott Rossillo
Smartling | Senior Software Engineer
srossillo(a)smartling.com
[image: Powered by Sigstr] <http://www.sigstr.com/>
On Dec 14, 2015, at 5:32 PM, Marek Posolda <mposolda(a)redhat.com> wrote:
CCing Alan Field from RH Infinispan team and forwarding his question:
I'd like to know which configuration files you are using and why is is
harder to use with Amazon’s Docker service (ECS) or Beanstalk. I'd also be
interested in how big a cluster you are using in AWS.
On 14/12/15 22:24, Scott Rossillo wrote:
AWS was why we didn’t use Infinispan to begin with. That and it’s even
more complicated when you deploy using Amazon’s Docker service (ECS) or
Beanstalk.
It’s too bad Infinispan / JGroups are beasts when the out of the box
configuration can’t be used. I’m planning to document this as we fix but
I’d avoid S3_PING and use JDBC_PING. You already need JDBC for the Keycloak
DB, unless you’re using Mongo and it’s easier to test locally.
TCPPING will bite you on AWS if Amazon decides to replace one of your
instances (which it does occasionally w/ECS or Beanstalk).
Best,
Scott
Scott Rossillo
Smartling | Senior Software Engineer
srossillo(a)smartling.com
[image: Powered by Sigstr] <http://www.sigstr.com/>
On Dec 14, 2015, at 10:59 AM, Marek Posolda <mposolda(a)redhat.com> wrote:
On 14/12/15 16:55, Marek Posolda wrote:
On 14/12/15 15:58, Bill Burke wrote:
On 12/14/2015 5:01 AM, Niko Köbler wrote:
Hi Marek,
Am 14.12.2015 um 08:50 schrieb Marek Posolda <mposolda(a)redhat.com
<mailto:mposolda@redhat.com <mposolda(a)redhat.com>>>:
Btv. what's your motivation to not use infinispan? If you afraid of
cluster communication, you don't need to worry much about it, because
if you run single keycloak through standalone.xml, the infinispan
automatically works in LOCAL mode and there is no any cluster
communication at all.
My current customer is running his apps in AWS. As known, multicast is
not available in cloud infrastructures. Wildfly/Infinispan Cluster works
pretty well with multicast w/o having to know too much about JGroups
config. S3_PING seams to be a viable way to get a cluster running in AWS.
But additionally, my customer doesn’t have any (deep) knowledge about
JBoss infrastructures and so I’m looking for a way to be able to run
Keycloak in a cluster in AWS without the need to build up deeper
knowlegde of JGroups config, for example in getting rid of Infinispan.
But I do understand all the concerns in doing this.
I still have to test S3_PING, if it works as easy as multicast. If yes,
we can use it, if no… I don’t know yet. But this gets offtopic for
Keycloak mailinglist, it’s more related to pure Wildfly/Infinispan.
seems to me it would be much easier to get Infinispan working on AWS
than to write and maintain an entire new caching mechanism and hope we
don't refactor the cache SPI.
+1
I am sure infinispan/JGroups has possibility to run in non-multicast
environment. You may just need to figure how exactly to configure it. So
I agree that this issue is more related to Wildfly/Infinispan itself
than to Keycloak.
You may need to use jgroups protocols like TCP instead of default UDP
and maybe TCPPING (this requires to manually list all your cluster
nodes. But still, it's much better option IMO than rewriting UserSession
SPI)
Btv. if TCPPING or S3_PING is an issue, there is also AWS_PING
http://www.jgroups.org/manual-3.x/html/protlist.html#d0e5100 , but it's
not official part of jgroups.
Marek
Marek
_______________________________________________
keycloak-user mailing list
keycloak-user(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/keycloak-user
_______________________________________________
keycloak-user mailing list
keycloak-user(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/keycloak-user
_______________________________________________ keycloak-user mailing list
keycloak-user(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/keycloak-user
_______________________________________________
keycloak-user mailing list
keycloak-user(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/keycloak-user
_______________________________________________
keycloak-user mailing list
keycloak-user(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/keycloak-user
_______________________________________________
keycloak-user mailing list
keycloak-user(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/keycloak-user
_______________________________________________
keycloak-user mailing list
keycloak-user(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/keycloak-user
1:40 p.m.
I will happily document once we validate on Wildfly. Did just the JGroups testing for
now.
Scott Rossillo
Smartling | Senior Software Engineer
srossillo(a)smartling.com
<http://www.sigstr.com/>
On Dec 18, 2015, at 2:56 AM, Stian Thorgersen
<sthorger(a)redhat.com> wrote:
If anyone (Scott?) wants to contribute how to configure KC clustering on EC2 to our
documentation that'd be great :)
On 16 December 2015 at 23:11, Scott Rossillo <srossillo(a)smartling.com
<mailto:srossillo@smartling.com>> wrote:
I actually set the jgroups.bind_addr to global. I need the EC2 instance's address for
jgroups.external.addr, see:
https://github.com/foo4u/aws-infinispan-poc/blob/master/ecs-jgroups-poc/e...
<https://github.com/foo4u/aws-infinispan-poc/blob/master/ecs-jgroups-poc/e...
Scott Rossillo
Smartling | Senior Software Engineer
srossillo(a)smartling.com <mailto:srossillo@smartling.com>
<http://www.sigstr.com/>
> On Dec 16, 2015, at 4:21 PM, Alan Field <afield(a)redhat.com
<mailto:afield@redhat.com>> wrote:
>
> Almost...
>
> I guess if the EC2 instance IP works for the bind address, why do you need to set
external_addr?
>
> Thanks for bearing with me on this! :-)
>
> Alan
>
> From: "Scott Rossillo" <srossillo(a)smartling.com
<mailto:srossillo@smartling.com>>
> To: "Alan Field" <afield(a)redhat.com
<mailto:afield@redhat.com>>
> Cc: "Niko Köbler" <niko(a)n-k.de <mailto:niko@n-k.de>>,
"keycloak-user" <keycloak-user(a)lists.jboss.org
<mailto:keycloak-user@lists.jboss.org>>
> Sent: Wednesday, December 16, 2015 4:17:29 PM
> Subject: Re: [keycloak-user] Replace use of Infinispan with User Sessions SPI ?
>
> Ah, sorry, my originally contrived example wasn’t using Amazon but just my local
Docker machine IP.
>
> In the case of my ECS tests, 172.16.0.0/16 <http://172.16.0.0/16> is the Docker
network’s IP, which is local to the machine / EC2 instance. Using ECS, my VPC has an IP
range of 172.31.0.0/16 <http://172.31.0.0/16>;, so the bind_addr has to be on this
network. On my small cluster, that’s either 172.31.44.109 or 172.31.45.191.
>
> Does that clear it up?
>
> Scott Rossillo
> Smartling | Senior Software Engineer
> srossillo(a)smartling.com <mailto:srossillo@smartling.com>
>
> <http://www.sigstr.com/>
> On Dec 16, 2015, at 4:11 PM, Alan Field <afield(a)redhat.com
<mailto:afield@redhat.com>> wrote:
>
> Hey Scott,
>
> Thanks, I think you answered all of my questions, but I'm confused by something
you said in your first email:
>
> "
> The 172.16 network is not routable between hosts (by design). Docker does port
forwarding for ports we wish to expose to this works fine for HTTP/HTTPS but not the
cluster traffic.
>
> So Node 1 will advertise itself as having IP 172.16.0.4 while Node 2 advertises
172.16.0.8. The two cannot talk to each other by default.
> "
>
> My understanding is that the 172.16 addresses are the Amazon EC2 instance’s internal
IP, so I'm confused why this didn't work for you before. Is the difference that
you were setting jgroups.bind_addr to this address and now you are setting it to global
and setting external_addr to the instance’s internal IP? Just trying to understand what
the problem was and how you fixed it!
>
> Thanks again,
> Alan
>
>
> From: "Scott Rossillo" <srossillo(a)smartling.com
<mailto:srossillo@smartling.com>>
> To: "Alan Field" <afield(a)redhat.com
<mailto:afield@redhat.com>>
> Cc: "Niko Köbler" <niko(a)n-k.de <mailto:niko@n-k.de>>,
"keycloak-user" <keycloak-user(a)lists.jboss.org
<mailto:keycloak-user@lists.jboss.org>>
> Sent: Wednesday, December 16, 2015 3:45:40 PM
> Subject: Re: [keycloak-user] Replace use of Infinispan with User Sessions SPI ?
>
> Hi Alan,
>
> > It is possible to use the TUNNEL with multiple gossip routers to avoid this, but
I understand not wanting to have to setup and maintain the extra gossip router processes.
>
> True, it’s mainly about maintaining extra components.
>
> > Which IP address from your example is retrieved with this command:
> > EXTERNAL_HOST_IP=$(curl http://169.254.169.254/latest/meta-data/local-ipv4
<http://169.254.169.254/latest/meta-data/local-ipv4>)”
>
> I get the Amazon EC2 instance’s internal IP. This is what I want. There’s another
endpoint for public but I don’t want to use it. What’s good about this is when called from
inside a Docker container, I manage to get the actual internal IP for the EC2 instance.
>
> > How are you setting the JGROUPS_INITIAL_HOSTS environment variable?
>
> Since this was a test with just 2 known hosts, I injected them as a Docker
environment variable with two fixed IPs. Once we switch to JDBC_PING, this will be
removed.
>
> > For my curiosity, can you tell me more about why you don't want to use
S3_PING? Is it the cost or something else? Just wondering and JDBC_PING should work fine.
>
> S3_PING, like Gossip Router adds an external dependency on another service. S3 has
had consistency issues 3 times in 2015 (at least in US East). I don’t want to rely another
component when I already need the database to be up. Less components, less chance of
failure. Also, there are ton of variables to set with S3 and it requires preliminary work.
I want something that scales well from dev to QA to prod. JDBC_PING has a
datasource_jndi_name property. I can just reuse the data source I set up for Keycloak.
>
> I hope I got all your questions.
>
> Best,
> Scott
>
> Scott Rossillo
> Smartling | Senior Software Engineer
> srossillo(a)smartling.com <mailto:srossillo@smartling.com>
>
> <http://www.sigstr.com/>
> On Dec 16, 2015, at 3:33 PM, Alan Field <afield(a)redhat.com
<mailto:afield@redhat.com>> wrote:
>
> Hey Scott,
>
> Thanks for following up and showing me your code. I have some questions inline for
you:
>
> From: "Scott Rossillo" <srossillo(a)smartling.com
<mailto:srossillo@smartling.com>>
> To: "Alan Field" <afield(a)redhat.com
<mailto:afield@redhat.com>>
> Cc: "Niko Köbler" <niko(a)n-k.de <mailto:niko@n-k.de>>,
"keycloak-user" <keycloak-user(a)lists.jboss.org
<mailto:keycloak-user@lists.jboss.org>>
> Sent: Wednesday, December 16, 2015 2:19:27 PM
> Subject: Re: [keycloak-user] Replace use of Infinispan with User Sessions SPI
?
>
> Hi Alan,
>
> Thanks for the informative email. The steps you outlined are similar to what I’ve
tested with ECS. The gossip router is definitely a no-go for production since it’s a
single point of failure.
> It is possible to use the TUNNEL with multiple gossip routers to avoid this, but I
understand not wanting to have to setup and maintain the extra gossip router processes.
>
> I am testing this down at the JGroups level right now and got it working with ECS.
There were two issues. On TCP you have to specify the external_addr to match the EC2 host
otherwise the nodes won’t form a cluster. Secondly, FD_SOCK attempts to connect back on a
random port. With Docker instances, this fails. Using a known client_bind_port works
well.
> Which IP address from your example is retrieved with this command:
>
> EXTERNAL_HOST_IP=$(curl http://169.254.169.254/latest/meta-data/local-ipv4
<http://169.254.169.254/latest/meta-data/local-ipv4>)"
>
> Is it the 172.16.0.4 address or the 10.10.0.100 address? When I use this command in
EC2, I get the internal IP address for the instance, but not the public IP address. In
your example, that would be the 172.16.0.4 address. Also which address is used for the
bind_addr when you use -Djgroups.bind_addr=global?
>
> Here’s the code I’m testing with: https://github.com/foo4u/aws-infinispan-poc
<https://github.com/foo4u/aws-infinispan-poc>
>
> Most interesting are probably:
>
> https://github.com/foo4u/aws-infinispan-poc/blob/master/ecs-jgroups-poc/e...
<https://github.com/foo4u/aws-infinispan-poc/blob/master/ecs-jgroups-poc/e...
> How are you setting the JGROUPS_INITIAL_HOSTS environment variable?
>
https://github.com/foo4u/aws-infinispan-poc/blob/master/ecs-jgroups-poc/s...
<https://github.com/foo4u/aws-infinispan-poc/blob/master/ecs-jgroups-poc/s...
>
> With this set up the nodes on different machines communicate without issue. I still
have to add in something other than TCP_PING, but that wasn’t the main issue. Will use
JDBC_PING most likely. Not a fan of S3 for coordination. Plus I already need an RDBMS for
Keycloak.
> For my curiosity, can you tell me more about why you don't want to use S3_PING?
Is it the cost or something else? Just wondering and JDBC_PING should work fine.
>
> Thanks,
> Alan
>
> Scott Rossillo
> Smartling | Senior Software Engineer
> srossillo(a)smartling.com <mailto:srossillo@smartling.com>
>
> <https://app.sigstr.com/uc/55e5d41c6533390d03580000>
> <http://www.sigstr.com/>
> On Dec 15, 2015, at 2:13 PM, Alan Field <afield(a)redhat.com
<mailto:afield@redhat.com>> wrote:
>
> Just to be clear, I have successfully tested Infinispan library and server mode
clusters on EC2 using S3_PING, TCP, and the internal EC2 IP addresses. None of the cloud
providers support multicast. The Docker case is a little different though, because of the
issues with getting access to the IP address.
>
> Thanks,
> Alan
>
> From: "Niko Köbler" <niko(a)n-k.de <mailto:niko@n-k.de>>
> To: "Paul Blair" <pblair(a)clearme.com
<mailto:pblair@clearme.com>>
> Cc: "keycloak-user" <keycloak-user(a)lists.jboss.org
<mailto:keycloak-user@lists.jboss.org>>
> Sent: Tuesday, December 15, 2015 1:53:18 PM
> Subject: Re: [keycloak-user] Replace use of Infinispan with User Sessions SPI
?
>
> We will go for the first run with EC2 and S3_PING, but w/o Docker.
> If we/you/whoever will find a proper solution (possibly on the jgroups mailinglist),
we will test this.
>
> Seams that everybody is aware of the Docker/Cloud/Multicast issues, but no-one has a
proper solution, only workarounds. :(
>
>
>
> Am 15.12.2015 um 15:47 schrieb Paul Blair <pblair(a)clearme.com
<mailto:pblair@clearme.com>>:
>
> I've also been working on setting up clustered Keycloak on Docker containers in
EC2 and would be interested in any potential solutions for this configuration.
>
> Alternatively I've set up on EC2 without Docker with S3_PING. I'd be
interested in hearing about the issues with this configuration.
>
> From: Scott Rossillo <srossillo(a)smartling.com
<mailto:srossillo@smartling.com>>
> Date: Mon, 14 Dec 2015 18:31:30 -0500
> To: Marek Posolda <mposolda(a)redhat.com <mailto:mposolda@redhat.com>>,
<afield(a)redhat.com <mailto:afield@redhat.com>>
> Cc: keycloak-user <keycloak-user(a)lists.jboss.org
<mailto:keycloak-user@lists.jboss.org>>
> Subject: Re: [keycloak-user] Replace use of Infinispan with User Sessions SPI ?
>
> There are two issues:
>
> 1. Infinispan relies on JGroups, which is difficult to configure correctly with the
various ping techniques that aren’t UDP multicast. I can elaborate on each one that we
tested but it’s just generally complex to get right. That’s not to say it’s impossible or
the biggest reason this is complicated on ECS or _insert container service here_, see #2
for that.
>
> 2. It is difficult to do discovery correctly with JGroups and Docker. Non-privileged
Docker instances - the default and recommend type - do not implicitly know their host’s
IP. This causes IP mismatches between what JGroups thinks the machine’s IP is and what it
actually is when connecting to hosts on different machines. This is the main issue and
it’s not the fault of JGroups per se, but there’s no simple work around.
>
> Take for example a simple 2 node cluster:
>
> Node 1 comes up on the docker0 interface of host A with the IP address 172.16.0.4.
The host A IP is 10.10.0.100.
> Node 2 comes up on the docker0 interface of host B with the IP address 172.16.0.8.
The host B IP is 10.10.0.108.
>
> The 172.16 network is not routable between hosts (by design). Docker does port
forwarding for ports we wish to expose to this works fine for HTTP/HTTPS but not the
cluster traffic.
>
> So Node 1 will advertise itself as having IP 172.16.0.4 while Node 2 advertises
172.16.0.8. The two cannot talk to each other by default. However, using the hard coded
IPs and TCP PING, we can set external_addr on Node 1 to 10.10.0.100 and external_addr on
Node 2 to 10.10.0.108 and set initial_hosts to 10.10.0.100, 10.10.0.108. This will cause
the nodes to discover each other. However, they will not form a cluster. The nodes will
reject the handshake thinking they’re not actually 10.10.0.100 or 10.10.0.108
respectively.
>
> I’d like to discuss further and I can share where we’ve gotten so far with
workarounds to this but it may be better to get into the weeds on another list.
>
> Let me know what you think.
>
> Best,
> Scott
>
> Scott Rossillo
> Smartling | Senior Software Engineer
> srossillo(a)smartling.com <mailto:srossillo@smartling.com>
>
> <http://www.sigstr.com/>
> On Dec 14, 2015, at 5:32 PM, Marek Posolda <mposolda(a)redhat.com
<mailto:mposolda@redhat.com>> wrote:
>
> CCing Alan Field from RH Infinispan team and forwarding his question:
> I'd like to know which configuration files you are using and why is is
> harder to use with Amazon’s Docker service (ECS) or Beanstalk. I'd also be
> interested in how big a cluster you are using in AWS.
>
>
>
> On 14/12/15 22:24, Scott Rossillo wrote:
> AWS was why we didn’t use Infinispan to begin with. That and it’s even more
complicated when you deploy using Amazon’s Docker service (ECS) or Beanstalk.
>
> It’s too bad Infinispan / JGroups are beasts when the out of the box configuration
can’t be used. I’m planning to document this as we fix but I’d avoid S3_PING and use
JDBC_PING. You already need JDBC for the Keycloak DB, unless you’re using Mongo and it’s
easier to test locally.
>
> TCPPING will bite you on AWS if Amazon decides to replace one of your instances
(which it does occasionally w/ECS or Beanstalk).
>
> Best,
> Scott
>
> Scott Rossillo
> Smartling | Senior Software Engineer
> srossillo(a)smartling.com <mailto:srossillo@smartling.com>
>
> <http://www.sigstr.com/>
> On Dec 14, 2015, at 10:59 AM, Marek Posolda <mposolda(a)redhat.com
<mailto:mposolda@redhat.com>> wrote:
>
> On 14/12/15 16:55, Marek Posolda wrote:
> On 14/12/15 15:58, Bill Burke wrote:
> On 12/14/2015 5:01 AM, Niko Köbler wrote:
> Hi Marek,
>
> Am 14.12.2015 um 08:50 schrieb Marek Posolda <mposolda(a)redhat.com
<mailto:mposolda@redhat.com>
> <mailto:mposolda@redhat.com <mailto:mposolda@redhat.com>>>:
>
> Btv. what's your motivation to not use infinispan? If you afraid of
> cluster communication, you don't need to worry much about it, because
> if you run single keycloak through standalone.xml, the infinispan
> automatically works in LOCAL mode and there is no any cluster
> communication at all.
> My current customer is running his apps in AWS. As known, multicast is
> not available in cloud infrastructures. Wildfly/Infinispan Cluster works
> pretty well with multicast w/o having to know too much about JGroups
> config. S3_PING seams to be a viable way to get a cluster running in AWS.
> But additionally, my customer doesn’t have any (deep) knowledge about
> JBoss infrastructures and so I’m looking for a way to be able to run
> Keycloak in a cluster in AWS without the need to build up deeper
> knowlegde of JGroups config, for example in getting rid of Infinispan.
> But I do understand all the concerns in doing this.
> I still have to test S3_PING, if it works as easy as multicast. If yes,
> we can use it, if no… I don’t know yet. But this gets offtopic for
> Keycloak mailinglist, it’s more related to pure Wildfly/Infinispan.
>
> seems to me it would be much easier to get Infinispan working on AWS
> than to write and maintain an entire new caching mechanism and hope we
> don't refactor the cache SPI.
>
>
> +1
>
> I am sure infinispan/JGroups has possibility to run in non-multicast
> environment. You may just need to figure how exactly to configure it. So
> I agree that this issue is more related to Wildfly/Infinispan itself
> than to Keycloak.
>
> You may need to use jgroups protocols like TCP instead of default UDP
> and maybe TCPPING (this requires to manually list all your cluster
> nodes. But still, it's much better option IMO than rewriting UserSession
> SPI)
> Btv. if TCPPING or S3_PING is an issue, there is also AWS_PING
> http://www.jgroups.org/manual-3.x/html/protlist.html#d0e5100
<http://www.jgroups.org/manual-3.x/html/protlist.html#d0e5100> , but it's
> not official part of jgroups.
>
> Marek
>
> Marek
> _______________________________________________
> keycloak-user mailing list
> keycloak-user(a)lists.jboss.org <mailto:keycloak-user@lists.jboss.org>
> https://lists.jboss.org/mailman/listinfo/keycloak-user
<https://lists.jboss.org/mailman/listinfo/keycloak-user>
>
> _______________________________________________
> keycloak-user mailing list
> keycloak-user(a)lists.jboss.org <mailto:keycloak-user@lists.jboss.org>
> https://lists.jboss.org/mailman/listinfo/keycloak-user
<https://lists.jboss.org/mailman/listinfo/keycloak-user>
>
>
>
> _______________________________________________ keycloak-user mailing list
keycloak-user(a)lists.jboss.org <mailto:keycloak-user@lists.jboss.org>
https://lists.jboss.org/mailman/listinfo/keycloak-user
<https://lists.jboss.org/mailman/listinfo/keycloak-user>_______________...
> keycloak-user mailing list
> keycloak-user(a)lists.jboss.org <mailto:keycloak-user@lists.jboss.org>
> https://lists.jboss.org/mailman/listinfo/keycloak-user
<https://lists.jboss.org/mailman/listinfo/keycloak-user>
>
> _______________________________________________
> keycloak-user mailing list
> keycloak-user(a)lists.jboss.org <mailto:keycloak-user@lists.jboss.org>
> https://lists.jboss.org/mailman/listinfo/keycloak-user
<https://lists.jboss.org/mailman/listinfo/keycloak-user>
>
> _______________________________________________
> keycloak-user mailing list
> keycloak-user(a)lists.jboss.org <mailto:keycloak-user@lists.jboss.org>
> https://lists.jboss.org/mailman/listinfo/keycloak-user
<https://lists.jboss.org/mailman/listinfo/keycloak-user>
>
>
>
_______________________________________________
keycloak-user mailing list
keycloak-user(a)lists.jboss.org <mailto:keycloak-user@lists.jboss.org>
https://lists.jboss.org/mailman/listinfo/keycloak-user
<https://lists.jboss.org/mailman/listinfo/keycloak-user>
1:52 p.m.
Hey Scott,
(I'm resending this with a little more information, since I can now post without being
moderated) :-)
----- Original Message -----
From: "Scott Rossillo" <srossillo(a)smartling.com>
To: "Marek Posolda" <mposolda(a)redhat.com>, afield(a)redhat.com
Cc: "keycloak-user" <keycloak-user(a)lists.jboss.org>, "Bill
Burke"
<bburke(a)redhat.com>
Sent: Monday, December 14, 2015 6:31:30 PM
Subject: Re: [keycloak-user] Replace use of Infinispan with User Sessions SPI
?
There are two issues:
1. Infinispan relies on JGroups, which is difficult to configure
correctly
with the various ping techniques that aren’t UDP multicast. I can elaborate
on each one that we tested but it’s just generally complex to get right.
That’s not to say it’s impossible or the biggest reason this is complicated
on ECS or _insert container service here_, see #2 for that.
The Infinispan server and JBoss EAP include a TCP-based stack in the configuration to run
on EC2 that looks like this:
<stack name="s3">
<transport type="TCP" socket-binding="jgroups-tcp"/>
<protocol type="S3_PING">
<property name="location">${jgroups.s3.bucket:}</property>
<property name="access_key">${jgroups.s3.access_key:}</property>
<property
name="secret_access_key">${jgroups.s3.secret_access_key:}</property>
<property
name="pre_signed_delete_url">${jgroups.s3.pre_signed_delete_url:}</property>
<property
name="pre_signed_put_url">${jgroups.s3.pre_signed_put_url:}</property>
<property name="prefix">${jgroups.s3.prefix:}</property>
</protocol>
<protocol type="MERGE3"/>
<protocol type="FD_SOCK" socket-binding="jgroups-tcp-fd"/>
<protocol type="FD_ALL"/>
<protocol type="VERIFY_SUSPECT"/>
<protocol type="pbcast.NAKACK2">
<property name="use_mcast_xmit">false</property>
</protocol>
<protocol type="UNICAST3"/>
<protocol type="pbcast.STABLE"/>
<protocol type="pbcast.GMS"/>
<protocol type="MFC"/>
<protocol type="FRAG2"/>
</stack>
With this in the configuration file, you can start the server with the following system
properties defined:
bin/clustered.sh -Djboss.node.name=node0 -Djboss.socket.binding.port-offset=0
-Djboss.default.jgroups.stack=s3 -Djgroups.s3.bucket=<s3_bucket_name>
-Djgroups.s3.access_key=<access_key>
-Djgroups.s3.secret_access_key=<secret_access_key> -Djboss.bind.address=$IP
-Djboss.bind.address.management=$IP
This will cause the server to start and the nodes will write to a file in the S3 bucket to
allow the nodes to discover each other. I do not see this stack defined in the
configuration used by WildFly 9, but it should work there as well. It is also possible to
use the JGroups Gossip Router for discovery, but it requires running a separate process
that all of the nodes contact during the discovery phase. I have the following in my
.bashrc to set the IP environment variable:
export IP=`GET http://169.254.169.254/latest/meta-data/local-ipv4`
I have verified that I can cluster plain EC2 instances on the internal IP addresses: (i.e.
172.31.4.165 and 172.31.18.207) These addresses are not publically accessible though, but
this cluster can be a cache for applications running in EC2.
2. It is difficult to do discovery correctly with JGroups and
Docker.
Non-privileged Docker instances - the default and recommend type - do not
implicitly know their host’s IP. This causes IP mismatches between what
JGroups thinks the machine’s IP is and what it actually is when connecting
to hosts on different machines. This is the main issue and it’s not the
fault of JGroups per se, but there’s no simple work around.
Take for example a simple 2 node cluster:
Node 1 comes up on the docker0 interface of host A with the IP
address
172.16.0.4. The host A IP is 10.10.0.100.
Node 2 comes up on the docker0 interface of host B with the IP address
172.16.0.8. The host B IP is 10.10.0.108.
The 172.16 network is not routable between hosts (by design). Docker
does
port forwarding for ports we wish to expose to this works fine for
HTTP/HTTPS but not the cluster traffic.
I've been trying to dig through the documentation to find out how you create
multi-container applications that need to network with each other on Amazon's ECS, but
so far I haven't gotten very far. Feel free to send me pointers, if you have any.
So Node 1 will advertise itself as having IP 172.16.0.4 while Node 2
advertises 172.16.0.8. The two cannot talk to each other by default.
However, using the hard coded IPs and TCP PING, we can set external_addr on
Node 1 to 10.10.0.100 and external_addr on Node 2 to 10.10.0.108 and set
initial_hosts to 10.10.0.100, 10.10.0.108. This will cause the nodes to
discover each other. However, they will not form a cluster. The nodes will
reject the handshake thinking they’re not actually 10.10.0.100 or
10.10.0.108 respectively.
I’d like to discuss further and I can share where we’ve gotten so far
with
workarounds to this but it may be better to get into the weeds on another
list.
Let me know what you think.
This issue is a little trickier, and I think we should probably move the discussion to the
jgroups-users list which you can subscribe to here. [1] Bela Ban may have some ideas about
how to set the binding address or interface to get around this. The Fabric8 project is
also using a JGroups discovery protocol that relies on Kubernetes, but I don't think
ECS uses Kubernetes.
Thanks,
Alan
[1] https://lists.sourceforge.net/lists/listinfo/javagroups-users
Best,
Scott
Scott Rossillo
Smartling | Senior Software Engineer
srossillo(a)smartling.com
> On Dec 14, 2015, at 5:32 PM, Marek Posolda <
mposolda(a)redhat.com > wrote:
> CCing Alan Field from RH Infinispan team and forwarding his
question:
> I'd like to know which configuration files you are using and why is is
> harder to use with Amazon’s Docker service (ECS) or Beanstalk. I'd also be
> interested in how big a cluster you are using in AWS.
> On 14/12/15 22:24, Scott Rossillo wrote:
> > AWS was why we didn’t use Infinispan to begin with. That
and it’s even
> > more
> > complicated when you deploy using Amazon’s Docker service (ECS) or
> > Beanstalk.
>
> > It’s too bad Infinispan / JGroups are beasts when the out
of the box
> > configuration can’t be used. I’m planning to document this as we fix but
> > I’d
> > avoid S3_PING and use JDBC_PING. You already need JDBC for the Keycloak
> > DB,
> > unless you’re using Mongo and it’s easier to test locally.
>
> > TCPPING will bite you on AWS if Amazon decides to replace
one of your
> > instances (which it does occasionally w/ECS or Beanstalk).
>
> > Best,
>
> > Scott
>
> > Scott Rossillo
>
> > Smartling | Senior Software Engineer
>
> > srossillo(a)smartling.com
>
> > > On Dec 14, 2015, at 10:59 AM, Marek Posolda <
mposolda(a)redhat.com >
> > > wrote:
> >
>
> > > On 14/12/15 16:55, Marek Posolda wrote:
> >
>
> > > > On 14/12/15 15:58, Bill Burke wrote:
> > >
> >
>
> > > > > On 12/14/2015 5:01 AM, Niko Köbler wrote:
> > > >
> > >
> >
>
> > > > > > Hi Marek,
> > > > >
> > > >
> > >
> >
>
> > > > > > > Am 14.12.2015 um 08:50 schrieb
Marek Posolda <
> > > > > > > mposolda(a)redhat.com
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > < mailto:mposolda@redhat.com >>:
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > Btv. what's your motivation to
not use infinispan? If you
> > > > > > > afraid
> > > > > > > of
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > cluster communication, you don't need to worry
much about it,
> > > > > > > because
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > if you run single keycloak through standalone.xml,
the
> > > > > > > infinispan
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > automatically works in LOCAL mode and there is no any
cluster
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > communication at all.
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > My current customer is running his apps
in AWS. As known,
> > > > > > multicast
> > > > > > is
> > > > >
> > > >
> > >
> >
>
> > > > > > not available in cloud infrastructures. Wildfly/Infinispan
> > > > > > Cluster
> > > > > > works
> > > > >
> > > >
> > >
> >
>
> > > > > > pretty well with multicast w/o having to know too much
about
> > > > > > JGroups
> > > > >
> > > >
> > >
> >
>
> > > > > > config. S3_PING seams to be a viable way to get a cluster
running
> > > > > > in
> > > > > > AWS.
> > > > >
> > > >
> > >
> >
>
> > > > > > But additionally, my customer doesn’t have any (deep)
knowledge
> > > > > > about
> > > > >
> > > >
> > >
> >
>
> > > > > > JBoss infrastructures and so I’m looking for a way to be
able to
> > > > > > run
> > > > >
> > > >
> > >
> >
>
> > > > > > Keycloak in a cluster in AWS without the need to build up
deeper
> > > > >
> > > >
> > >
> >
>
> > > > > > knowlegde of JGroups config, for example in getting rid of
> > > > > > Infinispan.
> > > > >
> > > >
> > >
> >
>
> > > > > > But I do understand all the concerns in doing this.
> > > > >
> > > >
> > >
> >
>
> > > > > > I still have to test S3_PING, if it works as easy as
multicast.
> > > > > > If
> > > > > > yes,
> > > > >
> > > >
> > >
> >
>
> > > > > > we can use it, if no… I don’t know yet. But this gets
offtopic
> > > > > > for
> > > > >
> > > >
> > >
> >
>
> > > > > > Keycloak mailinglist, it’s more related to pure
> > > > > > Wildfly/Infinispan.
> > > > >
> > > >
> > >
> >
>
> > > > > seems to me it would be much easier to get
Infinispan working on
> > > > > AWS
> > > >
> > >
> >
>
> > > > > than to write and maintain an entire new caching mechanism and
hope
> > > > > we
> > > >
> > >
> >
>
> > > > > don't refactor the cache SPI.
> > > >
> > >
> >
>
> > > > +1
> > >
> >
>
> > > > I am sure infinispan/JGroups has possibility to
run in non-multicast
> > >
> >
>
> > > > environment. You may just need to figure how exactly to configure
it.
> > > > So
> > >
> >
>
> > > > I agree that this issue is more related to Wildfly/Infinispan itself
> > >
> >
>
> > > > than to Keycloak.
> > >
> >
>
> > > > You may need to use jgroups protocols like TCP
instead of default UDP
> > >
> >
>
> > > > and maybe TCPPING (this requires to manually list all your cluster
> > >
> >
>
> > > > nodes. But still, it's much better option IMO than rewriting
> > > > UserSession
> > >
> >
>
> > > > SPI)
> > >
> >
>
> > > Btv. if TCPPING or S3_PING is an issue, there is also
AWS_PING
> >
>
> > > http://www.jgroups.org/manual-3.x/html/protlist.html#d0e5100 , but
it's
> >
>
> > > not official part of jgroups.
> >
>
> > > Marek
> >
>
> > > > Marek
> > >
> >
>
> > > > _______________________________________________
> > >
> >
>
> > > > keycloak-user mailing list
> > >
> >
>
> > > > keycloak-user(a)lists.jboss.org
> > >
> >
>
> > > > https://lists.jboss.org/mailman/listinfo/keycloak-user
> > >
> >
>
> > > _______________________________________________
> >
>
> > > keycloak-user mailing list
> >
>
> > > keycloak-user(a)lists.jboss.org
> >
>
> > > https://lists.jboss.org/mailman/listinfo/keycloak-user
> >
>
3291
days inactive
3303
days old
24 comments
7 participants
participants (7)
-
Alan Field -
Bill Burke -
Marek Posolda -
Niko Köbler -
Paul Blair -
Scott Rossillo -
Stian Thorgersen