3:53 a.m.
You are absolutely right, but at the time this was all the information I had and just
wanted to check if anyone else had a similar experience.
We narrowed it down to 1 realm receiving a lot of requests. All the requests are
originating from the customer's ADFS that fills up our log. It seems that those
requests aren't even logged at the keycloak realm, but the "logout all
sessions" button in the sessions part of the realm does stop the storm.
To answer your questions, customers didn't see the login page and keycloak didn't
process HTTP requests anymore. KeyCloak is just one instance. I don't know the amount
of database connections at that time. Will certainly look into those metrics next time.
-----Original Message-----
From: keycloak-user-bounces(a)lists.jboss.org [mailto:keycloak-user-bounces@lists.jboss.org]
On Behalf Of Bill Burke
Sent: dinsdag 1 augustus 2017 16:31
To: keycloak-user(a)lists.jboss.org
Subject: Re: [keycloak-user] KeyCloak 3.1.0 on OpenShift randomly unresponsive
You'll need to narrow down the problem more. i.e. What does "can't login
anymore" mean? Do customers still see login pages? Can Keycloak still receive and
process HTTP requests? Or is there connection denied? Is Keycloak clustered? Or is it
one instance? How many open database connections does the DB have?
On 8/1/17 5:47 AM, Anton Arntz wrote:
We are currently facing an issue on our production environment in
which the KeyCloak server becomes unresponsive at (what still looks like) random.
Tried to look into memory, cpu load and disk usage of the specific OpenShift gear and
gone through all of the logs but nothing out of the ordinary could be found.
Looks like the application continues to run and still keeps logging, but none of the
customers (realms) is able to login anymore.
Has anyone experienced the same with this KeyCloak version?
Kind regards,
Anton Arntz
_______________________________________________
keycloak-user mailing list
keycloak-user(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/keycloak-user
_______________________________________________
keycloak-user mailing list
keycloak-user(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/keycloak-user
8:35 a.m.
New subject: KeyCloak 3.1.0 on OpenShift randomly unresponsive
How can ADFS make requests to Keycloak? Wouldn't it be other way around?
On 8/8/17 4:53 AM, Anton Arntz wrote:
You are absolutely right, but at the time this was all the
information I had and just wanted to check if anyone else had a similar experience.
We narrowed it down to 1 realm receiving a lot of requests. All the requests are
originating from the customer's ADFS that fills up our log. It seems that those
requests aren't even logged at the keycloak realm, but the "logout all
sessions" button in the sessions part of the realm does stop the storm.
To answer your questions, customers didn't see the login page and keycloak didn't
process HTTP requests anymore. KeyCloak is just one instance. I don't know the amount
of database connections at that time. Will certainly look into those metrics next time.
-----Original Message-----
From: keycloak-user-bounces(a)lists.jboss.org
[mailto:keycloak-user-bounces@lists.jboss.org] On Behalf Of Bill Burke
Sent: dinsdag 1 augustus 2017 16:31
To: keycloak-user(a)lists.jboss.org
Subject: Re: [keycloak-user] KeyCloak 3.1.0 on OpenShift randomly unresponsive
You'll need to narrow down the problem more. i.e. What does "can't login
anymore" mean? Do customers still see login pages? Can Keycloak still receive and
process HTTP requests? Or is there connection denied? Is Keycloak clustered? Or is it
one instance? How many open database connections does the DB have?
On 8/1/17 5:47 AM, Anton Arntz wrote:
> We are currently facing an issue on our production environment in which the KeyCloak
server becomes unresponsive at (what still looks like) random.
> Tried to look into memory, cpu load and disk usage of the specific OpenShift gear and
gone through all of the logs but nothing out of the ordinary could be found.
> Looks like the application continues to run and still keeps logging, but none of the
customers (realms) is able to login anymore.
> Has anyone experienced the same with this KeyCloak version?
>
> Kind regards,
> Anton Arntz
>
> _______________________________________________
> keycloak-user mailing list
> keycloak-user(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/keycloak-user
_______________________________________________
keycloak-user mailing list
keycloak-user(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/keycloak-user
7:50 a.m.
New subject: KeyCloak 3.1.0 on OpenShift randomly unresponsive
Correct, the first step is a redirect from KeyCloak to the ADFS server to authenticate the
user.
This initial redirect happens only once.
After that we see more than 1000 requests hitting our KeyCloak URL with a redirect from
that ADFS server and redirecting back to the ADFS server.
I mean like this:
KeyCloak -> ADFS
ADFS -> KeyCloak
KeyCloak -> ADFS
ADFS -> KeyCloak
KeyCloak -> ADFS
ADFS -> KeyCloak
We reached out to our customer to check if one of their users has a different cookie
configuration in his/her browser.
Best way forward seems to be to locate the storm generator first.
-----Original Message-----
From: Bill Burke [mailto:bburke@redhat.com]
Sent: dinsdag 8 augustus 2017 15:35
To: Anton Arntz <Anton.Arntz(a)planonsoftware.com>; keycloak-user(a)lists.jboss.org
Subject: Re: [keycloak-user] KeyCloak 3.1.0 on OpenShift randomly unresponsive
How can ADFS make requests to Keycloak? Wouldn't it be other way around?
On 8/8/17 4:53 AM, Anton Arntz wrote:
You are absolutely right, but at the time this was all the
information I had and just wanted to check if anyone else had a similar experience.
We narrowed it down to 1 realm receiving a lot of requests. All the requests are
originating from the customer's ADFS that fills up our log. It seems that those
requests aren't even logged at the keycloak realm, but the "logout all
sessions" button in the sessions part of the realm does stop the storm.
To answer your questions, customers didn't see the login page and keycloak didn't
process HTTP requests anymore. KeyCloak is just one instance. I don't know the amount
of database connections at that time. Will certainly look into those metrics next time.
-----Original Message-----
From: keycloak-user-bounces(a)lists.jboss.org
[mailto:keycloak-user-bounces@lists.jboss.org] On Behalf Of Bill Burke
Sent: dinsdag 1 augustus 2017 16:31
To: keycloak-user(a)lists.jboss.org
Subject: Re: [keycloak-user] KeyCloak 3.1.0 on OpenShift randomly unresponsive
You'll need to narrow down the problem more. i.e. What does "can't login
anymore" mean? Do customers still see login pages? Can Keycloak still receive and
process HTTP requests? Or is there connection denied? Is Keycloak clustered? Or is it
one instance? How many open database connections does the DB have?
On 8/1/17 5:47 AM, Anton Arntz wrote:
> We are currently facing an issue on our production environment in which the KeyCloak
server becomes unresponsive at (what still looks like) random.
> Tried to look into memory, cpu load and disk usage of the specific OpenShift gear and
gone through all of the logs but nothing out of the ordinary could be found.
> Looks like the application continues to run and still keeps logging, but none of the
customers (realms) is able to login anymore.
> Has anyone experienced the same with this KeyCloak version?
>
> Kind regards,
> Anton Arntz
>
> _______________________________________________
> keycloak-user mailing list
> keycloak-user(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/keycloak-user
_______________________________________________
keycloak-user mailing list
keycloak-user(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/keycloak-user
8:45 a.m.
New subject: KeyCloak 3.1.0 on OpenShift randomly unresponsive
I assume the protocol for accessing ADFS is SAML, is that correct? Can
anything relevant be found in ADFS Event log? Is the Keycloak source
trusted? What is content of the ADFS messages? If that is SAML status
response with error code, what is the error code? You can view the
content of SAML messages either in browser (if fronchannel is used) or
by raising debug level in keycloak [1]. Is the certificate KeyInfo set
correctly to CERT_SUBJECT?
[1] Troubleshooting section of
http://blog.keycloak.org/2017/03/how-to-setup-ms-ad-fs-30-as-brokered.html
On Wed, Aug 9, 2017 at 2:50 PM, Anton Arntz
<Anton.Arntz(a)planonsoftware.com> wrote:
Correct, the first step is a redirect from KeyCloak to the ADFS
server to authenticate the user.
This initial redirect happens only once.
After that we see more than 1000 requests hitting our KeyCloak URL with a redirect from
that ADFS server and redirecting back to the ADFS server.
I mean like this:
KeyCloak -> ADFS
ADFS -> KeyCloak
KeyCloak -> ADFS
ADFS -> KeyCloak
KeyCloak -> ADFS
ADFS -> KeyCloak
We reached out to our customer to check if one of their users has a different cookie
configuration in his/her browser.
Best way forward seems to be to locate the storm generator first.
-----Original Message-----
From: Bill Burke [mailto:bburke@redhat.com]
Sent: dinsdag 8 augustus 2017 15:35
To: Anton Arntz <Anton.Arntz(a)planonsoftware.com>; keycloak-user(a)lists.jboss.org
Subject: Re: [keycloak-user] KeyCloak 3.1.0 on OpenShift randomly unresponsive
How can ADFS make requests to Keycloak? Wouldn't it be other way around?
On 8/8/17 4:53 AM, Anton Arntz wrote:
> You are absolutely right, but at the time this was all the information I had and just
wanted to check if anyone else had a similar experience.
> We narrowed it down to 1 realm receiving a lot of requests. All the requests are
originating from the customer's ADFS that fills up our log. It seems that those
requests aren't even logged at the keycloak realm, but the "logout all
sessions" button in the sessions part of the realm does stop the storm.
> To answer your questions, customers didn't see the login page and keycloak
didn't process HTTP requests anymore. KeyCloak is just one instance. I don't know
the amount of database connections at that time. Will certainly look into those metrics
next time.
>
> -----Original Message-----
> From: keycloak-user-bounces(a)lists.jboss.org
[mailto:keycloak-user-bounces@lists.jboss.org] On Behalf Of Bill Burke
> Sent: dinsdag 1 augustus 2017 16:31
> To: keycloak-user(a)lists.jboss.org
> Subject: Re: [keycloak-user] KeyCloak 3.1.0 on OpenShift randomly unresponsive
>
> You'll need to narrow down the problem more. i.e. What does "can't
login anymore" mean? Do customers still see login pages? Can Keycloak still receive
and process HTTP requests? Or is there connection denied? Is Keycloak clustered? Or is
it one instance? How many open database connections does the DB have?
>
> On 8/1/17 5:47 AM, Anton Arntz wrote:
>> We are currently facing an issue on our production environment in which the
KeyCloak server becomes unresponsive at (what still looks like) random.
>> Tried to look into memory, cpu load and disk usage of the specific OpenShift gear
and gone through all of the logs but nothing out of the ordinary could be found.
>> Looks like the application continues to run and still keeps logging, but none of
the customers (realms) is able to login anymore.
>> Has anyone experienced the same with this KeyCloak version?
>>
>> Kind regards,
>> Anton Arntz
>>
>> _______________________________________________
>> keycloak-user mailing list
>> keycloak-user(a)lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/keycloak-user
> _______________________________________________
> keycloak-user mailing list
> keycloak-user(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/keycloak-user
_______________________________________________
keycloak-user mailing list
keycloak-user(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/keycloak-user
--
--Hynek
9 a.m.
New subject: KeyCloak 3.1.0 on OpenShift randomly unresponsive
Yes, SAML is used for accessing ADFS. The ADFS/Domain manager of our customer is
troubleshooting on their end. The KeyCloak source is trusted, in fact it seems to be one
of their users causing this issue. For the rest of the users this setup works fine and it
doesn't generate the redirect storm. We didn't enable the debug logging yet since
it's our production gear and that would fill up our gear really fast. I do believe if
the customer can't find the issue on the ADFS end, we have no option left but to
enable it so we can see the contents of the response. The certificate KeyInfo is set to
CERT_SUBJECT. I think we assisted the customer and used the guide you are referring to :)
Thanks for writing it by the way.
-----Original Message-----
From: Hynek Mlnarik [mailto:hmlnarik@redhat.com]
Sent: woensdag 9 augustus 2017 15:45
To: Anton Arntz <Anton.Arntz(a)planonsoftware.com>
Cc: Bill Burke <bburke(a)redhat.com>; keycloak-user(a)lists.jboss.org
Subject: Re: [keycloak-user] KeyCloak 3.1.0 on OpenShift randomly unresponsive
I assume the protocol for accessing ADFS is SAML, is that correct? Can anything relevant
be found in ADFS Event log? Is the Keycloak source trusted? What is content of the ADFS
messages? If that is SAML status response with error code, what is the error code? You can
view the content of SAML messages either in browser (if fronchannel is used) or by raising
debug level in keycloak [1]. Is the certificate KeyInfo set correctly to CERT_SUBJECT?
[1] Troubleshooting section of
http://blog.keycloak.org/2017/03/how-to-setup-ms-ad-fs-30-as-brokered.html
On Wed, Aug 9, 2017 at 2:50 PM, Anton Arntz <Anton.Arntz(a)planonsoftware.com> wrote:
Correct, the first step is a redirect from KeyCloak to the ADFS
server to authenticate the user.
This initial redirect happens only once.
After that we see more than 1000 requests hitting our KeyCloak URL with a redirect from
that ADFS server and redirecting back to the ADFS server.
I mean like this:
KeyCloak -> ADFS
ADFS -> KeyCloak
KeyCloak -> ADFS
ADFS -> KeyCloak
KeyCloak -> ADFS
ADFS -> KeyCloak
We reached out to our customer to check if one of their users has a different cookie
configuration in his/her browser.
Best way forward seems to be to locate the storm generator first.
-----Original Message-----
From: Bill Burke [mailto:bburke@redhat.com]
Sent: dinsdag 8 augustus 2017 15:35
To: Anton Arntz <Anton.Arntz(a)planonsoftware.com>;
keycloak-user(a)lists.jboss.org
Subject: Re: [keycloak-user] KeyCloak 3.1.0 on OpenShift randomly
unresponsive
How can ADFS make requests to Keycloak? Wouldn't it be other way around?
On 8/8/17 4:53 AM, Anton Arntz wrote:
> You are absolutely right, but at the time this was all the information I had and just
wanted to check if anyone else had a similar experience.
> We narrowed it down to 1 realm receiving a lot of requests. All the requests are
originating from the customer's ADFS that fills up our log. It seems that those
requests aren't even logged at the keycloak realm, but the "logout all
sessions" button in the sessions part of the realm does stop the storm.
> To answer your questions, customers didn't see the login page and keycloak
didn't process HTTP requests anymore. KeyCloak is just one instance. I don't know
the amount of database connections at that time. Will certainly look into those metrics
next time.
>
> -----Original Message-----
> From: keycloak-user-bounces(a)lists.jboss.org
> [mailto:keycloak-user-bounces@lists.jboss.org] On Behalf Of Bill
> Burke
> Sent: dinsdag 1 augustus 2017 16:31
> To: keycloak-user(a)lists.jboss.org
> Subject: Re: [keycloak-user] KeyCloak 3.1.0 on OpenShift randomly
> unresponsive
>
> You'll need to narrow down the problem more. i.e. What does "can't
login anymore" mean? Do customers still see login pages? Can Keycloak still receive
and process HTTP requests? Or is there connection denied? Is Keycloak clustered? Or is
it one instance? How many open database connections does the DB have?
>
> On 8/1/17 5:47 AM, Anton Arntz wrote:
>> We are currently facing an issue on our production environment in which the
KeyCloak server becomes unresponsive at (what still looks like) random.
>> Tried to look into memory, cpu load and disk usage of the specific OpenShift gear
and gone through all of the logs but nothing out of the ordinary could be found.
>> Looks like the application continues to run and still keeps logging, but none of
the customers (realms) is able to login anymore.
>> Has anyone experienced the same with this KeyCloak version?
>>
>> Kind regards,
>> Anton Arntz
>>
>> _______________________________________________
>> keycloak-user mailing list
>> keycloak-user(a)lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/keycloak-user
> _______________________________________________
> keycloak-user mailing list
> keycloak-user(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/keycloak-user
_______________________________________________
keycloak-user mailing list
keycloak-user(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/keycloak-user
--
--Hynek
2699
days inactive
2701
days old
4 comments
3 participants
participants (3)
-
Anton Arntz -
Bill Burke -
Hynek Mlnarik