[keycloak-user] KeyCloak 3.1.0 on OpenShift randomly unresponsive

[Anton Arntz]

Yes, SAML is used for accessing ADFS. The ADFS/Domain manager of our customer is troubleshooting on their end. The KeyCloak source is trusted, in fact it seems to be one of their users causing this issue. For the rest of the users this setup works fine and it doesn't generate the redirect storm. We didn't enable the debug logging yet since it's our production gear and that would fill up our gear really fast. I do believe if the customer can't find the issue on the ADFS end, we have no option left but to enable it so we can see the contents of the response. The certificate KeyInfo is set to CERT_SUBJECT. I think we assisted the customer and used the guide you are referring to :) Thanks for writing it by the way.

-----Original Message-----
From: Hynek Mlnarik [mailto:hmlnarik at redhat.com] 
Sent: woensdag 9 augustus 2017 15:45
To: Anton Arntz <Anton.Arntz at planonsoftware.com>
Cc: Bill Burke <bburke at redhat.com>; keycloak-user at lists.jboss.org
Subject: Re: [keycloak-user] KeyCloak 3.1.0 on OpenShift randomly unresponsive

I assume the protocol for accessing ADFS is SAML, is that correct? Can anything relevant be found in ADFS Event log? Is the Keycloak source trusted? What is content of the ADFS messages? If that is SAML status response with error code, what is the error code? You can view the content of SAML messages either in browser (if fronchannel is used) or by raising debug level in keycloak [1]. Is the certificate KeyInfo set correctly to CERT_SUBJECT?

[1] Troubleshooting section of
http://blog.keycloak.org/2017/03/how-to-setup-ms-ad-fs-30-as-brokered.html

On Wed, Aug 9, 2017 at 2:50 PM, Anton Arntz <Anton.Arntz at planonsoftware.com> wrote:
> Correct, the first step is a redirect from KeyCloak to the ADFS server to authenticate the user.
> This initial redirect happens only once.
> After that we see more than 1000 requests hitting our KeyCloak URL with a redirect from that ADFS server and redirecting back to the ADFS server.
> I mean like this:
> KeyCloak -> ADFS
> ADFS -> KeyCloak
> KeyCloak -> ADFS
> ADFS -> KeyCloak
> KeyCloak -> ADFS
> ADFS -> KeyCloak
> We reached out to our customer to check if one of their users has a different cookie configuration in his/her browser.
> Best way forward seems to be to locate the storm generator first.
>
> -----Original Message-----
> From: Bill Burke [mailto:bburke at redhat.com]
> Sent: dinsdag 8 augustus 2017 15:35
> To: Anton Arntz <Anton.Arntz at planonsoftware.com>; 
> keycloak-user at lists.jboss.org
> Subject: Re: [keycloak-user] KeyCloak 3.1.0 on OpenShift randomly 
> unresponsive
>
> How can ADFS make requests to Keycloak?  Wouldn't it be other way around?
>
>
> On 8/8/17 4:53 AM, Anton Arntz wrote:
>> You are absolutely right, but at the time this was all the information I had and just wanted to check if anyone else had a similar experience.
>> We narrowed it down to 1 realm receiving a lot of requests. All the requests are originating from the customer's ADFS that fills up our log. It seems that those requests aren't even logged at the keycloak realm, but the "logout all sessions" button in the sessions part of the realm does stop the storm.
>> To answer your questions, customers didn't see the login page and keycloak didn't process HTTP requests anymore. KeyCloak is just one instance. I don't know the amount of database connections at that time. Will certainly look into those metrics next time.
>>
>> -----Original Message-----
>> From: keycloak-user-bounces at lists.jboss.org 
>> [mailto:keycloak-user-bounces at lists.jboss.org] On Behalf Of Bill 
>> Burke
>> Sent: dinsdag 1 augustus 2017 16:31
>> To: keycloak-user at lists.jboss.org
>> Subject: Re: [keycloak-user] KeyCloak 3.1.0 on OpenShift randomly 
>> unresponsive
>>
>> You'll need to narrow down the problem more.  i.e. What does "can't login anymore" mean?  Do customers still see login pages? Can Keycloak still receive and process HTTP requests?  Or is there connection denied?  Is Keycloak clustered?  Or is it one instance?  How many open database connections does the DB have?
>>
>> On 8/1/17 5:47 AM, Anton Arntz wrote:
>>> We are currently facing an issue on our production environment in which the KeyCloak server becomes unresponsive at (what still looks like) random.
>>> Tried to look into memory, cpu load and disk usage of the specific OpenShift gear and gone through all of the logs but nothing out of the ordinary could be found.
>>> Looks like the application continues to run and still keeps logging, but none of the customers (realms) is able to login anymore.
>>> Has anyone experienced the same with this KeyCloak version?
>>>
>>> Kind regards,
>>> Anton Arntz
>>>
>>> _______________________________________________
>>> keycloak-user mailing list
>>> keycloak-user at lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/keycloak-user
>> _______________________________________________
>> keycloak-user mailing list
>> keycloak-user at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/keycloak-user
>
>
> _______________________________________________
> keycloak-user mailing list
> keycloak-user at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/keycloak-user



-- 

--Hynek