[keycloak-user] Keycloak token exchange failure behind loadbalancer and reverse proxy

[Niels Bertram]

I am scratching my head with a specific setup problem which does not
generate any usable error messages.

I am running a haproxy as load balancer in a vm in front an apache web
server configured as reverse proxy connecting to the keycloak server via
ajp in another VM.

client browser (192.168.33.1)

       login.vagrant.v8 (192.168.33.80) aka proxy.vagrant.v8 is haproxy
adds X-Forwarded-For X-Forwarded-Port X-Forwarded-Proto and X-Real-Ip

                kc01.vagrant.v8 (192.168.33.81) apache reverse proxies to
wildfly on ajp port


Followed all the setup instructions in the documentation and if I connect
to apache proxying through to keycloak everything works fine. All web
resources are donwloaded fine however when I request a token exchange on
/auth/realms/master/protocol/openid-connect/token I get a 400 response. The
kc server log shows the corect IP address of the originating client and the
request dump from wildfly also shows the correct  X-Forwarded-For header
coming in. However the query string remoteAddr=/192.168.33.80:54672 which I
believe is the one sent to the ajp connector shows some half valid IP
address which is that of the load balancer. Did anyone come across this
before? Looks like a bug of some sort.

The symptom is a endless loop trying to log into the admin panel.

Cheers
Niels


# cat standalone/log/server.log | grep -A 58 '2016-05-24 09:19:27,672'
2016-05-24 09:19:27,672 WARN  [org.keycloak.events] (default task-19)
type=CODE_TO_TOKEN_ERROR, realmId=master, clientId=admin, userId=null,
ipAddress=*192.168.33.1*, error=invalid_client_credentials,
grant_type=authorization_code
2016-05-24 09:19:27,673 INFO  [io.undertow.request.dump] (default task-19)
----------------------------REQUEST---------------------------
               URI=/auth/realms/master/protocol/openid-connect/token
 characterEncoding=null
     contentLength=229
       contentType=[application/x-www-form-urlencoded]

cookie=KC_RESTART=eyJhbGciOiJIUzI1NiJ9.eyJjcyI6IjM5YTVkNTlmLWMyNDYtNDkwZi04ZGZkLTZhYzVhNzgyZDI5ZCIsImNpZCI6InNlY3VyaXR5LWFkbWluLWNvbnNvbGUiLCJwdHkiOiJvcGVuaWQtY29ubmVjdCIsInJ1cmkiOiJodHRwczovL2xvZ2luLnZhZ3JhbnQudjgvYXV0aC9hZG1pbi9tYXN0ZXIvY29uc29sZS8iLCJhY3QiOiJBVVRIRU5USUNBVEUiLCJub3RlcyI6eyJhY3Rpb25fa2V5IjoiOGQ5ZGI4MzctMjY2Ni00NDcxLTk0ZDgtZmFkMmIxMjA3NDQxIiwiYXV0aF90eXBlIjoiY29kZSIsImlzcyI6Imh0dHBzOi8vbG9naW4udmFncmFudC52OC9hdXRoL3JlYWxtcy9tYXN0ZXIiLCJyZXNwb25zZV90eXBlIjoiY29kZSIsInJlZGlyZWN0X3VyaSI6Imh0dHBzOi8vbG9naW4udmFncmFudC52OC9hdXRoL2FkbWluL21hc3Rlci9jb25zb2xlLyIsInN0YXRlIjoiNTQ4ZDE5ZTUtMjlkMS00NTU2LWFjNjAtYjZjZTM1ZmJiMGU2Iiwibm9uY2UiOiI5OGY3NDFiYS03MmQwLTQ0ZDUtOGQ0ZC1jZTAxNzZhYjMyMmUiLCJyZXNwb25zZV9tb2RlIjoiZnJhZ21lbnQifX0.I0jI4nDhbYtKNrVjdlwjjBe5mtd0a8u6Dm7rQXwLE60

cookie=KEYCLOAK_IDENTITY=eyJhbGciOiJSUzI1NiJ9.eyJqdGkiOiJhNjY5OWJkOS00MWQ4LTQyNWYtYjE5Ni04Y2QzNmJiZjBmNjQiLCJleHAiOjE0NjQxMTc1NjcsIm5iZiI6MCwiaWF0IjoxNDY0MDgxNTY3LCJpc3MiOiJodHRwczovL2xvZ2luLnZhZ3JhbnQudjgvYXV0aC9yZWFsbXMvbWFzdGVyIiwic3ViIjoiY2YyNDg4MTEtNmQ4Mi00N2U3LWJmOWEtN2IxOTdmYjk4OGQwIiwic2Vzc2lvbl9zdGF0ZSI6IjFiYTljODRlLTBlMzctNGE4Mi1hNDg0LWMyNWQyYzRhODBmYyIsInJlc291cmNlX2FjY2VzcyI6e319.E0vEe9XQJ_6IbDC_TEUfumQCJ0fS1_AOYsHh7svyGp16VC89sH9J1FQuLJfHYFVJlDTcE6o2ktLg0fLw2nLIdLOv-WXMseYr0KzudZveiLy1CZbRoPS9w9vlN-_EuXojiz0ORcyh90keUhqW5tMShccHvEaq_wpXOJQ6ITIglsgUXNhlSuEfpEcBy4CCqKQW98bRQiTKQOtoOfgc-Ez1RHR-7esTw-U22P_H-EMk23jI3nwuYGtqOn4Vvqb4-cHOzdyE_xaVWZxeteNKhU-RexfrMaHx1PSy3T796aY7gIljcqkxra-SA1dbOsRBawwlhJwFtojzBHEs1841gJ4bgg

cookie=KEYCLOAK_SESSION=master/cf248811-6d82-47e7-bf9a-7b197fb988d0/1ba9c84e-0e37-4a82-a484-c25d2c4a80fc
            header=Accept=*/*
            header=Accept-Language=en-US,en;q=0.8,de;q=0.6
            header=Accept-Encoding=gzip, deflate
            header=DNT=1
            header=Origin=https://login.vagrant.v8
            header=X-Original-To=192.168.33.80
            header=User-Agent=Mozilla/5.0 (Windows NT 6.1; WOW64)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36
            header=X-Forwarded-Proto=https
            header=X-Forwarded-Port=443
            header=X-Forwarded-For=192.168.33.1
            header=Content-Length=229
            header=Content-Type=application/x-www-form-urlencoded

header=Cookie=KC_RESTART=eyJhbGciOiJIUzI1NiJ9.eyJjcyI6IjM5YTVkNTlmLWMyNDYtNDkwZi04ZGZkLTZhYzVhNzgyZDI5ZCIsImNpZCI6InNlY3VyaXR5LWFkbWluLWNvbnNvbGUiLCJwdHkiOiJvcGVuaWQtY29ubmVjdCIsInJ1cmkiOiJodHRwczovL2xvZ2luLnZhZ3JhbnQudjgvYXV0aC9hZG1pbi9tYXN0ZXIvY29uc29sZS8iLCJhY3QiOiJBVVRIRU5USUNBVEUiLCJub3RlcyI6eyJhY3Rpb25fa2V5IjoiOGQ5ZGI4MzctMjY2Ni00NDcxLTk0ZDgtZmFkMmIxMjA3NDQxIiwiYXV0aF90eXBlIjoiY29kZSIsImlzcyI6Imh0dHBzOi8vbG9naW4udmFncmFudC52OC9hdXRoL3JlYWxtcy9tYXN0ZXIiLCJyZXNwb25zZV90eXBlIjoiY29kZSIsInJlZGlyZWN0X3VyaSI6Imh0dHBzOi8vbG9naW4udmFncmFudC52OC9hdXRoL2FkbWluL21hc3Rlci9jb25zb2xlLyIsInN0YXRlIjoiNTQ4ZDE5ZTUtMjlkMS00NTU2LWFjNjAtYjZjZTM1ZmJiMGU2Iiwibm9uY2UiOiI5OGY3NDFiYS03MmQwLTQ0ZDUtOGQ0ZC1jZTAxNzZhYjMyMmUiLCJyZXNwb25zZV9tb2RlIjoiZnJhZ21lbnQifX0.I0jI4nDhbYtKNrVjdlwjjBe5mtd0a8u6Dm7rQXwLE60;
KEYCLOAK_IDENTITY=eyJhbGciOiJSUzI1NiJ9.eyJqdGkiOiJhNjY5OWJkOS00MWQ4LTQyNWYtYjE5Ni04Y2QzNmJiZjBmNjQiLCJleHAiOjE0NjQxMTc1NjcsIm5iZiI6MCwiaWF0IjoxNDY0MDgxNTY3LCJpc3MiOiJodHRwczovL2xvZ2luLnZhZ3JhbnQudjgvYXV0aC9yZWFsbXMvbWFzdGVyIiwic3ViIjoiY2YyNDg4MTEtNmQ4Mi00N2U3LWJmOWEtN2IxOTdmYjk4OGQwIiwic2Vzc2lvbl9zdGF0ZSI6IjFiYTljODRlLTBlMzctNGE4Mi1hNDg0LWMyNWQyYzRhODBmYyIsInJlc291cmNlX2FjY2VzcyI6e319.E0vEe9XQJ_6IbDC_TEUfumQCJ0fS1_AOYsHh7svyGp16VC89sH9J1FQuLJfHYFVJlDTcE6o2ktLg0fLw2nLIdLOv-WXMseYr0KzudZveiLy1CZbRoPS9w9vlN-_EuXojiz0ORcyh90keUhqW5tMShccHvEaq_wpXOJQ6ITIglsgUXNhlSuEfpEcBy4CCqKQW98bRQiTKQOtoOfgc-Ez1RHR-7esTw-U22P_H-EMk23jI3nwuYGtqOn4Vvqb4-cHOzdyE_xaVWZxeteNKhU-RexfrMaHx1PSy3T796aY7gIljcqkxra-SA1dbOsRBawwlhJwFtojzBHEs1841gJ4bgg;
KEYCLOAK_SESSION=master/cf248811-6d82-47e7-bf9a-7b197fb988d0/1ba9c84e-0e37-4a82-a484-c25d2c4a80fc
            header=Referer=
https://login.vagrant.v8/auth/admin/master/console/
            header=Host=login.vagrant.v8
            locale=[en_US, en, de]
            method=POST
          protocol=HTTP/1.1
       queryString=
*        remoteAddr=/192.168.33.80:54672 <http://192.168.33.80:54672>*
        remoteHost=proxy.vagrant.v8
            scheme=https
              host=login.vagrant.v8
        serverPort=443
--------------------------RESPONSE--------------------------
     contentLength=123
       contentType=application/json
            header=X-Powered-By=Undertow/1
            header=Server=WildFly/10
            header=Content-Type=application/json
            header=Content-Length=123
            header=Date=Tue, 24 May 2016 09:19:27 GMT
            status=400
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/keycloak-user/attachments/20160524/bd6878a8/attachment-0001.html