[keycloak-user] Running two Keycloak instances in parallel

[Knurr, Michael]

Hi Hynek

When passing the parameter -Djboss.as.management.blocking.timeout=900 to standalone.sh, I still get an transaction timeout after 5 minutes. The JBoss deployment timeout follows 10 minutes later

13:48:58,480 INFO  [org.keycloak.connections.jpa.updater.liquibase.LiquibaseJpaUpdaterProvider] (ServerService Thread Pool -- 54) Updating database. Using changelog META-INF/jpa-changelog-master.xml
13:53:56,978 WARN  [com.arjuna.ats.arjuna] (Transaction Reaper) ARJUNA012117: TransactionReaper::check timeout for TX 0:ffff7f000101:3bafd8ef:5a5df4b7:d in state  RUN
13:53:56,986 WARN  [com.arjuna.ats.arjuna] (Transaction Reaper Worker 0) ARJUNA012121: TransactionReaper::doCancellations worker Thread[Transaction Reaper Worker 0,5,main] successfully canceled TX 0:ffff7f000101:3bafd8ef:5a5df4b7:d
13:53:57,240 WARN  [com.arjuna.ats.arjuna] (Transaction Reaper) ARJUNA012117: TransactionReaper::check timeout for TX 0:ffff7f000101:3bafd8ef:5a5df4b7:10 in state  RUN
13:53:57,241 WARN  [com.arjuna.ats.arjuna] (Transaction Reaper Worker 0) ARJUNA012121: TransactionReaper::doCancellations worker Thread[Transaction Reaper Worker 0,5,main] successfully canceled TX 0:ffff7f000101:3bafd8ef:5a5df4b7:10
14:03:55,819 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0348: Timeout after [900] seconds waiting for service container stability. Operation will roll back. Step that first updated the service container was 'add' at address '[
    ("core-service" => "management"),
    ("management-interface" => "http-interface")
]'
14:04:00,825 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0190: Step handler org.jboss.as.server.DeployerChainAddHandler$FinalRuntimeStepHandler at 3cf14117<mailto:org.jboss.as.server.DeployerChainAddHandler$FinalRuntimeStepHandler at 3cf14117> for operation add-deployer-chains at address [] failed handling operation rollback -- java.util.concurrent.TimeoutException: java.util.concurrent.TimeoutException
        at org.jboss.as.controller.OperationContextImpl.waitForRemovals(OperationContextImpl.java:522)
…

However, when the previous keycloak instance is stopped, the upgrade runs through within 55 seconds for about 25k users. As far as i could see, keycloak only uses the java:jboss/datasources/KeycloakDS which points to jdbc:h2:${jboss.server.data.dir}/keycloak;AUTO_SERVER=TRUE

For persistence I use the standard builtin H2 database connection so I just assumed that each wildfly installation would use its own h2 database and infinispan cache. Apparently this is not the case.

I already tried the upgrade using 900, 1200 and 1800 seconds for timeout, without any luck.

Do you have any other idea how I can debug this?

Brgds
Michael


Von: Hynek Mlnarik [mailto:hmlnarik at redhat.com]
Gesendet: Dienstag, 16. Januar 2018 11:31
An: Knurr, Michael <Michael.Knurr at adesso.ch>
Cc: keycloak-user <keycloak-user at lists.jboss.org>
Betreff: Re: [keycloak-user] Running two Keycloak instances in parallel

It could be for many reasons. Just guessing - since it is terminating after 300 seconds (5 minutes) while the last log entry is about updating database, maybe the database connection/operation is too slow. Hence you can try increasing the startup period value to e.g. 900 seconds by passing -Djboss.as.management.blocking.timeout=900 parameter to standalone.sh. Also make sure that the database of the new Keycloak instance is distinct from the original one.

On Tue, Jan 16, 2018 at 10:30 AM, Knurr, Michael <Michael.Knurr at adesso.ch<mailto:Michael.Knurr at adesso.ch>> wrote:
Hi all

When trying to upgrade Keycloak to the latest version I ran into problems. Unfortunately It turned out that we ran into different upgrade behaviors on different machines. With the latest version, the upgrade is running locally, on test and on stage. As a last step I now tried to test the upgrade procedure on the production machine with a keycloak instance running on a different port.

When running the upgrade on a server with an already running keycloak instance, the new server runs into a timeout because of concurrency.

10:16:24,926 INFO  [org.keycloak.connections.jpa.updater.liquibase.LiquibaseJpaUpdaterProvider] (ServerService Thread Pool -- 58) Updating database. Using changelog META-INF/jpa-changelog-master.xml
10:21:22,322 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0348: Timeout after [300] seconds waiting for service container stability. Operation will roll back. Step that first updated the service container was 'add' at address '[
    ("core-service" => "management"),
    ("management-interface" => "http-interface")
]'
10:21:23,397 WARN  [com.arjuna.ats.arjuna] (Transaction Reaper) ARJUNA012117: TransactionReaper::check timeout for TX 0:ffff7f000101:73d290d3:5a5dc2e6:d in state  RUN
10:21:23,402 WARN  [com.arjuna.ats.arjuna] (Transaction Reaper Worker 0) ARJUNA012121: TransactionReaper::doCancellations worker Thread[Transaction Reaper Worker 0,5,main] successfully canceled TX 0:ffff7f000101:73d290d3:5a5dc2e6:d
10:21:23,668 WARN  [com.arjuna.ats.arjuna] (Transaction Reaper) ARJUNA012117: TransactionReaper::check timeout for TX 0:ffff7f000101:73d290d3:5a5dc2e6:10 in state  RUN
10:21:23,669 WARN  [com.arjuna.ats.arjuna] (Transaction Reaper Worker 0) ARJUNA012121: TransactionReaper::doCancellations worker Thread[Transaction Reaper Worker 0,5,main] successfully canceled TX 0:ffff7f000101:73d290d3:5a5dc2e6:10
10:21:27,328 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0190: Step handler org.jboss.as.server.DeployerChainAddHandler$FinalRuntimeStepHandler at 20b4c4cb<mailto:org.jboss.as.server.DeployerChainAddHandler$FinalRuntimeStepHandler at 20b4c4cb> for operation add-deployer-chains at address [] failed handling operation rollback -- java.util.concurrent.TimeoutException: java.util.concurrent.TimeoutException
        at org.jboss.as.controller.OperationContextImpl.waitForRemovals(OperationContextImpl.java:522)
...

What might be the reason for this? Shouldn't it possible to run two keycloak servers (with a different port offset) on a single machine? How can I test the keycloak upgrade on the production machine without stopping the running server instance?


Best regards
Michael
_______________________________________________
keycloak-user mailing list
keycloak-user at lists.jboss.org<mailto:keycloak-user at lists.jboss.org>
https://lists.jboss.org/mailman/listinfo/keycloak-user



--

--Hynek