[Note: you may get 2 copies of this email, I sent one previously from my private email account and it was held for moderator approval, re-sending this under my redhat account which is subscribed to this list.] Using the latest release candidate /devel/candidates/jboss/sso/RHSSO-7.0.0 built on 6/13 the server will not initialize, server.log has a number of errors the following being the significant one I believe. Failed to start service jboss.undertow.deployment.default-server.default-host./auth For a while now we've seen errors related to database operations in the log, those errors are also present in the attached log but even with those database errors the server had seemed to start OK. Would someone be kind enough to look at the attached log and suggest why the server won't start and what errors in the log should be concerning? Many thanks, -- John _______________________________________________ keycloak-dev mailing list keycloak-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/keycloak-dev

On 06/20/2016 06:13 AM, Marko Strukelj wrote: > The first error means that there are existing tables in the local H2 > database (under standalone/data there are keycloak.* files). > > It looks like the logic determined that they are of some previous db > schema version, and tried to upgrade the schema to latest version, but > unexpectedly the schema in place already seems to contain the tables it > wasn't supposed to contain. > > I suppose that could happen if upgrade process is interrupted by > restarting the server? > > Since you are using the default H2 database I assume you don't care > about any existing data. The solution for you then is to stop the > server, delete the database (rm standalone/data/keycloak.*), and start > the server again. > Thank you Marko, I've got a few more questions for you. These errors occur during automated installation and configuration via ansible. One of the operations performed is invoking bin/add-user-keycloak to add the admin user. I seem to recall add-user-keycloak operates on static files which are read during start up. Could the use of add-user-keycloak trigger the schema errors seen in the log?

This is a brand new install so why would there be an upgrade process running?

The ansible scripts do restart the server. Starting the server is done via bin/standalone.sh but stopping the server is performed by systemd sending a SIGTERM, waiting and then sending a SIGKILL (or so I believe). Does the upgrade process gracefully handle SIGTERM such that it continues to run until complete and then exit?

On 06/21/2016 05:27 AM, Marko Strukelj wrote: > > > On Tue, Jun 21, 2016 at 2:07 AM, John Dennis <jdennis(a)redhat.com > <mailto:jdennis@redhat.com>> wrote: > > On 06/20/2016 06:13 AM, Marko Strukelj wrote: > > The first error means that there are existing tables in the local > H2 > database (under standalone/data there are keycloak.* files). > > It looks like the logic determined that they are of some previous > db > schema version, and tried to upgrade the schema to latest > version, but > unexpectedly the schema in place already seems to contain the > tables it > wasn't supposed to contain. > > I suppose that could happen if upgrade process is interrupted by > restarting the server? > > Since you are using the default H2 database I assume you don't > care > about any existing data. The solution for you then is to stop the > server, delete the database (rm standalone/data/keycloak.*), and > start > the server again. > > > Thank you Marko, I've got a few more questions for you. > > These errors occur during automated installation and configuration > via ansible. > > One of the operations performed is invoking bin/add-user-keycloak to > add the admin user. I seem to recall add-user-keycloak operates on > static files which are read during start up. Could the use of > add-user-keycloak trigger the schema errors seen in the log? > > > No. > > > > This is a brand new install so why would there be an upgrade process > running? > > > One possibility would be that the packaged installation already contains > standalone/data/keycloak.* files, which it shouldn't - maybe it's a > custom packaging you put together? > Another possibility is that the error does not happen on first start, > but on subsequent start, after server is forcefully restarted. > > > The ansible scripts do restart the server. Starting the server is > done via bin/standalone.sh but stopping the server is performed by > systemd sending a SIGTERM, waiting and then sending a SIGKILL (or so > I believe). Does the upgrade process gracefully handle SIGTERM such > that it continues to run until complete and then exit? > > > This is the most likely culprit. I don't think upgrade procedure will > properly complete when java process is set to shutdown no matter what > signal is used. > The shutdown should only be performed after server has reached started > state. > > That's when you see an entry in the log similar to: > > 11:26:41,410 INFO [org.jboss.as <http://org.jboss.as>] (Controller Boot > Thread) WFLYSRV0025: Keycloak 1.9.7.Final (WildFly Core 2.0.10.Final) > started in 10219ms - Started 416 of 782 services (526 services are lazy, > passive or on-demand) > From reading the log (I've attached a copy from another run) I think the server is stopping before it properly initialized. I'm not familiar with all your log messages but I base this conclusion on the fact "stopping" messages appear in the log just after the database errors, followed by this "stop" message: [org.jboss.as] (MSC service thread 1-4) WFLYSRV0050: rh-sso 7.0.0.GA (WildFly Core 2.1.2.Final-redhat-1) stopped in 2100ms The "start" message [org.jboss.modules] (main) JBoss Modules version 1.5.1.Final-redhat-1 occurs next followed by more database error messages and then finally the "ready to run" message you cite: [org.jboss.as] (Controller Boot Thread) WFLYSRV0026: rh-sso 7.0.0.GA (WildFly Core 2.1.2.Final-redhat-1) started (with errors) in 12451ms - Started 473 of 839 services (2 services failed or missing dependencies, 583 services are lazy, passive or on-demand) Question: Does the Keycloak team have a working script to start and stop the service in RHEL? When we first started working with Keycloak we were told no and we would need to cobble together something by calling the bin/standalone.sh script. The "wait for full initialization" problem is not new to us with daemons. It's come up a number of times with IPA and other daemons we work with. The way we've dealt with it is to have our service scripts that start and stop services write to one of the primary sockets and only when it gets a valid response back conclude the service is in fact up (handling timeouts of course). Systemd came along later and might have some support for socket detection, I'll investigate that option. We really need a script that can start and stop the service reliably without errors. -- John

On 6/21/16 8:21 AM, John Dennis wrote: > > The "wait for full initialization" problem is not new to us with > daemons. It's come up a number of times with IPA and other daemons we > work with. The way we've dealt with it is to have our service scripts > that start and stop services write to one of the primary sockets and > only when it gets a valid response back conclude the service is in fact > up (handling timeouts of course). Systemd came along later and might > have some support for socket detection, I'll investigate that option. > This is good feedback. I'm not sure what you mean by writing to a primary socket. You mean HTTP(S) 80/443? I'm pretty sure HTTP(S) sockets are set up before Keycloak is even deployed. This is because internally Keycloak has a dependency on HTTP subsystem and won't be initialized until that subsystem is started.

Can you parse System output for a specific string? Is that viable? I'll ping Wildfly team to see how they've handled stuff like this.

Initializing the database can take a while (1-2min), so you'd need a relatively long timeout on a first startup. Either pinging ' http://localhost:8080/auth/' or looking in the log would work just fine. You can also use jboss-cli to check the status, for example to check if server is started fully: bin/jboss-cli.sh --connect ':read-attribute(name=server-state)' Or check the status of Keycloak individually: bin/jboss-cli.sh --connect 'deployment-info --name=keycloak-server.war' An alternative is to use Liqubase to extract the SQL schema for your database. That way Ansible can setup the database fully prior to starting Keycloak in the first place. If you're interested in that approach let me know and I'll let you know the details for that. On 22 June 2016 at 00:46, John Dennis <jdennis(a)redhat.com&gt; wrote: > FYI, the following Jira issue was opened for this. > > https://issues.jboss.org/browse/KEYCLOAK-3150 > > -- > John > _______________________________________________ > keycloak-dev mailing list > keycloak-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/keycloak-dev >