2:44 p.m.
[Note: you may get 2 copies of this email, I sent one previously from my
private email account and it was held for moderator approval, re-sending
this under my redhat account which is subscribed to this list.]
Using the latest release candidate
/devel/candidates/jboss/sso/RHSSO-7.0.0 built on 6/13
the server will not initialize, server.log has a number of errors the
following being the significant one I believe.
Failed to start service
jboss.undertow.deployment.default-server.default-host./auth
For a while now we've seen errors related to database operations in the
log, those errors are also present in the attached log but even with
those database errors the server had seemed to start OK.
Would someone be kind enough to look at the attached log and suggest why
the server won't start and what errors in the log should be concerning?
Many thanks,
--
John
12:13 p.m.
The first error means that there are existing tables in the local H2
database (under standalone/data there are keycloak.* files).
It looks like the logic determined that they are of some previous db schema
version, and tried to upgrade the schema to latest version, but
unexpectedly the schema in place already seems to contain the tables it
wasn't supposed to contain.
I suppose that could happen if upgrade process is interrupted by restarting
the server?
Since you are using the default H2 database I assume you don't care about
any existing data. The solution for you then is to stop the server, delete
the database (rm standalone/data/keycloak.*), and start the server again.
On Sun, Jun 19, 2016 at 2:44 PM, John Dennis <jdennis(a)redhat.com> wrote:
[Note: you may get 2 copies of this email, I sent one previously from
my
private email account and it was held for moderator approval, re-sending
this under my redhat account which is subscribed to this list.]
Using the latest release candidate /devel/candidates/jboss/sso/RHSSO-7.0.0
built on 6/13
the server will not initialize, server.log has a number of errors the
following being the significant one I believe.
Failed to start service
jboss.undertow.deployment.default-server.default-host./auth
For a while now we've seen errors related to database operations in the
log, those errors are also present in the attached log but even with those
database errors the server had seemed to start OK.
Would someone be kind enough to look at the attached log and suggest why
the server won't start and what errors in the log should be concerning?
Many thanks,
--
John
_______________________________________________
keycloak-dev mailing list
keycloak-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/keycloak-dev
2:07 a.m.
On 06/20/2016 06:13 AM, Marko Strukelj wrote:
The first error means that there are existing tables in the local H2
database (under standalone/data there are keycloak.* files).
It looks like the logic determined that they are of some previous db
schema version, and tried to upgrade the schema to latest version, but
unexpectedly the schema in place already seems to contain the tables it
wasn't supposed to contain.
I suppose that could happen if upgrade process is interrupted by
restarting the server?
Since you are using the default H2 database I assume you don't care
about any existing data. The solution for you then is to stop the
server, delete the database (rm standalone/data/keycloak.*), and start
the server again.
Thank you Marko, I've got a few more questions for you.
These errors occur during automated installation and configuration via
ansible.
One of the operations performed is invoking bin/add-user-keycloak to add
the admin user. I seem to recall add-user-keycloak operates on static
files which are read during start up. Could the use of add-user-keycloak
trigger the schema errors seen in the log?
This is a brand new install so why would there be an upgrade process
running?
The ansible scripts do restart the server. Starting the server is done
via bin/standalone.sh but stopping the server is performed by systemd
sending a SIGTERM, waiting and then sending a SIGKILL (or so I believe).
Does the upgrade process gracefully handle SIGTERM such that it
continues to run until complete and then exit?
--
John
11:27 a.m.
On Tue, Jun 21, 2016 at 2:07 AM, John Dennis <jdennis(a)redhat.com> wrote:
On 06/20/2016 06:13 AM, Marko Strukelj wrote:
> The first error means that there are existing tables in the local H2
> database (under standalone/data there are keycloak.* files).
>
> It looks like the logic determined that they are of some previous db
> schema version, and tried to upgrade the schema to latest version, but
> unexpectedly the schema in place already seems to contain the tables it
> wasn't supposed to contain.
>
> I suppose that could happen if upgrade process is interrupted by
> restarting the server?
>
> Since you are using the default H2 database I assume you don't care
> about any existing data. The solution for you then is to stop the
> server, delete the database (rm standalone/data/keycloak.*), and start
> the server again.
>
Thank you Marko, I've got a few more questions for you.
These errors occur during automated installation and configuration via
ansible.
One of the operations performed is invoking bin/add-user-keycloak to add
the admin user. I seem to recall add-user-keycloak operates on static files
which are read during start up. Could the use of add-user-keycloak trigger
the schema errors seen in the log?
No.
This is a brand new install so why would there be an upgrade process
running?
One possibility would be that the packaged installation already contains
standalone/data/keycloak.* files, which it shouldn't - maybe it's a custom
packaging you put together?
Another possibility is that the error does not happen on first start, but
on subsequent start, after server is forcefully restarted.
The ansible scripts do restart the server. Starting the server is
done via
bin/standalone.sh but stopping the server is performed by systemd sending a
SIGTERM, waiting and then sending a SIGKILL (or so I believe). Does the
upgrade process gracefully handle SIGTERM such that it continues to run
until complete and then exit?
This is the most likely culprit. I don't think upgrade procedure will
properly complete when java process is set to shutdown no matter what
signal is used.
The shutdown should only be performed after server has reached started
state.
That's when you see an entry in the log similar to:
11:26:41,410 INFO [org.jboss.as] (Controller Boot Thread) WFLYSRV0025:
Keycloak 1.9.7.Final (WildFly Core 2.0.10.Final) started in 10219ms -
Started 416 of 782 services (526 services are lazy, passive or on-demand)
2:21 p.m.
On 06/21/2016 05:27 AM, Marko Strukelj wrote:
On Tue, Jun 21, 2016 at 2:07 AM, John Dennis <jdennis(a)redhat.com
<mailto:jdennis@redhat.com>> wrote:
On 06/20/2016 06:13 AM, Marko Strukelj wrote:
The first error means that there are existing tables in the local H2
database (under standalone/data there are keycloak.* files).
It looks like the logic determined that they are of some previous db
schema version, and tried to upgrade the schema to latest
version, but
unexpectedly the schema in place already seems to contain the
tables it
wasn't supposed to contain.
I suppose that could happen if upgrade process is interrupted by
restarting the server?
Since you are using the default H2 database I assume you don't care
about any existing data. The solution for you then is to stop the
server, delete the database (rm standalone/data/keycloak.*), and
start
the server again.
Thank you Marko, I've got a few more questions for you.
These errors occur during automated installation and configuration
via ansible.
One of the operations performed is invoking bin/add-user-keycloak to
add the admin user. I seem to recall add-user-keycloak operates on
static files which are read during start up. Could the use of
add-user-keycloak trigger the schema errors seen in the log?
No.
This is a brand new install so why would there be an upgrade process
running?
One possibility would be that the packaged installation already contains
standalone/data/keycloak.* files, which it shouldn't - maybe it's a
custom packaging you put together?
Another possibility is that the error does not happen on first start,
but on subsequent start, after server is forcefully restarted.
The ansible scripts do restart the server. Starting the server is
done via bin/standalone.sh but stopping the server is performed by
systemd sending a SIGTERM, waiting and then sending a SIGKILL (or so
I believe). Does the upgrade process gracefully handle SIGTERM such
that it continues to run until complete and then exit?
This is the most likely culprit. I don't think upgrade procedure will
properly complete when java process is set to shutdown no matter what
signal is used.
The shutdown should only be performed after server has reached started
state.
That's when you see an entry in the log similar to:
11:26:41,410 INFO [org.jboss.as <http://org.jboss.as>] (Controller Boot
Thread) WFLYSRV0025: Keycloak 1.9.7.Final (WildFly Core 2.0.10.Final)
started in 10219ms - Started 416 of 782 services (526 services are lazy,
passive or on-demand)
From reading the log (I've attached a copy from another run) I think
the server is stopping before it properly initialized. I'm not familiar
with all your log messages but I base this conclusion on the fact
"stopping" messages appear in the log just after the database errors,
followed by this "stop" message:
[org.jboss.as] (MSC service thread 1-4) WFLYSRV0050: rh-sso 7.0.0.GA
(WildFly Core 2.1.2.Final-redhat-1) stopped in 2100ms
The "start" message
[org.jboss.modules] (main) JBoss Modules version 1.5.1.Final-redhat-1
occurs next followed by more database error messages and then finally
the "ready to run" message you cite:
[org.jboss.as] (Controller Boot Thread) WFLYSRV0026: rh-sso 7.0.0.GA
(WildFly Core 2.1.2.Final-redhat-1) started (with errors) in 12451ms -
Started 473 of 839 services (2 services failed or missing dependencies,
583 services are lazy, passive or on-demand)
Question:
Does the Keycloak team have a working script to start and stop the
service in RHEL? When we first started working with Keycloak we were
told no and we would need to cobble together something by calling the
bin/standalone.sh script.
The "wait for full initialization" problem is not new to us with
daemons. It's come up a number of times with IPA and other daemons we
work with. The way we've dealt with it is to have our service scripts
that start and stop services write to one of the primary sockets and
only when it gets a valid response back conclude the service is in fact
up (handling timeouts of course). Systemd came along later and might
have some support for socket detection, I'll investigate that option.
We really need a script that can start and stop the service reliably
without errors.
--
John
3:18 p.m.
I don't know about availability of RHEL service script. But the log is very
clear about what happens:
2016-06-20 18:30:13,282 INFO
[org.keycloak.connections.jpa.updater.liquibase.LiquibaseJpaUpdaterProvider]
(ServerService Thread Pool -- 54) Initializing database schema
2016-06-20 18:30:13,351 INFO [org.jboss.as.server] (Thread-2) WFLYSRV0220:
Server shutdown has been requested.
Server shutdown is initiated while server is still in the process of
booting up.
On Tue, Jun 21, 2016 at 2:21 PM, John Dennis <jdennis(a)redhat.com> wrote:
On 06/21/2016 05:27 AM, Marko Strukelj wrote:
>
>
> On Tue, Jun 21, 2016 at 2:07 AM, John Dennis <jdennis(a)redhat.com
> <mailto:jdennis@redhat.com>> wrote:
>
> On 06/20/2016 06:13 AM, Marko Strukelj wrote:
>
> The first error means that there are existing tables in the local
> H2
> database (under standalone/data there are keycloak.* files).
>
> It looks like the logic determined that they are of some previous
> db
> schema version, and tried to upgrade the schema to latest
> version, but
> unexpectedly the schema in place already seems to contain the
> tables it
> wasn't supposed to contain.
>
> I suppose that could happen if upgrade process is interrupted by
> restarting the server?
>
> Since you are using the default H2 database I assume you don't
> care
> about any existing data. The solution for you then is to stop the
> server, delete the database (rm standalone/data/keycloak.*), and
> start
> the server again.
>
>
> Thank you Marko, I've got a few more questions for you.
>
> These errors occur during automated installation and configuration
> via ansible.
>
> One of the operations performed is invoking bin/add-user-keycloak to
> add the admin user. I seem to recall add-user-keycloak operates on
> static files which are read during start up. Could the use of
> add-user-keycloak trigger the schema errors seen in the log?
>
>
> No.
>
>
>
> This is a brand new install so why would there be an upgrade process
> running?
>
>
> One possibility would be that the packaged installation already contains
> standalone/data/keycloak.* files, which it shouldn't - maybe it's a
> custom packaging you put together?
> Another possibility is that the error does not happen on first start,
> but on subsequent start, after server is forcefully restarted.
>
>
> The ansible scripts do restart the server. Starting the server is
> done via bin/standalone.sh but stopping the server is performed by
> systemd sending a SIGTERM, waiting and then sending a SIGKILL (or so
> I believe). Does the upgrade process gracefully handle SIGTERM such
> that it continues to run until complete and then exit?
>
>
> This is the most likely culprit. I don't think upgrade procedure will
> properly complete when java process is set to shutdown no matter what
> signal is used.
> The shutdown should only be performed after server has reached started
> state.
>
> That's when you see an entry in the log similar to:
>
> 11:26:41,410 INFO [org.jboss.as <http://org.jboss.as>] (Controller Boot
> Thread) WFLYSRV0025: Keycloak 1.9.7.Final (WildFly Core 2.0.10.Final)
> started in 10219ms - Started 416 of 782 services (526 services are lazy,
> passive or on-demand)
>
From reading the log (I've attached a copy from another run) I think the
server is stopping before it properly initialized. I'm not familiar with
all your log messages but I base this conclusion on the fact "stopping"
messages appear in the log just after the database errors, followed by this
"stop" message:
[org.jboss.as] (MSC service thread 1-4) WFLYSRV0050: rh-sso 7.0.0.GA
(WildFly Core 2.1.2.Final-redhat-1) stopped in 2100ms
The "start" message
[org.jboss.modules] (main) JBoss Modules version 1.5.1.Final-redhat-1
occurs next followed by more database error messages and then finally the
"ready to run" message you cite:
[org.jboss.as] (Controller Boot Thread) WFLYSRV0026: rh-sso 7.0.0.GA
(WildFly Core 2.1.2.Final-redhat-1) started (with errors) in 12451ms -
Started 473 of 839 services (2 services failed or missing dependencies, 583
services are lazy, passive or on-demand)
Question:
Does the Keycloak team have a working script to start and stop the service
in RHEL? When we first started working with Keycloak we were told no and we
would need to cobble together something by calling the bin/standalone.sh
script.
The "wait for full initialization" problem is not new to us with daemons.
It's come up a number of times with IPA and other daemons we work with. The
way we've dealt with it is to have our service scripts that start and stop
services write to one of the primary sockets and only when it gets a valid
response back conclude the service is in fact up (handling timeouts of
course). Systemd came along later and might have some support for socket
detection, I'll investigate that option.
We really need a script that can start and stop the service reliably
without errors.
--
John
3:34 p.m.
On 6/21/16 8:21 AM, John Dennis wrote:
The "wait for full initialization" problem is not new to us with
daemons. It's come up a number of times with IPA and other daemons we
work with. The way we've dealt with it is to have our service scripts
that start and stop services write to one of the primary sockets and
only when it gets a valid response back conclude the service is in fact
up (handling timeouts of course). Systemd came along later and might
have some support for socket detection, I'll investigate that option.
This is good feedback. I'm not sure what you mean by writing to a
primary socket. You mean HTTP(S) 80/443? I'm pretty sure HTTP(S)
sockets are set up before Keycloak is even deployed. This is because
internally Keycloak has a dependency on HTTP subsystem and won't be
initialized until that subsystem is started.
Can you parse System output for a specific string? Is that viable?
I'll ping Wildfly team to see how they've handled stuff like this.
Bill
4:02 p.m.
On 06/21/2016 09:34 AM, Bill Burke wrote:
On 6/21/16 8:21 AM, John Dennis wrote:
>
> The "wait for full initialization" problem is not new to us with
> daemons. It's come up a number of times with IPA and other daemons we
> work with. The way we've dealt with it is to have our service scripts
> that start and stop services write to one of the primary sockets and
> only when it gets a valid response back conclude the service is in fact
> up (handling timeouts of course). Systemd came along later and might
> have some support for socket detection, I'll investigate that option.
>
This is good feedback. I'm not sure what you mean by writing to a
primary socket. You mean HTTP(S) 80/443? I'm pretty sure HTTP(S)
sockets are set up before Keycloak is even deployed. This is because
internally Keycloak has a dependency on HTTP subsystem and won't be
initialized until that subsystem is started.
Sorry, I guess I wasn't clear. What I meant was to send a request and
wait for a valid response. It makes sense the servlet container would
have it's listening sockets set up much earlier but Keycloak won't
respond to a request on a valid endpoint until it's fully initialized.
What I was trying to describe was something like this, a loop iterates
trying to send a request, it ignores common socket errors, if a valid
response is received it terminates the loop with a success status. If
after a specified time interval or iteration count a valid response has
not been received it terminates the loop with a failure status.
Of course this means there is some endpoint which does not require
authentication nor has any side-effect. I'm guessing the REST API
exposes something which could be used for this purpose but I haven't
looked into it.
Can you parse System output for a specific string? Is that viable?
I'll ping Wildfly team to see how they've handled stuff like this.
Trying to parse the log is problematic. Normally the log is appended to
(until log rotation occurs). It could be tricky to identify the matching
log message associated with the server lifetime. Also in some scenarios
log messages may not be flushed immediately, especially if logging is
configured to go to a network location. It also would make the service
script dependent on knowing the logging configuration and exact messages
to look for which may change between versions, a maintenance issue.
--
John
4:46 p.m.
I think you can ping http://localhost:8080/auth/ - until you get 200 OK.
Don't forget the final / in the url, otherwise you'll get 303 See Other
rather than 200 OK.
On Tue, Jun 21, 2016 at 4:02 PM, John Dennis <jdennis(a)redhat.com> wrote:
On 06/21/2016 09:34 AM, Bill Burke wrote:
>
>
> On 6/21/16 8:21 AM, John Dennis wrote:
>>
>> The "wait for full initialization" problem is not new to us with
>> daemons. It's come up a number of times with IPA and other daemons we
>> work with. The way we've dealt with it is to have our service scripts
>> that start and stop services write to one of the primary sockets and
>> only when it gets a valid response back conclude the service is in fact
>> up (handling timeouts of course). Systemd came along later and might
>> have some support for socket detection, I'll investigate that option.
>>
>
> This is good feedback. I'm not sure what you mean by writing to a
> primary socket. You mean HTTP(S) 80/443? I'm pretty sure HTTP(S)
> sockets are set up before Keycloak is even deployed. This is because
> internally Keycloak has a dependency on HTTP subsystem and won't be
> initialized until that subsystem is started.
Sorry, I guess I wasn't clear. What I meant was to send a request and
wait for a valid response. It makes sense the servlet container would
have it's listening sockets set up much earlier but Keycloak won't
respond to a request on a valid endpoint until it's fully initialized.
What I was trying to describe was something like this, a loop iterates
trying to send a request, it ignores common socket errors, if a valid
response is received it terminates the loop with a success status. If
after a specified time interval or iteration count a valid response has
not been received it terminates the loop with a failure status.
Of course this means there is some endpoint which does not require
authentication nor has any side-effect. I'm guessing the REST API
exposes something which could be used for this purpose but I haven't
looked into it.
> Can you parse System output for a specific string? Is that viable?
> I'll ping Wildfly team to see how they've handled stuff like this.
Trying to parse the log is problematic. Normally the log is appended to
(until log rotation occurs). It could be tricky to identify the matching
log message associated with the server lifetime. Also in some scenarios
log messages may not be flushed immediately, especially if logging is
configured to go to a network location. It also would make the service
script dependent on knowing the logging configuration and exact messages
to look for which may change between versions, a maintenance issue.
--
John
_______________________________________________
keycloak-dev mailing list
keycloak-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/keycloak-dev
12:46 a.m.
FYI, the following Jira issue was opened for this.
https://issues.jboss.org/browse/KEYCLOAK-3150
--
John
8:37 a.m.
Initializing the database can take a while (1-2min), so you'd need a
relatively long timeout on a first startup. Either pinging '
http://localhost:8080/auth/' or looking in the log would work just fine.
You can also use jboss-cli to check the status, for example to check if
server is started fully:
bin/jboss-cli.sh --connect ':read-attribute(name=server-state)'
Or check the status of Keycloak individually:
bin/jboss-cli.sh --connect 'deployment-info --name=keycloak-server.war'
An alternative is to use Liqubase to extract the SQL schema for your
database. That way Ansible can setup the database fully prior to starting
Keycloak in the first place. If you're interested in that approach let me
know and I'll let you know the details for that.
On 22 June 2016 at 00:46, John Dennis <jdennis(a)redhat.com> wrote:
FYI, the following Jira issue was opened for this.
https://issues.jboss.org/browse/KEYCLOAK-3150
--
John
_______________________________________________
keycloak-dev mailing list
keycloak-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/keycloak-dev
2:05 p.m.
Another thing to try would be:
bin/jboss-cli.sh --connect shutdown
This one will wait until first time boot is completed properly, and only
then perform a shutdown.
If it times out while waiting, the process will exit with error code, and
display a message:
Failed to connect to the controller: Timeout waiting for the system to boot.
On Wed, Jun 22, 2016 at 8:37 AM, Stian Thorgersen <sthorger(a)redhat.com>
wrote:
Initializing the database can take a while (1-2min), so you'd
need a
relatively long timeout on a first startup. Either pinging '
http://localhost:8080/auth/' or looking in the log would work just fine.
You can also use jboss-cli to check the status, for example to check if
server is started fully:
bin/jboss-cli.sh --connect ':read-attribute(name=server-state)'
Or check the status of Keycloak individually:
bin/jboss-cli.sh --connect 'deployment-info --name=keycloak-server.war'
An alternative is to use Liqubase to extract the SQL schema for your
database. That way Ansible can setup the database fully prior to starting
Keycloak in the first place. If you're interested in that approach let me
know and I'll let you know the details for that.
On 22 June 2016 at 00:46, John Dennis <jdennis(a)redhat.com> wrote:
> FYI, the following Jira issue was opened for this.
>
> https://issues.jboss.org/browse/KEYCLOAK-3150
>
> --
> John
> _______________________________________________
> keycloak-dev mailing list
> keycloak-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/keycloak-dev
>
3:04 p.m.
Thank you Stian and Marko for your suggestions regarding the use of
bin/jboss-cli.sh. I added your suggestions into the Jira so the
information would be collected in one place for future reference.
bin/jboss-cli.sh probably has a role to play in the start up and
shutdown scripts.
--
John
2864
days inactive
2867
days old
12 comments
4 participants
participants (4)
-
Bill Burke -
John Dennis -
Marko Strukelj -
Stian Thorgersen