[keycloak-user] Lots of errors in cross-datacenter replication mode with Infinispan Server 9.4.x

Fri Aug 2 15:00:51 EDT 2019

Hello,

We've built an installation of Keycloak 6.0.1 with cross-datacenter 
replication using Infinispan Server 9.4.15.Final.

About three minutes after starting Keycloak, we start getting lots of 
sequences of the following three errors:
ISPN000136: Error executing command RemoveExpiredCommand on Cache 'work'
ISPN000136: Error executing command RemoveCommand on Cache 'work'
ISPN000071: Caught exception when handling command SingleXSiteRpcCommand

Please, see the attached log for more detail.

The attached log is the server.log one of the Infinispan servers.
There you can see that around 17:49 the Keycloak server started and a 
couple of warnings regarding cache activity are shown. These warnings are 
normal as per Keycloak documentation troubleshooting.
At 17:52, the errors start appearing in a non-stop fashion.
This goes on until one of the infinispan sites automatically goes offline 
(only one repetition included in the log file, because it was too big for 
the mailing list).
The errors are always related to the work cache only.
Note that the log has been anonymized and host, node, site names, IP's, 
ports, etc. do not correspond to the real ones. We have made a best effort 
not to obfuscate any relevant information, however.

This log corresponds to an Infinispan 9.4.15 and a Keycloak 4.8.3, both in 
domain mode, with one node per data center each.
No error appears in Keycloak logs, even in debug mode (or we haven't found 
it).

To discard possible mistakes in the configuration, we have also tested it 
with the exact configuration in:
https://www.keycloak.org/docs/latest/server_installation/index.html#setup

These are the results for serveral combinations of versions, always with 
the exact example configuration from the docs:

Keycloak        Infinispan      Result
========        ========        ======
6.0.1           9.4.15          Error
4.8.3           9.4.15          Error
4.8.3           9.4.0           Error
4.8.3           9.3.6           Works
6.0.1           9.3.6           Does not connect
5.0.0           9.3.6           Does not connect

In the three first setups, the log looks exactly the same as the attached 
log.
In the fourth setup, everything works fine.
In the two last setups, Keycloak couldn't connect to infinispan, probably 
because it's too old.

Our conclusion is that there is a bug showing up only for Infinispan 9.4.x 
and not for Infinispan 9.3.x.
Given that the same version of Keycloak works with Infinispan 9.3 and 
doesn't work with Infinispan 9.4, this may be an Infinispan bug, but we 
have not been able to reproduce it by directly writing onto the work 
cache; we have always needed to start Keycloak to reproduce it.
For this reason, we cannot totally discard a problem in Keycloak which 
would show up only with later versions of Infinispan.

Any help will be appreciated. If you need any further information, don't 
hesitate to ask.

Thank you very much,

Juan

------------------------------------------------------------------------------------------------------------------------------------
Este mensaje va dirigido únicamente a la(s) persona(s) y/o entidad(es) arriba relacionada(s).
Puede contener información confidencial o legalmente protegida. Si no es usted el destinatario
señalado, le rogamos borre del sistema inmediatamente el mensaje y sus copias. Asimismo le
informamos que cualquier copia, divulgación, distribución o uso de los contenidos está prohibida.
---------------
This message is addressed only to the person (people) and / or entities listed above. It may contain confidential or legally protected information. If you are not the recipient indicated, please delete the message and its copies immediately from the system. We also inform that any copy, disclosure, distribution or use of the contents is forbidden
---------------------------------------------------------------------------------------------------------------
Viewnext, S.A. Domicilio Social: Avda. de Burgos 8-A 28036 de Madrid. 
telf: 913834060, Fax: 913834090. Reg. M. Madrid: Tomo 3238, Libro:0, Folio: 78, Seccion: 8ª, Hoja M-55112, N.I.F.: A-80157746

-------------- next part --------------

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/jpeg
Size: 16163 bytes
Desc: not available
Url : http://lists.jboss.org/pipermail/keycloak-user/attachments/20190802/2170f12a/attachment-0003.jpe 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/jpeg
Size: 9338 bytes
Desc: not available
Url : http://lists.jboss.org/pipermail/keycloak-user/attachments/20190802/2170f12a/attachment-0004.jpe 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/jpeg
Size: 9338 bytes
Desc: not available
Url : http://lists.jboss.org/pipermail/keycloak-user/attachments/20190802/2170f12a/attachment-0005.jpe 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: server.log
Type: application/octet-stream
Size: 46953 bytes
Desc: not available
Url : http://lists.jboss.org/pipermail/keycloak-user/attachments/20190802/2170f12a/attachment-0001.obj 