[keycloak-user] Shared datastore?

Thu Nov 15 13:38:59 EST 2018

OK, so I setup one KC node with a remote store, and one JDG (ISPN) server
with a shared JDBC store and it seems to work fine. Thanks!
I would have preferred not to add a JDG server but at least I have a
solution.

But a few things remain unclear:
  - each time I update/insert/delete a session entity from KC, the JDG
server will always be updated, right?
  - what about read operations? Does it mean there are 2 independent
caches, one in KC and one in ISPN (JMX shows same numEntriesInMemory for
both)?

So let's say I am not interested in Cross DC yet, but i want to set up a
cluster of 3 VMs (with HAProxy in front of them), each VM holding one KC
process + one ISPN process:

* VM#1 : kc_node11 && ispn_node11 (kc_node11 uses ispn_node11 as remote
cache)
* VM#2 : kc_node21 && ispn_node21 (kc_node21 uses ispn_node21 as remote
cache)
* VM#3 : kc_node31 && ispn_node31 (kc_node31 uses ispn_node31 as remote
cache)

Which cache mode
<http://infinispan.org/docs/stable/user_guide/user_guide.html#which_cache_mode_should_i_use>
should I use then and how many owners, for both KC and ISPN?  Invalidation
mode and numOfOwner=2 for KC, and distributed mode and numOfOwner=2 for
ISPN? Would that work?

Thanks again
--nick

Le jeu. 15 nov. 2018 à 11:55, Marek Posolda <mposolda at redhat.com> a écrit :

> Yes, true. We're using SKIP_CACHE_STORE when writing to sessions. We never
> tested with CacheStores enabled.
>
> The only store, which we're tested with, is the "remote-store" which we're
> using for the cross-datacenter setup. We have lots of places when we're not
> just writing data to the "cache" directly and let the "remote-store" to
> propagate it, but instead we obtain "remoteCache" instance from the
> underlying remote-store and CRUD data directly to remoteCache to have some
> optimizations and guaranteed consistency and atomicity for remoteCache
> operations (EG. putIfAbsent, replace etc). That's also the reason why we're
> using SKIP_CACHE_STORE flag.
>
> Feel free to create JIRA for better support of other CacheStores.
>
> The other possibility to workaround this (besides what Sebastian already
> mentioned) is to have JDG server and configure your cache with the
> remote-store as described in our "Cross-Datacenter setup" documentation. On
> JDG side, you can configure the JDBC store to your cache. In other words,
> the session will be always written to JDG and JDG will write it to the
> undrlying JDBC. I know this option is far from ideal (you need to add JDG
> server just to workaround things), just mentioning it for completeness.
>
> Marek
>
>
> On 09/11/18 14:29, Sebastian Laskawiec wrote:
>
> Yes, I think that could be case, I see a plenty of places where we
> use SKIP_CACHE_STORE.
>
> Let me ask Marek for help here since it has been implemented long before I
> joined the team and I don't know the history behind it...
>
> On Thu, Nov 8, 2018 at 8:48 PM William Burns <wburns at redhat.com> wrote:
>
>>
>>
>> ----- Original Message -----
>> > From: "Sebastian Laskawiec" <slaskawi at redhat.com>
>> > To: "Nicolas Ocquidant" <nocquidant at gmail.com>
>> > Cc: keycloak-user at lists.jboss.org, "Will Burns Rosenquist Burns" <
>> wburns at redhat.com>
>> > Sent: Thursday, November 8, 2018 12:33:47 PM
>> > Subject: Re: [keycloak-user] Shared datastore?
>> >
>> > So I think there are at least two ways to address this problem. This
>> first
>> > one is to use Offline Tokens [1]. I'm not sure if that fits into your
>> > application since it requires your client applications to store the
>> token.
>> > In other words you can simply delegate this problem one layer below in
>> your
>> > system.
>> >
>> > If that doesn't work for you, yes passivation is a way to go. Frankly, I
>> > haven't used passivation but from the manual I see it works hand in hand
>> > with eviction [2][3]. Will (on CC) can probably correct me here, but my
>> > understanding is that whenever an entry gets evicted, the passivation
>> > mechanism picks it up and stores somewhere.
>>
>> It does and it works, the problem is that passivation doesn't play well
>> with shared stores in Infinispan. We prevent this configuration in 9.4 or
>> newer even.
>>
>> I recommended that Nicolas just use eviction and a shared store without
>> passivation. However it seems that entries are not written to the store in
>> this configuration. My guess is that KeyCloak performs write operations
>> with the SKIP_CACHE_STORE flag and assumes entries will only be written to
>> the store due to passivation. Is there a reason for that?
>>
>> >
>> > [1] http://blog.keycloak.org/2015/12/offline-tokens-in-keycloak.html
>> > [2]
>> >
>> http://infinispan.org/docs/stable/user_guide/user_guide.html#cache_passivation
>> > [3]
>> >
>> https://github.com/infinispan/infinispan/blob/master/core/src/test/java/org/infinispan/eviction/impl/EvictionWithPassivationTest.java#L61-L69
>> >
>> > On Thu, Nov 8, 2018 at 5:40 PM Nicolas Ocquidant <nocquidant at gmail.com>
>> > wrote:
>> >
>> > > My requirements are the following: store tokens emitted by KC during
>> one
>> > > year.
>> > >
>> > > I don't know how many users there are, but here are the number I get:
>> > >   * the number of connections a week is about 700k.
>> > >   * the number of session refresh a week is about 200k.
>> > >
>> > > I approximated around 1M of sessions a week, thus 52M a year.
>> > > In memory, a user session has been estimated around 4KB (about 1KB in
>> > > file/DB).
>> > >
>> > > But I guess a refresh does not create another session isn't it? And
>> maybe
>> > > it's possible to ask KC to delete previous emitted tokens when a new
>> one is
>> > > created for a same user?
>> > >
>> > > If yes, my estimation is probably a little bit too high here, but I
>> > > certainly have several millions of tokens to keep (and maybe dozens of
>> > > millions).
>> > >
>> > > Thanks
>> > > --nick
>> > >
>> > > Le mer. 7 nov. 2018 à 18:17, Nicolas Ocquidant <nocquidant at gmail.com>
>> a
>> > > écrit :
>> > >
>> > > > Hi,
>> > > >
>> > > > According to Infinispan, when passivation is disabled, every update
>> to
>> > > the
>> > > > cache should always write to the store.
>> > > >
>> > > > But I can't manage to get it work with Keycloak. If I disable
>> > > passivation,
>> > > > my SQL store (Postgres) stays empty, even if the cache is full.
>> > > >
>> > > > So, if passivation is needed for Keycloak to write to the DB, it
>> means
>> > > > that the use of a shared DB is not possible...
>> > > >
>> > > > But this leads to another issue for me. Enable passivation without a
>> > > > shared DB seems to imply that either 'fetch-state' or 'purge'
>> should be
>> > > > enabled on startup, in order for the cache to not contain stale
>> entries.
>> > > >
>> > > > 15:27:44,626 WARN
>> > > >
>> [org.infinispan.configuration.cache.AbstractStoreConfigurationBuilder]
>> > > (MSC
>> > > > service thread 1-6) ISPN000149: Fetch persistent state and purge on
>> > > startup
>> > > > are both disabled, cache may contain stale entries on startup
>> > > >
>> > > > As I need to keep millions of sessions, this will considerably slow
>> down
>> > > > the startup of my node (when started again after a crash for
>> instance).
>> > > >
>> > > > So, is shared datastore allowed in Keycloak? If yes, how to enable
>> it?
>> > > > Otherwise what other options do I have to improve my startup time,
>> if
>> > > > millions of sessions are in the store?
>> > > >
>> > > > Thanks
>> > > > --nick
>> > > >
>> > > _______________________________________________
>> > > keycloak-user mailing list
>> > > keycloak-user at lists.jboss.org
>> > > https://lists.jboss.org/mailman/listinfo/keycloak-user
>> >
>>
>
>