Database types for primary and foreign keys
by Hynek Mlnarik
Hi All,
Apologies for a long e-mail.
TLDR: We need to define format of primary keys (UUID) so that it is possible to transform the primary and foreign keys from VARCHAR(36) columns into database-native binary format. This is in particular important to document in 2.5 in Storage Ids section of the new User Storage SPI [1]
Long version:
I have looked at current database model and while generally it looks well, there is an interesting issue with primary / foreign keys that causes performance degradation on both Java and - most importantly - database side, causing even deadlocks for some databases.
The issue comes from database handling of IDs. IDs are in fact UUIDs, i.e. series of bytes that are represented by Strings in KC JPA classes. Why this causes performance degradation is due to various representations conversions (byte array vs String in Java) and - most importantly for database - character set conversions. In the worst case, The conversions occur both in JDBC driver and the database. The consequences are demonstrated by Jira issue KEYCLOAK-3210 [2] when several simultaneous requests lead to deadlocking the database.
When JDBC driver obtains a string, it converts its representation into a character set understood by database. Database might need to convert the string to a character set specified for the column. This is nicely illustrated in MSSQL which makes distinction between VARCHAR (8-bit codepages) and NVARCHAR (UCS-2 Unicode charset). IDs are VARCHARs which is indeed an efficient way to use strings that consist of ASCII-only characters (though not optimal for UUID, read below). However, if Unicode characters are to be supported, MSSQL JDBC driver sends all character parameters as Unicode Strings [3]. Database then performs a conversion from Unicode to 8-bit charset which generally loses some data. To account for this loss, instead of performing an index scan that directly points to a requested row, it returns a range where the requested record should be. This has fatal impact on performance. For more detailed analysis of the resulting plans, see comment in [2].
Clearly, the scan by id should be fast and the format of IDs in database matters. It should avoid conversions as much as possible. Hence the following plan came:
* In the result, all primary keys and corresponding keys have to be represented by binary UUID data type (where supported, some databases represent UUID as e.g. VARBINARY(16)), i.e. 16 bytes instead of 36 bytes
* All keys in the JPA classes should be of type UUID, not String
As a result, database indices get smaller (16 bytes of indexed data per record vs 36 bytes as it is now in case of 8-bit storage of characters), and no character conversions are in place, hence the overall performance increases.
This task is a slighty big one so it won't fit into KC 2.5 timeframe, but we should definitely aim for 3.0.
This has several preconditions:
* The String keys in keycloak JPA classes, wherever used, are restricted to UUID format
* This format is documented and respected by all custom implementations, namely User Storage implementations.
* There exists conversion from String to native UUID for used databases (this is certainly possible for PostgreSQL MSSQL, DB2, and MySQL, most likely others)
Similarly to JPA, Infinispan classes should be revisited and optimized to save some bytes that might be important for cluster replication by replacing String with UUIDs
Thoughts?
--Hynek
[1] https://github.com/keycloak/server_development_guide/blob/6b82f0868c0d7a1...
[2] https://issues.jboss.org/browse/KEYCLOAK-3210
[3] https://msdn.microsoft.com/en-us/library/ms378857(v=sql.110).aspx
7 years, 4 months
Re: IE login in new session logs out the other user
by Michael Gerber
That's true. It shares the cookie which does not have set httpOnly to true.
It's obviously an IE fail, however, I need a workaround for that :)
Do you have any idea how to solve this?
Am 16. Dezember 2016 um 15:14 schrieb Stian Thorgersen <sthorger(a)redhat.com>:
... Doesn't
On 16 December 2016 at 15:13, Stian Thorgersen <sthorger(a)redhat.com> wrote:
Does sound like IE actually creates a clean new session as it's sharing some cookies.
On 16 December 2016 at 13:10, Michael Gerber <gerbermichi(a)me.com> wrote:
Hi,
I am using Windows 7 and Internet Explorer 11.
IE can create a new window with a new session. It should be possible to work with two different users in this two windows. However, the second login logs the older user out, because of the KEYCLOAK_SESSION cookie which is stored in the "C:\Users\{username}\AppData\Roaming\Microsoft\Windows\Cookies" directory. The problem is, that this cookie is not set to httpOnly.
Is this a known bug? Or can I solve this problem?
kind regards
Michael
_______________________________________________
keycloak-dev mailing list
keycloak-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/keycloak-dev
7 years, 4 months
IE login in new session logs out the other user
by Michael Gerber
Hi,
I am using Windows 7 and Internet Explorer 11.
IE can create a new window with a new session. It should be possible to work with two different users in this two windows. However, the second login logs the older user out, because of the KEYCLOAK_SESSION cookie which is stored in the "C:\Users\{username}\AppData\Roaming\Microsoft\Windows\Cookies" directory. The problem is, that this cookie is not set to httpOnly.
Is this a known bug? Or can I solve this problem?
kind regards
Michael
7 years, 4 months
Keycloak 2.5.0.CR1 release
by Stian Thorgersen
2.5.0.CR1 will be released on 21st December. Due to Christmas breaks
2.5.0.Final will not be released until 4th January.
I want all changes in by end of 19th December. Everyone around should help
with testing on Tuesday 20th.
Please check what issues are assigned to you for Keycloak 2.5.0.CR1 and
RH-SSO 7.1.0.ER3. If you don't believe you can complete yours by end of
Monday 19th December let me know.
7 years, 4 months
broker import should be local only?
by Bill Burke
I'm looking at the broker flow code and it seems that we import users
into whatever storage provider supports adding users. Should this import
be local only and bypass any User Storage Providers? This breaks
backwards compatbility, but I'm not sure the old approach was the
correct one.
Thoughts?
7 years, 4 months
Federated Users
by gambol
Hiya
What would the authentication flow for the following scenario; I've added
SAML provider to Office365, but I only want self-registration and associate
to if a user with the same email address has already been created in the
realm. I've tried disabling the "Create if unique" but it complains with
"No duplication detected"
Rohith
7 years, 4 months
Admin client interfaces not implemented in services
by Scott Rossillo
I’ve been doing some work around the admin client and endpoints. I noticed that org.keycloak.services.resources.admin.UsersResource does not implement the org.keycloak.admin.client.resource.UsersResource interface. Is there an intentional reason for this?
It would be easier to keep the server implementation honest to the APIs if the interfaces were implemented plus simplify implementation discovery. Seems there are redundant POJOs as a result of this too.
What do you guys think about modifying the admin service to implement the client interfaces?
Thanks,
Scott
Scott Rossillo
Smartling | Senior Software Engineer
srossillo(a)smartling.com
7 years, 4 months
ServletFilter Adapter Cookie Token Store
by Laghuvaram, Raghu
I see that cookie token-store would not be supported until 2.x as per the
comments in https://issues.jboss.org/browse/KEYCLOAK-2662, Is it fixed in
any of the recent versions? It seems like its not working in 2.3.0 Final.
Thanks,
Raghu
________________________________
Notice: This communication may contain privileged and/or confidential information. If you are not the intended recipient, please notify the sender by email, and immediately delete the message and any attachments without copying or disclosing them. LB may, for any reason, intercept, access, use, and disclose any information that is communicated by or through, or which is stored on, its networks, applications, services, and devices.
7 years, 4 months
Re: [keycloak-dev] [keycloak-user] Considering removing Mongo support
by Stian Thorgersen
You can't go wrong with Oracle (other than price obviously) and PostgreSQL
is a good database as well. That's my 2 cents at least, but then again I'm
not a db guru ;)
On 3 December 2016 at 10:09, Byte Flinger <byteflinger(a)gmail.com> wrote:
> Does that mean that the only supported backends would be SQL databases? I
> have recently started to look into Keycloak and I was thinking that Mongodb
> support was nice for scalability as it can be sharded, something SQL dbs
> cannot. Wouldn't that mean giving up on scalability for large deployments?
>
> Are there plans to support any other more scalable type of database such
> as Cassandra?
>
> On Fri, 2 Dec 2016, 11:30 Stian Thorgersen, <sthorger(a)redhat.com> wrote:
>
>> All,
>>
>> We are considering removing Mongo support from Keycloak in 3.x. The
>> reasons
>> behind it is that there are a fair few issues in the current
>> implementation, especially around consistency due to lack of transaction
>> support in Mongo and often we update multiple documents. In many cases we
>> rely on transactions to rollback to prevent partial updates, but this
>> obviously doesn't work in Mongo.
>>
>> With the fact that Mongo is already partially broken and the constant
>> maintenance involved we're considering removing it and rather focus purely
>> on the relational database back-end.
>>
>> Another point to make is that we are not considering supporting Mongo in
>> the supported version of Keycloak (Red Hat Single Sign-On). So we are
>> never
>> able to provide the same level of care and attention to it as we can for
>> relational databases.
>>
>> If we do decide to remove it we would make sure we provide a seamless and
>> easy option to migrate from Mongo to a relational database!
>>
>> I would like to gather some feedback from the community before doing
>> anything. So please vote on the following Doodle:
>>
>> http://doodle.com/poll/nnimebpkx774ppus
>>
>> Also, comments to this thread is more than welcome!
>>
>> I'll end with a comment - Time spent by core developer on maintaining
>> Mongo
>> could be better spent on awesome new features, testing and bug fixing!
>> _______________________________________________
>> keycloak-user mailing list
>> keycloak-user(a)lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/keycloak-user
>>
>
7 years, 4 months
