Re: [keycloak-dev] i18n/l10n: English text in java code

By the way I'm pretty sure our translation theme will take our English bundle and translate it to a language. They won't need to start/stop the server and open the admin console to check that they've translated all keys. They should have a tool/script for that. On 9 October 2015 at 13:56, Stian Thorgersen <sthorger(a)redhat.com&gt; wrote: > > > On 9 October 2015 at 13:46, Stan Silvert <ssilvert(a)redhat.com&gt; wrote: > >> On 10/9/2015 7:32 AM, Stian Thorgersen wrote: >> >> #1 You're not going to catch all missing keys this way - as I said >> there's 2 types, custom defined as well which could be missing from default >> bundle >> >> It catches it at load time. As it loads each bundle, it checks against >> the previously loaded bundle. That will indeed catch all missing keys in >> any bundle you try to test. >> >> I don't know exactly what you mean by "custom defined". Somehow a >> third-party bundle must be merged with our default bundle. Unless I >> completely misunderstand, the code I wrote will still work. >> > > New keys can be defined by using keys in client descriptions, client > names, etc, etc.. These won't be in our message bundle, but would be in the > bundle in a customers theme. This is why message bundles inherit from the > parent theme. Custom providers and themes can further introduce new keys. > > >> >> #2 Rendering the whole bundle useless just because you're missing one >> key is just daft >> >> It is the correct thing to do. A missing key is like a null pointer. >> It deserves a RuntimeException. >> > > No it's not the correct thing to do. > > >> >> #3 There will quite likely be separate teams that do translations to >> those that do development, which means stack traces and log output is not >> the solution >> >> I don't see what that has to do with anything. You start with a set of >> bundles containing all the correct keys. Then you translate each bundle. >> If you accidentally delete a key then you want to know that right away. >> But we should indeed ask the translation team what they want to see. >> > > Sure, go ahead and ask them if they want to look for RuntimeExceptions in > the server log. > > >> >> #4 Doing a check each time you pull a message bundle to compare with the >> base bundle is probably not that expensive, but still pretty daft thing to >> do >> >> You only load each bundle once. So the check only happens the first >> time you request the bundle. >> >> #5 A proper util that's used to translate bundles is much better - we >> can implement a page in the admin console that allows you to validate a >> bundle and print out all missing bundles. This is something that would be >> more developer friendly and also would be usable by non-developers (aka >> people with other language skills than Java) >> >> We should ask the translation team what they want to see and how they do >> their work. I'm sure that they don't expect a tool to be built into the >> product. None of our other products have that. >> > > I don't just care about our translation team, they will translate the > built in keys, but they won't translate the ones introduced by users. > > > >> >> >> >> On 9 October 2015 at 13:24, Stan Silvert <ssilvert(a)redhat.com&gt; wrote: >> >>> On 10/9/2015 6:21 AM, Marko Strukelj wrote: >>> >>> And we can always log the missing key situation into server log - that >>> should be enough for developers to notice it, and fix it. >>> >>> This is basically what happens with the code I wrote for the fix: >>> https://github.com/keycloak/keycloak/pull/1690 >>> >>> You get an error in the console and then a stack trace on the server. >>> The stack trace tells you exactly which key is missing. But the console >>> doesn't crash or anything like that. You just switch back to your original >>> language and everything works fine. >>> >>> >>> On Fri, Oct 9, 2015 at 8:09 AM, Stian Thorgersen <sthorger(a)redhat.com&gt; >>> wrote: >>> >>>> There's two places where keys can be missing: >>>> >>>> * In a translation - this can be an honest mistake, or the translation >>>> wasn't updated when KC was updated >>>> * Custom keys added - for example when keys are used for display names >>>> of clients, roles, etc.. >>>> >>>> Manually having to go through all sorts of pages to look for missing >>>> keys is very error prone and time consuming, so will not be the best option >>>> for developers. In both cases above the correct way to do this would be to >>>> have a way to verify a message bundle. We need a tool that can quickly >>>> identify if there are missing keys and we could expose that through the >>>> admin console. We currently have a student looking at providing a UI for >>>> defining locales and she is also going to look at adding some way of >>>> identifying if a locale is missing keys and also to easily list only >>>> missing keys. >>>> >>>> For end users as I've said they will have no clue what ???key??? is, >>>> and even worse if we throw an exception/error just because a missing key >>>> we'll actually break the whole console just because of a missing key. It's >>>> a much better option to look for the key in another translation and display >>>> that. Chances are they will be able to interpret one or two English words. >>>> Certainly higher chance of that then them being able to interpret ???key???. >>>> >>>> >>>> On 9 October 2015 at 07:51, Thomas Raehalme < >>>> thomas.raehalme(a)aitiofinland.com&gt; wrote: >>>> >>>>> How about returning something noticeable like ???key??? for example? >>>>> On Oct 9, 2015 8:10 AM, "Stian Thorgersen" <sthorger(a)redhat.com&gt; >>>>> wrote: >>>>> >>>>>> That's not putting it to rest at all! Throwing a RuntimeException >>>>>> and rendering the whole admin console useless just because there's a >>>>>> missing key is a horrible idea. >>>>>> >>>>>> On 8 October 2015 at 20:33, Stan Silvert <ssilvert(a)redhat.com&gt; >>>>>> wrote: >>>>>> >>>>>>> What if English is the bundle that has a missing key? >>>>>>> >>>>>>> Let's just put this to rest and solve it once and for all. The >>>>>>> simplest solution I can think of is to just compare keys when a new bundle >>>>>>> is loaded. If any bundle has a missing key or it has key not found in the >>>>>>> previous loaded bundle, we throw a RuntimeException. I can submit a patch >>>>>>> for that in just a few minutes. >>>>>>> >>>>>>> >>>>>>> On 10/8/2015 1:28 PM, Stian Thorgersen wrote: >>>>>>> >>>>>>> I'm not sure I'm buying into the argument that displaying the key >>>>>>> is better for developers. Having English suddenly pop-up in a German >>>>>>> translation is just as obvious as a key. Besides as Stan points out you >>>>>>> catch missing keys by comparing missing keys between English and German. >>>>>>> >>>>>>> However, if there is a mistake in a translation then a user may >>>>>>> quite likely be able to interpret English text, while a user will not be >>>>>>> able to interpret a key. So if a key is missing in a translation (which is >>>>>>> obviously a "bug") it's better to display English than to display the key. >>>>>>> >>>>>>> On 8 October 2015 at 14:13, Stan Silvert <ssilvert(a)redhat.com&gt; >>>>>>> wrote: >>>>>>> >>>>>>>> On 10/8/2015 12:48 AM, Thomas Raehalme wrote: >>>>>>>> >>>>>>>> >>>>>>>> On Oct 8, 2015 6:53 AM, "Stian Thorgersen" <sthorger(a)redhat.com&gt; >>>>>>>> wrote: >>>>>>>> > >>>>>>>> > With regards to internationalization I have two questions: >>>>>>>> > >>>>>>>> > * Should we fallback to English messages if a key is missing in >>>>>>>> a translation? Alternative is to show key, but that's not going to help >>>>>>>> anyone >>>>>>>> >>>>>>>> A missing key is a bug and showing the message in the default >>>>>>>> locale may hide the problem. >>>>>>>> >>>>>>>> Even though showing the key does not help the end user it helps >>>>>>>> the developer and identifies the problem. For this reason I think showing >>>>>>>> the key would be a good idea. >>>>>>>> >>>>>>>> For our bundles, we could catch missing keys at build time. >>>>>>>> >>>>>>>> Failing that, I agree that displaying the key is better than >>>>>>>> falling back to English. This is especially true right now while we >>>>>>>> haven't completed the task of converting everything. If we fall back to >>>>>>>> English we won't know if the problem is a missing key or if the text just >>>>>>>> hasn't been converted yet. >>>>>>>> >>>>>>>> > * Should we change message bundles to UTF-8? Or is ISO 8859-1 >>>>>>>> going to work for all languages? >>>>>>>> >>>>>>>> Depends what those all languages are :-) >>>>>>>> >>>>>>>> I think UTF-8 is the best choice as it will handle practically any >>>>>>>> character. >>>>>>>> >>>>>>>> But if you're referring to Java resource bundles the encoding for >>>>>>>> .properties is ISO-8859-1 but there are means to handle any UTF-8 character. >>>>>>>> >>>>>>>> Yes, an UTF-8 character can be encoded in ISO-8859-1. Java >>>>>>>> provides a native2ascii tool for converting entire files. The resource >>>>>>>> bundle tools in most IDE's do this for you automatically. So you just edit >>>>>>>> as UTF-8 and it saves the bundle as ISO-8859-1. >>>>>>>> >>>>>>>> We can read our bundles as UTF-8 if we want to do that. I'd >>>>>>>> rather not, because I'm not sure what we might run into down the road with >>>>>>>> Java assuming resource bundles are always ISO-8859-1. >>>>>>>> >>>>>>>> But I'd like to get the perspective of people who have handled >>>>>>>> resource bundles in languages that are not fully supported by ISO-8859-1. >>>>>>>> Is it too much of a pain to do a conversion or do the tools make the >>>>>>>> process seamless? >>>>>>>> >>>>>>>> Best regards, >>>>>>>> Thomas >>>>>>>> >>>>>>>> > >>>>>>>> > On 7 October 2015 at 18:42, Stan Silvert <ssilvert(a)redhat.com&gt; >>>>>>>> wrote: >>>>>>>> >> >>>>>>>> >> Marko brought this to my attention yesterday. For some things, >>>>>>>> we >>>>>>>> >> dynamically create UI. In this case, the java code contains >>>>>>>> the English >>>>>>>> >> text and it needs to be localized. Luckily, the solution was >>>>>>>> pretty >>>>>>>> >> straightforward. We just replace the English text with a key >>>>>>>> into the >>>>>>>> >> message bundle. The html template that displays this text >>>>>>>> already pulls >>>>>>>> >> from an Angular scope so we just leave that alone and pass it >>>>>>>> through >>>>>>>> >> the |translate filter. You do need to also add the >>>>>>>> double-colon. >>>>>>>> >> >>>>>>>> >> One nice side effect is that if the key is not found in the >>>>>>>> bundle then >>>>>>>> >> the output of the translate filter is the unchanged text. This >>>>>>>> means >>>>>>>> >> that any code which has not converted to using bundle keys will >>>>>>>> still >>>>>>>> >> work as expected. And, any third-party providers can just >>>>>>>> pass in >>>>>>>> >> plain text if they don't care about l10n. If they ever do care >>>>>>>> about >>>>>>>> >> l10n we will just need to provide a means for them to add >>>>>>>> key/value >>>>>>>> >> pairs to the resource bundles. >>>>>>>> >> >>>>>>>> >> Here is an example for anyone who needs to localize English text >>>>>>>> >> embedded in java: >>>>>>>> >> >>>>>>>> https://github.com/ssilvert/keycloak/commit/c9437595b70810c4472325373dd88... >>>>>>>> >> >>>>>>>> >> Stan >>>>>>>> >> _______________________________________________ >>>>>>>> >> keycloak-dev mailing list >>>>>>>> >> keycloak-dev(a)lists.jboss.org >>>>>>>> >> https://lists.jboss.org/mailman/listinfo/keycloak-dev >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > _______________________________________________ >>>>>>>> > keycloak-dev mailing list >>>>>>>> > keycloak-dev(a)lists.jboss.org >>>>>>>> > https://lists.jboss.org/mailman/listinfo/keycloak-dev >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>> >>>> >>>> _______________________________________________ >>>> keycloak-dev mailing list >>>> keycloak-dev(a)lists.jboss.org >>>> https://lists.jboss.org/mailman/listinfo/keycloak-dev >>>> >>> >>> >>> >>> _______________________________________________ >>> keycloak-dev mailing listkeycloak-dev@lists.jboss.orghttps://lists.jboss.org/mailman/listinfo/keycloak-dev >>> >>> >>> >>> _______________________________________________ >>> keycloak-dev mailing list >>> keycloak-dev(a)lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/keycloak-dev >>> >> >> >> >