Re: [keycloak-dev] Support UTF-8 encoding theme message properites

BTW do you have a reliable way to convert the files? I've tried various > tools, but none worked for me. > Yes, we can use native2ascii command. Example: native2ascii -encoding UTF-8 -reverse <input> <output> > On 15 July 2016 at 08:14, Stian Thorgersen <sthorger(a)redhat.com&gt; wrote: > >> I'd like the current messages to be converted by the people that have >> contributed them. They will be able to determine what encoding they prefer >> to use and also confirm that the conversion has been successful. >> >> On 15 July 2016 at 07:59, Hiroyuki Wada <wadahiro(a)gmail.com&gt; wrote: >> >>> I mean I'd like to change the message files encoding only. >>> For example, admin-messages for Russian language is currently written >>> in ISO-8859-1 encoding. >>> >>> >>> https://github.com/keycloak/keycloak/blob/master/themes/src/main/resource... >>> >>> Can I convert the language messages except for English? >>> >>> >>> 2016年7月15日(金) 14:03 Stian Thorgersen <sthorger(a)redhat.com&gt;: >>> >>>> No, we don't want the theme properties (and other properties files) to >>>> be utf-8. They should stick with the default encoding used by Java >>>> properties files. >>>> >>>> Messages are a special case, which is why we accepted it for them. >>>> >>>> On 14 July 2016 at 13:29, Hiroyuki Wada <wadahiro(a)gmail.com&gt; wrote: >>>> >>>>> Thank you for merging the pull request. Could I send a pull request >>>>> converting to UTF-8 encoding against the theme properties? >>>>> >>>>> 2016年7月7日(木) 21:39 Hiroyuki Wada <wadahiro(a)gmail.com&gt;: >>>>> >>>>>> Yes. I will send a pull request later! >>>>>> >>>>>> 2016年7月7日(木) 21:20 Stian Thorgersen <sthorger(a)redhat.com&gt;: >>>>>> >>>>>>> Are you doing a PR as well? >>>>>>> >>>>>>> On 7 July 2016 at 12:33, Hiroyuki Wada <wadahiro(a)gmail.com&gt; wrote: >>>>>>> >>>>>>>> Ok. I created a JIRA issue. >>>>>>>> https://issues.jboss.org/browse/KEYCLOAK-3278 >>>>>>>> >>>>>>>> 2016年7月6日(水) 19:36 Stian Thorgersen <sthorger(a)redhat.com&gt;: >>>>>>>> >>>>>>>>> Let's go for it. Please create a JIRA and send PR. Please just >>>>>>>>> use "# encoding: utf-8" and not "# -*- coding: utf-8 -*-". The >>>>>>>>> latter is just weird if you ask me. >>>>>>>>> >>>>>>>>> Thomas: It's optional, defaults to iso-8859-1 and the header will >>>>>>>>> clearly state that a file uses a different encoding, so I disagree that >>>>>>>>> there will be any confusion. >>>>>>>>> >>>>>>>>> On 6 July 2016 at 12:30, Hiroyuki Wada <wadahiro(a)gmail.com&gt; >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> On Wed, Jul 6, 2016 at 6:53 PM, Thomas Raehalme >>>>>>>>>> <thomas.raehalme(a)aitiofinland.com&gt; wrote: >>>>>>>>>> > From the javadoc of java.util.ResourceBundle the encoding for >>>>>>>>>> .properties is >>>>>>>>>> > still expected to be ISO-8859-1: >>>>>>>>>> > >>>>>>>>>> >> Constructing a PropertyResourceBundle instance from an >>>>>>>>>> InputStream >>>>>>>>>> >> requires that the input stream be encoded in ISO-8859-1. In >>>>>>>>>> that case, >>>>>>>>>> >> characters that cannot be represented in ISO-8859-1 encoding >>>>>>>>>> must be >>>>>>>>>> >> represented by Unicode Escapes as defined in section 3.3 of >>>>>>>>>> The Java™ >>>>>>>>>> >> Language Specification whereas the other constructor which >>>>>>>>>> takes a Reader >>>>>>>>>> >> does not have that limitation. >>>>>>>>>> >>>>>>>>>> It's a case of using constructor which takes an InputStream. >>>>>>>>>> It says we can use any encoding when we use the other constructor >>>>>>>>>> which takes a Reader. >>>>>>>>>> >>>>>>>>>> I understand your concern. So what do you think supporting utf-8 >>>>>>>>>> with >>>>>>>>>> a header like Stian suggested? >>>>>>>>>> I think it can avoid confusion because the encoding is noted in >>>>>>>>>> the >>>>>>>>>> file itself... >>>>>>>>>> >>>>>>>>>> Regards, >>>>>>>>>> >>>>>>>>>> > >>>>>>>>>> https://docs.oracle.com/javase/8/docs/api/java/util/PropertyResourceBundl... >>>>>>>>>> > >>>>>>>>>> > I understand that Spring allows one to read .properties using >>>>>>>>>> any encoding, >>>>>>>>>> > but you need to specify the encoding in Spring configuration >>>>>>>>>> external to the >>>>>>>>>> > .properties file. >>>>>>>>>> > >>>>>>>>>> > In my opinion it would cause unnecessary confusion amongst >>>>>>>>>> developers to use >>>>>>>>>> > any other encoding than the one defined by the official >>>>>>>>>> documentation for >>>>>>>>>> > Properties. >>>>>>>>>> > >>>>>>>>>> > Best regards, >>>>>>>>>> > Thomas >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > On Wed, Jul 6, 2016 at 12:40 PM, Hiroyuki Wada < >>>>>>>>>> wadahiro(a)gmail.com&gt; wrote: >>>>>>>>>> >> >>>>>>>>>> >> I think it was true before Java 1.6 because Java standard >>>>>>>>>> library >>>>>>>>>> >> (java.util.ResourceBundle or java.util.Properties) only >>>>>>>>>> supported >>>>>>>>>> >> ISO-8859-1 encoding file. >>>>>>>>>> >> But they support any encoding after Java 1.6. There are some >>>>>>>>>> >> frameworks which can read .properties with any encoding. For >>>>>>>>>> example, >>>>>>>>>> >> Spring Framework can read .properties with any encoding. I >>>>>>>>>> think it's >>>>>>>>>> >> very useful to read UTF-8 directly for multibyte language >>>>>>>>>> country. >>>>>>>>>> >> >>>>>>>>>> >> On Wed, Jul 6, 2016 at 3:32 PM, Thomas Raehalme >>>>>>>>>> >> <thomas.raehalme(a)aitiofinland.com&gt; wrote: >>>>>>>>>> >> > Please correct me if I am wrong, but I have been under the >>>>>>>>>> impression >>>>>>>>>> >> > that >>>>>>>>>> >> > Java .properties files should always use encoding >>>>>>>>>> ISO-8859-1. Characters >>>>>>>>>> >> > not >>>>>>>>>> >> > present in ISO-8859-1 can be written in \uxxxx. Won't it >>>>>>>>>> make things >>>>>>>>>> >> > confusing to developers if another encoding is used here >>>>>>>>>> instead? >>>>>>>>>> >> > >>>>>>>>>> >> > >>>>>>>>>> >> > >>>>>>>>>> https://docs.oracle.com/javase/8/docs/api/java/util/Properties.html#load-... >>>>>>>>>> >> > >>>>>>>>>> >> > >>>>>>>>>> https://docs.oracle.com/javase/8/docs/api/java/util/Properties.html#store... >>>>>>>>>> >> > >>>>>>>>>> >> > If alternate encodings are desired how about supporting the >>>>>>>>>> XML format >>>>>>>>>> >> > of >>>>>>>>>> >> > Properties? >>>>>>>>>> >> > >>>>>>>>>> >> > Best regards, >>>>>>>>>> >> > Thomas >>>>>>>>>> >> > >>>>>>>>>> >> > >>>>>>>>>> >> > On Wed, Jul 6, 2016 at 9:04 AM, Stian Thorgersen < >>>>>>>>>> sthorger(a)redhat.com&gt; >>>>>>>>>> >> > wrote: >>>>>>>>>> >> >> >>>>>>>>>> >> >> Both iso-8859-1 and utf-8 message bundles should be able >>>>>>>>>> to co-exist. >>>>>>>>>> >> >> >>>>>>>>>> >> >> We can allow specifying the encoding in a comment on the >>>>>>>>>> first line >>>>>>>>>> >> >> like >>>>>>>>>> >> >> this: >>>>>>>>>> >> >> >>>>>>>>>> >> >> # encoding=utf-8 >>>>>>>>>> >> >> key=value >>>>>>>>>> >> >> >>>>>>>>>> >> >> # encoding=iso-8859-1 >>>>>>>>>> >> >> key=value >>>>>>>>>> >> >> >>>>>>>>>> >> >> If the first line in the file doesn't contain the comment >>>>>>>>>> with the >>>>>>>>>> >> >> encoding then we should default to iso-8859-1 for backwards >>>>>>>>>> >> >> compatibility >>>>>>>>>> >> >> >>>>>>>>>> >> >> On 5 July 2016 at 09:49, Hiroyuki Wada <wadahiro(a)gmail.com&gt; >>>>>>>>>> wrote: >>>>>>>>>> >> >>> >>>>>>>>>> >> >>> Thanks for your comment. >>>>>>>>>> >> >>> >>>>>>>>>> >> >>> > If we want to change to utf-8 we'd still need to >>>>>>>>>> support iso.. for >>>>>>>>>> >> >>> > backwards compatibility. >>>>>>>>>> >> >>> >>>>>>>>>> >> >>> If we change to UTF-8, we can still read unicode >>>>>>>>>> codepoint like >>>>>>>>>> >> >>> '\u00e8'. >>>>>>>>>> >> >>> There is an incompatibility when non-ascii characters are >>>>>>>>>> used in >>>>>>>>>> >> >>> message properties. >>>>>>>>>> >> >>> The non-ascii characters are 0xA0 - 0xFF codes (please >>>>>>>>>> refer codepage >>>>>>>>>> >> >>> layout: >>>>>>>>>> https://en.wikipedia.org/wiki/ISO/IEC_8859-1#Codepage_layout ) >>>>>>>>>> >> >>> >>>>>>>>>> >> >>> I think the non-ascii characters might be used in French >>>>>>>>>> messages like >>>>>>>>>> >> >>> 'à' so I agree to support ISO-8859-1 for backwards >>>>>>>>>> compatibility. >>>>>>>>>> >> >>> To support this, I think we can add a property like >>>>>>>>>> "messageEncoding" >>>>>>>>>> >> >>> in keycloak-server.json as below. Is it a good idea? >>>>>>>>>> >> >>> >>>>>>>>>> >> >>> "theme": { >>>>>>>>>> >> >>> "staticMaxAge": 2592000, >>>>>>>>>> >> >>> "cacheTemplates": true, >>>>>>>>>> >> >>> "cacheThemes": true, >>>>>>>>>> >> >>> "messageEncoding": "ISO-8859-1", >>>>>>>>>> >> >>> "folder": { >>>>>>>>>> >> >>> "dir": "${jboss.home.dir}/themes" >>>>>>>>>> >> >>> } >>>>>>>>>> >> >>> }, >>>>>>>>>> >> >>> >>>>>>>>>> >> >>> Regards, >>>>>>>>>> >> >>> >>>>>>>>>> >> >>> On Mon, Jul 4, 2016 at 10:33 PM, Stian Thorgersen >>>>>>>>>> >> >>> <sthorger(a)redhat.com&gt; >>>>>>>>>> >> >>> wrote: >>>>>>>>>> >> >>> > We have in the past discussed this and decided to stick >>>>>>>>>> with >>>>>>>>>> >> >>> > ISO-8859-1. >>>>>>>>>> >> >>> > That was probably not the best idea though. If we want >>>>>>>>>> to change to >>>>>>>>>> >> >>> > utf-8 >>>>>>>>>> >> >>> > we'd still need to support iso.. for backwards >>>>>>>>>> compatibility. >>>>>>>>>> >> >>> > >>>>>>>>>> >> >>> > On 4 July 2016 at 14:37, Bruno Oliveira < >>>>>>>>>> bruno(a)abstractj.org&gt; wrote: >>>>>>>>>> >> >>> >> >>>>>>>>>> >> >>> >> It makes sense, maybe file a Jira associated with: >>>>>>>>>> >> >>> >> https://issues.jboss.org/browse/KEYCLOAK-3259 ? >>>>>>>>>> >> >>> >> >>>>>>>>>> >> >>> >> On 2016-07-04, Hiroyuki Wada wrote: >>>>>>>>>> >> >>> >> > Hello all, >>>>>>>>>> >> >>> >> > >>>>>>>>>> >> >>> >> > I am trying to translate all base theme messages to >>>>>>>>>> my country >>>>>>>>>> >> >>> >> > language, Japanese. And I'd like to contribute them. >>>>>>>>>> Before that >>>>>>>>>> >> >>> >> > work, >>>>>>>>>> >> >>> >> > I'd like to propose about the files encoding. >>>>>>>>>> >> >>> >> > >>>>>>>>>> >> >>> >> > Currently, the message files (*.properties) are >>>>>>>>>> loaded with >>>>>>>>>> >> >>> >> > ISO-8859-1 >>>>>>>>>> >> >>> >> > encoding. Therefore, it is necessary to convert the >>>>>>>>>> files by >>>>>>>>>> >> >>> >> > 'native2ascii' command beforehand. However we can >>>>>>>>>> directly read >>>>>>>>>> >> >>> >> > the >>>>>>>>>> >> >>> >> > property files with UTF-8 encoding in java 1.6 or >>>>>>>>>> later because >>>>>>>>>> >> >>> >> > 'java.util.Properties#load(java.io.Reader)' method >>>>>>>>>> was introduced >>>>>>>>>> >> >>> >> > as >>>>>>>>>> >> >>> >> > below. >>>>>>>>>> >> >>> >> > >>>>>>>>>> >> >>> >> > >>>>>>>>>> >> >>> >> > >>>>>>>>>> >> >>> >> > >>>>>>>>>> >> >>> >> > >>>>>>>>>> http://docs.oracle.com/javase/6/docs/api/java/util/Properties.html#load(j... >>>>>>>>>> >> >>> >> > >>>>>>>>>> >> >>> >> > So, my proposal is supporting the message files with >>>>>>>>>> UTF-8 >>>>>>>>>> >> >>> >> > encoding. >>>>>>>>>> >> >>> >> > I >>>>>>>>>> >> >>> >> > believe that it's very developers/customers >>>>>>>>>> friendly. In >>>>>>>>>> >> >>> >> > addition, >>>>>>>>>> >> >>> >> > we >>>>>>>>>> >> >>> >> > can easily review the translated messages on the >>>>>>>>>> github pull >>>>>>>>>> >> >>> >> > request >>>>>>>>>> >> >>> >> > view and so on. What do you think? >>>>>>>>>> >> >>> >> > >>>>>>>>>> >> >>> >> > If it's ok, I'll create a JIRA issue and create a >>>>>>>>>> pull request. >>>>>>>>>> >> >>> >> > >>>>>>>>>> >> >>> >> > Regards, >>>>>>>>>> >> >>> >> > >>>>>>>>>> >> >>> >> > -- >>>>>>>>>> >> >>> >> > Hiroyuki Wada, >>>>>>>>>> >> >>> >> > Developer, >>>>>>>>>> >> >>> >> > Nomura Research Institute, Ltd. >>>>>>>>>> >> >>> >> > _______________________________________________ >>>>>>>>>> >> >>> >> > keycloak-dev mailing list >>>>>>>>>> >> >>> >> > keycloak-dev(a)lists.jboss.org >>>>>>>>>> >> >>> >> > >>>>>>>>>> https://lists.jboss.org/mailman/listinfo/keycloak-dev >>>>>>>>>> >> >>> >> >>>>>>>>>> >> >>> >> -- >>>>>>>>>> >> >>> >> >>>>>>>>>> >> >>> >> abstractj >>>>>>>>>> >> >>> >> PGP: 0x84DC9914 >>>>>>>>>> >> >>> >> _______________________________________________ >>>>>>>>>> >> >>> >> keycloak-dev mailing list >>>>>>>>>> >> >>> >> keycloak-dev(a)lists.jboss.org >>>>>>>>>> >> >>> >> https://lists.jboss.org/mailman/listinfo/keycloak-dev >>>>>>>>>> >> >>> > >>>>>>>>>> >> >>> > >>>>>>>>>> >> >> >>>>>>>>>> >> >> >>>>>>>>>> >> >> >>>>>>>>>> >> >> _______________________________________________ >>>>>>>>>> >> >> keycloak-dev mailing list >>>>>>>>>> >> >> keycloak-dev(a)lists.jboss.org >>>>>>>>>> >> >> https://lists.jboss.org/mailman/listinfo/keycloak-dev >>>>>>>>>> >> > >>>>>>>>>> >> > >>>>>>>>>> >> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> keycloak-dev mailing list >>>>>>>>>> keycloak-dev(a)lists.jboss.org >>>>>>>>>> https://lists.jboss.org/mailman/listinfo/keycloak-dev >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>> >>>> >> > _______________________________________________ keycloak-dev mailing list keycloak-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/keycloak-dev