Re: [keycloak-dev] Support UTF-8 encoding theme message properites

BTW do you have a reliable way to convert the files? I've tried various tools, but none worked for me.

On 15 July 2016 at 08:14, Stian Thorgersen <sthorger(a)redhat.com&gt; wrote: > I'd like the current messages to be converted by the people that have > contributed them. They will be able to determine what encoding they prefer > to use and also confirm that the conversion has been successful. > > On 15 July 2016 at 07:59, Hiroyuki Wada <wadahiro(a)gmail.com&gt; wrote: > >> I mean I'd like to change the message files encoding only. >> For example, admin-messages for Russian language is currently written in >> ISO-8859-1 encoding. >> >> >> https://github.com/keycloak/keycloak/blob/master/themes/src/main/resource... >> >> Can I convert the language messages except for English? >> >> >> 2016年7月15日(金) 14:03 Stian Thorgersen <sthorger(a)redhat.com&gt;: >> >>> No, we don't want the theme properties (and other properties files) to >>> be utf-8. They should stick with the default encoding used by Java >>> properties files. >>> >>> Messages are a special case, which is why we accepted it for them. >>> >>> On 14 July 2016 at 13:29, Hiroyuki Wada <wadahiro(a)gmail.com&gt; wrote: >>> >>>> Thank you for merging the pull request. Could I send a pull request >>>> converting to UTF-8 encoding against the theme properties? >>>> >>>> 2016年7月7日(木) 21:39 Hiroyuki Wada <wadahiro(a)gmail.com&gt;: >>>> >>>>> Yes. I will send a pull request later! >>>>> >>>>> 2016年7月7日(木) 21:20 Stian Thorgersen <sthorger(a)redhat.com&gt;: >>>>> >>>>>> Are you doing a PR as well? >>>>>> >>>>>> On 7 July 2016 at 12:33, Hiroyuki Wada <wadahiro(a)gmail.com&gt; wrote: >>>>>> >>>>>>> Ok. I created a JIRA issue. >>>>>>> https://issues.jboss.org/browse/KEYCLOAK-3278 >>>>>>> >>>>>>> 2016年7月6日(水) 19:36 Stian Thorgersen <sthorger(a)redhat.com&gt;: >>>>>>> >>>>>>>> Let's go for it. Please create a JIRA and send PR. Please just use >>>>>>>> "# encoding: utf-8" and not "# -*- coding: utf-8 -*-". The latter >>>>>>>> is just weird if you ask me. >>>>>>>> >>>>>>>> Thomas: It's optional, defaults to iso-8859-1 and the header will >>>>>>>> clearly state that a file uses a different encoding, so I disagree that >>>>>>>> there will be any confusion. >>>>>>>> >>>>>>>> On 6 July 2016 at 12:30, Hiroyuki Wada <wadahiro(a)gmail.com&gt; wrote: >>>>>>>> >>>>>>>>> On Wed, Jul 6, 2016 at 6:53 PM, Thomas Raehalme >>>>>>>>> <thomas.raehalme(a)aitiofinland.com&gt; wrote: >>>>>>>>> > From the javadoc of java.util.ResourceBundle the encoding for >>>>>>>>> .properties is >>>>>>>>> > still expected to be ISO-8859-1: >>>>>>>>> > >>>>>>>>> >> Constructing a PropertyResourceBundle instance from an >>>>>>>>> InputStream >>>>>>>>> >> requires that the input stream be encoded in ISO-8859-1. In >>>>>>>>> that case, >>>>>>>>> >> characters that cannot be represented in ISO-8859-1 encoding >>>>>>>>> must be >>>>>>>>> >> represented by Unicode Escapes as defined in section 3.3 of >>>>>>>>> The Java™ >>>>>>>>> >> Language Specification whereas the other constructor which >>>>>>>>> takes a Reader >>>>>>>>> >> does not have that limitation. >>>>>>>>> >>>>>>>>> It's a case of using constructor which takes an InputStream. >>>>>>>>> It says we can use any encoding when we use the other constructor >>>>>>>>> which takes a Reader. >>>>>>>>> >>>>>>>>> I understand your concern. So what do you think supporting utf-8 >>>>>>>>> with >>>>>>>>> a header like Stian suggested? >>>>>>>>> I think it can avoid confusion because the encoding is noted in >>>>>>>>> the >>>>>>>>> file itself... >>>>>>>>> >>>>>>>>> Regards, >>>>>>>>> >>>>>>>>> > >>>>>>>>> https://docs.oracle.com/javase/8/docs/api/java/util/PropertyResourceBundl... >>>>>>>>> > >>>>>>>>> > I understand that Spring allows one to read .properties using >>>>>>>>> any encoding, >>>>>>>>> > but you need to specify the encoding in Spring configuration >>>>>>>>> external to the >>>>>>>>> > .properties file. >>>>>>>>> > >>>>>>>>> > In my opinion it would cause unnecessary confusion amongst >>>>>>>>> developers to use >>>>>>>>> > any other encoding than the one defined by the official >>>>>>>>> documentation for >>>>>>>>> > Properties. >>>>>>>>> > >>>>>>>>> > Best regards, >>>>>>>>> > Thomas >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > On Wed, Jul 6, 2016 at 12:40 PM, Hiroyuki Wada < >>>>>>>>> wadahiro(a)gmail.com&gt; wrote: >>>>>>>>> >> >>>>>>>>> >> I think it was true before Java 1.6 because Java standard >>>>>>>>> library >>>>>>>>> >> (java.util.ResourceBundle or java.util.Properties) only >>>>>>>>> supported >>>>>>>>> >> ISO-8859-1 encoding file. >>>>>>>>> >> But they support any encoding after Java 1.6. There are some >>>>>>>>> >> frameworks which can read .properties with any encoding. For >>>>>>>>> example, >>>>>>>>> >> Spring Framework can read .properties with any encoding. I >>>>>>>>> think it's >>>>>>>>> >> very useful to read UTF-8 directly for multibyte language >>>>>>>>> country. >>>>>>>>> >> >>>>>>>>> >> On Wed, Jul 6, 2016 at 3:32 PM, Thomas Raehalme >>>>>>>>> >> <thomas.raehalme(a)aitiofinland.com&gt; wrote: >>>>>>>>> >> > Please correct me if I am wrong, but I have been under the >>>>>>>>> impression >>>>>>>>> >> > that >>>>>>>>> >> > Java .properties files should always use encoding >>>>>>>>> ISO-8859-1. Characters >>>>>>>>> >> > not >>>>>>>>> >> > present in ISO-8859-1 can be written in \uxxxx. Won't it >>>>>>>>> make things >>>>>>>>> >> > confusing to developers if another encoding is used here >>>>>>>>> instead? >>>>>>>>> >> > >>>>>>>>> >> > >>>>>>>>> >> > >>>>>>>>> https://docs.oracle.com/javase/8/docs/api/java/util/Properties.html#load-... >>>>>>>>> >> > >>>>>>>>> >> > >>>>>>>>> https://docs.oracle.com/javase/8/docs/api/java/util/Properties.html#store... >>>>>>>>> >> > >>>>>>>>> >> > If alternate encodings are desired how about supporting the >>>>>>>>> XML format >>>>>>>>> >> > of >>>>>>>>> >> > Properties? >>>>>>>>> >> > >>>>>>>>> >> > Best regards, >>>>>>>>> >> > Thomas >>>>>>>>> >> > >>>>>>>>> >> > >>>>>>>>> >> > On Wed, Jul 6, 2016 at 9:04 AM, Stian Thorgersen < >>>>>>>>> sthorger(a)redhat.com&gt; >>>>>>>>> >> > wrote: >>>>>>>>> >> >> >>>>>>>>> >> >> Both iso-8859-1 and utf-8 message bundles should be able to >>>>>>>>> co-exist. >>>>>>>>> >> >> >>>>>>>>> >> >> We can allow specifying the encoding in a comment on the >>>>>>>>> first line >>>>>>>>> >> >> like >>>>>>>>> >> >> this: >>>>>>>>> >> >> >>>>>>>>> >> >> # encoding=utf-8 >>>>>>>>> >> >> key=value >>>>>>>>> >> >> >>>>>>>>> >> >> # encoding=iso-8859-1 >>>>>>>>> >> >> key=value >>>>>>>>> >> >> >>>>>>>>> >> >> If the first line in the file doesn't contain the comment >>>>>>>>> with the >>>>>>>>> >> >> encoding then we should default to iso-8859-1 for backwards >>>>>>>>> >> >> compatibility >>>>>>>>> >> >> >>>>>>>>> >> >> On 5 July 2016 at 09:49, Hiroyuki Wada <wadahiro(a)gmail.com&gt; >>>>>>>>> wrote: >>>>>>>>> >> >>> >>>>>>>>> >> >>> Thanks for your comment. >>>>>>>>> >> >>> >>>>>>>>> >> >>> > If we want to change to utf-8 we'd still need to support >>>>>>>>> iso.. for >>>>>>>>> >> >>> > backwards compatibility. >>>>>>>>> >> >>> >>>>>>>>> >> >>> If we change to UTF-8, we can still read unicode codepoint >>>>>>>>> like >>>>>>>>> >> >>> '\u00e8'. >>>>>>>>> >> >>> There is an incompatibility when non-ascii characters are >>>>>>>>> used in >>>>>>>>> >> >>> message properties. >>>>>>>>> >> >>> The non-ascii characters are 0xA0 - 0xFF codes (please >>>>>>>>> refer codepage >>>>>>>>> >> >>> layout: >>>>>>>>> https://en.wikipedia.org/wiki/ISO/IEC_8859-1#Codepage_layout ) >>>>>>>>> >> >>> >>>>>>>>> >> >>> I think the non-ascii characters might be used in French >>>>>>>>> messages like >>>>>>>>> >> >>> 'à' so I agree to support ISO-8859-1 for backwards >>>>>>>>> compatibility. >>>>>>>>> >> >>> To support this, I think we can add a property like >>>>>>>>> "messageEncoding" >>>>>>>>> >> >>> in keycloak-server.json as below. Is it a good idea? >>>>>>>>> >> >>> >>>>>>>>> >> >>> "theme": { >>>>>>>>> >> >>> "staticMaxAge": 2592000, >>>>>>>>> >> >>> "cacheTemplates": true, >>>>>>>>> >> >>> "cacheThemes": true, >>>>>>>>> >> >>> "messageEncoding": "ISO-8859-1", >>>>>>>>> >> >>> "folder": { >>>>>>>>> >> >>> "dir": "${jboss.home.dir}/themes" >>>>>>>>> >> >>> } >>>>>>>>> >> >>> }, >>>>>>>>> >> >>> >>>>>>>>> >> >>> Regards, >>>>>>>>> >> >>> >>>>>>>>> >> >>> On Mon, Jul 4, 2016 at 10:33 PM, Stian Thorgersen >>>>>>>>> >> >>> <sthorger(a)redhat.com&gt; >>>>>>>>> >> >>> wrote: >>>>>>>>> >> >>> > We have in the past discussed this and decided to stick >>>>>>>>> with >>>>>>>>> >> >>> > ISO-8859-1. >>>>>>>>> >> >>> > That was probably not the best idea though. If we want >>>>>>>>> to change to >>>>>>>>> >> >>> > utf-8 >>>>>>>>> >> >>> > we'd still need to support iso.. for backwards >>>>>>>>> compatibility. >>>>>>>>> >> >>> > >>>>>>>>> >> >>> > On 4 July 2016 at 14:37, Bruno Oliveira < >>>>>>>>> bruno(a)abstractj.org&gt; wrote: >>>>>>>>> >> >>> >> >>>>>>>>> >> >>> >> It makes sense, maybe file a Jira associated with: >>>>>>>>> >> >>> >> https://issues.jboss.org/browse/KEYCLOAK-3259 ? >>>>>>>>> >> >>> >> >>>>>>>>> >> >>> >> On 2016-07-04, Hiroyuki Wada wrote: >>>>>>>>> >> >>> >> > Hello all, >>>>>>>>> >> >>> >> > >>>>>>>>> >> >>> >> > I am trying to translate all base theme messages to >>>>>>>>> my country >>>>>>>>> >> >>> >> > language, Japanese. And I'd like to contribute them. >>>>>>>>> Before that >>>>>>>>> >> >>> >> > work, >>>>>>>>> >> >>> >> > I'd like to propose about the files encoding. >>>>>>>>> >> >>> >> > >>>>>>>>> >> >>> >> > Currently, the message files (*.properties) are >>>>>>>>> loaded with >>>>>>>>> >> >>> >> > ISO-8859-1 >>>>>>>>> >> >>> >> > encoding. Therefore, it is necessary to convert the >>>>>>>>> files by >>>>>>>>> >> >>> >> > 'native2ascii' command beforehand. However we can >>>>>>>>> directly read >>>>>>>>> >> >>> >> > the >>>>>>>>> >> >>> >> > property files with UTF-8 encoding in java 1.6 or >>>>>>>>> later because >>>>>>>>> >> >>> >> > 'java.util.Properties#load(java.io.Reader)' method >>>>>>>>> was introduced >>>>>>>>> >> >>> >> > as >>>>>>>>> >> >>> >> > below. >>>>>>>>> >> >>> >> > >>>>>>>>> >> >>> >> > >>>>>>>>> >> >>> >> > >>>>>>>>> >> >>> >> > >>>>>>>>> >> >>> >> > >>>>>>>>> http://docs.oracle.com/javase/6/docs/api/java/util/Properties.html#load(j... >>>>>>>>> >> >>> >> > >>>>>>>>> >> >>> >> > So, my proposal is supporting the message files with >>>>>>>>> UTF-8 >>>>>>>>> >> >>> >> > encoding. >>>>>>>>> >> >>> >> > I >>>>>>>>> >> >>> >> > believe that it's very developers/customers friendly. >>>>>>>>> In >>>>>>>>> >> >>> >> > addition, >>>>>>>>> >> >>> >> > we >>>>>>>>> >> >>> >> > can easily review the translated messages on the >>>>>>>>> github pull >>>>>>>>> >> >>> >> > request >>>>>>>>> >> >>> >> > view and so on. What do you think? >>>>>>>>> >> >>> >> > >>>>>>>>> >> >>> >> > If it's ok, I'll create a JIRA issue and create a >>>>>>>>> pull request. >>>>>>>>> >> >>> >> > >>>>>>>>> >> >>> >> > Regards, >>>>>>>>> >> >>> >> > >>>>>>>>> >> >>> >> > -- >>>>>>>>> >> >>> >> > Hiroyuki Wada, >>>>>>>>> >> >>> >> > Developer, >>>>>>>>> >> >>> >> > Nomura Research Institute, Ltd. >>>>>>>>> >> >>> >> > _______________________________________________ >>>>>>>>> >> >>> >> > keycloak-dev mailing list >>>>>>>>> >> >>> >> > keycloak-dev(a)lists.jboss.org >>>>>>>>> >> >>> >> > https://lists.jboss.org/mailman/listinfo/keycloak-dev >>>>>>>>> >> >>> >> >>>>>>>>> >> >>> >> -- >>>>>>>>> >> >>> >> >>>>>>>>> >> >>> >> abstractj >>>>>>>>> >> >>> >> PGP: 0x84DC9914 >>>>>>>>> >> >>> >> _______________________________________________ >>>>>>>>> >> >>> >> keycloak-dev mailing list >>>>>>>>> >> >>> >> keycloak-dev(a)lists.jboss.org >>>>>>>>> >> >>> >> https://lists.jboss.org/mailman/listinfo/keycloak-dev >>>>>>>>> >> >>> > >>>>>>>>> >> >>> > >>>>>>>>> >> >> >>>>>>>>> >> >> >>>>>>>>> >> >> >>>>>>>>> >> >> _______________________________________________ >>>>>>>>> >> >> keycloak-dev mailing list >>>>>>>>> >> >> keycloak-dev(a)lists.jboss.org >>>>>>>>> >> >> https://lists.jboss.org/mailman/listinfo/keycloak-dev >>>>>>>>> >> > >>>>>>>>> >> > >>>>>>>>> >> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> keycloak-dev mailing list >>>>>>>>> keycloak-dev(a)lists.jboss.org >>>>>>>>> https://lists.jboss.org/mailman/listinfo/keycloak-dev >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>> >>> >