Re: [keycloak-dev] Support UTF-8 encoding theme message properites

No, we don't want the theme properties (and other properties files) to be utf-8. They should stick with the default encoding used by Java properties files. Messages are a special case, which is why we accepted it for them. On 14 July 2016 at 13:29, Hiroyuki Wada <wadahiro(a)gmail.com&gt; wrote: > Thank you for merging the pull request. Could I send a pull request > converting to UTF-8 encoding against the theme properties? > > 2016年7月7日(木) 21:39 Hiroyuki Wada <wadahiro(a)gmail.com&gt;: > >> Yes. I will send a pull request later! >> >> 2016年7月7日(木) 21:20 Stian Thorgersen <sthorger(a)redhat.com&gt;: >> >>> Are you doing a PR as well? >>> >>> On 7 July 2016 at 12:33, Hiroyuki Wada <wadahiro(a)gmail.com&gt; wrote: >>> >>>> Ok. I created a JIRA issue. >>>> https://issues.jboss.org/browse/KEYCLOAK-3278 >>>> >>>> 2016年7月6日(水) 19:36 Stian Thorgersen <sthorger(a)redhat.com&gt;: >>>> >>>>> Let's go for it. Please create a JIRA and send PR. Please just use "# >>>>> encoding: utf-8" and not "# -*- coding: utf-8 -*-". The latter is >>>>> just weird if you ask me. >>>>> >>>>> Thomas: It's optional, defaults to iso-8859-1 and the header will >>>>> clearly state that a file uses a different encoding, so I disagree that >>>>> there will be any confusion. >>>>> >>>>> On 6 July 2016 at 12:30, Hiroyuki Wada <wadahiro(a)gmail.com&gt; wrote: >>>>> >>>>>> On Wed, Jul 6, 2016 at 6:53 PM, Thomas Raehalme >>>>>> <thomas.raehalme(a)aitiofinland.com&gt; wrote: >>>>>> > From the javadoc of java.util.ResourceBundle the encoding for >>>>>> .properties is >>>>>> > still expected to be ISO-8859-1: >>>>>> > >>>>>> >> Constructing a PropertyResourceBundle instance from an InputStream >>>>>> >> requires that the input stream be encoded in ISO-8859-1. In that >>>>>> case, >>>>>> >> characters that cannot be represented in ISO-8859-1 encoding must >>>>>> be >>>>>> >> represented by Unicode Escapes as defined in section 3.3 of The >>>>>> Java™ >>>>>> >> Language Specification whereas the other constructor which takes >>>>>> a Reader >>>>>> >> does not have that limitation. >>>>>> >>>>>> It's a case of using constructor which takes an InputStream. >>>>>> It says we can use any encoding when we use the other constructor >>>>>> which takes a Reader. >>>>>> >>>>>> I understand your concern. So what do you think supporting utf-8 with >>>>>> a header like Stian suggested? >>>>>> I think it can avoid confusion because the encoding is noted in the >>>>>> file itself... >>>>>> >>>>>> Regards, >>>>>> >>>>>> > >>>>>> https://docs.oracle.com/javase/8/docs/api/java/util/PropertyResourceBundl... >>>>>> > >>>>>> > I understand that Spring allows one to read .properties using any >>>>>> encoding, >>>>>> > but you need to specify the encoding in Spring configuration >>>>>> external to the >>>>>> > .properties file. >>>>>> > >>>>>> > In my opinion it would cause unnecessary confusion amongst >>>>>> developers to use >>>>>> > any other encoding than the one defined by the official >>>>>> documentation for >>>>>> > Properties. >>>>>> > >>>>>> > Best regards, >>>>>> > Thomas >>>>>> > >>>>>> > >>>>>> > On Wed, Jul 6, 2016 at 12:40 PM, Hiroyuki Wada <wadahiro(a)gmail.com&gt; >>>>>> wrote: >>>>>> >> >>>>>> >> I think it was true before Java 1.6 because Java standard library >>>>>> >> (java.util.ResourceBundle or java.util.Properties) only supported >>>>>> >> ISO-8859-1 encoding file. >>>>>> >> But they support any encoding after Java 1.6. There are some >>>>>> >> frameworks which can read .properties with any encoding. For >>>>>> example, >>>>>> >> Spring Framework can read .properties with any encoding. I think >>>>>> it's >>>>>> >> very useful to read UTF-8 directly for multibyte language country. >>>>>> >> >>>>>> >> On Wed, Jul 6, 2016 at 3:32 PM, Thomas Raehalme >>>>>> >> <thomas.raehalme(a)aitiofinland.com&gt; wrote: >>>>>> >> > Please correct me if I am wrong, but I have been under the >>>>>> impression >>>>>> >> > that >>>>>> >> > Java .properties files should always use encoding ISO-8859-1. >>>>>> Characters >>>>>> >> > not >>>>>> >> > present in ISO-8859-1 can be written in \uxxxx. Won't it make >>>>>> things >>>>>> >> > confusing to developers if another encoding is used here >>>>>> instead? >>>>>> >> > >>>>>> >> > >>>>>> >> > >>>>>> https://docs.oracle.com/javase/8/docs/api/java/util/Properties.html#load-... >>>>>> >> > >>>>>> >> > >>>>>> https://docs.oracle.com/javase/8/docs/api/java/util/Properties.html#store... >>>>>> >> > >>>>>> >> > If alternate encodings are desired how about supporting the XML >>>>>> format >>>>>> >> > of >>>>>> >> > Properties? >>>>>> >> > >>>>>> >> > Best regards, >>>>>> >> > Thomas >>>>>> >> > >>>>>> >> > >>>>>> >> > On Wed, Jul 6, 2016 at 9:04 AM, Stian Thorgersen < >>>>>> sthorger(a)redhat.com&gt; >>>>>> >> > wrote: >>>>>> >> >> >>>>>> >> >> Both iso-8859-1 and utf-8 message bundles should be able to >>>>>> co-exist. >>>>>> >> >> >>>>>> >> >> We can allow specifying the encoding in a comment on the first >>>>>> line >>>>>> >> >> like >>>>>> >> >> this: >>>>>> >> >> >>>>>> >> >> # encoding=utf-8 >>>>>> >> >> key=value >>>>>> >> >> >>>>>> >> >> # encoding=iso-8859-1 >>>>>> >> >> key=value >>>>>> >> >> >>>>>> >> >> If the first line in the file doesn't contain the comment with >>>>>> the >>>>>> >> >> encoding then we should default to iso-8859-1 for backwards >>>>>> >> >> compatibility >>>>>> >> >> >>>>>> >> >> On 5 July 2016 at 09:49, Hiroyuki Wada <wadahiro(a)gmail.com&gt; >>>>>> wrote: >>>>>> >> >>> >>>>>> >> >>> Thanks for your comment. >>>>>> >> >>> >>>>>> >> >>> > If we want to change to utf-8 we'd still need to support >>>>>> iso.. for >>>>>> >> >>> > backwards compatibility. >>>>>> >> >>> >>>>>> >> >>> If we change to UTF-8, we can still read unicode codepoint >>>>>> like >>>>>> >> >>> '\u00e8'. >>>>>> >> >>> There is an incompatibility when non-ascii characters are >>>>>> used in >>>>>> >> >>> message properties. >>>>>> >> >>> The non-ascii characters are 0xA0 - 0xFF codes (please refer >>>>>> codepage >>>>>> >> >>> layout: >>>>>> https://en.wikipedia.org/wiki/ISO/IEC_8859-1#Codepage_layout ) >>>>>> >> >>> >>>>>> >> >>> I think the non-ascii characters might be used in French >>>>>> messages like >>>>>> >> >>> 'à' so I agree to support ISO-8859-1 for backwards >>>>>> compatibility. >>>>>> >> >>> To support this, I think we can add a property like >>>>>> "messageEncoding" >>>>>> >> >>> in keycloak-server.json as below. Is it a good idea? >>>>>> >> >>> >>>>>> >> >>> "theme": { >>>>>> >> >>> "staticMaxAge": 2592000, >>>>>> >> >>> "cacheTemplates": true, >>>>>> >> >>> "cacheThemes": true, >>>>>> >> >>> "messageEncoding": "ISO-8859-1", >>>>>> >> >>> "folder": { >>>>>> >> >>> "dir": "${jboss.home.dir}/themes" >>>>>> >> >>> } >>>>>> >> >>> }, >>>>>> >> >>> >>>>>> >> >>> Regards, >>>>>> >> >>> >>>>>> >> >>> On Mon, Jul 4, 2016 at 10:33 PM, Stian Thorgersen >>>>>> >> >>> <sthorger(a)redhat.com&gt; >>>>>> >> >>> wrote: >>>>>> >> >>> > We have in the past discussed this and decided to stick with >>>>>> >> >>> > ISO-8859-1. >>>>>> >> >>> > That was probably not the best idea though. If we want to >>>>>> change to >>>>>> >> >>> > utf-8 >>>>>> >> >>> > we'd still need to support iso.. for backwards >>>>>> compatibility. >>>>>> >> >>> > >>>>>> >> >>> > On 4 July 2016 at 14:37, Bruno Oliveira < >>>>>> bruno(a)abstractj.org&gt; wrote: >>>>>> >> >>> >> >>>>>> >> >>> >> It makes sense, maybe file a Jira associated with: >>>>>> >> >>> >> https://issues.jboss.org/browse/KEYCLOAK-3259 ? >>>>>> >> >>> >> >>>>>> >> >>> >> On 2016-07-04, Hiroyuki Wada wrote: >>>>>> >> >>> >> > Hello all, >>>>>> >> >>> >> > >>>>>> >> >>> >> > I am trying to translate all base theme messages to my >>>>>> country >>>>>> >> >>> >> > language, Japanese. And I'd like to contribute them. >>>>>> Before that >>>>>> >> >>> >> > work, >>>>>> >> >>> >> > I'd like to propose about the files encoding. >>>>>> >> >>> >> > >>>>>> >> >>> >> > Currently, the message files (*.properties) are loaded >>>>>> with >>>>>> >> >>> >> > ISO-8859-1 >>>>>> >> >>> >> > encoding. Therefore, it is necessary to convert the >>>>>> files by >>>>>> >> >>> >> > 'native2ascii' command beforehand. However we can >>>>>> directly read >>>>>> >> >>> >> > the >>>>>> >> >>> >> > property files with UTF-8 encoding in java 1.6 or later >>>>>> because >>>>>> >> >>> >> > 'java.util.Properties#load(java.io.Reader)' method was >>>>>> introduced >>>>>> >> >>> >> > as >>>>>> >> >>> >> > below. >>>>>> >> >>> >> > >>>>>> >> >>> >> > >>>>>> >> >>> >> > >>>>>> >> >>> >> > >>>>>> >> >>> >> > >>>>>> http://docs.oracle.com/javase/6/docs/api/java/util/Properties.html#load(j... >>>>>> >> >>> >> > >>>>>> >> >>> >> > So, my proposal is supporting the message files with >>>>>> UTF-8 >>>>>> >> >>> >> > encoding. >>>>>> >> >>> >> > I >>>>>> >> >>> >> > believe that it's very developers/customers friendly. In >>>>>> >> >>> >> > addition, >>>>>> >> >>> >> > we >>>>>> >> >>> >> > can easily review the translated messages on the github >>>>>> pull >>>>>> >> >>> >> > request >>>>>> >> >>> >> > view and so on. What do you think? >>>>>> >> >>> >> > >>>>>> >> >>> >> > If it's ok, I'll create a JIRA issue and create a pull >>>>>> request. >>>>>> >> >>> >> > >>>>>> >> >>> >> > Regards, >>>>>> >> >>> >> > >>>>>> >> >>> >> > -- >>>>>> >> >>> >> > Hiroyuki Wada, >>>>>> >> >>> >> > Developer, >>>>>> >> >>> >> > Nomura Research Institute, Ltd. >>>>>> >> >>> >> > _______________________________________________ >>>>>> >> >>> >> > keycloak-dev mailing list >>>>>> >> >>> >> > keycloak-dev(a)lists.jboss.org >>>>>> >> >>> >> > https://lists.jboss.org/mailman/listinfo/keycloak-dev >>>>>> >> >>> >> >>>>>> >> >>> >> -- >>>>>> >> >>> >> >>>>>> >> >>> >> abstractj >>>>>> >> >>> >> PGP: 0x84DC9914 >>>>>> >> >>> >> _______________________________________________ >>>>>> >> >>> >> keycloak-dev mailing list >>>>>> >> >>> >> keycloak-dev(a)lists.jboss.org >>>>>> >> >>> >> https://lists.jboss.org/mailman/listinfo/keycloak-dev >>>>>> >> >>> > >>>>>> >> >>> > >>>>>> >> >> >>>>>> >> >> >>>>>> >> >> >>>>>> >> >> _______________________________________________ >>>>>> >> >> keycloak-dev mailing list >>>>>> >> >> keycloak-dev(a)lists.jboss.org >>>>>> >> >> https://lists.jboss.org/mailman/listinfo/keycloak-dev >>>>>> >> > >>>>>> >> > >>>>>> >> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> >>>>>> _______________________________________________ >>>>>> keycloak-dev mailing list >>>>>> keycloak-dev(a)lists.jboss.org >>>>>> https://lists.jboss.org/mailman/listinfo/keycloak-dev >>>>>> >>>>> >>>>> >>>