Re: [keycloak-dev] Support UTF-8 encoding theme message properites

iconv has been a good tool to convert files from one encoding to another. Best regards, Thomas On Jul 15, 2016 09:57, "Hiroyuki Wada" <wadahiro(a)gmail.com&gt; wrote: > > BTW do you have a reliable way to convert the files? I've tried various >> tools, but none worked for me. >> > > Yes, we can use native2ascii command. > > Example: > native2ascii -encoding UTF-8 -reverse <input> <output> > > > > >> On 15 July 2016 at 08:14, Stian Thorgersen <sthorger(a)redhat.com&gt; wrote: >> >>> I'd like the current messages to be converted by the people that have >>> contributed them. They will be able to determine what encoding they prefer >>> to use and also confirm that the conversion has been successful. >>> >>> On 15 July 2016 at 07:59, Hiroyuki Wada <wadahiro(a)gmail.com&gt; wrote: >>> >>>> I mean I'd like to change the message files encoding only. >>>> For example, admin-messages for Russian language is currently written >>>> in ISO-8859-1 encoding. >>>> >>>> >>>> https://github.com/keycloak/keycloak/blob/master/themes/src/main/resource... >>>> >>>> Can I convert the language messages except for English? >>>> >>>> >>>> 2016年7月15日(金) 14:03 Stian Thorgersen <sthorger(a)redhat.com&gt;: >>>> >>>>> No, we don't want the theme properties (and other properties files) >>>>> to be utf-8. They should stick with the default encoding used by Java >>>>> properties files. >>>>> >>>>> Messages are a special case, which is why we accepted it for them. >>>>> >>>>> On 14 July 2016 at 13:29, Hiroyuki Wada <wadahiro(a)gmail.com&gt; wrote: >>>>> >>>>>> Thank you for merging the pull request. Could I send a pull request >>>>>> converting to UTF-8 encoding against the theme properties? >>>>>> >>>>>> 2016年7月7日(木) 21:39 Hiroyuki Wada <wadahiro(a)gmail.com&gt;: >>>>>> >>>>>>> Yes. I will send a pull request later! >>>>>>> >>>>>>> 2016年7月7日(木) 21:20 Stian Thorgersen <sthorger(a)redhat.com&gt;: >>>>>>> >>>>>>>> Are you doing a PR as well? >>>>>>>> >>>>>>>> On 7 July 2016 at 12:33, Hiroyuki Wada <wadahiro(a)gmail.com&gt; wrote: >>>>>>>> >>>>>>>>> Ok. I created a JIRA issue. >>>>>>>>> https://issues.jboss.org/browse/KEYCLOAK-3278 >>>>>>>>> >>>>>>>>> 2016年7月6日(水) 19:36 Stian Thorgersen <sthorger(a)redhat.com&gt;: >>>>>>>>> >>>>>>>>>> Let's go for it. Please create a JIRA and send PR. Please just >>>>>>>>>> use "# encoding: utf-8" and not "# -*- coding: utf-8 -*-". The >>>>>>>>>> latter is just weird if you ask me. >>>>>>>>>> >>>>>>>>>> Thomas: It's optional, defaults to iso-8859-1 and the header >>>>>>>>>> will clearly state that a file uses a different encoding, so I disagree >>>>>>>>>> that there will be any confusion. >>>>>>>>>> >>>>>>>>>> On 6 July 2016 at 12:30, Hiroyuki Wada <wadahiro(a)gmail.com&gt; >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> On Wed, Jul 6, 2016 at 6:53 PM, Thomas Raehalme >>>>>>>>>>> <thomas.raehalme(a)aitiofinland.com&gt; wrote: >>>>>>>>>>> > From the javadoc of java.util.ResourceBundle the encoding for >>>>>>>>>>> .properties is >>>>>>>>>>> > still expected to be ISO-8859-1: >>>>>>>>>>> > >>>>>>>>>>> >> Constructing a PropertyResourceBundle instance from an >>>>>>>>>>> InputStream >>>>>>>>>>> >> requires that the input stream be encoded in ISO-8859-1. In >>>>>>>>>>> that case, >>>>>>>>>>> >> characters that cannot be represented in ISO-8859-1 encoding >>>>>>>>>>> must be >>>>>>>>>>> >> represented by Unicode Escapes as defined in section 3.3 of >>>>>>>>>>> The Java™ >>>>>>>>>>> >> Language Specification whereas the other constructor which >>>>>>>>>>> takes a Reader >>>>>>>>>>> >> does not have that limitation. >>>>>>>>>>> >>>>>>>>>>> It's a case of using constructor which takes an InputStream. >>>>>>>>>>> It says we can use any encoding when we use the other >>>>>>>>>>> constructor >>>>>>>>>>> which takes a Reader. >>>>>>>>>>> >>>>>>>>>>> I understand your concern. So what do you think supporting >>>>>>>>>>> utf-8 with >>>>>>>>>>> a header like Stian suggested? >>>>>>>>>>> I think it can avoid confusion because the encoding is noted in >>>>>>>>>>> the >>>>>>>>>>> file itself... >>>>>>>>>>> >>>>>>>>>>> Regards, >>>>>>>>>>> >>>>>>>>>>> > >>>>>>>>>>> https://docs.oracle.com/javase/8/docs/api/java/util/PropertyResourceBundl... >>>>>>>>>>> > >>>>>>>>>>> > I understand that Spring allows one to read .properties using >>>>>>>>>>> any encoding, >>>>>>>>>>> > but you need to specify the encoding in Spring configuration >>>>>>>>>>> external to the >>>>>>>>>>> > .properties file. >>>>>>>>>>> > >>>>>>>>>>> > In my opinion it would cause unnecessary confusion amongst >>>>>>>>>>> developers to use >>>>>>>>>>> > any other encoding than the one defined by the official >>>>>>>>>>> documentation for >>>>>>>>>>> > Properties. >>>>>>>>>>> > >>>>>>>>>>> > Best regards, >>>>>>>>>>> > Thomas >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > On Wed, Jul 6, 2016 at 12:40 PM, Hiroyuki Wada < >>>>>>>>>>> wadahiro(a)gmail.com&gt; wrote: >>>>>>>>>>> >> >>>>>>>>>>> >> I think it was true before Java 1.6 because Java standard >>>>>>>>>>> library >>>>>>>>>>> >> (java.util.ResourceBundle or java.util.Properties) only >>>>>>>>>>> supported >>>>>>>>>>> >> ISO-8859-1 encoding file. >>>>>>>>>>> >> But they support any encoding after Java 1.6. There are some >>>>>>>>>>> >> frameworks which can read .properties with any encoding. For >>>>>>>>>>> example, >>>>>>>>>>> >> Spring Framework can read .properties with any encoding. I >>>>>>>>>>> think it's >>>>>>>>>>> >> very useful to read UTF-8 directly for multibyte language >>>>>>>>>>> country. >>>>>>>>>>> >> >>>>>>>>>>> >> On Wed, Jul 6, 2016 at 3:32 PM, Thomas Raehalme >>>>>>>>>>> >> <thomas.raehalme(a)aitiofinland.com&gt; wrote: >>>>>>>>>>> >> > Please correct me if I am wrong, but I have been under the >>>>>>>>>>> impression >>>>>>>>>>> >> > that >>>>>>>>>>> >> > Java .properties files should always use encoding >>>>>>>>>>> ISO-8859-1. Characters >>>>>>>>>>> >> > not >>>>>>>>>>> >> > present in ISO-8859-1 can be written in \uxxxx. Won't it >>>>>>>>>>> make things >>>>>>>>>>> >> > confusing to developers if another encoding is used here >>>>>>>>>>> instead? >>>>>>>>>>> >> > >>>>>>>>>>> >> > >>>>>>>>>>> >> > >>>>>>>>>>> https://docs.oracle.com/javase/8/docs/api/java/util/Properties.html#load-... >>>>>>>>>>> >> > >>>>>>>>>>> >> > >>>>>>>>>>> https://docs.oracle.com/javase/8/docs/api/java/util/Properties.html#store... >>>>>>>>>>> >> > >>>>>>>>>>> >> > If alternate encodings are desired how about supporting >>>>>>>>>>> the XML format >>>>>>>>>>> >> > of >>>>>>>>>>> >> > Properties? >>>>>>>>>>> >> > >>>>>>>>>>> >> > Best regards, >>>>>>>>>>> >> > Thomas >>>>>>>>>>> >> > >>>>>>>>>>> >> > >>>>>>>>>>> >> > On Wed, Jul 6, 2016 at 9:04 AM, Stian Thorgersen < >>>>>>>>>>> sthorger(a)redhat.com&gt; >>>>>>>>>>> >> > wrote: >>>>>>>>>>> >> >> >>>>>>>>>>> >> >> Both iso-8859-1 and utf-8 message bundles should be able >>>>>>>>>>> to co-exist. >>>>>>>>>>> >> >> >>>>>>>>>>> >> >> We can allow specifying the encoding in a comment on the >>>>>>>>>>> first line >>>>>>>>>>> >> >> like >>>>>>>>>>> >> >> this: >>>>>>>>>>> >> >> >>>>>>>>>>> >> >> # encoding=utf-8 >>>>>>>>>>> >> >> key=value >>>>>>>>>>> >> >> >>>>>>>>>>> >> >> # encoding=iso-8859-1 >>>>>>>>>>> >> >> key=value >>>>>>>>>>> >> >> >>>>>>>>>>> >> >> If the first line in the file doesn't contain the comment >>>>>>>>>>> with the >>>>>>>>>>> >> >> encoding then we should default to iso-8859-1 for >>>>>>>>>>> backwards >>>>>>>>>>> >> >> compatibility >>>>>>>>>>> >> >> >>>>>>>>>>> >> >> On 5 July 2016 at 09:49, Hiroyuki Wada < >>>>>>>>>>> wadahiro(a)gmail.com&gt; wrote: >>>>>>>>>>> >> >>> >>>>>>>>>>> >> >>> Thanks for your comment. >>>>>>>>>>> >> >>> >>>>>>>>>>> >> >>> > If we want to change to utf-8 we'd still need to >>>>>>>>>>> support iso.. for >>>>>>>>>>> >> >>> > backwards compatibility. >>>>>>>>>>> >> >>> >>>>>>>>>>> >> >>> If we change to UTF-8, we can still read unicode >>>>>>>>>>> codepoint like >>>>>>>>>>> >> >>> '\u00e8'. >>>>>>>>>>> >> >>> There is an incompatibility when non-ascii characters >>>>>>>>>>> are used in >>>>>>>>>>> >> >>> message properties. >>>>>>>>>>> >> >>> The non-ascii characters are 0xA0 - 0xFF codes (please >>>>>>>>>>> refer codepage >>>>>>>>>>> >> >>> layout: >>>>>>>>>>> https://en.wikipedia.org/wiki/ISO/IEC_8859-1#Codepage_layout ) >>>>>>>>>>> >> >>> >>>>>>>>>>> >> >>> I think the non-ascii characters might be used in French >>>>>>>>>>> messages like >>>>>>>>>>> >> >>> 'à' so I agree to support ISO-8859-1 for backwards >>>>>>>>>>> compatibility. >>>>>>>>>>> >> >>> To support this, I think we can add a property like >>>>>>>>>>> "messageEncoding" >>>>>>>>>>> >> >>> in keycloak-server.json as below. Is it a good idea? >>>>>>>>>>> >> >>> >>>>>>>>>>> >> >>> "theme": { >>>>>>>>>>> >> >>> "staticMaxAge": 2592000, >>>>>>>>>>> >> >>> "cacheTemplates": true, >>>>>>>>>>> >> >>> "cacheThemes": true, >>>>>>>>>>> >> >>> "messageEncoding": "ISO-8859-1", >>>>>>>>>>> >> >>> "folder": { >>>>>>>>>>> >> >>> "dir": "${jboss.home.dir}/themes" >>>>>>>>>>> >> >>> } >>>>>>>>>>> >> >>> }, >>>>>>>>>>> >> >>> >>>>>>>>>>> >> >>> Regards, >>>>>>>>>>> >> >>> >>>>>>>>>>> >> >>> On Mon, Jul 4, 2016 at 10:33 PM, Stian Thorgersen >>>>>>>>>>> >> >>> <sthorger(a)redhat.com&gt; >>>>>>>>>>> >> >>> wrote: >>>>>>>>>>> >> >>> > We have in the past discussed this and decided to >>>>>>>>>>> stick with >>>>>>>>>>> >> >>> > ISO-8859-1. >>>>>>>>>>> >> >>> > That was probably not the best idea though. If we want >>>>>>>>>>> to change to >>>>>>>>>>> >> >>> > utf-8 >>>>>>>>>>> >> >>> > we'd still need to support iso.. for backwards >>>>>>>>>>> compatibility. >>>>>>>>>>> >> >>> > >>>>>>>>>>> >> >>> > On 4 July 2016 at 14:37, Bruno Oliveira < >>>>>>>>>>> bruno(a)abstractj.org&gt; wrote: >>>>>>>>>>> >> >>> >> >>>>>>>>>>> >> >>> >> It makes sense, maybe file a Jira associated with: >>>>>>>>>>> >> >>> >> https://issues.jboss.org/browse/KEYCLOAK-3259 ? >>>>>>>>>>> >> >>> >> >>>>>>>>>>> >> >>> >> On 2016-07-04, Hiroyuki Wada wrote: >>>>>>>>>>> >> >>> >> > Hello all, >>>>>>>>>>> >> >>> >> > >>>>>>>>>>> >> >>> >> > I am trying to translate all base theme messages to >>>>>>>>>>> my country >>>>>>>>>>> >> >>> >> > language, Japanese. And I'd like to contribute >>>>>>>>>>> them. Before that >>>>>>>>>>> >> >>> >> > work, >>>>>>>>>>> >> >>> >> > I'd like to propose about the files encoding. >>>>>>>>>>> >> >>> >> > >>>>>>>>>>> >> >>> >> > Currently, the message files (*.properties) are >>>>>>>>>>> loaded with >>>>>>>>>>> >> >>> >> > ISO-8859-1 >>>>>>>>>>> >> >>> >> > encoding. Therefore, it is necessary to convert the >>>>>>>>>>> files by >>>>>>>>>>> >> >>> >> > 'native2ascii' command beforehand. However we can >>>>>>>>>>> directly read >>>>>>>>>>> >> >>> >> > the >>>>>>>>>>> >> >>> >> > property files with UTF-8 encoding in java 1.6 or >>>>>>>>>>> later because >>>>>>>>>>> >> >>> >> > 'java.util.Properties#load(java.io.Reader)' method >>>>>>>>>>> was introduced >>>>>>>>>>> >> >>> >> > as >>>>>>>>>>> >> >>> >> > below. >>>>>>>>>>> >> >>> >> > >>>>>>>>>>> >> >>> >> > >>>>>>>>>>> >> >>> >> > >>>>>>>>>>> >> >>> >> > >>>>>>>>>>> >> >>> >> > >>>>>>>>>>> http://docs.oracle.com/javase/6/docs/api/java/util/Properties.html#load(j... >>>>>>>>>>> >> >>> >> > >>>>>>>>>>> >> >>> >> > So, my proposal is supporting the message files >>>>>>>>>>> with UTF-8 >>>>>>>>>>> >> >>> >> > encoding. >>>>>>>>>>> >> >>> >> > I >>>>>>>>>>> >> >>> >> > believe that it's very developers/customers >>>>>>>>>>> friendly. In >>>>>>>>>>> >> >>> >> > addition, >>>>>>>>>>> >> >>> >> > we >>>>>>>>>>> >> >>> >> > can easily review the translated messages on the >>>>>>>>>>> github pull >>>>>>>>>>> >> >>> >> > request >>>>>>>>>>> >> >>> >> > view and so on. What do you think? >>>>>>>>>>> >> >>> >> > >>>>>>>>>>> >> >>> >> > If it's ok, I'll create a JIRA issue and create a >>>>>>>>>>> pull request. >>>>>>>>>>> >> >>> >> > >>>>>>>>>>> >> >>> >> > Regards, >>>>>>>>>>> >> >>> >> > >>>>>>>>>>> >> >>> >> > -- >>>>>>>>>>> >> >>> >> > Hiroyuki Wada, >>>>>>>>>>> >> >>> >> > Developer, >>>>>>>>>>> >> >>> >> > Nomura Research Institute, Ltd. >>>>>>>>>>> >> >>> >> > _______________________________________________ >>>>>>>>>>> >> >>> >> > keycloak-dev mailing list >>>>>>>>>>> >> >>> >> > keycloak-dev(a)lists.jboss.org >>>>>>>>>>> >> >>> >> > >>>>>>>>>>> https://lists.jboss.org/mailman/listinfo/keycloak-dev >>>>>>>>>>> >> >>> >> >>>>>>>>>>> >> >>> >> -- >>>>>>>>>>> >> >>> >> >>>>>>>>>>> >> >>> >> abstractj >>>>>>>>>>> >> >>> >> PGP: 0x84DC9914 >>>>>>>>>>> >> >>> >> _______________________________________________ >>>>>>>>>>> >> >>> >> keycloak-dev mailing list >>>>>>>>>>> >> >>> >> keycloak-dev(a)lists.jboss.org >>>>>>>>>>> >> >>> >> https://lists.jboss.org/mailman/listinfo/keycloak-dev >>>>>>>>>>> >> >>> > >>>>>>>>>>> >> >>> > >>>>>>>>>>> >> >> >>>>>>>>>>> >> >> >>>>>>>>>>> >> >> >>>>>>>>>>> >> >> _______________________________________________ >>>>>>>>>>> >> >> keycloak-dev mailing list >>>>>>>>>>> >> >> keycloak-dev(a)lists.jboss.org >>>>>>>>>>> >> >> https://lists.jboss.org/mailman/listinfo/keycloak-dev >>>>>>>>>>> >> > >>>>>>>>>>> >> > >>>>>>>>>>> >> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> keycloak-dev mailing list >>>>>>>>>>> keycloak-dev(a)lists.jboss.org >>>>>>>>>>> https://lists.jboss.org/mailman/listinfo/keycloak-dev >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>> >>>>> >>> >> > _______________________________________________ > keycloak-dev mailing list > keycloak-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/keycloak-dev >