Re: [keycloak-dev] Support UTF-8 encoding theme message properites

I tried iconv, but it doesn't do anything. Just outputs the iso-8859-1 as before: iconv -f iso-8859-1 -t utf-8 ./themes/src/main/resources/theme/base/login/messages/messages_ru.properties On 15 July 2016 at 09:04, Thomas Raehalme < thomas.raehalme(a)aitiofinland.com&gt; wrote: > iconv has been a good tool to convert files from one encoding to another. > > Best regards, > Thomas > > On Jul 15, 2016 09:57, "Hiroyuki Wada" <wadahiro(a)gmail.com&gt; wrote: > >> >> BTW do you have a reliable way to convert the files? I've tried various >>> tools, but none worked for me. >>> >> >> Yes, we can use native2ascii command. >> >> Example: >> native2ascii -encoding UTF-8 -reverse <input> <output> >> >> >> >> >>> On 15 July 2016 at 08:14, Stian Thorgersen <sthorger(a)redhat.com&gt; wrote: >>> >>>> I'd like the current messages to be converted by the people that have >>>> contributed them. They will be able to determine what encoding they prefer >>>> to use and also confirm that the conversion has been successful. >>>> >>>> On 15 July 2016 at 07:59, Hiroyuki Wada <wadahiro(a)gmail.com&gt; wrote: >>>> >>>>> I mean I'd like to change the message files encoding only. >>>>> For example, admin-messages for Russian language is currently written >>>>> in ISO-8859-1 encoding. >>>>> >>>>> >>>>> https://github.com/keycloak/keycloak/blob/master/themes/src/main/resource... >>>>> >>>>> Can I convert the language messages except for English? >>>>> >>>>> >>>>> 2016年7月15日(金) 14:03 Stian Thorgersen <sthorger(a)redhat.com&gt;: >>>>> >>>>>> No, we don't want the theme properties (and other properties files) >>>>>> to be utf-8. They should stick with the default encoding used by Java >>>>>> properties files. >>>>>> >>>>>> Messages are a special case, which is why we accepted it for them. >>>>>> >>>>>> On 14 July 2016 at 13:29, Hiroyuki Wada <wadahiro(a)gmail.com&gt; wrote: >>>>>> >>>>>>> Thank you for merging the pull request. Could I send a pull request >>>>>>> converting to UTF-8 encoding against the theme properties? >>>>>>> >>>>>>> 2016年7月7日(木) 21:39 Hiroyuki Wada <wadahiro(a)gmail.com&gt;: >>>>>>> >>>>>>>> Yes. I will send a pull request later! >>>>>>>> >>>>>>>> 2016年7月7日(木) 21:20 Stian Thorgersen <sthorger(a)redhat.com&gt;: >>>>>>>> >>>>>>>>> Are you doing a PR as well? >>>>>>>>> >>>>>>>>> On 7 July 2016 at 12:33, Hiroyuki Wada <wadahiro(a)gmail.com&gt; >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Ok. I created a JIRA issue. >>>>>>>>>> https://issues.jboss.org/browse/KEYCLOAK-3278 >>>>>>>>>> >>>>>>>>>> 2016年7月6日(水) 19:36 Stian Thorgersen <sthorger(a)redhat.com&gt;: >>>>>>>>>> >>>>>>>>>>> Let's go for it. Please create a JIRA and send PR. Please just >>>>>>>>>>> use "# encoding: utf-8" and not "# -*- coding: utf-8 -*-". The >>>>>>>>>>> latter is just weird if you ask me. >>>>>>>>>>> >>>>>>>>>>> Thomas: It's optional, defaults to iso-8859-1 and the header >>>>>>>>>>> will clearly state that a file uses a different encoding, so I disagree >>>>>>>>>>> that there will be any confusion. >>>>>>>>>>> >>>>>>>>>>> On 6 July 2016 at 12:30, Hiroyuki Wada <wadahiro(a)gmail.com&gt; >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> On Wed, Jul 6, 2016 at 6:53 PM, Thomas Raehalme >>>>>>>>>>>> <thomas.raehalme(a)aitiofinland.com&gt; wrote: >>>>>>>>>>>> > From the javadoc of java.util.ResourceBundle the encoding >>>>>>>>>>>> for .properties is >>>>>>>>>>>> > still expected to be ISO-8859-1: >>>>>>>>>>>> > >>>>>>>>>>>> >> Constructing a PropertyResourceBundle instance from an >>>>>>>>>>>> InputStream >>>>>>>>>>>> >> requires that the input stream be encoded in ISO-8859-1. In >>>>>>>>>>>> that case, >>>>>>>>>>>> >> characters that cannot be represented in ISO-8859-1 >>>>>>>>>>>> encoding must be >>>>>>>>>>>> >> represented by Unicode Escapes as defined in section 3.3 of >>>>>>>>>>>> The Java™ >>>>>>>>>>>> >> Language Specification whereas the other constructor which >>>>>>>>>>>> takes a Reader >>>>>>>>>>>> >> does not have that limitation. >>>>>>>>>>>> >>>>>>>>>>>> It's a case of using constructor which takes an InputStream. >>>>>>>>>>>> It says we can use any encoding when we use the other >>>>>>>>>>>> constructor >>>>>>>>>>>> which takes a Reader. >>>>>>>>>>>> >>>>>>>>>>>> I understand your concern. So what do you think supporting >>>>>>>>>>>> utf-8 with >>>>>>>>>>>> a header like Stian suggested? >>>>>>>>>>>> I think it can avoid confusion because the encoding is noted >>>>>>>>>>>> in the >>>>>>>>>>>> file itself... >>>>>>>>>>>> >>>>>>>>>>>> Regards, >>>>>>>>>>>> >>>>>>>>>>>> > >>>>>>>>>>>> https://docs.oracle.com/javase/8/docs/api/java/util/PropertyResourceBundl... >>>>>>>>>>>> > >>>>>>>>>>>> > I understand that Spring allows one to read .properties >>>>>>>>>>>> using any encoding, >>>>>>>>>>>> > but you need to specify the encoding in Spring configuration >>>>>>>>>>>> external to the >>>>>>>>>>>> > .properties file. >>>>>>>>>>>> > >>>>>>>>>>>> > In my opinion it would cause unnecessary confusion amongst >>>>>>>>>>>> developers to use >>>>>>>>>>>> > any other encoding than the one defined by the official >>>>>>>>>>>> documentation for >>>>>>>>>>>> > Properties. >>>>>>>>>>>> > >>>>>>>>>>>> > Best regards, >>>>>>>>>>>> > Thomas >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > On Wed, Jul 6, 2016 at 12:40 PM, Hiroyuki Wada < >>>>>>>>>>>> wadahiro(a)gmail.com&gt; wrote: >>>>>>>>>>>> >> >>>>>>>>>>>> >> I think it was true before Java 1.6 because Java standard >>>>>>>>>>>> library >>>>>>>>>>>> >> (java.util.ResourceBundle or java.util.Properties) only >>>>>>>>>>>> supported >>>>>>>>>>>> >> ISO-8859-1 encoding file. >>>>>>>>>>>> >> But they support any encoding after Java 1.6. There are some >>>>>>>>>>>> >> frameworks which can read .properties with any encoding. >>>>>>>>>>>> For example, >>>>>>>>>>>> >> Spring Framework can read .properties with any encoding. I >>>>>>>>>>>> think it's >>>>>>>>>>>> >> very useful to read UTF-8 directly for multibyte language >>>>>>>>>>>> country. >>>>>>>>>>>> >> >>>>>>>>>>>> >> On Wed, Jul 6, 2016 at 3:32 PM, Thomas Raehalme >>>>>>>>>>>> >> <thomas.raehalme(a)aitiofinland.com&gt; wrote: >>>>>>>>>>>> >> > Please correct me if I am wrong, but I have been under >>>>>>>>>>>> the impression >>>>>>>>>>>> >> > that >>>>>>>>>>>> >> > Java .properties files should always use encoding >>>>>>>>>>>> ISO-8859-1. Characters >>>>>>>>>>>> >> > not >>>>>>>>>>>> >> > present in ISO-8859-1 can be written in \uxxxx. Won't it >>>>>>>>>>>> make things >>>>>>>>>>>> >> > confusing to developers if another encoding is used here >>>>>>>>>>>> instead? >>>>>>>>>>>> >> > >>>>>>>>>>>> >> > >>>>>>>>>>>> >> > >>>>>>>>>>>> https://docs.oracle.com/javase/8/docs/api/java/util/Properties.html#load-... >>>>>>>>>>>> >> > >>>>>>>>>>>> >> > >>>>>>>>>>>> https://docs.oracle.com/javase/8/docs/api/java/util/Properties.html#store... >>>>>>>>>>>> >> > >>>>>>>>>>>> >> > If alternate encodings are desired how about supporting >>>>>>>>>>>> the XML format >>>>>>>>>>>> >> > of >>>>>>>>>>>> >> > Properties? >>>>>>>>>>>> >> > >>>>>>>>>>>> >> > Best regards, >>>>>>>>>>>> >> > Thomas >>>>>>>>>>>> >> > >>>>>>>>>>>> >> > >>>>>>>>>>>> >> > On Wed, Jul 6, 2016 at 9:04 AM, Stian Thorgersen < >>>>>>>>>>>> sthorger(a)redhat.com&gt; >>>>>>>>>>>> >> > wrote: >>>>>>>>>>>> >> >> >>>>>>>>>>>> >> >> Both iso-8859-1 and utf-8 message bundles should be able >>>>>>>>>>>> to co-exist. >>>>>>>>>>>> >> >> >>>>>>>>>>>> >> >> We can allow specifying the encoding in a comment on the >>>>>>>>>>>> first line >>>>>>>>>>>> >> >> like >>>>>>>>>>>> >> >> this: >>>>>>>>>>>> >> >> >>>>>>>>>>>> >> >> # encoding=utf-8 >>>>>>>>>>>> >> >> key=value >>>>>>>>>>>> >> >> >>>>>>>>>>>> >> >> # encoding=iso-8859-1 >>>>>>>>>>>> >> >> key=value >>>>>>>>>>>> >> >> >>>>>>>>>>>> >> >> If the first line in the file doesn't contain the >>>>>>>>>>>> comment with the >>>>>>>>>>>> >> >> encoding then we should default to iso-8859-1 for >>>>>>>>>>>> backwards >>>>>>>>>>>> >> >> compatibility >>>>>>>>>>>> >> >> >>>>>>>>>>>> >> >> On 5 July 2016 at 09:49, Hiroyuki Wada < >>>>>>>>>>>> wadahiro(a)gmail.com&gt; wrote: >>>>>>>>>>>> >> >>> >>>>>>>>>>>> >> >>> Thanks for your comment. >>>>>>>>>>>> >> >>> >>>>>>>>>>>> >> >>> > If we want to change to utf-8 we'd still need to >>>>>>>>>>>> support iso.. for >>>>>>>>>>>> >> >>> > backwards compatibility. >>>>>>>>>>>> >> >>> >>>>>>>>>>>> >> >>> If we change to UTF-8, we can still read unicode >>>>>>>>>>>> codepoint like >>>>>>>>>>>> >> >>> '\u00e8'. >>>>>>>>>>>> >> >>> There is an incompatibility when non-ascii characters >>>>>>>>>>>> are used in >>>>>>>>>>>> >> >>> message properties. >>>>>>>>>>>> >> >>> The non-ascii characters are 0xA0 - 0xFF codes (please >>>>>>>>>>>> refer codepage >>>>>>>>>>>> >> >>> layout: >>>>>>>>>>>> https://en.wikipedia.org/wiki/ISO/IEC_8859-1#Codepage_layout ) >>>>>>>>>>>> >> >>> >>>>>>>>>>>> >> >>> I think the non-ascii characters might be used in >>>>>>>>>>>> French messages like >>>>>>>>>>>> >> >>> 'à' so I agree to support ISO-8859-1 for backwards >>>>>>>>>>>> compatibility. >>>>>>>>>>>> >> >>> To support this, I think we can add a property like >>>>>>>>>>>> "messageEncoding" >>>>>>>>>>>> >> >>> in keycloak-server.json as below. Is it a good idea? >>>>>>>>>>>> >> >>> >>>>>>>>>>>> >> >>> "theme": { >>>>>>>>>>>> >> >>> "staticMaxAge": 2592000, >>>>>>>>>>>> >> >>> "cacheTemplates": true, >>>>>>>>>>>> >> >>> "cacheThemes": true, >>>>>>>>>>>> >> >>> "messageEncoding": "ISO-8859-1", >>>>>>>>>>>> >> >>> "folder": { >>>>>>>>>>>> >> >>> "dir": "${jboss.home.dir}/themes" >>>>>>>>>>>> >> >>> } >>>>>>>>>>>> >> >>> }, >>>>>>>>>>>> >> >>> >>>>>>>>>>>> >> >>> Regards, >>>>>>>>>>>> >> >>> >>>>>>>>>>>> >> >>> On Mon, Jul 4, 2016 at 10:33 PM, Stian Thorgersen >>>>>>>>>>>> >> >>> <sthorger(a)redhat.com&gt; >>>>>>>>>>>> >> >>> wrote: >>>>>>>>>>>> >> >>> > We have in the past discussed this and decided to >>>>>>>>>>>> stick with >>>>>>>>>>>> >> >>> > ISO-8859-1. >>>>>>>>>>>> >> >>> > That was probably not the best idea though. If we >>>>>>>>>>>> want to change to >>>>>>>>>>>> >> >>> > utf-8 >>>>>>>>>>>> >> >>> > we'd still need to support iso.. for backwards >>>>>>>>>>>> compatibility. >>>>>>>>>>>> >> >>> > >>>>>>>>>>>> >> >>> > On 4 July 2016 at 14:37, Bruno Oliveira < >>>>>>>>>>>> bruno(a)abstractj.org&gt; wrote: >>>>>>>>>>>> >> >>> >> >>>>>>>>>>>> >> >>> >> It makes sense, maybe file a Jira associated with: >>>>>>>>>>>> >> >>> >> https://issues.jboss.org/browse/KEYCLOAK-3259 ? >>>>>>>>>>>> >> >>> >> >>>>>>>>>>>> >> >>> >> On 2016-07-04, Hiroyuki Wada wrote: >>>>>>>>>>>> >> >>> >> > Hello all, >>>>>>>>>>>> >> >>> >> > >>>>>>>>>>>> >> >>> >> > I am trying to translate all base theme messages >>>>>>>>>>>> to my country >>>>>>>>>>>> >> >>> >> > language, Japanese. And I'd like to contribute >>>>>>>>>>>> them. Before that >>>>>>>>>>>> >> >>> >> > work, >>>>>>>>>>>> >> >>> >> > I'd like to propose about the files encoding. >>>>>>>>>>>> >> >>> >> > >>>>>>>>>>>> >> >>> >> > Currently, the message files (*.properties) are >>>>>>>>>>>> loaded with >>>>>>>>>>>> >> >>> >> > ISO-8859-1 >>>>>>>>>>>> >> >>> >> > encoding. Therefore, it is necessary to convert >>>>>>>>>>>> the files by >>>>>>>>>>>> >> >>> >> > 'native2ascii' command beforehand. However we can >>>>>>>>>>>> directly read >>>>>>>>>>>> >> >>> >> > the >>>>>>>>>>>> >> >>> >> > property files with UTF-8 encoding in java 1.6 or >>>>>>>>>>>> later because >>>>>>>>>>>> >> >>> >> > 'java.util.Properties#load(java.io.Reader)' method >>>>>>>>>>>> was introduced >>>>>>>>>>>> >> >>> >> > as >>>>>>>>>>>> >> >>> >> > below. >>>>>>>>>>>> >> >>> >> > >>>>>>>>>>>> >> >>> >> > >>>>>>>>>>>> >> >>> >> > >>>>>>>>>>>> >> >>> >> > >>>>>>>>>>>> >> >>> >> > >>>>>>>>>>>> http://docs.oracle.com/javase/6/docs/api/java/util/Properties.html#load(j... >>>>>>>>>>>> >> >>> >> > >>>>>>>>>>>> >> >>> >> > So, my proposal is supporting the message files >>>>>>>>>>>> with UTF-8 >>>>>>>>>>>> >> >>> >> > encoding. >>>>>>>>>>>> >> >>> >> > I >>>>>>>>>>>> >> >>> >> > believe that it's very developers/customers >>>>>>>>>>>> friendly. In >>>>>>>>>>>> >> >>> >> > addition, >>>>>>>>>>>> >> >>> >> > we >>>>>>>>>>>> >> >>> >> > can easily review the translated messages on the >>>>>>>>>>>> github pull >>>>>>>>>>>> >> >>> >> > request >>>>>>>>>>>> >> >>> >> > view and so on. What do you think? >>>>>>>>>>>> >> >>> >> > >>>>>>>>>>>> >> >>> >> > If it's ok, I'll create a JIRA issue and create a >>>>>>>>>>>> pull request. >>>>>>>>>>>> >> >>> >> > >>>>>>>>>>>> >> >>> >> > Regards, >>>>>>>>>>>> >> >>> >> > >>>>>>>>>>>> >> >>> >> > -- >>>>>>>>>>>> >> >>> >> > Hiroyuki Wada, >>>>>>>>>>>> >> >>> >> > Developer, >>>>>>>>>>>> >> >>> >> > Nomura Research Institute, Ltd. >>>>>>>>>>>> >> >>> >> > _______________________________________________ >>>>>>>>>>>> >> >>> >> > keycloak-dev mailing list >>>>>>>>>>>> >> >>> >> > keycloak-dev(a)lists.jboss.org >>>>>>>>>>>> >> >>> >> > >>>>>>>>>>>> https://lists.jboss.org/mailman/listinfo/keycloak-dev >>>>>>>>>>>> >> >>> >> >>>>>>>>>>>> >> >>> >> -- >>>>>>>>>>>> >> >>> >> >>>>>>>>>>>> >> >>> >> abstractj >>>>>>>>>>>> >> >>> >> PGP: 0x84DC9914 >>>>>>>>>>>> >> >>> >> _______________________________________________ >>>>>>>>>>>> >> >>> >> keycloak-dev mailing list >>>>>>>>>>>> >> >>> >> keycloak-dev(a)lists.jboss.org >>>>>>>>>>>> >> >>> >> >>>>>>>>>>>> https://lists.jboss.org/mailman/listinfo/keycloak-dev >>>>>>>>>>>> >> >>> > >>>>>>>>>>>> >> >>> > >>>>>>>>>>>> >> >> >>>>>>>>>>>> >> >> >>>>>>>>>>>> >> >> >>>>>>>>>>>> >> >> _______________________________________________ >>>>>>>>>>>> >> >> keycloak-dev mailing list >>>>>>>>>>>> >> >> keycloak-dev(a)lists.jboss.org >>>>>>>>>>>> >> >> https://lists.jboss.org/mailman/listinfo/keycloak-dev >>>>>>>>>>>> >> > >>>>>>>>>>>> >> > >>>>>>>>>>>> >> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> >>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>> keycloak-dev mailing list >>>>>>>>>>>> keycloak-dev(a)lists.jboss.org >>>>>>>>>>>> https://lists.jboss.org/mailman/listinfo/keycloak-dev >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>> >>>>>> >>>> >>> >> _______________________________________________ >> keycloak-dev mailing list >> keycloak-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/keycloak-dev >> >