[keycloak-dev] Support UTF-8 encoding theme message properites

Stian Thorgersen sthorger at redhat.com
Fri Jul 15 04:41:56 EDT 2016 

I tried iconv, but it doesn't do anything. Just outputs the iso-8859-1 as
before:

iconv -f iso-8859-1 -t utf-8
./themes/src/main/resources/theme/base/login/messages/messages_ru.properties

On 15 July 2016 at 09:04, Thomas Raehalme <thomas.raehalme at aitiofinland.com>
wrote:

> iconv has been a good tool to convert files from one encoding to another.
>
> Best regards,
> Thomas
>
> On Jul 15, 2016 09:57, "Hiroyuki Wada" <wadahiro at gmail.com> wrote:
>
>>
>> BTW do you have a reliable way to convert the files? I've tried various
>>> tools, but none worked for me.
>>>
>>
>> Yes, we can use native2ascii command.
>>
>> Example:
>> native2ascii -encoding UTF-8 -reverse <input> <output>
>>
>>
>>
>>
>>> On 15 July 2016 at 08:14, Stian Thorgersen <sthorger at redhat.com> wrote:
>>>
>>>> I'd like the current messages to be converted by the people that have
>>>> contributed them. They will be able to determine what encoding they prefer
>>>> to use and also confirm that the conversion has been successful.
>>>>
>>>> On 15 July 2016 at 07:59, Hiroyuki Wada <wadahiro at gmail.com> wrote:
>>>>
>>>>> I mean I'd like to change the message files encoding only.
>>>>> For example, admin-messages for Russian language is currently written
>>>>> in ISO-8859-1 encoding.
>>>>>
>>>>>
>>>>> https://github.com/keycloak/keycloak/blob/master/themes/src/main/resources/theme/base/admin/messages/admin-messages_ru.properties
>>>>>
>>>>> Can I convert the language messages except for English?
>>>>>
>>>>>
>>>>> 2016年7月15日(金) 14:03 Stian Thorgersen <sthorger at redhat.com>:
>>>>>
>>>>>> No, we don't want the theme properties (and other properties files)
>>>>>> to be utf-8. They should stick with the default encoding used by Java
>>>>>> properties files.
>>>>>>
>>>>>> Messages are a special case, which is why we accepted it for them.
>>>>>>
>>>>>> On 14 July 2016 at 13:29, Hiroyuki Wada <wadahiro at gmail.com> wrote:
>>>>>>
>>>>>>> Thank you for merging the pull request. Could I send a pull request
>>>>>>> converting to UTF-8 encoding against the theme properties?
>>>>>>>
>>>>>>> 2016年7月7日(木) 21:39 Hiroyuki Wada <wadahiro at gmail.com>:
>>>>>>>
>>>>>>>> Yes. I will send a pull request later!
>>>>>>>>
>>>>>>>> 2016年7月7日(木) 21:20 Stian Thorgersen <sthorger at redhat.com>:
>>>>>>>>
>>>>>>>>> Are you doing a PR as well?
>>>>>>>>>
>>>>>>>>> On 7 July 2016 at 12:33, Hiroyuki Wada <wadahiro at gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Ok. I created a JIRA issue.
>>>>>>>>>> https://issues.jboss.org/browse/KEYCLOAK-3278
>>>>>>>>>>
>>>>>>>>>> 2016年7月6日(水) 19:36 Stian Thorgersen <sthorger at redhat.com>:
>>>>>>>>>>
>>>>>>>>>>> Let's go for it. Please create a JIRA and send PR. Please just
>>>>>>>>>>> use "# encoding: utf-8" and not "# -*- coding: utf-8 -*-". The
>>>>>>>>>>> latter is just weird if you ask me.
>>>>>>>>>>>
>>>>>>>>>>> Thomas: It's optional, defaults to iso-8859-1 and the header
>>>>>>>>>>> will clearly state that a file uses a different encoding, so I disagree
>>>>>>>>>>> that there will be any confusion.
>>>>>>>>>>>
>>>>>>>>>>> On 6 July 2016 at 12:30, Hiroyuki Wada <wadahiro at gmail.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> On Wed, Jul 6, 2016 at 6:53 PM, Thomas Raehalme
>>>>>>>>>>>> <thomas.raehalme at aitiofinland.com> wrote:
>>>>>>>>>>>> > From the javadoc of java.util.ResourceBundle the encoding for
>>>>>>>>>>>> .properties is
>>>>>>>>>>>> > still expected to be ISO-8859-1:
>>>>>>>>>>>> >
>>>>>>>>>>>> >> Constructing a PropertyResourceBundle instance from an
>>>>>>>>>>>> InputStream
>>>>>>>>>>>> >> requires that the input stream be encoded in ISO-8859-1. In
>>>>>>>>>>>> that case,
>>>>>>>>>>>> >> characters that cannot be represented in ISO-8859-1 encoding
>>>>>>>>>>>> must be
>>>>>>>>>>>> >> represented by Unicode Escapes as defined in section 3.3 of
>>>>>>>>>>>> The Java™
>>>>>>>>>>>> >> Language Specification whereas the other constructor which
>>>>>>>>>>>> takes a Reader
>>>>>>>>>>>> >> does not have that limitation.
>>>>>>>>>>>>
>>>>>>>>>>>> It's a case of using constructor which takes an InputStream.
>>>>>>>>>>>> It says we can use any encoding when we use the other
>>>>>>>>>>>> constructor
>>>>>>>>>>>> which takes a Reader.
>>>>>>>>>>>>
>>>>>>>>>>>> I understand your concern. So what do you think supporting
>>>>>>>>>>>> utf-8 with
>>>>>>>>>>>> a header like Stian suggested?
>>>>>>>>>>>> I think it can avoid confusion because the encoding is noted in
>>>>>>>>>>>> the
>>>>>>>>>>>> file itself...
>>>>>>>>>>>>
>>>>>>>>>>>> Regards,
>>>>>>>>>>>>
>>>>>>>>>>>> >
>>>>>>>>>>>> https://docs.oracle.com/javase/8/docs/api/java/util/PropertyResourceBundle.html
>>>>>>>>>>>> >
>>>>>>>>>>>> > I understand that Spring allows one to read .properties using
>>>>>>>>>>>> any encoding,
>>>>>>>>>>>> > but you need to specify the encoding in Spring configuration
>>>>>>>>>>>> external to the
>>>>>>>>>>>> > .properties file.
>>>>>>>>>>>> >
>>>>>>>>>>>> > In my opinion it would cause unnecessary confusion amongst
>>>>>>>>>>>> developers to use
>>>>>>>>>>>> > any other encoding than the one defined by the official
>>>>>>>>>>>> documentation for
>>>>>>>>>>>> > Properties.
>>>>>>>>>>>> >
>>>>>>>>>>>> > Best regards,
>>>>>>>>>>>> > Thomas
>>>>>>>>>>>> >
>>>>>>>>>>>> >
>>>>>>>>>>>> > On Wed, Jul 6, 2016 at 12:40 PM, Hiroyuki Wada <
>>>>>>>>>>>> wadahiro at gmail.com> wrote:
>>>>>>>>>>>> >>
>>>>>>>>>>>> >> I think it was true before Java 1.6 because Java standard
>>>>>>>>>>>> library
>>>>>>>>>>>> >> (java.util.ResourceBundle or java.util.Properties) only
>>>>>>>>>>>> supported
>>>>>>>>>>>> >> ISO-8859-1 encoding file.
>>>>>>>>>>>> >> But they support any encoding after Java 1.6. There are some
>>>>>>>>>>>> >> frameworks which can read .properties with any encoding. For
>>>>>>>>>>>> example,
>>>>>>>>>>>> >> Spring Framework can read .properties with any encoding. I
>>>>>>>>>>>> think it's
>>>>>>>>>>>> >> very useful to read UTF-8 directly for multibyte language
>>>>>>>>>>>> country.
>>>>>>>>>>>> >>
>>>>>>>>>>>> >> On Wed, Jul 6, 2016 at 3:32 PM, Thomas Raehalme
>>>>>>>>>>>> >> <thomas.raehalme at aitiofinland.com> wrote:
>>>>>>>>>>>> >> > Please correct me if I am wrong, but I have been under the
>>>>>>>>>>>> impression
>>>>>>>>>>>> >> > that
>>>>>>>>>>>> >> > Java .properties files should always use encoding
>>>>>>>>>>>> ISO-8859-1. Characters
>>>>>>>>>>>> >> > not
>>>>>>>>>>>> >> > present in ISO-8859-1 can be written in \uxxxx. Won't it
>>>>>>>>>>>> make things
>>>>>>>>>>>> >> > confusing to developers if another encoding is used here
>>>>>>>>>>>> instead?
>>>>>>>>>>>> >> >
>>>>>>>>>>>> >> >
>>>>>>>>>>>> >> >
>>>>>>>>>>>> https://docs.oracle.com/javase/8/docs/api/java/util/Properties.html#load-java.io.InputStream-
>>>>>>>>>>>> >> >
>>>>>>>>>>>> >> >
>>>>>>>>>>>> https://docs.oracle.com/javase/8/docs/api/java/util/Properties.html#store-java.io.OutputStream-java.lang.String-
>>>>>>>>>>>> >> >
>>>>>>>>>>>> >> > If alternate encodings are desired how about supporting
>>>>>>>>>>>> the XML format
>>>>>>>>>>>> >> > of
>>>>>>>>>>>> >> > Properties?
>>>>>>>>>>>> >> >
>>>>>>>>>>>> >> > Best regards,
>>>>>>>>>>>> >> > Thomas
>>>>>>>>>>>> >> >
>>>>>>>>>>>> >> >
>>>>>>>>>>>> >> > On Wed, Jul 6, 2016 at 9:04 AM, Stian Thorgersen <
>>>>>>>>>>>> sthorger at redhat.com>
>>>>>>>>>>>> >> > wrote:
>>>>>>>>>>>> >> >>
>>>>>>>>>>>> >> >> Both iso-8859-1 and utf-8 message bundles should be able
>>>>>>>>>>>> to co-exist.
>>>>>>>>>>>> >> >>
>>>>>>>>>>>> >> >> We can allow specifying the encoding in a comment on the
>>>>>>>>>>>> first line
>>>>>>>>>>>> >> >> like
>>>>>>>>>>>> >> >> this:
>>>>>>>>>>>> >> >>
>>>>>>>>>>>> >> >> # encoding=utf-8
>>>>>>>>>>>> >> >> key=value
>>>>>>>>>>>> >> >>
>>>>>>>>>>>> >> >> # encoding=iso-8859-1
>>>>>>>>>>>> >> >> key=value
>>>>>>>>>>>> >> >>
>>>>>>>>>>>> >> >> If the first line in the file doesn't contain the comment
>>>>>>>>>>>> with the
>>>>>>>>>>>> >> >> encoding then we should default to iso-8859-1 for
>>>>>>>>>>>> backwards
>>>>>>>>>>>> >> >> compatibility
>>>>>>>>>>>> >> >>
>>>>>>>>>>>> >> >> On 5 July 2016 at 09:49, Hiroyuki Wada <
>>>>>>>>>>>> wadahiro at gmail.com> wrote:
>>>>>>>>>>>> >> >>>
>>>>>>>>>>>> >> >>> Thanks for your comment.
>>>>>>>>>>>> >> >>>
>>>>>>>>>>>> >> >>> > If we want to change to utf-8 we'd still need to
>>>>>>>>>>>> support iso.. for
>>>>>>>>>>>> >> >>> > backwards compatibility.
>>>>>>>>>>>> >> >>>
>>>>>>>>>>>> >> >>> If we change to UTF-8, we can still read unicode
>>>>>>>>>>>> codepoint like
>>>>>>>>>>>> >> >>> '\u00e8'.
>>>>>>>>>>>> >> >>> There is an incompatibility when non-ascii characters
>>>>>>>>>>>> are used in
>>>>>>>>>>>> >> >>> message properties.
>>>>>>>>>>>> >> >>> The non-ascii characters are 0xA0 - 0xFF codes (please
>>>>>>>>>>>> refer codepage
>>>>>>>>>>>> >> >>> layout:
>>>>>>>>>>>> https://en.wikipedia.org/wiki/ISO/IEC_8859-1#Codepage_layout )
>>>>>>>>>>>> >> >>>
>>>>>>>>>>>> >> >>> I think the non-ascii characters might be used in French
>>>>>>>>>>>> messages like
>>>>>>>>>>>> >> >>> 'à' so I agree to support ISO-8859-1 for backwards
>>>>>>>>>>>> compatibility.
>>>>>>>>>>>> >> >>> To support this, I think we can add a property like
>>>>>>>>>>>> "messageEncoding"
>>>>>>>>>>>> >> >>> in keycloak-server.json as below. Is it a good idea?
>>>>>>>>>>>> >> >>>
>>>>>>>>>>>> >> >>>     "theme": {
>>>>>>>>>>>> >> >>>         "staticMaxAge": 2592000,
>>>>>>>>>>>> >> >>>         "cacheTemplates": true,
>>>>>>>>>>>> >> >>>         "cacheThemes": true,
>>>>>>>>>>>> >> >>>         "messageEncoding": "ISO-8859-1",
>>>>>>>>>>>> >> >>>         "folder": {
>>>>>>>>>>>> >> >>>           "dir": "${jboss.home.dir}/themes"
>>>>>>>>>>>> >> >>>         }
>>>>>>>>>>>> >> >>>     },
>>>>>>>>>>>> >> >>>
>>>>>>>>>>>> >> >>> Regards,
>>>>>>>>>>>> >> >>>
>>>>>>>>>>>> >> >>> On Mon, Jul 4, 2016 at 10:33 PM, Stian Thorgersen
>>>>>>>>>>>> >> >>> <sthorger at redhat.com>
>>>>>>>>>>>> >> >>> wrote:
>>>>>>>>>>>> >> >>> > We have in the past discussed this and decided to
>>>>>>>>>>>> stick with
>>>>>>>>>>>> >> >>> > ISO-8859-1.
>>>>>>>>>>>> >> >>> > That was probably not the best idea though. If we want
>>>>>>>>>>>> to change to
>>>>>>>>>>>> >> >>> > utf-8
>>>>>>>>>>>> >> >>> > we'd still need to support iso.. for backwards
>>>>>>>>>>>> compatibility.
>>>>>>>>>>>> >> >>> >
>>>>>>>>>>>> >> >>> > On 4 July 2016 at 14:37, Bruno Oliveira <
>>>>>>>>>>>> bruno at abstractj.org> wrote:
>>>>>>>>>>>> >> >>> >>
>>>>>>>>>>>> >> >>> >> It makes sense, maybe file a Jira associated with:
>>>>>>>>>>>> >> >>> >> https://issues.jboss.org/browse/KEYCLOAK-3259 ?
>>>>>>>>>>>> >> >>> >>
>>>>>>>>>>>> >> >>> >> On 2016-07-04, Hiroyuki Wada wrote:
>>>>>>>>>>>> >> >>> >> > Hello all,
>>>>>>>>>>>> >> >>> >> >
>>>>>>>>>>>> >> >>> >> > I am trying to translate all base theme messages to
>>>>>>>>>>>> my country
>>>>>>>>>>>> >> >>> >> > language, Japanese. And I'd like to contribute
>>>>>>>>>>>> them. Before that
>>>>>>>>>>>> >> >>> >> > work,
>>>>>>>>>>>> >> >>> >> > I'd like to propose about the files encoding.
>>>>>>>>>>>> >> >>> >> >
>>>>>>>>>>>> >> >>> >> > Currently, the message files (*.properties) are
>>>>>>>>>>>> loaded with
>>>>>>>>>>>> >> >>> >> > ISO-8859-1
>>>>>>>>>>>> >> >>> >> > encoding. Therefore, it is necessary to convert the
>>>>>>>>>>>> files by
>>>>>>>>>>>> >> >>> >> > 'native2ascii' command beforehand. However we can
>>>>>>>>>>>> directly read
>>>>>>>>>>>> >> >>> >> > the
>>>>>>>>>>>> >> >>> >> > property files with UTF-8 encoding in java 1.6 or
>>>>>>>>>>>> later because
>>>>>>>>>>>> >> >>> >> > 'java.util.Properties#load(java.io.Reader)' method
>>>>>>>>>>>> was introduced
>>>>>>>>>>>> >> >>> >> > as
>>>>>>>>>>>> >> >>> >> > below.
>>>>>>>>>>>> >> >>> >> >
>>>>>>>>>>>> >> >>> >> >
>>>>>>>>>>>> >> >>> >> >
>>>>>>>>>>>> >> >>> >> >
>>>>>>>>>>>> >> >>> >> >
>>>>>>>>>>>> http://docs.oracle.com/javase/6/docs/api/java/util/Properties.html#load(java.io.Reader)
>>>>>>>>>>>> >> >>> >> >
>>>>>>>>>>>> >> >>> >> > So, my proposal is supporting the message files
>>>>>>>>>>>> with UTF-8
>>>>>>>>>>>> >> >>> >> > encoding.
>>>>>>>>>>>> >> >>> >> > I
>>>>>>>>>>>> >> >>> >> > believe that it's very developers/customers
>>>>>>>>>>>> friendly. In
>>>>>>>>>>>> >> >>> >> > addition,
>>>>>>>>>>>> >> >>> >> > we
>>>>>>>>>>>> >> >>> >> > can easily review the translated messages on the
>>>>>>>>>>>> github pull
>>>>>>>>>>>> >> >>> >> > request
>>>>>>>>>>>> >> >>> >> > view and so on. What do you think?
>>>>>>>>>>>> >> >>> >> >
>>>>>>>>>>>> >> >>> >> > If it's ok, I'll create a JIRA issue and create a
>>>>>>>>>>>> pull request.
>>>>>>>>>>>> >> >>> >> >
>>>>>>>>>>>> >> >>> >> > Regards,
>>>>>>>>>>>> >> >>> >> >
>>>>>>>>>>>> >> >>> >> > --
>>>>>>>>>>>> >> >>> >> > Hiroyuki Wada,
>>>>>>>>>>>> >> >>> >> > Developer,
>>>>>>>>>>>> >> >>> >> > Nomura Research Institute, Ltd.
>>>>>>>>>>>> >> >>> >> > _______________________________________________
>>>>>>>>>>>> >> >>> >> > keycloak-dev mailing list
>>>>>>>>>>>> >> >>> >> > keycloak-dev at lists.jboss.org
>>>>>>>>>>>> >> >>> >> >
>>>>>>>>>>>> https://lists.jboss.org/mailman/listinfo/keycloak-dev
>>>>>>>>>>>> >> >>> >>
>>>>>>>>>>>> >> >>> >> --
>>>>>>>>>>>> >> >>> >>
>>>>>>>>>>>> >> >>> >> abstractj
>>>>>>>>>>>> >> >>> >> PGP: 0x84DC9914
>>>>>>>>>>>> >> >>> >> _______________________________________________
>>>>>>>>>>>> >> >>> >> keycloak-dev mailing list
>>>>>>>>>>>> >> >>> >> keycloak-dev at lists.jboss.org
>>>>>>>>>>>> >> >>> >> https://lists.jboss.org/mailman/listinfo/keycloak-dev
>>>>>>>>>>>> >> >>> >
>>>>>>>>>>>> >> >>> >
>>>>>>>>>>>> >> >>
>>>>>>>>>>>> >> >>
>>>>>>>>>>>> >> >>
>>>>>>>>>>>> >> >> _______________________________________________
>>>>>>>>>>>> >> >> keycloak-dev mailing list
>>>>>>>>>>>> >> >> keycloak-dev at lists.jboss.org
>>>>>>>>>>>> >> >> https://lists.jboss.org/mailman/listinfo/keycloak-dev
>>>>>>>>>>>> >> >
>>>>>>>>>>>> >> >
>>>>>>>>>>>> >> >
>>>>>>>>>>>> >
>>>>>>>>>>>> >
>>>>>>>>>>>> >
>>>>>>>>>>>> >
>>>>>>>>>>>>
>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>> keycloak-dev mailing list
>>>>>>>>>>>> keycloak-dev at lists.jboss.org
>>>>>>>>>>>> https://lists.jboss.org/mailman/listinfo/keycloak-dev
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>
>>>>>>
>>>>
>>>
>> _______________________________________________
>> keycloak-dev mailing list
>> keycloak-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/keycloak-dev
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/keycloak-dev/attachments/20160715/de7fe675/attachment-0001.html

More information about the keycloak-dev mailing list