[keycloak-dev] Support UTF-8 encoding theme message properites

Hiroyuki Wada wadahiro at gmail.com
Fri Jul 15 05:10:34 EDT 2016 

I tried native2ascii. It looks good.

native2ascii -encoding UTF-8 -reverse messages_ru.properties
messages_ru.properties

== Before
invalidPasswordMinLengthMessage=\u041D\u0435\u043A\u043E\u0440\u0440\u0435\u043A\u0442\u043D\u044B\u0439
\u043F\u0430\u0440\u043E\u043B\u044C:
\u043C\u0438\u043D\u0438\u043C\u0430\u043B\u044C\u043D\u0430\u044F
\u0434\u043B\u0438\u043D\u0430 {0}.

== After
invalidPasswordMinLengthMessage=Некорректный пароль: минимальная длина {0}.



2016年7月15日(金) 17:41 Stian Thorgersen <sthorger at redhat.com>:

> I tried iconv, but it doesn't do anything. Just outputs the iso-8859-1 as
> before:
>
> iconv -f iso-8859-1 -t utf-8
> ./themes/src/main/resources/theme/base/login/messages/messages_ru.properties
>
> On 15 July 2016 at 09:04, Thomas Raehalme <
> thomas.raehalme at aitiofinland.com> wrote:
>
>> iconv has been a good tool to convert files from one encoding to another.
>>
>> Best regards,
>> Thomas
>>
>> On Jul 15, 2016 09:57, "Hiroyuki Wada" <wadahiro at gmail.com> wrote:
>>
>>>
>>> BTW do you have a reliable way to convert the files? I've tried various
>>>> tools, but none worked for me.
>>>>
>>>
>>> Yes, we can use native2ascii command.
>>>
>>> Example:
>>> native2ascii -encoding UTF-8 -reverse <input> <output>
>>>
>>>
>>>
>>>
>>>> On 15 July 2016 at 08:14, Stian Thorgersen <sthorger at redhat.com> wrote:
>>>>
>>>>> I'd like the current messages to be converted by the people that have
>>>>> contributed them. They will be able to determine what encoding they prefer
>>>>> to use and also confirm that the conversion has been successful.
>>>>>
>>>>> On 15 July 2016 at 07:59, Hiroyuki Wada <wadahiro at gmail.com> wrote:
>>>>>
>>>>>> I mean I'd like to change the message files encoding only.
>>>>>> For example, admin-messages for Russian language is currently written
>>>>>> in ISO-8859-1 encoding.
>>>>>>
>>>>>>
>>>>>> https://github.com/keycloak/keycloak/blob/master/themes/src/main/resources/theme/base/admin/messages/admin-messages_ru.properties
>>>>>>
>>>>>> Can I convert the language messages except for English?
>>>>>>
>>>>>>
>>>>>> 2016年7月15日(金) 14:03 Stian Thorgersen <sthorger at redhat.com>:
>>>>>>
>>>>>>> No, we don't want the theme properties (and other properties files)
>>>>>>> to be utf-8. They should stick with the default encoding used by Java
>>>>>>> properties files.
>>>>>>>
>>>>>>> Messages are a special case, which is why we accepted it for them.
>>>>>>>
>>>>>>> On 14 July 2016 at 13:29, Hiroyuki Wada <wadahiro at gmail.com> wrote:
>>>>>>>
>>>>>>>> Thank you for merging the pull request. Could I send a pull request
>>>>>>>> converting to UTF-8 encoding against the theme properties?
>>>>>>>>
>>>>>>>> 2016年7月7日(木) 21:39 Hiroyuki Wada <wadahiro at gmail.com>:
>>>>>>>>
>>>>>>>>> Yes. I will send a pull request later!
>>>>>>>>>
>>>>>>>>> 2016年7月7日(木) 21:20 Stian Thorgersen <sthorger at redhat.com>:
>>>>>>>>>
>>>>>>>>>> Are you doing a PR as well?
>>>>>>>>>>
>>>>>>>>>> On 7 July 2016 at 12:33, Hiroyuki Wada <wadahiro at gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Ok. I created a JIRA issue.
>>>>>>>>>>> https://issues.jboss.org/browse/KEYCLOAK-3278
>>>>>>>>>>>
>>>>>>>>>>> 2016年7月6日(水) 19:36 Stian Thorgersen <sthorger at redhat.com>:
>>>>>>>>>>>
>>>>>>>>>>>> Let's go for it. Please create a JIRA and send PR. Please just
>>>>>>>>>>>> use "# encoding: utf-8" and not "# -*- coding: utf-8 -*-". The
>>>>>>>>>>>> latter is just weird if you ask me.
>>>>>>>>>>>>
>>>>>>>>>>>> Thomas: It's optional, defaults to iso-8859-1 and the header
>>>>>>>>>>>> will clearly state that a file uses a different encoding, so I disagree
>>>>>>>>>>>> that there will be any confusion.
>>>>>>>>>>>>
>>>>>>>>>>>> On 6 July 2016 at 12:30, Hiroyuki Wada <wadahiro at gmail.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> On Wed, Jul 6, 2016 at 6:53 PM, Thomas Raehalme
>>>>>>>>>>>>> <thomas.raehalme at aitiofinland.com> wrote:
>>>>>>>>>>>>> > From the javadoc of java.util.ResourceBundle the encoding
>>>>>>>>>>>>> for .properties is
>>>>>>>>>>>>> > still expected to be ISO-8859-1:
>>>>>>>>>>>>> >
>>>>>>>>>>>>> >> Constructing a PropertyResourceBundle instance from an
>>>>>>>>>>>>> InputStream
>>>>>>>>>>>>> >> requires that the input stream be encoded in ISO-8859-1. In
>>>>>>>>>>>>> that case,
>>>>>>>>>>>>> >> characters that cannot be represented in ISO-8859-1
>>>>>>>>>>>>> encoding must be
>>>>>>>>>>>>> >> represented by Unicode Escapes as defined in section 3.3 of
>>>>>>>>>>>>> The Java™
>>>>>>>>>>>>> >> Language Specification whereas the other constructor which
>>>>>>>>>>>>> takes a Reader
>>>>>>>>>>>>> >> does not have that limitation.
>>>>>>>>>>>>>
>>>>>>>>>>>>> It's a case of using constructor which takes an InputStream.
>>>>>>>>>>>>> It says we can use any encoding when we use the other
>>>>>>>>>>>>> constructor
>>>>>>>>>>>>> which takes a Reader.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I understand your concern. So what do you think supporting
>>>>>>>>>>>>> utf-8 with
>>>>>>>>>>>>> a header like Stian suggested?
>>>>>>>>>>>>> I think it can avoid confusion because the encoding is noted
>>>>>>>>>>>>> in the
>>>>>>>>>>>>> file itself...
>>>>>>>>>>>>>
>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>
>>>>>>>>>>>>> >
>>>>>>>>>>>>> https://docs.oracle.com/javase/8/docs/api/java/util/PropertyResourceBundle.html
>>>>>>>>>>>>> >
>>>>>>>>>>>>> > I understand that Spring allows one to read .properties
>>>>>>>>>>>>> using any encoding,
>>>>>>>>>>>>> > but you need to specify the encoding in Spring configuration
>>>>>>>>>>>>> external to the
>>>>>>>>>>>>> > .properties file.
>>>>>>>>>>>>> >
>>>>>>>>>>>>> > In my opinion it would cause unnecessary confusion amongst
>>>>>>>>>>>>> developers to use
>>>>>>>>>>>>> > any other encoding than the one defined by the official
>>>>>>>>>>>>> documentation for
>>>>>>>>>>>>> > Properties.
>>>>>>>>>>>>> >
>>>>>>>>>>>>> > Best regards,
>>>>>>>>>>>>> > Thomas
>>>>>>>>>>>>> >
>>>>>>>>>>>>> >
>>>>>>>>>>>>> > On Wed, Jul 6, 2016 at 12:40 PM, Hiroyuki Wada <
>>>>>>>>>>>>> wadahiro at gmail.com> wrote:
>>>>>>>>>>>>> >>
>>>>>>>>>>>>> >> I think it was true before Java 1.6 because Java standard
>>>>>>>>>>>>> library
>>>>>>>>>>>>> >> (java.util.ResourceBundle or java.util.Properties) only
>>>>>>>>>>>>> supported
>>>>>>>>>>>>> >> ISO-8859-1 encoding file.
>>>>>>>>>>>>> >> But they support any encoding after Java 1.6. There are some
>>>>>>>>>>>>> >> frameworks which can read .properties with any encoding.
>>>>>>>>>>>>> For example,
>>>>>>>>>>>>> >> Spring Framework can read .properties with any encoding. I
>>>>>>>>>>>>> think it's
>>>>>>>>>>>>> >> very useful to read UTF-8 directly for multibyte language
>>>>>>>>>>>>> country.
>>>>>>>>>>>>> >>
>>>>>>>>>>>>> >> On Wed, Jul 6, 2016 at 3:32 PM, Thomas Raehalme
>>>>>>>>>>>>> >> <thomas.raehalme at aitiofinland.com> wrote:
>>>>>>>>>>>>> >> > Please correct me if I am wrong, but I have been under
>>>>>>>>>>>>> the impression
>>>>>>>>>>>>> >> > that
>>>>>>>>>>>>> >> > Java .properties files should always use encoding
>>>>>>>>>>>>> ISO-8859-1. Characters
>>>>>>>>>>>>> >> > not
>>>>>>>>>>>>> >> > present in ISO-8859-1 can be written in \uxxxx. Won't it
>>>>>>>>>>>>> make things
>>>>>>>>>>>>> >> > confusing to developers if another encoding is used here
>>>>>>>>>>>>> instead?
>>>>>>>>>>>>> >> >
>>>>>>>>>>>>> >> >
>>>>>>>>>>>>> >> >
>>>>>>>>>>>>> https://docs.oracle.com/javase/8/docs/api/java/util/Properties.html#load-java.io.InputStream-
>>>>>>>>>>>>> >> >
>>>>>>>>>>>>> >> >
>>>>>>>>>>>>> https://docs.oracle.com/javase/8/docs/api/java/util/Properties.html#store-java.io.OutputStream-java.lang.String-
>>>>>>>>>>>>> >> >
>>>>>>>>>>>>> >> > If alternate encodings are desired how about supporting
>>>>>>>>>>>>> the XML format
>>>>>>>>>>>>> >> > of
>>>>>>>>>>>>> >> > Properties?
>>>>>>>>>>>>> >> >
>>>>>>>>>>>>> >> > Best regards,
>>>>>>>>>>>>> >> > Thomas
>>>>>>>>>>>>> >> >
>>>>>>>>>>>>> >> >
>>>>>>>>>>>>> >> > On Wed, Jul 6, 2016 at 9:04 AM, Stian Thorgersen <
>>>>>>>>>>>>> sthorger at redhat.com>
>>>>>>>>>>>>> >> > wrote:
>>>>>>>>>>>>> >> >>
>>>>>>>>>>>>> >> >> Both iso-8859-1 and utf-8 message bundles should be able
>>>>>>>>>>>>> to co-exist.
>>>>>>>>>>>>> >> >>
>>>>>>>>>>>>> >> >> We can allow specifying the encoding in a comment on the
>>>>>>>>>>>>> first line
>>>>>>>>>>>>> >> >> like
>>>>>>>>>>>>> >> >> this:
>>>>>>>>>>>>> >> >>
>>>>>>>>>>>>> >> >> # encoding=utf-8
>>>>>>>>>>>>> >> >> key=value
>>>>>>>>>>>>> >> >>
>>>>>>>>>>>>> >> >> # encoding=iso-8859-1
>>>>>>>>>>>>> >> >> key=value
>>>>>>>>>>>>> >> >>
>>>>>>>>>>>>> >> >> If the first line in the file doesn't contain the
>>>>>>>>>>>>> comment with the
>>>>>>>>>>>>> >> >> encoding then we should default to iso-8859-1 for
>>>>>>>>>>>>> backwards
>>>>>>>>>>>>> >> >> compatibility
>>>>>>>>>>>>> >> >>
>>>>>>>>>>>>> >> >> On 5 July 2016 at 09:49, Hiroyuki Wada <
>>>>>>>>>>>>> wadahiro at gmail.com> wrote:
>>>>>>>>>>>>> >> >>>
>>>>>>>>>>>>> >> >>> Thanks for your comment.
>>>>>>>>>>>>> >> >>>
>>>>>>>>>>>>> >> >>> > If we want to change to utf-8 we'd still need to
>>>>>>>>>>>>> support iso.. for
>>>>>>>>>>>>> >> >>> > backwards compatibility.
>>>>>>>>>>>>> >> >>>
>>>>>>>>>>>>> >> >>> If we change to UTF-8, we can still read unicode
>>>>>>>>>>>>> codepoint like
>>>>>>>>>>>>> >> >>> '\u00e8'.
>>>>>>>>>>>>> >> >>> There is an incompatibility when non-ascii characters
>>>>>>>>>>>>> are used in
>>>>>>>>>>>>> >> >>> message properties.
>>>>>>>>>>>>> >> >>> The non-ascii characters are 0xA0 - 0xFF codes (please
>>>>>>>>>>>>> refer codepage
>>>>>>>>>>>>> >> >>> layout:
>>>>>>>>>>>>> https://en.wikipedia.org/wiki/ISO/IEC_8859-1#Codepage_layout )
>>>>>>>>>>>>> >> >>>
>>>>>>>>>>>>> >> >>> I think the non-ascii characters might be used in
>>>>>>>>>>>>> French messages like
>>>>>>>>>>>>> >> >>> 'à' so I agree to support ISO-8859-1 for backwards
>>>>>>>>>>>>> compatibility.
>>>>>>>>>>>>> >> >>> To support this, I think we can add a property like
>>>>>>>>>>>>> "messageEncoding"
>>>>>>>>>>>>> >> >>> in keycloak-server.json as below. Is it a good idea?
>>>>>>>>>>>>> >> >>>
>>>>>>>>>>>>> >> >>>     "theme": {
>>>>>>>>>>>>> >> >>>         "staticMaxAge": 2592000,
>>>>>>>>>>>>> >> >>>         "cacheTemplates": true,
>>>>>>>>>>>>> >> >>>         "cacheThemes": true,
>>>>>>>>>>>>> >> >>>         "messageEncoding": "ISO-8859-1",
>>>>>>>>>>>>> >> >>>         "folder": {
>>>>>>>>>>>>> >> >>>           "dir": "${jboss.home.dir}/themes"
>>>>>>>>>>>>> >> >>>         }
>>>>>>>>>>>>> >> >>>     },
>>>>>>>>>>>>> >> >>>
>>>>>>>>>>>>> >> >>> Regards,
>>>>>>>>>>>>> >> >>>
>>>>>>>>>>>>> >> >>> On Mon, Jul 4, 2016 at 10:33 PM, Stian Thorgersen
>>>>>>>>>>>>> >> >>> <sthorger at redhat.com>
>>>>>>>>>>>>> >> >>> wrote:
>>>>>>>>>>>>> >> >>> > We have in the past discussed this and decided to
>>>>>>>>>>>>> stick with
>>>>>>>>>>>>> >> >>> > ISO-8859-1.
>>>>>>>>>>>>> >> >>> > That was probably not the best idea though. If we
>>>>>>>>>>>>> want to change to
>>>>>>>>>>>>> >> >>> > utf-8
>>>>>>>>>>>>> >> >>> > we'd still need to support iso.. for backwards
>>>>>>>>>>>>> compatibility.
>>>>>>>>>>>>> >> >>> >
>>>>>>>>>>>>> >> >>> > On 4 July 2016 at 14:37, Bruno Oliveira <
>>>>>>>>>>>>> bruno at abstractj.org> wrote:
>>>>>>>>>>>>> >> >>> >>
>>>>>>>>>>>>> >> >>> >> It makes sense, maybe file a Jira associated with:
>>>>>>>>>>>>> >> >>> >> https://issues.jboss.org/browse/KEYCLOAK-3259 ?
>>>>>>>>>>>>> >> >>> >>
>>>>>>>>>>>>> >> >>> >> On 2016-07-04, Hiroyuki Wada wrote:
>>>>>>>>>>>>> >> >>> >> > Hello all,
>>>>>>>>>>>>> >> >>> >> >
>>>>>>>>>>>>> >> >>> >> > I am trying to translate all base theme messages
>>>>>>>>>>>>> to my country
>>>>>>>>>>>>> >> >>> >> > language, Japanese. And I'd like to contribute
>>>>>>>>>>>>> them. Before that
>>>>>>>>>>>>> >> >>> >> > work,
>>>>>>>>>>>>> >> >>> >> > I'd like to propose about the files encoding.
>>>>>>>>>>>>> >> >>> >> >
>>>>>>>>>>>>> >> >>> >> > Currently, the message files (*.properties) are
>>>>>>>>>>>>> loaded with
>>>>>>>>>>>>> >> >>> >> > ISO-8859-1
>>>>>>>>>>>>> >> >>> >> > encoding. Therefore, it is necessary to convert
>>>>>>>>>>>>> the files by
>>>>>>>>>>>>> >> >>> >> > 'native2ascii' command beforehand. However we can
>>>>>>>>>>>>> directly read
>>>>>>>>>>>>> >> >>> >> > the
>>>>>>>>>>>>> >> >>> >> > property files with UTF-8 encoding in java 1.6 or
>>>>>>>>>>>>> later because
>>>>>>>>>>>>> >> >>> >> > 'java.util.Properties#load(java.io.Reader)' method
>>>>>>>>>>>>> was introduced
>>>>>>>>>>>>> >> >>> >> > as
>>>>>>>>>>>>> >> >>> >> > below.
>>>>>>>>>>>>> >> >>> >> >
>>>>>>>>>>>>> >> >>> >> >
>>>>>>>>>>>>> >> >>> >> >
>>>>>>>>>>>>> >> >>> >> >
>>>>>>>>>>>>> >> >>> >> >
>>>>>>>>>>>>> http://docs.oracle.com/javase/6/docs/api/java/util/Properties.html#load(java.io.Reader)
>>>>>>>>>>>>> >> >>> >> >
>>>>>>>>>>>>> >> >>> >> > So, my proposal is supporting the message files
>>>>>>>>>>>>> with UTF-8
>>>>>>>>>>>>> >> >>> >> > encoding.
>>>>>>>>>>>>> >> >>> >> > I
>>>>>>>>>>>>> >> >>> >> > believe that it's very developers/customers
>>>>>>>>>>>>> friendly. In
>>>>>>>>>>>>> >> >>> >> > addition,
>>>>>>>>>>>>> >> >>> >> > we
>>>>>>>>>>>>> >> >>> >> > can easily review the translated messages on the
>>>>>>>>>>>>> github pull
>>>>>>>>>>>>> >> >>> >> > request
>>>>>>>>>>>>> >> >>> >> > view and so on. What do you think?
>>>>>>>>>>>>> >> >>> >> >
>>>>>>>>>>>>> >> >>> >> > If it's ok, I'll create a JIRA issue and create a
>>>>>>>>>>>>> pull request.
>>>>>>>>>>>>> >> >>> >> >
>>>>>>>>>>>>> >> >>> >> > Regards,
>>>>>>>>>>>>> >> >>> >> >
>>>>>>>>>>>>> >> >>> >> > --
>>>>>>>>>>>>> >> >>> >> > Hiroyuki Wada,
>>>>>>>>>>>>> >> >>> >> > Developer,
>>>>>>>>>>>>> >> >>> >> > Nomura Research Institute, Ltd.
>>>>>>>>>>>>> >> >>> >> > _______________________________________________
>>>>>>>>>>>>> >> >>> >> > keycloak-dev mailing list
>>>>>>>>>>>>> >> >>> >> > keycloak-dev at lists.jboss.org
>>>>>>>>>>>>> >> >>> >> >
>>>>>>>>>>>>> https://lists.jboss.org/mailman/listinfo/keycloak-dev
>>>>>>>>>>>>> >> >>> >>
>>>>>>>>>>>>> >> >>> >> --
>>>>>>>>>>>>> >> >>> >>
>>>>>>>>>>>>> >> >>> >> abstractj
>>>>>>>>>>>>> >> >>> >> PGP: 0x84DC9914
>>>>>>>>>>>>> >> >>> >> _______________________________________________
>>>>>>>>>>>>> >> >>> >> keycloak-dev mailing list
>>>>>>>>>>>>> >> >>> >> keycloak-dev at lists.jboss.org
>>>>>>>>>>>>> >> >>> >>
>>>>>>>>>>>>> https://lists.jboss.org/mailman/listinfo/keycloak-dev
>>>>>>>>>>>>> >> >>> >
>>>>>>>>>>>>> >> >>> >
>>>>>>>>>>>>> >> >>
>>>>>>>>>>>>> >> >>
>>>>>>>>>>>>> >> >>
>>>>>>>>>>>>> >> >> _______________________________________________
>>>>>>>>>>>>> >> >> keycloak-dev mailing list
>>>>>>>>>>>>> >> >> keycloak-dev at lists.jboss.org
>>>>>>>>>>>>> >> >> https://lists.jboss.org/mailman/listinfo/keycloak-dev
>>>>>>>>>>>>> >> >
>>>>>>>>>>>>> >> >
>>>>>>>>>>>>> >> >
>>>>>>>>>>>>> >
>>>>>>>>>>>>> >
>>>>>>>>>>>>> >
>>>>>>>>>>>>> >
>>>>>>>>>>>>>
>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>> keycloak-dev mailing list
>>>>>>>>>>>>> keycloak-dev at lists.jboss.org
>>>>>>>>>>>>> https://lists.jboss.org/mailman/listinfo/keycloak-dev
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>
>>>>>
>>>>
>>> _______________________________________________
>>> keycloak-dev mailing list
>>> keycloak-dev at lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/keycloak-dev
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/keycloak-dev/attachments/20160715/a7c43755/attachment-0001.html

More information about the keycloak-dev mailing list