[keycloak-dev] Support UTF-8 encoding theme message properites

Thomas Raehalme thomas.raehalme at aitiofinland.com
Fri Jul 15 03:04:00 EDT 2016 

iconv has been a good tool to convert files from one encoding to another.

Best regards,
Thomas

On Jul 15, 2016 09:57, "Hiroyuki Wada" <wadahiro at gmail.com> wrote:

>
> BTW do you have a reliable way to convert the files? I've tried various
>> tools, but none worked for me.
>>
>
> Yes, we can use native2ascii command.
>
> Example:
> native2ascii -encoding UTF-8 -reverse <input> <output>
>
>
>
>
>> On 15 July 2016 at 08:14, Stian Thorgersen <sthorger at redhat.com> wrote:
>>
>>> I'd like the current messages to be converted by the people that have
>>> contributed them. They will be able to determine what encoding they prefer
>>> to use and also confirm that the conversion has been successful.
>>>
>>> On 15 July 2016 at 07:59, Hiroyuki Wada <wadahiro at gmail.com> wrote:
>>>
>>>> I mean I'd like to change the message files encoding only.
>>>> For example, admin-messages for Russian language is currently written
>>>> in ISO-8859-1 encoding.
>>>>
>>>>
>>>> https://github.com/keycloak/keycloak/blob/master/themes/src/main/resources/theme/base/admin/messages/admin-messages_ru.properties
>>>>
>>>> Can I convert the language messages except for English?
>>>>
>>>>
>>>> 2016年7月15日(金) 14:03 Stian Thorgersen <sthorger at redhat.com>:
>>>>
>>>>> No, we don't want the theme properties (and other properties files) to
>>>>> be utf-8. They should stick with the default encoding used by Java
>>>>> properties files.
>>>>>
>>>>> Messages are a special case, which is why we accepted it for them.
>>>>>
>>>>> On 14 July 2016 at 13:29, Hiroyuki Wada <wadahiro at gmail.com> wrote:
>>>>>
>>>>>> Thank you for merging the pull request. Could I send a pull request
>>>>>> converting to UTF-8 encoding against the theme properties?
>>>>>>
>>>>>> 2016年7月7日(木) 21:39 Hiroyuki Wada <wadahiro at gmail.com>:
>>>>>>
>>>>>>> Yes. I will send a pull request later!
>>>>>>>
>>>>>>> 2016年7月7日(木) 21:20 Stian Thorgersen <sthorger at redhat.com>:
>>>>>>>
>>>>>>>> Are you doing a PR as well?
>>>>>>>>
>>>>>>>> On 7 July 2016 at 12:33, Hiroyuki Wada <wadahiro at gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Ok. I created a JIRA issue.
>>>>>>>>> https://issues.jboss.org/browse/KEYCLOAK-3278
>>>>>>>>>
>>>>>>>>> 2016年7月6日(水) 19:36 Stian Thorgersen <sthorger at redhat.com>:
>>>>>>>>>
>>>>>>>>>> Let's go for it. Please create a JIRA and send PR. Please just
>>>>>>>>>> use "# encoding: utf-8" and not "# -*- coding: utf-8 -*-". The
>>>>>>>>>> latter is just weird if you ask me.
>>>>>>>>>>
>>>>>>>>>> Thomas: It's optional, defaults to iso-8859-1 and the header will
>>>>>>>>>> clearly state that a file uses a different encoding, so I disagree that
>>>>>>>>>> there will be any confusion.
>>>>>>>>>>
>>>>>>>>>> On 6 July 2016 at 12:30, Hiroyuki Wada <wadahiro at gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> On Wed, Jul 6, 2016 at 6:53 PM, Thomas Raehalme
>>>>>>>>>>> <thomas.raehalme at aitiofinland.com> wrote:
>>>>>>>>>>> > From the javadoc of java.util.ResourceBundle the encoding for
>>>>>>>>>>> .properties is
>>>>>>>>>>> > still expected to be ISO-8859-1:
>>>>>>>>>>> >
>>>>>>>>>>> >> Constructing a PropertyResourceBundle instance from an
>>>>>>>>>>> InputStream
>>>>>>>>>>> >> requires that the input stream be encoded in ISO-8859-1. In
>>>>>>>>>>> that case,
>>>>>>>>>>> >> characters that cannot be represented in ISO-8859-1 encoding
>>>>>>>>>>> must be
>>>>>>>>>>> >> represented by Unicode Escapes as defined in section 3.3 of
>>>>>>>>>>> The Java™
>>>>>>>>>>> >> Language Specification whereas the other constructor which
>>>>>>>>>>> takes a Reader
>>>>>>>>>>> >> does not have that limitation.
>>>>>>>>>>>
>>>>>>>>>>> It's a case of using constructor which takes an InputStream.
>>>>>>>>>>> It says we can use any encoding when we use the other constructor
>>>>>>>>>>> which takes a Reader.
>>>>>>>>>>>
>>>>>>>>>>> I understand your concern. So what do you think supporting utf-8
>>>>>>>>>>> with
>>>>>>>>>>> a header like Stian suggested?
>>>>>>>>>>> I think it can avoid confusion because the encoding is noted in
>>>>>>>>>>> the
>>>>>>>>>>> file itself...
>>>>>>>>>>>
>>>>>>>>>>> Regards,
>>>>>>>>>>>
>>>>>>>>>>> >
>>>>>>>>>>> https://docs.oracle.com/javase/8/docs/api/java/util/PropertyResourceBundle.html
>>>>>>>>>>> >
>>>>>>>>>>> > I understand that Spring allows one to read .properties using
>>>>>>>>>>> any encoding,
>>>>>>>>>>> > but you need to specify the encoding in Spring configuration
>>>>>>>>>>> external to the
>>>>>>>>>>> > .properties file.
>>>>>>>>>>> >
>>>>>>>>>>> > In my opinion it would cause unnecessary confusion amongst
>>>>>>>>>>> developers to use
>>>>>>>>>>> > any other encoding than the one defined by the official
>>>>>>>>>>> documentation for
>>>>>>>>>>> > Properties.
>>>>>>>>>>> >
>>>>>>>>>>> > Best regards,
>>>>>>>>>>> > Thomas
>>>>>>>>>>> >
>>>>>>>>>>> >
>>>>>>>>>>> > On Wed, Jul 6, 2016 at 12:40 PM, Hiroyuki Wada <
>>>>>>>>>>> wadahiro at gmail.com> wrote:
>>>>>>>>>>> >>
>>>>>>>>>>> >> I think it was true before Java 1.6 because Java standard
>>>>>>>>>>> library
>>>>>>>>>>> >> (java.util.ResourceBundle or java.util.Properties) only
>>>>>>>>>>> supported
>>>>>>>>>>> >> ISO-8859-1 encoding file.
>>>>>>>>>>> >> But they support any encoding after Java 1.6. There are some
>>>>>>>>>>> >> frameworks which can read .properties with any encoding. For
>>>>>>>>>>> example,
>>>>>>>>>>> >> Spring Framework can read .properties with any encoding. I
>>>>>>>>>>> think it's
>>>>>>>>>>> >> very useful to read UTF-8 directly for multibyte language
>>>>>>>>>>> country.
>>>>>>>>>>> >>
>>>>>>>>>>> >> On Wed, Jul 6, 2016 at 3:32 PM, Thomas Raehalme
>>>>>>>>>>> >> <thomas.raehalme at aitiofinland.com> wrote:
>>>>>>>>>>> >> > Please correct me if I am wrong, but I have been under the
>>>>>>>>>>> impression
>>>>>>>>>>> >> > that
>>>>>>>>>>> >> > Java .properties files should always use encoding
>>>>>>>>>>> ISO-8859-1. Characters
>>>>>>>>>>> >> > not
>>>>>>>>>>> >> > present in ISO-8859-1 can be written in \uxxxx. Won't it
>>>>>>>>>>> make things
>>>>>>>>>>> >> > confusing to developers if another encoding is used here
>>>>>>>>>>> instead?
>>>>>>>>>>> >> >
>>>>>>>>>>> >> >
>>>>>>>>>>> >> >
>>>>>>>>>>> https://docs.oracle.com/javase/8/docs/api/java/util/Properties.html#load-java.io.InputStream-
>>>>>>>>>>> >> >
>>>>>>>>>>> >> >
>>>>>>>>>>> https://docs.oracle.com/javase/8/docs/api/java/util/Properties.html#store-java.io.OutputStream-java.lang.String-
>>>>>>>>>>> >> >
>>>>>>>>>>> >> > If alternate encodings are desired how about supporting the
>>>>>>>>>>> XML format
>>>>>>>>>>> >> > of
>>>>>>>>>>> >> > Properties?
>>>>>>>>>>> >> >
>>>>>>>>>>> >> > Best regards,
>>>>>>>>>>> >> > Thomas
>>>>>>>>>>> >> >
>>>>>>>>>>> >> >
>>>>>>>>>>> >> > On Wed, Jul 6, 2016 at 9:04 AM, Stian Thorgersen <
>>>>>>>>>>> sthorger at redhat.com>
>>>>>>>>>>> >> > wrote:
>>>>>>>>>>> >> >>
>>>>>>>>>>> >> >> Both iso-8859-1 and utf-8 message bundles should be able
>>>>>>>>>>> to co-exist.
>>>>>>>>>>> >> >>
>>>>>>>>>>> >> >> We can allow specifying the encoding in a comment on the
>>>>>>>>>>> first line
>>>>>>>>>>> >> >> like
>>>>>>>>>>> >> >> this:
>>>>>>>>>>> >> >>
>>>>>>>>>>> >> >> # encoding=utf-8
>>>>>>>>>>> >> >> key=value
>>>>>>>>>>> >> >>
>>>>>>>>>>> >> >> # encoding=iso-8859-1
>>>>>>>>>>> >> >> key=value
>>>>>>>>>>> >> >>
>>>>>>>>>>> >> >> If the first line in the file doesn't contain the comment
>>>>>>>>>>> with the
>>>>>>>>>>> >> >> encoding then we should default to iso-8859-1 for backwards
>>>>>>>>>>> >> >> compatibility
>>>>>>>>>>> >> >>
>>>>>>>>>>> >> >> On 5 July 2016 at 09:49, Hiroyuki Wada <wadahiro at gmail.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>> >> >>>
>>>>>>>>>>> >> >>> Thanks for your comment.
>>>>>>>>>>> >> >>>
>>>>>>>>>>> >> >>> > If we want to change to utf-8 we'd still need to
>>>>>>>>>>> support iso.. for
>>>>>>>>>>> >> >>> > backwards compatibility.
>>>>>>>>>>> >> >>>
>>>>>>>>>>> >> >>> If we change to UTF-8, we can still read unicode
>>>>>>>>>>> codepoint like
>>>>>>>>>>> >> >>> '\u00e8'.
>>>>>>>>>>> >> >>> There is an incompatibility when non-ascii characters are
>>>>>>>>>>> used in
>>>>>>>>>>> >> >>> message properties.
>>>>>>>>>>> >> >>> The non-ascii characters are 0xA0 - 0xFF codes (please
>>>>>>>>>>> refer codepage
>>>>>>>>>>> >> >>> layout:
>>>>>>>>>>> https://en.wikipedia.org/wiki/ISO/IEC_8859-1#Codepage_layout )
>>>>>>>>>>> >> >>>
>>>>>>>>>>> >> >>> I think the non-ascii characters might be used in French
>>>>>>>>>>> messages like
>>>>>>>>>>> >> >>> 'à' so I agree to support ISO-8859-1 for backwards
>>>>>>>>>>> compatibility.
>>>>>>>>>>> >> >>> To support this, I think we can add a property like
>>>>>>>>>>> "messageEncoding"
>>>>>>>>>>> >> >>> in keycloak-server.json as below. Is it a good idea?
>>>>>>>>>>> >> >>>
>>>>>>>>>>> >> >>>     "theme": {
>>>>>>>>>>> >> >>>         "staticMaxAge": 2592000,
>>>>>>>>>>> >> >>>         "cacheTemplates": true,
>>>>>>>>>>> >> >>>         "cacheThemes": true,
>>>>>>>>>>> >> >>>         "messageEncoding": "ISO-8859-1",
>>>>>>>>>>> >> >>>         "folder": {
>>>>>>>>>>> >> >>>           "dir": "${jboss.home.dir}/themes"
>>>>>>>>>>> >> >>>         }
>>>>>>>>>>> >> >>>     },
>>>>>>>>>>> >> >>>
>>>>>>>>>>> >> >>> Regards,
>>>>>>>>>>> >> >>>
>>>>>>>>>>> >> >>> On Mon, Jul 4, 2016 at 10:33 PM, Stian Thorgersen
>>>>>>>>>>> >> >>> <sthorger at redhat.com>
>>>>>>>>>>> >> >>> wrote:
>>>>>>>>>>> >> >>> > We have in the past discussed this and decided to stick
>>>>>>>>>>> with
>>>>>>>>>>> >> >>> > ISO-8859-1.
>>>>>>>>>>> >> >>> > That was probably not the best idea though. If we want
>>>>>>>>>>> to change to
>>>>>>>>>>> >> >>> > utf-8
>>>>>>>>>>> >> >>> > we'd still need to support iso.. for backwards
>>>>>>>>>>> compatibility.
>>>>>>>>>>> >> >>> >
>>>>>>>>>>> >> >>> > On 4 July 2016 at 14:37, Bruno Oliveira <
>>>>>>>>>>> bruno at abstractj.org> wrote:
>>>>>>>>>>> >> >>> >>
>>>>>>>>>>> >> >>> >> It makes sense, maybe file a Jira associated with:
>>>>>>>>>>> >> >>> >> https://issues.jboss.org/browse/KEYCLOAK-3259 ?
>>>>>>>>>>> >> >>> >>
>>>>>>>>>>> >> >>> >> On 2016-07-04, Hiroyuki Wada wrote:
>>>>>>>>>>> >> >>> >> > Hello all,
>>>>>>>>>>> >> >>> >> >
>>>>>>>>>>> >> >>> >> > I am trying to translate all base theme messages to
>>>>>>>>>>> my country
>>>>>>>>>>> >> >>> >> > language, Japanese. And I'd like to contribute them.
>>>>>>>>>>> Before that
>>>>>>>>>>> >> >>> >> > work,
>>>>>>>>>>> >> >>> >> > I'd like to propose about the files encoding.
>>>>>>>>>>> >> >>> >> >
>>>>>>>>>>> >> >>> >> > Currently, the message files (*.properties) are
>>>>>>>>>>> loaded with
>>>>>>>>>>> >> >>> >> > ISO-8859-1
>>>>>>>>>>> >> >>> >> > encoding. Therefore, it is necessary to convert the
>>>>>>>>>>> files by
>>>>>>>>>>> >> >>> >> > 'native2ascii' command beforehand. However we can
>>>>>>>>>>> directly read
>>>>>>>>>>> >> >>> >> > the
>>>>>>>>>>> >> >>> >> > property files with UTF-8 encoding in java 1.6 or
>>>>>>>>>>> later because
>>>>>>>>>>> >> >>> >> > 'java.util.Properties#load(java.io.Reader)' method
>>>>>>>>>>> was introduced
>>>>>>>>>>> >> >>> >> > as
>>>>>>>>>>> >> >>> >> > below.
>>>>>>>>>>> >> >>> >> >
>>>>>>>>>>> >> >>> >> >
>>>>>>>>>>> >> >>> >> >
>>>>>>>>>>> >> >>> >> >
>>>>>>>>>>> >> >>> >> >
>>>>>>>>>>> http://docs.oracle.com/javase/6/docs/api/java/util/Properties.html#load(java.io.Reader)
>>>>>>>>>>> >> >>> >> >
>>>>>>>>>>> >> >>> >> > So, my proposal is supporting the message files with
>>>>>>>>>>> UTF-8
>>>>>>>>>>> >> >>> >> > encoding.
>>>>>>>>>>> >> >>> >> > I
>>>>>>>>>>> >> >>> >> > believe that it's very developers/customers
>>>>>>>>>>> friendly. In
>>>>>>>>>>> >> >>> >> > addition,
>>>>>>>>>>> >> >>> >> > we
>>>>>>>>>>> >> >>> >> > can easily review the translated messages on the
>>>>>>>>>>> github pull
>>>>>>>>>>> >> >>> >> > request
>>>>>>>>>>> >> >>> >> > view and so on. What do you think?
>>>>>>>>>>> >> >>> >> >
>>>>>>>>>>> >> >>> >> > If it's ok, I'll create a JIRA issue and create a
>>>>>>>>>>> pull request.
>>>>>>>>>>> >> >>> >> >
>>>>>>>>>>> >> >>> >> > Regards,
>>>>>>>>>>> >> >>> >> >
>>>>>>>>>>> >> >>> >> > --
>>>>>>>>>>> >> >>> >> > Hiroyuki Wada,
>>>>>>>>>>> >> >>> >> > Developer,
>>>>>>>>>>> >> >>> >> > Nomura Research Institute, Ltd.
>>>>>>>>>>> >> >>> >> > _______________________________________________
>>>>>>>>>>> >> >>> >> > keycloak-dev mailing list
>>>>>>>>>>> >> >>> >> > keycloak-dev at lists.jboss.org
>>>>>>>>>>> >> >>> >> >
>>>>>>>>>>> https://lists.jboss.org/mailman/listinfo/keycloak-dev
>>>>>>>>>>> >> >>> >>
>>>>>>>>>>> >> >>> >> --
>>>>>>>>>>> >> >>> >>
>>>>>>>>>>> >> >>> >> abstractj
>>>>>>>>>>> >> >>> >> PGP: 0x84DC9914
>>>>>>>>>>> >> >>> >> _______________________________________________
>>>>>>>>>>> >> >>> >> keycloak-dev mailing list
>>>>>>>>>>> >> >>> >> keycloak-dev at lists.jboss.org
>>>>>>>>>>> >> >>> >> https://lists.jboss.org/mailman/listinfo/keycloak-dev
>>>>>>>>>>> >> >>> >
>>>>>>>>>>> >> >>> >
>>>>>>>>>>> >> >>
>>>>>>>>>>> >> >>
>>>>>>>>>>> >> >>
>>>>>>>>>>> >> >> _______________________________________________
>>>>>>>>>>> >> >> keycloak-dev mailing list
>>>>>>>>>>> >> >> keycloak-dev at lists.jboss.org
>>>>>>>>>>> >> >> https://lists.jboss.org/mailman/listinfo/keycloak-dev
>>>>>>>>>>> >> >
>>>>>>>>>>> >> >
>>>>>>>>>>> >> >
>>>>>>>>>>> >
>>>>>>>>>>> >
>>>>>>>>>>> >
>>>>>>>>>>> >
>>>>>>>>>>>
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> keycloak-dev mailing list
>>>>>>>>>>> keycloak-dev at lists.jboss.org
>>>>>>>>>>> https://lists.jboss.org/mailman/listinfo/keycloak-dev
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>
>>>
>>
> _______________________________________________
> keycloak-dev mailing list
> keycloak-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/keycloak-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/keycloak-dev/attachments/20160715/37888ecb/attachment-0001.html

More information about the keycloak-dev mailing list