Hi Galder, this seems great, I plan to try out soon implementing the ones for Lucene for a little extra performance boost. Just to double-check, no annotations are required on the externalized classes right? That would be a showstopper for example in the JIRA case as I don't have control over all marshalled source code.

The lucene-infinispan module will definitely use it, do you think there's a way to make it as simple as possible to have the ones I'll write registered for the end user? Maybe in a future release, could we add an SPI style discovery to self-register externalizers found on classpath?

And in case we have something like that in place, how will we guarantee ID unique assignments and also consistency across the cluster; as a starter should I reserve a range of identifiers in core, even without introducing dependencies; maybe later we should have a consistency check at node join to verify the same types are registered.

Sanne 2010/12/7 Galder Zamarreño <galder(a)redhat.com&gt;: > Hi all, > > Re: https://jira.jboss.org/browse/ISPN-244 > > This is trunk now and I've written preliminary documentation for it in http://community.jboss.org/docs/DOC-16198. Have a read through it and let me know what you think. > > Now, there's a couple of improvements I'm considering doing even for alpha1: > > 1. Internally, Infinispan has been mapping Externalizer classes and the types they marshall via @Marshallable interface. I don't think this annotation is really usable as it is for users cos there can be situations where users might not be able to modify the class that they wanna serialize. However, they for sure should be able to modify the Externalizer implementation classes, so I was wondering of converting this annotation into an annotation that an Externalizer class takes and that specifies the type of class that it externalizes and its id. Example: > > class Person { > ... > } > > @Marshalls(typeClass=Person.class, id=768) > class PersonExternalizer Implements Externalizer<Person> { > ... > } > > This could be a mechanism that could be used in equal manner by both internal and user defined externalizers. We don't wanna be doing annotation scanning, so in the same fashion that for internal externalizers we list the type classes that can be extermalized, this could lead to a user configuration simplification, i.e: > > <global> > <serialization> > <externalizers> > <externalizer class="PersonExternalizer" /> > </externalizers> > </serialization> > </global> > > On startup we'd load the externalizers defined in configuration and search for the @Marshalls annotation which would give us the rest of info which is: typeClass and id. Thoughts?? I think this would make internal and user defined externalizers more homogeneous when it comes to their definition and would simplify configuration. > > Chaging @Marshallable to @Marshalls should be easy to do particularly cos this annotation was only used for internal purpouses. > > 2. The second improvement I had in mind is around programmatic configuration. As you can see in the docu, configuring programmatically user defined externalizers requires quite a bit of code. I was thinking of adding a helper method in global configuration that would hide all of that away. If you take in account my recommendations for the previous point, that would be limited to something along the lines of: > > globalCfg.addExternalizer(PersonExternalizer.class); > > Where the method would have a signature like: public void addExternalizer(Class<? externds Externalizer> externalizerClass); > > We could also have an alternative that uses reflection to load the class and add it: public void addExternalizer(String externalizerClassName); > > Cheers, > -- > Galder Zamarreño > Sr. Software Engineer > Infinispan, JBoss Cache > > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev > _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

All looks great.  +1 to annotating externalizers rather than the class being externalized, but how would you match them up?  When the marshalling framework sees Class clazz, would it try and match the classes using clazz.equals()?  We'd need to consider class loader implications... +1 to making the config easier too. On 7 Dec 2010, at 09:49, Galder Zamarreño wrote: > Hi all, > > Re: https://jira.jboss.org/browse/ISPN-244 > > This is trunk now and I've written preliminary documentation for it in http://community.jboss.org/docs/DOC-16198. Have a read through it and let me know what you think. > > Now, there's a couple of improvements I'm considering doing even for alpha1: > > 1. Internally, Infinispan has been mapping Externalizer classes and the types they marshall via @Marshallable interface. I don't think this annotation is really usable as it is for users cos there can be situations where users might not be able to modify the class that they wanna serialize. However, they for sure should be able to modify the Externalizer implementation classes, so I was wondering of converting this annotation into an annotation that an Externalizer class takes and that specifies the type of class that it externalizes and its id. Example: > > class Person { > ... > } > > @Marshalls(typeClass=Person.class, id=768) > class PersonExternalizer Implements Externalizer<Person> { > ... > } > > This could be a mechanism that could be used in equal manner by both internal and user defined externalizers. We don't wanna be doing annotation scanning, so in the same fashion that for internal externalizers we list the type classes that can be extermalized, this could lead to a user configuration simplification, i.e: > > <global> >  <serialization> >    <externalizers> >      <externalizer class="PersonExternalizer" /> >    </externalizers> >  </serialization> > </global> > > On startup we'd load the externalizers defined in configuration and search for the @Marshalls annotation which would give us the rest of info which is: typeClass and id. Thoughts?? I think this would make internal and user defined externalizers more homogeneous when it comes to their definition and would simplify configuration. > > Chaging @Marshallable to @Marshalls should be easy to do particularly cos this annotation was only used for internal purpouses. > > 2. The second improvement I had in mind is around programmatic configuration. As you can see in the docu, configuring programmatically user defined externalizers requires quite a bit of code. I was thinking of adding a helper method in global configuration that would hide all of that away. If you take in account my recommendations for the previous point, that would be limited to something along the lines of: > > globalCfg.addExternalizer(PersonExternalizer.class); > > Where the method would have a signature like: public void addExternalizer(Class<? externds Externalizer> externalizerClass); > > We could also have an alternative that uses reflection to load the class and add it: public void addExternalizer(String externalizerClassName); > > Cheers, > -- > Galder Zamarreño > Sr. Software Engineer > Infinispan, JBoss Cache > > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev -- Manik Surtani manik(a)jboss.org Lead, Infinispan Lead, JBoss Cache http://www.infinispan.org http://www.jbosscache.org _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

On Dec 7, 2010, at 7:16 PM, Sanne Grinovero wrote: > Hi, > Is the Clazz going to be matched with some degree of polymorphism? No, it'll be matched on equality, so no polymorphism there. > Or will I be able to use the same externaliser for multiple types, so > effectively having something like > > marshallable.setTypeClasses(Class... types) > > and > @Marshalls(typeClasses={Person.class}, id=768) That could certainly be handy. Good suggestion Sanne :). It's not uncommon to see the same Externalizer type being used for different classes. For example, within Infinispan you have: JDK_EXTERNALIZERS.put(HashMap.class.getName(), MapExternalizer.class.getName()); JDK_EXTERNALIZERS.put(TreeMap.class.getName(), MapExternalizer.class.getName());

Where MapExternalizer externalizes both HashMap and TreeMap. > > Cheers, > Sanne > > 2010/12/7 Manik Surtani <manik(a)jboss.org&gt;: >> All looks great. +1 to annotating externalizers rather than the class being externalized, but how would you match them up? When the marshalling framework sees Class clazz, would it try and match the classes using clazz.equals()? We'd need to consider class loader implications... >> >> +1 to making the config easier too. >> >> >> On 7 Dec 2010, at 09:49, Galder Zamarreño wrote: >> >>> Hi all, >>> >>> Re: https://jira.jboss.org/browse/ISPN-244 >>> >>> This is trunk now and I've written preliminary documentation for it in http://community.jboss.org/docs/DOC-16198. Have a read through it and let me know what you think. >>> >>> Now, there's a couple of improvements I'm considering doing even for alpha1: >>> >>> 1. Internally, Infinispan has been mapping Externalizer classes and the types they marshall via @Marshallable interface. I don't think this annotation is really usable as it is for users cos there can be situations where users might not be able to modify the class that they wanna serialize. However, they for sure should be able to modify the Externalizer implementation classes, so I was wondering of converting this annotation into an annotation that an Externalizer class takes and that specifies the type of class that it externalizes and its id. Example: >>> >>> class Person { >>> ... >>> } >>> >>> @Marshalls(typeClass=Person.class, id=768) >>> class PersonExternalizer Implements Externalizer<Person> { >>> ... >>> } >>> >>> This could be a mechanism that could be used in equal manner by both internal and user defined externalizers. We don't wanna be doing annotation scanning, so in the same fashion that for internal externalizers we list the type classes that can be extermalized, this could lead to a user configuration simplification, i.e: >>> >>> <global> >>> <serialization> >>> <externalizers> >>> <externalizer class="PersonExternalizer" /> >>> </externalizers> >>> </serialization> >>> </global> >>> >>> On startup we'd load the externalizers defined in configuration and search for the @Marshalls annotation which would give us the rest of info which is: typeClass and id. Thoughts?? I think this would make internal and user defined externalizers more homogeneous when it comes to their definition and would simplify configuration. >>> >>> Chaging @Marshallable to @Marshalls should be easy to do particularly cos this annotation was only used for internal purpouses. >>> >>> 2. The second improvement I had in mind is around programmatic configuration. As you can see in the docu, configuring programmatically user defined externalizers requires quite a bit of code. I was thinking of adding a helper method in global configuration that would hide all of that away. If you take in account my recommendations for the previous point, that would be limited to something along the lines of: >>> >>> globalCfg.addExternalizer(PersonExternalizer.class); >>> >>> Where the method would have a signature like: public void addExternalizer(Class<? externds Externalizer> externalizerClass); >>> >>> We could also have an alternative that uses reflection to load the class and add it: public void addExternalizer(String externalizerClassName); >>> >>> Cheers, >>> -- >>> Galder Zamarreño >>> Sr. Software Engineer >>> Infinispan, JBoss Cache >>> >>> >>> _______________________________________________ >>> infinispan-dev mailing list >>> infinispan-dev(a)lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >> >> -- >> Manik Surtani >> manik(a)jboss.org >> Lead, Infinispan >> Lead, JBoss Cache >> http://www.infinispan.org >> http://www.jbosscache.org >> >> >> >> >> >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev >> > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev -- Galder Zamarreño Sr. Software Engineer Infinispan, JBoss Cache _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

On Wed, Dec 8, 2010 at 1:05 PM, Mircea Markus <mircea.markus(a)jboss.com&gt; wrote: > Isn't this possible even now through XML? > <marshallable id="456" typeClass="com.acme.Person"  externalizerClass="com.acme.SameExternalizer"/> > <marshallable id="457" typeClass="com.acme.HotPerson"  externalizerClass="com.acme.SameExternalizer"/> There are a few things that came to mind here: 1. In terms of assigning IDs, it needs to be fairly clear there's no "global" conflict and having IDs at the annotation level makes it a bit unclear what actual number goes with what class. Additionally, Infinispan (and users themselves) might want to provide externalizers, and unless there's assigned ranges to those teams, putting the ID within the annotation it somewhat difficult to reuse these classes. How would a team provide a combined cache from two separate instances that happen to use different externalizers with the same ID? Still, having ranges reserved by Infinispan itself would make sense. If at all possible, I would restrict them to less than, say 64 or 32, thus allowing custom types to be passed as byte values. I feel having the IDs in one place also makes things easier to debug. For network analysis, it's helpful as a reference. 2. Can't the configuration be simply: <marshallable id="456" externalizerClass="com.acme.SameExternalizer"/> and then you simply call SameExternalizer.class.getTypeParameters()[0] to determine the type class? _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

On Dec 9, 2010, at 12:11 AM, Sanne Grinovero wrote: > I'm liking Elias's suggestion to define the ids in xml, at least I > would have a single place where I can clearly see > and inspect usage of my used ids; You can always have an interface with constants defining your IDs. That's what the Infinispan Core code does to allow a central view of the IDs that are allocated. I don't want Infinispan Core having to fiddle with GlobalConfiguration passed, or having to have some XML for internal works. I understand your point though from a users perspective, and so I'll allow both IDs to be defined via XML or annotation. If there's clashes (i.e. same ID points to diff externalizers), a WARN/ERROR (?) message will be shown, but the end winner, just like in the EE world, will be XML. > btw having the map logged in > readable format at startup would be nice. Both maps in operation will be logged on startup, but so far not doing anything fancy, just toString(). Any suggestion to what format you'd want to see? > However getTypeParameters()[0] seems to me insufficient as for the > requirement to reuse each externalizer for multiple cases. +1 > > I'd rather use Galder's original proposal to have type(s) and ids > provided by the class, but the ids would be a default only, > especially useful for the autodiscovered externalizers, and then let > the user override all ids in the XML if he wishes so. Exactly. That's the conclusion to which I arrived in my comment above. At the end of the day, I'm trying to use the same mechanism to define Infinispan Core externalizers and foreign, or user-defined, ones, so your suggestion is what IMO can fit both use cases at the same. > > Cheers, > Sanne > > 2010/12/8 Elias Ross <genman(a)noderunner.net&gt;: >> On Wed, Dec 8, 2010 at 1:05 PM, Mircea Markus <mircea.markus(a)jboss.com&gt; wrote: >>> Isn't this possible even now through XML? >>> <marshallable id="456" typeClass="com.acme.Person" externalizerClass="com.acme.SameExternalizer"/> >>> <marshallable id="457" typeClass="com.acme.HotPerson" externalizerClass="com.acme.SameExternalizer"/> >> >> There are a few things that came to mind here: >> >> 1. In terms of assigning IDs, it needs to be fairly clear there's no >> "global" conflict and having IDs at the annotation level makes it a >> bit unclear what actual number goes with what class. Additionally, >> Infinispan (and users themselves) might want to provide externalizers, >> and unless there's assigned ranges to those teams, putting the ID >> within the annotation it somewhat difficult to reuse these classes. >> How would a team provide a combined cache from two separate instances >> that happen to use different externalizers with the same ID? >> >> Still, having ranges reserved by Infinispan itself would make sense. >> If at all possible, I would restrict them to less than, say 64 or 32, >> thus allowing custom types to be passed as byte values. >> >> I feel having the IDs in one place also makes things easier to debug. >> For network analysis, it's helpful as a reference. >> >> 2. Can't the configuration be simply: >> >> <marshallable id="456" externalizerClass="com.acme.SameExternalizer"/> >> >> and then you simply call SameExternalizer.class.getTypeParameters()[0] >> to determine the type class? >> >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev >> > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev -- Galder Zamarreño Sr. Software Engineer Infinispan, JBoss Cache _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

On 8 Dec 2010, at 08:49, Galder Zamarreño wrote: > > On Dec 7, 2010, at 7:16 PM, Sanne Grinovero wrote: > >> Hi, >> Is the Clazz going to be matched with some degree of polymorphism? > > No, it'll be matched on equality, so no polymorphism there. > >> Or will I be able to use the same externaliser for multiple types, so >> effectively having something like >> >> marshallable.setTypeClasses(Class... types) >> >> and >> @Marshalls(typeClasses={Person.class}, id=768) > > That could certainly be handy. Good suggestion Sanne :). It's not uncommon to see the same Externalizer type being used for different classes. For example, within Infinispan you have: > > JDK_EXTERNALIZERS.put(HashMap.class.getName(), MapExternalizer.class.getName()); > JDK_EXTERNALIZERS.put(TreeMap.class.getName(), MapExternalizer.class.getName()); Isn't this possible even now through XML? <marshallable id="456" typeClass="com.acme.Person" externalizerClass="com.acme.SameExternalizer"/> <marshallable id="457" typeClass="com.acme.HotPerson" externalizerClass="com.acme.SameExternalizer"/>

> > Where MapExternalizer externalizes both HashMap and TreeMap. > >> >> Cheers, >> Sanne >> >> 2010/12/7 Manik Surtani <manik(a)jboss.org&gt;: >>> All looks great. +1 to annotating externalizers rather than the class being externalized, but how would you match them up? When the marshalling framework sees Class clazz, would it try and match the classes using clazz.equals()? We'd need to consider class loader implications... >>> >>> +1 to making the config easier too. >>> >>> >>> On 7 Dec 2010, at 09:49, Galder Zamarreño wrote: >>> >>>> Hi all, >>>> >>>> Re: https://jira.jboss.org/browse/ISPN-244 >>>> >>>> This is trunk now and I've written preliminary documentation for it in http://community.jboss.org/docs/DOC-16198. Have a read through it and let me know what you think. >>>> >>>> Now, there's a couple of improvements I'm considering doing even for alpha1: >>>> >>>> 1. Internally, Infinispan has been mapping Externalizer classes and the types they marshall via @Marshallable interface. I don't think this annotation is really usable as it is for users cos there can be situations where users might not be able to modify the class that they wanna serialize. However, they for sure should be able to modify the Externalizer implementation classes, so I was wondering of converting this annotation into an annotation that an Externalizer class takes and that specifies the type of class that it externalizes and its id. Example: >>>> >>>> class Person { >>>> ... >>>> } >>>> >>>> @Marshalls(typeClass=Person.class, id=768) >>>> class PersonExternalizer Implements Externalizer<Person> { >>>> ... >>>> } >>>> >>>> This could be a mechanism that could be used in equal manner by both internal and user defined externalizers. We don't wanna be doing annotation scanning, so in the same fashion that for internal externalizers we list the type classes that can be extermalized, this could lead to a user configuration simplification, i.e: >>>> >>>> <global> >>>> <serialization> >>>> <externalizers> >>>> <externalizer class="PersonExternalizer" /> >>>> </externalizers> >>>> </serialization> >>>> </global> >>>> >>>> On startup we'd load the externalizers defined in configuration and search for the @Marshalls annotation which would give us the rest of info which is: typeClass and id. Thoughts?? I think this would make internal and user defined externalizers more homogeneous when it comes to their definition and would simplify configuration. >>>> >>>> Chaging @Marshallable to @Marshalls should be easy to do particularly cos this annotation was only used for internal purpouses. >>>> >>>> 2. The second improvement I had in mind is around programmatic configuration. As you can see in the docu, configuring programmatically user defined externalizers requires quite a bit of code. I was thinking of adding a helper method in global configuration that would hide all of that away. If you take in account my recommendations for the previous point, that would be limited to something along the lines of: >>>> >>>> globalCfg.addExternalizer(PersonExternalizer.class); >>>> >>>> Where the method would have a signature like: public void addExternalizer(Class<? externds Externalizer> externalizerClass); >>>> >>>> We could also have an alternative that uses reflection to load the class and add it: public void addExternalizer(String externalizerClassName); >>>> >>>> Cheers, >>>> -- >>>> Galder Zamarreño >>>> Sr. Software Engineer >>>> Infinispan, JBoss Cache >>>> >>>> >>>> _______________________________________________ >>>> infinispan-dev mailing list >>>> infinispan-dev(a)lists.jboss.org >>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>> >>> -- >>> Manik Surtani >>> manik(a)jboss.org >>> Lead, Infinispan >>> Lead, JBoss Cache >>> http://www.infinispan.org >>> http://www.jbosscache.org >>> >>> >>> >>> >>> >>> _______________________________________________ >>> infinispan-dev mailing list >>> infinispan-dev(a)lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>> >> >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev > > -- > Galder Zamarreño > Sr. Software Engineer > Infinispan, JBoss Cache > > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev