6:43 a.m.
Hi,
Re: https://jira.jboss.org/jira/browse/ISPN-44
I've attached a patch to the JIRA containing a 1st implementation of a
JBoss Marshalling based marshalling layer for Infinispan. A few notes:
- JBMAR based marshalling is optional for the moment. To enable it,
configure the marshaller to be
org.infinispan.marshall.jboss.JBossMarshaller.
- I created a new profile that runs the testsuite with this marshaller.
To run, just do: mvn -Ptest-jbossmarshaller test.
- I've run the testsuite for core and there're no regressions with new
marshaller. I also wanted to test jdbc but the testsuite seems to need a
bit of tidying up or I need to preconfigure something (Tests run: 177,
Failures: 4, Errors: 0, Skipped: 164, Time elapsed: 0.631 sec <<<
FAILURE!). Regardless, jdbc uses a dummy marshaller so would be
pointless for the moment.
- Different type marshalling has been implemented using
org.jboss.marshalling.Externalizer implementations mapped from
HorizonMarshaller class. JBMAR does support Externalizable classes but I
figured it out later and wanted to touch as less of existing code as
possible. So, for Beta1, some of Externalizer implementations (except
JDK classes) will probably go in favour of Externalizable implementations.
- StateTransferManagerImpl uses marshaller in a different way to the
rest. It first opens an ObjectOutput and then makes multiple calls to
stream stuff. StateTransferManagerImpl was hardcoded to use OOS and OIS
so I had to add org.infinispan.marshall.Marshaller that would
start/finish these OO/OI implementations. This is to enable
JBossMarshaller to provide OO/OI implementations coming from JBMAR.
- I had one doubt about the scope of the marshaller. I gave it
@Scope(Scopes.GLOBAL) since it'd be something shared among all caches.
Now, I assume its @Stop method would be called when all caches are
finished and the CacheManager is stopped? Wanted to make sure nothing is
leaked!.
- Application level versioning (VAM) is not yet supported in JBMAR but
David is happy to listen for ideas. I'll address that for Beta1.
- For Beta1, I'd like David to have a look at what I've done and see if
he can review it and then do perf tests to see whether this new
implementation is faster than our current one. If it is, next step would
be to design a way for people to extend/modify via configuration magic
numbers for their classes, provide Externalizers for their own
classes...etc.
Cheers,
--
Galder Zamarreño
Sr. Software Maintenance Engineer
JBoss, a division of Red Hat
9:24 a.m.
On 04/10/2009 06:43 AM, Galder Zamarreno wrote:
- For Beta1, I'd like David to have a look at what I've done
and see if
he can review it...
Regarding externalizers - they are potentially slightly faster than
Externalizable due to the fact that reflection is not needed to construct
the instance. But, I'll leave the proof of that to the benchmarks. :-)
Also, org.infinispan.marshall.jboss.ExternalizerClassFactory#EXTERNALIZERS
is a Map<String, String> so you don't want to use a WeakHashMap.
Otherwise it looks correct from a JBoss Marshalling perspective. Moving
from Object*Stream to ObjectInput/Output where you can is a smart move in
my opinion.
- DML
11:52 a.m.
David M. Lloyd wrote:
On 04/10/2009 06:43 AM, Galder Zamarreno wrote:
> - For Beta1, I'd like David to have a look at what I've done and see
> if he can review it...
Regarding externalizers - they are potentially slightly faster than
Externalizable due to the fact that reflection is not needed to
construct the instance. But, I'll leave the proof of that to the
benchmarks. :-)
That was precisely one of the questions I had for you. Which of the two
options would be faster in your opinion. I'll definitely try to test
both options.
Also,
org.infinispan.marshall.jboss.ExternalizerClassFactory#EXTERNALIZERS is
a Map<String, String> so you don't want to use a WeakHashMap.
Indeed. I had copy pasted from the one with Class as key and forgot to
change it.
Otherwise it looks correct from a JBoss Marshalling perspective. Moving
from Object*Stream to ObjectInput/Output where you can is a smart move
in my opinion.
What's the reason your saying this?
- DML
Thanks for having a look to this!
--
Galder Zamarreño
Sr. Software Maintenance Engineer
JBoss, a division of Red Hat
12:15 p.m.
On 04/14/2009 11:52 AM, Galder Zamarreno wrote:
David M. Lloyd wrote:
> On 04/10/2009 06:43 AM, Galder Zamarreno wrote:
>> - For Beta1, I'd like David to have a look at what I've done and see
>> if he can review it...
>
> Regarding externalizers - they are potentially slightly faster than
> Externalizable due to the fact that reflection is not needed to
> construct the instance. But, I'll leave the proof of that to the
> benchmarks. :-)
That was precisely one of the questions I had for you. Which of the two
options would be faster in your opinion. I'll definitely try to test
both options.
It's really hard to say since it could even depend on the message size and
composition. I think for large messages with lots of repetitions of that
class, Exteralizers will be a little faster. For small messages with less
repetition, maybe not. The cost of sending the Externalizer instance might
outweigh the cost of the reflection invocations required to support
Externalizable. Of course if you use ObjectTable to mitigate this, then
that might tip the scale the other way.
Either one will be tons faster than Serializable though. :-)
> Otherwise it looks correct from a JBoss Marshalling perspective.
> Moving from Object*Stream to ObjectInput/Output where you can is a
> smart move in my opinion.
What's the reason your saying this?
Just because Object*Stream is a big clunky API with lots of extra, useless
junk, and instances of those classes tend to have a lot of overhead
associated with them. Using the interfaces also gives more flexibility;
and of course (Unm|M)arshaller extends Object(In|Out)put as well so you can
work with either mechanism.
- DML
2:16 a.m.
I suggest you also take a look at performance and the size of te
marshalled data, compared to the current marshalling code.
I recall we had a bad experience with JBoss Serialization, which blew up
our data
Galder Zamarreno wrote:
Hi,
Re: https://jira.jboss.org/jira/browse/ISPN-44
I've attached a patch to the JIRA containing a 1st implementation of a
JBoss Marshalling based marshalling layer for Infinispan. A few notes:
- JBMAR based marshalling is optional for the moment. To enable it,
configure the marshaller to be
org.infinispan.marshall.jboss.JBossMarshaller.
- I created a new profile that runs the testsuite with this
marshaller. To run, just do: mvn -Ptest-jbossmarshaller test.
- I've run the testsuite for core and there're no regressions with new
marshaller. I also wanted to test jdbc but the testsuite seems to need
a bit of tidying up or I need to preconfigure something (Tests run:
177, Failures: 4, Errors: 0, Skipped: 164, Time elapsed: 0.631 sec <<<
FAILURE!). Regardless, jdbc uses a dummy marshaller so would be
pointless for the moment.
- Different type marshalling has been implemented using
org.jboss.marshalling.Externalizer implementations mapped from
HorizonMarshaller class. JBMAR does support Externalizable classes but
I figured it out later and wanted to touch as less of existing code as
possible. So, for Beta1, some of Externalizer implementations (except
JDK classes) will probably go in favour of Externalizable
implementations.
- StateTransferManagerImpl uses marshaller in a different way to the
rest. It first opens an ObjectOutput and then makes multiple calls to
stream stuff. StateTransferManagerImpl was hardcoded to use OOS and
OIS so I had to add org.infinispan.marshall.Marshaller that would
start/finish these OO/OI implementations. This is to enable
JBossMarshaller to provide OO/OI implementations coming from JBMAR.
- I had one doubt about the scope of the marshaller. I gave it
@Scope(Scopes.GLOBAL) since it'd be something shared among all caches.
Now, I assume its @Stop method would be called when all caches are
finished and the CacheManager is stopped? Wanted to make sure nothing
is leaked!.
- Application level versioning (VAM) is not yet supported in JBMAR but
David is happy to listen for ideas. I'll address that for Beta1.
- For Beta1, I'd like David to have a look at what I've done and see
if he can review it and then do perf tests to see whether this new
implementation is faster than our current one. If it is, next step
would be to design a way for people to extend/modify via configuration
magic numbers for their classes, provide Externalizers for their own
classes...etc.
Cheers,
--
Bela Ban
Lead JGroups / Clustering Team
JBoss - a division of Red Hat
9:26 a.m.
New subject: First version of JBoss Marshalling based Marshaller
Remember that if there's a problem with data size, you can always use the
Serial implementation (this implementation implements the Java
Serialization spec so there won't be a size difference from what was there
before; my crappy benchmarks show that even Serial is roughly 2x faster
than Java Serialization for the small data sets that I measured with). But
the River implementation should be the same size or smaller (since many
common classes, like primitive wrappers, are collapsed down to a single
byte). Since you're using ClassTables it should be much smaller.
Of course if you really miss JBoss Serialization, Ron has a JBoss
Serialization implementation that will probably go in to 1.2.0.GA. :-)
- DML
On 04/11/2009 02:16 AM, Bela Ban wrote:
I suggest you also take a look at performance and the size of te
marshalled data, compared to the current marshalling code.
I recall we had a bad experience with JBoss Serialization, which blew up
our data
Galder Zamarreno wrote:
> Hi,
>
> Re: https://jira.jboss.org/jira/browse/ISPN-44
>
> I've attached a patch to the JIRA containing a 1st implementation of a
> JBoss Marshalling based marshalling layer for Infinispan. A few notes:
>
> - JBMAR based marshalling is optional for the moment. To enable it,
> configure the marshaller to be
> org.infinispan.marshall.jboss.JBossMarshaller.
>
> - I created a new profile that runs the testsuite with this
> marshaller. To run, just do: mvn -Ptest-jbossmarshaller test.
>
> - I've run the testsuite for core and there're no regressions with new
> marshaller. I also wanted to test jdbc but the testsuite seems to need
> a bit of tidying up or I need to preconfigure something (Tests run:
> 177, Failures: 4, Errors: 0, Skipped: 164, Time elapsed: 0.631 sec <<<
> FAILURE!). Regardless, jdbc uses a dummy marshaller so would be
> pointless for the moment.
>
> - Different type marshalling has been implemented using
> org.jboss.marshalling.Externalizer implementations mapped from
> HorizonMarshaller class. JBMAR does support Externalizable classes but
> I figured it out later and wanted to touch as less of existing code as
> possible. So, for Beta1, some of Externalizer implementations (except
> JDK classes) will probably go in favour of Externalizable
> implementations.
>
> - StateTransferManagerImpl uses marshaller in a different way to the
> rest. It first opens an ObjectOutput and then makes multiple calls to
> stream stuff. StateTransferManagerImpl was hardcoded to use OOS and
> OIS so I had to add org.infinispan.marshall.Marshaller that would
> start/finish these OO/OI implementations. This is to enable
> JBossMarshaller to provide OO/OI implementations coming from JBMAR.
>
> - I had one doubt about the scope of the marshaller. I gave it
> @Scope(Scopes.GLOBAL) since it'd be something shared among all caches.
> Now, I assume its @Stop method would be called when all caches are
> finished and the CacheManager is stopped? Wanted to make sure nothing
> is leaked!.
>
> - Application level versioning (VAM) is not yet supported in JBMAR but
> David is happy to listen for ideas. I'll address that for Beta1.
>
> - For Beta1, I'd like David to have a look at what I've done and see
> if he can review it and then do perf tests to see whether this new
> implementation is faster than our current one. If it is, next step
> would be to design a way for people to extend/modify via configuration
> magic numbers for their classes, provide Externalizers for their own
> classes...etc.
>
> Cheers,
6:20 a.m.
On 11 Apr 2009, at 08:16, Bela Ban wrote:
I suggest you also take a look at performance and the size of te
marshalled data, compared to the current marshalling code.
I recall we had a bad experience with JBoss Serialization, which
blew up our data
This is an important point. We need to make sure we aren't
unnecessarily inflating the number of bytes being transported,
although I seriously doubt we are. :-)
Perhaps a quick unit test to prove this? This marshaller [1] in our
test suite is a dummy marshaller that just uses java object streams.
Could be useful to compare resulting byte array sizes against.
[1]
http://fisheye.jboss.org/browse/Infinispan/trunk/core/src/test/java/org/i...
Cheers
--
Manik Surtani
manik(a)jboss.org
Lead, Infinispan
Lead, JBoss Cache
http://www.infinispan.org
http://www.jbosscache.org
11:54 a.m.
Manik Surtani wrote:
On 11 Apr 2009, at 08:16, Bela Ban wrote:
> I suggest you also take a look at performance and the size of te
> marshalled data, compared to the current marshalling code.
>
> I recall we had a bad experience with JBoss Serialization, which blew
> up our data
This is an important point. We need to make sure we aren't
unnecessarily inflating the number of bytes being transported, although
I seriously doubt we are. :-)
Perhaps a quick unit test to prove this? This marshaller [1] in our
test suite is a dummy marshaller that just uses java object streams.
Could be useful to compare resulting byte array sizes against.
Good idea. I could run the same smoke tests I have in
JBossMarshallerTest with MarshallerImpl and ObjectStreamMarshaller and
compare the sizes.
I'll see if I can get this done in time for alpha deadline tomorrow COB.
[1]
http://fisheye.jboss.org/browse/Infinispan/trunk/core/src/test/java/org/i...
Cheers
--
Manik Surtani
manik(a)jboss.org
Lead, Infinispan
Lead, JBoss Cache
http://www.infinispan.org
http://www.jbosscache.org
--
Galder Zamarreño
Sr. Software Maintenance Engineer
JBoss, a division of Red Hat
12:46 p.m.
New subject: First version of JBoss Marshalling based Marshaller
Galder Zamarreno wrote:
Manik Surtani wrote:
>
> On 11 Apr 2009, at 08:16, Bela Ban wrote:
>
>> I suggest you also take a look at performance and the size of te
>> marshalled data, compared to the current marshalling code.
>>
>> I recall we had a bad experience with JBoss Serialization, which blew
>> up our data
>
> This is an important point. We need to make sure we aren't
> unnecessarily inflating the number of bytes being transported,
> although I seriously doubt we are. :-)
>
> Perhaps a quick unit test to prove this? This marshaller [1] in our
> test suite is a dummy marshaller that just uses java object streams.
> Could be useful to compare resulting byte array sizes against.
Good idea. I could run the same smoke tests I have in
JBossMarshallerTest with MarshallerImpl and ObjectStreamMarshaller and
compare the sizes.
I'll see if I can get this done in time for alpha deadline tomorrow COB.
I don't think I'm gonna have time for this but I've created a subtask to
make sure I add it: https://jira.jboss.org/jira/browse/ISPN-59
>
> [1]
>
http://fisheye.jboss.org/browse/Infinispan/trunk/core/src/test/java/org/i...
>
>
> Cheers
> --
> Manik Surtani
> manik(a)jboss.org
> Lead, Infinispan
> Lead, JBoss Cache
> http://www.infinispan.org
> http://www.jbosscache.org
>
>
>
>
--
Galder Zamarreño
Sr. Software Maintenance Engineer
JBoss, a division of Red Hat
6:15 a.m.
Looks good, my comments inline:
On 10 Apr 2009, at 12:43, Galder Zamarreno wrote:
Hi,
Re: https://jira.jboss.org/jira/browse/ISPN-44
I've attached a patch to the JIRA containing a 1st implementation of
a JBoss Marshalling based marshalling layer for Infinispan. A few
notes:
- JBMAR based marshalling is optional for the moment. To enable it,
configure the marshaller to be
org.infinispan.marshall.jboss.JBossMarshaller.
- I created a new profile that runs the testsuite with this
marshaller. To run, just do: mvn -Ptest-jbossmarshaller test.
I am guessing this is just a temporary measure to compare the two
approaches to marshalling? In the end I don't want to ship with 2
different approaches to marshalling.
- I've run the testsuite for core and there're no regressions
with
new marshaller. I also wanted to test jdbc but the testsuite seems
to need a bit of tidying up or I need to preconfigure something
(Tests run: 177, Failures: 4, Errors: 0, Skipped: 164, Time elapsed:
0.631 sec <<< FAILURE!). Regardless, jdbc uses a dummy marshaller so
would be pointless for the moment.
Yes, ignore this, I believe Mircea is looking at this. And, as you
said, this uses a dummy marshaller so it should not affect you.
- Different type marshalling has been implemented using
org.jboss.marshalling.Externalizer implementations mapped from
HorizonMarshaller class. JBMAR does support Externalizable classes
but I figured it out later and wanted to touch as less of existing
code as possible. So, for Beta1, some of Externalizer
implementations (except JDK classes) will probably go in favour of
Externalizable implementations.
- StateTransferManagerImpl uses marshaller in a different way to the
rest. It first opens an ObjectOutput and then makes multiple calls
to stream stuff. StateTransferManagerImpl was hardcoded to use OOS
and OIS so I had to add org.infinispan.marshall.Marshaller that
would start/finish these OO/OI implementations. This is to enable
JBossMarshaller to provide OO/OI implementations coming from JBMAR.
- I had one doubt about the scope of the marshaller. I gave it
@Scope(Scopes.GLOBAL) since it'd be something shared among all
caches. Now, I assume its @Stop method would be called when all
caches are finished and the CacheManager is stopped? Wanted to make
sure nothing is leaked!.
Yes, your assumption is correct. The whole thing with scopes is that
2 separate component registries are created, one for global components
and one for each cache instance. And lifecycle (such as @Stop) is
controlled by the CR so even if you stop individual caches, as long as
the cache manager is running global components won't be stopped.
Global components are only stopped when the cache manager is stopped.
- Application level versioning (VAM) is not yet supported in JBMAR
but David is happy to listen for ideas. I'll address that for Beta1.
For now we could write our own version headers (as we do with the VAM).
- For Beta1, I'd like David to have a look at what I've done
and see
if he can review it and then do perf tests to see whether this new
implementation is faster than our current one. If it is, next step
would be to design a way for people to extend/modify via
configuration magic numbers for their classes, provide Externalizers
for their own classes...etc.
I know this is jumping the gun a little, but how does this play with
ISPN-42: Object stream pooling? That's an optimisation I would really
like to see in place as soon as possible. :-) Is this a trivial
"extra" on top of ISPN-44?
Cheers
--
Manik Surtani
manik(a)jboss.org
Lead, Infinispan
Lead, JBoss Cache
http://www.infinispan.org
http://www.jbosscache.org
12:02 p.m.
Manik Surtani wrote:
Looks good, my comments inline:
On 10 Apr 2009, at 12:43, Galder Zamarreno wrote:
> </snip>
>
> - I created a new profile that runs the testsuite with this
> marshaller. To run, just do: mvn -Ptest-jbossmarshaller test.
I am guessing this is just a temporary measure to compare the two
approaches to marshalling? In the end I don't want to ship with 2
different approaches to marshalling.
+1
It's just there to enable me to run the full testsuite with the
currently optional marshaller.
> </snip>
>
> - I had one doubt about the scope of the marshaller. I gave it
> @Scope(Scopes.GLOBAL) since it'd be something shared among all caches.
> Now, I assume its @Stop method would be called when all caches are
> finished and the CacheManager is stopped? Wanted to make sure nothing
> is leaked!.
Yes, your assumption is correct. The whole thing with scopes is that 2
separate component registries are created, one for global components and
one for each cache instance. And lifecycle (such as @Stop) is
controlled by the CR so even if you stop individual caches, as long as
the cache manager is running global components won't be stopped. Global
components are only stopped when the cache manager is stopped.
Excellent. Thanks for the clarification.
> </snip>
I know this is jumping the gun a little, but how does this play with
ISPN-42: Object stream pooling? That's an optimisation I would really
like to see in place as soon as possible. :-) Is this a trivial
"extra" on top of ISPN-44?
Since streams are now retrieved from a factory in JBoss Marshalling, I
need to see what it happens behind the scenes (David could give us a
hint here). He might be already doing pooling of the streams returned by
the factory. If he isn't, since he's already implemented the factory, it
might make more sense to implement it in JBMAR than in JBC? Otherwise,
we can always implement this ourselves. David, thoughts?
This is certainly on my plate for Beta1. My plan was to verify raw
performance before looking into this.
Cheers
--
Manik Surtani
manik(a)jboss.org
Lead, Infinispan
Lead, JBoss Cache
http://www.infinispan.org
http://www.jbosscache.org
--
Galder Zamarreño
Sr. Software Maintenance Engineer
JBoss, a division of Red Hat
12:10 p.m.
New subject: First version of JBoss Marshalling based Marshaller
On 04/14/2009 12:02 PM, Galder Zamarreno wrote:
> I know this is jumping the gun a little, but how does this play
with
> ISPN-42: Object stream pooling? That's an optimisation I would really
> like to see in place as soon as possible. :-) Is this a trivial
> "extra" on top of ISPN-44?
Since streams are now retrieved from a factory in JBoss Marshalling, I
need to see what it happens behind the scenes (David could give us a
hint here). He might be already doing pooling of the streams returned by
the factory. If he isn't, since he's already implemented the factory, it
might make more sense to implement it in JBMAR than in JBC? Otherwise,
we can always implement this ourselves. David, thoughts?
Nope, the marshallers aren't pooled by default because they're very
lightweight to create (compared to Object*Streams). You can however
implement pooling if you want to (the start/finish methods exist for this
purpose). The potential benefit to pooling Marshaller/Unmarshaller
instances is that if you have Serializable objects which have
readObject/writeObject methods, the special Object*Stream that is used to
support this API is cached on the instance. I haven't seen conclusive
evidence that pooling would improve performance overall though; because my
Object*Streams completely override the API, it doesn't have the same costs
that are normally associated with creating these objects (however there is
still a permission check involved).
- DML
2:15 p.m.
New subject: First version of JBoss Marshalling based Marshaller
On 14 Apr 2009, at 18:10, David M. Lloyd wrote:
On 04/14/2009 12:02 PM, Galder Zamarreno wrote:
>> I know this is jumping the gun a little, but how does this play
>> with ISPN-42: Object stream pooling? That's an optimisation I
>> would really like to see in place as soon as possible. :-) Is
>> this a trivial "extra" on top of ISPN-44?
> Since streams are now retrieved from a factory in JBoss
> Marshalling, I need to see what it happens behind the scenes (David
> could give us a hint here). He might be already doing pooling of
> the streams returned by the factory. If he isn't, since he's
> already implemented the factory, it might make more sense to
> implement it in JBMAR than in JBC? Otherwise, we can always
> implement this ourselves. David, thoughts?
Nope, the marshallers aren't pooled by default because they're very
lightweight to create (compared to Object*Streams). You can however
implement pooling if you want to (the start/finish methods exist for
this purpose). The potential benefit to pooling Marshaller/
Unmarshaller instances is that if you have Serializable objects
which have readObject/writeObject methods, the special Object*Stream
that is used to support this API is cached on the instance. I
haven't seen conclusive evidence that pooling would improve
performance overall though; because my Object*Streams completely
override the API, it doesn't have the same costs that are normally
associated with creating these objects (however there is still a
permission check involved).
Interesting. Maybe stream pooling will not be necessary at all then
if your streams are that much lighter/quicker to create.
--
Manik Surtani
manik(a)jboss.org
Lead, Infinispan
Lead, JBoss Cache
http://www.infinispan.org
http://www.jbosscache.org
5510
days inactive
5514
days old
infinispan-dev@lists.jboss.org
12 comments
4 participants
participants (4)
-
Bela Ban -
David M. Lloyd -
Galder Zamarreno -
Manik Surtani