Re: [infinispan-dev] @Externalize in conjunction with Infinispan?
by Galder Zamarreño
On Mar 28, 2011, at 3:50 PM, David M. Lloyd wrote:
> On 03/28/2011 03:53 AM, Galder Zamarreño wrote:
>> Hey David,
>>
>> I'm gonna start looking into @Externalize this week and I was
>> wondering whether we'll be able to use it as is with Infinispan?
>>
>> I mean, we have our own Externalizer interface which then maps to a
>> JBMAR Writer class via an adapter. This adapter is what makes a
>> Writer use our own Externalizer.writeObject(). The reading part is
>> done via an implementation of ObjectTable.readObject() where we map
>> an Id that's written by each of our Externalizer implementations to
>> the corresponding Externalizer so that it can read the object.
>>
>> I get the feeling that the way we use JBMAR might not suit the way
>> you can ship externalizer classes around to unmarshall stuff?
>
> The Externalizer interface and the @Externalize annotation rely on a different mechanism than object tables. You set up the AnnotationClassExternalizerFactory as your class externalizer factory in the configuration and the annotation will be recognized.
>
>> Also, in JBMAR, how does the marshaller side know that it does not
>> need to send the externalizer class again? I ask this cos you'd only
>> want the Externalizer class to be sent once to the receiving end, and
>> only if it's not aware of it? Or does it do it for all nodes in the
>> first request?
>
> It always sends the externalizer instance once for each externalizer. If you send the same externalizer twice, it will of course use a very compact backreference. The size of this initial message is dominated by the length of the class name, unless for some reason you use a complex externalizer.
>
>> Finally, if we can't take advantage of JBMAR to do this for us, how
>> much work do you think it would take to implement something similar
>> at the Infinispan level?
>
> Well ultimately the problem boils down to, how will you communicate which externalizer to use between sender and recipient. Using JBMAR Externalizers is the most simple and flexible option, however writing the class name can consume substantially more space than, say, an integer or a short. That said, the cost disappears into larger messages. The downside of using an integer or short however is that you have to preallocate and/or coordinate the identifier space.
>
> So I guess I don't really have an easy answer. Externalizers are simple but use a little more space. Using numeric IDs are somewhat more difficult (especially for end-users) but use less space.
Thanks David, your reply made me see it clearly.
In fact, I think we'll end up using the two methods. For internal and framework developers, the current set up works nicely. No annotations, so they can abstract Infinispan externalizers easily and we have very small payloads which is very efficient. Remember that I originally tried your Externalizer interface but the end result were payloads that were a bit big for our liking. That's how I ended up with the current set up.
However, for the end user, or average Joe, your Externalizer might be more useful because it uses annotations and needs no pre-registration. So, much easier to use at first glance though some bigger payloads.
CC'ing the infinispan-dev list.
>
> --
> - DML
--
Galder Zamarreño
Sr. Software Engineer
Infinispan, JBoss Cache
13 years
distributed execution - invoking commands on self
by Vladimir Blagojevic
Hi,
I discovered a problem with distributed framework in cases where
Callables submitted for distributed execution contain mutable instance
fields. Just before Callable is dispersed across cluster it gets invoked
locally where instance fields of a submitted Callable can be possibly
mutated; Callable is in turn sent to remote nodes with mutated values
instead of "original" field values as submitted by user. For example,
consider pi approximation example from wiki [1]. If CircleTest Callable
declared insideCircleCount as an instance field instead of local field
of call method we would ultimately get a wrong final result.
Therefore we need to ensure that each Infinispan node gets an unmodified
instance of Callable for execution. I was looking at the possibility to
send command to self thus causing creation of a marshaled copy of a
Callable, pretty much like any other remote node receives it. However, I
was unable to invoke a command on self. Any way to do this? How about
invoking marshaling locally to create a Callable copy without going
through rpc invocation layer? I though about skipping master/self node
altogether but that's an easy way out;-)
Got suggestions, ideas?
Regards,
Vladimir
[1] http://community.jboss.org/wiki/InfinispanDistributedExecutionFramework
13 years
TxRecovery design update
by Mircea Markus
Hi,
I've updated tx recovery design document[1] after feedback received from Manik: the most relevant changed is that all recovery information is now stored in a local cache (user can configure it). This way the user is in control of the memory consumed by recovery, being able to passivate to disk this information when/if needed. And all this out-of-the box.
Cheers,
Mircea
[1] http://community.jboss.org/wiki/Transactionrecoverydesign
13 years
Spring Infinispan finished for now
by Olaf Bergner
Hello,
I've decided that Spring Infinispan is where it ought to be right now,
feature-wise and documentation-wise. There's obviously a lot that could
be done to improve it. Yet I don't deem it prudent to go any further as
long as the fine folks over in Spring land are pondering whether and in
what way to accept our offer. If you wish, you may have a look at what
I've done so far at
https://github.com/obergner/spring-infinispan
So, time permitting, I'm willing to tackle the next challenge and
thought about turning my attention to either
https://issues.jboss.org/browse/ISPN-78 - Large Object Support
or
https://issues.jboss.org/browse/ISPN-984 - Add ability to retrieve cache
names remotely in Hot Rod,
depending on which of the two you consider to be more pressing at the
moment. Of course, I'm willing to take on other tasks in case there's
something that needs to be done more urgently. What do you think?
Cheers,
Olaf
13 years
[ISPN-78] Alternative interface for writing large objects
by Olaf Bergner
I've started working on ISPN-78 - Large Object Support - closely
following Manik's design document
http://community.jboss.org/wiki/LargeObjectSupport. As a starting point
I'm currently trying to implement
OutputStream writeToKey(K key),
which, for the time being, I chose to declare on AdvancedCache rather
that on Cache proper.
While thinking about the implications, I stumbled upon a few questions
which may well be owing to my lack of knowledge about Infinispan's inner
workings.
1. OutputStream writeToKey(K key) implies that the interaction between
user code and Infinispan happens in the OutputStream returned. Contrary
to existing methods, there would be no well defined *single* point where
control passes from user code to Infinispan. Instead, a user would write
a few bytes, passing control to Infinispan. Infinispan would buffer
those bytes and return control to the user until a preconfigured chunk
size is reached, whereupon Infinispan would probably issue a more or
less standard request to store that chunk on some node. Rinse and repeat.
This is certainly doable but leaves me wondering where that proposed
ChunkingInterceptor might come into play. It is my current understanding
that interceptors, well, intercept commands, and in this scenario there
could not be such a PutLargeObjectCommand as, as I said, there is no
single point where control passes from user code to Infinispan. Instead,
chunking would have to be done directly within the CacheOutputStream
returned, and the only commands involved would be the more or less
standard PutKeyValueCommands mentioned above. In this scenario there
wouldn't be any PutLargeObjectCommand to encapsulate the whole process.
Does this make sense? If so, I'd prefer to have
void writeToKey(K key, InputStream largeObject)
instead. Thus, after calling this method control would be handed over to
Infinispan until that LargeObject is stored, and we could have indeed
have some PutLargeObject command to encapsulate the whole process.
2. For the mapping Key -> LargeObjectMetadata I would intuitively choose
to use a dedicated "system" cache. Is this the Infinispan way of doing
things? If so, where can I find some code to use as a template? If not,
what would be an alternative approach that is in keeping with
Infinispan's architecture?
3. The design suggests to use a fresh UUID as the key for each new
chunk. While this in all likelihood gives us a unique new key for each
chunk I currently fail to see how that guarantees that this key maps to
a node that is different from all the nodes already used to store chunks
of the same Large Object. But then again I know next to nothing about
Infinispan's constant hashing algorithm.
4. Finally, the problem regarding eager locking and transactions
mentioned in Manik's comment seems rather ... hairy. If we indeed forego
transactions readers of a key just being written shouldn't be affected
provided we write the LargeObjectMetadata object only after all chunks
have been written. But what about writers?
I hope this makes sense. Otherwise, don't hesitate to point out where I
went wrong and I will happily retreat to the drawing board.
Cheers,
Olaf
13 years
Emails on pull req comments
by Galder Zamarreño
Hi,
I've noticed that when someone sends a pull req, we all get an email. However, the moment someone comments, that comment is only received as email by the person who created the pull req. And if the person that created the pull req replies, only the other person that commented gets the email.
It'd be much better if all people that can push pull reqs could receive all emails about comments, cos even though you don't deal with the pull req comment, you might have something to say about someone's comment.
Thoughts? I'm not sure where this falls, whether in Manik's script or something that we have to configure in Github
Cheers,
--
Galder Zamarreño
Sr. Software Engineer
Infinispan, JBoss Cache
13 years
5.0.0.BETA2 this coming Friday?
by Galder Zamarreño
Hi all,
I'm currently targeting BETA2 for this Friday. How does this sound? I know some of you will be busy with JAX London.
Note that this is the last expected BETA, with CR1 planned for the week of 25th April.
Please note that I'll be in Miracle Open World this week so my work will be cut out.
Cheers,
--
Galder Zamarreño
Sr. Software Engineer
Infinispan, JBoss Cache
13 years
