exceptions and locking policy
by Mircea Markus
Hi,
This is about what happens with locking and transactions when an exception appears during an invocation. It might be a e.g. TimeoutException or an CacheStore related exception or any type of exception.
Deadlock detection code relies on this policy, so I need to clearly define it.
Current logic is rather unclear and spreads over multiple interceptors:
- CallInterceptor would mark the tx for rollback if ant exception happen. This won't include TE as lock acquisition happen earlier in the call stack
- InvocationContextInterceptor releases locks on all keys associated with this call, but doesn't care if the call is in a tx or not so it doesn't rollback tx or release tx locks (this logic rather belongs in LockingInterceptor)
- TxInterceptor and LockInterceptor just ignore exceptions
As discussed in a previous email, the desired behaviour in case of a TimeoutException is to mark the tx for rollback. I think this can be extended as a rule for any type of Exception happening during a transaction - e.g. store exception, remote communication exception etc. On the short, if the user receives any kind of exception while writing in a tx (e.g. cache.put() throws an exception) he should know that tx was marked as rollback only. Wdyt?
Cheers,
Mircea
15 years, 10 months
Findings of ISPN-508 - Portable serialization marshaller
by Galder Zamarreño
Hi all,
Re: https://jira.jboss.org/browse/ISPN-508. This is a bit of a long one, so take your time :)
Over past few days I've been trying to come up with a Infinispan Marshaller implementation based on one of those portable serialization libraries (i.e. protobuf, thrift...etc) and I've reached the conclusion that it is possible to do so but requires a fair bit of work on our side and performance/usability would decrease. Before explaining my findings, let me explain the model that I'm trying to achieve:
I want to build a generic marshalling/unmarshalling implementation that's based on language neutral serialization libraries and which does not require the unmarshalling code to have knowledge of the target type. Think about Java Serialization or JBoss Marshalling where the code written can be generic enough w/o needing to know what we're trying to deserialize. The payload in these cases has enough information for the underlying library to figure out the type, instantiate it and populate it accordingly.
The problem is that there's no such library in the space of portable serialization libraries that provides this out of the box and let me explain why:
Pretty much any code using those libraries must use class type or schema name to deserialize the payload. However, Infinispan based Marshaller does not have such information when it's trying to deserialize the payload. Infinispan Marshaller based on JDK serialization or JBoss Marshalling uses information in the payload to instantiate the object. However, none of the libraries out there use this mechanism and instead some (i.e. Protobufs) force the client code to do things like: Pojo.parseFrom(byte[]) to generate instances of Pojo. In these cases for example, nothing stops you from writing a UTF-8 string with the class name and put it at the beginning of the payload so the deserialization part can be class agnostic, but this payload would not be portable. What would a python client do with a String containing the java class name?
Based on the FileDescriptorSet information in http://code.google.com/apis/protocolbuffers/docs/techniques.html I was able to hack something that might work in a portable way. Given a FileDescriptorSet generated a class compile time, I was able to match the protobuf name of a class with its java counter part. So, before writing the protobuf generated byte[], i prepend it with protobuf class name so that when reading, I can take the name, get the java class name and using reflection call parseFrom method to convert the byte[] into a pojo. Note that DynamicMessage class hinted in the the techniques page won't work cos it cannot create instances of pojos. It can only create generic objects with fields that are accessed in a reflection style.
I also looked at what Avro offers but it does not fully fit either. They have a reflection based serialization mechanism that doesn't require any precompilation, but it requires some kind of type knowledge on the client code to deserialize, plus Avro themselves recommend against it and I'm not sure how performant it'd be. Avro also includes other marshalling mechanisms called specific (like protobufs one with precompiled classes) and generic used to build dynamic objects on the fly. None of these two fit the bill. The specific one is like protobufs with the disadvantage of having ugly code like http://is.gd/dzwj3 where a static object has a strong reference to a <String,Class> CHM, which would leak in an AS env.
Thrift has the same problems as stated for Protobufs but coudn't see an equivalent way to get find the file descriptor set. Docu is way below what Protobuf offers and latest version which is 0.2 has issues generating classes as stated in JIRA.
MessagePack has the capability to deserialize an object given a String representation of the schema, so a solution like the protobuf one might be hackable. However, the generated classes do not have an equals implementation (??) which is rather odd. Maybe it's due to lack of maturity? Latest version is 0.3, so that might explain it. API wise, MessagePack provides the API that suits best to what we'd want to do and avoids having to use reflection to resolve the payload. However, looking around I couldn't see similar API for the python language for example and similar to Protobuf, we'd have to prepend the schema name key to the payload to then have the reading part lookup the entire schema based on it.
So, I can see two solutions here bearing in mind that pretty much any solution would require precompiling some classes:
- Try to build a marshaller using Protobuf or MessagePack where we enhance it to pass a string key that permits the reading part to deserialize the payload in a generic way.
- Or try to build some wikis on how to integrate Protobuf/Thrift/MessagePack with Hot Rod client so that they can generate the byte[] with these libraries and pass it to the corresponding Hot Rod client. For the moment, we'd do this for the Hot Rod java client. We would add more info once other language clients are available. The reason I said about potentially showing various libraries is cos whereas Protobuf only supports Java/Python/C++, Thrift supports loads more languages ( C++, Java, Python, PHP, Ruby, Erlang, Perl, Haskell, C#, Cocoa, Smalltalk, and OCaml)
In terms of usability , I think the second option is a bit better because building the marshaller requires steps for clients to generate the FileDescriptorSet and somehow pass this to Infinispan so that the marshaller can use it. The second option has none of this and the only added code on the client side is retrieving the byte[] and calling Pojo.parseFrom(byte[]). Performance wise, the 2nd option would be better cos there's no extra hacking needed to pass the String key around and so less bytes are sent around, and no need to use reflection to resolve the payload.
It's a pity that I couldn't find a tool that fully fits our use case and I wonder whether the need to deal with different languages makes coming up with such solution difficult. I believe reflection based instantiation is present in C++ or Python but not sure about other languages. It's also true that our use case is very specific where we're building a tool where we don't have control over what people will put in the cache. Generics could give us some hints for example but it's not mandatory and would not solve the issue entirely.
Thoughts?
--
Galder Zamarreño
Sr. Software Engineer
Infinispan, JBoss Cache
15 years, 11 months
Cutting 4.1.0.Final
by Manik Surtani
How do people feel about the status of branches/4.1.x? I think it's pretty stable and ready for 4.1.0.Final. Please let me know in the next few hours if there is any good reason to delay this a bit - otherwise I'm going to start the release process.
Cheers
Manik
--
Manik Surtani
manik(a)jboss.org
Lead, Infinispan
Lead, JBoss Cache
http://www.infinispan.org
http://www.jbosscache.org
15 years, 11 months
tx and lock acquisition timeouts
by Mircea Markus
Hi,
Doesn't seem to be documented what happens when a timeout exception is being thrown during a transaction: shall we mark it for rollback or shall we allow the user to continue working on it (retry perhaps).
Any idea on this? Not sure what database approach to this is either, perhaps we should follow the same.
Cheers,
Mircea
15 years, 11 months
eventually...
by Mircea Markus
Hi,
While testing, especially on async behaviour, one needs to wait until a certain state is reached.
E.g. let's say we want to test _async_ repl works fine, one way to do it is:
1. first approach
cache1.put(k,v);
//now expect this to be replicated at some point in future on cache2
for (int i=0; i < 10; i++) {
if (v.equals(cache2.get(k)) break;
Thread.sleep(1000);
}
assert v.equals(cache2.get(k));
the code bellow waits 10 seconds for the replication to occur, if not fails the test.
Problem with this code is that it is rather hard to read, and not reusable.
This code can be found in many paces in our tests.
2. second approach.
I've added AbstractInfinispanTest.eventually(Condition ec) method that does the same.
Code bellow now looks like:
eventually(new Condition() {
public boolean isSatisfied() throws Exception {
return v.equals(cache2.get(k));
}
});
While the amount of code itself hasn't change significantly, the second approach is more readable and much more quicker to write as it is generated by the IDE.
I've added this in 4.2, let me know your thoughts around it.
Cheers,
Mircea
15 years, 11 months
Flag.CACHE_MODE_LOCAL and eager locking
by Mircea Markus
if one uses Flag.CACHE_MODE_LOCAL and eager locking is enabled shall we acquire remote locks or only local?
IMO local-only locking should happen ( not the case at this time).
Cheers,
Mircea
15 years, 11 months
command id duplicated
by Mircea Markus
Hi,
Looks like a command can uniquely be identified by two ids. E.g.
@Marshallable(externalizer = ReplicableCommandExternalizer.class, id = Ids.REMOVE_COMMAND)
public class RemoveCommand extends AbstractDataWriteCommand {
...
public static final byte COMMAND_ID = 10;
...
It has an Marshallable id and an command id. Can't we just use one of them? Ids.REMOVE_COMMAND is my favourite as this way one can see the ids for all commands in a single place.
Cheers,
Mircea
15 years, 11 months
