Configuration visitor - Re: [JBoss JIRA] Commented: (ISPN-145) No transport and singleton store enabled should not be allowed
by Vladimir Blagojevic
Hi,
Galder and I talked about this offline. Time to involve you guys!
I just completed visitor pattern for our configuration objects. Visitor
is passed from root of configuration - InfinispanConfiguration object.
InfinispanConfiguration class has a new method:
public void accept(ConfigurationBeanVisitor v)
How do we want to integrate this visitor into existing structure?
1) We add a new factory method to InfinispanConfiguration with
additional ConfigurationBeanVisitor parameter
2) We leave everything as is and if there is a need to pass some visitor
we pass it to InfinispanConfiguration instance directly (from
DefaultCacheManager)
DefaultCacheManager will pass ValidationVisitor to
InfinispanConfiguration that will verify configuration semantically.
Regards,
Vladimir
On 09-09-09 10:19 AM, Galder Zamarreno wrote:
> Good idea :)
>
> On 09/09/2009 04:13 PM, Vladimir Blagojevic wrote:
>> Yeah,
>>
>> I was thinking that we can make a visitor for configuration tree and
>> then you can do verification of any node and other things as well. Use
>> cases will come up in the future for sure.
>>
>> Cheers
>>
>>
>>
>> On 09-09-09 3:29 AM, Galder Zamarreno (JIRA) wrote:
>>> [
>>> https://jira.jboss.org/jira/browse/ISPN-145?page=com.atlassian.jira.plugi...
>>>
>>> ]
>>>
>>> Galder Zamarreno commented on ISPN-145:
>>> ---------------------------------------
>>>
>>> Not sure I understand what you mean by generic though. You mean any
>>> component to have a validation step of some sort?
>>>
>>> Thanks for taking this on :)
>>>
>>>> No transport and singleton store enabled should not be allowed
>>>> --------------------------------------------------------------
>>>>
>>>> Key: ISPN-145
>>>> URL: https://jira.jboss.org/jira/browse/ISPN-145
>>>> Project: Infinispan
>>>> Issue Type: Bug
>>>> Components: Loaders and Stores
>>>> Affects Versions: 4.0.0.ALPHA6
>>>> Reporter: Galder Zamarreno
>>>> Assignee: Vladimir Blagojevic
>>>> Priority: Minor
>>>> Fix For: 4.0.0.CR1
>>>>
>>>>
>>>> Throw configuration exception if singleton store configured without
>>>> transport having been configured.
>>>> It makes no sense to have singleton store enabled when there's no
>>>> transport.
>>
>
12 years, 6 months
ISPN 200
by Israel Lacerra
Manik,
What you mean by:
" * The calling node returns a CacheQuery impl that lazily fetches
and collates results from the cluster." (JIRA)
Is enough if each node returns a list of keys and then, we lazily get the
values using the keys? Or the process has to be more lazy yet?
thanks!
Israel Lacerra
13 years, 3 months
Using Coverity scan?
by Sanne Grinovero
Hello,
Did you consider enabling Infinispan to be monitored by coverity's
code analysis services? They are free for OSS projects, I saw a demo
recently and was quite amazed. It's similar to FindBugs, but not only
about static code checks. They checkout your code from trunk and then
run several analysis on it periodically, one of them is about dynamic
thread behavior to predict deadlocks or missing fences instrumenting
the code, and produce nice public reports; AFAIK you don't need to
setup anything yourself, besides getting in touch to ask for it.
It's only available for C and Java code, and they have an impressive
list of OSS projects in the C world (linux kernel, httpd server,
samba, gnome, GCC, PostgreSQL, ...) but not much on Java.
http://scan.coverity.com/
No, I'm not affiliated :-) Just thinking that it might be useful to
have if it's not too hard to setup.
Cheers,
Sanne
13 years, 4 months
Findings of ISPN-508 - Portable serialization marshaller
by Galder Zamarreño
Hi all,
Re: https://jira.jboss.org/browse/ISPN-508. This is a bit of a long one, so take your time :)
Over past few days I've been trying to come up with a Infinispan Marshaller implementation based on one of those portable serialization libraries (i.e. protobuf, thrift...etc) and I've reached the conclusion that it is possible to do so but requires a fair bit of work on our side and performance/usability would decrease. Before explaining my findings, let me explain the model that I'm trying to achieve:
I want to build a generic marshalling/unmarshalling implementation that's based on language neutral serialization libraries and which does not require the unmarshalling code to have knowledge of the target type. Think about Java Serialization or JBoss Marshalling where the code written can be generic enough w/o needing to know what we're trying to deserialize. The payload in these cases has enough information for the underlying library to figure out the type, instantiate it and populate it accordingly.
The problem is that there's no such library in the space of portable serialization libraries that provides this out of the box and let me explain why:
Pretty much any code using those libraries must use class type or schema name to deserialize the payload. However, Infinispan based Marshaller does not have such information when it's trying to deserialize the payload. Infinispan Marshaller based on JDK serialization or JBoss Marshalling uses information in the payload to instantiate the object. However, none of the libraries out there use this mechanism and instead some (i.e. Protobufs) force the client code to do things like: Pojo.parseFrom(byte[]) to generate instances of Pojo. In these cases for example, nothing stops you from writing a UTF-8 string with the class name and put it at the beginning of the payload so the deserialization part can be class agnostic, but this payload would not be portable. What would a python client do with a String containing the java class name?
Based on the FileDescriptorSet information in http://code.google.com/apis/protocolbuffers/docs/techniques.html I was able to hack something that might work in a portable way. Given a FileDescriptorSet generated a class compile time, I was able to match the protobuf name of a class with its java counter part. So, before writing the protobuf generated byte[], i prepend it with protobuf class name so that when reading, I can take the name, get the java class name and using reflection call parseFrom method to convert the byte[] into a pojo. Note that DynamicMessage class hinted in the the techniques page won't work cos it cannot create instances of pojos. It can only create generic objects with fields that are accessed in a reflection style.
I also looked at what Avro offers but it does not fully fit either. They have a reflection based serialization mechanism that doesn't require any precompilation, but it requires some kind of type knowledge on the client code to deserialize, plus Avro themselves recommend against it and I'm not sure how performant it'd be. Avro also includes other marshalling mechanisms called specific (like protobufs one with precompiled classes) and generic used to build dynamic objects on the fly. None of these two fit the bill. The specific one is like protobufs with the disadvantage of having ugly code like http://is.gd/dzwj3 where a static object has a strong reference to a <String,Class> CHM, which would leak in an AS env.
Thrift has the same problems as stated for Protobufs but coudn't see an equivalent way to get find the file descriptor set. Docu is way below what Protobuf offers and latest version which is 0.2 has issues generating classes as stated in JIRA.
MessagePack has the capability to deserialize an object given a String representation of the schema, so a solution like the protobuf one might be hackable. However, the generated classes do not have an equals implementation (??) which is rather odd. Maybe it's due to lack of maturity? Latest version is 0.3, so that might explain it. API wise, MessagePack provides the API that suits best to what we'd want to do and avoids having to use reflection to resolve the payload. However, looking around I couldn't see similar API for the python language for example and similar to Protobuf, we'd have to prepend the schema name key to the payload to then have the reading part lookup the entire schema based on it.
So, I can see two solutions here bearing in mind that pretty much any solution would require precompiling some classes:
- Try to build a marshaller using Protobuf or MessagePack where we enhance it to pass a string key that permits the reading part to deserialize the payload in a generic way.
- Or try to build some wikis on how to integrate Protobuf/Thrift/MessagePack with Hot Rod client so that they can generate the byte[] with these libraries and pass it to the corresponding Hot Rod client. For the moment, we'd do this for the Hot Rod java client. We would add more info once other language clients are available. The reason I said about potentially showing various libraries is cos whereas Protobuf only supports Java/Python/C++, Thrift supports loads more languages ( C++, Java, Python, PHP, Ruby, Erlang, Perl, Haskell, C#, Cocoa, Smalltalk, and OCaml)
In terms of usability , I think the second option is a bit better because building the marshaller requires steps for clients to generate the FileDescriptorSet and somehow pass this to Infinispan so that the marshaller can use it. The second option has none of this and the only added code on the client side is retrieving the byte[] and calling Pojo.parseFrom(byte[]). Performance wise, the 2nd option would be better cos there's no extra hacking needed to pass the String key around and so less bytes are sent around, and no need to use reflection to resolve the payload.
It's a pity that I couldn't find a tool that fully fits our use case and I wonder whether the need to deal with different languages makes coming up with such solution difficult. I believe reflection based instantiation is present in C++ or Python but not sure about other languages. It's also true that our use case is very specific where we're building a tool where we don't have control over what people will put in the cache. Generics could give us some hints for example but it's not mandatory and would not solve the issue entirely.
Thoughts?
--
Galder Zamarreño
Sr. Software Engineer
Infinispan, JBoss Cache
13 years, 7 months
CR3: async stores, again
by Sanne Grinovero
Hello,
I'm testing CR3, I got it so far that I'm testing the stop-restart
application cycle now, I'm using the JdbcStringBasedCacheStore in
async mode,
and every time I stop the application - which issues a clean Stop() on
the CacheManager, it appears that some values are lost.
This is fixed if I remove the async.
Is this expected? I hope not, as this way I can't shutdown the cluster
when using async.
This time I can reproduce it with a UT.
Cheers,
Sanne
13 years, 7 months
Re: [infinispan-dev] CR3: async stores, again
by Sanne Grinovero
yes personally I'd like to set infinite, but the API doesn't accept it.
Also others might want to set it low?
Il giorno 31/ago/2010 17:14, "Emmanuel Bernard" <emmanuel(a)hibernate.org> ha
scritto:
>>>
>>
>> +1, changing that.
>>
>> Now I'm in need of feedback about adding the "shutdown timeout...
If you want something that high, how about infinite as the default? It seems
it's what you really want.
_______________________________________________
infinispan-dev mailing list
infinispan-dev@lists.jbo...
13 years, 7 months
Use of Inifnispan across data centers
by Bela Ban
The other day I had an idea how we could use Infinispan across data centers:
Say we have 2 data centers: NYC and SFO. There are nodes A,B,C,D in NYC
and W,X,Y,Z in SFO.
We'd like to use Infinispan in DIST mode with numOwners == 2, and
ideally place 1 copy of a data in the NYC center and one in the SFO
center. (Of course, this would also work with numOwners > 2, we'd just
have to make sure the copies are distributed more or less evenly between
the 2 data centers).
To make this work, there are 2 things that need to be done:
1. The nodes in NYC and SFO have to be in the same JGroups cluster.
This way Infinispan has insight into the membership and gets
notified of changes. The cluster would usually be TCP based, but
if there's multicasting available between the centers, then of
course multicasting could be used, too.
2. The nodes in NYC would have to have a logical name starting with
"nyc-", the ones in SFO "sfo-".
#2 is needed by Infinispan as input into its consistent hash function in
order to determine the nodes to store a given key/value. Say if we have
a view of
{nyc-D, sfo-Y, sfo-W, nyc-B, nyc-C, sfo-Z, nyc-A, sfo-X}, then we could
come up with a consistent hash function which knows that there are 2
data centers and could place the nodes onto 2 different hash wheels and
pick one from each (if numOwners=2).
Yes, I know #1 sucks, but until we've implemented the more advanced
solutions (e.g. RELAY in JGroups, data center replication in
Infinispan), this seems to me a workable solution that could be
implemented in a short time.
WDYT ?
--
Bela Ban
Lead JGroups / Clustering Team
JBoss
13 years, 7 months
about cache.getCacheManager()
by Sanne Grinovero
Hello,
I was quite surprised to see that
cache.getCacheManager() doesn't return the CacheManager but a
CacheContainer, which I then need to cast to an EmbeddedCacheManager.
Shouldn't the signature of the interface be updated to return a CacheManager ?
The methods exposed by a CacheContainer are rather useless, especially
when returned from a cache instance as the only two I could use are
getCache() and getCache(String cacheName); a use case to explain it
better:
((EmbeddedCacheManager)cache.getCacheManager()).getGlobalConfiguration()
Cheers,
Sanne
13 years, 8 months
eager locking and async replication
by Mircea Markus
Hi,
Here is the scenario, N=numOwners
TM.start()
c.lock(a); //this makes N (might me less) RPCs to acquire locks
c.get(a)
...
TM.commit(); // this would do an N calls for prepare/commit. Might happen async.
By using the key affinity service, one might enforce a tx to operate on "local" keys (i.e. keys that are hashed on the same node where the tx was started).
Now, if we would be able to *only* eager lock the main data owner (v.s. N RPCs for lock acquisition locks) than eager locking would be as fast as the non-eager locking for this scenario.
What happens with the TX if the data owner crashes and only one copy is locked? We would need to invalidate the transaction at originator's side, which I think is possible.
For async repl with N >= 2 and key affinity the performance benefit for eager locking would be close to local puts: which is huge.
This use case was brought by Erik ( cc)- please add you comments if something is missing. What do you think about this optimisation?
Cheers,
Mircea
13 years, 8 months
