July 2010 - infinispan-dev - Jboss List Archives

Configuration visitor - Re: [JBoss JIRA] Commented: (ISPN-145) No transport and singleton store enabled should not be allowed

by Vladimir Blagojevic

Hi, Galder and I talked about this offline. Time to involve you guys! I just completed visitor pattern for our configuration objects. Visitor is passed from root of configuration - InfinispanConfiguration object. InfinispanConfiguration class has a new method: public void accept(ConfigurationBeanVisitor v) How do we want to integrate this visitor into existing structure? 1) We add a new factory method to InfinispanConfiguration with additional ConfigurationBeanVisitor parameter 2) We leave everything as is and if there is a need to pass some visitor we pass it to InfinispanConfiguration instance directly (from DefaultCacheManager) DefaultCacheManager will pass ValidationVisitor to InfinispanConfiguration that will verify configuration semantically. Regards, Vladimir On 09-09-09 10:19 AM, Galder Zamarreno wrote: > Good idea :) > > On 09/09/2009 04:13 PM, Vladimir Blagojevic wrote: >> Yeah, >> >> I was thinking that we can make a visitor for configuration tree and >> then you can do verification of any node and other things as well. Use >> cases will come up in the future for sure. >> >> Cheers >> >> >> >> On 09-09-09 3:29 AM, Galder Zamarreno (JIRA) wrote: >>> [ >>> https://jira.jboss.org/jira/browse/ISPN-145?page=com.atlassian.jira.plugi... >>> >>> ] >>> >>> Galder Zamarreno commented on ISPN-145: >>> --------------------------------------- >>> >>> Not sure I understand what you mean by generic though. You mean any >>> component to have a validation step of some sort? >>> >>> Thanks for taking this on :) >>> >>>> No transport and singleton store enabled should not be allowed >>>> -------------------------------------------------------------- >>>> >>>> Key: ISPN-145 >>>> URL: https://jira.jboss.org/jira/browse/ISPN-145 >>>> Project: Infinispan >>>> Issue Type: Bug >>>> Components: Loaders and Stores >>>> Affects Versions: 4.0.0.ALPHA6 >>>> Reporter: Galder Zamarreno >>>> Assignee: Vladimir Blagojevic >>>> Priority: Minor >>>> Fix For: 4.0.0.CR1 >>>> >>>> >>>> Throw configuration exception if singleton store configured without >>>> transport having been configured. >>>> It makes no sense to have singleton store enabled when there's no >>>> transport. >> >

14 years, 10 months

4
10
0 / 0

Using Coverity scan?

by Sanne Grinovero

Hello, Did you consider enabling Infinispan to be monitored by coverity's code analysis services? They are free for OSS projects, I saw a demo recently and was quite amazed. It's similar to FindBugs, but not only about static code checks. They checkout your code from trunk and then run several analysis on it periodically, one of them is about dynamic thread behavior to predict deadlocks or missing fences instrumenting the code, and produce nice public reports; AFAIK you don't need to setup anything yourself, besides getting in touch to ask for it. It's only available for C and Java code, and they have an impressive list of OSS projects in the C world (linux kernel, httpd server, samba, gnome, GCC, PostgreSQL, ...) but not much on Java. http://scan.coverity.com/ No, I'm not affiliated :-) Just thinking that it might be useful to have if it's not too hard to setup. Cheers, Sanne

15 years, 7 months

2
3
0 / 0

Findings of ISPN-508 - Portable serialization marshaller

by Galder Zamarreño

Hi all, Re: https://jira.jboss.org/browse/ISPN-508. This is a bit of a long one, so take your time :) Over past few days I've been trying to come up with a Infinispan Marshaller implementation based on one of those portable serialization libraries (i.e. protobuf, thrift...etc) and I've reached the conclusion that it is possible to do so but requires a fair bit of work on our side and performance/usability would decrease. Before explaining my findings, let me explain the model that I'm trying to achieve: I want to build a generic marshalling/unmarshalling implementation that's based on language neutral serialization libraries and which does not require the unmarshalling code to have knowledge of the target type. Think about Java Serialization or JBoss Marshalling where the code written can be generic enough w/o needing to know what we're trying to deserialize. The payload in these cases has enough information for the underlying library to figure out the type, instantiate it and populate it accordingly. The problem is that there's no such library in the space of portable serialization libraries that provides this out of the box and let me explain why: Pretty much any code using those libraries must use class type or schema name to deserialize the payload. However, Infinispan based Marshaller does not have such information when it's trying to deserialize the payload. Infinispan Marshaller based on JDK serialization or JBoss Marshalling uses information in the payload to instantiate the object. However, none of the libraries out there use this mechanism and instead some (i.e. Protobufs) force the client code to do things like: Pojo.parseFrom(byte[]) to generate instances of Pojo. In these cases for example, nothing stops you from writing a UTF-8 string with the class name and put it at the beginning of the payload so the deserialization part can be class agnostic, but this payload would not be portable. What would a python client do with a String containing the java class name? Based on the FileDescriptorSet information in http://code.google.com/apis/protocolbuffers/docs/techniques.html I was able to hack something that might work in a portable way. Given a FileDescriptorSet generated a class compile time, I was able to match the protobuf name of a class with its java counter part. So, before writing the protobuf generated byte[], i prepend it with protobuf class name so that when reading, I can take the name, get the java class name and using reflection call parseFrom method to convert the byte[] into a pojo. Note that DynamicMessage class hinted in the the techniques page won't work cos it cannot create instances of pojos. It can only create generic objects with fields that are accessed in a reflection style. I also looked at what Avro offers but it does not fully fit either. They have a reflection based serialization mechanism that doesn't require any precompilation, but it requires some kind of type knowledge on the client code to deserialize, plus Avro themselves recommend against it and I'm not sure how performant it'd be. Avro also includes other marshalling mechanisms called specific (like protobufs one with precompiled classes) and generic used to build dynamic objects on the fly. None of these two fit the bill. The specific one is like protobufs with the disadvantage of having ugly code like http://is.gd/dzwj3 where a static object has a strong reference to a <String,Class> CHM, which would leak in an AS env. Thrift has the same problems as stated for Protobufs but coudn't see an equivalent way to get find the file descriptor set. Docu is way below what Protobuf offers and latest version which is 0.2 has issues generating classes as stated in JIRA. MessagePack has the capability to deserialize an object given a String representation of the schema, so a solution like the protobuf one might be hackable. However, the generated classes do not have an equals implementation (??) which is rather odd. Maybe it's due to lack of maturity? Latest version is 0.3, so that might explain it. API wise, MessagePack provides the API that suits best to what we'd want to do and avoids having to use reflection to resolve the payload. However, looking around I couldn't see similar API for the python language for example and similar to Protobuf, we'd have to prepend the schema name key to the payload to then have the reading part lookup the entire schema based on it. So, I can see two solutions here bearing in mind that pretty much any solution would require precompiling some classes: - Try to build a marshaller using Protobuf or MessagePack where we enhance it to pass a string key that permits the reading part to deserialize the payload in a generic way. - Or try to build some wikis on how to integrate Protobuf/Thrift/MessagePack with Hot Rod client so that they can generate the byte[] with these libraries and pass it to the corresponding Hot Rod client. For the moment, we'd do this for the Hot Rod java client. We would add more info once other language clients are available. The reason I said about potentially showing various libraries is cos whereas Protobuf only supports Java/Python/C++, Thrift supports loads more languages ( C++, Java, Python, PHP, Ruby, Erlang, Perl, Haskell, C#, Cocoa, Smalltalk, and OCaml) In terms of usability , I think the second option is a bit better because building the marshaller requires steps for clients to generate the FileDescriptorSet and somehow pass this to Infinispan so that the marshaller can use it. The second option has none of this and the only added code on the client side is retrieving the byte[] and calling Pojo.parseFrom(byte[]). Performance wise, the 2nd option would be better cos there's no extra hacking needed to pass the String key around and so less bytes are sent around, and no need to use reflection to resolve the payload. It's a pity that I couldn't find a tool that fully fits our use case and I wonder whether the need to deal with different languages makes coming up with such solution difficult. I believe reflection based instantiation is present in C++ or Python but not sure about other languages. It's also true that our use case is very specific where we're building a tool where we don't have control over what people will put in the cache. Generics could give us some hints for example but it's not mandatory and would not solve the issue entirely. Thoughts? -- Galder Zamarreño Sr. Software Engineer Infinispan, JBoss Cache

15 years, 11 months

4
19
0 / 0

Finalising stuff for 4.1.0.GA

by Manik Surtani

Guys, Can we please aim to have all 4.1.0 tasks closed up this week? http://links.infinispan.org/iypht There aren't many, and most of them seem to be documentation-related, so if we can spend a few cycles closing these up that would be great. That would put us in a very good position to address any issues that crop up on the user forums quickly and efficiently, and hopefully push out a 4.1.0.Final fairly soon. Cheers Manik -- Manik Surtani manik(a)jboss.org Lead, Infinispan Lead, JBoss Cache http://www.infinispan.org http://www.jbosscache.org

15 years, 11 months

3
3
0 / 0

Comments on latest iteration of Distributed Task Execution design

by Manik Surtani

Vladimir, Here are some thoughts on your latest iteration (Version 8) of this page: http://community.jboss.org/wiki/InfinispanDistributedExecutionFramework There are actually 2 distinct use cases that this feature would touch: 1) pushing a Callable to the node where state is located and pull back a result 2) breaking up a task into Callables and then pushing the Callables out as per (1). Now only (2) is true map/reduce (and relevant to fork/join) but for many cases, (1) alone is enough. So whatever API we propose should support both. Simple remote code execution to leverage data locality where the user doesn't care about breaking up tasks into subtasks, as well as proper task decomposition. But we should be clear about these differences - even though on an impl level they are very closely related. The stuff you have so far handles case (2) pretty well but we should also expose (1). Some other feedback: * I presume distributed task monitoring and annotations are sections that come under "outside of scope"? They seem to be on the same level so I wasn't sure of your intentions here. * Proposed interfaces - Not sure if I understand the purpose of DistributedCallable#mapped(). You already assign a cache to the callable in DistributedCallable#initialized(), right? * DistributedCallable#preferredExecutionNodes() - do we really want to support this at this stage? Or is it better to not support arbitrary node selection by end-user code? Simpler would be to add a DistributedCallable#getRelatedKeys() which returns a Set of keys which the callable would be expected to touch, so we can decide where to route the task. Or maybe you want to offer both forms, so that a DistributedCallable could *either* provide a set of nodes to execute on or a set of keys which it expects to touch. Cheers Manik -- Manik Surtani manik(a)jboss.org Lead, Infinispan Lead, JBoss Cache http://www.infinispan.org http://www.jbosscache.org

15 years, 12 months

3
8
0 / 0

transaction over hotrod

by Mircea Markus

Hi, I'be just added a design draft for transactions over hotrod[1]. Feedback appreciated! Cheers, Mircea [1]http://community.jboss.org/wiki/TransactionsOverHotRod

16 years

3
13
0 / 0

Re: [infinispan-dev] Distributed execution Framework

by Bowers, Ronald (Civ, ARL/SLAD)

Hi Vladimir, Malik, and Jeff, I too have been working on a Rio-based distributed execution framework and have also done some work deploying Infinispan under Rio. This project (Gomez) is available at https://rio-substrates.dev.java.net/svn/rio-substrates/branches/maven-gomez. Although not quite production ready, it is mostly stable and reliable. Gomez is used as the basis of a distributed simulation system. Our simulation problem is easily parallelizable and use the F/J pattern extensively. The Gomez distribute execution framework consists of three types of services: 1. Worker Service - executes tasks 2. Tasks Space - a JavaSpace that is used to distribute tasks and events 3. Job Monitor - watches the progress of a job and acts on certain events. The collection of these services is referred to a Sim Pool. Multiple worker services and task spaces can be deployed to nodes in the cluster. There is only one Job Monitor per Sim Pool. Distributed F/J in implemented in Gomez in the following fashion. An initial task is dropped into a Task Space by the client. This task is taken from the space by a Worker, who executes the task. The task can fork into multiple subordinate tasks. These subordinate tasks are then written to the task space to be executed by any worker in the sim pool. The initial (parent) task then generates a join task. The join task is attached to a message that is sent through the task space to the Job Monitor. The message tells the Job Monitor the number of subordinate tasks and an identifier by which they can be recognized. This identifier is necessary because the worker can be expanding and distributing multiple F/J hierarchies concurrently. After writing the message to the JobMonitor, the parent task returns. Results from tasks are written to a separate distributed cache. Currently it is also implemented using JavaSpaces. The Job Monitor listens for the completion of the subordinate tasks. When all subordinate tasks have completed, the Job Monitor submits the "join" task. This task gathers the results from the subordinate tasks and performs whatever actions (usually statistics) are required. When the join task has completed it also notifies the Job Monitor and the simulation can continue to roll back up. In my case I have only started investigating Infinispan in the role as a distributed cache for results. I have not considered it for task distribution since I couldn't see a clear way of mapping the JavaSpace take() pattern to the Cache API. -Ron

16 years

2
1
0 / 0

Distributed execution framework

by Vladimir Blagojevic

Hi, I wanted to gather feedback from community regarding proposed scope and objectives of Distributed execution framework targeted for Infinispan 5.0. The document is a gentle introduction to the subject of distributed execution and as such is accessible to wider audience. Eager to hear your thoughts. Regards, Vladimir https://community.jboss.org/wiki/InfinispanDistributedExecutionFramework -- Vladimir Blagojevic JBoss Clustering Team JBoss, by Red Hat

16 years

6
14
0 / 0

Adding isMarshallable() to Marshaller interface?

by Galder Zamarreño

Hi, Re: https://jira.jboss.org/browse/ISPN-552 This came from the user forum (http://community.jboss.org/message/554531#554531). I had originally thought of removing the if statement in MV ctor but that's not a good idea since it can delay discovery and in single node envs it won't detect the problem. Instead of hardcoding what the marshaller can do in MV ctor, I suggest adding a isMarshallable method to Marshaller interface. More details in JIRA. Thoughts? -- Galder Zamarreño Sr. Software Engineer Infinispan, JBoss Cache

16 years

2
3
0 / 0

Hibernate Search 3.3.0.Alpha1

by Emmanuel Bernard

As discussed, here is the semi public release of Hibernate Search for use by Infinispan. The doc is not up to date. http://repository.jboss.org/nexus/content/groups/public/org/hibernate/hib... Emmanuel

16 years

2
1
0 / 0

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

infinispan-dev July 2010