[infinispan-dev] [hibernate-dev] Distributed queries

[Ray Hilton]

Phase 2 sounds exactly like what I was imagining, but Non-shared Phase
1 is a good substitute for what we need.  I'll take a look at the JIRA
issues and see if there is anything I can lend myself to.

Ray

On Fri, Sep 25, 2009 at 1:15 AM, Manik Surtani <manik at jboss.org> wrote:
> Hmm, this is what I have in mind (sorry if some of it is repetitive
> and has already been discussed).
>
> Phase 1 (Basic index handling)
>
> Indexes may either be shared (stored on NFS or shared disk), or shared
> via Infinispan (see comments re: Lukasz's InfinispanDirectory,
> somewhat based o my earlier JBoss Cache Directory), or local (local
> disk, RAMDirectory).
>
> * Shared
>    * Indexes need only be generated by the node where changes occur.
> For this, you use the -Dinfinispan.query.indexLocalOnly=true param. [1]
>    * Queries can be executed on any node, and correct results will be
> returned since indexes are shared.
>    * Will work with any cache mode.
>
> * Non-shared
>    * Indexes need to be generated by everyone.  For this, you use the
> -Dinfinispan.query.indexLocalOnly=false.
>    * Queries can be executed on any node.
>    * But only works with REPLICATED cache mode, so all nodes see
> changes and can build/update indexes.
>
> Phase 2 (Distributed querying)
>
> This is dependent on some other work in Infinispan [2] to provide a
> mechanism to distribute workloads.  This is the 'holy grail' I want to
> reach.  Essentially, this is what it entails:
>
> * Indexes are _never_ shared.
>    * Each node maintains local indexes for state it is responsible
> for (-Dinfinispan.query.indexLocalOnly=true).
>    * Indexes could be in memory or disk.
> * Queries themselves are distributed.
>    * The query object is built and broadcast to the entire cluster.
>    * Each node executes the query on its own _local_ index, returning
> results.
>    * The calling node returns a CacheQuery impl that lazily fetches
> and collates results from the cluster.
>    * I expect this Map/Reduce model to perform very well since the
> workload is split up and happens in parallel across multiple CPUs
> against much smaller (individual) datasets.
>    * Works with all cache modes, including DIST.
>    * Need to make sure duplicates are handled, as well as failover.
>
> TODO
> ====
>
> We already have Phase 1 in tech preview.  For Phase 1 to come out of
> tech preview, we need:
>    * A few more JIRAs to be resolved (see [3] for what is outstanding)
>    * Detailed docs (perhaps explaining the options available, as I
> did in the bullet points under "Phase 1" above, maybe with some pretty
> pictures and diagrams)
>    * Lukasz's work to be rolled into a proper release
>
> And for Phase 2, we need [2] in place, and the issue with dupes/
> failover for indexes to be resolved.
>
> If folks want to help with any of the above, please raise your hand -
> this is pretty exciting stuff!  ;)
>
> Cheers
> Manik
>
> [1] This param will be replaced with a proper config option in
> Infinispan 4.1.0, once the Query API is out of 'tech preview'.
> [2] https://jira.jboss.org/jira/browse/ISPN-39
> [3] https://jira.jboss.org/jira/browse/ISPN-194
>
>
>
> On 22 Sep 2009, at 10:27, Michael Neale wrote:
>
>> well as part of a separate project to do with "cloud" stuff we will
>> have appliances/images (or will have) which can help setting up
>> reasonably large clusters for testing (and if we can do the testing
>> within an hour, should only cost a couple of dollars at a time for
>> 100's of nodes).
>>
>> On Tue, Sep 22, 2009 at 7:23 PM, Navin Surtani <nsurtani at redhat.com>
>> wrote:
>>>
>>> On 22 Sep 2009, at 03:00, Michael Neale wrote:
>>>
>>>> I guess that could make sense for some cases - if the data changes
>>>> are
>>>> small-ish, and the index calculation cost isn't huge... I guess if
>>>> the
>>>> objects you are parking in infinispan are large, then it could end
>>>> up
>>>> more efficient to only do the index once and then spread it around
>>>> (sticky to the data that it represents).
>>>
>>>
>>> I believe the plan is to build in a few different configs that would
>>> work for different use-cases. For example, a lot of "small" objects
>>> but not necessarily many nodes so they all share the same index or a
>>> lot of "big" objects sitting on disk on each individual node (where
>>> replication could be expensive).
>>>
>>> Or this is what I understood when speaking with Manik a couple of
>>> weeks ago that is :-).
>>>
>>>
>>>
>>>>
>>>> On Mon, Sep 21, 2009 at 11:09 PM, Ray Hilton <ray at wirestorm.net>
>>>> wrote:
>>>>> Im guessing that something similar already happens so that
>>>>> infinispan
>>>>> can re-jigg the data around the grid.  Forgive my lack of intimate
>>>>> knowledge of how infinispan works here, but at some point the data
>>>>> that was hosted by a bad node needs to be re-distributed?
>>>>>
>>>>> On Mon, Sep 21, 2009 at 11:02 PM, Emmanuel Bernard
>>>>> <emmanuel at hibernate.org> wrote:
>>>>>> could be possible. That would likely be chatty though each time a
>>>>>> node
>>>>>> comes or go.
>>>>>> Typically when a node goes down potentially due to network error,
>>>>>> you
>>>>>> don't wanna be chatty I imagine ;)
>>>>>>
>>>>>> On 21 sept. 09, at 14:59, Ray Hilton wrote:
>>>>>>
>>>>>>> Yes, point taken.
>>>>>>>
>>>>>>> Is there perhaps a way to only index an object on one node.  For
>>>>>>> example, if each node new there were currently 3 copies, and it
>>>>>>> was
>>>>>>> the node with the lowest id, for example, it would index the
>>>>>>> document.
>>>>>>> When a new node joins or a node fails, the strategy is re-applied
>>>>>>> and
>>>>>>> the node-local indices are updated accordingly.
>>>>>>>
>>>>>>> On Mon, Sep 21, 2009 at 6:06 PM, Emmanuel Bernard
>>>>>>> <emmanuel at hibernate.org> wrote:
>>>>>>>> Hello
>>>>>>>> See inline
>>>>>>>>
>>>>>>>> On 20 sept. 09, at 06:01, Ray Hilton wrote:
>>>>>>>>
>>>>>>>>> Hi guys,
>>>>>>>>>
>>>>>>>>> I've been following the distributed query stuff with interest,
>>>>>>>>> but
>>>>>>>>> this is the first time I'm posting, so please excuse the lack
>>>>>>>>> of
>>>>>>>>> intimate knowledge of Infinispan.  Basically, I have been
>>>>>>>>> working
>>>>>>>>> on a
>>>>>>>>> project that could really do with the Holy Grail of a
>>>>>>>>> distributed
>>>>>>>>> query-able cache and I really liked the look of using
>>>>>>>>> JBossCache +
>>>>>>>>> the
>>>>>>>>> Lucene Directory implementation that Manik wrote a while
>>>>>>>>> back.  I
>>>>>>>>> then
>>>>>>>>> noticed Infinispan and talk of building querying directly into
>>>>>>>>> the
>>>>>>>>> project and figured that it would be worthwhile waiting to see
>>>>>>>>> how
>>>>>>>>> that panned out.
>>>>>>>>>
>>>>>>>>> I've thought a bit about how something like this might work,
>>>>>>>>> I'm not
>>>>>>>>> sure if this will be in any way helpful, but here goes:  I
>>>>>>>>> guess
>>>>>>>>> there
>>>>>>>>> are two approaches:  1) store the index (or partitioned
>>>>>>>>> indices) in
>>>>>>>>> the grid and sync it to a node to do a particular query or 2)
>>>>>>>>> each
>>>>>>>>> node has an index for the data it currently caches.  We
>>>>>>>>> preferred
>>>>>>>>> the
>>>>>>>>> second idea as it offers a natural way to partition the indices
>>>>>>>>> (i.e.
>>>>>>>>> however infinispan is configured to do it).  The first option
>>>>>>>>> would
>>>>>>>>> mean you end up with either a monolithic index in the grid, or
>>>>>>>>> partitions based on, say, date, that have to be sync'd en-
>>>>>>>>> mass to
>>>>>>>>> whichever node(s) are doing a query.  I realise that the second
>>>>>>>>> technique would produce duplicates, but Im sure there would be
>>>>>>>>> a way
>>>>>>>>> to eliminate dupes based on the object's uuid (something im
>>>>>>>>> pretty
>>>>>>>>> sure infinispan already has a notion of).
>>>>>>>>
>>>>>>>> Well 2 looks nicer but I don't know an obvious way to solve the
>>>>>>>> duplication issues:
>>>>>>>>  - returning several times the same content does alter the
>>>>>>>> scoring of
>>>>>>>> other documents
>>>>>>>>  - it prevent efficient pagination as somehow you need to jump
>>>>>>>> several results.
>>>>>>>>
>>>>>>>>>
>>>>>>>>> We would also need to come up with a way or normalising the
>>>>>>>>> scoring
>>>>>>>>> across all partitions (regardless of which method is used).  I
>>>>>>>>> have
>>>>>>>>> seen this done before, and it would basically involve, per-
>>>>>>>>> query,
>>>>>>>>> finding out the term frequency of the various keywords across
>>>>>>>>> the
>>>>>>>>> entire index, or at least enough of it to produce a
>>>>>>>>> representative
>>>>>>>>> value.  This would be used to calculate the score for each hit
>>>>>>>>> when
>>>>>>>>> doing the actual search, and thus the ranking.
>>>>>>>>
>>>>>>>> I believe Lucene does normalize the score properly when using
>>>>>>>> the
>>>>>>>> remote IndexSearcher as the normalization is done on the
>>>>>>>> "client"
>>>>>>>> side.
>>>>>>>>
>>>>>>>>>
>>>>>>>>> We have had issues with index corruption in the past as well
>>>>>>>>> (probably
>>>>>>>>> due to programming bugs rather than lucene).  Making each node
>>>>>>>>> responsible for its own index will make it very easy to throw
>>>>>>>>> corrupt
>>>>>>>>> indices away and re-generate new ones.
>>>>>>>>>
>>>>>>>>> I did take a look at the visitor stuff in Infinispan before,
>>>>>>>>> but I
>>>>>>>>> wasn't really sure where the best place to hook into would be
>>>>>>>>> to
>>>>>>>>> find
>>>>>>>>> out which objects are being stored locally or evicted.  If
>>>>>>>>> someone
>>>>>>>>> has
>>>>>>>>> a good idea of where to start, I'd be happy to lend a hand to
>>>>>>>>> to
>>>>>>>>> this
>>>>>>>>> effort!
>>>>>>>>>
>>>>>>>>> Ray
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Sat, Sep 19, 2009 at 8:43 PM, Michael Neale <michael.neale at gmail.com
>>>>>>>>>> wrote:
>>>>>>>>>> I think you just stuck a pin in the bubble that normally says
>>>>>>>>>> "magic
>>>>>>>>>> happens here" ;)
>>>>>>>>>>
>>>>>>>>>> How much of this did you tackle regarding hibernate search
>>>>>>>>>> that
>>>>>>>>>> could
>>>>>>>>>> be applied here?
>>>>>>>>>>
>>>>>>>>>> (you final point re duplication may have some "flexibility" I
>>>>>>>>>> think ?)
>>>>>>>>>>
>>>>>>>>>> On Fri, Sep 18, 2009 at 6:18 PM, Emmanuel Bernard
>>>>>>>>>> <emmanuel at hibernate.org> wrote:
>>>>>>>>>>> Neither 1 nor 2 imply *distributed* queries.
>>>>>>>>>>>
>>>>>>>>>>> The hard parts with distributed queries (ie executed on a
>>>>>>>>>>> grid and
>>>>>>>>>>> recomposed) are:
>>>>>>>>>>>  - making sure you ask all the nodes where the index is
>>>>>>>>>>> distributed
>>>>>>>>>>> (you can't miss a node)
>>>>>>>>>>>  - find a way to index only a subset of the data in a given
>>>>>>>>>>> index
>>>>>>>>>>> (on
>>>>>>>>>>> a given node). Applying the Infinispan distribution routine
>>>>>>>>>>> to the
>>>>>>>>>>> InfinispanDirectory does not do that, it chunks data
>>>>>>>>>>> arbitrarily.
>>>>>>>>>>>  - be able to rebuild a given index on a givne node (ie
>>>>>>>>>>> remember
>>>>>>>>>>> which element were indexed)
>>>>>>>>>>>  - you need to find a way to distribute your data without
>>>>>>>>>>> duplication. If a key is indexed multiple times, then you end
>>>>>>>>>>> up
>>>>>>>>>>> with
>>>>>>>>>>> duplicated results that can't trivially be de-duplicated.
>>>>>>>>>>>
>>>>>>>>>>> Happy thinking.
>>>>>>>>>>>
>>>>>>>>>>> On 17 sept. 09, at 10:32, Sanne Grinovero wrote:
>>>>>>>>>>>
>>>>>>>>>>>> 2009/9/17 Michael Neale <michael.neale at gmail.com>:
>>>>>>>>>>>>> I am still not entirely sure what I am asking, but look
>>>>>>>>>>>>> forward
>>>>>>>>>>>>> for
>>>>>>>>>>>>> your merged in changes (they are in another branch right
>>>>>>>>>>>>> now
>>>>>>>>>>>>> yes?).
>>>>>>>>>>>>>
>>>>>>>>>>>>> Yes I mean querying objects - I was under the impression
>>>>>>>>>>>>> that
>>>>>>>>>>>>> lucene
>>>>>>>>>>>>> was used for the indexing of the data to service these
>>>>>>>>>>>>> queries?
>>>>>>>>>>>>
>>>>>>>>>>>> Sure, to clarify: there's work going on on two different
>>>>>>>>>>>> aspects,
>>>>>>>>>>>> which
>>>>>>>>>>>> complement each other in the ideal setup:
>>>>>>>>>>>>
>>>>>>>>>>>> 1) Be able to query a Lucene index (wherever you store that)
>>>>>>>>>>>> to
>>>>>>>>>>>> find
>>>>>>>>>>>> objects
>>>>>>>>>>>> which are located inside Infinispan; this is about how to
>>>>>>>>>>>> search
>>>>>>>>>>>> them and how
>>>>>>>>>>>> to maintain the index in synch with Infinispan's content.
>>>>>>>>>>>>
>>>>>>>>>>>> 2) Store a Lucene index inside Infinispan, instead of, for
>>>>>>>>>>>> example,
>>>>>>>>>>>> filesystem.
>>>>>>>>>>>> In this case we're not concerned about what you index, the
>>>>>>>>>>>> Lucene
>>>>>>>>>>>> interface
>>>>>>>>>>>> is the usual one and you should be able to replace the
>>>>>>>>>>>> Directory
>>>>>>>>>>>> implementation in existing applications.
>>>>>>>>>>>>
>>>>>>>>>>>> So 1) is the branch you've found, and Navin is working on
>>>>>>>>>>>> that,
>>>>>>>>>>>> 2)
>>>>>>>>>>>> is not yet
>>>>>>>>>>>> in subversion, the latest patch is attached to other
>>>>>>>>>>>> thread by
>>>>>>>>>>>> Łukasz,
>>>>>>>>>>>> and is to be applied
>>>>>>>>>>>> on Hibernate Search's trunk (and depends on Infinispan).
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Wed, Sep 16, 2009 at 10:32 PM, Navin Surtani
>>>>>>>>>>>>> <nsurtani at redhat.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On 16 Sep 2009, at 12:25, Michael Neale wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> oh ok nice - could you point me at which branch to try to
>>>>>>>>>>>>>>> find
>>>>>>>>>>>>>>> some
>>>>>>>>>>>>>>> tests to play with?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> If you're talking about Querying objects in Infinispan: -
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> The eventual goal is to be able to have different
>>>>>>>>>>>>>> configurations on
>>>>>>>>>>>>>> how you want to index your data. Manik has given me the
>>>>>>>>>>>>>> 'OK' to
>>>>>>>>>>>>>> push a
>>>>>>>>>>>>>> simple query interface for CR1 for Monday/Tuesday.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I'm kind-of pressed with getting the code working for this
>>>>>>>>>>>>>> and
>>>>>>>>>>>>>> also
>>>>>>>>>>>>>> between moving house and lack of internet there I'll be a
>>>>>>>>>>>>>> bit
>>>>>>>>>>>>>> quiet.
>>>>>>>>>>>>>> However, I'll get a wiki up by the end of the week about
>>>>>>>>>>>>>> how
>>>>>>>>>>>>>> this
>>>>>>>>>>>>>> all
>>>>>>>>>>>>>> works.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> However if you're not then I assume you're talking about
>>>>>>>>>>>>>> using
>>>>>>>>>>>>>> Lucene
>>>>>>>>>>>>>> to index into Infinispan?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Wed, Sep 16, 2009 at 6:05 PM, Sanne Grinovero
>>>>>>>>>>>>>>> <sanne.grinovero at gmail.com> wrote:
>>>>>>>>>>>>>>>> 2009/9/16 Michael Neale <michael.neale at gmail.com>:
>>>>>>>>>>>>>>>>> regarding indexing and queries - is the current aim to
>>>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>> require
>>>>>>>>>>>>>>>>> that the index for the entire data grid exist on a
>>>>>>>>>>>>>>>>> single
>>>>>>>>>>>>>>>>> node?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> (asking as a potential user who is wrestling with
>>>>>>>>>>>>>>>>> lucene
>>>>>>>>>>>>>>>>> indexes at
>>>>>>>>>>>>>>>>> the moment is curious).
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Yes the concept is to store the Lucene index itself in
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> grid,
>>>>>>>>>>>>>>>> so
>>>>>>>>>>>>>>>> it will
>>>>>>>>>>>>>>>> be distributed, and the segments you use most get cached
>>>>>>>>>>>>>>>> locally.
>>>>>>>>>>>>>>>> At the moment you have to select only one node to
>>>>>>>>>>>>>>>> write to
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> index,
>>>>>>>>>>>>>>>> but all other nodes should be able to read.
>>>>>>>>>>>>>>>> Feel free to test it as we are needing feedback.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>> Michael D Neale
>>>>>>>>>>>>>>>>> home: www.michaelneale.net
>>>>>>>>>>>>>>>>> blog: michaelneale.blogspot.com
>>>>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>>>>> infinispan-dev mailing list
>>>>>>>>>>>>>>>>> infinispan-dev at lists.jboss.org
>>>>>>>>>>>>>>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>>>> infinispan-dev mailing list
>>>>>>>>>>>>>>>> infinispan-dev at lists.jboss.org
>>>>>>>>>>>>>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>> Michael D Neale
>>>>>>>>>>>>>>> home: www.michaelneale.net
>>>>>>>>>>>>>>> blog: michaelneale.blogspot.com
>>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>>> infinispan-dev mailing list
>>>>>>>>>>>>>>> infinispan-dev at lists.jboss.org
>>>>>>>>>>>>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Navin Surtani
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Intern Infinispan
>>>>>>>>>>>>>> Intern JBoss Cache Searchable
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>> infinispan-dev mailing list
>>>>>>>>>>>>>> infinispan-dev at lists.jboss.org
>>>>>>>>>>>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> Michael D Neale
>>>>>>>>>>>>> home: www.michaelneale.net
>>>>>>>>>>>>> blog: michaelneale.blogspot.com
>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>> infinispan-dev mailing list
>>>>>>>>>>>>> infinispan-dev at lists.jboss.org
>>>>>>>>>>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>> hibernate-dev mailing list
>>>>>>>>>>>> hibernate-dev at lists.jboss.org
>>>>>>>>>>>> https://lists.jboss.org/mailman/listinfo/hibernate-dev
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> infinispan-dev mailing list
>>>>>>>>>>> infinispan-dev at lists.jboss.org
>>>>>>>>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Michael D Neale
>>>>>>>>>> home: www.michaelneale.net
>>>>>>>>>> blog: michaelneale.blogspot.com
>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> infinispan-dev mailing list
>>>>>>>>>> infinispan-dev at lists.jboss.org
>>>>>>>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Ray Hilton
>>>>>>>>> -
>>>>>>>>>         email: ray at wirestorm.net
>>>>>>>>> melbourne: +61 (0) 3 9077 0513
>>>>>>>>>       mobile: +61 (0) 430 484 708
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> infinispan-dev mailing list
>>>>>>>>> infinispan-dev at lists.jboss.org
>>>>>>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> infinispan-dev mailing list
>>>>>>>> infinispan-dev at lists.jboss.org
>>>>>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Ray Hilton
>>>>>>> -
>>>>>>>         email: ray at wirestorm.net
>>>>>>> melbourne: +61 (0) 3 9077 0513
>>>>>>>       mobile: +61 (0) 430 484 708
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> infinispan-dev mailing list
>>>>>>> infinispan-dev at lists.jboss.org
>>>>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> infinispan-dev mailing list
>>>>>> infinispan-dev at lists.jboss.org
>>>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Ray Hilton
>>>>> -
>>>>>         email: ray at wirestorm.net
>>>>>  melbourne: +61 (0) 3 9077 0513
>>>>>       mobile: +61 (0) 430 484 708
>>>>>
>>>>> _______________________________________________
>>>>> infinispan-dev mailing list
>>>>> infinispan-dev at lists.jboss.org
>>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>>>
>>>>
>>>>
>>>> --
>>>> Michael D Neale
>>>> home: www.michaelneale.net
>>>> blog: michaelneale.blogspot.com
>>>>
>>>> _______________________________________________
>>>> infinispan-dev mailing list
>>>> infinispan-dev at lists.jboss.org
>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>>
>>> Navin Surtani
>>>
>>> Intern Infinispan
>>> Intern JBoss Cache Searchable
>>>
>>>
>>> _______________________________________________
>>> infinispan-dev mailing list
>>> infinispan-dev at lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>
>>
>>
>> --
>> Michael D Neale
>> home: www.michaelneale.net
>> blog: michaelneale.blogspot.com
>>
>> _______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
> --
> Manik Surtani
> manik at jboss.org
> Lead, Infinispan
> Lead, JBoss Cache
> http://www.infinispan.org
> http://www.jbosscache.org
>
>
>
>
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev



-- 
Ray Hilton
-
         email: ray at wirestorm.net
 melbourne: +61 (0) 3 9077 0513
       mobile: +61 (0) 430 484 708