Re: [infinispan-dev] [infinispan-internal] async processing documentation (+ nice inconsistency scenario example)
by Manik Surtani
On 19 Mar 2013, at 15:07, Sanne Grinovero <sanne(a)redhat.com> wrote:
>
>
> ----- Original Message -----
>>
>> On 19 Mar 2013, at 12:21, Mircea Markus <mmarkus(a)redhat.com> wrote:
>>
>>> On 19 Mar 2013, at 11:05, Sanne Grinovero wrote:
>>>> Does Marshalling really need to be performed in a separate thread
>>>> pool?
>>>> I think we have too many pools, too much context switching, and
>>>> situations like this one which should be avoided.
>>>>
>>>> We could document it but all these details are making it very
>>>> hard to feel comfortable with, and for this specific use case I
>>>> wonder if there
>>>> is a strong benefit: plain serial operations seem so much cleaner
>>>> to me.
>>> +1 for dropping it in 6.0. It isn't enabled by default and AFAIK it
>>> created more confusion through the users than benefits.
>>
>> Why? I don't agree. If network transfer is the most expensive part
>> of performing a write, then marshalling is the second-most
>> expensive. If you don't take the marshalling offline as well,
>> you're only realising a part of the performance gains of using
>> async.
>
> Of course. I didn't mean to put it on the thread of the invoker, I would expect
> this to happen "behind the scenes" when using async, but in the same thread which
> is managing the lower IO so to reduce both context switching and these weird
> race conditions.. so removing the option only.
Well, when using the same lower IO pool, while common sense, isn't as easy since it is a JGroups pool. If we pass the marshaller itself into JGroups, the marshalling still happens online, and just the IO happening in a separate thread. Also, JGroups allows you to register one marshaller and unmarshaller per channel - which doesn't work when you have a transport shared by multiple cache instances potentially on different class loaders.
So yes, this can be done much better, but that means a fair few changes in JGroups such that:
* Marshalling happens in the async thread (the same one that puts the message on the wire) rather than in the caller's thread
* sendMessage() should accept a marshaller and unmarshaller per invocation
Then we can drop this additional thread pool.
>
>>
>>> On top of that the number of pools is growing (5.3 adds another
>>> pool in the scope of ISPN-2808).
>>
>> You can configure to use a single thread pool for all these tasks, if
>> hanging on to multiple thread pools is too complex.
>
> I don't believe you can always do that, if you don't keep tasks isolated
> in different pools deadlocks could happen. So unless you can come up with
> a nice diagram and explain which ones are safe to share, it is very
> complex to handle.
>
> Would be nice to have these discussions on the public mailing list.
+1. Adding infinispan-dev in cc.
>
> Sanne
>
>>
>> - M
>>
>> --
>> Manik Surtani
>> manik(a)jboss.org
>> twitter.com/maniksurtani
>>
>> Platform Architect, JBoss Data Grid
>> http://red.ht/data-grid
>>
>>
>>
>
--
Manik Surtani
manik(a)jboss.org
twitter.com/maniksurtani
Platform Architect, JBoss Data Grid
http://red.ht/data-grid
12 years, 9 months
Re: [infinispan-dev] CacheLoaders, Distribution mode and Interceptors
by James Aley
Apologies - forgot to copy list.
On 15 March 2013 15:48, James Aley <james.aley(a)swiftkey.net> wrote:
> Hey Adrian,
>
> Thanks for the response. I was chatting to Sanne on IRC yesterday, and
> he suggested this to me. Actually the logging I attached was from a
> cluster of 4 servers with numOwners=2. Sorry, I should have mentioned
> this actually, but I thought seeing as it didn't appear to make any
> difference that I'd just keep things simple in my previous email.
>
> While it seemed not to make a difference in this case... I can see why
> that would make sense. In future tests I guess I should probably stick
> with numOwners > 1.
>
>
> James.
>
> On 15 March 2013 15:44, Adrian Nistor <anistor(a)redhat.com> wrote:
>> Hi James,
>>
>> I'm not an expert on InfinispanDirectory but I've noticed in [1] that the
>> lucene-index cache is distributed with numOwners = 1. That means each cache
>> entry is owned by just one cluster node and there's nowhere else to go in
>> the cluster if the key is not available in local memory, thus it needs
>> fetching from the cache store. This can be solved with numOwners > 1.
>> Please let me know if this solves your problem.
>>
>> Cheers!
>>
>>
>> On 03/15/2013 05:03 PM, James Aley wrote:
>>>
>>> Hey all,
>>>
>>> <OT>
>>> Seeing as this is my first post, I wanted to just quickly thank you
>>> all for Infinispan. So far I'm really enjoying working with it - great
>>> product!
>>> </OT>
>>>
>>> I'm using the InfinispanDirectory for a Lucene project at the moment.
>>> We use Lucene directly to build a search product, which has high read
>>> requirements and likely very large indexes. I'm hoping to make use of
>>> a distribution mode cache to keep the whole index in memory across a
>>> cluster of machines (the index will be too big for one server).
>>>
>>> The problem I'm having is that after loading a filesystem-based Lucene
>>> directory into InfinispanDirectory via LuceneCacheLoader, no nodes are
>>> retrieving data from the cluster - they instead look up keys in their
>>> local CacheLoaders, which involves lots of disk I/O and is very slow.
>>> I was hoping to just use the CacheLoader to initialize the caches, but
>>> from there on read only from RAM (and network, of course). Is this
>>> supported? Maybe I've misunderstood the purpose of the CacheLoader?
>>>
>>> To explain my observations in a little more detail:
>>> * I start a cluster of two servers, using [1] as the cache config.
>>> Both have a local copy of the Lucene index that will be loaded into
>>> the InfinispanDirectory via the loader. This is a test configuration,
>>> where I've set numOwners=1 so that I only need two servers for
>>> distribution to happen.
>>> * Upon startup, things look good. I see the memory usage of the JVM
>>> reflect a pretty near 50/50 split of the data across both servers.
>>> Logging indicates both servers are in the cluster view, all seems
>>> fine.
>>> * When I send a search query to either one of the nodes, I notice the
>>> following:
>>> - iotop shows huge (~100MB/s) disk I/O on that node alone from the
>>> JVM process.
>>> - no change in network activity between nodes (~300b/s, same as when
>>> idle)
>>> - memory usage on the node running the query increases dramatically,
>>> and stays higher even after the query is finished.
>>>
>>> So it seemed to me like each node was favouring use of the CacheLoader
>>> to retrieve keys that are not in memory, instead of using the cluster.
>>> Does that seem reasonable? Is this the expected behaviour?
>>>
>>> I started to investigate this by turning on trace logging, in this
>>> made me think perhaps the cause was that the CacheLoader's interceptor
>>> is higher priority in the chain than the the distribution interceptor?
>>> I'm not at all familiar with the design in any level of detail - just
>>> what I picked up in the last 24 hours from browsing the code, so I
>>> could easily be way off. I've attached the log snippets I thought
>>> relevant in [2].
>>>
>>> Any advice offered much appreciated.
>>> Thanks!
>>>
>>> James.
>>>
>>>
>>> [1] https://www.refheap.com/paste/12531
>>> [2] https://www.refheap.com/paste/12543
>>> _______________________________________________
>>> infinispan-dev mailing list
>>> infinispan-dev(a)lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>
>>
12 years, 9 months
From JSR-107 list, Fwd: Do we need the ReadListener?
by Galder Zamarreño
@Manik, thoughts?
I haven't seen Infinispan cache visit listeners being used that extensively…
Cheers,
Begin forwarded message:
> From: Greg Luck <gluck(a)gregluck.com>
> Subject: Do we need the ReadListener?
> Date: March 14, 2013 12:41:57 AM GMT+01:00
> To: "jsr107(a)googlegroups.com" <jsr107(a)googlegroups.com>
> Reply-To: jsr107(a)googlegroups.com
>
> Guys
>
> Brian has raised https://github.com/jsr107/jsr107spec/issues/104 on whether we need ReadListeners.
>
> I recall that Manik was a proponent of them. Can anyone interested in retaining them please visit the issue and comment, otherwise we might remove them.
>
>
> Regards
>
> Greg Luck
>
> web: http://gregluck.com
> skype: gregrluck
> yahoo: gregrluck
> mobile: +61 408 061 622
>
>
> --
> You received this message because you are subscribed to the Google Groups "jsr107" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to jsr107+unsubscribe(a)googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>
--
Galder Zamarreño
galder(a)redhat.com
twitter.com/galderz
Project Lead, Escalante
http://escalante.io
Engineer, Infinispan
http://infinispan.org
12 years, 9 months
Conditional cache operations useless with Optimistic locking?
by Galder Zamarreño
Hi,
Re: https://issues.jboss.org/browse/ISPN-2891
Not sure what previous Infinispan version the AS instance Immutant guys used (5.1?), but seems like they're having more issues with the clojure test linked in the JIRA.
Again, we're talking about issues with replace(), but in a single node environment. They seem to have issues with PL and OL, but let's focus on OL for which there are logs attached.
Do conditional operations make any sense at all with OL? For example, can the return of replace() be taken a truthful in a transaction with OL?
As shown in the bad.log, this is not possible because lock acquisition and write skew checks are only done when the transaction is committed. So, what's the point of the conditional operations with OL? Their returns provide no guarantees whatsoever.
If this is known thing, I'm not aware of any docu on the topic.
Thoughts?
Cheers,
--
Galder Zamarreño
galder(a)redhat.com
twitter.com/galderz
Project Lead, Escalante
http://escalante.io
Engineer, Infinispan
http://infinispan.org
12 years, 10 months
Infinispan Server repository
by Tristan Tarrant
Hi devs,
as we agreed a while ago, I have renamed the infinispan/jdg repository
on GitHub to infinispan/infinispan-server.
I will also issue pull requests soon to remove any references to jdg
from the code/docs/examples.
Please update your local remotes to point to:
git@github.com:infinispan/infinispan-server.git
Tristan
12 years, 10 months
