Spliting Lucene segments in Infinispan Directory
by Łukasz Moreń
Hi,
The JBoss Cache directory for Lucene splits each Lucene segment into pieces
- chunks. Similar solutions exists in Lucene RamDirectory implementation.
Are there some pros to use such splitting approach in Infinispan directory
case? Some buffer size is recommended?
Lukasz
15 years, 4 months
Re: [infinispan-dev] [ISPN-116] Async cache store: aggregation of multiple changes on a single key
by Galder Zamarreno
Did other list readers ever got this message?
I did send it last week but never actually got it via the dev list, so
double checking...
On 07/15/2009 07:58 PM, Galder Zamarreno wrote:
> Hi,
>
> As part of this JIRA, I'm thinking whether we wanna leave the current
> queue based async store.
>
> For the work I've done, I've created a new class called
> CoalescedAsyncStore that works in the way agreed below, iow, instead of
> queying mods, it keeps a ConcurrentMap and which is swapped by a new
> instance when the async thread is going to apply the changes.
>
> I've run some perf tests between CoalescedAsyncStore and AsyncStore and
> when doing 1 million store where each key is different, the performance
> is fairly similar. The difference comes when trying to run a similar
> test where the same key is always updated, this results in lesser
> underlying store calls and hence CoalescedAsyncStore solution is faster
> here.
>
> So, rather than keeping both, I'd suggest replacing AsyncStore with the
> impl in CoalescedAsyncStore. We also need to look at the configuration
> for the new AsyncStore:
>
> Configuration wise in AsyncStoreConfig, I'd leave threadPoolSize,
> enabled and add the read/write lock acquisition timeout. The rest I'd
> get rid as no longer apply:
>
> int batchSize = 100;
> long pollWait = 100;
> int queueSize = 10000;
>
> By default, I'd give 5000ms to the read/write lock acquisition timeout.
>
> Thoughts?
>
> On 07/09/2009 01:08 PM, Manik Surtani wrote:
>>
>> On 9 Jul 2009, at 11:37, Galder Zamarreno wrote:
>>
>>>
>>>
>>> On 07/09/2009 11:55 AM, Manik Surtani wrote:
>>>>
>>>> On 8 Jul 2009, at 19:53, Jason T. Greene wrote:
>>>>
>>>>> Manik Surtani wrote:
>>>>>> * Make the concurrent map volatile
>>>>>> * When iterating, first create a new ConcurrentMap and replace the
>>>>>> old one with the new one so all concurrent threads write to the new
>>>>>> Map
>>>>>> * Iterate over the old map
>>>>>
>>>>> That would lead to race conditions since a concurrent writing thread
>>>>> could write to the "old" map, either by getting a recent yet incorrect
>>>>> read off the volatile, or reading it right before it changes.
>>>>
>>>> True, since referencing the map and writing to it isn't atomic.
>>>>
>>>> We could guard access to the map with a read/write lock. Safe, if a
>>>> little heavy-handed... map writers would acquire a RL (since we want
>>>> concurrent access here) but the async flushing thread would need to
>>>> wait
>>>> for a WL to swap the map reference, releasing the lock after the map
>>>> reference has been swapped.
>>>
>>> Yeah, I don't think it could be volatile because it only applies to
>>> the variable itself, not the contents pointed at. The R/W lock
>>> approach looks like a valid one and better than using synchronized
>>> blocks. Another thing I had in my mind is whether we need the copy to
>>> be a ConcurrentMap, we could probably just use a
>>> Immutables.immutableMapCopy().
>>
>> Sync blocks would suck since it is effectively an exclusive lock. We
>> want non-exclusive (read) locks for threads writing to this map - we
>> want multiple application threads to do this concurrently otherwise this
>> becomes a bottleneck. And hence the need for a ConcurrentMap.
>>
>> And we don't make a copy at all - all the async flushing thread does is
>> that it creates a new, empty ConcurrentMap and swaps the 'async queue'
>> Map so that application threads can continue logging their changes
>> somewhere while old changes can be iterated through and flushed. And for
>> this, the field still needs to be volatile (not strictly so, since we
>> can rely on the fencing that will happen in the lock as a side-effect)
>>
>>> I looked at the two implementations (ConcurrentHashMap(Map)
>>> constructor vs Immutables.immutableMapCopy() which would
>>> Object.clone() on the map) but I'm not sure which one would be faster.
>>> Since clone() would rely on a native clone() impl, I'd imagine that to
>>> be faster.
>>
>> Like I said, we don't need a copy. We need a new, empty map for threads
>> to log changes to while the old one's being flushed.
>>
>>>
>>>> --
>>>> Manik Surtani
>>>> manik(a)jboss.org
>>>> Lead, Infinispan
>>>> Lead, JBoss Cache
>>>> http://www.infinispan.org
>>>> http://www.jbosscache.org
>>>>
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> infinispan-dev mailing list
>>>> infinispan-dev(a)lists.jboss.org
>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>>
>>> --
>>> Galder Zamarreño
>>> Sr. Software Engineer
>>> Infinispan, JBoss Cache
>>
>> --
>> Manik Surtani
>> manik(a)jboss.org
>> Lead, Infinispan
>> Lead, JBoss Cache
>> http://www.infinispan.org
>> http://www.jbosscache.org
>>
>>
>>
>>
>
--
Galder Zamarreño
Sr. Software Engineer
Infinispan, JBoss Cache
15 years, 4 months
Infinispan Persistence Manager and JBoss Messaging Journal
by Clebert Suconic
I had a quick look on the Persistence code for Infinispan yesterday, and
it looks like the JBoss Messaging Journal was just made for infinispan.
The only thing I couldn' t understand was the streaming operations
fromStream and toStream. What they are used for, and how they translate
into multiple records?
We have talked about making the Journal a sub project of JBoss
Messaging, in a way other projects would be able to reuse (if that
interests you guys).
Cheers,
Clebert
15 years, 4 months
Re: [infinispan-dev] [jboss-dev] Infinispan configuration - all in one solution
by Vladimir Blagojevic
We have sucesfully converted to JAXB based configuration reading and
schema generation.
Along the way I noticed a few things that JAXB 2.x is missing, the most
important in our case being:
No clear support for custom annotations and hooking up their processing
into JAXB library
Annotation for xs:documentation i.e ability to generate xs:documentation
elements in generated schema
Overall it was well worth the switch to JAXB annotations to map
configuration class hierarchy to XML and thus achieve seamless XML
configuration files reading into object model as well as automatic XML
schema generation.
On 7/22/09 2:41 PM, Alexey Loubyansky wrote:
> Sorry for the delayed response.
>
> I quickly looked through the code. From pure Java/XML binding point, now
> that you looked a bit into the JAXB annotations, do you see something
> that you need but missing there?
> JBossXB annotations based binding is based on JAXB2.0, although it has a
> number of its own annotations and features that allow to go a bit beyond
> the standard. But also JBossXB doesn't support everything in JAXB yet.
> Just because no one asked for it.
>
> Another thing, JBossXB doesn't support marshalling for binding based on
> Java annotations. We also don't generate schemas and documentation.
> So, if you decide to switch to JAXB/JBossXB annotations and decide to
> also write some tools like that, it would be great if you contributed
> them to JBossXB.
>
>
> Alexey
>
> Vladimir Blagojevic wrote:
>
>> On 7/17/09 5:16 PM, Bill Burke wrote:
>>
>>> Wouldn't it be better to write a javadoc-like generator that
>>> introspected for JAXB and Inifispan annotations instead of writing your
>>> own XML parser and schema generator? Or maybe I'm misunderstanding your
>>> post.
>>>
>>> In other words, use JAXB for marshalling/unmarshalling/schema
>>> generation. Use inifinispan javadoc generator for documentation.
>>>
>>>
>> I looked around a bit more to see that now in JAXB 2.x we have
>> annotations as well :) If we annotate our configuration beans with JAXB
>> annotations I suppose we can do exactly what you suggest - use JAXB for
>> marshalling/unmarshalling/schema generation and at the same time use
>> those annotation in a custom tool that creates reference documentation.
>>
>>> By creating your own XML parsing you've just complicated the maintenance
>>> problems for future maintainers of infinispan.
>>>
>>>
>> I agree.
>>
>> Cheers,
>> Vladimir
>> _______________________________________________
>> jboss-development mailing list
>> jboss-development(a)lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/jboss-development
>>
> _______________________________________________
> jboss-development mailing list
> jboss-development(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/jboss-development
>
15 years, 4 months
JAXB based configuration reading and schema creation
by Vladimir Blagojevic
Ok guys I am ready to commit this change. This is a large commit and
currently I have no svn collisions. Test suite passes 100% and I am
confident that we might experience only minor hiccups after the commit.
I had to slightly change configuration file. From now on we have to wrap
all list of instances of <property> element with <properties> element.
We cannot have it, like we do now, a little bit of variation where we
use <properties> wrapper and how sometimes we don't.
Things that remain to be done are:
- figure out how we can deal with CacheMode enumeration of options so
that users do not have to spell out REPL_SYNC, REPL_ASYNC in
configuration files but something more user friendly
- convert Mircea's conversion tests to conform to a new format (I
disabled those tests for now)
- remove all files related to previous configuration parsing
For now, I created schema generator task in tools project and I manually
copy created schema to old location. We'll figure out how to automate
this completely soon.
Regards,
Vladimir
15 years, 4 months
[ISPN-31]JPA based cache store
by Alejandro Montenegro
Hi guys,some thought about this issue. Mainly there are two option for the
implementation:
1- Objects stored in cache are by themselves Entites (marked with an
@Entity)
2- Create a kind of wrapper Entity for non-entities objects
[1] Are more straight forward and should perform better, but are restricted
to Enities
[2] Are more complex to develop, as have to figure out a good structure fot
the wrapper, but satifies a lot of more use cases.
--
Alejandro Montenegro del Pino.
Viña del Mar - Chile
phone: (+56) 9-68358690
15 years, 5 months
JOPR and managing Infinispan
by Manik Surtani
So while in Stuttgart presenting at the JUG there, I had a chance to
catch up with Heiko and discuss some details of upcoming management/
admin features for Infinispan.
Immediately, we have a JOPR plugin that allows JOPR to manage
Infinispan instances.
Where this does need to improve though is that the JOPR plugin is
built with an XML definitions file that is hand written.
* This should be generated as a part of the build process, by parsing
@ManagedAttribute and@ManagedOperation annotations in Infinispan.
* @ManagedAttribute needs to contain more information:
* Type. JOPR has a concept of different types:
* informational types (labels that cannot be compared or measured),
e.g., Version number
* measurable types (numeric) that can be compared, plotted on a
graph, visualised in other ways, e.g., cache size
* two sub-types of measurable types: incrementing and decrementing,
e.g., number of commits
* a timing-type, (represented in milliseconds as a long), e.g.,
average replication time
* This should be encapsulated as an enum and declared on each managed
attribute so JOPR can provide better visuals on these
* Some metrics need to be resettable to be meaningful, and the JOPR
plugin would need to reset these whenever the metrics are collected.
E.g., average replication time would need to be reset and recalculated
every time JOPR picks up the last average and plots it on a graph.
* @ManagedAttribute also needs to be able to describe units of
measurement
* JOPR has some allowed values here and these should be encapsulated
in an enum. E.g., milliseconds. Allows JOPR to display the
appropriate widgets based on the unit of data being displayed.
Thoughts and comments on these, before I start writing up JIRAs for
the above?
Cheers
--
Manik Surtani
manik(a)jboss.org
Lead, Infinispan
Lead, JBoss Cache
http://www.infinispan.org
http://www.jbosscache.org
15 years, 5 months
Maven dependency and java runtime
by Vladimir Blagojevic
Hi,
So how do I specify dependency that is triggered only on certain java
runtimes? Say, jaxb is available in java 6 and not java 5 so if user is
running java 5 we should download jaxb dependencies.
Thanks,
Vladimir
15 years, 5 months
