Design change in Infinispan Query
by Sanne Grinovero
Hello all,
currently Infinispan Query is an interceptor registering on the
specific Cache instance which has indexing enabled; one such
interceptor is doing all what it needs to do in the sole scope of the
cache it was registered in.
If you enable indexing - for example - on 3 different caches, there
will be 3 different Hibernate Search engines started in background,
and they are all unaware of each other.
After some design discussions with Ales for CapeDwarf, but also
calling attention on something that bothered me since some time, I'd
evaluate the option to have a single Hibernate Search Engine
registered in the CacheManager, and have it shared across indexed
caches.
Current design limitations:
A- If they are all configured to use the same base directory to
store indexes, and happen to have same-named indexes, they'll share
the index without being aware of each other. This is going to break
unless the user configures some tricky parameters, and even so
performance won't be great: instances will lock each other out, or at
best write in alternate turns.
B- The search engine isn't particularly "heavy", still it would be
nice to share some components and internal services.
C- Configuration details which need some care - like injecting a
JGroups channel for clustering - needs to be done right isolating each
instance (so large parts of configuration would be quite similar but
not totally equal)
D- Incoming messages into a JGroups Receiver need to be routed not
only among indexes, but also among Engine instances. This prevents
Query to reuse code from Hibernate Search.
Problems with a unified Hibernate Search Engine:
1#- Isolation of types / indexes. If the same indexed class is
stored in different (indexed) caches, they'll share the same index. Is
it a problem? I'm tempted to consider this a good thing, but wonder if
it would surprise some users. Would you expect that?
2#- configuration format overhaul: indexing options won't be set on
the cache section but in the global section. I'm looking forward to
use the schema extensions anyway to provide a better configuration
experience than the current <properties />.
3#- Assuming 1# is fine, when a search hit is found I'd need to be
able to figure out from which cache the value should be loaded.
3#A we could have the cache name encoded in the index, as part
of the identifier: {PK,cacheName}
3#B we actually shard the index, keeping a physically separate
index per cache. This would mean searching on the joint index view but
extracting hits from specific indexes to keep track of "which index"..
I think we can do that but it's definitely tricky.
It's likely easier to keep indexed values from different caches in
different indexes. that would mean to reject #1 and mess with the user
defined index name, to add for example the cache name to the user
defined string.
Any comment?
Cheers,
Sanne
10 years, 2 months
singleton @Listeners
by Mircea Markus
This is a problem that pops up constantly:
User: "I add a listener to my distributed/replicated cache but this gets invoked numOwners times - can I make that to be invoked only once cluster wise?"
Developer: "Yes, you can! You have to do that and that..."
What about a "singleton" attribute on the Listener? Would make the reply shorter:
Developer: "Use @Listener(singleton=true)"
Cheers,
Mircea
10 years, 8 months
Re: [infinispan-dev] Removing Infinispan dependency on the Hibernate-Infinispan module in 4.x
by Galder Zamarreño
Scott, what do you suggest doing instead then? Without the commands, evictAll invalidation won't work.
Are you suggesting that I revert back to using the cache as a notification bus so that regions are invalidated?
On Feb 8, 2012, at 4:13 PM, Scott Marlow wrote:
> http://lists.jboss.org/pipermail/infinispan-dev/2012-February/010125.html has more context.
>
> Since there are no easy/quick fixes that can be applied at this time, to remove the AS7 Infinispan dependency on the Hibernate-Infinispan module, I think we should avoid depending on the service loader way to supply the custom commands (in the Hibernate-Infinispan module), at least until this can be addressed elsewhere.
>
> I propose that the Hibernate-Infinispan second level cache should not use the Service Loader to pass custom commands into Infinispan. If we agree, I'll create a jira for this.
>
> Scott
--
Galder Zamarreño
Sr. Software Engineer
Infinispan, JBoss Cache
10 years, 10 months
Testsuite error jiras
by Galder Zamarreño
When you open JIRAs to fix randomly failing tets, i.e. https://issues.jboss.org/browse/ISPN-2102, please make sure the test is disabled too.
This avoids confusion with the tests that have been identified and the ones that have not been.
Cheers,
--
Galder Zamarreño
Sr. Software Engineer
Infinispan, JBoss Cache
11 years, 5 months
Providing a context for object de-serialization
by Sanne Grinovero
Imagine I have a value object which needs to be stored in Infinispan:
class Person {
final String nationality = ...
final String fullName = ...
[constructor]
}
And now let's assume that - as you could expect - most Person
instances have the same value for the nationality String, but a
different name.
I want to define a custom Externalizer for my type, but the current
Externalizer API doesn't allow to refer to some common application
context, which might be extremely useful to deserialize this Person
instance:
we could avoid filling the memory of my Grid by having multiple copies
of the nationality String repeated all over, when a String [1] could
be reused.
Would it be a good idea to have the Externalizer instances have an
initialization phase receiving a ComponentRegistry, so I could look up
some custom service to de-duplicate or otherwise optimize my in-memory
data representation?
Personally I'd prefer to receive it injected via the constructor so
that I could use a final field when my custom Externalizer is
constructed.
This is OGM related.
Cheers,
Sanne
1 - or any immutable object: I'm using String as an example so let's
forget about the static String pool optimizations the JVM might
enable..
11 years, 9 months
IRC meeting log
by Galder Zamarreño
Meeting bot was not working (??), so find attached the log.
Cheers,
--
Galder Zamarreño
Sr. Software Engineer
Infinispan, JBoss Cache
11 years, 9 months
new module: integration tests
by Sanne Grinovero
In projects like Hibernate Search and OGM we added a new module
"integrationtests" where we can add specific tests needing non-core
dependencies, like booting the framework on AS7 via Arquillian or
using a custom TransactionManager.
I need now to make a test which requires both Infinispan Query and the
Infinispan Lucene Directory. If I add the dependency of one to the
other - even just as test scope - I'll be binding the dependent
version to each other which brings me in a circular dependency problem
with Hibernate Search (a bit because of how Maven works, a bit of how
Eclipse works...).
So are you all fine if I create a new module
/integrationtests/clusteredquerytests
?
We can then add more integration tests as siblings.
Cheers,
Sanne
11 years, 9 months
Dynamic data placement
by Pedro Ruivo
Hi all,
In the Cloud-TM project we are working on a technique to change
dynamically the distribution of data across the nodes of an ISPN data
grid. The idea is basically to determine at run-time which are the keys
most accessed by each member/node and move these keys to that nodes. The
ultimate goal is to maximize data locality and reduce the number of
remote get operation.
One of the ingredients of the solution is to define a mechanism that
allows the entire data grid, which is running in a configuration A,
having a given data distribution strategy (determined by the current
hash function implementation), to switch to a configuration B, in which
all nodes have agreed on a new data distribution strategy, and have
completed to relocate accordingly the keys.
Our idea therefore is to add a functionality in ISPN, which allows to
support the dynamic change of the data distribution criterion (namely
the consistent hash function).
To us this mechanism resembles closely the state transfer problem,
except that in the latter case the keys are relocated when a node joins
or leaves the system.
We were thinking to implement this by introducing a command (say,
RedistributedDataCommand) that would take as input a new hash function,
and that would internally trigger a state transfer based on the hash
function passed as input parameter. Another alternative would be to
change the CacheViewControlCommand, allowing it to distributed across
all the nodes the new hash function, as well as to trigger the state
transfer.
The point is that we would like to do this in a clean way, in order to
simplify a possible future integration. We will therefore be grateful
for any feedback that you will be able to provide.
Cheers,
Pedro Ruivo
11 years, 9 months
infinispan-arquillian-container - please review
by Martin Gencur
Hello,
I've just done a thorough clean up of this project and my changes along
with other commits (some of them are quite old now) are waiting at
https://github.com/infinispan/infinispan-arquillian-container/pull/18
I'm planning to release the first version of this project (so far it was
only as a snapshot 1.0.0-SNAPSHOT). Can anyone please review my commits
in the pull request? Previously, these reviews were done by Manik or
Pete but others are welcome as well.
I did not set versions to 1.0.0.Final or CRx so far as I'm not sure what
is the right approach for releasing such a project. Can someone give me
a hand with this release? I would like to push the maven artifacts to
jboss public maven repo afterwards.
For those who are not familiar with this framework, see
https://github.com/infinispan/infinispan-arquillian-container/blob/master...
We've been using it for testing of JDG during the last year.
Thanks a lot in advance
--
Martin Gencur
--
QA Engineer, JBoss Data Grid
Desk phone: +420 532 294 192, ext. 62192
11 years, 10 months
Unexpected partial state transfer
by Rusk, Patrick
We recently upgraded our Infinispan from 4.2 to 5.1.5. A behavior that we encountered afterwards was that some of our important replicated caches would only have partial state after their initial state transfer was complete. The rest of the state never seems to make it over, either.
Under 4.2, if any keys became available in the cache, they were all there. We have always considered full state transfer to be appropriate for our app because our caches are small and all of the data is needed for our rules engine calculations.
Is there any discussion of this change in behavior in the documentation or forums? Is it driven by parameters that we might be unaware of?
Thanks.
Pat
11 years, 10 months
