Distributed index?
by Radim Vansa
Hi,
as we've discovered some imperfections in current distributed index
implementation, I'd like to know whether it could be possible to store
on each node only index of those entries that are primary-owned on that
node. Then, each query would be broadcast to other nodes and the results
would be merged.
From what I understood from Coherence documentation, they do that this
way - this seems quite reasonable to me, and does not introduce any
bottleneck as our index-master node (and also it does not require any
synchronization on shared index). It's also different from sharding
which introduces multiple indices but shares the index across nodes.
I can easily imagine simple ... WHERE x = 'y' queries, ORDER BY or
projections wouldn't be complicated either (unless sorting by
non-projected field). Effective offsets and limits would require a bit
more work, but the simplistic implementation (non-distributed merge)
shouldn't be hard either.
Could this approach be used with Lucene easily, or are there any caveats?
Radim
--
Radim Vansa <rvansa(a)redhat.com>
JBoss DataGrid QA
9 years, 9 months
Early Access build for JDK 9 b27 is available on java.net
by Rory O'Donnell Oracle, Dublin Ireland
Hi Galder,
Early Access build for JDK 9 b27 <https://jdk9.java.net/download/> is
available on java.net, summary of changes here
<http://www.java.net/download/jdk9/changes/jdk9-b27.html>
I'd also like to use this opportunity to point you to ongoing work in
OpenJDK on Project Jigsaw.
- JDK 9's source code is now modular:
http://mail.openjdk.java.net/pipermail/jdk9-dev/2014-August/001220.html
- Mark Reinhold's post providing some context is available on his
blog: http://mreinhold.org/blog/jigsaw-phase-two
- The first two Project Jigsaw JEPs have been posted at
http://openjdk.java.net/jeps/200 & http://openjdk.java.net/jeps/201 .
You can also track the progress on the JEPs in the JDK Bug System now -
the corresponding JBS issue for JEP 201 is
https://bugs.openjdk.java.net/browse/JDK-8051619 , for example.
Comments, questions, and suggestions are welcome on the jigsaw-dev
mailing list. (If you haven’t already subscribed to that list then
please do so first, otherwise your message will be discarded as spam.)
Rgds,Rory
--
Rgds,Rory O'Donnell
Quality Engineering Manager
Oracle EMEA , Dublin, Ireland
9 years, 9 months
[Search] @Transformable vs @ProvidedId
by Sanne Grinovero
There are two annotations clashing for same responsibilities:
- org.infinispan.query.Transformable
- org.hibernate.search.annotations.ProvidedId
as documented at the following link, these two different ways to apply "Id
indexing options" in Infinispan Query, IMHO quite unclear when a user
should use one vs. the other.
-
http://infinispan.org/docs/7.0.x/user_guide/user_guide.html#_requirements...
The benefit of @Transformable is that Infinispan provides one out of the
box which will work for any user case: it will serialize the whole object
representing the id, then hex-encode the buffer into a String: horribly
inefficient but works on any serializable type.
@ProvidedId originally marked the indexed entry in such a way that the
indexing engine would consider the id "provided externally", i.e. given at
runtime. It would also assume that its type would be static for a specific
type - which is I think a reasonable expectation but doesn't really hold as
an absolute truth in the case of Infinispan: nothing prevents me to store
an indexed entry of type "Person" for index "personindex" with an Integer
typed key in the cache, and also duplicate the same information under a say
String typed key.
So there's an expectation mismatch: in ORM world the key type is strongly
related to the value type, but when indexing Infinispan entries the reality
is that we're indexing two independent "modules".
I was hoping to drop @ProvidedId today as the original "marker"
functionality is no longer needed: since we have
org.hibernate.search.cfg.spi.SearchConfiguration.isIdProvidedImplicit()
the option can be implicitly applied to all indexed entries, and the
annotation is mostly redundant in Infinispan since we added this.
But actually it turns out it's a bit more complex as it servers a second
function as well: it's the only way for users to be able to specify a
FieldBridge for the ID.. so the functionality of this annotation is not
consumed yet.
So my proposal is to get rid of both @Transformable and @ProvidedId. There
needs to be a single way in Infinispan to define both the indexing options
and transformation; ideally this should be left to the Search Engine and
its provided collection of FieldBridge implementations.
Since the id type and the value type in Infinispan are not necessarily
strongly related (still the id is unique of course), I think this option
doesn't even belong on the @Indexed value but should be specified on the
key type.
Problem is that to define a class-level annotation to be used on the
Infinispan keys doesn't really belong in the collection of annotations of
Hibernate Search; I'm tempted to require that the key used for the type
must be one of those for which an out-of-the-box FieldBridge is provided:
the good thing is that now the set is extensible. In a second phase
Infinispan could opt to create a custom annotation like @Transformable to
register these options in a simplified way.
Even more, I've witnessed cases in which in Infinispan it makes sense to
encode some more information in the key than what's strictly necessary to
identify the key (like having attributes which are not included in the
hashcode and equals definitions). It sounds like the user should be allowed
to annotate the Key types, to allow such additional properties to
contribute to the index definition.
Comments welcome, but I feel strongly that these two annotations need to be
removed to make room for better solutions: we have an opportunity now as
I'm rewriting the mapping engine.
Sanne
9 years, 9 months
Fwd: Infinispan
by Radim Vansa
Hi Galder,
as HotRod protocol is portable and this functionality will be eventually
implemented in other languages, how is the marshalling of parameters to
the factory supposed to work?
Radim
-------- Original Message --------
Subject: Infinispan
Date: Thu, 21 Aug 2014 04:09:47 +0000
From: Infinispan <manik.surtani(a)gmail.com>
To: rvansa(a)redhat.com
Infinispan
Infinispan <http://blog.infinispan.org/>
------------------------------------------------------------------------
Hot Rod Remote Events #2: Filtering events
<http://feedproxy.google.com/%7Er/Infinispan/%7E3/CmE_v40pVRk/hot-rod-remo...>
Posted: 20 Aug 2014 08:19 AM PDT
This blog post is the second in a series that looks at the forthcoming
Hot Rod Remote Events functionality included in Infinispan 7.0. In the
first blog post
<http://blog.infinispan.org/2014/08/hot-rod-remote-events-1-getting-starte...>
we looked at how to get started receiving remote events from Hot Rod
servers. This time we are going to focus on how to filter events
directly in the server.
Sending events to remote clients has a cost which increases as the
number of clients. The more clients register remote listeners, the more
events the server has to send. This cost also goes up as the number of
modifications are executed against the cache. The more cache
modifications, the more events that need to be sent.
A way to reduce this cost is by filtering the events to send
server-side. If at the server level custom code decides that clients are
not interested in a particular event, the event does not even need to
leave the server, improving the overall performance of the system.
Remote event filters are created by implementing a
org.infinispan.filter.KeyValueFilterFactory class. Each factory must
have a name associated to it via the org.infinispan.filter.NamedFactory
annotation.
When a listener is added, we can provide the name of a key value filter
factory to use with this listener, and when the listener is added, the
server will look up the factory and invoke getKeyValueFilter method to
get a org.infinispan.filter.KeyValueFilter class instance to filter
events server side.
Filtering can be done based on key or value information, and even based
on cached entry metadata. Here's a sample implementation which will
filter key "2" out of the events sent to clients:
Plugging the server with this key value filter requires deploying this
filter factory (and associated filter class) within a jar file including
a service definition inside the
META-INF/services/org.infinispan.filter.KeyValueFilterFactory file:
With the server plugged with the filter, the next step is adding a
remote client listener that will use this filter. For this example,
we'll extend the EventLogListener implementation provided in the first
blog post
<http://blog.infinispan.org/2014/08/hot-rod-remote-events-1-getting-starte...>
in the series and we override the @ClientListener annotation to indicate
the filter factory to use with this listener:
Next, we add the listener via the RemoteCache API and we execute some
operations against the remote cache:
If we checkout the system output we'll see that the client receives
events for all keys except those that have been filtered:
Finally, with Hot Rod remote events we have tried to provide additional
flexibility at the client side, which is why when adding client
listeners, users can provide parameters to the filter factory so that
filter instances with different behaviours can be generated out of a
single filter factory based on client side information. To show this in
action, we are going to enhance the filter factory above so that instead
of filtering on a statically given key, it can filter dynamically based
on the key provided when adding the listener. Here's the revised version:
Finally, here's how we can now filter by "3" instead of "2":
And the output:
To summarise, we've seen how Hot Rod remote events can be filtered
providing key/value filter factories that can create instances that
filter which events are sent to clients, and how these filters can act
on client provided information.
In the next blog post, we'll look at how to customize remote events in
order to reduce the amount of information sent to the clients, or on the
contrary, provide even more information to our clients.
Cheers,
Galder
You are subscribed to email updates from Infinispan
<http://blog.infinispan.org/>
To stop receiving these emails, you may unsubscribe now
<http://feedburner.google.com/fb/a/mailunsubscribe?k=oCjJAXg1MUgjCxUM7TkP1...>.
Email delivery powered by Google
Google Inc., 20 West Kinzie, Chicago IL USA 60610
9 years, 9 months
Removal of ConfigurationBuilder.classLoader(...)
by Paul Ferraro
It seems that the ability to associate a cache with a specific classloader has been removed in 7.0 by this commit:
https://github.com/infinispan/infinispan/commit/39a21a025db2e0f85019b93d0...
I don't fully understand the reason for the removal. WildFly previously relied on this mechanism to define the classloader from which Infinispan should load any classes when building its configuration. In general, WF builds its configuration using object instances instead of class names, so normally this isn't a problem. However, there isn't always such a mechanism (e.g. https://issues.jboss.org/browse/ISPN-3979)
However, now that ConfigurationBuilder.classloader(...) is gone, the classloader used to build a Configuration is effectively hardcoded (usually as this.getClass().getClassLoader()).
This directly affects the ability for a WildFly using to configure a cache with querying. IndexingConfigurationBuilder.validate(...) previously used the configured classloader to validate that the query module is loadable.
https://github.com/infinispan/infinispan/blob/6.0.x/core/src/main/java/or...
However, this is now hardcoded to use the classloader that loaded the IndexingConfigurationBuilder class itself.
https://github.com/infinispan/infinispan/blob/master/core/src/main/java/o...
The WF distribution uses distinct modules for infinispan-core vs infinispan-query. Consequently, if your cache don't need querying, the query module is not loaded. WF8 let the user configure a cache with query support via <distributed-cache ... module="org.infinispan.query"/>.
Currently, however, the only way we can satisfy the validation logic in IndexingConfigurationBuilder.validate(...) is to bloat our core "org.infinispan" module with the infinispan-query module and its dependencies. I don't want to do that. Is there some way we can re-enable the ability to configure a cache with a classloader that still satisfies the reasons for its original removal? GlobalConfigurationBuilder still supports the ability to configure a classloader, why remove this from ConfigurationBuilder?
That said, however, the IndexingConfigurationBuilder validation itself is wrong. Ultimately, the infinispan-query module will be loaded by the classloader with which the GlobalConfiguration was built (i.e. the default classloader of the cache), so really, at the very least, the validation logic in IndexingConfigurationBuilder.validate(...) should reflect this. I've opened https://issues.jboss.org/browse/ISPN-4639 to track this specific bug.
Thoughts?
9 years, 9 months
Hot Rod not starting
by Gerard Fulton
We are having issues with Hot Rod when we stop and stop a member in a four
node cluster. The server will go through the start up process but then
throw a StateTransfer timeout exception. When this issue happens the client
port never binds on the node and we are unable to successfully start the
node and have it join the cluster until all nodes in the cluster are
restarted. Also note the problem cascades if any of the other nodes are
restarted in the cluster. They to are not able to rejoin the cluster
successfully.
I have posted my logs, configuration, and tcpdump capture on the form.
Form Link: https://community.jboss.org/thread/243451
Any help is appreciated.
Gerard
9 years, 9 months
Log message categories
by Tristan Tarrant
Dear all,
currently the Infinispan log messages "fall" in the categories named
from the originating class. While this isfine for TRACE/DEBUG messages,
there are some high-level INFO events which warrant their own specific
categories. I think that user-triggered events (such as JMX ops) should
also be treated like this.
Examples:
org.infinispan.CLUSTER (for important view change, state transfer and
rebalancing messages)
org.infinispan.CACHE (for cache lifecycle events)
org.infinispan.PERSISTENCE
What do you think ?
Any other suggestions ?
Tristan
9 years, 9 months
Re: [infinispan-dev] DIST-SYNC, put(), a problem and a solution
by Bela Ban
Seems like this discussion has died with the general agreement that this
is broken and with a few proposals on how to fix it, but without any
follow-up action items.
I think we (= someone from the ISPN team) need to create a JIRA,
preferably blocking.
WDYT ?
If not, here's what our options are:
#1 I'll create a JIRA
#2 We'll hold the team meeting in Krasnojarsk, Russia
#3 There will be only vodka, no beers in #2
#4 Bela will join the ISPN team
Thoughts ?
--
Bela Ban, JGroups lead (http://www.jgroups.org)
9 years, 9 months
