Fwd: Infinspan configuration Distributed Node with distributed index and HotRod usage
by Guillaume terral
Hello Infinispan Dev Team,
After several days of testing and research with Sanne Grinovero, he asked
me to send an email to this mailing list to expose you my issue with I
nfinispan.
I'm actually working on a project to build an infinispan cluster with the
following requirement:
- Clustered nodes in Distributed mode
- Usage of HotRod to send and query data from the cluster
- Low latency for read and write
- Distributed lucene index
- JGroups back-end
I posted the configuration i have for my cluster and for the hot rod client
on a github under:
https://github.com/guillaumeterral/infinispan-test
You will see a little readme explaining how to use this java project.
I faced some issues along my development,
What i noticed is:
If i remove the line *
.transaction().transactionMode(TransactionMode.NON_TRANSACTIONAL)*, I have
good latencies showed in the statistics MBean for averageWritingTime but I
get an unstable cluster with the following issues:
- lock issue, one node lock the index and try to write in it, at the same
time another node is trying to lock the index, this behavior lead to a lock
timeout.
- send message timeout, some node timeout to send message to others, i was
not able to understand the root code.
- compound file not found, this appears only if I use compound file
(default behavior) I think those errors are due to previous ones.
But with a non transactional cluster i get a pretty stable cluster but with
really bad latencies (~ 1.5s) showed in the cluster and in the GUI client i
wrote, this client is not present of GitHub I can push a console client on
demand if you want.
We tried to investigate the issue with Sanne, but across IRC it's not that
easy.
Could you provide me some help/advice, we are really motivated to use
Infinispan
and to give you some feedback on it, especially the HotRod client/server
component.
We plan to test Infinispan 7.0 and provide you some feedback when we will
complete our requirements on Infinispan 6.0 cluster.
You can join me at this mail adress at any time, or contact me on IRC
(#infinispan) my nickname is guillaume__ , i'm almost always connected on
it.
Thanks in advance,
Guillaume
10 years, 1 month
Disabling unstable tests: push on master directly?
by Sanne Grinovero
I'm blocked as usual on some core tests failing.
Since I've found evidence that it's not "just me", I'm using the new
cute annotation to mark them as unstable.
Would we all be good for me to push such findings - when I'm sure -
directly to master skipping the Pull Request process?
My reasoning is that:
- these are urgent as they block other work and rise FUD among other
occasional contributors
- you seem overwhelmed with PRs which deserve actual attention
- worst case you can revert: these commits can't go unnoticed as there
are JIRAs created to track these, so essentially it's not skipping
peer review but postponing it to a more suitable point in time.
Good idea?
Cheers,
Sanne
10 years, 2 months
Deprecating parent dependencies & migrating to JUnit
by Sanne Grinovero
TLDR: I'm banning TestNG from the Query modules, and this will affect
the parent poms.
--
I needed to run some quick experiment rounds in Query, but *some*
tests fail, and I simply want to know which ones to evaluate some
approaches.
But I can't get to know the full list easily, as failing a test in our
current testing framework apparently implies you'll be leaking
resources, and the suite slowly grinds to a slow death running out of
resources.
The problem in this, is when I need a quick evaluation which is
blocking other people in other teams, embarking in such witch hunting
is not an option; while I'm co-responsible of some of the embarassing
ugliness in our test's resource handling, I do primarily blame to
tooling rather than ourselves: especially TestNG's ridiculously
unreliable handling of shutdown hooks - and have enough of it.
Compared to that, the resource handling of JUnit is extremely handy
and reliable: requires a bit more coding as it provides less features
than TestNG, but can be relied on.
I can't really afford spending time to migrate all tests to JUnit
today, but I will start doing so in smaller iterations.
Now one problem is that TestNG, and several other dependencies, are
included by default in the parent classpath. Having a set of default
dependencies - even just in provided or test mode - forcefully
provided to all modules has created some other trouble too; for
example I really don't want the slf4j logger in my classpath as it has
hidden problems in classpath structure of our own modules.
## -> Would you all be ok if I move the dependencies to the dependency
management section?
(You can all use TestNG in your own modules if you want, but please
allow me to opt out)
That implies each module will have to explicitly pick what it needs; I
consider that a good thing as explicit is good, and getting utilities
on classpath from out-of-the-blue is confusing. For example, moving
just one dependency yesterday I could figure out that the migration to
Netty4 was actually incomplete and some tests where still using
Netty3. I consider that a proof of a more healthy structure.
Sanne
10 years, 2 months
Intermediate cache in M/R API
by Vladimir Blagojevic
Guys,
We need some input on how to design API regarding use of intermediate
caches [1]. As you might know one of the requirements for improving our
M/R is allowing applications to use custom defined intermediate
key/value cache used to store keys/values of map/combine phase before
being reduced in reduced phase.
Currently we have a constructor where one can specify whether to use
shared or per-task intermediate cache. And now we wanted to add an
additional method:
usingIntermediateCache(String cacheName, String cacheConfigurationName);
that will enable use of custom intermediate cache.
Now, Dan, and rightly so, thought this was a bit confusing. Are we
referring to intermediate shared or per-task intermediate cache when
using the above mentioned method.
His proposal is touse a per-task intermediate cache with our default
specified intermediate cache configuration. Remove the constructor
parameter in MapReduceTask regarding shared or non shared cache and add
configuration methods for both caches:
usingIntermediateCache(String configName) - use a per-task
intermediate cache with the given configuration
usingSharedIntermediateCache(String cache) - use a shared cache
with our default configuration
usingSharedIntermediateCache(String cache, String configName) - use
a shared cache with the given configuration
Note that we need a name for shared cache because we want to enable
application to easily remove/inspect that cache after all m/r tasks
sharing that intermediate cache have been executed.
What are your thoughts here?
Vladimir
[1] https://issues.jboss.org/browse/ISPN-4021
10 years, 2 months
