test fine, but network exceptions are logged
by Sanne Grinovero
Hello,
I'd appreciate some advice to understand if my tests are wrong, or if
my environment is screwed.
Having ported some of Łukasz's tests for the Lucene Directory, I have 6 tests:
one test fails and others appear to work, but in all logs I see this
kind of errors:
2009-10-08 00:55:37,493 ERROR [org.jgroups.protocols.UDP]
(Timer-1,bluestar.tana-50817) failed sending message to null (138
bytes)
java.lang.Exception: dest=/228.10.10.10:45588 (141 bytes)
at org.jgroups.protocols.UDP._send(UDP.java:213)
when running "mvn clean test"
I get environment info:
~~~~~~~~~~~~~~~~~~~~~~~~~ ENVIRONMENT INFO ~~~~~~~~~~~~~~~~~~~~~~~~~~
bind.address = 127.0.0.1
java.runtime.version = 1.6.0_16-b01
java.runtime.name =Java(TM) SE Runtime Environment
java.vm.version = 14.2-b01
java.vm.vendor = Sun Microsystems Inc.
os.name = Linux
os.version = 2.6.30.8-64.fc11.x86_64
sun.arch.data.model = 64
sun.cpu.endian = little
protocol.stack = tcp
infinispan.marshaller.class = ${marshaller.class}
java.net.preferIPv4Stack = true
java.net.preferIPv6Stack = null
MAVEN_OPTS = null
~~~~~~~~~~~~~~~~~~~~~~~~~ ENVIRONMENT INFO ~~~~~~~~~~~~~~~~~~~~~~~~~~
So it appears the parent pom.xml is picked up correctly (as IPV4
preference and bind-address properties are defined there)
But if I run:
mvn clean test -Djava.net.preferIPv4Stack=true -Dbind.address=127.0.0.1
I get the same environment info but the tests behave differently: the
mentioned exceptions are not being logged any more,
but I get other kind of failures.
Also disabling the parallel execution of tests solves this, but
introduces other problems.
I can't understand the effect of setting these parameters on
infinispan/core, as many tests are failing already and have different
errors each time I relaunch the testsuite.
I really hate not being able to reproduce the errors, as they change
often, so it doesn't make much sense to post some stacktrace.
Is there any problem you might recognize in my environment?
Are tests in trunk having some known problem?
thanks in advance,
Sanne
16 years, 10 months
ComponentRegistry, rewiring and @NonVolatile
by Manik Surtani
I have spent some time with this over the weekend, and here is where I
think we stand:
The main reason why we needed the @NonVolatile annotation and the
ability to rewire components prior to starting is because JBC had a 2-
stage start sequence (create() followed by start()), and in between
these 2 stages the configuration could have changed. This doesn't
apply to Infinispan anymore.
The other reason this was important is to be able to restart a cache,
after stopping it. This still applies to Infinispan. I have
revisited the code and the reasons for rewiring. Since there is no
create phase anymore, we don't need to rewire every time a cache
starts, only if it had been stopped before. And as such, a lot fewer
components need to be marked as @NonVolatile - only a bare minimum
critical set of components that are needed to bootstrap the starting
of a cache after it has been stopped.
Also, I have renamed @NonVolatile to @SurvivesRestarts, to be a bit
more specific about its purpose, and also to break away from the old
intention of @NonVolatile in JBoss Cache 3.x. I have updated the
javadocs of @SurvivesRestarts accordingly as well.
Please have a look, and let me know what you think.
Cheers
--
Manik Surtani
manik(a)jboss.org
Lead, Infinispan
Lead, JBoss Cache
http://www.infinispan.org
http://www.jbosscache.org
16 years, 10 months
The reason(s) why we have so many threads are spawned in the testsuite
by Galder Zamarreno
This is a rather long but please bear with me:
So, I've been looking at the number of threads created in our testsuite
because quite often I'm getting OOM "unable to create thread" and note
that tests have evolved to be quite complex with several CacheManagers,
Caches...etc.
In summary, I see plenty of maxThread=1 ThreadPoolExecutor (TPE)
instances created that then are either useless (i.e. they get overriden
after rewiring) or are there even though they're not gonna be used in
the test (i.e. async listener notifier). Side note: Note that maxThread
is used for the core pool size which means that this means that there's
1 thread available.
For the useless TPEs, we need to avoid creating them in the 1st place.
For the useful but not used in test, we should have an option that
allows for such executors to be created lazily in a testsuite env. I see
the point of creating these in advance in a prod or user env but when
running the testsuite, it'd be better and would safe resources,
specially wrt to threads if we created them lazily.
Examples:
For each CacheManager created, 2 x TPEs for ASYNC_NOTIFICATION_EXECUTOR
are created. The second one during the re-wiring process. The result is
that the 1st one is never used any more and it's left sitting there. Why
is a 2nd one created? Something to do with non-volatility? Indeed it is.
The j.u.e.ExecutorService instance is a j.u.c.TPE which has no
NonVolatile annotation and hence it gets recreated. Note that this
happens for all TPE usages (ASYNC_NOTIFICATION_EXECUTOR,
ASYNC_TRANSPORT_EXECUTOR, EVICTION_SCHEDULED_EXECUTOR,
ASYNC_REPLICATION_QUEUE_EXECUTOR)! For each one created, there's another
one left there taking up resources!! So, what are the options? Two:
- A question that seems to come up more and more, get rid of the
Volatility/NonVolatility difference and treat everything as NonVolatile?
Apart from simplyfying the code, it'd make easier to debug cos figuring
out these type of issues are a royal PITA. I remember hearing Manik that
since create and start phases were gone, this should be no problem.
- Or provide our own ExecutorService and TPE facades that allows use to
annotate TPE as @NonVolatile.
For each Cache created, the same thing, 2 x TPE for
ASYNC_NOTIFICATION_EXECUTOR are created.
The lazyness I was refering about earlier comes into play here in the
sense that, why not start the TPE the 1st time we register an async
Listener? If we did so, we'd be saving resources if no async listeners
are enabled (This looks to me the most common case right now). I can see
the point of having this ready for normal use but certainly for
testsuite use, this is waste when we're creating so many CacheManagers,
Caches instances and have several threads running different tests at the
same time.
Would it make sense to apply such lazyness to ASYNC_TRANSPORT_EXECUTOR?
Hmmmm, even in a configured SYNC configuration, there might be stuff we
wanna send asynchronously, i.e. PFERs, so don't think so.
With regards to EVICTION_SCHEDULED_EXECUTOR, this is a cache component.
Currently, a TPE is created regardless of whether eviction is enabled or
not and eviction is not a dynamic property. Now, if eviction is set to
NONE, neither eviction nor expiration happens, correct? In that case, I
don't see why TPE should be maintained if eviction is disabled.
To finish with the lazy TPE startup topic, with
ASYNC_REPLICATION_QUEUE_EXECUTOR, we're certainly doing a good job. In
ReplicationQueueFactory, we only construct a ReplicationQueue and
corresponding TPE if replication queue is enabled
More comments now. Since there are 4 different usages of
ThreadPoolExecutor each should have, *by default*, a different thread
name prefix. Currently, there's no such thing which means that figuring
out what a particular thread belongs to is difficult unless
threadNamePrefix are set in each set of properties.
Another thing that is very confusing is the
DefaultExecutorFactory.counter. What is it trying to represent? For
example, in the testsuite thread dumps, I was seeing thread names like:
asyncThread-150 and this is very confusing when taking in account that
maxThreads in each ThreadPoolExecutor in the testsuite is 1.
Finally, KnownComponentNames should probably be an enum, shouldn't it?
If you made it this far, thx!!
--
Galder Zamarreño
Sr. Software Engineer
Infinispan, JBoss Cache
16 years, 10 months
jclouds wire logging examples
by Adrian Cole
Hi, guys.
I'm polishing up jclouds for release. I'd like some feedback on logging,
especially as relates to integration with other libraries. In this case,
I'm referring to wire logging, where we trace the inputs and outputs to the
cloud service into a separate log file. Wire logging is extremely important
when troubleshooting with cloud vendors, as they can read the stacks without
knowing java. Wire logging is course optional, and removes redundant content
logging from other parts of the code.
The design is largely borrowed from apache hc wire logging. It differs in
implementation, where we make it asynchronous and portable across http
engines. In other words, java, nio, and whatever engines we come up with
next will have identical output like below [1]. This specific logging is
controlled by categories jclouds.http.headers (for headers) and
jclouds.http.wire (for content). Note that these are not org.jclouds
categories on purpose, so as to avoid accidentally turning them on with
wildcards.
Please let me know what you think.
Cheers,
-Adrian
[1]
2009-10-10 00:44:37,566 DEBUG [jclouds.http.headers] (pool-1-thread-2) >>
POST
https://node1.nirvanix.com/Upload.ashx?output=json&uploadToken=40Z0PFwy%7...
rPath=container&sessionToken=61b0ca55-3782-44f8-bdf1-177a9d118d28 HTTP/1.1
2009-10-10 00:44:37,566 DEBUG [jclouds.http.headers] (pool-1-thread-2) >>
Content-Length: 141
2009-10-10 00:44:37,566 DEBUG [jclouds.http.headers] (pool-1-thread-2) >>
Content-Type: multipart/form-data; boundary=--JCLOUDS--
2009-10-10 00:44:37,751 DEBUG [jclouds.http.wire] (pool-1-thread-1) >>
"----JCLOUDS--[\r][\n]"
2009-10-10 00:44:37,751 DEBUG [jclouds.http.wire] (pool-1-thread-1) >>
"Content-Disposition: form-data; name="key"; filename="key"[\r][\n]"
2009-10-10 00:44:37,751 DEBUG [jclouds.http.wire] (pool-1-thread-1) >>
"Content-Type: application/octet-stream[\r][\n]"
2009-10-10 00:44:37,751 DEBUG [jclouds.http.wire] (pool-1-thread-1) >>
"[\r][\n]"
2009-10-10 00:44:37,751 DEBUG [jclouds.http.wire] (pool-1-thread-1) >>
"value[\r][\n]"
2009-10-10 00:44:37,751 DEBUG [jclouds.http.wire] (pool-1-thread-1) >>
"----JCLOUDS----[\r][\n]"
2009-10-10 00:44:37,926 DEBUG [jclouds.http.headers] (pool-1-thread-2) <<
HTTP/1.1 200 OK
2009-10-10 00:44:37,926 DEBUG [jclouds.http.headers] (pool-1-thread-2) <<
Content-Length: 152
2009-10-10 00:44:37,926 DEBUG [jclouds.http.headers] (pool-1-thread-2) <<
X-Powered-By: Nirvanix SDN
2009-10-10 00:44:37,926 DEBUG [jclouds.http.headers] (pool-1-thread-2) <<
X-Powered-By: ASP.NET
2009-10-10 00:44:37,926 DEBUG [jclouds.http.headers] (pool-1-thread-2) <<
Expires: -1
2009-10-10 00:44:37,926 DEBUG [jclouds.http.headers] (pool-1-thread-2) <<
X-AspNet-Version: 2.0.50727
2009-10-10 00:44:37,927 DEBUG [jclouds.http.headers] (pool-1-thread-2) <<
Date: Sat, 10 Oct 2009 07:46:25 GMT
2009-10-10 00:44:37,927 DEBUG [jclouds.http.headers] (pool-1-thread-2) <<
Pragma: no-cache
2009-10-10 00:44:37,927 DEBUG [jclouds.http.headers] (pool-1-thread-2) <<
Content-Type: text/xml; charset=utf-8
2009-10-10 00:44:37,927 DEBUG [jclouds.http.headers] (pool-1-thread-2) <<
Server: Nirvanix IMFS
2009-10-10 00:44:37,927 DEBUG [jclouds.http.headers] (pool-1-thread-2) <<
Server: Microsoft-IIS/6.0
2009-10-10 00:44:37,927 DEBUG [jclouds.http.headers] (pool-1-thread-2) <<
Cache-Control: no-cache
16 years, 10 months
refresh ahead - do we want this?
by Mircea Markus
For example, assume that the expiration time for entries in the cache
is set to 60 seconds and the refresh-ahead factor is set to 0.5. If
the cached object is accessed after 60 seconds, a synchronous read
from the cache store is performed to refresh its value. On the other
hand, if a request is performed for an entry that is more than 30 but
less than 60 seconds old, the current value in the cache is returned
and an asynchronous reload from the cache store is scheduled.
For frequently accessed data, requests won't block waiting for data to
be retrieved from persistent store if data expires.
Wdyt?
Cheers,
Mircea
16 years, 10 months
Generics usage in org.infinispan.util.ClassFinder
by Galder Zamarreno
Hi,
I'm a bit concerned about the usage of Class<?> in
org.infinispan.util.ClassFinder because in pretty much all those
methods, we can't say anything about the Class type returned and by
keeping Class<?>, you're forcing clients to deal with generics that
don't add anything.
Thoughts?
--
Galder Zamarreño
Sr. Software Engineer
Infinispan, JBoss Cache
16 years, 10 months
Issues in creation of Issues
by Sanne Grinovero
Hello,
I'm getting errors at JIRA creation, I've been trying both "Task" and
"New Feature Request" to track the inclusion of Łukcasz's work in
Infinispan.
After four hours, still the same error. Any maintenance going on?
BTW what do you think about creating a component (on JIRA) about the
Lucene Directory? I'm having several ideas about improvements yet to
implement, and also have already some bugs to track; would be cool to
group these issues together?
Cheers,
Sanne
16 years, 10 months
