11:19 a.m.
Hi Dan,
thanks! Yes I concur: while this module seems to have some high leaks
I also suspect now that it's unrelated to the failure.
Looks like the "too many files" exception can be thrown for a wide
number of reasons,
FYI these tests fail "almost consistently" for me:
Failed tests:
ByteArrayKeyDistEmbeddedHotRodTest.setup:27 » IllegalState failed to
create a ...
ByteArrayKeyReplEmbeddedHotRodTest.setup:87 » IllegalState failed to
create a ...
ByteArrayValueDistEmbeddedHotRodTest.setup:27 » IllegalState failed
to create ...
ByteArrayValueReplEmbeddedHotRodTest.setup:87 » IllegalState failed
to create ...
DistEmbeddedHotRodBulkTest.setup:36 » IllegalState failed to create
a child ev...
DistEmbeddedRestHotRodTest.setup:25 » IllegalState failed to create
a child ev...
DistL1EmbeddedHotRodTest.setup:30 » IllegalState failed to create a
child even...
DistMemcachedEmbeddedTest.setup:39 » Transport Could not fetch transport
org.infinispan.it.compatibility.EmbeddedHotRodCacheListenerTest.setup(org.infinispan.it.compatibility.EmbeddedHotRodCacheListenerTest)
Run 1: EmbeddedHotRodCacheListenerTest.setup:36 » IllegalState
failed to create a chi...
Run 2: PASS
Run 3: PASS
EmbeddedMemcachedCacheListenerTest.setup:39 » IllegalState failed to
create a ...
EmbeddedRestMemcachedHotRodTest.setup:50 » IllegalState failed to
create a chi...
ReplEmbeddedRestHotRodTest.setup:38 » Channel Unable to create
Channel from cl...
Tests run: 103, Failures: 12, Errors: 0, Skipped: 49
If I run it a dozen times, I get the same report (exactly the same
tests failing) in 11 cases, and in one case it just passes all test.
I'm pretty sure I'm not running out of FDs anymore.
I'll open a JIRA and move on, as I just needed to verify some PRs but
need to switch focus to my own projects:
I hope someone with more Netty experience will be interested enough to
have a look, I can try reproducing
eventual fixes.
- https://issues.jboss.org/browse/ISPN-7517
Thanks all!
Sanne
On 23 February 2017 at 08:29, Dan Berindei <dan.berindei(a)gmail.com> wrote:
After looking into this some more...
TestNGTestListener does report 1262 leaked HotRodServerWorker and
MemcachedServer worker threads at the end of the compatibility-mode-it
test suite, so I'd say we probably do have a leak. The tests do seem
to shut down their servers at the end, maybe someone more familiar
with the server could look into why this happens? And maybe change the
thread naming scheme so that we can tell from the thread name which
test didn't shut down its servers properly?
But I'm pretty sure the leak isn't the problem here, compared to the
fact that we run 15 tests in parallel, AND each test starts 2
HotRod/Memcached/REST servers and 2 clients, AND they use the default
thread pool sizes instead of the 20x smaller thread pool sizes that we
use in the core suite.
Cheers
Dan
On Thu, Feb 23, 2017 at 9:56 AM, Vojtech Juranek <vjuranek(a)redhat.com> wrote:
> On středa 22. února 2017 23:31:55 CET Sanne Grinovero wrote:
>> Good point! You're right, I verified and all instances of e.g. the
>> jgroups jar were using the same FD, just a different thread id.
>
> yes, maybe there are some stale threads? Recently (well, actually it quite
> some time and it's still on my TODO list to investigate it more:-) testsuite
> started to fail with OOM issue "cannto create native thread" on some
machines
> (rarely on RHEL7, very often on RHEL6). I tried to increase limit on user
> threads and end up with unlimited, so this sound like some leak to me as well.
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev