[infinispan-dev] Classloader leaks?

[Sanne Grinovero]

Hi Dan,
thanks! Yes I concur: while this module seems to have some high leaks
I also suspect now that it's unrelated to the failure.

Looks like the "too many files" exception can be thrown for a wide
number of reasons,

FYI these tests fail "almost consistently" for me:

Failed tests:
  ByteArrayKeyDistEmbeddedHotRodTest.setup:27 » IllegalState failed to
create a ...
  ByteArrayKeyReplEmbeddedHotRodTest.setup:87 » IllegalState failed to
create a ...
  ByteArrayValueDistEmbeddedHotRodTest.setup:27 » IllegalState failed
to create ...
  ByteArrayValueReplEmbeddedHotRodTest.setup:87 » IllegalState failed
to create ...
  DistEmbeddedHotRodBulkTest.setup:36 » IllegalState failed to create
a child ev...
  DistEmbeddedRestHotRodTest.setup:25 » IllegalState failed to create
a child ev...
  DistL1EmbeddedHotRodTest.setup:30 » IllegalState failed to create a
child even...
  DistMemcachedEmbeddedTest.setup:39 » Transport Could not fetch transport
org.infinispan.it.compatibility.EmbeddedHotRodCacheListenerTest.setup(org.infinispan.it.compatibility.EmbeddedHotRodCacheListenerTest)
  Run 1: EmbeddedHotRodCacheListenerTest.setup:36 » IllegalState
failed to create a chi...
  Run 2: PASS
  Run 3: PASS

  EmbeddedMemcachedCacheListenerTest.setup:39 » IllegalState failed to
create a ...
  EmbeddedRestMemcachedHotRodTest.setup:50 » IllegalState failed to
create a chi...
  ReplEmbeddedRestHotRodTest.setup:38 » Channel Unable to create
Channel from cl...

Tests run: 103, Failures: 12, Errors: 0, Skipped: 49

If I run it a dozen times, I get the same report (exactly the same
tests failing) in 11 cases, and in one case it just passes all test.
I'm pretty sure I'm not running out of FDs anymore.

I'll open a JIRA and move on, as I just needed to verify some PRs but
need to switch focus to my own projects:
I hope someone with more Netty experience will be interested enough to
have a look, I can try reproducing
eventual fixes.

 - https://issues.jboss.org/browse/ISPN-7517

Thanks all!

Sanne







On 23 February 2017 at 08:29, Dan Berindei <dan.berindei at gmail.com> wrote:
> After looking into this some more...
>
> TestNGTestListener does report 1262 leaked HotRodServerWorker and
> MemcachedServer worker threads at the end of the compatibility-mode-it
> test suite, so I'd say we probably do have a leak. The tests do seem
> to shut down their servers at the end, maybe someone more familiar
> with the server could look into why this happens? And maybe change the
> thread naming scheme so that we can tell from the thread name which
> test didn't shut down its servers properly?
>
> But I'm pretty sure the leak isn't the problem here, compared to the
> fact that we run 15 tests in parallel, AND each test starts 2
> HotRod/Memcached/REST servers and 2 clients, AND they use the default
> thread pool sizes instead of the 20x smaller thread pool sizes that we
> use in the core suite.
>
> Cheers
> Dan
>
> On Thu, Feb 23, 2017 at 9:56 AM, Vojtech Juranek <vjuranek at redhat.com> wrote:
>> On středa 22. února 2017 23:31:55 CET Sanne Grinovero wrote:
>>> Good point! You're right, I verified and all instances of e.g. the
>>> jgroups jar were using the same FD, just a different thread id.
>>
>> yes, maybe there are some stale threads? Recently (well, actually it quite
>> some time and it's still on my TODO list to investigate it more:-) testsuite
>> started to fail with OOM issue "cannto create native thread" on some machines
>> (rarely on RHEL7, very often on RHEL6). I tried to increase limit on user
>> threads and end up with unlimited, so this sound like some leak to me as well.
>>
>> _______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev