[jboss-jira] [JBoss JIRA] (WFLY-8843) Calling an EJB hangs at getPeerIdentity

Calling an EJB hangs at getPeerIdentity --------------------------------------- Key: WFLY-8843 URL: https://issues.jboss.org/browse/WFLY-8843 Project: WildFly Issue Type: Bug Components: EJB, Security Reporter: Richard Janík Assignee: David Lloyd Attachments: server-dump-1.txt, server-dump-2.txt, surefire-booter-dump-1.txt I'm looking up a stateless EJB and the call hangs indefinitely with current wildfly master. I'm doing the following: {code:java} @Before public void setUpAdminObjects() throws NamingException { server1Proxy = lookupAdminObject(TestSuiteEnvironment.getServerAddress(), "8080"); server2Proxy = lookupAdminObject(TestSuiteEnvironment.getServerAddress(), "8180"); Assert.assertNotNull(server1Proxy); Assert.assertNotNull(server2Proxy); } private DwmAdminObjectEjb lookupAdminObject(String address, String port) throws NamingException { Properties properties = new Properties(); properties.put(Context.INITIAL_CONTEXT_FACTORY, "org.jboss.naming.remote.client.InitialContextFactory"); properties.put(Context.PROVIDER_URL, String.format("%s%s:%s", "http-remoting://", address, port)); Context context = new InitialContext(properties); String ejbExportedName = String.format("%s/%s/%s!%s", "dwm-ejb-application", "dwm-ejb-module", DwmAdminObjectEjbImpl.class.getSimpleName(), DwmAdminObjectEjb.class.getCanonicalName()); return (DwmAdminObjectEjb) context.lookup(ejbExportedName); } {code} That hangs at {{context.lookup()}}. Note that I am aware that the old {{InitialContextFactory}} is deprecated, but the new one doesn't make any difference. I'll eventually replace it anyway. You can reproduce it with something like: {noformat} git remote add rjanik git@github.com:LittleJohnII/wildfly.git git fetch rjanik ; git checkout rjanik/eap7-495 mvn clean install -DskipTests ; cd testsuite/integration/manualmode mvn clean test -Dtest=Dwm*TestCase -Djboss.server.config.file.name=standalone-ha.xml -DtrimStackTrace=false {noformat} Warning: the issue looks somewhat random - I reproduce it on every test run, but sometimes one or two testcases pass before it hangs. I also took a thread dump of the two servers and the client that is started by the test. The server thread dumps contain nothing interesting to me, but the client dump shows that it is waiting to get the {{ConnectionPeerIdentity}}. {noformat} "main" #1 prio=5 os_prio=0 tid=0x00007fa824009000 nid=0x4ad8 waiting on condition [0x00007fa82a6f3000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000f7a68820> (a org.xnio.FutureResult$1) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at org.xnio.AbstractIoFuture.await(AbstractIoFuture.java:419) at org.xnio.AbstractIoFuture.get(AbstractIoFuture.java:535) at org.wildfly.naming.client.remote.SingleRemoteNamingProvider.getPeerIdentity(SingleRemoteNamingProvider.java:110) at org.wildfly.naming.client.remote.RemoteNamingProvider.getPeerIdentityForNaming(RemoteNamingProvider.java:69) at org.wildfly.naming.client.remote.RemoteContext.lambda$lookupNative$0(RemoteContext.java:109) at org.wildfly.naming.client.remote.RemoteContext$$Lambda$122/532677950.apply(Unknown Source) at org.wildfly.naming.client.NamingProvider.performExceptionAction(NamingProvider.java:115) at org.wildfly.naming.client.remote.RemoteContext.lookupNative(RemoteContext.java:108) at org.wildfly.naming.client.AbstractFederatingContext.lookup(AbstractFederatingContext.java:78) at org.wildfly.naming.client.AbstractFederatingContext.lookup(AbstractFederatingContext.java:64) at org.wildfly.naming.client.WildFlyRootContext.lookup(WildFlyRootContext.java:153) at javax.naming.InitialContext.lookup(InitialContext.java:417) at org.jboss.as.test.manualmode.jca.workmanager.distributed.AbstractDwmTestCase.lookupAdminObject(AbstractDwmTestCase.java:342) at org.jboss.as.test.manualmode.jca.workmanager.distributed.AbstractDwmTestCase.setUpAdminObjects(AbstractDwmTestCase.java:326) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24) at org.jboss.arquillian.junit.Arquillian$StatementLifecycleExecutor.invoke(Arquillian.java:463) at org.jboss.arquillian.container.test.impl.execution.ClientBeforeAfterLifecycleEventExecuter.execute(ClientBeforeAfterLifecycleEventExecuter.java:99) at org.jboss.arquillian.container.test.impl.execution.ClientBeforeAfterLifecycleEventExecuter.on(ClientBeforeAfterLifecycleEventExecuter.java:72) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:94) at org.jboss.arquillian.core.impl.EventContextImpl.invokeObservers(EventContextImpl.java:99) at org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:81) at org.jboss.arquillian.container.test.impl.client.ContainerEventController.createContext(ContainerEventController.java:142) at org.jboss.arquillian.container.test.impl.client.ContainerEventController.createBeforeContext(ContainerEventController.java:124) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:94) at org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:88) at org.jboss.arquillian.test.impl.TestContextHandler.createTestContext(TestContextHandler.java:130) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:94) at org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:88) at org.jboss.arquillian.test.impl.TestContextHandler.createClassContext(TestContextHandler.java:92) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:94) at org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:88) at org.jboss.arquillian.test.impl.TestContextHandler.createSuiteContext(TestContextHandler.java:73) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:94) at org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:88) at org.jboss.arquillian.core.impl.ManagerImpl.fire(ManagerImpl.java:145) at org.jboss.arquillian.core.impl.ManagerImpl.fire(ManagerImpl.java:116) at org.jboss.arquillian.test.impl.EventTestRunnerAdaptor.before(EventTestRunnerAdaptor.java:108) at org.jboss.arquillian.junit.Arquillian$4.evaluate(Arquillian.java:241) at org.jboss.arquillian.junit.Arquillian.multiExecute(Arquillian.java:426) at org.jboss.arquillian.junit.Arquillian.access$200(Arquillian.java:54) at org.jboss.arquillian.junit.Arquillian$5.evaluate(Arquillian.java:259) at org.jboss.arquillian.junit.Arquillian$7$1.invoke(Arquillian.java:319) at org.jboss.arquillian.container.test.impl.execution.ClientBeforeAfterLifecycleEventExecuter.execute(ClientBeforeAfterLifecycleEventExecuter.java:99) at org.jboss.arquillian.container.test.impl.execution.ClientBeforeAfterLifecycleEventExecuter.on(ClientBeforeAfterLifecycleEventExecuter.java:72) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:94) at org.jboss.arquillian.core.impl.EventContextImpl.invokeObservers(EventContextImpl.java:99) at org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:81) at org.jboss.arquillian.container.test.impl.client.ContainerEventController.createContext(ContainerEventController.java:142) at org.jboss.arquillian.container.test.impl.client.ContainerEventController.createBeforeContext(ContainerEventController.java:124) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:94) at org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:88) at org.jboss.arquillian.test.impl.TestContextHandler.createTestContext(TestContextHandler.java:130) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:94) at org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:88) at org.jboss.arquillian.test.impl.TestContextHandler.createClassContext(TestContextHandler.java:92) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:94) at org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:88) at org.jboss.arquillian.test.impl.TestContextHandler.createSuiteContext(TestContextHandler.java:73) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:94) at org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:88) at org.jboss.arquillian.core.impl.ManagerImpl.fire(ManagerImpl.java:145) at org.jboss.arquillian.core.impl.ManagerImpl.fire(ManagerImpl.java:116) at org.jboss.arquillian.test.impl.EventTestRunnerAdaptor.fireCustomLifecycle(EventTestRunnerAdaptor.java:159) at org.jboss.arquillian.junit.Arquillian$7.evaluate(Arquillian.java:312) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.jboss.arquillian.junit.Arquillian$2.evaluate(Arquillian.java:204) at org.jboss.arquillian.junit.Arquillian.multiExecute(Arquillian.java:426) at org.jboss.arquillian.junit.Arquillian.access$200(Arquillian.java:54) at org.jboss.arquillian.junit.Arquillian$3.evaluate(Arquillian.java:218) at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at org.jboss.arquillian.junit.Arquillian.run(Arquillian.java:166) at org.junit.runners.Suite.runChild(Suite.java:128) at org.junit.runners.Suite.runChild(Suite.java:27) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at org.junit.runner.JUnitCore.run(JUnitCore.java:137) at org.junit.runner.JUnitCore.run(JUnitCore.java:115) at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.createRequestAndRun(JUnitCoreWrapper.java:108) at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.executeEager(JUnitCoreWrapper.java:78) at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:54) at org.apache.maven.surefire.junitcore.JUnitCoreProvider.invoke(JUnitCoreProvider.java:144) at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:203) at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:155) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103) {noformat} I'll attach all the thread dumps too. Bisecting the commits in WFLY identified this commit as the culprit: {{deebbdf}} Upgrade WildFly Core to 3.0.0.Beta22. Notable upgrade in 3.0.0.Beta22 was Elytron. Because of that and because the client hangs at {{getPeerIdentity()}}, I added the security component. The same scenario works without issues prior to this commit, so I assume I'm not just doing things wrong. Even if I'm doing something wrong, which is possible, I believe the client shouldn't just hang, but should instead fire an exception. If you see any issues with my code I'll gladly fix them too. Not yet found in JBEAP, current version DR18 has wildfly-core Beta21, this will be present in DR19. The current master is 4a705cf89c90b5893b777fb0aa8188a01942a019.