[JBoss JIRA] (WFLY-4805) Distributable app causes TransferQueueBundler Invalid argument and Topology errors
by Romain Pelisse (JIRA)
[ https://issues.jboss.org/browse/WFLY-4805?page=com.atlassian.jira.plugin.... ]
Romain Pelisse reassigned WFLY-4805:
------------------------------------
Assignee: Romain Pelisse (was: Paul Ferraro)
> Distributable app causes TransferQueueBundler Invalid argument and Topology errors
> ----------------------------------------------------------------------------------
>
> Key: WFLY-4805
> URL: https://issues.jboss.org/browse/WFLY-4805
> Project: WildFly
> Issue Type: Bug
> Components: Clustering
> Affects Versions: 10.0.0.Alpha4
> Environment: Oracle JDK 1.8, Open JDK 1.8, RHEL 7
> Reporter: Michal Karm Babacek
> Assignee: Romain Pelisse
> Attachments: clusterbench.war, configs.zip, logs.zip
>
>
> Simple shutdown failover tests are failing due to apparent error in / misconfiguration of the distributed cache.
> If one follows the aforementioned steps to reproduce, the following symptoms appear:
> * Variations on {noformat}SEVERE [org.jgroups.protocols.UDP] (TransferQueueBundler,ee,rhel7gax86-64) JGRP000029: rhel7gax86-64: failed sending message to rhel7gax86-64 (59 bytes): java.io.IOException: Invalid argument, headers: UNICAST3: ACK, seqno=9410, ts=217, UDP: [cluster_name=ee]{noformat}
> * Failures of this kind: {noformat}ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (transport-thread--p2-t12) ISPN000136: Execution error: org.infinispan.util.concurrent.TimeoutException: Timed out waiting for topology 11
> at org.infinispan.statetransfer.StateTransferLockImpl.waitForTransactionData(StateTransferLockImpl.java:92)
> at org.infinispan.interceptors.base.BaseStateTransferInterceptor.waitForTransactionData(BaseStateTransferInterceptor.java:96)
> at org.infinispan.statetransfer.StateTransferInterceptor.handleTxWriteCommand(StateTransferInterceptor.java:285)
> at org.infinispan.statetransfer.StateTransferInterceptor.handleWriteCommand(StateTransferInterceptor.java:254)
> at org.infinispan.statetransfer.StateTransferInterceptor.visitRemoveCommand(StateTransferInterceptor.java:130)
> at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:58)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
> at org.infinispan.interceptors.CacheMgmtInterceptor.visitRemoveCommand(CacheMgmtInterceptor.java:209)
> at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:58)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:102)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:71)
> at org.infinispan.commands.AbstractVisitor.visitRemoveCommand(AbstractVisitor.java:49)
> at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:58)
> at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:336)
> at org.infinispan.cache.impl.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1617)
> at org.infinispan.cache.impl.CacheImpl.removeInternal(CacheImpl.java:579)
> at org.infinispan.cache.impl.CacheImpl.remove(CacheImpl.java:572)
> at org.infinispan.cache.impl.DecoratedCache.remove(DecoratedCache.java:442)
> at org.infinispan.cache.impl.AbstractDelegatingCache.remove(AbstractDelegatingCache.java:297)
> at org.wildfly.clustering.server.registry.CacheRegistry.topologyChanged(CacheRegistry.java:152)
> at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at org.infinispan.notifications.impl.AbstractListenerImpl$ListenerInvocationImpl$1.run(AbstractListenerImpl.java:286)
> at org.infinispan.util.concurrent.WithinThreadExecutor.execute(WithinThreadExecutor.java:22)
> at org.infinispan.notifications.impl.AbstractListenerImpl$ListenerInvocationImpl.invoke(AbstractListenerImpl.java:309)
> at org.infinispan.notifications.cachelistener.CacheNotifierImpl$BaseCacheEntryListenerInvocation.doRealInvocation(CacheNotifierImpl.java:1180)
> at org.infinispan.notifications.cachelistener.CacheNotifierImpl$BaseCacheEntryListenerInvocation.invoke(CacheNotifierImpl.java:1139)
> at org.infinispan.notifications.cachelistener.CacheNotifierImpl$BaseCacheEntryListenerInvocation.invoke(CacheNotifierImpl.java:1105)
> at org.infinispan.notifications.cachelistener.CacheNotifierImpl.notifyTopologyChanged(CacheNotifierImpl.java:560)
> at org.infinispan.statetransfer.StateTransferManagerImpl.doTopologyUpdate(StateTransferManagerImpl.java:201)
> at org.infinispan.statetransfer.StateTransferManagerImpl.access$000(StateTransferManagerImpl.java:45)
> at org.infinispan.statetransfer.StateTransferManagerImpl$1.updateConsistentHash(StateTransferManagerImpl.java:113)
> at org.infinispan.topology.LocalTopologyManagerImpl.doHandleTopologyUpdate(LocalTopologyManagerImpl.java:285)
> at org.infinispan.topology.LocalTopologyManagerImpl$1.run(LocalTopologyManagerImpl.java:218)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at org.infinispan.executors.SemaphoreCompletionService$QueueingTask.runInternal(SemaphoreCompletionService.java:166)
> at org.infinispan.executors.SemaphoreCompletionService$QueueingTask.run(SemaphoreCompletionService.java:144)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745){noformat}
> * Warnings such as: {noformat}WARN [org.infinispan.transaction.impl.TransactionTable] (TxCleanupService,dist,rhel7gax86-64) ISPN000326: Remote transaction GlobalTransaction:<rhel7gax86-64>:9373:remote timed out. Rolling back after 70810 ms{noformat}
> Logs from one of such play & test scenarios are attached: [^logs.zip], [^configs.zip].
> Any ideas which configuration directive or application setting might be at the bottom of this?
> Needless to say any such test passes without any error with EAP 6.4 and 6.3 both with shutdown and undeploy failover scenarios.
> Thx for comments.
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
8 years, 5 months
[JBoss JIRA] (WFCORE-1330) Deployment error after reboot [WFLYSRV0137]
by Tobi Tobias (JIRA)
[ https://issues.jboss.org/browse/WFCORE-1330?page=com.atlassian.jira.plugi... ]
Tobi Tobias commented on WFCORE-1330:
-------------------------------------
Sorry for the doubled upload - I had problems with the JIRA upload...
> Deployment error after reboot [WFLYSRV0137]
> -------------------------------------------
>
> Key: WFCORE-1330
> URL: https://issues.jboss.org/browse/WFCORE-1330
> Project: WildFly Core
> Issue Type: Bug
> Components: Domain Management
> Affects Versions: 1.0.2.Final
> Environment: Ubuntu 14.04 LTS 64bit, jre-1.8.0_65
> Several deployed (JavaEE) web applications including non XA datasources that connect to a MySQL instance.
> Reporter: Tobi Tobias
> Assignee: ehsavoie Hugonnet
> Priority: Critical
> Attachments: server.log, server.log
>
>
> I have a working configuration on wildfly 9.0.2 final with several web applications (JavaEE).
> After a couple of days, I deployed another war file via jboss CLI. This application worked correctly and no deployment error occurred.
> But if I restart the server now, I get following error message:
> 10:36:01,893 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0013: Operation ("add") failed - address: ([("deployment" => "MM-Controller-0.1.0-SNAPSHOT.war")]) - failure description: "WFLYSRV0137: No deployment content with hash 966847a6f5f5bf8c3470f07ea9e65b7bbcdcd7b7 is available in the deployment content repository for deployment 'MM-Controller-0.1.0-SNAPSHOT.war'. This is a fatal boot error. To correct the problem, either restart with the --admin-only switch set and use the CLI to install the missing content or remove it from the configuration, or remove the deployment from the xml configuration file and restart."
> 10:36:01,990 FATAL [org.jboss.as.server] (Controller Boot Thread) WFLYSRV0056: Server boot has failed in an unrecoverable manner; exiting. See previous messages for details.
> I reproduced this at least 30 times - even with different jars. I always get this error. The server works fine as long as I don't reboot.
> The only way to fix the configuration is to manually remove the deployments from the standalone.xml.
> But this is not an option for me as I want to have the wildfly running as production server where I have several automatic deployments every day.
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
8 years, 5 months
[JBoss JIRA] (WFCORE-1330) Deployment error after reboot [WFLYSRV0137]
by Tobi Tobias (JIRA)
[ https://issues.jboss.org/browse/WFCORE-1330?page=com.atlassian.jira.plugi... ]
Tobi Tobias updated WFCORE-1330:
--------------------------------
Attachment: server.log
> Deployment error after reboot [WFLYSRV0137]
> -------------------------------------------
>
> Key: WFCORE-1330
> URL: https://issues.jboss.org/browse/WFCORE-1330
> Project: WildFly Core
> Issue Type: Bug
> Components: Domain Management
> Affects Versions: 1.0.2.Final
> Environment: Ubuntu 14.04 LTS 64bit, jre-1.8.0_65
> Several deployed (JavaEE) web applications including non XA datasources that connect to a MySQL instance.
> Reporter: Tobi Tobias
> Assignee: ehsavoie Hugonnet
> Priority: Critical
> Attachments: server.log, server.log
>
>
> I have a working configuration on wildfly 9.0.2 final with several web applications (JavaEE).
> After a couple of days, I deployed another war file via jboss CLI. This application worked correctly and no deployment error occurred.
> But if I restart the server now, I get following error message:
> 10:36:01,893 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0013: Operation ("add") failed - address: ([("deployment" => "MM-Controller-0.1.0-SNAPSHOT.war")]) - failure description: "WFLYSRV0137: No deployment content with hash 966847a6f5f5bf8c3470f07ea9e65b7bbcdcd7b7 is available in the deployment content repository for deployment 'MM-Controller-0.1.0-SNAPSHOT.war'. This is a fatal boot error. To correct the problem, either restart with the --admin-only switch set and use the CLI to install the missing content or remove it from the configuration, or remove the deployment from the xml configuration file and restart."
> 10:36:01,990 FATAL [org.jboss.as.server] (Controller Boot Thread) WFLYSRV0056: Server boot has failed in an unrecoverable manner; exiting. See previous messages for details.
> I reproduced this at least 30 times - even with different jars. I always get this error. The server works fine as long as I don't reboot.
> The only way to fix the configuration is to manually remove the deployments from the standalone.xml.
> But this is not an option for me as I want to have the wildfly running as production server where I have several automatic deployments every day.
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
8 years, 5 months
[JBoss JIRA] (WFCORE-1330) Deployment error after reboot [WFLYSRV0137]
by Tobi Tobias (JIRA)
[ https://issues.jboss.org/browse/WFCORE-1330?page=com.atlassian.jira.plugi... ]
Tobi Tobias updated WFCORE-1330:
--------------------------------
Attachment: server.log
> Deployment error after reboot [WFLYSRV0137]
> -------------------------------------------
>
> Key: WFCORE-1330
> URL: https://issues.jboss.org/browse/WFCORE-1330
> Project: WildFly Core
> Issue Type: Bug
> Components: Domain Management
> Affects Versions: 1.0.2.Final
> Environment: Ubuntu 14.04 LTS 64bit, jre-1.8.0_65
> Several deployed (JavaEE) web applications including non XA datasources that connect to a MySQL instance.
> Reporter: Tobi Tobias
> Assignee: ehsavoie Hugonnet
> Priority: Critical
> Attachments: server.log
>
>
> I have a working configuration on wildfly 9.0.2 final with several web applications (JavaEE).
> After a couple of days, I deployed another war file via jboss CLI. This application worked correctly and no deployment error occurred.
> But if I restart the server now, I get following error message:
> 10:36:01,893 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0013: Operation ("add") failed - address: ([("deployment" => "MM-Controller-0.1.0-SNAPSHOT.war")]) - failure description: "WFLYSRV0137: No deployment content with hash 966847a6f5f5bf8c3470f07ea9e65b7bbcdcd7b7 is available in the deployment content repository for deployment 'MM-Controller-0.1.0-SNAPSHOT.war'. This is a fatal boot error. To correct the problem, either restart with the --admin-only switch set and use the CLI to install the missing content or remove it from the configuration, or remove the deployment from the xml configuration file and restart."
> 10:36:01,990 FATAL [org.jboss.as.server] (Controller Boot Thread) WFLYSRV0056: Server boot has failed in an unrecoverable manner; exiting. See previous messages for details.
> I reproduced this at least 30 times - even with different jars. I always get this error. The server works fine as long as I don't reboot.
> The only way to fix the configuration is to manually remove the deployments from the standalone.xml.
> But this is not an option for me as I want to have the wildfly running as production server where I have several automatic deployments every day.
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
8 years, 5 months
[JBoss JIRA] (WFCORE-1330) Deployment error after reboot [WFLYSRV0137]
by Tobi Tobias (JIRA)
[ https://issues.jboss.org/browse/WFCORE-1330?page=com.atlassian.jira.plugi... ]
Tobi Tobias commented on WFCORE-1330:
-------------------------------------
I could successfully nailed the problem down to a server log. Please see attached log file.
I deployed two applications. Both have their own datasource. Both apps worked fine.
After deployment I requested a server shutdown via jboss-cli. -> worked fine
Then, I started the server again. The first log entries where "Removing content..." and then the server couldn't finish booting because of missing exactly these deployments.
> Deployment error after reboot [WFLYSRV0137]
> -------------------------------------------
>
> Key: WFCORE-1330
> URL: https://issues.jboss.org/browse/WFCORE-1330
> Project: WildFly Core
> Issue Type: Bug
> Components: Domain Management
> Affects Versions: 1.0.2.Final
> Environment: Ubuntu 14.04 LTS 64bit, jre-1.8.0_65
> Several deployed (JavaEE) web applications including non XA datasources that connect to a MySQL instance.
> Reporter: Tobi Tobias
> Assignee: ehsavoie Hugonnet
> Priority: Critical
>
> I have a working configuration on wildfly 9.0.2 final with several web applications (JavaEE).
> After a couple of days, I deployed another war file via jboss CLI. This application worked correctly and no deployment error occurred.
> But if I restart the server now, I get following error message:
> 10:36:01,893 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0013: Operation ("add") failed - address: ([("deployment" => "MM-Controller-0.1.0-SNAPSHOT.war")]) - failure description: "WFLYSRV0137: No deployment content with hash 966847a6f5f5bf8c3470f07ea9e65b7bbcdcd7b7 is available in the deployment content repository for deployment 'MM-Controller-0.1.0-SNAPSHOT.war'. This is a fatal boot error. To correct the problem, either restart with the --admin-only switch set and use the CLI to install the missing content or remove it from the configuration, or remove the deployment from the xml configuration file and restart."
> 10:36:01,990 FATAL [org.jboss.as.server] (Controller Boot Thread) WFLYSRV0056: Server boot has failed in an unrecoverable manner; exiting. See previous messages for details.
> I reproduced this at least 30 times - even with different jars. I always get this error. The server works fine as long as I don't reboot.
> The only way to fix the configuration is to manually remove the deployments from the standalone.xml.
> But this is not an option for me as I want to have the wildfly running as production server where I have several automatic deployments every day.
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
8 years, 5 months
[JBoss JIRA] (WFLY-5566) org.jboss.as.test.clustering.xsite.XSiteSimpleTestCase#testPutToBackupNotRelayed
by Radoslav Husar (JIRA)
[ https://issues.jboss.org/browse/WFLY-5566?page=com.atlassian.jira.plugin.... ]
Radoslav Husar reassigned WFLY-5566:
------------------------------------
Assignee: Richard Achmatowicz (was: Paul Ferraro)
> org.jboss.as.test.clustering.xsite.XSiteSimpleTestCase#testPutToBackupNotRelayed
> --------------------------------------------------------------------------------
>
> Key: WFLY-5566
> URL: https://issues.jboss.org/browse/WFLY-5566
> Project: WildFly
> Issue Type: Bug
> Components: Clustering, Test Suite
> Affects Versions: 10.0.0.CR2
> Reporter: Ivan Straka
> Assignee: Richard Achmatowicz
> Labels: clustering_revalidate
>
> There is org.jboss.as.test.clustering.xsite.XSiteSimpleTestCase#testPutToBackupNotRelayed failing with exception:
> {code:java}
> Error Message
> expected:<500> but was:<200>
> Stacktrace
> java.lang.AssertionError: expected:<500> but was:<200>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at org.junit.Assert.assertEquals(Assert.java:542)
> at org.jboss.as.test.clustering.xsite.XSiteSimpleTestCase.testPutToBackupNotRelayed(XSiteSimpleTestCase.java:202)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
8 years, 5 months