[JBoss JIRA] (AS7-4957) Cancelling deployment update hangs deployment repository until server is restarted
by Kabir Khan (JIRA)
Kabir Khan created AS7-4957:
-------------------------------
Summary: Cancelling deployment update hangs deployment repository until server is restarted
Key: AS7-4957
URL: https://issues.jboss.org/browse/AS7-4957
Project: Application Server 7
Issue Type: Bug
Components: Domain Management
Reporter: Kabir Khan
Assignee: Kabir Khan
If I deploy a large deployment in CLI and cancel before it has completed uploading the lock on the deployment repository is never released since it hangs in PipedInputStream.read:
{code}
"management-handler-thread - 3" prio=10 tid=0x00007fd230006000 nid=0x3f51 in Object.wait() [0x00007fd20d9d7000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000d5d50078> (a java.io.PipedInputStream)
at java.io.PipedInputStream.read(PipedInputStream.java:327)
- locked <0x00000000d5d50078> (a java.io.PipedInputStream)
at org.jboss.as.controller.remote.OperationAttachmentsProxy$ProxiedInputStream.read(OperationAttachmentsProxy.java:120)
at java.io.InputStream.read(InputStream.java:170)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
- locked <0x00000000d5d520d8> (a java.io.BufferedInputStream)
at java.io.FilterInputStream.read(FilterInputStream.java:107)
at org.jboss.as.repository.ContentRepository$Factory$ContentRepositoryImpl.addContent(ContentRepository.java:149)
- locked <0x00000000daae77d8> (a java.security.MessageDigest$Delegate)
at org.jboss.as.domain.controller.operations.deployment.DeploymentUploadUtil.storeDeploymentContent(DeploymentUploadUtil.java:35)
at org.jboss.as.domain.controller.operations.coordination.OperationCoordinatorStepHandler.storeDeploymentContent(OperationCoordinatorStepHandler.java:251)
at org.jboss.as.domain.controller.operations.coordination.OperationCoordinatorStepHandler.storeDeploymentContent(OperationCoordinatorStepHandler.java:245)
at org.jboss.as.domain.controller.operations.coordination.OperationCoordinatorStepHandler.executeTwoPhaseOperation(OperationCoordinatorStepHandler.java:180)
at org.jboss.as.domain.controller.operations.coordination.OperationCoordinatorStepHandler.execute(OperationCoordinatorStepHandler.java:118)
at org.jboss.as.domain.controller.operations.coordination.PrepareStepHandler.execute(PrepareStepHandler.java:85)
at org.jboss.as.controller.AbstractOperationContext.executeStep(AbstractOperationContext.java:397)
at org.jboss.as.controller.AbstractOperationContext.doCompleteStep(AbstractOperationContext.java:284)
at org.jboss.as.controller.AbstractOperationContext.completeStep(AbstractOperationContext.java:211)
at org.jboss.as.controller.ModelControllerImpl.internalExecute(ModelControllerImpl.java:126)
at org.jboss.as.controller.ModelControllerImpl.execute(ModelControllerImpl.java:111)
at org.jboss.as.controller.remote.ModelControllerClientOperationHandler$ExecuteRequestHandler.doExecute(ModelControllerClientOperationHandler.java:139)
at org.jboss.as.controller.remote.ModelControllerClientOperationHandler$ExecuteRequestHandler$1.execute(ModelControllerClientOperationHandler.java:108)
at org.jboss.as.protocol.mgmt.AbstractMessageHandler$2$1.doExecute(AbstractMessageHandler.java:295)
at org.jboss.as.protocol.mgmt.AbstractMessageHandler$AsyncTaskRunner.run(AbstractMessageHandler.java:512)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
at org.jboss.threads.JBossThread.run(JBossThread.java:122)
{code}
Further attempts to deploy things hang here, meaning the server needs restarting before deployments can be made again:
{code}
"management-handler-thread - 7" prio=10 tid=0x00007fd230047800 nid=0x4002 waiting for monitor entry [0x00007fd20dbd9000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.jboss.as.repository.ContentRepository$Factory$ContentRepositoryImpl.addContent(ContentRepository.java:143)
- waiting to lock <0x00000000daae77d8> (a java.security.MessageDigest$Delegate)
at org.jboss.as.domain.controller.operations.deployment.DeploymentUploadUtil.storeDeploymentContent(DeploymentUploadUtil.java:35)
at org.jboss.as.domain.controller.operations.coordination.OperationCoordinatorStepHandler.storeDeploymentContent(OperationCoordinatorStepHandler.java:251)
at org.jboss.as.domain.controller.operations.coordination.OperationCoordinatorStepHandler.storeDeploymentContent(OperationCoordinatorStepHandler.java:245)
at org.jboss.as.domain.controller.operations.coordination.OperationCoordinatorStepHandler.executeTwoPhaseOperation(OperationCoordinatorStepHandler.java:180)
at org.jboss.as.domain.controller.operations.coordination.OperationCoordinatorStepHandler.execute(OperationCoordinatorStepHandler.java:118)
at org.jboss.as.domain.controller.operations.coordination.PrepareStepHandler.execute(PrepareStepHandler.java:85)
at org.jboss.as.controller.AbstractOperationContext.executeStep(AbstractOperationContext.java:397)
at org.jboss.as.controller.AbstractOperationContext.doCompleteStep(AbstractOperationContext.java:284)
at org.jboss.as.controller.AbstractOperationContext.completeStep(AbstractOperationContext.java:211)
at org.jboss.as.controller.ModelControllerImpl.internalExecute(ModelControllerImpl.java:126)
at org.jboss.as.controller.ModelControllerImpl.execute(ModelControllerImpl.java:111)
at org.jboss.as.controller.remote.ModelControllerClientOperationHandler$ExecuteRequestHandler.doExecute(ModelControllerClientOperationHandler.java:139)
at org.jboss.as.controller.remote.ModelControllerClientOperationHandler$ExecuteRequestHandler$1.execute(ModelControllerClientOperationHandler.java:108)
at org.jboss.as.protocol.mgmt.AbstractMessageHandler$2$1.doExecute(AbstractMessageHandler.java:295)
at org.jboss.as.protocol.mgmt.AbstractMessageHandler$AsyncTaskRunner.run(AbstractMessageHandler.java:512)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
at org.jboss.threads.JBossThread.run(JBossThread.java:122)
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
13 years, 11 months
[JBoss JIRA] (LOGTOOL-51) Logging generation processing breaks APT rules by loading classes via reflection
by Max Rydahl Andersen (JIRA)
[ https://issues.jboss.org/browse/LOGTOOL-51?page=com.atlassian.jira.plugin... ]
Max Rydahl Andersen commented on LOGTOOL-51:
--------------------------------------------
Meaning the issue will be ignored or ?
> Logging generation processing breaks APT rules by loading classes via reflection
> --------------------------------------------------------------------------------
>
> Key: LOGTOOL-51
> URL: https://issues.jboss.org/browse/LOGTOOL-51
> Project: Log Tool
> Issue Type: Bug
> Security Level: Public(Everyone can see)
> Environment: JBDS 5.5.0.CR1
> Reporter: Fred Bricon
> Assignee: David Lloyd
> Priority: Blocker
>
> JBDS 5.5.0.CR1 embeds [m2e-apt|https://github.com/jbosstools/m2e-apt], which automatically enables Eclipse JDT Annotation Processor Toolkit.
> When importing the [server|https://github.com/jbossas/jboss-as/tree/master/server] module from http://github.com/jbossas/jboss-as.git (also happens for other modules), Eclipse starts crashing (see JBIDE-12087) with errors like :
> {noformat}
> javax.xml.parsers.FactoryConfigurationError: Provider __redirected.__DocumentBuilderFactory not found
> at javax.xml.parsers.DocumentBuilderFactory.newInstance(DocumentBuilderFactory.java:127)
> at org.eclipse.jdt.apt.core.internal.util.FactoryPathUtil.decodeFactoryPath(FactoryPathUtil.java:222)
> at org.eclipse.jdt.apt.core.internal.util.FactoryPathUtil.readFactoryPathFile(FactoryPathUtil.java:115)
> at org.eclipse.jdt.apt.core.internal.util.FactoryPathUtil.calculatePath(FactoryPathUtil.java:342)
> at org.eclipse.jdt.apt.core.internal.util.FactoryPathUtil.getFactoryPath(FactoryPathUtil.java:429)
> at org.eclipse.jdt.apt.core.internal.AnnotationProcessorFactoryLoader.getJava6FactoriesAndAttributesForProject(AnnotationProcessorFactoryLoader.java:420)
> at org.eclipse.jdt.internal.apt.pluggable.core.dispatch.IdeAnnotationProcessorManager.processAnnotations(IdeAnnotationProcessorManager.java:130)
> at org.eclipse.jdt.internal.compiler.Compiler.processAnnotations(Compiler.java:813)
> at org.eclipse.jdt.internal.compiler.Compiler.compile(Compiler.java:432)
> at org.eclipse.jdt.internal.core.builder.AbstractImageBuilder.compile(AbstractImageBuilder.java:364)
> at org.eclipse.jdt.internal.core.builder.BatchImageBuilder.compile(BatchImageBuilder.java:178)
> at org.eclipse.jdt.internal.core.builder.AbstractImageBuilder.compile(AbstractImageBuilder.java:301)
> at org.eclipse.jdt.internal.core.builder.BatchImageBuilder.build(BatchImageBuilder.java:60)
> at org.eclipse.jdt.internal.core.builder.JavaBuilder.buildAll(JavaBuilder.java:254)
> at org.eclipse.jdt.internal.core.builder.JavaBuilder.build(JavaBuilder.java:173)
> at org.eclipse.core.internal.events.BuildManager$2.run(BuildManager.java:728)
> at org.eclipse.core.runtime.SafeRunner.run(SafeRunner.java:42)
> at org.eclipse.core.internal.events.BuildManager.basicBuild(BuildManager.java:199)
> at org.eclipse.core.internal.events.BuildManager.basicBuild(BuildManager.java:239)
> at org.eclipse.core.internal.events.BuildManager$1.run(BuildManager.java:292)
> at org.eclipse.core.runtime.SafeRunner.run(SafeRunner.java:42)
> at org.eclipse.core.internal.events.BuildManager.basicBuild(BuildManager.java:295)
> at org.eclipse.core.internal.events.BuildManager.basicBuildLoop(BuildManager.java:351)
> at org.eclipse.core.internal.events.BuildManager.build(BuildManager.java:374)
> at org.eclipse.core.internal.events.AutoBuildJob.doBuild(AutoBuildJob.java:143)
> at org.eclipse.core.internal.events.AutoBuildJob.run(AutoBuildJob.java:241)
> at org.eclipse.core.internal.jobs.Worker.run(Worker.java:54)
> Caused by: java.lang.ClassNotFoundException: __redirected/__DocumentBuilderFactory
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:264)
> at javax.xml.parsers.FactoryFinder.getProviderClass(FactoryFinder.java:119)
> at javax.xml.parsers.FactoryFinder.newInstance(FactoryFinder.java:144)
> at javax.xml.parsers.FactoryFinder.find(FactoryFinder.java:185)
> at javax.xml.parsers.DocumentBuilderFactory.newInstance(DocumentBuilderFactory.java:121)
> {noformat}
> It turns out the Eclipse System Properties have been changed to use javax.xml.parsers.SAXParserFactory=__redirected/__DocumentBuilderFactory
> I tracked the code responsible with messing with the properties :
> {noformat}
> Thread [Worker-24] (Suspended (breakpoint at line 779 in System))
> System.setProperty(String, String) line: 779
> __SAXParserFactory.<clinit>() line: 68
> __JAXPRedirected.initAll() line: 80
> Module$1.run() line: 126
> Module$1.run() line: 113
> AccessController.doPrivileged(PrivilegedAction<T>) line: not available [native method]
> Module.<clinit>() line: 113
> Class<T>.forName0(String, boolean, ClassLoader) line: not available [native method]
> Class<T>.forName(String) line: 186 <====== org.jboss.modules.Module
> JCodeModel.ref(String) line: 363
> MessageBundleImplementor(ImplementationClassModel).createBundleMethod(MessageMethod, JMethod, JMethod, JVar) line: 150
> MessageBundleImplementor.generateModel() line: 86
> MessageBundleImplementor(ClassModel).create(JavaFileObject) line: 104
> ImplementationClassGenerator.processTypeElement(TypeElement, TypeElement, MessageInterface) line: 63
> LoggingToolsProcessor.process(Set<TypeElement>, RoundEnvironment) line: 155
> RoundDispatcher.handleProcessor(ProcessorInfo) line: 139
> RoundDispatcher.round() line: 121
> IdeAnnotationProcessorManager(BaseAnnotationProcessorManager).processAnnotations(CompilationUnitDeclaration[], ReferenceBinding[], boolean) line: 159
> IdeAnnotationProcessorManager.processAnnotations(CompilationUnitDeclaration[], ReferenceBinding[], boolean) line: 134
> Compiler.processAnnotations() line: 813
> Compiler.compile(ICompilationUnit[]) line: 432
> BatchImageBuilder(AbstractImageBuilder).compile(SourceFile[], SourceFile[], boolean) line: 364
> BatchImageBuilder.compile(SourceFile[], SourceFile[], boolean) line: 178
> BatchImageBuilder(AbstractImageBuilder).compile(SourceFile[]) line: 301
> BatchImageBuilder.build() line: 60
> JavaBuilder.buildAll() line: 254
> JavaBuilder.build(int, Map, IProgressMonitor) line: 173
> BuildManager$2.run() line: 728
> SafeRunner.run(ISafeRunnable) line: 42
> BuildManager.basicBuild(int, IncrementalProjectBuilder, Map<String,String>, MultiStatus, IProgressMonitor) line: 199
> BuildManager.basicBuild(IBuildConfiguration, int, IBuildContext, ICommand[], MultiStatus, IProgressMonitor) line: 239
> BuildManager$1.run() line: 292
> SafeRunner.run(ISafeRunnable) line: 42
> BuildManager.basicBuild(IBuildConfiguration, int, IBuildContext, MultiStatus, IProgressMonitor) line: 295
> BuildManager.basicBuildLoop(IBuildConfiguration[], IBuildConfiguration[], int, MultiStatus, IProgressMonitor) line: 351
> BuildManager.build(IBuildConfiguration[], IBuildConfiguration[], int, IProgressMonitor) line: 374
> AutoBuildJob.doBuild(IProgressMonitor) line: 143
> AutoBuildJob.run(IProgressMonitor) line: 241
> Worker.run() line: 54
> {noformat}
> That stacktrace represents the processing of https://github.com/jbossas/jboss-as/blob/master/server/src/main/java/org/...
> So basically, whenever a Logger has a dependency to the [org.jboss.modules.Module|https://github.com/jbossas/jboss-modules/blob/ma...] class, stuff starts hitting the fan.
> Classes should not be loaded by reflection during APT processing to avoid such issues - it should use the API provided by APT to iterate over the classes metadata provided by the compiler and not depend on com.sun.*.internal classes.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
13 years, 11 months
[JBoss JIRA] (JGRP-1348) Getting exception- at java.lang.Thread.run(Thread.java:595) Caused by: java.io.IOException: No such device
by Bela Ban (JIRA)
[ https://issues.jboss.org/browse/JGRP-1348?page=com.atlassian.jira.plugin.... ]
Bela Ban deleted JGRP-1348:
---------------------------
> Getting exception- at java.lang.Thread.run(Thread.java:595) Caused by: java.io.IOException: No such device
> ----------------------------------------------------------------------------------------------------------
>
> Key: JGRP-1348
> URL: https://issues.jboss.org/browse/JGRP-1348
> Project: JGroups
> Issue Type: Bug
> Environment: Java 1.5.0
> Reporter: namrata sarda
> Assignee: Bela Ban
>
> We have faced this problem for 3 of our customer hosted by 3rd party vendor in just one week -
> Customer 1
> Param :
> UDP = {
> discard_incompatible_packets = true;
> down_thread = false;
> enable_bundling = false;
> enable_diagnostics = false;
> ip_mcast = true;
> ip_ttl = 1;
> loopback = false;
> mcast_addr = 224.135.43.60;
> mcast_port = 6979;
> mcast_recv_buf_size = 1000000;
> mcast_send_buf_size = 640000;
> receive_on_all_interfaces = true;
> tos = 16;
> ucast_recv_buf_size = 1000000;
> ucast_send_buf_size = 640000;
> up_thread = false;
> use_incoming_packet_handler = true;
> use_outgoing_packet_handler = true;
> };
> Mon Jul 11 04:26:14 PDT 2011 (T62) (org.jgroups.protocols.UDP:ERROR) : exception sending packet
> { ThreadName = "Thread[OutgoingPacketHandler,5,Pooled Threads]";}
> java.lang.Exception: dest=/224.135.43.60:6979 (58 bytes)
> at org.jgroups.protocols.UDP._send(UDP.java:356)
> at org.jgroups.protocols.UDP.sendToAllMembers(UDP.java:300)
> at org.jgroups.protocols.TP.doSend(TP.java:952)
> at org.jgroups.protocols.TP.send(TP.java:941)
> at org.jgroups.protocols.TP.access$1600(TP.java:49)
> at org.jgroups.protocols.TP$OutgoingPacketHandler.handleMessage(TP.java:1305)
> at org.jgroups.protocols.TP$OutgoingPacketHandler.run(TP.java:1288)
> at java.lang.Thread.run(Thread.java:595)
> Caused by: java.io.IOException: No such device
> at java.net.PlainDatagramSocketImpl.send(Native Method)
> at java.net.DatagramSocket.send(DatagramSocket.java:612)
> at org.jgroups.protocols.UDP._send(UDP.java:330)
> ... 7 more
> Customer 2
> UDP = {
> discard_incompatible_packets = true;
> down_thread = false;
> enable_bundling = false;
> enable_diagnostics = false;
> ip_mcast = true;
> ip_ttl = 1;
> loopback = false;
> mcast_addr = 228.62.152.231;
> mcast_port = 6999;
> mcast_recv_buf_size = 1000000;
> mcast_send_buf_size = 640000;
> receive_on_all_interfaces = true;
> tos = 16;
> ucast_recv_buf_size = 1000000;
> ucast_send_buf_size = 640000;
> up_thread = false;
> use_incoming_packet_handler = true;
> use_outgoing_packet_handler = true;
> };
> java.lang.Exception: dest=/228.62.152.231:6999 (58 bytes)
> at org.jgroups.protocols.UDP._send(UDP.java:356)
> at org.jgroups.protocols.UDP.sendToAllMembers(UDP.java:300)
> at org.jgroups.protocols.TP.doSend(TP.java:952)
> at org.jgroups.protocols.TP.send(TP.java:941)
> at org.jgroups.protocols.TP.access$1600(TP.java:49)
> at org.jgroups.protocols.TP$OutgoingPacketHandler.handleMessage(TP.java:1305)
> at org.jgroups.protocols.TP$OutgoingPacketHandler.run(TP.java:1288)
> at java.lang.Thread.run(Thread.java:595)
> Caused by: java.io.IOException: No such device
> at java.net.PlainDatagramSocketImpl.send(Native Method)
> at java.net.DatagramSocket.send(DatagramSocket.java:612)
> at org.jgroups.protocols.UDP._send(UDP.java:330)
> ... 7 more
> Customer 3
> UDP = {
> discard_incompatible_packets = true;
> down_thread = false;
> enable_bundling = false;
> enable_diagnostics = false;
> ip_mcast = true;
> ip_ttl = 1;
> loopback = false;
> mcast_addr = 228.135.43.59;
> mcast_port = 6129;
> mcast_recv_buf_size = 1000000;
> mcast_send_buf_size = 640000;
> receive_on_all_interfaces = true;
> tos = 16;
> ucast_recv_buf_size = 1000000;
> ucast_send_buf_size = 640000;
> up_thread = false;
> use_incoming_packet_handler = true;
> use_outgoing_packet_handler = true;
> };
> Thu Apr 14 19:17:43 EDT 2011 (T1165) (org.jgroups.protocols.UDP:ERROR) : exception sending packet
> { ThreadName = "Thread[OutgoingPacketHandler,5,Pooled Threads]";}
> java.lang.Exception: dest=/228.135.43.59:6129 (166 bytes)
> at org.jgroups.protocols.UDP._send(UDP.java:356)
> at org.jgroups.protocols.UDP.sendToAllMembers(UDP.java:300)
> at org.jgroups.protocols.TP.doSend(TP.java:952)
> at org.jgroups.protocols.TP.send(TP.java:941)
> at org.jgroups.protocols.TP.access$1600(TP.java:49)
> at org.jgroups.protocols.TP$OutgoingPacketHandler.handleMessage(TP.java:1305)
> at org.jgroups.protocols.TP$OutgoingPacketHandler.run(TP.java:1288)
> at java.lang.Thread.run(Thread.java:595)
> Caused by: java.io.IOException: No such device
> at java.net.PlainDatagramSocketImpl.send(Native Method)
> at java.net.DatagramSocket.send(DatagramSocket.java:612)
> at org.jgroups.protocols.UDP._send(UDP.java:330)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
13 years, 11 months
[JBoss JIRA] (JGRP-1334) TIMED_WAITING loop prevents message propagation to cluster due to FC blockage
by Bela Ban (JIRA)
[ https://issues.jboss.org/browse/JGRP-1334?page=com.atlassian.jira.plugin.... ]
Bela Ban resolved JGRP-1334.
----------------------------
Try on a more recent version, e.g. 3.x, and create a new JIRA if you can still reproduce it
> TIMED_WAITING loop prevents message propagation to cluster due to FC blockage
> -----------------------------------------------------------------------------
>
> Key: JGRP-1334
> URL: https://issues.jboss.org/browse/JGRP-1334
> Project: JGroups
> Issue Type: Bug
> Environment: Sparc Solaris 10
> Reporter: Simone Scarduzio
> Assignee: Bela Ban
> Labels: FC, credits
>
> Cannot send messages to cluster because node is waiting to have FC credit which never arrives.
> JGroups sender thread stays constantly in TIMED_WAITING loop having not enough credit to send any bytes.
> We have been able to reproduce the issue which occurs after some days of activity, but the actual triggering event is unclear, no clear test case available.
> Cluster is made of about 25 nodes, we're using Oracle jdk 1.6.0_24. No GC time is observed being bigger than 3.6s around the cluster nodes during problem happening. Still we read a lot of VERIFY_SUSPECT in the logs, mentioning many different node address.
> See the stack trace in which we see the blockage:
> "QuotaDistributor" daemon prio=3 tid=0x00a9cc00 nid=0x71 waiting on condition [0x63f1f000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x7b8392e0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2116)
> at org.jgroups.protocols.FC.handleDownMessage(FC.java:550)
> at org.jgroups.protocols.FC.down(FC.java:424)
> at org.jgroups.protocols.FRAG2.down(FRAG2.java:154)
> at org.jgroups.protocols.pbcast.STATE_TRANSFER.down(STATE_TRANSFER.java:216)
> at org.jgroups.stack.ProtocolStack.down(ProtocolStack.java:836)
> at org.jgroups.JChannel.down(JChannel.java:1626)
> at org.jgroups.JChannel.send(JChannel.java:721)
> at org.jgroups.JChannel.send(JChannel.java:737)
> at com.firsthop.common.platform.jgroups.JGroupsManager.send(JGroupsManager.java:193)
> Please advise or give us some pointers on how to progress the investigation
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
13 years, 11 months
[JBoss JIRA] (JGRP-1334) TIMED_WAITING loop prevents message propagation to cluster due to FC blockage
by Bela Ban (JIRA)
[ https://issues.jboss.org/browse/JGRP-1334?page=com.atlassian.jira.plugin.... ]
Bela Ban deleted JGRP-1334:
---------------------------
> TIMED_WAITING loop prevents message propagation to cluster due to FC blockage
> -----------------------------------------------------------------------------
>
> Key: JGRP-1334
> URL: https://issues.jboss.org/browse/JGRP-1334
> Project: JGroups
> Issue Type: Bug
> Environment: Sparc Solaris 10
> Reporter: Simone Scarduzio
> Assignee: Bela Ban
> Labels: FC, credits
>
> Cannot send messages to cluster because node is waiting to have FC credit which never arrives.
> JGroups sender thread stays constantly in TIMED_WAITING loop having not enough credit to send any bytes.
> We have been able to reproduce the issue which occurs after some days of activity, but the actual triggering event is unclear, no clear test case available.
> Cluster is made of about 25 nodes, we're using Oracle jdk 1.6.0_24. No GC time is observed being bigger than 3.6s around the cluster nodes during problem happening. Still we read a lot of VERIFY_SUSPECT in the logs, mentioning many different node address.
> See the stack trace in which we see the blockage:
> "QuotaDistributor" daemon prio=3 tid=0x00a9cc00 nid=0x71 waiting on condition [0x63f1f000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x7b8392e0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2116)
> at org.jgroups.protocols.FC.handleDownMessage(FC.java:550)
> at org.jgroups.protocols.FC.down(FC.java:424)
> at org.jgroups.protocols.FRAG2.down(FRAG2.java:154)
> at org.jgroups.protocols.pbcast.STATE_TRANSFER.down(STATE_TRANSFER.java:216)
> at org.jgroups.stack.ProtocolStack.down(ProtocolStack.java:836)
> at org.jgroups.JChannel.down(JChannel.java:1626)
> at org.jgroups.JChannel.send(JChannel.java:721)
> at org.jgroups.JChannel.send(JChannel.java:737)
> at com.firsthop.common.platform.jgroups.JGroupsManager.send(JGroupsManager.java:193)
> Please advise or give us some pointers on how to progress the investigation
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
13 years, 11 months
[JBoss JIRA] (JGRP-1272) FC: all threads are blocked at credits_available.await
by Bela Ban (JIRA)
[ https://issues.jboss.org/browse/JGRP-1272?page=com.atlassian.jira.plugin.... ]
Bela Ban deleted JGRP-1272:
---------------------------
> FC: all threads are blocked at credits_available.await
> ------------------------------------------------------
>
> Key: JGRP-1272
> URL: https://issues.jboss.org/browse/JGRP-1272
> Project: JGroups
> Issue Type: Bug
> Environment: CentOS release 5.2 (Final)
> kernel 2.6.27.24rootserver-20090525a
> JDK 1.6.0_22 64-bit
> Reporter: Victor N
> Assignee: Bela Ban
>
> After several days of working one node can not rejoin because of FC protocol issue. The situation is reproduced once or several times per week.
> The problematic node is called "gate9", its view is outdated:
> [gate10.mydomain|1175] [gate10.mydomain, gate7.mydomain, gate8.mydomain, gate9.mydomain, gate11.mydomain, gate14.mydomain, gate5.mydomain, gate12.mydomain, gate2.mydomain, gate4.mydomain, gate3.mydomain, gate6.mydomain]
> Actual view (seen by all other nodes) is: [gate10.mydomain|1176] [gate10.mydomain, gate7.mydomain, gate8.mydomain, gate11.mydomain, gate14.mydomain, gate5.mydomain, gate12.mydomain, gate2.mydomain, gate4.mydomain, gate3.mydomain, gate6.mydomain]
> In log file at gate9 I can see that it sends CREDIT_REQUEST constantly:
> 17/01/2011 09:55:52 UTC| TRACE | org.jgroups.protocols.TP.down(): sending msg to gate10.mydomain, src=gate9.mydomain, headers are FC: CREDIT_REQUEST, UNICAST: DATA, seqno=4308, conn_id=14, TCP: [channel_name=GateCluster]
> 17/01/2011 09:55:52 UTC| TRACE | org.jgroups.protocols.BasicTCP.sendUnicast(): dest=192.168.1.10:40001 (94 bytes)
> 17/01/2011 09:55:52 UTC| TRACE | org.jgroups.protocols.UNICAST.retransmit(): gate9.mydomain --> XMIT(gate10.mydomain: #2014)
> 17/01/2011 09:55:52 UTC| TRACE | org.jgroups.protocols.TP.down(): sending msg to gate10.mydomain, src=gate9.mydomain, headers are FC: CREDIT_REQUEST, UNICAST: DATA, seqno=2014, conn_id=14, TCP: [channel_name=GateCluster]
> 17/01/2011 09:55:52 UTC| TRACE | org.jgroups.protocols.BasicTCP.sendUnicast(): dest=192.168.1.10:40001 (94 bytes)
> 17/01/2011 09:55:52 UTC| TRACE | org.jgroups.protocols.UNICAST.retransmit(): gate9.mydomain --> XMIT(gate10.mydomain: #1124)
> ...
> At gate10 (the coordinator) I see that it receives the requests and responds:
> 17/01/2011 09:55:55 UTC| TRACE | org.jgroups.protocols.TP.passMessageUp(): received [dst: gate10.mydomain, src: gate9.mydomain (3 headers), size=9 bytes], headers are FC: CREDIT_REQUEST, UNICAST: DATA, seqno=2397, conn_id=14, TCP: [channel_name=GateCluster]
> 17/01/2011 09:55:55 UTC| TRACE | org.jgroups.protocols.UNICAST.handleDataReceived(): gate10.mydomain <-- DATA(gate9.mydomain: #2397, conn_id=14)
> 17/01/2011 09:55:55 UTC| TRACE | org.jgroups.protocols.UNICAST.sendRequestForFirstSeqno(): gate10.mydomain --> SEND_FIRST_SEQNO(gate9.mydomain)
> 17/01/2011 09:55:55 UTC| TRACE | org.jgroups.protocols.TP.down(): sending msg to gate9.mydomain, src=gate10.mydomain, headers are UNICAST: SEND_FIRST_SEQNO, seqno=0, TCP: [channel_name=GateCluster]
> 17/01/2011 09:55:55 UTC| TRACE | org.jgroups.protocols.TP.passMessageUp(): received [dst: gate10.mydomain, src: gate9.mydomain (3 headers), size=9 bytes], headers are FC: CREDIT_REQUEST, UNICAST: DATA, seqno=1202, conn_id=14, TCP: [channel_name=GateCluster]
> 17/01/2011 09:55:55 UTC| TRACE | org.jgroups.protocols.UNICAST.handleDataReceived(): gate10.mydomain <-- DATA(gate9.mydomain: #1202, conn_id=14)
> 17/01/2011 09:55:55 UTC| TRACE | org.jgroups.protocols.UNICAST.sendRequestForFirstSeqno(): gate10.mydomain --> SEND_FIRST_SEQNO(gate9.mydomain)
> 17/01/2011 09:55:55 UTC| TRACE | org.jgroups.protocols.TP.down(): sending msg to gate9.mydomain, src=gate10.mydomain, headers are UNICAST: SEND_FIRST_SEQNO, seqno=0, TCP: [channel_name=GateCluster]
> ...
> but I do not see any answer to this at gate9.
> My config and stack traces are attached below. I do not know how to reproduce the problem in tests. But it occurs, so I can provide you with any details, just let me know.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
13 years, 11 months
[JBoss JIRA] (JGRP-1166) Fix processing of Windows batch file arguments
by Bela Ban (JIRA)
[ https://issues.jboss.org/browse/JGRP-1166?page=com.atlassian.jira.plugin.... ]
Bela Ban deleted JGRP-1166:
---------------------------
> Fix processing of Windows batch file arguments
> ----------------------------------------------
>
> Key: JGRP-1166
> URL: https://issues.jboss.org/browse/JGRP-1166
> Project: JGroups
> Issue Type: Bug
> Environment: Windows
> Reporter: Richard Achmatowicz
> Assignee: Richard Achmatowicz
>
> The file build.bat can sometimes take a rather long set of command line arguments, due to specification of JVM system properties; for example,
> $ build.bat -Djgroups.bind_addr=%MYTESTIP_1% -Djgroups.udp.mcast_addr=%MCAST_ADDR% -Djgroups.tcpping.initial_hosts="%MYTESTIP_1%[7800],%MYTESTIP_1[7801]" all-tests-functional
> Windows batch files provide positional parameters %0 through %9 to collect the first nine arguments. Windows batch files also process a single parameter -Dfred=barney as two parameters -Dfred and barney. This leads to a limit on the number of system properties we can pass to the build.bat file, due to the way the command line arguments are processed (i.e. using the maximum number of positional parameters). In the worst case, some system properties get lost.
> One way to fix this is to gather together the all the command line arguments using the shift operator and store them in a variable, BUILD_ARGS. Here is the code:
> REM ******************************************************
> REM ********Save the command line arguments***************
> REM ******************************************************
> set BUILD_ARGS=%1
> if ""%1""=="""" goto completedArgs
> shift
> :processArg
> if ""%1""=="""" goto completedArgs
> set BUILD_ARGS=%BUILD_ARGS% %1
> shift
> goto processArg
> :completedArgs
> The variable BUILD_ARGS can then be passed to the
> %JAVA_HOME%\bin\java -classpath "%CP%" org.apache.tools.ant.Main -buildfile build.xml %BUILD_ARGS%
> This gets rid of the limit on command line parameters for batch files.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
13 years, 11 months