[JBoss JIRA] (WFLY-6926) TimeoutException: Replication timeout when handling request with DIST cache
by Paul Ferraro (JIRA)
[ https://issues.jboss.org/browse/WFLY-6926?page=com.atlassian.jira.plugin.... ]
Paul Ferraro commented on WFLY-6926:
------------------------------------
[~mvinkler] There seems to be an upstream Infinispan bug with staggered clustered gets. Can you try retesting using the following system property? -Dinfinispan.stagger.delay=0
That should workaround the issue. I'll submit a PR that temporarily disable staggered gets.
> TimeoutException: Replication timeout when handling request with DIST cache
> ---------------------------------------------------------------------------
>
> Key: WFLY-6926
> URL: https://issues.jboss.org/browse/WFLY-6926
> Project: WildFly
> Issue Type: Bug
> Components: Clustering
> Affects Versions: 10.1.0.CR1
> Reporter: Michal Vinkler
> Assignee: Paul Ferraro
> Priority: Blocker
> Fix For: 10.1.0.Final
>
>
> Seen in http-session scenarios with *DIST* cache only. REPL cache scenarios seem not to be affected.
> TimeoutExceptions occur right away after sending first request and definitely result in client getting 500:
> {code}
> [JBossINF] [0m[31m04:31:39,033 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (default task-1) ISPN000136: Error executing command GetKeyValueCommand, writing keys []: org.infinispan.util.concurrent.TimeoutException: Replication timeout for perf19
> [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:801)
> [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$1(JGroupsTransport.java:642)
> [JBossINF] at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
> [JBossINF] at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
> [JBossINF] at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
> [JBossINF] at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
> [JBossINF] at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.staggeredProcessNext(CommandAwareRpcDispatcher.java:375)
> [JBossINF] at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.lambda$processCallsStaggered$3(CommandAwareRpcDispatcher.java:357)
> [JBossINF] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [JBossINF] at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> [JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> [JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [JBossINF] at java.lang.Thread.run(Thread.java:745)
> {code}
> Client:
> {code}
> 2016/08/08 04:31:39:068 EDT [WARN ][Runner - 4] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Error sampling data: Internal Server Error>
> org.jboss.smartfrog.loaddriver.RequestProcessingException: Invalid response code: 500 Content: Internal Server Error
> at org.jboss.smartfrog.loaddriver.http.HttpRequestProcessorFactoryImpl$HttpRequestProcessor.processRequest(HttpRequestProcessorFactoryImpl.java:163)
> at org.jboss.smartfrog.loaddriver.CompoundRequestProcessorFactoryImpl$CompoundRequestProcessor.processRequest(CompoundRequestProcessorFactoryImpl.java:52)
> at org.jboss.smartfrog.loaddriver.Runner.run(Runner.java:103)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> I was able to reproduce it even with a very small load of 30 sessions.
> Server link (run with 30 sessions):
> http://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/mvinkler_eap-7x-failo...
> Client link (run with 30 sessions):
> http://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/mvinkler_eap-7x-failo...
> This issue occurs under different conditions than JBEAP-3779 (which is similar), that's why I opened new issue.
> Number of occurrences of TimeoutExceptions is very high. The server logs are full of them.
> This issue makes clustering with *DIST* cache unusable, thus giving blocker priority.
> Also, there is a issue with shutting down the servers in the end of the test, I don't know if it can be related, but I am hitting it constantly in the http-session DIST scenarios.
> The server shutdown usually gets stuck on this command:
> {code}
> [JBossINF] [0m[0m04:38:51,694 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-7) ISPN000082: Stopping the RpcDispatcher for channel web
> {code}
> *EDIT:*
> Also ejb-remote tests with *DIST* cache are affected (even if only 20 sessions are created).
> Server log for ejb-remote run:
> http://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/mvinkler_eap-7x-failo...
> Client log for ejb-remote run:
> http://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/mvinkler_eap-7x-failo...
> *EDIT #2:*
> Can be very easily reproduced locally. Just start a cluster of at least 3 servers with clusterbench deployed. Then do few requests to various nodes until you get 500 as a response. Also notice, when shutting down a node, which logged TimeoutException before, the node won't stop and get stuck during the shutdown.
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
9 years, 8 months
[JBoss JIRA] (WFLY-6953) Wildfly 10.1.0.CR1 does not start as a windows service
by James Perkins (JIRA)
[ https://issues.jboss.org/browse/WFLY-6953?page=com.atlassian.jira.plugin.... ]
James Perkins updated WFLY-6953:
--------------------------------
Description:
Wildfly fails to start as a windows service.
Installation works fine:
{code}
.\service.bat install
Using the X86-64bit version of prunsrv
"C:\applications\wildfly-10.1.0.CR1-test\bin\service\amd64\wildfly-service" install Wildfly --DisplayName=WildFly --Description=""WildFly Application Server"" --LogLevel=INFO --LogPath="C:\applications\wildfly-10.1.0.CR1-test\standalone\log" --LogPrefix=service --StdOutput=auto --StdError=auto --StartMode=exe --Startup=manual --StartImage=cmd.exe --StartPath="C:\applications\wildfly-10.1.0.CR1-test\bin" ++StartParams="/c#set#NOPAUSE=Y#&&#standalone.bat#-Djboss.server.base.dir=C:\applications\wildfly-10.1.0.CR1-test\standalone#--server-config=standalone.xml" --StopMode=exe --StopImage=cmd.exe --StopPath="C:\applications\wildfly-10.1.0.CR1-test\bin" ++StopParams="/c jboss-cli.bat --controller=localhost:9990 --connect --command=:shutdown"
Service Wildfly installed
{code}
Windows reports this error:
{quote}
Windows could not start the Wildfly on Local Computer. For more information, review the System Event Log. If this is a non-Microsoft service, contact the service vendor, and refer to service-specific error code1.
{quote}
{code}
Event log report:
The Wildfly service terminated with the following service-specific error:
Incorrect function.
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<System>
<Provider Name="Service Control Manager" Guid="{555908d1-a6d7-4695-8e1e-26931d2012f4}" EventSourceName="Service Control Manager"/>
<EventID Qualifiers="49152">7024</EventID>
<Version>0</Version>
<Level>2</Level>
<Task>0</Task>
<Opcode>0</Opcode>
<Keywords>0x8080000000000000</Keywords>
<TimeCreated SystemTime="2016-08-13T21:40:42.402218400Z"/>
<EventRecordID>196500</EventRecordID>
<Correlation/>
<Execution ProcessID="668" ThreadID="10184"/>
<Channel>System</Channel>
<Computer>ITPC7.intra.rfgh.net</Computer>
<Security/>
</System>
-
<EventData>
<Data Name="param1">Wildfly</Data>
<Data Name="param2">%%1</Data>
<Binary>570069006C00640066006C0079000000</Binary>
</EventData>
</Event>
{code}
was:
Wildfly fails to start as a windows service.
Installation works fine:
bq. .\service.bat install
bq. Using the X86-64bit version of prunsrv
bq.
bq. "C:\applications\wildfly-10.1.0.CR1-test\bin\service\amd64\wildfly-service" install Wildfl
bq. y --DisplayName=WildFly --Description=""WildFly Application Server"" --LogLevel=INFO --Lo
bq. gPath="C:\applications\wildfly-10.1.0.CR1-test\standalone\log" --LogPrefix=service --StdOu
bq. tput=auto --StdError=auto --StartMode=exe --Startup=manual --StartImage=cmd.exe --StartPat
bq. h="C:\applications\wildfly-10.1.0.CR1-test\bin" ++StartParams="/c#set#NOPAUSE=Y#&&#standal
bq. one.bat#-Djboss.server.base.dir=C:\applications\wildfly-10.1.0.CR1-test\standalone#--serve
bq. r-config=standalone.xml" --StopMode=exe --StopImage=cmd.exe --StopPath="C:\applications\wi
bq. ldfly-10.1.0.CR1-test\bin" ++StopParams="/c jboss-cli.bat --controller=localhost:9990 --c
bq. onnect --command=:shutdown"
bq. Service Wildfly installed
bq.
Windows reports this error:
{{Windows could not start the Wildfly on Local Computer. For more information, review the System Event Log. If this is a non-Microsoft service, contact the service vendor, and refer to service-specific error code1.
}}
Event log report:
The Wildfly service terminated with the following service-specific error:
Incorrect function.
- <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
- <System>
<Provider Name="Service Control Manager" Guid="{555908d1-a6d7-4695-8e1e-26931d2012f4}" EventSourceName="Service Control Manager" />
<EventID Qualifiers="49152">7024</EventID>
<Version>0</Version>
<Level>2</Level>
<Task>0</Task>
<Opcode>0</Opcode>
<Keywords>0x8080000000000000</Keywords>
<TimeCreated SystemTime="2016-08-13T21:40:42.402218400Z" />
<EventRecordID>196500</EventRecordID>
<Correlation />
<Execution ProcessID="668" ThreadID="10184" />
<Channel>System</Channel>
<Computer>ITPC7.intra.rfgh.net</Computer>
<Security />
</System>
- <EventData>
<Data Name="param1">Wildfly</Data>
<Data Name="param2">%%1</Data>
<Binary>570069006C00640066006C0079000000</Binary>
</EventData>
</Event>
> Wildfly 10.1.0.CR1 does not start as a windows service
> ------------------------------------------------------
>
> Key: WFLY-6953
> URL: https://issues.jboss.org/browse/WFLY-6953
> Project: WildFly
> Issue Type: Bug
> Affects Versions: 10.1.0.CR1
> Environment: Windows 8/Windows Server 2012
> Oracle Java 8
> Reporter: Anton Yudin
> Assignee: Jason Greene
>
> Wildfly fails to start as a windows service.
> Installation works fine:
> {code}
> .\service.bat install
> Using the X86-64bit version of prunsrv
> "C:\applications\wildfly-10.1.0.CR1-test\bin\service\amd64\wildfly-service" install Wildfly --DisplayName=WildFly --Description=""WildFly Application Server"" --LogLevel=INFO --LogPath="C:\applications\wildfly-10.1.0.CR1-test\standalone\log" --LogPrefix=service --StdOutput=auto --StdError=auto --StartMode=exe --Startup=manual --StartImage=cmd.exe --StartPath="C:\applications\wildfly-10.1.0.CR1-test\bin" ++StartParams="/c#set#NOPAUSE=Y#&&#standalone.bat#-Djboss.server.base.dir=C:\applications\wildfly-10.1.0.CR1-test\standalone#--server-config=standalone.xml" --StopMode=exe --StopImage=cmd.exe --StopPath="C:\applications\wildfly-10.1.0.CR1-test\bin" ++StopParams="/c jboss-cli.bat --controller=localhost:9990 --connect --command=:shutdown"
> Service Wildfly installed
> {code}
> Windows reports this error:
> {quote}
> Windows could not start the Wildfly on Local Computer. For more information, review the System Event Log. If this is a non-Microsoft service, contact the service vendor, and refer to service-specific error code1.
> {quote}
> {code}
> Event log report:
> The Wildfly service terminated with the following service-specific error:
> Incorrect function.
> <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
> <System>
> <Provider Name="Service Control Manager" Guid="{555908d1-a6d7-4695-8e1e-26931d2012f4}" EventSourceName="Service Control Manager"/>
> <EventID Qualifiers="49152">7024</EventID>
> <Version>0</Version>
> <Level>2</Level>
> <Task>0</Task>
> <Opcode>0</Opcode>
> <Keywords>0x8080000000000000</Keywords>
> <TimeCreated SystemTime="2016-08-13T21:40:42.402218400Z"/>
> <EventRecordID>196500</EventRecordID>
> <Correlation/>
> <Execution ProcessID="668" ThreadID="10184"/>
> <Channel>System</Channel>
> <Computer>ITPC7.intra.rfgh.net</Computer>
> <Security/>
> </System>
> -
> <EventData>
> <Data Name="param1">Wildfly</Data>
> <Data Name="param2">%%1</Data>
> <Binary>570069006C00640066006C0079000000</Binary>
> </EventData>
> </Event>
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
9 years, 8 months
[JBoss JIRA] (WFCORE-1687) WFLYCTL0357 warning upon undeploying any deployment
by Brian Stansberry (JIRA)
[ https://issues.jboss.org/browse/WFCORE-1687?page=com.atlassian.jira.plugi... ]
Brian Stansberry reassigned WFCORE-1687:
----------------------------------------
Fix Version/s: 3.0.0.Alpha6
Assignee: Chao Wang (was: Brian Stansberry)
Resolution: Done
> WFLYCTL0357 warning upon undeploying any deployment
> ---------------------------------------------------
>
> Key: WFCORE-1687
> URL: https://issues.jboss.org/browse/WFCORE-1687
> Project: WildFly Core
> Issue Type: Bug
> Components: Server
> Affects Versions: 3.0.0.Alpha4
> Reporter: Martin Simka
> Assignee: Chao Wang
> Fix For: 3.0.0.Alpha6
>
>
> EAP produces following warning upon undeployment:
> {code}WARN [org.jboss.as.controller] (management-handler-thread - 4) WFLYCTL0357: Notification of type deployment-undeployed is not described for the resource at the address []{code}
> The warning is produced for both managed and unmanaged deployments.
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
9 years, 8 months
[JBoss JIRA] (WFLY-6955) Application client has not assigned security permissions defined in parent EAR
by Ondrej Kotek (JIRA)
[ https://issues.jboss.org/browse/WFLY-6955?page=com.atlassian.jira.plugin.... ]
Ondrej Kotek moved JBEAP-5637 to WFLY-6955:
-------------------------------------------
Project: WildFly (was: JBoss Enterprise Application Platform)
Key: WFLY-6955 (was: JBEAP-5637)
Workflow: GIT Pull Request workflow (was: CDW with loose statuses v1)
Component/s: Application Client
Security Manager
(was: Application Client)
(was: Security Manager)
Affects Version/s: 11.0.0.Alpha1
(was: 7.1.0.DR2)
> Application client has not assigned security permissions defined in parent EAR
> ------------------------------------------------------------------------------
>
> Key: WFLY-6955
> URL: https://issues.jboss.org/browse/WFLY-6955
> Project: WildFly
> Issue Type: Bug
> Components: Application Client, Security Manager
> Affects Versions: 11.0.0.Alpha1
> Reporter: Ondrej Kotek
> Assignee: Stuart Douglas
> Priority: Critical
> Attachments: permissions.xml
>
>
> When application client runs with security manager, it should have assigned security permissions defined in permissions.xml in parent EAR.
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
9 years, 8 months
[JBoss JIRA] (JGRP-2095) FILE_PING rewrites files too often
by Bela Ban (JIRA)
[ https://issues.jboss.org/browse/JGRP-2095?page=com.atlassian.jira.plugin.... ]
Bela Ban commented on JGRP-2095:
--------------------------------
Hi Dennis,
what you suggest sounds reasonable. The many calls to the storage backend was one of the reasons why this was subsequently changed (in 3.5).
Can you make that change? You should have permissions, so no need to submit a PR. Do you need me to create a 3.2.17 in JIRA?
> FILE_PING rewrites files too often
> ----------------------------------
>
> Key: JGRP-2095
> URL: https://issues.jboss.org/browse/JGRP-2095
> Project: JGroups
> Issue Type: Bug
> Affects Versions: 3.2.13
> Reporter: Dennis Reed
> Assignee: Bela Ban
>
> FILE_PING.fetchClusterMembers writes its own data to file on every call.
> If there is an issue finding a cluster member's address, fetchClusterMembers can be called very frequently by TP's GET_PHYSICAL_ADDRESS calls.
> This can cause problems because while that file is being (unnecessarily) overwritten, any readers will see an empty or corrupt file. If it's bad enough JGRP-1448 is not sufficient to handle it, and readers will end up thinking the file is stale and delete it (even while it's still in progress of being written). That missing file can then cascade the problem, with other nodes not being able to find this member's address, and triggering the same problem again.
> There is already a scheduled task to rewrite the file occasionally, so rewriting it on every fetchClusterMembers call is not needed.
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
9 years, 8 months
[JBoss JIRA] (WFLY-6954) Binding Wildfly's management interface to IP address fails
by Eduard Dedu (JIRA)
[ https://issues.jboss.org/browse/WFLY-6954?page=com.atlassian.jira.plugin.... ]
Eduard Dedu edited comment on WFLY-6954 at 8/15/16 7:51 AM:
------------------------------------------------------------
Many thanks for the prompt reply, I'll take it to the forum.
*you can't bind to an IP address on a remote machine,*
I think the docs need to make this clear, there's no indication whatsoever
https://docs.jboss.org/author/display/AS71/Command+line+parameters
was (Author: eduarddedu):
Many thanks for the prompt reply, I'll take it to the forum.
*you can't bind to an IP address on a remote machine,*
*I think the docs need to make this clear, there's no indication whatsoever*
https://docs.jboss.org/author/display/AS71/Command+line+parameters
> Binding Wildfly's management interface to IP address fails
> ----------------------------------------------------------
>
> Key: WFLY-6954
> URL: https://issues.jboss.org/browse/WFLY-6954
> Project: WildFly
> Issue Type: Bug
> Reporter: Eduard Dedu
> Assignee: Jason Greene
> Labels: Wildfly-10.0.0.Final
> Attachments: Screen Shot 2016-08-15 at 12.34.42.png
>
>
> Binding management interface to a remote IPV4 address causes the server to start with errors :
> Failed to start service jboss.network.management: org.jboss.msc.service.StartException in service jboss.network.management: WFLYSRV0082: failed to resolve interface management
> The error occurs if the server is started with the -b switch:
> ./standalone.sh -bmanagement=my_IPV4_address
> ... or if I edit the configuration standalone.xml:
> <interface name="management">
> <inet-address value="${jboss.bind.address.management:my_IPV4_address}"/>
> </interface>
> The remote host located at the IPV4 address is up and running and responds to PING signals.
> TCP is enabled on port 9990.
> Wildfly starts w/o errors if I enable remote access from any host:
>
> ./standalone.sh -b 0.0.0.0 -bmanagement 0.0.0.0 &
> Perhaps I'm missing something, but this behaviour looks like a bug in a core WF feature.
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
9 years, 8 months
[JBoss JIRA] (WFLY-6954) Binding Wildfly's management interface to IP address fails
by Eduard Dedu (JIRA)
[ https://issues.jboss.org/browse/WFLY-6954?page=com.atlassian.jira.plugin.... ]
Eduard Dedu commented on WFLY-6954:
-----------------------------------
Many thanks for the prompt reply, I'll take it to the forum.
*you can't bind to an IP address on a remote machine,*
*I think the docs need to make this clear, there's no indication whatsoever*
https://docs.jboss.org/author/display/AS71/Command+line+parameters
> Binding Wildfly's management interface to IP address fails
> ----------------------------------------------------------
>
> Key: WFLY-6954
> URL: https://issues.jboss.org/browse/WFLY-6954
> Project: WildFly
> Issue Type: Bug
> Reporter: Eduard Dedu
> Assignee: Jason Greene
> Labels: Wildfly-10.0.0.Final
> Attachments: Screen Shot 2016-08-15 at 12.34.42.png
>
>
> Binding management interface to a remote IPV4 address causes the server to start with errors :
> Failed to start service jboss.network.management: org.jboss.msc.service.StartException in service jboss.network.management: WFLYSRV0082: failed to resolve interface management
> The error occurs if the server is started with the -b switch:
> ./standalone.sh -bmanagement=my_IPV4_address
> ... or if I edit the configuration standalone.xml:
> <interface name="management">
> <inet-address value="${jboss.bind.address.management:my_IPV4_address}"/>
> </interface>
> The remote host located at the IPV4 address is up and running and responds to PING signals.
> TCP is enabled on port 9990.
> Wildfly starts w/o errors if I enable remote access from any host:
>
> ./standalone.sh -b 0.0.0.0 -bmanagement 0.0.0.0 &
> Perhaps I'm missing something, but this behaviour looks like a bug in a core WF feature.
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
9 years, 8 months
[JBoss JIRA] (WFLY-6954) Binding Wildfly's management interface to IP address fails
by Darran Lofthouse (JIRA)
[ https://issues.jboss.org/browse/WFLY-6954?page=com.atlassian.jira.plugin.... ]
Darran Lofthouse resolved WFLY-6954.
------------------------------------
Resolution: Rejected
This kind of question should be taken to the user forms, you can't bind to an IP address on a remote machine, you can only bind to IP addresses on the local machine. There are various options out there but start a forum thread to discuss them.
> Binding Wildfly's management interface to IP address fails
> ----------------------------------------------------------
>
> Key: WFLY-6954
> URL: https://issues.jboss.org/browse/WFLY-6954
> Project: WildFly
> Issue Type: Bug
> Reporter: Eduard Dedu
> Assignee: Jason Greene
> Labels: Wildfly-10.0.0.Final
> Attachments: Screen Shot 2016-08-15 at 12.34.42.png
>
>
> Binding management interface to a remote IPV4 address causes the server to start with errors :
> Failed to start service jboss.network.management: org.jboss.msc.service.StartException in service jboss.network.management: WFLYSRV0082: failed to resolve interface management
> The error occurs if the server is started with the -b switch:
> ./standalone.sh -bmanagement=my_IPV4_address
> ... or if I edit the configuration standalone.xml:
> <interface name="management">
> <inet-address value="${jboss.bind.address.management:my_IPV4_address}"/>
> </interface>
> The remote host located at the IPV4 address is up and running and responds to PING signals.
> TCP is enabled on port 9990.
> Wildfly starts w/o errors if I enable remote access from any host:
>
> ./standalone.sh -b 0.0.0.0 -bmanagement 0.0.0.0 &
> Perhaps I'm missing something, but this behaviour looks like a bug in a core WF feature.
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
9 years, 8 months