[JBoss JIRA] (WFCORE-1016) NPE or AssertionError if op is cancelled on slave between sending prepare and receiving commit/rollback
by Kabir Khan (JIRA)
[ https://issues.jboss.org/browse/WFCORE-1016?page=com.atlassian.jira.plugi... ]
Kabir Khan updated WFCORE-1016:
-------------------------------
Fix Version/s: 2.0.0.CR7
(was: 2.0.0.CR6)
> NPE or AssertionError if op is cancelled on slave between sending prepare and receiving commit/rollback
> -------------------------------------------------------------------------------------------------------
>
> Key: WFCORE-1016
> URL: https://issues.jboss.org/browse/WFCORE-1016
> Project: WildFly Core
> Issue Type: Bug
> Components: Domain Management
> Affects Versions: 2.0.0.CR5
> Reporter: Brian Stansberry
> Assignee: Brian Stansberry
> Fix For: 2.0.0.CR7
>
>
> OperationCancellationTestCase showed this failure:
> {code}
> 2015-09-23 06:43:40,147 INFO [org.jboss.as.test.integration.management.extension.blocker.BlockerExtension] (Host Controller Service Threads - 82) block requested by testSlaveBlockCompletionCancelSlave for slave/undefined(COMMIT)
> 2015-09-23 06:43:40,147 INFO [org.jboss.as.test.integration.management.extension.blocker.BlockerExtension] (Host Controller Service Threads - 82) will block at COMMIT for 20000
> 2015-09-23 06:43:40,164 INFO [org.jboss.as.controller.management-operation] (Host Controller Service Threads - 71) WFLYCTL0352: Cancelling operation 'block' with id '1900489784' running on thread 'Host Controller Service Threads - 82'
> 2015-09-23 06:43:40,185 ERROR [stderr] (Host Controller Service Threads - 82) Exception in thread "Host Controller Service Threads - 82" java.lang.AssertionError
> 2015-09-23 06:43:40,185 ERROR [stderr] (Host Controller Service Threads - 82) at org.jboss.as.controller.remote.TransactionalProtocolOperationHandler$ExecuteRequestContext.completed(TransactionalProtocolOperationHandler.java:475)
> 2015-09-23 06:43:40,186 ERROR [stderr] (Host Controller Service Threads - 82) at org.jboss.as.controller.remote.TransactionalProtocolOperationHandler$ExecuteRequestHandler.doExecute(TransactionalProtocolOperationHandler.java:190)
> 2015-09-23 06:43:40,186 ERROR [stderr] (Host Controller Service Threads - 82) at org.jboss.as.controller.remote.TransactionalProtocolOperationHandler$ExecuteRequestHandler$1.run(TransactionalProtocolOperationHandler.java:136)
> 2015-09-23 06:43:40,186 ERROR [stderr] (Host Controller Service Threads - 82) at org.jboss.as.controller.remote.TransactionalProtocolOperationHandler$ExecuteRequestHandler$1.run(TransactionalProtocolOperationHandler.java:132)
> 2015-09-23 06:43:40,186 ERROR [stderr] (Host Controller Service Threads - 82) at java.security.AccessController.doPrivileged(Native Method)
> 2015-09-23 06:43:40,186 ERROR [stderr] (Host Controller Service Threads - 82) at javax.security.auth.Subject.doAs(Subject.java:360)
> 2015-09-23 06:43:40,186 ERROR [stderr] (Host Controller Service Threads - 82) at org.jboss.as.controller.AccessAuditContext.doAs(AccessAuditContext.java:81)
> 2015-09-23 06:43:40,186 ERROR [stderr] (Host Controller Service Threads - 82) at org.jboss.as.controller.remote.TransactionalProtocolOperationHandler$ExecuteRequestHandler$2$1.run(TransactionalProtocolOperationHandler.java:152)
> 2015-09-23 06:43:40,186 ERROR [stderr] (Host Controller Service Threads - 82) at org.jboss.as.controller.remote.TransactionalProtocolOperationHandler$ExecuteRequestHandler$2$1.run(TransactionalProtocolOperationHandler.java:148)
> 2015-09-23 06:43:40,187 ERROR [stderr] (Host Controller Service Threads - 82) at java.security.AccessController.doPrivileged(Native Method)
> 2015-09-23 06:43:40,187 ERROR [stderr] (Host Controller Service Threads - 82) at org.jboss.as.controller.remote.TransactionalProtocolOperationHandler$ExecuteRequestHandler$2.execute(TransactionalProtocolOperationHandler.java:148)
> 2015-09-23 06:43:40,187 ERROR [stderr] (Host Controller Service Threads - 82) at org.jboss.as.protocol.mgmt.AbstractMessageHandler$2$1.doExecute(AbstractMessageHandler.java:299)
> 2015-09-23 06:43:40,187 ERROR [stderr] (Host Controller Service Threads - 82) at org.jboss.as.protocol.mgmt.AbstractMessageHandler$AsyncTaskRunner.run(AbstractMessageHandler.java:519)
> 2015-09-23 06:43:40,187 ERROR [stderr] (Host Controller Service Threads - 82) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> 2015-09-23 06:43:40,187 ERROR [stderr] (Host Controller Service Threads - 82) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> 2015-09-23 06:43:40,187 ERROR [stderr] (Host Controller Service Threads - 82) at java.lang.Thread.run(Thread.java:745)
> 2015-09-23 06:43:40,187 ERROR [stderr] (Host Controller Service Threads - 82) at org.jboss.threads.JBossThread.run(JBossThread.java:320)
> {code}
> What happens here is the slave HC executing the 'block' op has sent a prepare message to the master, and then when the commit has not come back yet the test driver triggers a cancel on the slave. At this point the thread running the 'block' op returns and tries to send the final result to the master. But it fails because the prepare logic has nulled out the responseChannel field, and the commit message has not arrived yet to restore a new one.
> If assertions weren't enabled, the result would be an NPE when the null responseChannel is used.
> I don't see any reason for null out the responseChannel.
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
8 years, 7 months
[JBoss JIRA] (WFCORE-1027) Inconsistent read-resource results with host scoped roles
by Kabir Khan (JIRA)
[ https://issues.jboss.org/browse/WFCORE-1027?page=com.atlassian.jira.plugi... ]
Kabir Khan updated WFCORE-1027:
-------------------------------
Fix Version/s: 2.0.0.CR7
(was: 2.0.0.CR6)
> Inconsistent read-resource results with host scoped roles
> ---------------------------------------------------------
>
> Key: WFCORE-1027
> URL: https://issues.jboss.org/browse/WFCORE-1027
> Project: WildFly Core
> Issue Type: Bug
> Components: Domain Management
> Affects Versions: 2.0.0.CR5
> Reporter: Kabir Khan
> Assignee: Kabir Khan
> Fix For: 2.0.0.CR7
>
>
> Setting up host scoped roles as follows https://gist.github.com/heiko-braun/0dc810ed04db8739defd there are inconsistent results in the filtering. When using a role which only selects the master there is no access-control response header showing the filtered resources, and the slave wrongly appears in the results:
> {code}
> [domain@localhost:9990 /] /host=*:read-resource{roles=master-monitor}
> {
> "outcome" => "success",
> "result" => [
> {
> "address" => [("host" => "master")],
> "outcome" => "success",
> "result" => {
> "directory-grouping" => "by-server",
> "domain-controller" => {"local" => {}},
> "management-major-version" => 4,
> "management-micro-version" => 0,
> "management-minor-version" => 0,
> "master" => true,
> "name" => "master",
> "namespaces" => [],
> "organization" => undefined,
> "product-name" => "WildFly Core",
> "product-version" => "2.0.0.CR6-SNAPSHOT",
> "release-codename" => "Kenny",
> "release-version" => "2.0.0.CR6-SNAPSHOT",
> "schema-locations" => [],
> "core-service" => {
> "host-environment" => undefined,
> "platform-mbean" => undefined,
> "management" => undefined,
> "discovery-options" => undefined,
> "ignored-resources" => undefined,
> "patching" => undefined,
> "module-loading" => undefined
> },
> "extension" => {"org.jboss.as.jmx" => undefined},
> "interface" => {
> "management" => undefined,
> "public" => undefined,
> "unsecure" => undefined
> },
> "jvm" => {"default" => undefined},
> "path" => undefined,
> "server" => {
> "server-one" => undefined,
> "server-two" => undefined,
> "server-three" => undefined
> },
> "server-config" => {
> "server-one" => undefined,
> "server-two" => undefined,
> "server-three" => undefined
> },
> "socket-binding-group" => undefined,
> "subsystem" => {"jmx" => undefined},
> "system-property" => undefined
> }
> },
> {
> "address" => [("host" => "localhost")],
> "outcome" => "success",
> "result" => undefined
> }
> ]
> }
> {code}
> When using a role that only selects the slave we get a proper access-control header
> {code}
> [domain@localhost:9990 /] /host=*:read-resource{roles=slave-maintainer}
> {
> "outcome" => "success",
> "result" => [{
> "address" => [("host" => "localhost")],
> "outcome" => "success",
> "result" => undefined
> }],
> "response-headers" => {"access-control" => [{
> "absolute-address" => [],
> "relative-address" => [],
> "filtered-children-types" => ["host"]
> }]}
> {code}
> The same output on master with WFCORE-994 applied:
> {code}
> [domain@localhost:9990 /] /host=*:read-resource{roles=slave-maintainer}
> {
> "outcome" => "success",
> "result" => [{
> "address" => [("host" => "slave")],
> "outcome" => "success",
> "result" => {
> "directory-grouping" => "by-server",
> "domain-controller" => {"remote" => {
> "protocol" => undefined,
> "port" => undefined,
> "host" => undefined,
> "username" => undefined,
> "ignore-unused-configuration" => undefined,
> "admin-only-policy" => undefined,
> "security-realm" => "ManagementRealm"
> }},
> "management-major-version" => 4,
> "management-micro-version" => 0,
> "management-minor-version" => 0,
> "master" => false,
> "name" => "slave",
> "namespaces" => [],
> "organization" => undefined,
> "product-name" => undefined,
> "product-version" => undefined,
> "release-codename" => "Kenny",
> "release-version" => "2.0.0.CR6-SNAPSHOT",
> "schema-locations" => [],
> "core-service" => {
> "host-environment" => undefined,
> "platform-mbean" => undefined,
> "management" => undefined,
> "discovery-options" => undefined,
> "ignored-resources" => undefined,
> "patching" => undefined,
> "module-loading" => undefined
> },
> "extension" => {"org.jboss.as.jmx" => undefined},
> "interface" => {
> "management" => undefined,
> "public" => undefined,
> "unsecure" => undefined
> },
> "jvm" => {"default" => undefined},
> "path" => undefined,
> "server" => {
> "server-one" => undefined,
> "server-two" => undefined
> },
> "server-config" => {
> "server-one" => undefined,
> "server-two" => undefined
> },
> "socket-binding-group" => undefined,
> "subsystem" => {"jmx" => undefined},
> "system-property" => undefined
> }
> }],
> "response-headers" => {"access-control" => [{
> "absolute-address" => [],
> "relative-address" => [],
> "filtered-children-types" => ["host"]
> }]}
> }
> {code}
> master-monitor should behave the same as slave-maintainer.
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
8 years, 7 months
[JBoss JIRA] (WFCORE-1025) ManagementRequestContext executeAsync hides RejectedExecutionException
by Kabir Khan (JIRA)
[ https://issues.jboss.org/browse/WFCORE-1025?page=com.atlassian.jira.plugi... ]
Kabir Khan updated WFCORE-1025:
-------------------------------
Fix Version/s: 2.0.0.CR7
(was: 2.0.0.CR6)
> ManagementRequestContext executeAsync hides RejectedExecutionException
> ----------------------------------------------------------------------
>
> Key: WFCORE-1025
> URL: https://issues.jboss.org/browse/WFCORE-1025
> Project: WildFly Core
> Issue Type: Bug
> Components: Domain Management
> Affects Versions: 2.0.0.CR5
> Reporter: Brian Stansberry
> Assignee: Brian Stansberry
> Fix For: 2.0.0.CR7
>
>
> The impl of ManagementRequestContext executeAsync catches RejectedExecutionException and doesn't notify the calling thread. The handling itself seems ok (call failed on the result handler and send a failure response to the client), but not notifying the caller is problematic. There are a number of cases where the caller thread waits for a latch to be tripped, with the async task tripping. If the latch never trips, the caller thread will block forever. The testsuite hang at http://brontes.lab.eng.brq.redhat.com/viewLog.html?buildId=71233&buildTyp... looks to be a case of this, with a server not stopping due to this:
> {code}
> "Remoting "master:main-one:MANAGEMENT" task-13" #55 prio=5 os_prio=0 tid=0xc6c3bc00 nid=0x19d9 waiting on condition [0xc215c000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0xddbb4aa0> (a java.util.concurrent.CountDownLatch$Sync)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
> at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
> at org.jboss.as.controller.remote.TransactionalProtocolOperationHandler.sendResponse(TransactionalProtocolOperationHandler.java:540)
> at org.jboss.as.controller.remote.TransactionalProtocolOperationHandler$ExecuteRequestContext.failed(TransactionalProtocolOperationHandler.java:377)
> - locked <0xddbb2300> (a org.jboss.as.controller.remote.TransactionalProtocolOperationHandler$ExecuteRequestContext)
> at org.jboss.as.protocol.mgmt.ActiveOperationSupport$ActiveOperationImpl$2.handleFailed(ActiveOperationSupport.java:350)
> at org.jboss.threads.AsyncFutureTask$Reg.run(AsyncFutureTask.java:60)
> at org.jboss.as.protocol.mgmt.ActiveOperationSupport$1.execute(ActiveOperationSupport.java:55)
> at org.jboss.threads.AsyncFutureTask.safeExecute(AsyncFutureTask.java:169)
> at org.jboss.threads.AsyncFutureTask.setFailed(AsyncFutureTask.java:162)
> at org.jboss.as.protocol.mgmt.ActiveOperationSupport$ActiveOperationImpl.access$400(ActiveOperationSupport.java:295)
> at org.jboss.as.protocol.mgmt.ActiveOperationSupport$ActiveOperationImpl$1.failed(ActiveOperationSupport.java:315)
> at org.jboss.as.protocol.mgmt.AbstractMessageHandler$2.executeAsync(AbstractMessageHandler.java:333)
> at org.jboss.as.protocol.mgmt.AbstractMessageHandler$2.executeAsync(AbstractMessageHandler.java:315)
> at org.jboss.as.controller.remote.TransactionalProtocolOperationHandler$ExecuteRequestHandler.handleRequest(TransactionalProtocolOperationHandler.java:144)
> at org.jboss.as.protocol.mgmt.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:270)
> at org.jboss.as.protocol.mgmt.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:252)
> at org.jboss.as.protocol.mgmt.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:123)
> at org.jboss.as.protocol.mgmt.ManagementChannelReceiver$1.handleMessage(ManagementChannelReceiver.java:56)
> at org.jboss.as.protocol.mgmt.ManagementChannelReceiver.handleMessage(ManagementChannelReceiver.java:85)
> at org.jboss.remoting3.remote.RemoteConnectionChannel$5.run(RemoteConnectionChannel.java:463)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> The RejectedExecutionException indicates a thread pool has been shutdown before the response has gone out. Ideally we would prevent that (for which I've recently filed a JIRA) but in any case we should make this more robust.
> I think having executeAsync return a boolean should suffice.
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
8 years, 7 months
[JBoss JIRA] (WFCORE-1002) TransactionalProtocolOperationHandler.AbortOperationHandler must initialize the ExecuteRequestContext before responding
by Kabir Khan (JIRA)
[ https://issues.jboss.org/browse/WFCORE-1002?page=com.atlassian.jira.plugi... ]
Kabir Khan updated WFCORE-1002:
-------------------------------
Fix Version/s: 2.0.0.CR7
(was: 2.0.0.CR6)
> TransactionalProtocolOperationHandler.AbortOperationHandler must initialize the ExecuteRequestContext before responding
> -----------------------------------------------------------------------------------------------------------------------
>
> Key: WFCORE-1002
> URL: https://issues.jboss.org/browse/WFCORE-1002
> Project: WildFly Core
> Issue Type: Bug
> Components: Domain Management
> Affects Versions: 2.0.0.CR3
> Reporter: Brian Stansberry
> Assignee: Brian Stansberry
> Fix For: 2.0.0.CR7
>
>
> If in a race a COMPLETE_TX_REQUEST cancelling an op is processed by a server/slave HC before the original EXECUTE_TX_REQUEST is processed, when the EXECUTE_TX_REQUEST comes in the AbortOperationHandler attempts to respond with a "outcome"=>"cancelled" response. This will fail because the ExecutionRequestContext will not be initialized.
> In this situation the AbortOperationHandler should initialize the ERC.
> With assertions enabled, this appears in the log:
> {code}
> 20:58:47,947 ERROR [stderr] (Remoting "slave:main-three:MANAGEMENT" task-6) Exception in thread "Remoting "slave:main-three:MANAGEMENT" task-6" java.lang.AssertionError
> 20:58:47,947 ERROR [stderr] (Remoting "slave:main-three:MANAGEMENT" task-6) at org.jboss.as.controller.remote.TransactionalProtocolOperationHandler$ExecuteRequestContext.failed(TransactionalProtocolOperationHandler.java:458)
> 20:58:47,948 ERROR [stderr] (Remoting "slave:main-three:MANAGEMENT" task-6) at org.jboss.as.controller.remote.TransactionalProtocolOperationHandler$AbortOperationHandler.handleRequest(TransactionalProtocolOperationHandler.java:275)
> 20:58:47,948 ERROR [stderr] (Remoting "slave:main-three:MANAGEMENT" task-6) at org.jboss.as.protocol.mgmt.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:270)
> 20:58:47,948 ERROR [stderr] (Remoting "slave:main-three:MANAGEMENT" task-6) at org.jboss.as.protocol.mgmt.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:252)
> 20:58:47,948 ERROR [stderr] (Remoting "slave:main-three:MANAGEMENT" task-6) at org.jboss.as.protocol.mgmt.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:123)
> 20:58:47,949 ERROR [stderr] (Remoting "slave:main-three:MANAGEMENT" task-6) at org.jboss.as.protocol.mgmt.ManagementChannelReceiver$1.handleMessage(ManagementChannelReceiver.java:56)
> 20:58:47,949 ERROR [stderr] (Remoting "slave:main-three:MANAGEMENT" task-6) at org.jboss.as.protocol.mgmt.ManagementChannelReceiver.handleMessage(ManagementChannelReceiver.java:85)
> 20:58:47,949 ERROR [stderr] (Remoting "slave:main-three:MANAGEMENT" task-6) at org.jboss.remoting3.remote.RemoteConnectionChannel$5.run(RemoteConnectionChannel.java:451)
> 20:58:47,949 ERROR [stderr] (Remoting "slave:main-three:MANAGEMENT" task-6) at org.jboss.remoting3.EndpointImpl$TrackingExecutor$1.run(EndpointImpl.java:705)
> 20:58:47,950 ERROR [stderr] (Remoting "slave:main-three:MANAGEMENT" task-6) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> 20:58:47,950 ERROR [stderr] (Remoting "slave:main-three:MANAGEMENT" task-6) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> 20:58:47,950 ERROR [stderr] (Remoting "slave:main-three:MANAGEMENT" task-6) at java.lang.Thread.run(Thread.java:745)
> {code}
> Without assertions, a few lines later a call to sendResponse will result in an NPE.
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
8 years, 7 months
[JBoss JIRA] (WFCORE-994) Inconsistent :rrd(access-control) response when querying host resources
by Kabir Khan (JIRA)
[ https://issues.jboss.org/browse/WFCORE-994?page=com.atlassian.jira.plugin... ]
Kabir Khan updated WFCORE-994:
------------------------------
Fix Version/s: 2.0.0.CR7
(was: 2.0.0.CR6)
> Inconsistent :rrd(access-control) response when querying host resources
> -----------------------------------------------------------------------
>
> Key: WFCORE-994
> URL: https://issues.jboss.org/browse/WFCORE-994
> Project: WildFly Core
> Issue Type: Bug
> Components: Domain Management
> Reporter: Heiko Braun
> Assignee: Kabir Khan
> Priority: Critical
> Fix For: 2.0.0.CR7
>
>
> {noformat}
> [domain@localhost:9990 /] /host=*:read-resource-description(access-control=trim-descriptions){roles=Administrator}
> {
> "outcome" => "success",
> "result" => [
> {
> "address" => [("host" => "macbook-pro.fritz.box")],
> "outcome" => "success",
> "result" => {
> "access-control" => {
> "default" => {
> "read" => true,
> "write" => true,
> "attributes" => {}
> },
> "exceptions" => {}
> },
> "description" => undefined
> }
> },
> {
> "address" => [("host" => "master")],
> "outcome" => "success",
> "result" => {
> "description" => undefined,
> "attributes" => undefined,
> "operations" => undefined,
> "notifications" => undefined,
> "children" => {
> "core-service" => {"model-description" => undefined},
> "jvm" => {"model-description" => undefined},
> "socket-binding-group" => {"model-description" => undefined},
> "path" => {"model-description" => undefined},
> "server" => {"model-description" => undefined},
> "subsystem" => {"model-description" => undefined},
> "server-config" => {"model-description" => undefined},
> "interface" => {"model-description" => undefined},
> "system-property" => {"model-description" => undefined},
> "extension" => {"model-description" => undefined}
> },
> "access-control" => {
> "default" => {
> "read" => true,
> "write" => true,
> "attributes" => {
> "directory-grouping" => {
> "read" => true,
> "write" => true
> },
> "domain-controller" => {
> "read" => true,
> "write" => true
> },
> "domain-organization" => {
> "read" => true,
> "write" => true
> },
> "host-state" => {
> "read" => true,
> "write" => true
> },
> "management-major-version" => {
> "read" => true,
> "write" => true
> },
> "management-micro-version" => {
> "read" => true,
> "write" => true
> },
> "management-minor-version" => {
> "read" => true,
> "write" => true
> },
> "master" => {
> "read" => true,
> "write" => true
> },
> "name" => {
> "read" => true,
> "write" => true
> },
> "namespaces" => {
> "read" => true,
> "write" => true
> },
> "organization" => {
> "read" => true,
> "write" => true
> },
> "product-name" => {
> "read" => true,
> "write" => true
> },
> "product-version" => {
> "read" => true,
> "write" => true
> },
> "release-codename" => {
> "read" => true,
> "write" => true
> },
> "release-version" => {
> "read" => true,
> "write" => true
> },
> "running-mode" => {
> "read" => true,
> "write" => true
> },
> "schema-locations" => {
> "read" => true,
> "write" => true
> },
> "server-state" => {
> "read" => true,
> "write" => true
> },
> "suspend-state" => {
> "read" => true,
> "write" => true
> },
> "uuid" => {
> "read" => true,
> "write" => true
> }
> }
> },
> "exceptions" => {}
> }
> }
> }
> ]
> }
> [domain@localhost:9990 /]
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
8 years, 7 months
[JBoss JIRA] (WFCORE-989) Premature registration of reconnecting server
by Kabir Khan (JIRA)
[ https://issues.jboss.org/browse/WFCORE-989?page=com.atlassian.jira.plugin... ]
Kabir Khan updated WFCORE-989:
------------------------------
Fix Version/s: 2.0.0.CR7
(was: 2.0.0.CR6)
> Premature registration of reconnecting server
> ---------------------------------------------
>
> Key: WFCORE-989
> URL: https://issues.jboss.org/browse/WFCORE-989
> Project: WildFly Core
> Issue Type: Bug
> Components: Domain Management
> Affects Versions: 2.0.0.CR1
> Reporter: Brian Stansberry
> Assignee: Brian Stansberry
> Fix For: 2.0.0.CR7
>
>
> ServerInventoryImpl.reconnectServer is registering the ProxyController for the reconnecting server with the DomainModelControllerService. The problem is that ProxyController is not yet in a state where it can forward requests to the server. That won't happen until the server calls back with a DomainServerProtocol.SERVER_RECONNECT_REQUEST and the ServerToHostProtocolHandler.ServerReconnectRequestHandler handles it. The effect is if a request for the server comes in during this window, it will fail. If that request is part of a domain operation rollout (because during the window the HC regarded the server as 'live' and instructed the DC to send rollout ops to it), then the domain rollout will be affected. This happened in the testsuite failure I just saw. Most significant logging is as follows:
> {code}
> 2015-09-18 17:59:32,229 INFO [org.jboss.as] (Controller Boot Thread) WFLYSRV0025: WildFly Core 2.0.0.CR2-SNAPSHOT "Kenny" (Host Controller) started in 93ms - Started 48 of 51 services (15 services are lazy, passive or on-demand)
> 2015-09-18 17:59:32,247 INFO [org.jboss.as.controller.management-operation] (Remoting "slave:MANAGEMENT" task-16) Initialized for 1849055188
> 2015-09-18 17:59:32,247 INFO [org.jboss.as.host.controller] (Remoting "slave:MANAGEMENT" task-14) WFLYHC0021: Server [Server:main-three] connected using connection [Channel ID 08aa200b (inbound) of Remoting connection 27e3d90d to /127.0.0.1:49272]
> 2015-09-18 17:59:32,247 INFO [org.jboss.as.host.controller] (Remoting "slave:MANAGEMENT" task-15) WFLYHC0021: Server [Server:other-two] connected using connection [Channel ID 02bd2646 (inbound) of Remoting connection 64a8b7f2 to /127.0.0.1:49273]
> 2015-09-18 17:59:32,247 INFO [org.jboss.as.controller.management-operation] (Host Controller Service Threads - 34) Initialized for 1849055188
> 2015-09-18 17:59:32,252 INFO [org.jboss.as.controller.management-operation] (Host Controller Service Threads - 34) sending prepared response for 1849055188 --- interrupted: false
> 2015-09-18 17:59:32,254 INFO [org.jboss.as.controller.management-operation] (Remoting "slave:MANAGEMENT" task-1) Initialized for 1933145457
> 2015-09-18 17:59:32,254 INFO [org.jboss.as.controller.management-operation] (Remoting "slave:MANAGEMENT" task-1) Initialized for 481103151
> 2015-09-18 17:59:32,254 INFO [org.jboss.as.controller.management-operation] (Host Controller Service Threads - 36) Initialized for 1933145457
> 2015-09-18 17:59:32,254 INFO [org.jboss.as.controller.management-operation] (Host Controller Service Threads - 37) Initialized for 481103151
> 2015-09-18 17:59:32,255 INFO [org.jboss.as.controller.management-operation] (Host Controller Service Threads - 36) sending pre-prepare failed response for 1933145457 --- interrupted: false
> 2015-09-18 17:59:32,256 INFO [org.jboss.as.controller.management-operation] (Host Controller Service Threads - 37) sending pre-prepare failed response for 481103151 --- interrupted: false
> {code}
> The HC completes boot. Then servers main-three and other-two register. The HC sends a prepared response to a request. This is the host rollout request to the slave from the DC with the response being the ops to invoke on the servers. The HC reports sending a "pre-prepare failed response" to two requests. These are the requests the DC has asked it to proxy to the servers.
> The result of all this for the client is the following failure of a management op:
> {code}
> Failed operation:
> {
> "operation" => "add",
> "address" => [("extension" => "org.jboss.as.test.hc.extension")]
> }
> Response:
> {
> "outcome" => "failed",
> "result" => undefined,
> "failure-description" => {"WFLYDC0074: Operation failed or was rolled back on all servers. Server failures:" => {"server-group" => {
> "main-server-group" => {"host" => {"slave" => {"main-three" => "WFLYHC0153: Channel closed"}}},
> "other-server-group" => {"host" => {"slave" => {"other-two" => "WFLYHC0153: Channel closed"}}}
> }}},
> "rolled-back" => true,
> "server-groups" => {
> "main-server-group" => {"host" => {
> "master" => {"main-one" => {"response" => {
> "outcome" => "failed",
> "result" => [("HC" => "1.1.1")],
> "rolled-back" => true
> }}},
> "slave" => {"main-three" => {"response" => {
> "outcome" => "failed",
> "result" => undefined,
> "failure-description" => "WFLYHC0153: Channel closed",
> "rolled-back" => true
> }}}
> }},
> "other-server-group" => {"host" => {"slave" => {"other-two" => {"response" => {
> "outcome" => "failed",
> "result" => undefined,
> "failure-description" => "WFLYHC0153: Channel closed",
> "rolled-back" => true
> }}}}}
> }
> }
> {code}
> The "WFLYHC0153: Channel closed" failure is what is produced when an attempt is made to invoke on a disconnected proxy controller.
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
8 years, 7 months
[JBoss JIRA] (WFCORE-988) Host controller fails to roll out management operations if assertions are enabled
by Kabir Khan (JIRA)
[ https://issues.jboss.org/browse/WFCORE-988?page=com.atlassian.jira.plugin... ]
Kabir Khan updated WFCORE-988:
------------------------------
Fix Version/s: 2.0.0.CR7
(was: 2.0.0.CR6)
> Host controller fails to roll out management operations if assertions are enabled
> ---------------------------------------------------------------------------------
>
> Key: WFCORE-988
> URL: https://issues.jboss.org/browse/WFCORE-988
> Project: WildFly Core
> Issue Type: Bug
> Components: Domain Management
> Reporter: James Perkins
> Assignee: Brian Stansberry
> Fix For: 2.0.0.CR7
>
>
> Enabling assertions, {{-ea}}, for the host controller then trying to execute an operation that needs to be rolled out to servers fails. The issue seems to be in the [{{OperationContextImpl}}|https://github.com/wildfly/wildfly-core/blob/2.0.0.CR1/controller/src/main/java/org/jboss/as/controller/OperationContextImpl.java#L826].
> Replacing the assertion with an exception I see the stack trace below.
> {code:title=Stack Trace}
> [Host Controller] 14:08:33,124 DEBUG [org.jboss.as.host.controller] (Host Controller Service Threads - 8) failed to process task {
> [Host Controller] "address" => [
> [Host Controller] ("host" => "master"),
> [Host Controller] ("server" => "server-one"),
> [Host Controller] ("subsystem" => "logging")
> [Host Controller] ],
> [Host Controller] "operation" => "write-attribute",
> [Host Controller] "name" => "use-deployment-logging-config",
> [Host Controller] "value" => true,
> [Host Controller] "operation-headers" => {
> [Host Controller] "access-mechanism" => "NATIVE",
> [Host Controller] "domain-uuid" => "1763a478-a1ad-42b5-a206-03db6e8bbc69",
> [Host Controller] "execute-for-coordinator" => true
> [Host Controller] }
> [Host Controller] }: java.lang.RuntimeException: Thread.currentThread=Thread[Host Controller Service Threads - 8,5,Host Controller Service Threads], controllingThread=null, initiatingThreadThread[management-handler-thread - 2,5,management-handler-thread]
> [Host Controller] at org.jboss.as.controller.OperationContextImpl.readResourceFromRoot(OperationContextImpl.java:832)
> [Host Controller] at org.jboss.as.controller.transform.ResourceTransformationContextImpl.create(ResourceTransformationContextImpl.java:64)
> [Host Controller] at org.jboss.as.controller.transform.TransformersImpl.transformOperation(TransformersImpl.java:88)
> [Host Controller] at org.jboss.as.controller.TransformingProxyController$Factory$1.transformOperation(TransformingProxyController.java:102)
> [Host Controller] at org.jboss.as.controller.TransformingProxyController$TransformingProxyControllerImpl.transformOperation(TransformingProxyController.java:153)
> [Host Controller] at org.jboss.as.domain.controller.operations.coordination.MultiphaseOverallContext.transformServerOperation(MultiphaseOverallContext.java:176)
> [Host Controller] at org.jboss.as.domain.controller.operations.coordination.DomainRolloutStepHandler$2.execute(DomainRolloutStepHandler.java:315)
> [Host Controller] at org.jboss.as.domain.controller.plan.ServerTaskExecutor.executeTask(ServerTaskExecutor.java:83)
> [Host Controller] at org.jboss.as.domain.controller.plan.ConcurrentServerGroupUpdateTask.execute(ConcurrentServerGroupUpdateTask.java:53)
> [Host Controller] at org.jboss.as.domain.controller.plan.AbstractServerGroupRolloutTask$1.run(AbstractServerGroupRolloutTask.java:63)
> [Host Controller] at org.jboss.as.domain.controller.plan.AbstractServerGroupRolloutTask$1.run(AbstractServerGroupRolloutTask.java:59)
> [Host Controller] at java.security.AccessController.doPrivileged(Native Method)
> [Host Controller] at javax.security.auth.Subject.doAs(Subject.java:360)
> [Host Controller] at org.jboss.as.controller.AccessAuditContext.doAs(AccessAuditContext.java:81)
> [Host Controller] at org.jboss.as.domain.controller.plan.AbstractServerGroupRolloutTask.run(AbstractServerGroupRolloutTask.java:59)
> [Host Controller] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [Host Controller] at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> [Host Controller] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [Host Controller] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [Host Controller] at java.lang.Thread.run(Thread.java:745)
> [Host Controller] at org.jboss.threads.JBossThread.run(JBossThread.java:320)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
8 years, 7 months
[JBoss JIRA] (WFCORE-745) Standalone powershell script does't support --debug and -secmgr option
by Kabir Khan (JIRA)
[ https://issues.jboss.org/browse/WFCORE-745?page=com.atlassian.jira.plugin... ]
Kabir Khan updated WFCORE-745:
------------------------------
Fix Version/s: 2.0.0.CR7
(was: 2.0.0.CR6)
> Standalone powershell script does't support --debug and -secmgr option
> ----------------------------------------------------------------------
>
> Key: WFCORE-745
> URL: https://issues.jboss.org/browse/WFCORE-745
> Project: WildFly Core
> Issue Type: Bug
> Components: Scripts
> Affects Versions: 2.0.0.Alpha3
> Reporter: Marek Kopecký
> Assignee: Tomaz Cerar
> Fix For: 2.0.0.CR7
>
>
> Standalone powershell script does't support --debug option. Also -secmgr option and SECMGR=true environment variable settings is missing in posershell script.
> *Steps to Reproduce:*
> # .\standalone.ps1 --debug
> *Actual results:*
> Debug mode is not active
> *Expected results:*
> {noformat}
> ...
> Listening for transport dt_socket at address: 8787
> ...
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
8 years, 7 months