[jboss-jira] [JBoss JIRA] (WFCORE-218) wildfly web management console hangs during deploy from cli

[Brian Stansberry (JIRA)]

    [ https://issues.jboss.org/browse/WFCORE-218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13020158#comment-13020158 ] 

Brian Stansberry commented on WFCORE-218:
-----------------------------------------

The thread dump shows 2 console request threads blocking waiting for the exclusive controller lock that is held by the thread executing the deployment op.

The console request threads are asking for the exclusive lock so they can execute a 2-phase domain wide operation. All such ops get the exclusive lock so they can prevent domain topology changes during execution.

There are two ways I want to improve this:

1) I believe when the console sends a composite operation, it is getting routed into the 2-phase domain wide operation path more often than is necessary. Improving this needs to be done carefully, but should be a relatively straightforward change and probably will have the biggest impact, as most of the CRUD screens in the console don't need to invoke ops that involve more than the domain controller.

2) Look into having a separate lock for the domain topology, and not using the exclusive controller lock to guard it. That requires care though, as now there will be two separate locks involved in operation execution. We need to be certain that all code paths always acquire them in the same order or we'll be vulnerable to deadlocks. I believe the correct order should be 1) topology lock 2) controller lock. There are relatively few points where a topology lock would be needed, and I believe they are all at the outer edge of operation execution. So it's much simpler to control those points and ensure they always get topology before doing anything that could need the controller lock.

This second step will be more important once the feature discussed at http://lists.jboss.org/pipermail/wildfly-dev/2014-October/003241.html comes in, as that will result in numerous read operations that truly need the domain topology lock.

> wildfly web management console hangs during deploy from cli
> -----------------------------------------------------------
>
>                 Key: WFCORE-218
>                 URL: https://issues.jboss.org/browse/WFCORE-218
>             Project: WildFly Core
>          Issue Type: Bug
>          Components: Domain Management
>    Affects Versions: 1.0.0.Alpha1
>            Reporter: Ian Kent
>            Assignee: Brian Stansberry
>         Attachments: threaddump-1415735255304.tdump
>
>
> We are running wildfly in domain mode with the following configuration.
> host A running domain controlller
> host B running host controller with one app sever
> host C running host controller with one app server
> host D running host controller with one app server
> When we deloy war using jboss-cli the web console is blocked for usage until deploy completes. I have run jvisualvm and it does not appear that domain controller process is starved for resources (cpu, memory, threads).



--
This message was sent by Atlassian JIRA
(v6.3.8#6338)