[JBoss JIRA] (WFCORE-205) ManagedDMRContentTypeResource does not use a map with consistent ordering for storing content
by RH Bugzilla Integration (JIRA)
[ https://issues.jboss.org/browse/WFCORE-205?page=com.atlassian.jira.plugin... ]
RH Bugzilla Integration commented on WFCORE-205:
------------------------------------------------
Brian Stansberry <brian.stansberry(a)redhat.com> changed the Status of [bug 1078062|https://bugzilla.redhat.com/show_bug.cgi?id=1078062] from NEW to ASSIGNED
> ManagedDMRContentTypeResource does not use a map with consistent ordering for storing content
> ---------------------------------------------------------------------------------------------
>
> Key: WFCORE-205
> URL: https://issues.jboss.org/browse/WFCORE-205
> Project: WildFly Core
> Issue Type: Bug
> Components: Domain Management
> Affects Versions: 1.0.0.Alpha10
> Reporter: Brian Stansberry
> Assignee: Brian Stansberry
> Fix For: 1.0.0.Beta1
>
>
> ManagedDMRContentTypeResource.content is used for generating an overall hash for the stored content, but it does not use consistent ordering (i.e. it needs to be a LinkedHashMap or perhaps a TreeMap.) The result is when another node in the domain receives an update it may calculate a different overall hash.
> This is surfacing as failures in https://bugzilla.redhat.com/show_bug.cgi?id=1078062 when the master and slave are running on different JVM releases. Different VM releases often have different ordering behavior when iterating over the unordered collections.
> I believe this is unlikely to result in real-world problems since any backup will persist its own version of the hash to its local copy of domain.xml, so there won't be any mismatch. The importance of the overall hash is that it allows the process to find the content in the repo when it is added.
--
This message was sent by Atlassian JIRA
(v6.3.1#6329)
11 years, 9 months
[JBoss JIRA] (WFCORE-205) ManagedDMRContentTypeResource does not use a map with consistent ordering for storing content
by RH Bugzilla Integration (JIRA)
[ https://issues.jboss.org/browse/WFCORE-205?page=com.atlassian.jira.plugin... ]
RH Bugzilla Integration updated WFCORE-205:
-------------------------------------------
Bugzilla Update: Perform
Bugzilla References: https://bugzilla.redhat.com/show_bug.cgi?id=1078062
> ManagedDMRContentTypeResource does not use a map with consistent ordering for storing content
> ---------------------------------------------------------------------------------------------
>
> Key: WFCORE-205
> URL: https://issues.jboss.org/browse/WFCORE-205
> Project: WildFly Core
> Issue Type: Bug
> Components: Domain Management
> Affects Versions: 1.0.0.Alpha10
> Reporter: Brian Stansberry
> Assignee: Brian Stansberry
> Fix For: 1.0.0.Beta1
>
>
> ManagedDMRContentTypeResource.content is used for generating an overall hash for the stored content, but it does not use consistent ordering (i.e. it needs to be a LinkedHashMap or perhaps a TreeMap.) The result is when another node in the domain receives an update it may calculate a different overall hash.
> This is surfacing as failures in https://bugzilla.redhat.com/show_bug.cgi?id=1078062 when the master and slave are running on different JVM releases. Different VM releases often have different ordering behavior when iterating over the unordered collections.
> I believe this is unlikely to result in real-world problems since any backup will persist its own version of the hash to its local copy of domain.xml, so there won't be any mismatch. The importance of the overall hash is that it allows the process to find the content in the repo when it is added.
--
This message was sent by Atlassian JIRA
(v6.3.1#6329)
11 years, 9 months
[JBoss JIRA] (WFCORE-205) ManagedDMRContentTypeResource does not use a map with consistent ordering for storing content
by Brian Stansberry (JIRA)
Brian Stansberry created WFCORE-205:
---------------------------------------
Summary: ManagedDMRContentTypeResource does not use a map with consistent ordering for storing content
Key: WFCORE-205
URL: https://issues.jboss.org/browse/WFCORE-205
Project: WildFly Core
Issue Type: Bug
Components: Domain Management
Affects Versions: 1.0.0.Alpha10
Reporter: Brian Stansberry
Assignee: Brian Stansberry
Fix For: 1.0.0.Beta1
ManagedDMRContentTypeResource.content is used for generating an overall hash for the stored content, but it does not use consistent ordering (i.e. it needs to be a LinkedHashMap or perhaps a TreeMap.) The result is when another node in the domain receives an update it may calculate a different overall hash.
This is surfacing as failures in https://bugzilla.redhat.com/show_bug.cgi?id=1078062 when the master and slave are running on different JVM releases. Different VM releases often have different ordering behavior when iterating over the unordered collections.
I believe this is unlikely to result in real-world problems since any backup will persist its own version of the hash to its local copy of domain.xml, so there won't be any mismatch. The importance of the overall hash is that it allows the process to find the content in the repo when it is added.
--
This message was sent by Atlassian JIRA
(v6.3.1#6329)
11 years, 9 months
[JBoss JIRA] (JGRP-1895) FLUSH_NOT_COMPLETED race condition
by Dennis Reed (JIRA)
[ https://issues.jboss.org/browse/JGRP-1895?page=com.atlassian.jira.plugin.... ]
Dennis Reed commented on JGRP-1895:
-----------------------------------
Possible fix for the specific issue in my previous comment:
When handling FLUSH_NOT_COMPLETED, if flushCompleted is true (which will never be the case if
the FLUSH_NOT_COMPLETED is for the current flush), then just ignore it.
> FLUSH_NOT_COMPLETED race condition
> ----------------------------------
>
> Key: JGRP-1895
> URL: https://issues.jboss.org/browse/JGRP-1895
> Project: JGroups
> Issue Type: Bug
> Affects Versions: 2.6.16
> Reporter: Dennis Reed
> Assignee: Dennis Reed
>
> FLUSH_NOT_COMPLETED does not keep track of which FLUSH it was related to. FLUSH_COMPLETED only keeps track of whether the FLUSH was for the current view.
> If these responses are delayed (which can be caused by a long GC pause on the node sending it) where a new FLUSH has been started, these can be interpreted as responses for the wrong FLUSH.
--
This message was sent by Atlassian JIRA
(v6.3.1#6329)
11 years, 9 months
[JBoss JIRA] (JGRP-1895) FLUSH_NOT_COMPLETED race condition
by Dennis Reed (JIRA)
[ https://issues.jboss.org/browse/JGRP-1895?page=com.atlassian.jira.plugin.... ]
Dennis Reed updated JGRP-1895:
------------------------------
Description:
FLUSH_NOT_COMPLETED does not keep track of which FLUSH it was related to. FLUSH_COMPLETED only keeps track of whether the FLUSH was for the current view.
If these responses are delayed (which can be caused by
a long GC pause on the node sending it) where a new FLUSH has been started, these can be interpreted as responses for the wrong FLUSH.
was:
FLUSH_NOT_COMPLETED does not keep track of which FLUSH it was related to. FLUSH_COMPLETED only keeps track of whether the FLUSH was for the current view.
If these responses are delayed (which can be G/caused by
a long GC pause on the node sending it) where a new FLUSH has been started, these can be interpreted as responses for the wrong FLUSH.
> FLUSH_NOT_COMPLETED race condition
> ----------------------------------
>
> Key: JGRP-1895
> URL: https://issues.jboss.org/browse/JGRP-1895
> Project: JGroups
> Issue Type: Bug
> Affects Versions: 2.6.16
> Reporter: Dennis Reed
> Assignee: Dennis Reed
>
> FLUSH_NOT_COMPLETED does not keep track of which FLUSH it was related to. FLUSH_COMPLETED only keeps track of whether the FLUSH was for the current view.
> If these responses are delayed (which can be caused by
> a long GC pause on the node sending it) where a new FLUSH has been started, these can be interpreted as responses for the wrong FLUSH.
--
This message was sent by Atlassian JIRA
(v6.3.1#6329)
11 years, 9 months
[JBoss JIRA] (JGRP-1895) FLUSH_NOT_COMPLETED race condition
by Dennis Reed (JIRA)
[ https://issues.jboss.org/browse/JGRP-1895?page=com.atlassian.jira.plugin.... ]
Dennis Reed updated JGRP-1895:
------------------------------
Description:
FLUSH_NOT_COMPLETED does not keep track of which FLUSH it was related to. FLUSH_COMPLETED only keeps track of whether the FLUSH was for the current view.
If these responses are delayed (which can be caused by a long GC pause on the node sending it) where a new FLUSH has been started, these can be interpreted as responses for the wrong FLUSH.
was:
FLUSH_NOT_COMPLETED does not keep track of which FLUSH it was related to. FLUSH_COMPLETED only keeps track of whether the FLUSH was for the current view.
If these responses are delayed (which can be caused by
a long GC pause on the node sending it) where a new FLUSH has been started, these can be interpreted as responses for the wrong FLUSH.
> FLUSH_NOT_COMPLETED race condition
> ----------------------------------
>
> Key: JGRP-1895
> URL: https://issues.jboss.org/browse/JGRP-1895
> Project: JGroups
> Issue Type: Bug
> Affects Versions: 2.6.16
> Reporter: Dennis Reed
> Assignee: Dennis Reed
>
> FLUSH_NOT_COMPLETED does not keep track of which FLUSH it was related to. FLUSH_COMPLETED only keeps track of whether the FLUSH was for the current view.
> If these responses are delayed (which can be caused by a long GC pause on the node sending it) where a new FLUSH has been started, these can be interpreted as responses for the wrong FLUSH.
--
This message was sent by Atlassian JIRA
(v6.3.1#6329)
11 years, 9 months
[JBoss JIRA] (JGRP-1895) FLUSH_NOT_COMPLETED race condition
by Dennis Reed (JIRA)
[ https://issues.jboss.org/browse/JGRP-1895?page=com.atlassian.jira.plugin.... ]
Dennis Reed commented on JGRP-1895:
-----------------------------------
Specific case where this caused a major outage:
A new node was joining the cluster (JOIN_REQ_WITH_STATE_TRANSFER).
Cooordinator started flush.
nodeB was in GC and didn't respond, so the flush timed out.
Coordinator started another flush.
nodeB responded to the original flush with FLUSH_COMPLETED, and the coordinator thought it was a response for the current flush when it was not.
FLUSH completed (when it shouldn't have), and coordinator started a reconcile.
nodeB then responded to the current FLUSH with FLUSH_NOT_COMPLETED, which set flush_promise to false to indicate a failure.
Since the FLUSH actually succeeded, the coordinator did not abort it.
But since it was reported as failing, the new node did not join the cluster, and so never stopped the FLUSH.
FLUSH was then stuck in an incomplete state until the channel was restarted.
> FLUSH_NOT_COMPLETED race condition
> ----------------------------------
>
> Key: JGRP-1895
> URL: https://issues.jboss.org/browse/JGRP-1895
> Project: JGroups
> Issue Type: Bug
> Affects Versions: 2.6.16
> Reporter: Dennis Reed
> Assignee: Dennis Reed
>
> FLUSH_NOT_COMPLETED does not keep track of which FLUSH it was related to. FLUSH_COMPLETED only keeps track of whether the FLUSH was for the current view.
> If these responses are delayed (which can be G/caused by
> a long GC pause on the node sending it) where a new FLUSH has been started, these can be interpreted as responses for the wrong FLUSH.
--
This message was sent by Atlassian JIRA
(v6.3.1#6329)
11 years, 9 months
[JBoss JIRA] (JGRP-1895) FLUSH_NOT_COMPLETED race condition
by Dennis Reed (JIRA)
Dennis Reed created JGRP-1895:
---------------------------------
Summary: FLUSH_NOT_COMPLETED race condition
Key: JGRP-1895
URL: https://issues.jboss.org/browse/JGRP-1895
Project: JGroups
Issue Type: Bug
Affects Versions: 2.6.16
Reporter: Dennis Reed
Assignee: Dennis Reed
FLUSH_NOT_COMPLETED does not keep track of which FLUSH it was related to. FLUSH_COMPLETED only keeps track of whether the FLUSH was for the current view.
If these responses are delayed (which can be G/caused by
a long GC pause on the node sending it) where a new FLUSH has been started, these can be interpreted as responses for the wrong FLUSH.
--
This message was sent by Atlassian JIRA
(v6.3.1#6329)
11 years, 9 months