[JBoss JIRA] (JGRP-2150) More efficient message adding and draining
by Bela Ban (JIRA)
[ https://issues.jboss.org/browse/JGRP-2150?page=com.atlassian.jira.plugin.... ]
Bela Ban edited comment on JGRP-2150 at 1/5/17 7:43 AM:
--------------------------------------------------------
In pseudo-code, this could look like this (similar code for single messages):
{code:java}
final AtomicInteger counter=new AtomicInteger(0);
void add(MessageBatch batch) {
int size=batch.size();
if(counter.getAndAdd(size) == 0) {
do {
if(batch != null) { // initial batch
deliver(batch);
batch=null;
}
else {
batch=table.remove();
if(batch == null) --> continue
size=batch.size();
deliver(batch);
}
} while(counter.decrementAndGet(size) != 0);
}
else {
table.add(batch); // queues batch
return;
}
}
{code}
was (Author: belaban):
In pseudo-code, this could look like this (similar code for single messages):
{code:java}
final AtomicInteger counter=new AtomicInteger(0);
void add(MessageBatch batch) {
int size=batch.size();
if(counter.getAndAdd(size) == 0) {
do {
if(batch != null) { // initial batch
deliver(batch);
batch=null;
}
else {
batch=table.remove();
if(batch == null) --> continue
size=batch.size();
}
} while(counter.decrementAndGet(size) != 0);
}
else {
table.add(batch); // queues batch
return;
}
}
{code}
> More efficient message adding and draining
> ------------------------------------------
>
> Key: JGRP-2150
> URL: https://issues.jboss.org/browse/JGRP-2150
> Project: JGroups
> Issue Type: Enhancement
> Reporter: Bela Ban
> Assignee: Bela Ban
> Labels: CR1
> Fix For: 4.0
>
>
> In NAKACK2, UNICAST3 and in MaxOneThreadPerSenderPolicy, we have a pattern where one or more producers add messages (to a table in NAKACK2 and UNICAST3, or to a MessageBatch in MaxOneThreadPerSenderPolicy) and then only *a single thread* can remove and deliver messages up the stack.
> This requires synchronization around (1) determining the thread which will remove messages, (2) adding messages to the table (or batch) and (3) removing messages from the table or batch.
> Unit tests DrainTest and MessageBatchDrainTest show how a simple AtomicInteger can be used to do this.
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
9 years, 4 months
[JBoss JIRA] (JGRP-2150) More efficient message adding and draining
by Bela Ban (JIRA)
[ https://issues.jboss.org/browse/JGRP-2150?page=com.atlassian.jira.plugin.... ]
Bela Ban commented on JGRP-2150:
--------------------------------
In pseudo-code, this could look like this (similar code for single messages):
{code:java}
final AtomicInteger counter=new AtomicInteger(0);
void add(MessageBatch batch) {
int size=batch.size();
if(counter.getAndAdd(size) == 0) {
do {
if(batch != null) { // initial batch
deliver(batch);
batch=null;
}
else {
batch=table.remove();
if(batch == null) --> continue
size=batch.size();
}
} while(counter.decrementAndGet(size) != 0);
}
else {
table.add(batch); // queues batch
return;
}
}
{code}
> More efficient message adding and draining
> ------------------------------------------
>
> Key: JGRP-2150
> URL: https://issues.jboss.org/browse/JGRP-2150
> Project: JGroups
> Issue Type: Enhancement
> Reporter: Bela Ban
> Assignee: Bela Ban
> Labels: CR1
> Fix For: 4.0
>
>
> In NAKACK2, UNICAST3 and in MaxOneThreadPerSenderPolicy, we have a pattern where one or more producers add messages (to a table in NAKACK2 and UNICAST3, or to a MessageBatch in MaxOneThreadPerSenderPolicy) and then only *a single thread* can remove and deliver messages up the stack.
> This requires synchronization around (1) determining the thread which will remove messages, (2) adding messages to the table (or batch) and (3) removing messages from the table or batch.
> Unit tests DrainTest and MessageBatchDrainTest show how a simple AtomicInteger can be used to do this.
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
9 years, 4 months
[JBoss JIRA] (DROOLS-1397) Drools condition cannot retrieve object from Map with Enum key
by Patrick Mingard (JIRA)
[ https://issues.jboss.org/browse/DROOLS-1397?page=com.atlassian.jira.plugi... ]
Patrick Mingard updated DROOLS-1397:
------------------------------------
Workaround Description:
As proposed on the stackoverflow topic, this works :
{code:java}
rule comms
when
$conv: Convention()
$comm: Comm() from (ArrayList)($conv.getComms()).get(Type.AAA)
then [...]
{code}
Please note the difference is an ArrayList cast (with a few parentheses).
was:
As proposed on the stackoverflow topic, this works :
rule comms
when
$conv: Convention()
$comm: Comm() from (ArrayList)($conv.getComms()).get(Type.AAA)
then [...]
Please note the difference is an ArrayList cast (with a few parentheses).
> Drools condition cannot retrieve object from Map with Enum key
> --------------------------------------------------------------
>
> Key: DROOLS-1397
> URL: https://issues.jboss.org/browse/DROOLS-1397
> Project: Drools
> Issue Type: Bug
> Components: core engine, decision tables
> Affects Versions: 6.5.0.Final
> Environment: JDK 1.7, JUnit test, exact drools version is 6.5.0.Final-redhat-2
> Reporter: Patrick Mingard
> Assignee: Mario Fusco
> Priority: Minor
> Fix For: 7.0.0.CR1
>
>
> I'm migrating an application that used drools 5.3 on the newer drools 6.5 version.
> I'm using a decision table, And one of my condition is accessing a Map which has an enum class as a key. It is not compiling unless I do a strange ArrayList cast.
> Please refer to [this stackoverflow topic|http://stackoverflow.com/questions/41461468/drools-condition-cannot...] for details.
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
9 years, 4 months
[JBoss JIRA] (DROOLS-1397) Drools condition cannot retrieve object from Map with Enum key
by Patrick Mingard (JIRA)
[ https://issues.jboss.org/browse/DROOLS-1397?page=com.atlassian.jira.plugi... ]
Patrick Mingard updated DROOLS-1397:
------------------------------------
Steps to Reproduce:
A fact :
{code:java}
public class Convention {
private Map<ECommissionType, List<Commission>> commissions = new HashMap<>();
}
{code}
An enumeration :
{code:java}
public enum ECommissionType {
ACQ,RIS,POF,[...]
}
{code}
And a rule condition assuming $convention is properly initialized in a former condition and is an instance of the Convention class shown) :
{code:java}
$comm : Commission() from $convention.getCommissions().get(ECommissionType.$1)
{code}
Compilation using kie-maven-plugin is raising an error :
{code:java}
Unable to Analyse Expression $convention.getCommissions().get(ECommissionType.RIS)):
sun.reflect.generics.reflectiveObjects.ParameterizedTypeImpl cannot be cast to java.lang.Class : [Rule name='B_CommissionLigne_98']
{code}
(I'm sorry I'm completely unable to format correctly the code extracts on this site... more readable on stackoverflow)
was:
A fact :
{{ public class Convention {
[...]
private Map<ECommissionType, List<Commission>> commissions = new HashMap<>();
[...]
}}}
An enumeration :
{{ public enum ECommissionType {
ACQ,RIS,POF,[...]
}}}
And a rule condition assuming $convention is properly initialized in a former condition and is an instance of the Convention class shown) :
$comm : Commission() from $convention.getCommissions().get(ECommissionType.$1)
Compilation using kie-maven-plugin is raising an error :
Unable to Analyse Expression $convention.getCommissions().get(ECommissionType.RIS)):
sun.reflect.generics.reflectiveObjects.ParameterizedTypeImpl cannot be cast to java.lang.Class : [Rule name='B_CommissionLigne_98']
(I'm sorry I'm completely unable to format correctly the code extracts on this site... more readable on stackoverflow)
> Drools condition cannot retrieve object from Map with Enum key
> --------------------------------------------------------------
>
> Key: DROOLS-1397
> URL: https://issues.jboss.org/browse/DROOLS-1397
> Project: Drools
> Issue Type: Bug
> Components: core engine, decision tables
> Affects Versions: 6.5.0.Final
> Environment: JDK 1.7, JUnit test, exact drools version is 6.5.0.Final-redhat-2
> Reporter: Patrick Mingard
> Assignee: Mario Fusco
> Priority: Minor
> Fix For: 7.0.0.CR1
>
>
> I'm migrating an application that used drools 5.3 on the newer drools 6.5 version.
> I'm using a decision table, And one of my condition is accessing a Map which has an enum class as a key. It is not compiling unless I do a strange ArrayList cast.
> Please refer to [this stackoverflow topic|http://stackoverflow.com/questions/41461468/drools-condition-cannot...] for details.
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
9 years, 4 months
[JBoss JIRA] (DROOLS-1397) Drools condition cannot retrieve object from Map with Enum key
by Patrick Mingard (JIRA)
[ https://issues.jboss.org/browse/DROOLS-1397?page=com.atlassian.jira.plugi... ]
Patrick Mingard updated DROOLS-1397:
------------------------------------
Steps to Reproduce:
A fact :
{code:java}
public class Convention {
private Map<ECommissionType, List<Commission>> commissions = new HashMap<>();
}
{code}
An enumeration :
{code:java}
public enum ECommissionType {
ACQ,RIS,POF,[...]
}
{code}
And a rule condition assuming $convention is properly initialized in a former condition and is an instance of the Convention class shown) :
{code:java}
$comm : Commission() from $convention.getCommissions().get(ECommissionType.$1)
{code}
Compilation using kie-maven-plugin is raising an error :
{code:java}
Unable to Analyse Expression $convention.getCommissions().get(ECommissionType.RIS)):
sun.reflect.generics.reflectiveObjects.ParameterizedTypeImpl cannot be cast to java.lang.Class : [Rule name='B_CommissionLigne_98']
{code}
was:
A fact :
{code:java}
public class Convention {
private Map<ECommissionType, List<Commission>> commissions = new HashMap<>();
}
{code}
An enumeration :
{code:java}
public enum ECommissionType {
ACQ,RIS,POF,[...]
}
{code}
And a rule condition assuming $convention is properly initialized in a former condition and is an instance of the Convention class shown) :
{code:java}
$comm : Commission() from $convention.getCommissions().get(ECommissionType.$1)
{code}
Compilation using kie-maven-plugin is raising an error :
{code:java}
Unable to Analyse Expression $convention.getCommissions().get(ECommissionType.RIS)):
sun.reflect.generics.reflectiveObjects.ParameterizedTypeImpl cannot be cast to java.lang.Class : [Rule name='B_CommissionLigne_98']
{code}
(I'm sorry I'm completely unable to format correctly the code extracts on this site... more readable on stackoverflow)
> Drools condition cannot retrieve object from Map with Enum key
> --------------------------------------------------------------
>
> Key: DROOLS-1397
> URL: https://issues.jboss.org/browse/DROOLS-1397
> Project: Drools
> Issue Type: Bug
> Components: core engine, decision tables
> Affects Versions: 6.5.0.Final
> Environment: JDK 1.7, JUnit test, exact drools version is 6.5.0.Final-redhat-2
> Reporter: Patrick Mingard
> Assignee: Mario Fusco
> Priority: Minor
> Fix For: 7.0.0.CR1
>
>
> I'm migrating an application that used drools 5.3 on the newer drools 6.5 version.
> I'm using a decision table, And one of my condition is accessing a Map which has an enum class as a key. It is not compiling unless I do a strange ArrayList cast.
> Please refer to [this stackoverflow topic|http://stackoverflow.com/questions/41461468/drools-condition-cannot...] for details.
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
9 years, 4 months
[JBoss JIRA] (JGRP-2135) OOM with JGroups 3.6.11.
by Bela Ban (JIRA)
[ https://issues.jboss.org/browse/JGRP-2135?page=com.atlassian.jira.plugin.... ]
Bela Ban resolved JGRP-2135.
----------------------------
Resolution: Rejected
Please reopen if you can reproduce this issue (without ssh'ing to the JGroups port)...
> OOM with JGroups 3.6.11.
> ------------------------
>
> Key: JGRP-2135
> URL: https://issues.jboss.org/browse/JGRP-2135
> Project: JGroups
> Issue Type: Bug
> Affects Versions: 3.6.11
> Reporter: Zoltan Farkas
> Assignee: Bela Ban
> Fix For: 3.6.12
>
>
> We are running our JVMs with : -XX:OnOutOfMemoryError="kill -9 %p"
> we have been experiencing OOMs fairly often, and the OOMs happen at:
> {code}
> Object / Stack Frame |Name | Shallow Heap | Retained Heap |Context Class Loader |Is Daemon
> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> java.lang.Thread @ 0x81bdf838 |Connection.Receiver [144.77.77.53:50363 - 144.77.77.53:50363],sis-cluster.service,prodpmwsv5-6461| 120 | 456 |sun.misc.Launcher$AppClassLoader @ 0x800175a8|false
> |- at java.lang.OutOfMemoryError.<init>()V (OutOfMemoryError.java:48) | | | | |
> |- at org.jgroups.blocks.cs.TcpConnection$Receiver.run()V (TcpConnection.java:310)| | | | |
> |- at java.lang.Thread.run()V (Thread.java:745) | | | | |
> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> {code}
> the Code where it happens is in TcpConnection.java:
> {code}
> while(canRun()) {
> try {
> int len=in.readInt();
> if(buffer == null || buffer.length < len)
> buffer=new byte[len];
> in.readFully(buffer, 0, len);
> updateLastAccessed();
> server.receive(peer_addr, buffer, 0, len);
> }
> catch(OutOfMemoryError mem_ex) {
> t=mem_ex;
> break; // continue;
> }
> catch(IOException io_ex) {
> t=io_ex;
> break;
> }
> catch(Throwable e) {
> }
> }
> {code}
> when allocating: buffer=new byte[len];
> it looks to me that some invalid large value is received and the process OOMs when allocating a huge byte array
> Running JVMs without kill on OOM would make this issue "dissapear" in the sense that it is swallowed by:
> {code}
> catch(OutOfMemoryError mem_ex) {
> t=mem_ex;
> break; // continue;
> }
> {code}
> Handling OutOfMemoryError is a strange implementation choice...
> instead a size limit should be employed to protect from receiving invalid sizes...
> My heap limit is 1GB and my heap dumps are 50Mb so the attempted allocation size is huge...
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
9 years, 4 months
[JBoss JIRA] (JGRP-2030) GMS: view_ack_collection_timeout delay when last 2 members leave concurrently
by Bela Ban (JIRA)
[ https://issues.jboss.org/browse/JGRP-2030?page=com.atlassian.jira.plugin.... ]
Bela Ban resolved JGRP-2030.
----------------------------
Resolution: Won't Fix
> GMS: view_ack_collection_timeout delay when last 2 members leave concurrently
> -----------------------------------------------------------------------------
>
> Key: JGRP-2030
> URL: https://issues.jboss.org/browse/JGRP-2030
> Project: JGroups
> Issue Type: Bug
> Affects Versions: 3.6.8
> Reporter: Dan Berindei
> Assignee: Bela Ban
> Priority: Minor
> Fix For: 3.6.12
>
>
> When the coordinator ({{NodeE}}) leaves, it tries to install a new view on behalf of the new coordinator ({{NodeG}}, the last member).
> {noformat}
> 21:33:26,844 TRACE (ViewHandler,InitialClusterSizeTest-NodeE-42422:) [GMS] InitialClusterSizeTest-NodeE-42422: mcasting view [InitialClusterSizeTest-NodeG-30521|3] (1) [InitialClusterSizeTest-NodeG-30521] (1 mbrs)
> 21:33:26,844 TRACE (ViewHandler,InitialClusterSizeTest-NodeE-42422:) [TCP_NIO2] InitialClusterSizeTest-NodeE-42422: sending msg to null, src=InitialClusterSizeTest-NodeE-42422, headers are GMS: GmsHeader[VIEW], NAKACK2: [MSG, seqno=1], TP: [cluster_name=ISPN]
> {noformat}
> The message is actually sent later by the bundler, but {{NodeG}} is also sending its {{LEAVE_REQ}} message at the same time. Both nodes try to create a connection to each other, and only {{NodeG}} succeeds:
> {noformat}
> 21:33:26,844 TRACE (ForkThread-2,InitialClusterSizeTest:) [TCP_NIO2] InitialClusterSizeTest-NodeG-30521: sending msg to InitialClusterSizeTest-NodeE-42422, src=InitialClusterSizeTest-NodeG-30521, headers are GMS: GmsHeader[LEAVE_REQ]: mbr=InitialClusterSizeTest-NodeG-30521, UNICAST3: DATA, seqno=1, conn_id=1, first, TP: [cluster_name=ISPN]
> 21:33:26,865 TRACE (Timer-2,InitialClusterSizeTest-NodeG-30521:) [TCP_NIO2] InitialClusterSizeTest-NodeG-30521: sending 1 msgs (83 bytes (0.27% of max_bundle_size) to 1 dests(s): [ISPN:InitialClusterSizeTest-NodeE-42422]
> 21:33:26,865 TRACE (Timer-2,InitialClusterSizeTest-NodeE-42422:) [TCP_NIO2] InitialClusterSizeTest-NodeE-42422: sending 1 msgs (91 bytes (0.29% of max_bundle_size) to 1 dests(s): [ISPN]
> 21:33:26,865 TRACE (Timer-2,InitialClusterSizeTest-NodeG-30521:) [TCP_NIO2] dest=127.0.0.1:7900 (86 bytes)
> 21:33:26,865 TRACE (Timer-2,InitialClusterSizeTest-NodeE-42422:) [TCP_NIO2] dest=127.0.0.1:7920 (94 bytes)
> 21:33:26,865 TRACE (Timer-2,InitialClusterSizeTest-NodeE-42422:) [TCP_NIO2] 127.0.0.1:7900: connecting to 127.0.0.1:7920
> 21:33:26,865 TRACE (Timer-2,InitialClusterSizeTest-NodeG-30521:) [TCP_NIO2] 127.0.0.1:7920: connecting to 127.0.0.1:7900
> 21:33:26,866 TRACE (NioConnection.Reader [null],InitialClusterSizeTest-NodeG-30521:) [TCP_NIO2] 127.0.0.1:7920: rejected connection from 127.0.0.1:7900 (connection existed and my address won as it's higher)
> 21:33:26,867 TRACE (OOB-1,InitialClusterSizeTest-NodeE-42422:) [TCP_NIO2] InitialClusterSizeTest-NodeE-42422: received [dst: InitialClusterSizeTest-NodeE-42422, src: InitialClusterSizeTest-NodeG-30521 (3 headers), size=0 bytes, flags=OOB], headers are GMS: GmsHeader[LEAVE_REQ]: mbr=InitialClusterSizeTest-NodeG-30521, UNICAST3: DATA, seqno=1, conn_id=1, first, TP: [cluster_name=ISPN]
> {noformat}
> I'm guessing {{NodeE}} would need a {{STABLE}} round in order to retransmit the {{VIEW}} message, but I'm not sure if the stable round would work, since it already (partially?) installed the new view with {{NodeG}} as the only member. However, I think it should be possible for {{NodeE}} to remove {{NodeG}} from it's {{AckCollector}} once it receives its {{LEAVE_REQ}}, and stop blocking.
> This is a minor annoyance a few the Infinispan tests - most of them shut down the nodes serially, so they don't see this delay.
> The question is whether the concurrent connection setup can have an impact for other messages as well - e.g. during startup, when there aren't a lot of messages being sent around to trigger retransmission. Could the node that failed to open its connection retry immediately on the connection opened by the other node?
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
9 years, 4 months
[JBoss JIRA] (JGRP-2136) Merge installs the same view
by Bela Ban (JIRA)
[ https://issues.jboss.org/browse/JGRP-2136?page=com.atlassian.jira.plugin.... ]
Bela Ban resolved JGRP-2136.
----------------------------
Fix Version/s: (was: 3.6.12)
Resolution: Won't Fix
See reason for not fixing this in the comments at the tail of the issue
> Merge installs the same view
> ----------------------------
>
> Key: JGRP-2136
> URL: https://issues.jboss.org/browse/JGRP-2136
> Project: JGroups
> Issue Type: Bug
> Reporter: Bela Ban
> Assignee: Bela Ban
> Fix For: 4.0
>
> Attachments: DuplicateMergeViewTest.java
>
>
> The cluster is A,B with AUTH and ASYM_ENCRYPT. When a rogue member C (whith an incorrect auth_value in AUTH) attempts to join, it will be rejected by AUTH. However, while the subsequent merge attempts also fail (as designed), this leads to spurious MergeView installations in A and B, e.g.:
> {noformat}
> ** View=[belasmac-17416|1] (2) [belasmac-17416, belasmac-56188]
> ** MergeView::[belasmac-17416|2] (2) [belasmac-17416, belasmac-56188], 1 subgroups: [belasmac-17416|1] (2) [belasmac-17416, belasmac-56188]
> ** MergeView::[belasmac-17416|3] (2) [belasmac-17416, belasmac-56188], 1 subgroups: [belasmac-17416|2] (2) [belasmac-17416, belasmac-56188]
> ** MergeView::[belasmac-17416|4] (2) [belasmac-17416, belasmac-56188], 1 subgroups: [belasmac-17416|3] (2) [belasmac-17416, belasmac-56188]
> ** MergeView::[belasmac-17416|5] (2) [belasmac-17416, belasmac-56188], 1 subgroups: [belasmac-17416|4] (2) [belasmac-17416, belasmac-56188]
> ** MergeView::[belasmac-17416|6] (2) [belasmac-17416, belasmac-56188], 1 subgroups: [belasmac-17416|5] (2) [belasmac-17416, belasmac-56188]
> ** MergeView::[belasmac-17416|7] (2) [belasmac-17416, belasmac-56188], 1 subgroups: [belasmac-17416|6] (2) [belasmac-17416, belasmac-56188]
> ** MergeView::[belasmac-17416|8] (2) [belasmac-17416, belasmac-56188], 1 subgroups: [belasmac-17416|7] (2) [belasmac-17416, belasmac-56188]
> {noformat}
> This neither corrupts security of the system (the rogue member cannot merge-join) nor correctness, but we need to prevent the spurious views. Systems like Infinispan might start a rebalance on a view, regardless of whether they are the same as before or not.
> SOLUTION: the merge leader needs to see if the MergeView it is about to send out is the same as the current view, and simply drop it if that's the case.
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
9 years, 4 months