[JBoss JIRA] (JGRP-2171) New bundler with max_bundle_size for each destination
by Bela Ban (JIRA)
[ https://issues.jboss.org/browse/JGRP-2171?page=com.atlassian.jira.plugin.... ]
Bela Ban edited comment on JGRP-2171 at 6/9/17 2:06 AM:
--------------------------------------------------------
h2. Alternative designs
h3. Alternating bundler
This bundler sends a batch as soon as the target destination changes, e.g. for sequence {{\[A A B C C C B A A A\]}}, batches {{\[A A A\]}}, {{B}} (single message), {{\[C C C\]}}, {{B}} and {{\[A A\]}} will be sent.
Of course, {{max_bundle_size}} is still observed; if we encounter a sequence whose accumulated size exceeds it, then a batch is sent immediately.
The advantage is that messages or message batches are sent immediately (reducing latency) and that this is a simple design (similar to the one above with {{num_flips=1}}. The disadvantage is that for sequences such as {{\[A B A B A B\]}}, we'll send 6 single messages, so this degenerates into a {{NoBundler}}.
h3. Queue-based bundler
This bundler drains messages from the main queue (to which sender threads add their messages) into a remove-queue of fixed length. Then we iterate through the queue and add messages to lists keyed by the target destination and finally send a batch (or single message) for each destination.
In the above example, we'd send 3 batches {{\[A A A A A\]}}, {{\[B B\]}} and {{\[C C C\]}}. Contrast this to the 5 batches (or single messages) that we send with the alternating bundler above.
The size of the queue determines the max latency: a bigger queue will result in more throughput but also higher latency. A queue of 1 is more or less the {{NoBundler}}.
was (Author: belaban):
h2. Alternative designs
h3. Alternating bundler
This bundler sends a batch as soon as the target destination changes, e.g. for sequence {{\[A A B C C C B A A A\]}}, batches {{\[A A A\]}}, {{B}} (single message), {{\[C C C\]}}, {{B}} and {{\[A A\]}} will be sent.
Of course, {{max_bundle_size}} is still observed; if we encounter a sequence whose accumulated size exceeds it, then a batch is sent immediately.
The advantage is that messages or message batches are sent immediately (reducing latency) and that this is a simple design (similar to the one above with {{num_flips=1}}. The disadvantage is that for sequences such as {{\[A B A B A B\]}}, we'll send 6 single messages, so this degenerates into a {{NoBundler}}.
h3. Queue-based bundler
This bundler drains messages from the main queue (to which sender threads add their messages) into a remove-queue of fixed length. Then we iterate through the queue and add messages to lists keyed by the target destination and finally send a batch (or single message) for each destination.
In the above example, we'd send 3 batches {{\[A A A A A\]}}, {{\[B B\]}} and {{\[C C C\]}}. Contrast this to the 5 batches (or single messages) that we send with the alternating bundler above.
> New bundler with max_bundle_size for each destination
> -----------------------------------------------------
>
> Key: JGRP-2171
> URL: https://issues.jboss.org/browse/JGRP-2171
> Project: JGroups
> Issue Type: Feature Request
> Reporter: Bela Ban
> Assignee: Bela Ban
> Fix For: 4.0.4
>
>
> The current bundlers queue all messages and when the total size of all messages for all destinations would exceed {{max_bundle_size}}, message batches for each destination are sent.
> This negatively affects latency-sensitive applications, e.g. when we have a queue such as this: {{A B B C B B D B B}}, then the message for A has to wait until either the queue is full ({{max_bundle_size exceeded}}), or no more messages are received (and then we send the batches anyway).
> The goal is to write a new bundler which keeps a count for _each destination_ and sends batches to different destinations sooner. Also introduce a counter {{num_flips}} (find a better name!), which determines when a message batch is to be sent.
> This counter is decremented when a message to be sent has a destination that's different from the previous destination. When the counter is 0, we send the batch to the previous destination(s).
> We have a main queue, into which the senders write, and a runner thread (same as {{run()}} in TransferQueueBundler), which continually removes messages from the main queue and inserts them into queues for each destination.
> So 1 main queue and 1 queue for each destination.
> h4. Example:
> * {{num_flips}} is 2
> * A message for A is sent, added to the main queue and removed by the runner. It is queued in A's queue
> * Another message for A is sent. Also queued (A's queue: {{A A}})
> * A message to B is sent: A's {{num_flips}} is now 1. A's queue is {{A A}}, B's queue is {{B}}
> * Another message to A is sent. This resets A's {{num_flips}} to 2, B's {{num_flips}} is now 1
> * 2 messages to C are sent. This causes {{num_flips}} for A and B to be 0, so the batches to A (with 3 msgs) and B (1 msg) are also sent
> * No more messages are received, so the batch to C is also sent
> The value of {{num_flips}} should be computed as the rolling (weighted) average of the number of *adjacent messages to the same destination*. It is maintained for each destination separately (probably in the queue for that destination).
> h4. Misc
> * Should the sending of batches be delegated to a thread pool?
> * Should the senders add their messages directly to the destination queues instead of the main queue? That would result in less contention on the main queue, but it would also require 1 thread per destination queue, which creates too many threads...
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
9 years, 1 month
[JBoss JIRA] (JGRP-2171) New bundler with max_bundle_size for each destination
by Bela Ban (JIRA)
[ https://issues.jboss.org/browse/JGRP-2171?page=com.atlassian.jira.plugin.... ]
Bela Ban edited comment on JGRP-2171 at 6/9/17 1:54 AM:
--------------------------------------------------------
h2. Alternative designs
h3. Alternating bundler
This bundler sends a batch as soon as the target destination changes, e.g. for sequence {{\[A A B C C C B A A A\]}}, batches {{\[A A A\]}}, {{B}} (single message), {{\[C C C\]}}, {{B}} and {{\[A A\]}} will be sent.
Of course, {{max_bundle_size}} is still observed; if we encounter a sequence whose accumulated size exceeds it, then a batch is sent immediately.
The advantage is that messages or message batches are sent immediately (reducing latency) and that this is a simple design (similar to the one above with {{num_flips=1}}. The disadvantage is that for sequences such as {{\[A B A B A B\]}}, we'll send 6 single messages, so this degenerates into a {{NoBundler}}.
h3. Queue-based bundler
This bundler drains messages from the main queue (to which sender threads add their messages) into a remove-queue of fixed length. Then we iterate through the queue and add messages to lists keyed by the target destination and finally send a batch (or single message) for each destination.
In the above example, we'd send 3 batches {{\[A A A A A\]}}, {{\[B B\]}} and {{\[C C C\]}}. Contrast this to the 5 batches (or single messages) that we send with the alternating bundler above.
was (Author: belaban):
h2. Alternative designs
h3. Alternating bundler
This bundler sends a batch as soon as the target destination changes, e.g. for sequence {{\[A A B C C C B A A A\]}}, batches {{\[A A A\]}}, {{B}} (single message), {{\[C C C\]}}, {{B}} and {{\[A A\]}} will be sent.
Of course, {{max_bundle_size}} is still observed; if we encounter a sequence whose accumulated size exceeds it, then a batch is sent immediately.
The advantage is that messages or message batches are sent immediately (reducing latency) and that this is a simple design (similar to the one above with {{num_flips=1}}. The disadvantage is that for sequences such as {{\[A B A B A B\]}}, we'll send 6 single messages, so this degenerates into a {{NoBundler}}.
h3. Queue-based bundler
> New bundler with max_bundle_size for each destination
> -----------------------------------------------------
>
> Key: JGRP-2171
> URL: https://issues.jboss.org/browse/JGRP-2171
> Project: JGroups
> Issue Type: Feature Request
> Reporter: Bela Ban
> Assignee: Bela Ban
> Fix For: 4.0.4
>
>
> The current bundlers queue all messages and when the total size of all messages for all destinations would exceed {{max_bundle_size}}, message batches for each destination are sent.
> This negatively affects latency-sensitive applications, e.g. when we have a queue such as this: {{A B B C B B D B B}}, then the message for A has to wait until either the queue is full ({{max_bundle_size exceeded}}), or no more messages are received (and then we send the batches anyway).
> The goal is to write a new bundler which keeps a count for _each destination_ and sends batches to different destinations sooner. Also introduce a counter {{num_flips}} (find a better name!), which determines when a message batch is to be sent.
> This counter is decremented when a message to be sent has a destination that's different from the previous destination. When the counter is 0, we send the batch to the previous destination(s).
> We have a main queue, into which the senders write, and a runner thread (same as {{run()}} in TransferQueueBundler), which continually removes messages from the main queue and inserts them into queues for each destination.
> So 1 main queue and 1 queue for each destination.
> h4. Example:
> * {{num_flips}} is 2
> * A message for A is sent, added to the main queue and removed by the runner. It is queued in A's queue
> * Another message for A is sent. Also queued (A's queue: {{A A}})
> * A message to B is sent: A's {{num_flips}} is now 1. A's queue is {{A A}}, B's queue is {{B}}
> * Another message to A is sent. This resets A's {{num_flips}} to 2, B's {{num_flips}} is now 1
> * 2 messages to C are sent. This causes {{num_flips}} for A and B to be 0, so the batches to A (with 3 msgs) and B (1 msg) are also sent
> * No more messages are received, so the batch to C is also sent
> The value of {{num_flips}} should be computed as the rolling (weighted) average of the number of *adjacent messages to the same destination*. It is maintained for each destination separately (probably in the queue for that destination).
> h4. Misc
> * Should the sending of batches be delegated to a thread pool?
> * Should the senders add their messages directly to the destination queues instead of the main queue? That would result in less contention on the main queue, but it would also require 1 thread per destination queue, which creates too many threads...
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
9 years, 1 month
[JBoss JIRA] (JGRP-2171) New bundler with max_bundle_size for each destination
by Bela Ban (JIRA)
[ https://issues.jboss.org/browse/JGRP-2171?page=com.atlassian.jira.plugin.... ]
Bela Ban commented on JGRP-2171:
--------------------------------
h2. Alternative designs
h3. Alternating bundler
This bundler sends a batch as soon as the target destination changes, e.g. for sequence {{\[A A B C C C B A A A\]}}, batches {{\[A A A\]}}, {{B}} (single message), {{\[C C C\]}}, {{B}} and {{\[A A\]}} will be sent.
Of course, {{max_bundle_size}} is still observed; if we encounter a sequence whose accumulated size exceeds it, then a batch is sent immediately.
The advantage is that messages or message batches are sent immediately (reducing latency) and that this is a simple design (similar to the one above with {{num_flips=1}}. The disadvantage is that for sequences such as {{\[A B A B A B\]}}, we'll send 6 single messages, so this degenerates into a {{NoBundler}}.
h3. Queue-based bundler
> New bundler with max_bundle_size for each destination
> -----------------------------------------------------
>
> Key: JGRP-2171
> URL: https://issues.jboss.org/browse/JGRP-2171
> Project: JGroups
> Issue Type: Feature Request
> Reporter: Bela Ban
> Assignee: Bela Ban
> Fix For: 4.0.4
>
>
> The current bundlers queue all messages and when the total size of all messages for all destinations would exceed {{max_bundle_size}}, message batches for each destination are sent.
> This negatively affects latency-sensitive applications, e.g. when we have a queue such as this: {{A B B C B B D B B}}, then the message for A has to wait until either the queue is full ({{max_bundle_size exceeded}}), or no more messages are received (and then we send the batches anyway).
> The goal is to write a new bundler which keeps a count for _each destination_ and sends batches to different destinations sooner. Also introduce a counter {{num_flips}} (find a better name!), which determines when a message batch is to be sent.
> This counter is decremented when a message to be sent has a destination that's different from the previous destination. When the counter is 0, we send the batch to the previous destination(s).
> We have a main queue, into which the senders write, and a runner thread (same as {{run()}} in TransferQueueBundler), which continually removes messages from the main queue and inserts them into queues for each destination.
> So 1 main queue and 1 queue for each destination.
> h4. Example:
> * {{num_flips}} is 2
> * A message for A is sent, added to the main queue and removed by the runner. It is queued in A's queue
> * Another message for A is sent. Also queued (A's queue: {{A A}})
> * A message to B is sent: A's {{num_flips}} is now 1. A's queue is {{A A}}, B's queue is {{B}}
> * Another message to A is sent. This resets A's {{num_flips}} to 2, B's {{num_flips}} is now 1
> * 2 messages to C are sent. This causes {{num_flips}} for A and B to be 0, so the batches to A (with 3 msgs) and B (1 msg) are also sent
> * No more messages are received, so the batch to C is also sent
> The value of {{num_flips}} should be computed as the rolling (weighted) average of the number of *adjacent messages to the same destination*. It is maintained for each destination separately (probably in the queue for that destination).
> h4. Misc
> * Should the sending of batches be delegated to a thread pool?
> * Should the senders add their messages directly to the destination queues instead of the main queue? That would result in less contention on the main queue, but it would also require 1 thread per destination queue, which creates too many threads...
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
9 years, 1 month
[JBoss JIRA] (WFLY-8914) SPNEGOLoginModuleTestCase#testIdentityPropagation fails with IBM on some machines
by Martin Choma (JIRA)
Martin Choma created WFLY-8914:
----------------------------------
Summary: SPNEGOLoginModuleTestCase#testIdentityPropagation fails with IBM on some machines
Key: WFLY-8914
URL: https://issues.jboss.org/browse/WFLY-8914
Project: WildFly
Issue Type: Bug
Components: Test Suite
Affects Versions: 11.0.0.Alpha1
Reporter: Martin Choma
Priority: Minor
IBM java sends address in delegated kerberos ticket. ApacheDS includes this address into ticket and check that address with address of client (taken from connection). On some machines, these addresses doesn't match.
Those are machines when there are several virtual IPs and if node0 is set to non-first IP address, ApacheDS address check fails.
See details in https://issues.apache.org/jira/browse/DIRSERVER-2156
{code}
[31m15:14:11,302 ERROR [io.undertow.request] (default task-32) UT005023: Exception handling request to /f1eb2aa6-5139-4bce-bad8-ad9a49d3912f/protected/PropagateIdentityServlet: javax.servlet.ServletException: Propagation failed.
at org.jboss.eapqe.krbldap.eap.deployments.servlets.PropagateIdentityServlet.doGet(PropagateIdentityServlet.java:87)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:85)
at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)
at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)
at org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78)
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:131)
at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)
at io.undertow.server.handlers.DisableCacheHandler.handleRequest(DisableCacheHandler.java:33)
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at io.undertow.security.handlers.AuthenticationConstraintHandler.handleRequest(AuthenticationConstraintHandler.java:51)
at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)
at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)
at io.undertow.servlet.handlers.security.ServletSecurityConstraintHandler.handleRequest(ServletSecurityConstraintHandler.java:56)
at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)
at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)
at io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)
at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:285)
at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:264)
at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:175)
at io.undertow.server.Connectors.executeRootHandler(Connectors.java:202)
at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:792)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1153)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.lang.Thread.run(Thread.java:785)
Caused by: org.ietf.jgss.GSSException, major code: 11, minor code: 0
major string: General failure, unspecified at GSSAPI level
minor string: Error: java.lang.Exception: Error: com.ibm.security.krb5.KrbException, status code: 38
message: Incorrect net address
at com.ibm.security.jgss.i18n.I18NException.throwGSSException(I18NException.java:33)
at com.ibm.security.jgss.mech.krb5.g.a(g.java:23)
at com.ibm.security.jgss.mech.krb5.g.initSecContext(g.java:814)
at com.ibm.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:337)
at com.ibm.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:437)
at org.jboss.eapqe.krbldap.utils.krb.GSSTestClient.getName(GSSTestClient.java:100)
at org.jboss.eapqe.krbldap.eap.deployments.servlets.PropagateIdentityServlet.doGet(PropagateIdentityServlet.java:85)
... 32 more
{code}
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
9 years, 1 month
[JBoss JIRA] (ELY-559) Token-based authentication should forward authentication
by Pedro Igor (JIRA)
[ https://issues.jboss.org/browse/ELY-559?page=com.atlassian.jira.plugin.sy... ]
Pedro Igor edited comment on ELY-559 at 6/8/17 6:52 PM:
--------------------------------------------------------
Yeah, I've looked your PR which introduced this capability and the change is pretty straight forward. We also need to change {{org.wildfly.security.http.impl.BearerTokenAuthenticationMechanism}}.
Do you want me to push all those changes, [~dmlloyd] ?
was (Author: pcraveiro):
Yeah, I've looked your PR which introduced this capability and the change is pretty straight forward. We also need to change ```org.wildfly.security.http.impl.BearerTokenAuthenticationMechanism```.
Do you want me to push all those changes, [~dmlloyd] ?
> Token-based authentication should forward authentication
> --------------------------------------------------------
>
> Key: ELY-559
> URL: https://issues.jboss.org/browse/ELY-559
> Project: WildFly Elytron
> Issue Type: Enhancement
> Components: Authentication Mechanisms
> Reporter: David Lloyd
> Assignee: Pedro Igor
> Fix For: 1.1.0.Beta54
>
>
> Mechanisms that handle BearerTokenCredetials (ELY-557) should forward them as public or private credentials (ELY-473 / https://github.com/wildfly-security/wildfly-elytron/pull/434 ).
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
9 years, 1 month