[JBoss JIRA] (JGRP-2218) New buffers
by Bela Ban (JIRA)
[ https://issues.jboss.org/browse/JGRP-2218?page=com.atlassian.jira.plugin.... ]
Bela Ban updated JGRP-2218:
---------------------------
Description:
h3. Goal
Change payload in {{Message}} from byte[] arrays to a {{Buffer}} interface which can have multiple implementations.
h3. Motivation
Currently, having to pass a byte[] array to a message leads to unnecessary copying:
* When an application has a ref to an NIO (direct) {{ByteBuffer}}, the bytes in the byte buffer have to be copied into a byte[] array and then set in the message
* When the application sends around byte[] arrays, but also wants to add some additional metadata, e.g. type (1000-byte requests/responses), it needs to create a new byte[] array of (say) 1001 bytes and copy the data (1000 bytes) plus the request type (1 byte) into the new copy. Example: {{MPerf}} and {{UPerf}}
h3. Design
Instead of copying, the application creates an instance of {{Buffer}} and sets the buffer in {{Message}}. The {{Buffer}} is then passed all the way down into the transport where it is marshalled and sent. There can be a number of buffer implementations, e.g.
* {{ArrayBuffer}}: wraps a byte[] array with an offset and length
* {{NioDirectBuffer}}: wraps an NIO direct {{ByteBuffer}}
* {{NioHeapBuffer}}: wraps an NIO heap-based {{ByteBuffer}}
* {{CompositeBuffer}}: wraps multiple Buffers. E.g. type (1 byte) and data (1000 bytes) as described above
* {{IntBuffer}}: a single integer
* {{ObjectBuffer}}: has an Object and a ClassLoader (for reading), plus a Marshaller which know how to marshal the object, this allows for objects to be passed in buffers and they're only marshalled at the end (transport).
* {{PartialBuffer}}: a ref to a {{Buffer}}, with an offset and length
The {{Buffer}} interface has methods:
* {{size()}}
* {{writeTo(DataOutput)}}
* {{readFrom(DataInput)}}
* {{getInput()}}: this provides a {{DataInput}} stream for reading from the underlying buffer
and possibly also
* {{acquire()}} and
* {{release()}} (for ref-counting).
Each buffer impl has an ID and it should be possible to register new impls. A {{BufferFactory}} maintains a mapping between IDs and impl classes.
When marshalling a {{Buffer}}, the ID is written first, followed by the buffer's {{writeTo()}} method. When reading buffers, the {{BufferFactory}} is used to create instances from IDs.
h4. Fragmentation
When fragmenting a buffer, the fragments are instances of {{PartialBuffer}} which maintains an offset and length over an underlying buffer. When marshalling a {{PartialBuffer}}, only the part between offset and offset+length is written to the output stream.
h4. Reference counting
If we implement ref-counting, then buffers can be reused as soon as the ref-count is 0. For example, when sending a message, the buffer's ref-count could be incremented by the app calling {{acquire()}}. (Assuming the message is a unicast message), {{UNICAST3}} would increment the count to 2. This is needed because {{UNICAST3}} might have to retransmit the message if it was lost on the network, and meanwhile the buffer cannot be reused (changed). The app calls {{release()}} when the {{JChannel.send()}} call returns, but the buffer cannot be reused until {{UNICAST3}} calls {{release()}} as well. This will happen when an {{ACK}} for the given message has been received.
was:
h3. Goal
Change payload in {{Message}} from byte[] arrays to a {{Buffer}} interface which can have multiple implementations.
h3. Motivation
Currently, having to pass a byte[] array to a message leads to unnecessary copying:
* When an application has a ref to an NIO (direct) {{ByteBuffer}}, the bytes in the byte buffer have to be copied into a byte[] array and then set in the message
* When the application sends around byte[] arrays, but also wants to add some additional metadata, e.g. type (1000-byte requests/responses), it needs to create a new byte[] array of (say) 1001 bytes and copy the data (1000 bytes) plus the request type (1 byte) into the new copy. Example: {{MPerf}} and {{UPerf}}
h3. Design
Instead of copying, the application creates an instance of {{Buffer}} and sets the buffer in {{Message}}. The {{Buffer}} is then passed all the way down into the transport where it is marshalled and sent. There can be a number of buffer implementations, e.g.
* {{ArrayBuffer}}: wraps a byte[] array with an offset and length
* {{NioDirectBuffer}}: wraps an NIO direct {{ByteBuffer}}
* {{NioHeapBuffer}}: wraps an NIO heap-based {{ByteBuffer}}
* {{CompositeBuffer}}: wraps multiple Buffers. E.g. type (1 byte) and data (1000 bytes) as described above
* {{IntBuffer}}: a single integer
* {{ObjectBuffer}}: has an Object and a ClassLoader (for reading), plus a Marshaller which know how to marshal the object, this allows for objects to be passed in buffers and they're only marshalled at the end (transport).
* {{PartialBuffer}}: a ref to a {{Buffer}}, with an offset and length
The {{Buffer}} interface has methods {{size()}}, {{writeTo(DataOutput)}}, {{readFrom(DataInput)}}, possibly also {{acquire()}} and {{release()}} (for ref-counting).
Each buffer impl has an ID and it should be possible to register new impls. A {{BufferFactory}} maintains a mapping between IDs and impl classes.
When marshalling a {{Buffer}}, the ID is written first, followed by the buffer's {{writeTo()}} method. When reading buffers, the {{BufferFactory}} is used to create instances from IDs.
h4. Fragmentation
When fragmenting a buffer, the fragments are instances of {{PartialBuffer}} which maintains an offset and length over an underlying buffer. When marshalling a {{PartialBuffer}}, only the part between offset and offset+length is written to the output stream.
h4. Reference counting
If we implement ref-counting, then buffers can be reused as soon as the ref-count is 0. For example, when sending a message, the buffer's ref-count could be incremented by the app calling {{acquire()}}. (Assuming the message is a unicast message), {{UNICAST3}} would increment the count to 2. This is needed because {{UNICAST3}} might have to retransmit the message if it was lost on the network, and meanwhile the buffer cannot be reused (changed). The app calls {{release()}} when the {{JChannel.send()}} call returns, but the buffer cannot be reused until {{UNICAST3}} calls {{release()}} as well. This will happen when an {{ACK}} for the given message has been received.
> New buffers
> -----------
>
> Key: JGRP-2218
> URL: https://issues.jboss.org/browse/JGRP-2218
> Project: JGroups
> Issue Type: Feature Request
> Reporter: Bela Ban
> Assignee: Bela Ban
> Fix For: 5.0
>
>
> h3. Goal
> Change payload in {{Message}} from byte[] arrays to a {{Buffer}} interface which can have multiple implementations.
> h3. Motivation
> Currently, having to pass a byte[] array to a message leads to unnecessary copying:
> * When an application has a ref to an NIO (direct) {{ByteBuffer}}, the bytes in the byte buffer have to be copied into a byte[] array and then set in the message
> * When the application sends around byte[] arrays, but also wants to add some additional metadata, e.g. type (1000-byte requests/responses), it needs to create a new byte[] array of (say) 1001 bytes and copy the data (1000 bytes) plus the request type (1 byte) into the new copy. Example: {{MPerf}} and {{UPerf}}
> h3. Design
> Instead of copying, the application creates an instance of {{Buffer}} and sets the buffer in {{Message}}. The {{Buffer}} is then passed all the way down into the transport where it is marshalled and sent. There can be a number of buffer implementations, e.g.
> * {{ArrayBuffer}}: wraps a byte[] array with an offset and length
> * {{NioDirectBuffer}}: wraps an NIO direct {{ByteBuffer}}
> * {{NioHeapBuffer}}: wraps an NIO heap-based {{ByteBuffer}}
> * {{CompositeBuffer}}: wraps multiple Buffers. E.g. type (1 byte) and data (1000 bytes) as described above
> * {{IntBuffer}}: a single integer
> * {{ObjectBuffer}}: has an Object and a ClassLoader (for reading), plus a Marshaller which know how to marshal the object, this allows for objects to be passed in buffers and they're only marshalled at the end (transport).
> * {{PartialBuffer}}: a ref to a {{Buffer}}, with an offset and length
> The {{Buffer}} interface has methods:
> * {{size()}}
> * {{writeTo(DataOutput)}}
> * {{readFrom(DataInput)}}
> * {{getInput()}}: this provides a {{DataInput}} stream for reading from the underlying buffer
> and possibly also
> * {{acquire()}} and
> * {{release()}} (for ref-counting).
> Each buffer impl has an ID and it should be possible to register new impls. A {{BufferFactory}} maintains a mapping between IDs and impl classes.
> When marshalling a {{Buffer}}, the ID is written first, followed by the buffer's {{writeTo()}} method. When reading buffers, the {{BufferFactory}} is used to create instances from IDs.
> h4. Fragmentation
> When fragmenting a buffer, the fragments are instances of {{PartialBuffer}} which maintains an offset and length over an underlying buffer. When marshalling a {{PartialBuffer}}, only the part between offset and offset+length is written to the output stream.
> h4. Reference counting
> If we implement ref-counting, then buffers can be reused as soon as the ref-count is 0. For example, when sending a message, the buffer's ref-count could be incremented by the app calling {{acquire()}}. (Assuming the message is a unicast message), {{UNICAST3}} would increment the count to 2. This is needed because {{UNICAST3}} might have to retransmit the message if it was lost on the network, and meanwhile the buffer cannot be reused (changed). The app calls {{release()}} when the {{JChannel.send()}} call returns, but the buffer cannot be reused until {{UNICAST3}} calls {{release()}} as well. This will happen when an {{ACK}} for the given message has been received.
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
8 years, 9 months
[JBoss JIRA] (JGRP-2218) New buffers
by Bela Ban (JIRA)
[ https://issues.jboss.org/browse/JGRP-2218?page=com.atlassian.jira.plugin.... ]
Bela Ban updated JGRP-2218:
---------------------------
Description:
h3. Goal
Change payload in {{Message}} from byte[] arrays to a {{Buffer}} interface which can have multiple implementations.
h3. Motivation
Currently, having to pass a byte[] array to a message leads to unnecessary copying:
* When an application has a ref to an NIO (direct) {{ByteBuffer}}, the bytes in the byte buffer have to be copied into a byte[] array and then set in the message
* When the application sends around byte[] arrays, but also wants to add some additional metadata, e.g. type (1000-byte requests/responses), it needs to create a new byte[] array of (say) 1001 bytes and copy the data (1000 bytes) plus the request type (1 byte) into the new copy. Example: {{MPerf}} and {{UPerf}}
h3. Design
Instead of copying, the application creates an instance of {{Buffer}} and sets the buffer in {{Message}}. The {{Buffer}} is then passed all the way down into the transport where it is marshalled and sent. There can be a number of buffer implementations, e.g.
* {{ArrayBuffer}}: wraps a byte[] array with an offset and length
* {{NioDirectBuffer}}: wraps an NIO direct {{ByteBuffer}}
* {{NioHeapBuffer}}: wraps an NIO heap-based {{ByteBuffer}}
* {{CompositeBuffer}}: wraps multiple Buffers. E.g. type (1 byte) and data (1000 bytes) as described above
* {{IntBuffer}}: a single integer
* {{ObjectBuffer}}: has an Object and a ClassLoader (for reading), plus a Marshaller which know how to marshal the object, this allows for objects to be passed in buffers and they're only marshalled at the end (transport).
* {{PartialBuffer}}: a ref to a {{Buffer}}, with an offset and length
The {{Buffer}} interface has methods {{size()}}, {{writeTo(DataOutput)}}, {{readFrom(DataInput)}}, possibly also {{acquire()}} and {{release()}} (for ref-counting).
Each buffer impl has an ID and it should be possible to register new impls. A {{BufferFactory}} maintains a mapping between IDs and impl classes.
When marshalling a {{Buffer}}, the ID is written first, followed by the buffer's {{writeTo()}} method. When reading buffers, the {{BufferFactory}} is used to create instances from IDs.
h4. Fragmentation
When fragmenting a buffer, the fragments are instances of {{PartialBuffer}} which maintains an offset and length over an underlying buffer. When marshalling a {{PartialBuffer}}, only the part between offset and offset+length is written to the output stream.
h4. Reference counting
If we implement ref-counting, then buffers can be reused as soon as the ref-count is 0. For example, when sending a message, the buffer's ref-count could be incremented by the app calling {{acquire()}}. (Assuming the message is a unicast message), {{UNICAST3}} would increment the count to 2. This is needed because {{UNICAST3}} might have to retransmit the message if it was lost on the network, and meanwhile the buffer cannot be reused (changed). The app calls {{release()}} when the {{JChannel.send()}} call returns, but the buffer cannot be reused until {{UNICAST3}} calls {{release()}} as well. This will happen when an {{ACK}} for the given message has been received.
was:
h3. Goal
Change payload in {{Message}} from byte[] arrays to a {{Buffer}} interface which can have multiple implementations.
h3. Motivation
Currently, having to pass a byte[] array to a message leads to unnecessary copying:
* When an application has a ref to an NIO (direct) {{ByteBuffer}}, the bytes in the byte buffer have to be copied into a byte[] array and then set in the message
* When the application sends around byte[] arrays, but also wants to add some additional metadata, e.g. type (1000-byte requests/responses), it needs to create a new byte[] array of (say) 1001 bytes and copy the data (1000 bytes) plus the request type (1 byte) into the new copy. Example: {{MPerf}} and {{UPerf}}
h3. Design
Instead of copying, the application creates an instance of {{Buffer}} and sets the buffer in {{Message}}. The {{Buffer}} is then passed all the way down into the transport where it is marshalled and sent. There can be a number of buffer implementations, e.g.
* {{ArrayBuffer}}: wraps a byte[] array with an offset and length
* {{NioDirectBuffer}}: wraps an NIO direct {{ByteBuffer}}
* {{NioHeapBuffer}}: wraps an NIO heap-based {{ByteBuffer}}
* {{CompositeBuffer}}: wraps multiple Buffers. E.g. type (1 byte) and data (1000 bytes) as described above
* {{IntBuffer}}: a single integer
* {{PartialBuffer}}: a ref to a {{Buffer}}, with an offset and length
The {{Buffer}} interface has methods {{size()}}, {{writeTo(DataOutput)}}, {{readFrom(DataInput)}}, possibly also {{acquire()}} and {{release()}} (for ref-counting).
Each buffer impl has an ID and it should be possible to register new impls. A {{BufferFactory}} maintains a mapping between IDs and impl classes.
When marshalling a {{Buffer}}, the ID is written first, followed by the buffer's {{writeTo()}} method. When reading buffers, the {{BufferFactory}} is used to create instances from IDs.
h4. Fragmentation
When fragmenting a buffer, the fragments are instances of {{PartialBuffer}} which maintains an offset and length over an underlying buffer. When marshalling a {{PartialBuffer}}, only the part between offset and offset+length is written to the output stream.
h4. Reference counting
If we implement ref-counting, then buffers can be reused as soon as the ref-count is 0. For example, when sending a message, the buffer's ref-count could be incremented by the app calling {{acquire()}}. (Assuming the message is a unicast message), {{UNICAST3}} would increment the count to 2. This is needed because {{UNICAST3}} might have to retransmit the message if it was lost on the network, and meanwhile the buffer cannot be reused (changed). The app calls {{release()}} when the {{JChannel.send()}} call returns, but the buffer cannot be reused until {{UNICAST3}} calls {{release()}} as well. This will happen when an {{ACK}} for the given message has been received.
> New buffers
> -----------
>
> Key: JGRP-2218
> URL: https://issues.jboss.org/browse/JGRP-2218
> Project: JGroups
> Issue Type: Feature Request
> Reporter: Bela Ban
> Assignee: Bela Ban
> Fix For: 5.0
>
>
> h3. Goal
> Change payload in {{Message}} from byte[] arrays to a {{Buffer}} interface which can have multiple implementations.
> h3. Motivation
> Currently, having to pass a byte[] array to a message leads to unnecessary copying:
> * When an application has a ref to an NIO (direct) {{ByteBuffer}}, the bytes in the byte buffer have to be copied into a byte[] array and then set in the message
> * When the application sends around byte[] arrays, but also wants to add some additional metadata, e.g. type (1000-byte requests/responses), it needs to create a new byte[] array of (say) 1001 bytes and copy the data (1000 bytes) plus the request type (1 byte) into the new copy. Example: {{MPerf}} and {{UPerf}}
> h3. Design
> Instead of copying, the application creates an instance of {{Buffer}} and sets the buffer in {{Message}}. The {{Buffer}} is then passed all the way down into the transport where it is marshalled and sent. There can be a number of buffer implementations, e.g.
> * {{ArrayBuffer}}: wraps a byte[] array with an offset and length
> * {{NioDirectBuffer}}: wraps an NIO direct {{ByteBuffer}}
> * {{NioHeapBuffer}}: wraps an NIO heap-based {{ByteBuffer}}
> * {{CompositeBuffer}}: wraps multiple Buffers. E.g. type (1 byte) and data (1000 bytes) as described above
> * {{IntBuffer}}: a single integer
> * {{ObjectBuffer}}: has an Object and a ClassLoader (for reading), plus a Marshaller which know how to marshal the object, this allows for objects to be passed in buffers and they're only marshalled at the end (transport).
> * {{PartialBuffer}}: a ref to a {{Buffer}}, with an offset and length
> The {{Buffer}} interface has methods {{size()}}, {{writeTo(DataOutput)}}, {{readFrom(DataInput)}}, possibly also {{acquire()}} and {{release()}} (for ref-counting).
> Each buffer impl has an ID and it should be possible to register new impls. A {{BufferFactory}} maintains a mapping between IDs and impl classes.
> When marshalling a {{Buffer}}, the ID is written first, followed by the buffer's {{writeTo()}} method. When reading buffers, the {{BufferFactory}} is used to create instances from IDs.
> h4. Fragmentation
> When fragmenting a buffer, the fragments are instances of {{PartialBuffer}} which maintains an offset and length over an underlying buffer. When marshalling a {{PartialBuffer}}, only the part between offset and offset+length is written to the output stream.
> h4. Reference counting
> If we implement ref-counting, then buffers can be reused as soon as the ref-count is 0. For example, when sending a message, the buffer's ref-count could be incremented by the app calling {{acquire()}}. (Assuming the message is a unicast message), {{UNICAST3}} would increment the count to 2. This is needed because {{UNICAST3}} might have to retransmit the message if it was lost on the network, and meanwhile the buffer cannot be reused (changed). The app calls {{release()}} when the {{JChannel.send()}} call returns, but the buffer cannot be reused until {{UNICAST3}} calls {{release()}} as well. This will happen when an {{ACK}} for the given message has been received.
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
8 years, 9 months
[JBoss JIRA] (WFCORE-3257) The add-identity filesystem-realm operation in Elytron subsystem intermittently fails on NullPointerException
by Yeray Borges (JIRA)
[ https://issues.jboss.org/browse/WFCORE-3257?page=com.atlassian.jira.plugi... ]
Yeray Borges reassigned WFCORE-3257:
------------------------------------
Assignee: Yeray Borges
> The add-identity filesystem-realm operation in Elytron subsystem intermittently fails on NullPointerException
> -------------------------------------------------------------------------------------------------------------
>
> Key: WFCORE-3257
> URL: https://issues.jboss.org/browse/WFCORE-3257
> Project: WildFly Core
> Issue Type: Bug
> Components: Security
> Reporter: Josef Cacek
> Assignee: Yeray Borges
>
> Elytron filesystem-realm has an {{add-identity}} operation. I've experienced in few occasions it failed with NullPointerException.
> I'm executing set of CLI operations as batch using:
> {code}
> run-batch --file=/path/to/script.cli -v
> {code}
> When I hit the issue I see following in the log:
> {noformat}
> 17:23:19,182 ERROR [org.jboss.as.controller.management-operation] (management-handler-thread - 1) WFLYCTL0013: Operation ("add-identity") failed - address: ([
> ("subsystem" => "elytron"),
> ("filesystem-realm" => "SeccontextFsRealm")
> ]): java.lang.NullPointerException
> at org.wildfly.extension.elytron.ModifiableRealmDecorator$AddIdentityHandler.executeRuntimeStep(ModifiableRealmDecorator.java:128)
> at org.jboss.as.controller.AbstractRuntimeOnlyHandler$1.execute(AbstractRuntimeOnlyHandler.java:59)
> at org.jboss.as.controller.AbstractOperationContext.executeStep(AbstractOperationContext.java:976)
> at org.jboss.as.controller.AbstractOperationContext.processStages(AbstractOperationContext.java:722)
> at org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:446)
> at org.jboss.as.controller.OperationContextImpl.executeOperation(OperationContextImpl.java:1402)
> at org.jboss.as.controller.ModelControllerImpl.internalExecute(ModelControllerImpl.java:418)
> at org.jboss.as.controller.ModelControllerImpl.lambda$execute$1(ModelControllerImpl.java:243)
> at org.wildfly.security.auth.server.SecurityIdentity.runAs(SecurityIdentity.java:263)
> at org.wildfly.security.auth.server.SecurityIdentity.runAs(SecurityIdentity.java:229)
> at org.jboss.as.controller.ModelControllerImpl.execute(ModelControllerImpl.java:243)
> at org.jboss.as.controller.remote.ModelControllerClientOperationHandler$ExecuteRequestHandler.doExecute(ModelControllerClientOperationHandler.java:217)
> at org.jboss.as.controller.remote.ModelControllerClientOperationHandler$ExecuteRequestHandler.access$400(ModelControllerClientOperationHandler.java:137)
> at org.jboss.as.controller.remote.ModelControllerClientOperationHandler$ExecuteRequestHandler$1$1.run(ModelControllerClientOperationHandler.java:161)
> at org.jboss.as.controller.remote.ModelControllerClientOperationHandler$ExecuteRequestHandler$1$1.run(ModelControllerClientOperationHandler.java:157)
> at org.wildfly.security.auth.server.SecurityIdentity.runAs(SecurityIdentity.java:287)
> at org.wildfly.security.auth.server.SecurityIdentity.runAs(SecurityIdentity.java:244)
> at org.jboss.as.controller.AccessAuditContext.doAs(AccessAuditContext.java:254)
> at org.jboss.as.controller.AccessAuditContext.doAs(AccessAuditContext.java:225)
> at org.jboss.as.controller.remote.ModelControllerClientOperationHandler$ExecuteRequestHandler$1.execute(ModelControllerClientOperationHandler.java:157)
> at org.jboss.as.protocol.mgmt.ManagementRequestContextImpl$1.doExecute(ManagementRequestContextImpl.java:70)
> at org.jboss.as.protocol.mgmt.ManagementRequestContextImpl$AsyncTaskRunner.run(ManagementRequestContextImpl.java:160)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> at org.jboss.threads.JBossThread.run(JBossThread.java:320)
> {noformat}
> I'm attaching also the CLI file which caused the problem.
> The code which executes the script in Java seems like:
> {code:java}
> CommandContext commandCtx = ... //initialize
> commandCtx.handle("run-batch --file=/path/to/script.cli -v");
> {code}
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
8 years, 9 months
[JBoss JIRA] (JGRP-2218) New buffers
by Bela Ban (JIRA)
[ https://issues.jboss.org/browse/JGRP-2218?page=com.atlassian.jira.plugin.... ]
Bela Ban updated JGRP-2218:
---------------------------
Description:
h3. Goal
Change payload in {{Message}} from byte[] arrays to a {{Buffer}} interface which can have multiple implementations.
h3. Motivation
Currently, having to pass a byte[] array to a message leads to unnecessary copying:
* When an application has a ref to an NIO (direct) {{ByteBuffer}}, the bytes in the byte buffer have to be copied into a byte[] array and then set in the message
* When the application sends around byte[] arrays, but also wants to add some additional metadata, e.g. type (1000-byte requests/responses), it needs to create a new byte[] array of (say) 1001 bytes and copy the data (1000 bytes) plus the request type (1 byte) into the new copy. Example: {{MPerf}} and {{UPerf}}
h3. Design
Instead of copying, the application creates an instance of {{Buffer}} and sets the buffer in {{Message}}. The {{Buffer}} is then passed all the way down into the transport where it is marshalled and sent. There can be a number of buffer implementations, e.g.
* {{ArrayBuffer}}: wraps a byte[] array with an offset and length
* {{NioDirectBuffer}}: wraps an NIO direct {{ByteBuffer}}
* {{NioHeapBuffer}}: wraps an NIO heap-based {{ByteBuffer}}
* {{CompositeBuffer}}: wraps multiple Buffers. E.g. type (1 byte) and data (1000 bytes) as described above
* {{IntBuffer}}: a single integer
* {{PartialBuffer}}: a ref to a {{Buffer}}, with an offset and length
The {{Buffer}} interface has methods {{size()}}, {{writeTo(DataOutput)}}, {{readFrom(DataInput)}}, possibly also {{acquire()}} and {{release()}} (for ref-counting).
Each buffer impl has an ID and it should be possible to register new impls. A {{BufferFactory}} maintains a mapping between IDs and impl classes.
When marshalling a {{Buffer}}, the ID is written first, followed by the buffer's {{writeTo()}} method. When reading buffers, the {{BufferFactory}} is used to create instances from IDs.
h4. Fragmentation
When fragmenting a buffer, the fragments are instances of {{PartialBuffer}} which maintains an offset and length over an underlying buffer. When marshalling a {{PartialBuffer}}, only the part between offset and offset+length is written to the output stream.
h4. Reference counting
If we implement ref-counting, then buffers can be reused as soon as the ref-count is 0. For example, when sending a message, the buffer's ref-count could be incremented by the app calling {{acquire()}}. (Assuming the message is a unicast message), {{UNICAST3}} would increment the count to 2. This is needed because {{UNICAST3}} might have to retransmit the message if it was lost on the network, and meanwhile the buffer cannot be reused (changed). The app calls {{release()}} when the {{JChannel.send()}} call returns, but the buffer cannot be reused until {{UNICAST3}} calls {{release()}} as well. This will happen when an {{ACK}} for the given message has been received.
was:
h3. Goal
Change payload in {{Message}} from byte[] arrays to a {{Buffer}} interface which can have multiple implementations.
h3. Motivation
Currently, having to pass a byte[] array to a message leads to unnecessary copying:
* When an application has a ref to an NIO (direct) {{ByteBuffer}}, the bytes in the byte buffer have to be copied into a byte[] array and then set in the message
* When the application sends around byte[] arrays, but also wants to add some additional metadata, e.g. type (1000-byte requests/responses), it needs to create a new byte[] array of (say) 1001 bytes and copy the data (1000 bytes) plus the request type (1 byte) into the new copy. Example: {{MPerf}} and {{UPerf}}
h3. Design
Instead of copying, the application creates an instance of {{Buffer}} and sets the buffer in {{Message}}. The {{Buffer}} is then passed all the way down into the transport where it is marshalled and sent. There can be a number of buffer implementations, e.g.
* {{ArrayBuffer}}: wraps a byte[] array with an offset and length
* {{NioDirectBuffer}}: wraps an NIO direct {{ByteBuffer}}
* {{NioHeapBuffer}}: wraps an NIO heap-based {{ByteBuffer}}
* {{CompositeBuffer}}: wraps multiple Buffers. E.g. type (1 byte) and data (1000 bytes) as described above
* {{IntBuffer}}: a single integer
* {{PartialBuffer}}: a ref to a {{Buffer}}, with an offset and length
The {{Buffer}} interface has methods {{size()}}, {{writeTo(DataOutput)}}, {{readFrom(DataInput)}}, possibly also {{acquire()}} and {{release()}} (for ref-counting).
Each buffer impl has an ID and it should be possible to register new impls. A {{BufferFactory}} maintains a mapping between IDs and impl classes.
When marshalling a {{Buffer}}, the ID is written first, followed by the buffer's {{writeTo()}} method. When reading buffers, the {{BufferFactory}} is used to create instances from IDs.
h4. Fragmentation
When fragmenting a buffer, the fragments are instances of {{PartialBuffer}} which maintains an offset and length over an underlying buffer. When marshalling a {{PartialBuffer}}, only the part between offset and offset+length is written to the output stream.
> New buffers
> -----------
>
> Key: JGRP-2218
> URL: https://issues.jboss.org/browse/JGRP-2218
> Project: JGroups
> Issue Type: Feature Request
> Reporter: Bela Ban
> Assignee: Bela Ban
> Fix For: 5.0
>
>
> h3. Goal
> Change payload in {{Message}} from byte[] arrays to a {{Buffer}} interface which can have multiple implementations.
> h3. Motivation
> Currently, having to pass a byte[] array to a message leads to unnecessary copying:
> * When an application has a ref to an NIO (direct) {{ByteBuffer}}, the bytes in the byte buffer have to be copied into a byte[] array and then set in the message
> * When the application sends around byte[] arrays, but also wants to add some additional metadata, e.g. type (1000-byte requests/responses), it needs to create a new byte[] array of (say) 1001 bytes and copy the data (1000 bytes) plus the request type (1 byte) into the new copy. Example: {{MPerf}} and {{UPerf}}
> h3. Design
> Instead of copying, the application creates an instance of {{Buffer}} and sets the buffer in {{Message}}. The {{Buffer}} is then passed all the way down into the transport where it is marshalled and sent. There can be a number of buffer implementations, e.g.
> * {{ArrayBuffer}}: wraps a byte[] array with an offset and length
> * {{NioDirectBuffer}}: wraps an NIO direct {{ByteBuffer}}
> * {{NioHeapBuffer}}: wraps an NIO heap-based {{ByteBuffer}}
> * {{CompositeBuffer}}: wraps multiple Buffers. E.g. type (1 byte) and data (1000 bytes) as described above
> * {{IntBuffer}}: a single integer
> * {{PartialBuffer}}: a ref to a {{Buffer}}, with an offset and length
> The {{Buffer}} interface has methods {{size()}}, {{writeTo(DataOutput)}}, {{readFrom(DataInput)}}, possibly also {{acquire()}} and {{release()}} (for ref-counting).
> Each buffer impl has an ID and it should be possible to register new impls. A {{BufferFactory}} maintains a mapping between IDs and impl classes.
> When marshalling a {{Buffer}}, the ID is written first, followed by the buffer's {{writeTo()}} method. When reading buffers, the {{BufferFactory}} is used to create instances from IDs.
> h4. Fragmentation
> When fragmenting a buffer, the fragments are instances of {{PartialBuffer}} which maintains an offset and length over an underlying buffer. When marshalling a {{PartialBuffer}}, only the part between offset and offset+length is written to the output stream.
> h4. Reference counting
> If we implement ref-counting, then buffers can be reused as soon as the ref-count is 0. For example, when sending a message, the buffer's ref-count could be incremented by the app calling {{acquire()}}. (Assuming the message is a unicast message), {{UNICAST3}} would increment the count to 2. This is needed because {{UNICAST3}} might have to retransmit the message if it was lost on the network, and meanwhile the buffer cannot be reused (changed). The app calls {{release()}} when the {{JChannel.send()}} call returns, but the buffer cannot be reused until {{UNICAST3}} calls {{release()}} as well. This will happen when an {{ACK}} for the given message has been received.
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
8 years, 9 months
[JBoss JIRA] (JGRP-2218) New buffers
by Bela Ban (JIRA)
[ https://issues.jboss.org/browse/JGRP-2218?page=com.atlassian.jira.plugin.... ]
Bela Ban updated JGRP-2218:
---------------------------
Description:
h3. Goal
Change payload in {{Message}} from byte[] arrays to a {{Buffer}} interface which can have multiple implementations.
h3. Motivation
Currently, having to pass a byte[] array to a message leads to unnecessary copying:
* When an application has a ref to an NIO (direct) {{ByteBuffer}}, the bytes in the byte buffer have to be copied into a byte[] array and then set in the message
* When the application sends around byte[] arrays, but also wants to add some additional metadata, e.g. type (1000-byte requests/responses), it needs to create a new byte[] array of (say) 1001 bytes and copy the data (1000 bytes) plus the request type (1 byte) into the new copy. Example: {{MPerf}} and {{UPerf}}
h3. Design
Instead of copying, the application creates an instance of {{Buffer}} and sets the buffer in {{Message}}. The {{Buffer}} is then passed all the way down into the transport where it is marshalled and sent. There can be a number of buffer implementations, e.g.
* {{ArrayBuffer}}: wraps a byte[] array with an offset and length
* {{NioDirectBuffer}}: wraps an NIO direct {{ByteBuffer}}
* {{NioHeapBuffer}}: wraps an NIO heap-based {{ByteBuffer}}
* {{CompositeBuffer}}: wraps multiple Buffers. E.g. type (1 byte) and data (1000 bytes) as described above
* {{IntBuffer}}: a single integer
* {{PartialBuffer}}: a ref to a {{Buffer}}, with an offset and length
The {{Buffer}} interface has methods {{size()}}, {{writeTo(DataOutput)}}, {{readFrom(DataInput)}}, possibly also {{acquire()}} and {{release()}} (for ref-counting).
Each buffer impl has an ID and it should be possible to register new impls. A {{BufferFactory}} maintains a mapping between IDs and impl classes.
When marshalling a {{Buffer}}, the ID is written first, followed by the buffer's {{writeTo()}} method. When reading buffers, the {{BufferFactory}} is used to create instances from IDs.
h4. Fragmentation
When fragmenting a buffer, the fragments are instances of {{PartialBuffer}} which maintains an offset and length over an underlying buffer. When marshalling a {{PartialBuffer}}, only the part between offset and offset+length is written to the output stream.
was:
h3. Goal
Change payload in {{Message}} from byte[] arrays to a {{Buffer}} interface which can have multiple implementations.
h3. Motivation
Currently, having to pass a byte[] array to a message leads to unnecessary copying:
* When an application has a ref to an NIO (direct) {{ByteBuffer}}, the bytes in the byte buffer have to be copied into a byte[] array and then set in the message
* When the application sends around byte[] arrays, but also wants to add some additional metadata, e.g. type (1000-byte requests/responses), it needs to create a new byte[] array of (say) 1001 bytes and copy the data (1000 bytes) plus the request type (1 byte) into the new copy. Example: {{MPerf}} and {{UPerf}}
h3. Design
Instead of copying, the application creates an instance of {{Buffer}} and sets the buffer in {{Message}}. The {{Buffer}} is then passed all the way down into the transport where it is marshalled and sent. There can be a number of buffer implementations, e.g.
* {{ArrayBuffer}}: wraps a byte[] array with an offset and length
* {{NioDirectBuffer}}: wraps an NIO direct {{ByteBuffer}}
* {{NioHeapBuffer}}: wraps an NIO heap-based {{ByteBuffer}}
* {{CompositeBuffer}}: wraps multiple Buffers. E.g. type (1 byte) and data (1000 bytes) as described above
* {{IntBuffer}}: a single integer
* {{PartialBuffer}}: a ref to a {{Buffer}}, with an offset and length
The {{Buffer}} interface has methods {{size()}}, {{writeTo(DataOutput)}}, {{readFrom(DataInput)}}, possibly also {{acquire()}} and {{release()}} (for ref-counting).
Each buffer impl has an ID and it should be possible to register new impls. A {{BufferFactory}} maintains a mapping between IDs and impl classes.
When marshalling a {{Buffer}}, the ID is written first, followed by the buffer's {{writeTo()}} method. When reading buffers, the {{BufferFactory}} is used to create instances from IDs.
> New buffers
> -----------
>
> Key: JGRP-2218
> URL: https://issues.jboss.org/browse/JGRP-2218
> Project: JGroups
> Issue Type: Feature Request
> Reporter: Bela Ban
> Assignee: Bela Ban
> Fix For: 5.0
>
>
> h3. Goal
> Change payload in {{Message}} from byte[] arrays to a {{Buffer}} interface which can have multiple implementations.
> h3. Motivation
> Currently, having to pass a byte[] array to a message leads to unnecessary copying:
> * When an application has a ref to an NIO (direct) {{ByteBuffer}}, the bytes in the byte buffer have to be copied into a byte[] array and then set in the message
> * When the application sends around byte[] arrays, but also wants to add some additional metadata, e.g. type (1000-byte requests/responses), it needs to create a new byte[] array of (say) 1001 bytes and copy the data (1000 bytes) plus the request type (1 byte) into the new copy. Example: {{MPerf}} and {{UPerf}}
> h3. Design
> Instead of copying, the application creates an instance of {{Buffer}} and sets the buffer in {{Message}}. The {{Buffer}} is then passed all the way down into the transport where it is marshalled and sent. There can be a number of buffer implementations, e.g.
> * {{ArrayBuffer}}: wraps a byte[] array with an offset and length
> * {{NioDirectBuffer}}: wraps an NIO direct {{ByteBuffer}}
> * {{NioHeapBuffer}}: wraps an NIO heap-based {{ByteBuffer}}
> * {{CompositeBuffer}}: wraps multiple Buffers. E.g. type (1 byte) and data (1000 bytes) as described above
> * {{IntBuffer}}: a single integer
> * {{PartialBuffer}}: a ref to a {{Buffer}}, with an offset and length
> The {{Buffer}} interface has methods {{size()}}, {{writeTo(DataOutput)}}, {{readFrom(DataInput)}}, possibly also {{acquire()}} and {{release()}} (for ref-counting).
> Each buffer impl has an ID and it should be possible to register new impls. A {{BufferFactory}} maintains a mapping between IDs and impl classes.
> When marshalling a {{Buffer}}, the ID is written first, followed by the buffer's {{writeTo()}} method. When reading buffers, the {{BufferFactory}} is used to create instances from IDs.
> h4. Fragmentation
> When fragmenting a buffer, the fragments are instances of {{PartialBuffer}} which maintains an offset and length over an underlying buffer. When marshalling a {{PartialBuffer}}, only the part between offset and offset+length is written to the output stream.
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
8 years, 9 months