[JBoss JIRA] (WFWIP-205) tx recovery intermittently fails after jvm crash
by Martin Simka (Jira)
Martin Simka created WFWIP-205:
----------------------------------
Summary: tx recovery intermittently fails after jvm crash
Key: WFWIP-205
URL: https://issues.jboss.org/browse/WFWIP-205
Project: WildFly WIP
Issue Type: Bug
Components: OpenShift
Environment: image:
{noformat}
docker-registry.engineering.redhat.com/ochaloup/wildfly18-snapshot:190909...
{noformat}
operator:
{noformat}
docker-registry.engineering.redhat.com/jbossqe-eap/wildfly-operator:EAP7-...
{noformat}
operator built from https://github.com/ochaloup/wildfly-operator/tree/issue70-statefulset-hea..., head 8925e7f64b6fc02b4694da63d93c0a8ce03a566d)
Reporter: Martin Simka
Assignee: Ondrej Chaloupka
Attachments: tx-client-0.log, tx-server-0.log, tx-server-1.log, wildfly-operator-668fd79fb5-8chs8.log
While testing tx recovery in OpenShift I see that recovery after JVM crash intermittently fails
Scenario:
*ejb client* (app tx-client, pod tx-client-0):
* EJB business method
** lookup remote EJB
** enlist XA resource 1 to transaction
** enlist XA resource 2 to transaction
** call remote EJB
*ejb server* (app tx-server, pod tx-server-0):
* EJB business method
** enlist XA resource 1 to transaction
** enlist XA resource 2 to transaction
ejb server XA resource 2 crashes JVM in commit method phase.
Test waits until crashed pod is restarted, then forces periodic recovery twice and then checks that transaction log store is empty. But it is not empty.
Attached are logs from client and server pods.
It seems that it can be partially mitigated by clear openshift namespace before test ({{oc delete all --all}}). But it makes it just less frequent.
--
This message was sent by Atlassian Jira
(v7.13.5#713005)
6 years, 10 months
[JBoss JIRA] (WFWIP-205) tx recovery intermittently fails after jvm crash
by Martin Simka (Jira)
[ https://issues.jboss.org/browse/WFWIP-205?page=com.atlassian.jira.plugin.... ]
Martin Simka updated WFWIP-205:
-------------------------------
Description:
While testing tx recovery in OpenShift I see that recovery after JVM crash intermittently fails
Scenario:
*ejb client* (app tx-client, pod tx-client-0):
* EJB business method
** lookup remote EJB
** enlist XA resource 1 to transaction
** enlist XA resource 2 to transaction
** call remote EJB
*ejb server* (app tx-server, pod tx-server-0):
* EJB business method
** enlist XA resource 1 to transaction
** enlist XA resource 2 to transaction
ejb server XA resource 2 crashes JVM in commit method phase.
Test waits until crashed pod is restarted, then forces periodic recovery twice and then checks that transaction log store is empty. But it is not empty.
Attached are logs from client and server pods.
It seems that it can be partially mitigated by clearing openshift namespace before test ({{oc delete all --all}}). But it makes it just less frequent.
was:
While testing tx recovery in OpenShift I see that recovery after JVM crash intermittently fails
Scenario:
*ejb client* (app tx-client, pod tx-client-0):
* EJB business method
** lookup remote EJB
** enlist XA resource 1 to transaction
** enlist XA resource 2 to transaction
** call remote EJB
*ejb server* (app tx-server, pod tx-server-0):
* EJB business method
** enlist XA resource 1 to transaction
** enlist XA resource 2 to transaction
ejb server XA resource 2 crashes JVM in commit method phase.
Test waits until crashed pod is restarted, then forces periodic recovery twice and then checks that transaction log store is empty. But it is not empty.
Attached are logs from client and server pods.
It seems that it can be partially mitigated by clear openshift namespace before test ({{oc delete all --all}}). But it makes it just less frequent.
> tx recovery intermittently fails after jvm crash
> ------------------------------------------------
>
> Key: WFWIP-205
> URL: https://issues.jboss.org/browse/WFWIP-205
> Project: WildFly WIP
> Issue Type: Bug
> Components: OpenShift
> Environment: image:
> {noformat}
> docker-registry.engineering.redhat.com/ochaloup/wildfly18-snapshot:190909...
> {noformat}
> operator:
> {noformat}
> docker-registry.engineering.redhat.com/jbossqe-eap/wildfly-operator:EAP7-...
> {noformat}
> operator built from https://github.com/ochaloup/wildfly-operator/tree/issue70-statefulset-hea..., head 8925e7f64b6fc02b4694da63d93c0a8ce03a566d)
> Reporter: Martin Simka
> Assignee: Ondrej Chaloupka
> Priority: Blocker
> Attachments: tx-client-0.log, tx-server-0.log, tx-server-1.log, wildfly-operator-668fd79fb5-8chs8.log
>
>
> While testing tx recovery in OpenShift I see that recovery after JVM crash intermittently fails
> Scenario:
> *ejb client* (app tx-client, pod tx-client-0):
> * EJB business method
> ** lookup remote EJB
> ** enlist XA resource 1 to transaction
> ** enlist XA resource 2 to transaction
> ** call remote EJB
> *ejb server* (app tx-server, pod tx-server-0):
> * EJB business method
> ** enlist XA resource 1 to transaction
> ** enlist XA resource 2 to transaction
> ejb server XA resource 2 crashes JVM in commit method phase.
> Test waits until crashed pod is restarted, then forces periodic recovery twice and then checks that transaction log store is empty. But it is not empty.
> Attached are logs from client and server pods.
> It seems that it can be partially mitigated by clearing openshift namespace before test ({{oc delete all --all}}). But it makes it just less frequent.
--
This message was sent by Atlassian Jira
(v7.13.5#713005)
6 years, 10 months
[JBoss JIRA] (WFWIP-198) EAP Operator long-term strategy
by Martin Choma (Jira)
[ https://issues.jboss.org/browse/WFWIP-198?page=com.atlassian.jira.plugin.... ]
Martin Choma edited comment on WFWIP-198 at 9/17/19 8:39 AM:
-------------------------------------------------------------
[~jmesnil] I have in mind customers which need transactions, but need configure Openshift in way Operator does not support. When critical feature as transaction will be available only throught Operator they will be locked-in. As customers are creative, I expect customers will come with requests for RFEs to expose internal kubernetes object parameters. And we will actually end in state exposing most of internal objects through Operator anyway.
[~ochaloup] What about these customers who can't/don't want to use operator but do need transactions?
Jeff could you formulate clear EAP operator contract which will be communicated to customers. So far seems to me it is something like this:
- Operator does not provides all capabilities that working directly with Kubernetes objects do. And never will.
- Operator provides opinionated way how to configure Kubernetes environment for runing EAP. Not all ways.
- You cant touch internal Kubernetes objects created by Operator to change setting. Operator will overwrite your changes.
- If you need something what Operator does not provide you can raise RFE for operator or turn off operator and be on your own.
- Transactions are supported only when using Operator
was (Author: mchoma):
[~jmesnil] I have in mind customers which need transactions, but need configure Openshift in way Operator does not support. When critical feature as transaction will be available only throught Operator they will be locked-in. As customers are creative, I expect customers will come with requests for RFEs to expose internal kubernetes object parameters. And we will actually end in state exposing most of internal objects through Operator anyway.
[~ochaloup] What about these customers who can't/don't want to use operator but do need transactions?
Could you formulate clear EAP operator contract which will be communicated to customers. So far seems to me it is something like this:
- Operator does not provides all capabilities that working directly with Kubernetes objects do. And never will.
- Operator provides opinionated way how to configure Kubernetes environment for runing EAP. Not all ways.
- You cant touch internal Kubernetes objects created by Operator to change setting. Operator will overwrite your changes.
- If you need something what Operator does not provide you can raise RFE for operator or turn off operator and be on your own.
- Transactions are supported only when using Operator
> EAP Operator long-term strategy
> -------------------------------
>
> Key: WFWIP-198
> URL: https://issues.jboss.org/browse/WFWIP-198
> Project: WildFly WIP
> Issue Type: Bug
> Components: OpenShift
> Reporter: Martin Choma
> Assignee: Jeff Mesnil
> Priority: Blocker
>
> Facts:
> # Operator creates and manges internaly several kubernetes objects; StatefulSet, Pod, Service, Routes. By rough estimation these kubernetes objects expose approximately 50 options[2]. Operator now expose approximately 10 options [1].
> # EAP7-1192 is currently designed in way operator is integral part of solution. You can not turn off operator to EAP7-1192 properly work.
> # User cant touch internal Kubernetes objects. Operator manages them and can overwrite changes anytime.
> So given these facts question is what is long-term strategy with Operator?
> # Operator will provide all options of internal objects. Everything I am able to configure directly with Kubernetes objects I will be able to configure with Operator
> # Operator will expose just most useful options. If user wants to make something special he has to "turn off" operator and to be on his own. (Loosing transaction support provided by EAP7-1192)
> # Operator will expose just most useful options. If user wants to make something special he can and Operator won't overwritten his changes. (I dont know if it is feasible?)
> [1] https://github.com/wildfly/wildfly-operator/blob/master/doc/apis.adoc#wil...
> [2]
> https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.15/#pods...
> https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.15/#stat...
> https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.15/#serv...
> https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.15/#ingr...
--
This message was sent by Atlassian Jira
(v7.13.5#713005)
6 years, 10 months
[JBoss JIRA] (WFWIP-198) EAP Operator long-term strategy
by Martin Choma (Jira)
[ https://issues.jboss.org/browse/WFWIP-198?page=com.atlassian.jira.plugin.... ]
Martin Choma commented on WFWIP-198:
------------------------------------
[~jmesnil] I have in mind customers which need transactions, but need configure Openshift in way Operator does not support. When critical feature as transaction will be available only throught Operator they will be locked-in. As customers are creative, I expect customers will come with requests for RFEs to expose internal kubernetes object parameters. And we will actually end in state exposing most of internal objects through Operator anyway.
[~ochaloup] What about these customers who can't/don't want to use operator but do need transactions?
Could you formulate clear EAP operator contract which will be communicated to customers. So far seems to me it is something like this:
- Operator does not provides all capabilities that working directly with Kubernetes objects do. And never will.
- Operator provides opinionated way how to configure Kubernetes environment for runing EAP. Not all ways.
- You cant touch internal Kubernetes objects created by Operator to change setting. Operator will overwrite your changes.
- If you need something what Operator does not provide you can raise RFE for operator or turn off operator and be on your own.
- Transactions are supported only when using Operator
> EAP Operator long-term strategy
> -------------------------------
>
> Key: WFWIP-198
> URL: https://issues.jboss.org/browse/WFWIP-198
> Project: WildFly WIP
> Issue Type: Bug
> Components: OpenShift
> Reporter: Martin Choma
> Assignee: Jeff Mesnil
> Priority: Blocker
>
> Facts:
> # Operator creates and manges internaly several kubernetes objects; StatefulSet, Pod, Service, Routes. By rough estimation these kubernetes objects expose approximately 50 options[2]. Operator now expose approximately 10 options [1].
> # EAP7-1192 is currently designed in way operator is integral part of solution. You can not turn off operator to EAP7-1192 properly work.
> # User cant touch internal Kubernetes objects. Operator manages them and can overwrite changes anytime.
> So given these facts question is what is long-term strategy with Operator?
> # Operator will provide all options of internal objects. Everything I am able to configure directly with Kubernetes objects I will be able to configure with Operator
> # Operator will expose just most useful options. If user wants to make something special he has to "turn off" operator and to be on his own. (Loosing transaction support provided by EAP7-1192)
> # Operator will expose just most useful options. If user wants to make something special he can and Operator won't overwritten his changes. (I dont know if it is feasible?)
> [1] https://github.com/wildfly/wildfly-operator/blob/master/doc/apis.adoc#wil...
> [2]
> https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.15/#pods...
> https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.15/#stat...
> https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.15/#serv...
> https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.15/#ingr...
--
This message was sent by Atlassian Jira
(v7.13.5#713005)
6 years, 10 months
[JBoss JIRA] (JGRP-987) Make many classes immutable
by Bela Ban (Jira)
[ https://issues.jboss.org/browse/JGRP-987?page=com.atlassian.jira.plugin.s... ]
Bela Ban updated JGRP-987:
--------------------------
Description:
Currently, a lot of classes could be immutable if it wasn't for the readFrom() method to initialize itself from an input stream.
However, if we had a factory, which reads and constructs immutable instances of classes, then we could turn a lot of classes into immutable classes, for example all headers should be immutable.
The code for marshalling and unmarshalling could be generated, e.g. based on an XML description of the data layout of a class, or via annotations.
was:
Currently, a lot of classes could be immutable if it wasn't for the readFrom() method to initialize itself from an input stream.
However, if we had a factory, which reads and constructs immutable instances of classes, then we could turn a lot of classes into immutable classes, for example all headers should be immutable.
The code for marshalling and unmarshalling could be generated, e.g. based on an XML description of the data layout of a class.
> Make many classes immutable
> ---------------------------
>
> Key: JGRP-987
> URL: https://issues.jboss.org/browse/JGRP-987
> Project: JGroups
> Issue Type: Feature Request
> Reporter: Bela Ban
> Assignee: Bela Ban
> Priority: Major
> Fix For: Future
>
>
> Currently, a lot of classes could be immutable if it wasn't for the readFrom() method to initialize itself from an input stream.
> However, if we had a factory, which reads and constructs immutable instances of classes, then we could turn a lot of classes into immutable classes, for example all headers should be immutable.
> The code for marshalling and unmarshalling could be generated, e.g. based on an XML description of the data layout of a class, or via annotations.
--
This message was sent by Atlassian Jira
(v7.13.5#713005)
6 years, 10 months
[JBoss JIRA] (JGRP-653) Streaming API for large messages
by Bela Ban (Jira)
[ https://issues.jboss.org/browse/JGRP-653?page=com.atlassian.jira.plugin.s... ]
Bela Ban resolved JGRP-653.
---------------------------
Resolution: Won't Do
Won't do this, but its graveyard will be 5.0 instead of Future... unless someone convinces me that the code (in userland) above won't suffice...
The fact this has been dormant for 12 years shows that this feature is not required.
> Streaming API for large messages
> --------------------------------
>
> Key: JGRP-653
> URL: https://issues.jboss.org/browse/JGRP-653
> Project: JGroups
> Issue Type: Feature Request
> Reporter: Bela Ban
> Assignee: Bela Ban
> Priority: Major
> Fix For: 5.0
>
> Attachments: JGroupsInputStream.java, JGroupsOutputStream.java, StreamTest.java
>
>
> For large messages, to load the entire payload into memory might be bad because the payload might be bigger than the max memory available. It would be useful to have an API which allows for use of input and output streams, so that large payloads can be read iteratively by a user and streamed out to the cluster via the underlying channel breaking the data in the input stream into chunks, which are fed into the input stream on the receivers side.
> Issues: we have to have 1 input stream per sender on the receiver side, because a stream is always defined between 2 parties (sender, receiver). Maybe something like NIO, where we register interest in a stream, are notified of new streams ('accept()') and get notified when data on any of the stream is available, would be beneficial.
> Demo is attached
--
This message was sent by Atlassian Jira
(v7.13.5#713005)
6 years, 10 months
[JBoss JIRA] (JGRP-653) Streaming API for large messages
by Bela Ban (Jira)
[ https://issues.jboss.org/browse/JGRP-653?page=com.atlassian.jira.plugin.s... ]
Bela Ban updated JGRP-653:
--------------------------
Fix Version/s: 5.0
(was: Future)
> Streaming API for large messages
> --------------------------------
>
> Key: JGRP-653
> URL: https://issues.jboss.org/browse/JGRP-653
> Project: JGroups
> Issue Type: Feature Request
> Reporter: Bela Ban
> Assignee: Bela Ban
> Priority: Major
> Fix For: 5.0
>
> Attachments: JGroupsInputStream.java, JGroupsOutputStream.java, StreamTest.java
>
>
> For large messages, to load the entire payload into memory might be bad because the payload might be bigger than the max memory available. It would be useful to have an API which allows for use of input and output streams, so that large payloads can be read iteratively by a user and streamed out to the cluster via the underlying channel breaking the data in the input stream into chunks, which are fed into the input stream on the receivers side.
> Issues: we have to have 1 input stream per sender on the receiver side, because a stream is always defined between 2 parties (sender, receiver). Maybe something like NIO, where we register interest in a stream, are notified of new streams ('accept()') and get notified when data on any of the stream is available, would be beneficial.
> Demo is attached
--
This message was sent by Atlassian Jira
(v7.13.5#713005)
6 years, 10 months
[JBoss JIRA] (JGRP-653) Streaming API for large messages
by Bela Ban (Jira)
[ https://issues.jboss.org/browse/JGRP-653?page=com.atlassian.jira.plugin.s... ]
Bela Ban edited comment on JGRP-653 at 9/17/19 8:11 AM:
--------------------------------------------------------
I'm wondering if a streaming API is needed after all: when a user needs to transfer a large object (say a DVD ISO image of 4 GB size), then he would do the following:
- Open an input stream to the DVD's file
- Get a JGroupsOutputStream
- Allocate a small buffer (say 8K)
- While(end of file not reached on input stream)
- Read 8K from input stream into buffer
- Write buffer to output stream
On the receiver's side:
- Get JGroupsInputStream
- Create an output stream (file)
- Allocate a small buffer (say 8K)
- While(not end of file)
- Read data from input stream into buffer
- Write data to output stream (file)
- Close output stream
If we didn't have JGroups\{Input,Output\}Streams, we could still achieve the same:
- Open an input stream to the DVD's file
- Allocate a small buffer (say 8K)
- Send CREATE(filename) message
- While(end of file not reached on input stream)
- Read 8K from input stream into buffer
- Send 8K WRITE message (include filename in message)
- Send CLOSE(filename) message
On the receiver's side:
- On reception of CREATE(filename) message:
- Create an output stream (file) for filename
- On reception of WRITE message:
- Append the received byte buffer to the output stream (file)
- On reception of CLOSE message (filename)
- Close output stream for filename
was (Author: belaban):
I'm wondering if a streaming API is needed after all: when a user needs to transfer a large object (say a DVD ISO image of 4 GB size), then he would do the following:
- Open an input stream to the DVD's file
- Get a JGroupsOutputStream
- Allocate a small buffer (say 8K)
- While(end of file not reached on input stream)
- Read 8K from input stream into buffer
- Write buffer to output stream
On the receiver's side:
- Get JGroupsInputStream
- Create an output stream (file)
- Allocate a small buffer (say 8K)
- While(not end of file)
- Read data from input stream into buffer
- Write data to output stream (file)
- Close output stream
If we didn't have JGroups{Input,Output}Streams, we could still achieve the same:
- Open an input stream to the DVD's file
- Allocate a small buffer (say 8K)
- Send CREATE(filename) message
- While(end of file not reached on input stream)
- Read 8K from input stream into buffer
- Send 8K WRITE message (include filename in message)
- Send CLOSE(filename) message
On the receiver's side:
- On reception of CREATE(filename) message:
- Create an output stream (file) for filename
- On reception of WRITE message:
- Append the received byte buffer to the output stream (file)
- On reception of CLOSE message (filename)
- Close output stream for filename
> Streaming API for large messages
> --------------------------------
>
> Key: JGRP-653
> URL: https://issues.jboss.org/browse/JGRP-653
> Project: JGroups
> Issue Type: Feature Request
> Reporter: Bela Ban
> Assignee: Bela Ban
> Priority: Major
> Fix For: Future
>
> Attachments: JGroupsInputStream.java, JGroupsOutputStream.java, StreamTest.java
>
>
> For large messages, to load the entire payload into memory might be bad because the payload might be bigger than the max memory available. It would be useful to have an API which allows for use of input and output streams, so that large payloads can be read iteratively by a user and streamed out to the cluster via the underlying channel breaking the data in the input stream into chunks, which are fed into the input stream on the receivers side.
> Issues: we have to have 1 input stream per sender on the receiver side, because a stream is always defined between 2 parties (sender, receiver). Maybe something like NIO, where we register interest in a stream, are notified of new streams ('accept()') and get notified when data on any of the stream is available, would be beneficial.
> Demo is attached
--
This message was sent by Atlassian Jira
(v7.13.5#713005)
6 years, 10 months