[JBoss JIRA] (WFLY-11294) Artemis with JDBC store on slow or high latency database crashed on: Critical IO Error. Failed to process JDBC Record statements: java.sql.BatchUpdateException: IO Error: Socket read timed out
by Miroslav Novak (Jira)
Miroslav Novak created WFLY-11294:
-------------------------------------
Summary: Artemis with JDBC store on slow or high latency database crashed on: Critical IO Error. Failed to process JDBC Record statements: java.sql.BatchUpdateException: IO Error: Socket read timed out
Key: WFLY-11294
URL: https://issues.jboss.org/browse/WFLY-11294
Project: WildFly
Issue Type: Bug
Components: JMS
Affects Versions: 12.0.0.Final, 13.0.0.Final
Reporter: Miroslav Novak
Assignee: Martyn Taylor
Artemis configured with JDBC store is not robust if database is slow/under high load and when there is higher latency to database. Artemis has tendency to shutdown itself on "Critical IO Error" under really low load. It is visible when EAP QE moved Jenkins to Central CI where is higher latency to used Oracle12cR2 database. In following test scenario:
* Start 2 WF/EAP server in cluster. There is queue InQueue and OutQueue deployed on both of the servers.
* Send 10 000 small messages to InQueue to server 1
* Then deploy MDB on server 2. MDB consumes messages from InQueue and resends to OutQueue
* Wait until MDB processes all messages and check that 10 000 messages is in OutQueue
Result:
Sometimes happens that Artemis in server 1 crashes due to following exception:
{code}
20:13:45,641 WARN [org.apache.activemq.artemis.jdbc.store.journal.JDBCJournalImpl] (Thread-17 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$3@4de066d8)) IO Error: Socket read
timed out: java.sql.BatchUpdateException: IO Error: Socket read timed out
at oracle.jdbc.driver.OraclePreparedStatement.executeLargeBatch(OraclePreparedStatement.java:10032)
at oracle.jdbc.driver.T4CPreparedStatement.executeLargeBatch(T4CPreparedStatement.java:1364)
at oracle.jdbc.driver.OraclePreparedStatement.executeBatch(OraclePreparedStatement.java:9839)
at oracle.jdbc.driver.OracleStatementWrapper.executeBatch(OracleStatementWrapper.java:234)
at org.jboss.jca.adapters.jdbc.WrappedStatement.executeBatch(WrappedStatement.java:1180)
at org.apache.activemq.artemis.jdbc.store.journal.JDBCJournalImpl.sync(JDBCJournalImpl.java:228) [artemis-jdbc-store-1.5.5.jbossorg-012.jar:1.5.5.jbossorg-012]
at org.apache.activemq.artemis.jdbc.store.journal.JDBCJournalImpl$JDBCJournalSync.run(JDBCJournalImpl.java:991) [artemis-jdbc-store-1.5.5.jbossorg-012.jar:1.5.5.jbossorg-012]
at org.apache.activemq.artemis.core.server.ActiveMQScheduledComponent$2.run(ActiveMQScheduledComponent.java:284) [artemis-commons-1.5.5.jbossorg-012.jar:1.5.5.jbossorg-012]
at org.apache.activemq.artemis.utils.OrderedExecutorFactory$OrderedExecutor$ExecutorTask.run(OrderedExecutorFactory.java:122) [artemis-commons-1.5.5.jbossorg-012.jar:1.5.5.jbossorg-012]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [rt.jar:1.8.0_171]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [rt.jar:1.8.0_171]
at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_171]
20:13:45,652 WARN [org.apache.activemq.artemis.core.server] (Thread-17 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$3@4de066d8)) AMQ222010: Critical IO Error, shutting down the server. file=NULL, message=Critical IO Error. Failed to process JDBC Record statements: java.sql.BatchUpdateException: IO Error: Socket read timed out
at oracle.jdbc.driver.OraclePreparedStatement.executeLargeBatch(OraclePreparedStatement.java:10032)
at oracle.jdbc.driver.T4CPreparedStatement.executeLargeBatch(T4CPreparedStatement.java:1364)
at oracle.jdbc.driver.OraclePreparedStatement.executeBatch(OraclePreparedStatement.java:9839)
at oracle.jdbc.driver.OracleStatementWrapper.executeBatch(OracleStatementWrapper.java:234)
at org.jboss.jca.adapters.jdbc.WrappedStatement.executeBatch(WrappedStatement.java:1180)
at org.apache.activemq.artemis.jdbc.store.journal.JDBCJournalImpl.sync(JDBCJournalImpl.java:228) [artemis-jdbc-store-1.5.5.jbossorg-012.jar:1.5.5.jbossorg-012]
at org.apache.activemq.artemis.jdbc.store.journal.JDBCJournalImpl$JDBCJournalSync.run(JDBCJournalImpl.java:991) [artemis-jdbc-store-1.5.5.jbossorg-012.jar:1.5.5.jbossorg-012]
at org.apache.activemq.artemis.core.server.ActiveMQScheduledComponent$2.run(ActiveMQScheduledComponent.java:284) [artemis-commons-1.5.5.jbossorg-012.jar:1.5.5.jbossorg-012]
at org.apache.activemq.artemis.utils.OrderedExecutorFactory$OrderedExecutor$ExecutorTask.run(OrderedExecutorFactory.java:122) [artemis-commons-1.5.5.jbossorg-012.jar:1.5.5.jbossorg-012]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [rt.jar:1.8.0_171]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [rt.jar:1.8.0_171]
at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_171]
20:13:45,655 WARN [org.apache.activemq.artemis.jdbc.store.journal.JDBCJournalImpl] (Thread-17 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$3@4de066d8)) java.sql.SQLRecoverableException: Closed Connection
{code}
This thing can be very frustrating for customers who would rely on JDBC store in production. We should provide more robust solution and guide how to configure JDBC store to avoid this situation.
This problem will arise also when moving to OpenShift which runs in cloud environment with unreliable and slow network to database like Amazon AWS, MS Azure or OpenStack.
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
7 years, 6 months
[JBoss JIRA] (DROOLS-3268) [DMN Designer] Data-types: Check for use of built-in-type should be case sensitive
by Michael Anstis (Jira)
[ https://issues.jboss.org/browse/DROOLS-3268?page=com.atlassian.jira.plugi... ]
Michael Anstis commented on DROOLS-3268:
----------------------------------------
[~karreiro] Could you please provide [~dadossan] with a link to where you do the check for data type names equalling that of a built in type?
I suspect you're simply doing a comparison ignoring case.
> [DMN Designer] Data-types: Check for use of built-in-type should be case sensitive
> ----------------------------------------------------------------------------------
>
> Key: DROOLS-3268
> URL: https://issues.jboss.org/browse/DROOLS-3268
> Project: Drools
> Issue Type: Bug
> Components: DMN Editor
> Affects Versions: 7.14.0.Final
> Reporter: Michael Anstis
> Assignee: Daniel José dos Santos
> Priority: Blocker
> Labels: drools-tools
>
> On the Data Types dialog, creating a type/field called "Number" with an uppercase "N" should be allowed. FEEL/DMN is type sensitive, and only "number" lower case is reserved word. Check if the problem also happens for other types, except for "Any" and "Null" that have an upper case initial.
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
7 years, 6 months
[JBoss JIRA] (DROOLS-3268) [DMN Designer] Data-types: Check for use of built-in-type should be case sensitive
by Michael Anstis (Jira)
[ https://issues.jboss.org/browse/DROOLS-3268?page=com.atlassian.jira.plugi... ]
Michael Anstis reassigned DROOLS-3268:
--------------------------------------
Assignee: Daniel José dos Santos (was: Guilherme Carreiro)
> [DMN Designer] Data-types: Check for use of built-in-type should be case sensitive
> ----------------------------------------------------------------------------------
>
> Key: DROOLS-3268
> URL: https://issues.jboss.org/browse/DROOLS-3268
> Project: Drools
> Issue Type: Bug
> Components: DMN Editor
> Affects Versions: 7.14.0.Final
> Reporter: Michael Anstis
> Assignee: Daniel José dos Santos
> Priority: Blocker
> Labels: drools-tools
>
> On the Data Types dialog, creating a type/field called "Number" with an uppercase "N" should be allowed. FEEL/DMN is type sensitive, and only "number" lower case is reserved word. Check if the problem also happens for other types, except for "Any" and "Null" that have an upper case initial.
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
7 years, 6 months
[JBoss JIRA] (WFLY-10380) ThreadPoolExecutor leak under SM due to lack of privileged block
by Miroslav Novak (Jira)
[ https://issues.jboss.org/browse/WFLY-10380?page=com.atlassian.jira.plugin... ]
Miroslav Novak commented on WFLY-10380:
---------------------------------------
[~dmlloyd] [~jmesnil] Is this issue still valid?
> ThreadPoolExecutor leak under SM due to lack of privileged block
> ----------------------------------------------------------------
>
> Key: WFLY-10380
> URL: https://issues.jboss.org/browse/WFLY-10380
> Project: WildFly
> Issue Type: Bug
> Components: JMS
> Reporter: David Lloyd
> Priority: Critical
> Labels: security-manager
>
> Still researching the source of these leaks.
> The way the leak happens is, a {{java.util.concurrent.ThreadPoolExecutor}} is constructed from an unprivileged context. The pool starts up and threads are created without a problem, however, the thread pool is never shut down. The finalizer runs but since it tries to shut down the pool with an access control context that was captured during construction, it fails because the context did not have the {{modifyThread}} {{RuntimePermission}}, and the thread pool never shuts down.
> We need to identify the points where TPEs are being constructed without controlled privileges.
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
7 years, 6 months
[JBoss JIRA] (WFLY-10014) Unable to configure comma delimited enabled-protocols in remote-connector
by Miroslav Novak (Jira)
[ https://issues.jboss.org/browse/WFLY-10014?page=com.atlassian.jira.plugin... ]
Miroslav Novak commented on WFLY-10014:
---------------------------------------
As Artemis 2.x is present in WF master, marking jira as resolved with fix version Widfly 15.
> Unable to configure comma delimited enabled-protocols in remote-connector
> -------------------------------------------------------------------------
>
> Key: WFLY-10014
> URL: https://issues.jboss.org/browse/WFLY-10014
> Project: WildFly
> Issue Type: Bug
> Components: JMS
> Affects Versions: 12.0.0.Final
> Reporter: Nagaraju Chitimilla
> Assignee: Jeff Mesnil
> Priority: Critical
>
> When I want to configure enabled SSL protocols, Netty requires to use comma delimited list to be passed into enabled-protocols param in remote-connector
> {noformat}
> <param name="enabled-protocols" value="TLSv1.1,TLSv1.2"/>
> {noformat}
> But when comma is used in value in configuration string passed into ActiveMQRaUtils.parseConfig, parsing fails:
> {code:java}
> Caused by: java.lang.IllegalArgumentException: Invalid expression TLSv1.2 at enabledProtocols=TLSv1.1,TLSv1.2;trustStorePassword=N0tall0wed;keyStorePassword=N0tall0wed;port=61616;localAddress=0.0.0.0;sslEnabled=true;host=localhost;trustStorePath=C:/ActiveMQ/conf/keystore.p12;keyStorePath=C:/ActiveMQ/conf/keystore.p12
> at org.apache.activemq.artemis.ra.ActiveMQRaUtils.parseConfig(ActiveMQRaUtils.java:205)
> at org.apache.activemq.artemis.ra.ActiveMQResourceAdapter.setConnectionParameters(ActiveMQResourceAdapter.java:342)
> ... 18 more
> {code}
> I set this as critical as it prevent setting more then one allowed SSLContext protocol.
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
7 years, 6 months
[JBoss JIRA] (WFLY-10014) Unable to configure comma delimited enabled-protocols in remote-connector
by Miroslav Novak (Jira)
[ https://issues.jboss.org/browse/WFLY-10014?page=com.atlassian.jira.plugin... ]
Miroslav Novak resolved WFLY-10014.
-----------------------------------
Resolution: Done
> Unable to configure comma delimited enabled-protocols in remote-connector
> -------------------------------------------------------------------------
>
> Key: WFLY-10014
> URL: https://issues.jboss.org/browse/WFLY-10014
> Project: WildFly
> Issue Type: Bug
> Components: JMS
> Affects Versions: 12.0.0.Final
> Reporter: Nagaraju Chitimilla
> Assignee: Jeff Mesnil
> Priority: Critical
>
> When I want to configure enabled SSL protocols, Netty requires to use comma delimited list to be passed into enabled-protocols param in remote-connector
> {noformat}
> <param name="enabled-protocols" value="TLSv1.1,TLSv1.2"/>
> {noformat}
> But when comma is used in value in configuration string passed into ActiveMQRaUtils.parseConfig, parsing fails:
> {code:java}
> Caused by: java.lang.IllegalArgumentException: Invalid expression TLSv1.2 at enabledProtocols=TLSv1.1,TLSv1.2;trustStorePassword=N0tall0wed;keyStorePassword=N0tall0wed;port=61616;localAddress=0.0.0.0;sslEnabled=true;host=localhost;trustStorePath=C:/ActiveMQ/conf/keystore.p12;keyStorePath=C:/ActiveMQ/conf/keystore.p12
> at org.apache.activemq.artemis.ra.ActiveMQRaUtils.parseConfig(ActiveMQRaUtils.java:205)
> at org.apache.activemq.artemis.ra.ActiveMQResourceAdapter.setConnectionParameters(ActiveMQResourceAdapter.java:342)
> ... 18 more
> {code}
> I set this as critical as it prevent setting more then one allowed SSLContext protocol.
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
7 years, 6 months
[JBoss JIRA] (WFLY-10014) Unable to configure comma delimited enabled-protocols in remote-connector
by Miroslav Novak (Jira)
[ https://issues.jboss.org/browse/WFLY-10014?page=com.atlassian.jira.plugin... ]
Miroslav Novak updated WFLY-10014:
----------------------------------
Fix Version/s: 15.0.0.Alpha1
> Unable to configure comma delimited enabled-protocols in remote-connector
> -------------------------------------------------------------------------
>
> Key: WFLY-10014
> URL: https://issues.jboss.org/browse/WFLY-10014
> Project: WildFly
> Issue Type: Bug
> Components: JMS
> Affects Versions: 12.0.0.Final
> Reporter: Nagaraju Chitimilla
> Assignee: Jeff Mesnil
> Priority: Critical
> Fix For: 15.0.0.Alpha1
>
>
> When I want to configure enabled SSL protocols, Netty requires to use comma delimited list to be passed into enabled-protocols param in remote-connector
> {noformat}
> <param name="enabled-protocols" value="TLSv1.1,TLSv1.2"/>
> {noformat}
> But when comma is used in value in configuration string passed into ActiveMQRaUtils.parseConfig, parsing fails:
> {code:java}
> Caused by: java.lang.IllegalArgumentException: Invalid expression TLSv1.2 at enabledProtocols=TLSv1.1,TLSv1.2;trustStorePassword=N0tall0wed;keyStorePassword=N0tall0wed;port=61616;localAddress=0.0.0.0;sslEnabled=true;host=localhost;trustStorePath=C:/ActiveMQ/conf/keystore.p12;keyStorePath=C:/ActiveMQ/conf/keystore.p12
> at org.apache.activemq.artemis.ra.ActiveMQRaUtils.parseConfig(ActiveMQRaUtils.java:205)
> at org.apache.activemq.artemis.ra.ActiveMQResourceAdapter.setConnectionParameters(ActiveMQResourceAdapter.java:342)
> ... 18 more
> {code}
> I set this as critical as it prevent setting more then one allowed SSLContext protocol.
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
7 years, 6 months
[JBoss JIRA] (WFLY-10731) core-threads of ManagedScheduledExecutorService limits number of maximal threads
by gunter zeilinger (Jira)
[ https://issues.jboss.org/browse/WFLY-10731?page=com.atlassian.jira.plugin... ]
gunter zeilinger commented on WFLY-10731:
-----------------------------------------
The misleading description is also returned by
{noformat}
$ jboss-cli -c /subsystem=ee/managed-scheduled-executor-service=default:read-resource-description
:
"core-threads" => {
"type" => INT,
"description" => "The minimum number of threads to be used by the scheduled executor.",
:
{noformat}
> core-threads of ManagedScheduledExecutorService limits number of maximal threads
> --------------------------------------------------------------------------------
>
> Key: WFLY-10731
> URL: https://issues.jboss.org/browse/WFLY-10731
> Project: WildFly
> Issue Type: Bug
> Components: EE
> Affects Versions: 12.0.0.Final
> Reporter: gunter zeilinger
> Assignee: Jiri Ondrusek
> Priority: Major
>
> After changing the default configuration:
> {noformat}
> <managed-scheduled-executor-services>
> - <managed-scheduled-executor-service name="default" jndi-name="java:jboss/ee/concurrency/scheduler/default" context-service="default" hung-task-threshold="60000" keepalive-time="3000"/>
> + <managed-scheduled-executor-service name="default" jndi-name="java:jboss/ee/concurrency/scheduler/default" context-service="default" long-running-tasks="true" core-threads="2" keepalive-time="3000"/>
> </managed-scheduled-executor-services>
> {noformat}
> only 2 (= value of core-threads) threads are used by the Scheduled Executer Service:
> {code:bash}
> $ grep ManagedScheduledExecutorService $WILDFLY/standalone/log/server.log | tail
> 2018-07-24 12:03:22,901 INFO [org.dcm4che3.net.Association] (EE-ManagedScheduledExecutorService-default-Thread-2) ..
> 2018-07-24 12:03:22,911 INFO [org.dcm4che3.net.Association] (EE-ManagedScheduledExecutorService-default-Thread-1) ..
> 2018-07-24 12:03:22,923 INFO [org.dcm4che3.net.Association] (EE-ManagedScheduledExecutorService-default-Thread-2) ..
> 2018-07-24 12:03:22,963 INFO [org.dcm4che3.net.Association] (EE-ManagedScheduledExecutorService-default-Thread-1) ..
> 2018-07-24 12:03:22,967 INFO [org.dcm4che3.net.Association] (EE-ManagedScheduledExecutorService-default-Thread-2) ..
> 2018-07-24 12:03:22,981 INFO [org.dcm4che3.net.Association] (EE-ManagedScheduledExecutorService-default-Thread-1) ..
> 2018-07-24 12:03:22,996 INFO [org.dcm4che3.net.Association] (EE-ManagedScheduledExecutorService-default-Thread-2) ..
> 2018-07-24 12:03:23,016 INFO [org.dcm4che3.net.Association] (EE-ManagedScheduledExecutorService-default-Thread-1) ..
> 2018-07-24 12:03:23,035 INFO [org.dcm4che3.net.Association] (EE-ManagedScheduledExecutorService-default-Thread-2) ..
> 2018-07-24 12:03:23,039 INFO [org.dcm4che3.net.Association] (EE-ManagedScheduledExecutorService-default-Thread-1) ..
> {code}
> I have not yet tested if the issue also effects Wildfly 13.0.0.Final ( cannot use Wildfly 13.0.0.Final because of WFLY-10531 )
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
7 years, 6 months