[JBoss JIRA] (TEIID-5962) Teiid locks many threads when Salesforce is disconnected
by Steven Hawkins (Jira)
[ https://issues.redhat.com/browse/TEIID-5962?page=com.atlassian.jira.plugi... ]
Steven Hawkins updated TEIID-5962:
----------------------------------
Original Estimate: 1 hour
Remaining Estimate: 1 hour
Story Points: 0.1
Sprint: DV Sprint 65
> Teiid locks many threads when Salesforce is disconnected
> --------------------------------------------------------
>
> Key: TEIID-5962
> URL: https://issues.redhat.com/browse/TEIID-5962
> Project: Teiid
> Issue Type: Bug
> Components: Salesforce Connector
> Affects Versions: 12.3.1
> Reporter: Renat Eskenin
> Assignee: Steven Hawkins
> Priority: Major
> Attachments: Снимок экрана от 2020-05-31 11-09-10.png
>
> Original Estimate: 1 hour
> Remaining Estimate: 1 hour
>
> We had two incidents in our experimental service with teiid spring boot app. When Salesforce is down to 20min and service with teiid trying to call SQL from Salesforce we have many threads in tomcat server (in spring boot app). Then when 200 threads is activated service do not reply to requests because is "max threads" in tomcat of service.
> Why teiid do not send response as "Salesforce is down - error" or connection timeout?
> Micro schema:
> HTTP Requests to REST api -> Teiid spring boot app -> {broken 20min}Salesforce
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
4 years
[JBoss JIRA] (TEIID-5962) Teiid locks many threads when Salesforce is disconnected
by Steven Hawkins (Jira)
[ https://issues.redhat.com/browse/TEIID-5962?page=com.atlassian.jira.plugi... ]
Steven Hawkins resolved TEIID-5962.
-----------------------------------
Fix Version/s: (was: 15.0)
Resolution: Partially Completed
Updated the Teiid docs to talk more about the available salesforce properties. In the docs we've taken the convention of saying consult the cli rather than explicitly documenting. So a separate issue would be needed for teiid spring boot to ensure appropriate javadocs on the configuration classes or secondary docs.
If there are additional details that can be added here, such as a thread dump this can be reopened and likely moved to teiid spring boot.
> Teiid locks many threads when Salesforce is disconnected
> --------------------------------------------------------
>
> Key: TEIID-5962
> URL: https://issues.redhat.com/browse/TEIID-5962
> Project: Teiid
> Issue Type: Bug
> Components: Salesforce Connector
> Affects Versions: 12.3.1
> Reporter: Renat Eskenin
> Assignee: Steven Hawkins
> Priority: Major
> Attachments: Снимок экрана от 2020-05-31 11-09-10.png
>
>
> We had two incidents in our experimental service with teiid spring boot app. When Salesforce is down to 20min and service with teiid trying to call SQL from Salesforce we have many threads in tomcat server (in spring boot app). Then when 200 threads is activated service do not reply to requests because is "max threads" in tomcat of service.
> Why teiid do not send response as "Salesforce is down - error" or connection timeout?
> Micro schema:
> HTTP Requests to REST api -> Teiid spring boot app -> {broken 20min}Salesforce
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
4 years
[JBoss JIRA] (TEIID-5969) The "already removed" exception in case of copyLobs=true translator property for MSSQL
by Steven Hawkins (Jira)
[ https://issues.redhat.com/browse/TEIID-5969?page=com.atlassian.jira.plugi... ]
Steven Hawkins updated TEIID-5969:
----------------------------------
Fix Version/s: 15.0
> The "already removed" exception in case of copyLobs=true translator property for MSSQL
> --------------------------------------------------------------------------------------
>
> Key: TEIID-5969
> URL: https://issues.redhat.com/browse/TEIID-5969
> Project: Teiid
> Issue Type: Quality Risk
> Components: Query Engine
> Affects Versions: 13.1
> Reporter: Dmitrii Pogorelov
> Assignee: Steven Hawkins
> Priority: Major
> Fix For: 15.0
>
>
> We're getting the following exception:
> {code:java}
> 2020-06-10 16:01:45,326 WARN [org.teiid.PROCESSOR] (Worker21_QueryProcessorQueue272) 1bxj3NJqJLpO TEIID30020 Processing exception for request 1bxj3NJqJLpO.10 'TEIID30328 Unable to evaluate convert(ms_dwh.v1.expr1, string): TEIID30384 Error while evaluating function convert'. Originally ExpressionEvaluationException 'already removed' FileStore.java:156. Enable more detailed logging to see the entire stacktrace. {code}
> in the case of MSSQL connector and copyLobs=true in its translator property.
> After debugging I discovered that the error occurred actually because of not enough value in
> MAX_LOB_MEMORY_BYTES which is used in LobManager. The variable depends on
> org.teiid.maxStringLength which is set usually to 4000 by default. But if someone put a data with text field type in MSSQL having more than, say, 20000 symbols then the field won't be read by Teiid at all, Teiid will just throw out the exception mentioned above. Will it be possible to improve LobManager so that it can read Text data dynamically increasing needed memory for the process? Otherwise even setting the org.teiid.maxStringLength to 20000 Teiid can raise the exception because of a Text data with length more than 20000 symbols.
>
> Steps to reproduce:
> # create the v1 table in MSSQL:
> {code:java}
> CREATE TABLE [dbo].[v1](CREATE TABLE [dbo].[v1]( [expr1] [text] NULL, [b] [int] NULL) ON [PRIMARY] TEXTIMAGE_ON [PRIMARY] {code}
> 2. insert in the table the following data (I did it via standard MSSQL Server Management Studio):
> {code:java}
> insert into test_dwh.dbo.v1 values (replicate(convert(varchar(max),'a'), 20000), 1) ; {code}
> 3. add MSSQL as a source to Teiid:
> {code:java}
> <datasource jndi-name="java:/mssql-test-dwh" pool-name="mssql-test-dwh" enabled="true" use-java-context="true">
> <connection-url>jdbc:sqlserver://localhost:1433;databaseName=test_dwh</connection-url>
> <driver-class&gt;com.microsoft.sqlserver.jdbc.SQLServerDriver</driver-class&gt;
> <driver>com.microsoft.sqlserver</driver>
> <new-connection-sql>SET ANSI_WARNINGS OFF</new-connection-sql>
> <pool>
> <min-pool-size>2</min-pool-size>
> <max-pool-size>70</max-pool-size>
> </pool>
> <security>
> <user-name>*****</user-name>
> <password>*****</password>
> </security>
> <validation>
> <check-valid-connection-sql>select 1</check-valid-connection-sql>
> </validation>
> <timeout>
> <blocking-timeout-millis>120000</blocking-timeout-millis>
> <idle-timeout-minutes>5</idle-timeout-minutes>
> </timeout>
> </datasource>{code}
> 4. add ms model:
> {code:java}
> <model name="ms_dwh">
> <property name="importer.useFullSchemaName" value="false"/>
> <property name="importer.tableTypes" value="TABLE,VIEW"/>
> <property name="importer.importKeys" value="false"/>
> <property name="importer.schemaPattern" value="dbo"/>
> <source name="dwh_ms" translator-name="mySqlserver" connection-jndi-name="java:/mssql-test-dwh"/>
> </model>{code}
> 5. add mySqlserver translator:
> {code:java}
> <translator name="mySqlserver" type="sqlserver">
> <property name="SupportsNativeQueries" value="true"/>
> <property name="copyLobs" value="true"/>
> </translator> {code}
> 6. try to run the following query:
> {code:java}
> select cast(expr1 as string) from ms_dwh.v1 ;; {code}
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
4 years
[JBoss JIRA] (TEIID-5937) Union All followed by except returns wrong results
by Steven Hawkins (Jira)
[ https://issues.redhat.com/browse/TEIID-5937?focusedWorklogId=12451400&pag... ]
Steven Hawkins logged work on TEIID-5937:
-----------------------------------------
Author: Steven Hawkins
Created on: 10/Jun/20 4:45 PM
Start Date: 10/Jun/20 4:45 PM
Worklog Time Spent: 1 hour
Issue Time Tracking
-------------------
Time Spent: 5 hours, 30 minutes (was: 4 hours, 30 minutes)
Worklog Id: (was: 12451400)
> Union All followed by except returns wrong results
> --------------------------------------------------
>
> Key: TEIID-5937
> URL: https://issues.redhat.com/browse/TEIID-5937
> Project: Teiid
> Issue Type: Bug
> Components: Query Engine
> Affects Versions: 13.1
> Reporter: Dmitrii Pogorelov
> Assignee: Steven Hawkins
> Priority: Blocker
> Fix For: 15.0, 13.1.2, 14.0.1
>
> Original Estimate: 4 hours
> Time Spent: 5 hours, 30 minutes
> Remaining Estimate: 0 minutes
>
> The following queries return wrong results:
> {code:sql}
> -- <null> though should return 1
> select * from views.test2 except select * from views.test1
> UNION ALL
> select * from views.test1 except select * from views.test2 ;;
>
> -- <null> though should return 1
> (select * from views.test2 except select * from views.test1
> UNION ALL
> select * from views.test1) except select * from views.test2 ;;
> {code}
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
4 years
[JBoss JIRA] (TEIID-5937) Union All followed by except returns wrong results
by Steven Hawkins (Jira)
[ https://issues.redhat.com/browse/TEIID-5937?page=com.atlassian.jira.plugi... ]
Steven Hawkins commented on TEIID-5937:
---------------------------------------
This case is now resolved as well. I've simplified this - rather than trying to prevent the view removal everything works if we just keep the projection mapping using the constant added.
> Union All followed by except returns wrong results
> --------------------------------------------------
>
> Key: TEIID-5937
> URL: https://issues.redhat.com/browse/TEIID-5937
> Project: Teiid
> Issue Type: Bug
> Components: Query Engine
> Affects Versions: 13.1
> Reporter: Dmitrii Pogorelov
> Assignee: Steven Hawkins
> Priority: Blocker
> Fix For: 15.0, 13.1.2, 14.0.1
>
> Original Estimate: 4 hours
> Time Spent: 4 hours, 30 minutes
> Remaining Estimate: 0 minutes
>
> The following queries return wrong results:
> {code:sql}
> -- <null> though should return 1
> select * from views.test2 except select * from views.test1
> UNION ALL
> select * from views.test1 except select * from views.test2 ;;
>
> -- <null> though should return 1
> (select * from views.test2 except select * from views.test1
> UNION ALL
> select * from views.test1) except select * from views.test2 ;;
> {code}
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
4 years
[JBoss JIRA] (TEIID-5937) Union All followed by except returns wrong results
by Steven Hawkins (Jira)
[ https://issues.redhat.com/browse/TEIID-5937?page=com.atlassian.jira.plugi... ]
Steven Hawkins commented on TEIID-5937:
---------------------------------------
That is related, but slightly different the projection conflict is being propagated downward instead of upwards. That requires different checks.
> Union All followed by except returns wrong results
> --------------------------------------------------
>
> Key: TEIID-5937
> URL: https://issues.redhat.com/browse/TEIID-5937
> Project: Teiid
> Issue Type: Bug
> Components: Query Engine
> Affects Versions: 13.1
> Reporter: Dmitrii Pogorelov
> Assignee: Steven Hawkins
> Priority: Blocker
> Fix For: 15.0, 13.1.2, 14.0.1
>
> Original Estimate: 4 hours
> Time Spent: 4 hours, 30 minutes
> Remaining Estimate: 0 minutes
>
> The following queries return wrong results:
> {code:sql}
> -- <null> though should return 1
> select * from views.test2 except select * from views.test1
> UNION ALL
> select * from views.test1 except select * from views.test2 ;;
>
> -- <null> though should return 1
> (select * from views.test2 except select * from views.test1
> UNION ALL
> select * from views.test1) except select * from views.test2 ;;
> {code}
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
4 years
[JBoss JIRA] (TEIID-5968) enabled=true flag not visible in xa-datasource
by Steven Hawkins (Jira)
[ https://issues.redhat.com/browse/TEIID-5968?page=com.atlassian.jira.plugi... ]
Steven Hawkins updated TEIID-5968:
----------------------------------
Original Estimate: 2 hours, 30 minutes
Remaining Estimate: 2 hours, 30 minutes
Story Points: 0.5
Sprint: DV Sprint 65
looking at master, for example IntegrationTestDeployment.testCreateXADatasource I see the same behavior an xa datasource is only fully enabled after restart.
> enabled=true flag not visible in xa-datasource
> ----------------------------------------------
>
> Key: TEIID-5968
> URL: https://issues.redhat.com/browse/TEIID-5968
> Project: Teiid
> Issue Type: Bug
> Components: AdminApi
> Affects Versions: 11.1.2
> Reporter: Manoj Majumdar
> Assignee: Steven Hawkins
> Priority: Major
> Original Estimate: 2 hours, 30 minutes
> Remaining Estimate: 2 hours, 30 minutes
>
> While creating a XA-Datasource using admin Api, enabled="true" paramter is not set and is not visible in standalone-teiid.xml, have to add it manually.
>
> Because of this flag not set, connection pooling is not working and teiid is creating more connections than max-pool-size mentioned
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
4 years
[JBoss JIRA] (TEIID-5968) enabled=true flag not visible in xa-datasource
by Steven Hawkins (Jira)
[ https://issues.redhat.com/browse/TEIID-5968?page=com.atlassian.jira.plugi... ]
Steven Hawkins resolved TEIID-5968.
-----------------------------------
Fix Version/s: 15.0
13.1.2
14.0.1
Resolution: Done
Adding enabled true to all datasource creation.
> enabled=true flag not visible in xa-datasource
> ----------------------------------------------
>
> Key: TEIID-5968
> URL: https://issues.redhat.com/browse/TEIID-5968
> Project: Teiid
> Issue Type: Bug
> Components: AdminApi
> Affects Versions: 11.1.2
> Reporter: Manoj Majumdar
> Assignee: Steven Hawkins
> Priority: Major
> Fix For: 15.0, 13.1.2, 14.0.1
>
> Original Estimate: 2 hours, 30 minutes
> Remaining Estimate: 2 hours, 30 minutes
>
> While creating a XA-Datasource using admin Api, enabled="true" paramter is not set and is not visible in standalone-teiid.xml, have to add it manually.
>
> Because of this flag not set, connection pooling is not working and teiid is creating more connections than max-pool-size mentioned
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
4 years
[JBoss JIRA] (TEIID-5937) Union All followed by except returns wrong results
by Dmitrii Pogorelov (Jira)
[ https://issues.redhat.com/browse/TEIID-5937?page=com.atlassian.jira.plugi... ]
Dmitrii Pogorelov commented on TEIID-5937:
------------------------------------------
[~shawkins] we found also such case:
{code:java}
-- the value for the second column should be returned as true, while it is a null
select name, true From (select 'test' as name except select 'bla') x;; {code}
could you please check it as well?
> Union All followed by except returns wrong results
> --------------------------------------------------
>
> Key: TEIID-5937
> URL: https://issues.redhat.com/browse/TEIID-5937
> Project: Teiid
> Issue Type: Bug
> Components: Query Engine
> Affects Versions: 13.1
> Reporter: Dmitrii Pogorelov
> Assignee: Steven Hawkins
> Priority: Blocker
> Fix For: 15.0, 13.1.2, 14.0.1
>
> Original Estimate: 4 hours
> Time Spent: 4 hours, 30 minutes
> Remaining Estimate: 0 minutes
>
> The following queries return wrong results:
> {code:sql}
> -- <null> though should return 1
> select * from views.test2 except select * from views.test1
> UNION ALL
> select * from views.test1 except select * from views.test2 ;;
>
> -- <null> though should return 1
> (select * from views.test2 except select * from views.test1
> UNION ALL
> select * from views.test1) except select * from views.test2 ;;
> {code}
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
4 years
[JBoss JIRA] (TEIID-5969) The "already removed" exception in case of copyLobs=true translator property for MSSQL
by Dmitrii Pogorelov (Jira)
[ https://issues.redhat.com/browse/TEIID-5969?page=com.atlassian.jira.plugi... ]
Dmitrii Pogorelov commented on TEIID-5969:
------------------------------------------
[~shawkins] thx a lot for your reply.
> That is not expected. We should create a disk / memory backed copy that will live as long as the query is open.
will it be fixed by you?
> You may have found a code path though were that is inhibiting the full copying and thus seeing the already removed exception.
maybe, I just set the org.teiid.maxStringLength to 20000 thus the MAX_LOB_MEMORY_BYTES variable was increased as well and the exception disappeared.
> Again that is not expected. What should happen is that the initial copy will happen, then the cast will truncate the lob value.
yes, I got it, it should work if this is implemented: "We should create a disk / memory backed copy that will live as long as the query is open."
> No, that is really a separate consideration. You are asking about having dynamic / unlimited length string values by default - that is dangerous from a memory management perspective.
yes, I know that it can be dangerous as such a mechanism can consume all memory in case of big data. It was just one of the suggestions though not so successful.
> The "already removed" exception in case of copyLobs=true translator property for MSSQL
> --------------------------------------------------------------------------------------
>
> Key: TEIID-5969
> URL: https://issues.redhat.com/browse/TEIID-5969
> Project: Teiid
> Issue Type: Quality Risk
> Components: Query Engine
> Affects Versions: 13.1
> Reporter: Dmitrii Pogorelov
> Assignee: Steven Hawkins
> Priority: Major
>
> We're getting the following exception:
> {code:java}
> 2020-06-10 16:01:45,326 WARN [org.teiid.PROCESSOR] (Worker21_QueryProcessorQueue272) 1bxj3NJqJLpO TEIID30020 Processing exception for request 1bxj3NJqJLpO.10 'TEIID30328 Unable to evaluate convert(ms_dwh.v1.expr1, string): TEIID30384 Error while evaluating function convert'. Originally ExpressionEvaluationException 'already removed' FileStore.java:156. Enable more detailed logging to see the entire stacktrace. {code}
> in the case of MSSQL connector and copyLobs=true in its translator property.
> After debugging I discovered that the error occurred actually because of not enough value in
> MAX_LOB_MEMORY_BYTES which is used in LobManager. The variable depends on
> org.teiid.maxStringLength which is set usually to 4000 by default. But if someone put a data with text field type in MSSQL having more than, say, 20000 symbols then the field won't be read by Teiid at all, Teiid will just throw out the exception mentioned above. Will it be possible to improve LobManager so that it can read Text data dynamically increasing needed memory for the process? Otherwise even setting the org.teiid.maxStringLength to 20000 Teiid can raise the exception because of a Text data with length more than 20000 symbols.
>
> Steps to reproduce:
> # create the v1 table in MSSQL:
> {code:java}
> CREATE TABLE [dbo].[v1](CREATE TABLE [dbo].[v1]( [expr1] [text] NULL, [b] [int] NULL) ON [PRIMARY] TEXTIMAGE_ON [PRIMARY] {code}
> 2. insert in the table the following data (I did it via standard MSSQL Server Management Studio):
> {code:java}
> insert into test_dwh.dbo.v1 values (replicate(convert(varchar(max),'a'), 20000), 1) ; {code}
> 3. add MSSQL as a source to Teiid:
> {code:java}
> <datasource jndi-name="java:/mssql-test-dwh" pool-name="mssql-test-dwh" enabled="true" use-java-context="true">
> <connection-url>jdbc:sqlserver://localhost:1433;databaseName=test_dwh</connection-url>
> <driver-class&gt;com.microsoft.sqlserver.jdbc.SQLServerDriver</driver-class&gt;
> <driver>com.microsoft.sqlserver</driver>
> <new-connection-sql>SET ANSI_WARNINGS OFF</new-connection-sql>
> <pool>
> <min-pool-size>2</min-pool-size>
> <max-pool-size>70</max-pool-size>
> </pool>
> <security>
> <user-name>*****</user-name>
> <password>*****</password>
> </security>
> <validation>
> <check-valid-connection-sql>select 1</check-valid-connection-sql>
> </validation>
> <timeout>
> <blocking-timeout-millis>120000</blocking-timeout-millis>
> <idle-timeout-minutes>5</idle-timeout-minutes>
> </timeout>
> </datasource>{code}
> 4. add ms model:
> {code:java}
> <model name="ms_dwh">
> <property name="importer.useFullSchemaName" value="false"/>
> <property name="importer.tableTypes" value="TABLE,VIEW"/>
> <property name="importer.importKeys" value="false"/>
> <property name="importer.schemaPattern" value="dbo"/>
> <source name="dwh_ms" translator-name="mySqlserver" connection-jndi-name="java:/mssql-test-dwh"/>
> </model>{code}
> 5. add mySqlserver translator:
> {code:java}
> <translator name="mySqlserver" type="sqlserver">
> <property name="SupportsNativeQueries" value="true"/>
> <property name="copyLobs" value="true"/>
> </translator> {code}
> 6. try to run the following query:
> {code:java}
> select cast(expr1 as string) from ms_dwh.v1 ;; {code}
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
4 years