[JBoss JIRA] (TEIID-4283) External Materialization interleaving loads SYSADMIN.loadMatView
by Steven Hawkins (JIRA)
[ https://issues.jboss.org/browse/TEIID-4283?page=com.atlassian.jira.plugin... ]
Steven Hawkins closed TEIID-4283.
---------------------------------
> External Materialization interleaving loads SYSADMIN.loadMatView
> ----------------------------------------------------------------
>
> Key: TEIID-4283
> URL: https://issues.jboss.org/browse/TEIID-4283
> Project: Teiid
> Issue Type: Bug
> Components: Query Engine
> Affects Versions: 8.12.5
> Reporter: Jan Stastny
> Assignee: Steven Hawkins
> Priority: Blocker
> Fix For: 9.1, 8.12.5, 9.0.1
>
> Attachments: jdg-log.txt
>
>
> There is a possibility of two concurrent loads of single materialized view when using SYSADMIN.loadMatView function. I observed the issue while:
> # Waiting for ttl-driven reload and in right time I triggered another load explicitly by invoking the loadMatView function.
> # Invoking SYSADMIN.loadMatView function multiple times in quick succession.
> The view definition:
> {code:sql}
> CREATE VIEW external_long_ttl (
> customer_id integer NOT NULL,
> total_amount integer
> ) OPTIONS (MATERIALIZED 'TRUE', UPDATABLE 'FALSE',
> MATERIALIZED_TABLE 'Source.JSTASTNY.dv_matviews_mat_view',
> "teiid_rel:ALLOW_MATVIEW_MANAGEMENT" 'true',
> "teiid_rel:MATVIEW_STATUS_TABLE" 'Source.JSTASTNY.dv_matviews_statustable',
> "teiid_rel:ON_VDB_START_SCRIPT" 'MERGE INTO dv_matviews_check_table(id,vdb_create) SELECT id, vdb_create+1 FROM dv_matviews_check_table WHERE id=''external_long_ttl'';',
> "teiid_rel:ON_VDB_DROP_SCRIPT" 'MERGE INTO dv_matviews_check_table(id,vdb_drop) SELECT id, vdb_drop+1 FROM dv_matviews_check_table WHERE id=''external_long_ttl'';',
> "teiid_rel:MATVIEW_LOAD_SCRIPT" 'INSERT INTO dv_matviews_mat_view_stage(customer_id,total_amount) SELECT CONVERT(c.id,integer) AS customer_id, CONVERT(SUM(o.amount),integer) AS total_amount FROM dv_matviews_customers c INNER JOIN dv_matviews_orders o ON c.id = o.customer_id GROUP BY c.id;',
> "teiid_rel:MATVIEW_BEFORE_LOAD_SCRIPT" 'exec Source.native(''truncate table ${db.table.prefix}dv_matviews_mat_view_stage'');MERGE INTO dv_matviews_check_table(id,before_load) SELECT id, before_load+1 FROM dv_matviews_check_table WHERE id=''external_long_ttl'';',
> "teiid_rel:MATVIEW_AFTER_LOAD_SCRIPT" 'exec Source.native(''RENAME TABLE ${db.table.prefix}dv_matviews_mat_view_stage TO ${db.table.prefix}dv_matviews_mat_view_temp'');exec Source.native(''RENAME TABLE ${db.table.prefix}dv_matviews_mat_view TO ${db.table.prefix}dv_matviews_mat_view_stage'');exec Source.native(''RENAME TABLE ${db.table.prefix}dv_matviews_mat_view_temp TO ${db.table.prefix}dv_matviews_mat_view'');MERGE INTO dv_matviews_check_table(id,after_load) SELECT id, after_load+1 FROM dv_matviews_check_table WHERE id=''external_long_ttl'';',
> "teiid_rel:MATVIEW_ONERROR_ACTION" 'WAIT',
> "teiid_rel:MATVIEW_TTL" 20000)
> AS SELECT CONVERT(c.id,integer) AS customer_id, CONVERT(SUM(o.amount),integer) AS total_amount FROM dv_matviews_customers c INNER JOIN dv_matviews_orders o ON c.id = o.customer_id GROUP BY c.id;
> {code}
> Query to explicitly load the view:
> {code:sql}
> exec SYSADMIN.loadMatView(schemaName=>'View',viewname=>'external_long_ttl', invalidate=>'true')
> {code}
> I attached a log which starts with ttl-driven load, then the explicit load is performed. The issue noticed is in teiid_rel:MATVIEW_AFTER_LOAD_SCRIPT , where primary and staging tables are being swapped. The second load (explicit) can't find the original table, as the ttl-driven load had already renamed it.
> But this situation of two concurrent loads should never occur.
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
8 years, 5 months
[JBoss JIRA] (TEIID-4343) Impala Translator - With Clause Support
by Steven Hawkins (JIRA)
[ https://issues.jboss.org/browse/TEIID-4343?page=com.atlassian.jira.plugin... ]
Steven Hawkins updated TEIID-4343:
----------------------------------
Issue Type: Bug (was: Feature Request)
> Impala Translator - With Clause Support
> ---------------------------------------
>
> Key: TEIID-4343
> URL: https://issues.jboss.org/browse/TEIID-4343
> Project: Teiid
> Issue Type: Bug
> Components: Misc. Connectors
> Affects Versions: 8.13.3
> Reporter: Scott Wallace
> Assignee: Steven Hawkins
> Fix For: 9.1
>
>
> I'm getting an error using the With clause in Teiid connecting to Impala
> {code}
> with t as (select * from schema_name.table_name)
> select * from t;
> {code}
> Error:
> {code}
> org.teiid.runtime.client.TeiidClientException: java.lang.RuntimeException: Remote org.teiid.core.TeiidProcessingException: TEIID30504 warehouse: 11300 TEIID11008:TEIID11004 Error executing statement(s): [Prepared Values: [] SQL: WITH t (col1, col2, col3) AS (SELECT g_0.`col1`, g_0.`col2`, g_0.`col3` FROM schema_name.table_name g_0) SELECT g_0.col1, g_0.col2, g_0.col3 FROM t g_0]
> Elapsed Time: 0 hr, 0 min, 0 sec, 841 ms.
> {code}
> Full error stack:
> {code}
> 14:37:35,491 INFO [org.teiid.CONNECTOR] (Worker1_QueryProcessorQueue1) ImpalaExecutionFactory Commit=true;DatabaseProductName=Impala;DatabaseProductVersion=2.5.0-cdh5.7.0;DriverMajorVersion=2;DriverMajorVersion=5;DriverName=ImpalaJDBC;DriverVersion=02.05.31.1051;IsolationLevel=1
> 14:37:35,516 WARN [org.teiid.CONNECTOR] (Worker1_QueryProcessorQueue1) Connector worker process failed for atomic-request=u1RZzQGciJra.0.1.0: org.teiid.translator.jdbc.JDBCExecutionException: 11300 TEIID11008:TEIID11004 Error executing statement(s): [Prepared Values: [] SQL: WITH t (col1, col2, col3) AS (SELECT g_0.`col1`, g_0.`col2`, g_0.`col3` FROM schema_name.table_name g_0) SELECT g_0.col1, g_0.col2, g_0.col3 FROM t g_0]
> at org.teiid.translator.jdbc.JDBCQueryExecution.execute(JDBCQueryExecution.java:131) [translator-jdbc-8.13.3.jar:8.13.3]
> at org.teiid.dqp.internal.datamgr.ConnectorWorkItem.execute(ConnectorWorkItem.java:359)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.8.0_91]
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) [rt.jar:1.8.0_91]
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_91]
> at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_91]
> at org.teiid.dqp.internal.datamgr.ConnectorManager$1.invoke(ConnectorManager.java:211)
> at com.sun.proxy.$Proxy29.execute(Unknown Source)
> at org.teiid.dqp.internal.process.DataTierTupleSource.getResults(DataTierTupleSource.java:306)
> at org.teiid.dqp.internal.process.DataTierTupleSource$1.call(DataTierTupleSource.java:112)
> at org.teiid.dqp.internal.process.DataTierTupleSource$1.call(DataTierTupleSource.java:108)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_91]
> at org.teiid.dqp.internal.process.FutureWork.run(FutureWork.java:65)
> at org.teiid.dqp.internal.process.DQPWorkContext.runInContext(DQPWorkContext.java:276)
> at org.teiid.dqp.internal.process.ThreadReuseExecutor$RunnableWrapper.run(ThreadReuseExecutor.java:119)
> at org.teiid.dqp.internal.process.ThreadReuseExecutor$3.run(ThreadReuseExecutor.java:210)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_91]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_91]
> at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_91]
> Caused by: java.sql.SQLDataException: [Simba][JDBC](11300) A ResultSet was expected but not generated from query "WITH t (col1, col2, col3) AS (SELECT g_0.`col1`, g_0.`col2`, g_0.`col3` FROM schema_name.table_name g_0) SELECT g_0.col1, g_0.col2, g_0.col3 FROM t g_0". Query not executed.
> at com.cloudera.exceptions.ExceptionConverter.toSQLException(Unknown Source)
> at com.cloudera.jdbc.common.SStatement.checkCondition(Unknown Source)
> at com.cloudera.jdbc.common.SPreparedStatement.executeWithParams(Unknown Source)
> at com.cloudera.jdbc.common.SPreparedStatement.executeQuery(Unknown Source)
> at org.jboss.jca.adapters.jdbc.WrappedPreparedStatement.executeQuery(WrappedPreparedStatement.java:504)
> at org.teiid.translator.jdbc.JDBCQueryExecution.execute(JDBCQueryExecution.java:123) [translator-jdbc-8.13.3.jar:8.13.3]
> ... 18 more
> {code}
> Taking the prepared statement and executing against Impala directly, succeeds without any issues.
> Is the With clause supported in the Impala translator?
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
8 years, 5 months
[JBoss JIRA] (TEIID-4296) File translator cannot find a file without an extensioin
by Steven Hawkins (JIRA)
[ https://issues.jboss.org/browse/TEIID-4296?page=com.atlassian.jira.plugin... ]
Steven Hawkins reassigned TEIID-4296:
-------------------------------------
Assignee: Steven Hawkins
> File translator cannot find a file without an extensioin
> --------------------------------------------------------
>
> Key: TEIID-4296
> URL: https://issues.jboss.org/browse/TEIID-4296
> Project: Teiid
> Issue Type: Feature Request
> Components: Misc. Connectors
> Affects Versions: 8.12.5
> Reporter: Van Halbert
> Assignee: Steven Hawkins
> Fix For: 9.1
>
>
> Spark creates files with no extension, so using the following:
> EXEC Spark.getTextFiles('part-00001') will generate an error when trying any of the following ways: part-*.* or part*.* or part-*
> org.teiid.runtime.client.TeiidClientException: java.lang.RuntimeException: Remote org.teiid.core.TeiidProcessingException: TEIID30504 Spark: File not found part*.*
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
8 years, 5 months
[JBoss JIRA] (TEIID-4296) File translator cannot find a file without an extensioin
by Steven Hawkins (JIRA)
[ https://issues.jboss.org/browse/TEIID-4296?page=com.atlassian.jira.plugin... ]
Steven Hawkins resolved TEIID-4296.
-----------------------------------
Resolution: Done
Changed any \* to be treated like a glob wildcard.
> File translator cannot find a file without an extensioin
> --------------------------------------------------------
>
> Key: TEIID-4296
> URL: https://issues.jboss.org/browse/TEIID-4296
> Project: Teiid
> Issue Type: Feature Request
> Components: Misc. Connectors
> Affects Versions: 8.12.5
> Reporter: Van Halbert
> Fix For: 9.1
>
>
> Spark creates files with no extension, so using the following:
> EXEC Spark.getTextFiles('part-00001') will generate an error when trying any of the following ways: part-*.* or part*.* or part-*
> org.teiid.runtime.client.TeiidClientException: java.lang.RuntimeException: Remote org.teiid.core.TeiidProcessingException: TEIID30504 Spark: File not found part*.*
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
8 years, 5 months
[JBoss JIRA] (TEIID-4283) External Materialization interleaving loads SYSADMIN.loadMatView
by RH Bugzilla Integration (JIRA)
[ https://issues.jboss.org/browse/TEIID-4283?page=com.atlassian.jira.plugin... ]
RH Bugzilla Integration commented on TEIID-4283:
------------------------------------------------
Jan Stastny <jstastny(a)redhat.com> changed the Status of [bug 1348466|https://bugzilla.redhat.com/show_bug.cgi?id=1348466] from ON_QA to VERIFIED
> External Materialization interleaving loads SYSADMIN.loadMatView
> ----------------------------------------------------------------
>
> Key: TEIID-4283
> URL: https://issues.jboss.org/browse/TEIID-4283
> Project: Teiid
> Issue Type: Bug
> Components: Query Engine
> Affects Versions: 8.12.5
> Reporter: Jan Stastny
> Assignee: Steven Hawkins
> Priority: Blocker
> Fix For: 9.1, 8.12.5, 9.0.1
>
> Attachments: jdg-log.txt
>
>
> There is a possibility of two concurrent loads of single materialized view when using SYSADMIN.loadMatView function. I observed the issue while:
> # Waiting for ttl-driven reload and in right time I triggered another load explicitly by invoking the loadMatView function.
> # Invoking SYSADMIN.loadMatView function multiple times in quick succession.
> The view definition:
> {code:sql}
> CREATE VIEW external_long_ttl (
> customer_id integer NOT NULL,
> total_amount integer
> ) OPTIONS (MATERIALIZED 'TRUE', UPDATABLE 'FALSE',
> MATERIALIZED_TABLE 'Source.JSTASTNY.dv_matviews_mat_view',
> "teiid_rel:ALLOW_MATVIEW_MANAGEMENT" 'true',
> "teiid_rel:MATVIEW_STATUS_TABLE" 'Source.JSTASTNY.dv_matviews_statustable',
> "teiid_rel:ON_VDB_START_SCRIPT" 'MERGE INTO dv_matviews_check_table(id,vdb_create) SELECT id, vdb_create+1 FROM dv_matviews_check_table WHERE id=''external_long_ttl'';',
> "teiid_rel:ON_VDB_DROP_SCRIPT" 'MERGE INTO dv_matviews_check_table(id,vdb_drop) SELECT id, vdb_drop+1 FROM dv_matviews_check_table WHERE id=''external_long_ttl'';',
> "teiid_rel:MATVIEW_LOAD_SCRIPT" 'INSERT INTO dv_matviews_mat_view_stage(customer_id,total_amount) SELECT CONVERT(c.id,integer) AS customer_id, CONVERT(SUM(o.amount),integer) AS total_amount FROM dv_matviews_customers c INNER JOIN dv_matviews_orders o ON c.id = o.customer_id GROUP BY c.id;',
> "teiid_rel:MATVIEW_BEFORE_LOAD_SCRIPT" 'exec Source.native(''truncate table ${db.table.prefix}dv_matviews_mat_view_stage'');MERGE INTO dv_matviews_check_table(id,before_load) SELECT id, before_load+1 FROM dv_matviews_check_table WHERE id=''external_long_ttl'';',
> "teiid_rel:MATVIEW_AFTER_LOAD_SCRIPT" 'exec Source.native(''RENAME TABLE ${db.table.prefix}dv_matviews_mat_view_stage TO ${db.table.prefix}dv_matviews_mat_view_temp'');exec Source.native(''RENAME TABLE ${db.table.prefix}dv_matviews_mat_view TO ${db.table.prefix}dv_matviews_mat_view_stage'');exec Source.native(''RENAME TABLE ${db.table.prefix}dv_matviews_mat_view_temp TO ${db.table.prefix}dv_matviews_mat_view'');MERGE INTO dv_matviews_check_table(id,after_load) SELECT id, after_load+1 FROM dv_matviews_check_table WHERE id=''external_long_ttl'';',
> "teiid_rel:MATVIEW_ONERROR_ACTION" 'WAIT',
> "teiid_rel:MATVIEW_TTL" 20000)
> AS SELECT CONVERT(c.id,integer) AS customer_id, CONVERT(SUM(o.amount),integer) AS total_amount FROM dv_matviews_customers c INNER JOIN dv_matviews_orders o ON c.id = o.customer_id GROUP BY c.id;
> {code}
> Query to explicitly load the view:
> {code:sql}
> exec SYSADMIN.loadMatView(schemaName=>'View',viewname=>'external_long_ttl', invalidate=>'true')
> {code}
> I attached a log which starts with ttl-driven load, then the explicit load is performed. The issue noticed is in teiid_rel:MATVIEW_AFTER_LOAD_SCRIPT , where primary and staging tables are being swapped. The second load (explicit) can't find the original table, as the ttl-driven load had already renamed it.
> But this situation of two concurrent loads should never occur.
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
8 years, 5 months
[JBoss JIRA] (TEIID-4213) Cassandra translator: enable pushdown of aggregate functions
by JieRen JieRen (JIRA)
[ https://issues.jboss.org/browse/TEIID-4213?page=com.atlassian.jira.plugin... ]
JieRen JieRen reassigned TEIID-4213:
------------------------------------
Assignee: JieRen JieRen
> Cassandra translator: enable pushdown of aggregate functions
> ------------------------------------------------------------
>
> Key: TEIID-4213
> URL: https://issues.jboss.org/browse/TEIID-4213
> Project: Teiid
> Issue Type: Feature Request
> Components: Misc. Connectors
> Affects Versions: 8.12.x
> Reporter: Jan Stastny
> Assignee: JieRen JieRen
> Priority: Minor
> Fix For: Open To Community, 9.1
>
>
> Cassandra supports basic aggregate functions:
> MIN(),MAX(),SUM() and AVG()
> Currently Teiid doesn't push such queries.
> NOTE:In case of AVG() there is difference between the values returned from Teiid vs. Cassandra for an integer column. Cassandra returns only whole part of the number as integer, Teiid returns bigdecimal.
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
8 years, 5 months
[JBoss JIRA] (TEIID-4297) Add UDF functions based on OSDQ for POC
by JieRen JieRen (JIRA)
[ https://issues.jboss.org/browse/TEIID-4297?page=com.atlassian.jira.plugin... ]
JieRen JieRen resolved TEIID-4297.
----------------------------------
Resolution: Done
> Add UDF functions based on OSDQ for POC
> ---------------------------------------
>
> Key: TEIID-4297
> URL: https://issues.jboss.org/browse/TEIID-4297
> Project: Teiid
> Issue Type: Enhancement
> Components: Query Engine
> Reporter: Ramesh Reddy
> Assignee: JieRen JieRen
> Fix For: 9.1
>
>
> Please add following UDF functions to Teiid library based on maven module
> http://search.maven.org/#search%7Cga%7C1%7Carrahtec
> The UDF functions needs to from this class below
> {code}
> public class Maskutil {
> /**
> * @param a
> * The string that need to randomize
> * vivek singh' will become 'ihg vkeivh'
> */
> public static String toRandomValue(String a) {
> return ShuffleRTM.shuffleString(a);
> }
> /**
> * @param a
> * This function will retrun MD5 hashcode of the string
> * @return String
> */
> public static String toHashValue(String a) {
> if (a == null)
> return "d41d8cd98f00b204e9800998ecf8427e"; // null MD5 value
> return ResultsetToRTM.getMD5(a).toString();
> }
> /**
> * @param a
> * This function will return digit characters of the string
> * @return
> *
> */
> public static String toDigitValue(String a) {
> return StringCaseFormatUtil.digitString(a);
> }
> }
> {code}
> for example Teiid should have UDF functions like
> {code}
> string random(string)
> string hash(string)
> string digit(string)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
8 years, 5 months
[JBoss JIRA] (TEIID-4297) Add UDF functions based on OSDQ for POC
by Ramesh Reddy (JIRA)
[ https://issues.jboss.org/browse/TEIID-4297?page=com.atlassian.jira.plugin... ]
Ramesh Reddy commented on TEIID-4297:
-------------------------------------
No need yet, this is just a POC
> Add UDF functions based on OSDQ for POC
> ---------------------------------------
>
> Key: TEIID-4297
> URL: https://issues.jboss.org/browse/TEIID-4297
> Project: Teiid
> Issue Type: Enhancement
> Components: Query Engine
> Reporter: Ramesh Reddy
> Assignee: JieRen JieRen
> Fix For: 9.1
>
>
> Please add following UDF functions to Teiid library based on maven module
> http://search.maven.org/#search%7Cga%7C1%7Carrahtec
> The UDF functions needs to from this class below
> {code}
> public class Maskutil {
> /**
> * @param a
> * The string that need to randomize
> * vivek singh' will become 'ihg vkeivh'
> */
> public static String toRandomValue(String a) {
> return ShuffleRTM.shuffleString(a);
> }
> /**
> * @param a
> * This function will retrun MD5 hashcode of the string
> * @return String
> */
> public static String toHashValue(String a) {
> if (a == null)
> return "d41d8cd98f00b204e9800998ecf8427e"; // null MD5 value
> return ResultsetToRTM.getMD5(a).toString();
> }
> /**
> * @param a
> * This function will return digit characters of the string
> * @return
> *
> */
> public static String toDigitValue(String a) {
> return StringCaseFormatUtil.digitString(a);
> }
> }
> {code}
> for example Teiid should have UDF functions like
> {code}
> string random(string)
> string hash(string)
> string digit(string)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
8 years, 5 months