[JBoss JIRA] (TEIID-5657) VDB Migration Tool
by Ramesh Reddy (Jira)
[ https://issues.jboss.org/browse/TEIID-5657?page=com.atlassian.jira.plugin... ]
Ramesh Reddy commented on TEIID-5657:
-------------------------------------
I issue to think about multiple DDL file inclusions is with Operator model. We have double check if there a way we can do this or it is anti-pattern to supply file sources.
> VDB Migration Tool
> ------------------
>
> Key: TEIID-5657
> URL: https://issues.jboss.org/browse/TEIID-5657
> Project: Teiid
> Issue Type: Feature Request
> Components: Documentation, Quick Starts, Tooling
> Affects Versions: 12.x
> Reporter: Van Halbert
> Assignee: Van Halbert
> Priority: Major
> Fix For: 12.1
>
>
> A migration tool will be provided to convert customer vdbs into just ddl. More than likely this will be just a single ddl file - which will probably not be acceptable to customers with large vdbs.
> We have this utility already available as just a main method in one of our jars. It is anticipated that a maven repo will be distribution mechanism and minimal documentation will be provided.
> There should be optional validation available for the vdb ddl at build time. This breaks down into 3 parts:
> 1. static syntactic validation, potentially even fully resolving if all metadata is present. This makes sure that basic typos will be caught.
> 2. providing hard errors for things that are completely removed - an error for usage of soap or function models for example.
> 3. providing errors or warning for features that are not yet available - vdb imports, sources that aren't yet supported etc.
> To avoid introducing a new plugin or plugging into fmp it's been suggested that the validation could be run via a generated unit test, which ties into another task which is booster, base project, or last resort an arche type that scaffolds the developer project.
> This could possibly be included in the quickstarts as a means to facilitate its use.
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
7 years, 1 month
[JBoss JIRA] (TEIID-5522) Avoid pushing join to datasource if DS cannot handle 1600+ columns
by Steven Hawkins (Jira)
[ https://issues.jboss.org/browse/TEIID-5522?page=com.atlassian.jira.plugin... ]
Steven Hawkins commented on TEIID-5522:
---------------------------------------
See https://stackoverflow.com/questions/12606842/what-is-the-maximum-number-o... for an explanation of the limitation.
We already have a limit on the number of tables in a join. This will be similar to implement.
> Avoid pushing join to datasource if DS cannot handle 1600+ columns
> -------------------------------------------------------------------
>
> Key: TEIID-5522
> URL: https://issues.jboss.org/browse/TEIID-5522
> Project: Teiid
> Issue Type: Feature Request
> Components: Misc. Connectors, Query Engine
> Reporter: Norbert Funke
> Assignee: Steven Hawkins
> Priority: Major
> Fix For: 12.1
>
>
> Problem: I am trying to create a wide view (~5000 columns), which works across data sources fine JDV. However, when I try to create the view with a join on 2+ table from data source, the optimizer pushes down the join to the source. The current source cannot handle more then ~1600 columns.
> Example: When trying to join Member_DX1 and Member_DX2 at client, JDV pushes the enter code herecombined join to postgres as one getting the too max column error.
> /* TABLE 1 */
> CREATE VIEW Member_DX1 (
> MEMB_BID Integer
> , DX130402000000 Integer
> , DX180608000000 Integer
> , DX20401070000 Integer
> .... /* 1000 more */
> as
> SELECT dx.memb_bid
> , case dx.EPI_1_DX4 when 130402000000 then 1 else 0 END as DX130402000000
> , case dx.EPI_1_DX4 when 180608000000 then 1 else 0 END as DX180608000000
> , case dx.EPI_1_DX4 when 20401070000 then 1 else 0 END as DX20401070000
> ...
> FROM BDR.ENH_EPI_DETAIL dx
> /* TABLE 2 */
> CREATE VIEW Member_DX2 (
> MEMB_BID Integer
> , DX200102010000 Integer
> , DX90125000000 Integer
> , DX160603070000 Integer
> ... /* 1000 more ...
> SELECT dx.memb_bid /* FOREIGN TABLE */
> , case dx.EPI_1_DX4 when 200102010000 then 1 else 0 END as DX200102010000
> , case dx.EPI_1_DX4 when 90125000000 then 1 else 0 END as DX90125000000
> , case dx.EPI_1_DX4 when 160603070000 then 1 else 0 END as DX160603070000
> ...`enter code here`
> FROM BDR.ENH_EPI_DETAIL dx;
> then my query in (e.g. dBeaver) looks like this:
> SELECT * from Member_DX1 dx1
> join Member_DX2 dx2
> on dx1.MEMB_BID = dx2.MEMB_BID
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
7 years, 1 month
[JBoss JIRA] (TEIID-5636) Planning error. Could not find symbol disiz__1.DischargeTime
by Steven Hawkins (Jira)
[ https://issues.jboss.org/browse/TEIID-5636?page=com.atlassian.jira.plugin... ]
Steven Hawkins commented on TEIID-5636:
---------------------------------------
I don't see this issue with the patch applied to 10.2.1 either.
The difference in the query plan comes down to:
SELECT g_0."_PRICETYPORDRE" AS c_0, g_0."_CODEOPERATEUR" AS c_1, g_1."_NUMEROINT" AS c_2 FROM OmegaModel_qgWs.Omega.Cpte.ORDRES AS g_0 LEFT OUTER JOIN OmegaModel_qgWs.Omega.Cpte.OPERAT AS g_1 ON g_0."_CODEOPERATION" = g_1."_CODEOPERATION" ORDER BY c_0
which has missing projected symbol without the patch.
If you can still reproduce the issue with master or one of the other fix branches please let me know.
> Planning error. Could not find symbol disiz__1.DischargeTime
> ------------------------------------------------------------
>
> Key: TEIID-5636
> URL: https://issues.jboss.org/browse/TEIID-5636
> Project: Teiid
> Issue Type: Bug
> Components: Query Engine
> Affects Versions: 10.3.4
> Reporter: Bram Gadeyne
> Assignee: Steven Hawkins
> Priority: Critical
> Fix For: 12.1, 12.0.1, 11.2.3
>
> Attachments: Planning Error 21-02-2019.txt, planning_error.txt
>
>
> Hi,
> We're running into this exception. It happens when adding "LEFT OUTER JOIN izisprod.P_DischargeData". I unfortunitaly have no smaller example. We use this table all the time without any previous errors.
> 2019-01-31 15:32:36,438 ERROR [org.teiid.PROCESSOR] (Worker172_QueryProcessorQueue57002) JesgvCmnKfP5 TEIID30019 Unexpected exception for request JesgvCmnKfP5.187: org.teiid.core.TeiidRuntimeException: Planning error. Could not find symbol: disiz__1.DischargeTime
> at org.teiid.query.processor.relational.RelationalNode.getProjectionIndexes(RelationalNode.java:363)
> at org.teiid.query.processor.relational.JoinNode.initialize(JoinNode.java:129)
> at org.teiid.query.processor.relational.RelationalPlan.connectExternal(RelationalPlan.java:92)
> at org.teiid.query.processor.relational.RelationalPlan.connectExternal(RelationalPlan.java:98)
> at org.teiid.query.processor.relational.RelationalPlan.connectExternal(RelationalPlan.java:98)
> at org.teiid.query.processor.relational.RelationalPlan.initialize(RelationalPlan.java:87)
> at org.teiid.query.processor.QueryProcessor.init(QueryProcessor.java:223)
> at org.teiid.query.processor.QueryProcessor.nextBatchDirect(QueryProcessor.java:135)
> at org.teiid.query.processor.QueryProcessor.nextBatch(QueryProcessor.java:111)
> at org.teiid.query.processor.BatchCollector.collectTuples(BatchCollector.java:160)
> at org.teiid.query.processor.BatchCollector.collectTuples(BatchCollector.java:142)
> at org.teiid.dqp.internal.process.RequestWorkItem.processMore(RequestWorkItem.java:480)
> at org.teiid.dqp.internal.process.RequestWorkItem.process(RequestWorkItem.java:350)
> at org.teiid.dqp.internal.process.AbstractWorkItem.run(AbstractWorkItem.java:47)
> at org.teiid.dqp.internal.process.RequestWorkItem.run(RequestWorkItem.java:276)
> at org.teiid.dqp.internal.process.DQPWorkContext.runInContext(DQPWorkContext.java:277)
> at org.teiid.dqp.internal.process.ThreadReuseExecutor$RunnableWrapper.run(ThreadReuseExecutor.java:115)
> at org.teiid.dqp.internal.process.ThreadReuseExecutor$3.run(ThreadReuseExecutor.java:206)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
7 years, 1 month
[JBoss JIRA] (TEIID-5657) VDB Migration Tool
by Steven Hawkins (Jira)
[ https://issues.jboss.org/browse/TEIID-5657?page=com.atlassian.jira.plugin... ]
Steven Hawkins commented on TEIID-5657:
---------------------------------------
> How do database DBAs handle this? Say if I have PG database 500 tables. I have been thinking FlyDB support, but that is a different issue.
You create separate ddl scripts for given sets of logically related things:
- create schema 1 script
- load data to schema 1 script
- create schema 2 script
- migration schema 1 to next version script
...
It's up to the dba and applications how they divide these things up. We don't have to worry about the migration steps just yet, which is handled by tools like flydb. We should however allow for multiple ddl files - especially for legacy customers. This can be done with an include style statement in the main ddl or by convention in the ddl file names.
> VDB Migration Tool
> ------------------
>
> Key: TEIID-5657
> URL: https://issues.jboss.org/browse/TEIID-5657
> Project: Teiid
> Issue Type: Feature Request
> Components: Documentation, Quick Starts, Tooling
> Affects Versions: 12.x
> Reporter: Van Halbert
> Assignee: Van Halbert
> Priority: Major
> Fix For: 12.1
>
>
> A migration tool will be provided to convert customer vdbs into just ddl. More than likely this will be just a single ddl file - which will probably not be acceptable to customers with large vdbs.
> We have this utility already available as just a main method in one of our jars. It is anticipated that a maven repo will be distribution mechanism and minimal documentation will be provided.
> There should be optional validation available for the vdb ddl at build time. This breaks down into 3 parts:
> 1. static syntactic validation, potentially even fully resolving if all metadata is present. This makes sure that basic typos will be caught.
> 2. providing hard errors for things that are completely removed - an error for usage of soap or function models for example.
> 3. providing errors or warning for features that are not yet available - vdb imports, sources that aren't yet supported etc.
> To avoid introducing a new plugin or plugging into fmp it's been suggested that the validation could be run via a generated unit test, which ties into another task which is booster, base project, or last resort an arche type that scaffolds the developer project.
> This could possibly be included in the quickstarts as a means to facilitate its use.
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
7 years, 1 month
[JBoss JIRA] (TEIID-5657) VDB Migration Tool
by Ramesh Reddy (Jira)
[ https://issues.jboss.org/browse/TEIID-5657?page=com.atlassian.jira.plugin... ]
Ramesh Reddy commented on TEIID-5657:
-------------------------------------
> arbitrary files, and jars,
There will not be going to be any support there, IMO this migration tool needs to limit to schema migration, then rest accomplish through any documents.
> single potentially large ddl file, etc
How do database DBAs handle this? Say if I have PG database 500 tables. I have been thinking FlyDB support, but that is a different issue.
> It doesn't appear yet that tsb is handling override translators, do you want an issue on that?
Sure, log the issue.
> VDB Migration Tool
> ------------------
>
> Key: TEIID-5657
> URL: https://issues.jboss.org/browse/TEIID-5657
> Project: Teiid
> Issue Type: Feature Request
> Components: Documentation, Quick Starts, Tooling
> Affects Versions: 12.x
> Reporter: Van Halbert
> Assignee: Van Halbert
> Priority: Major
> Fix For: 12.1
>
>
> A migration tool will be provided to convert customer vdbs into just ddl. More than likely this will be just a single ddl file - which will probably not be acceptable to customers with large vdbs.
> We have this utility already available as just a main method in one of our jars. It is anticipated that a maven repo will be distribution mechanism and minimal documentation will be provided.
> There should be optional validation available for the vdb ddl at build time. This breaks down into 3 parts:
> 1. static syntactic validation, potentially even fully resolving if all metadata is present. This makes sure that basic typos will be caught.
> 2. providing hard errors for things that are completely removed - an error for usage of soap or function models for example.
> 3. providing errors or warning for features that are not yet available - vdb imports, sources that aren't yet supported etc.
> To avoid introducing a new plugin or plugging into fmp it's been suggested that the validation could be run via a generated unit test, which ties into another task which is booster, base project, or last resort an arche type that scaffolds the developer project.
> This could possibly be included in the quickstarts as a means to facilitate its use.
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
7 years, 1 month