[JBoss JIRA] (TEIID-4526) Integrate with Debezium for CDC for maintaining materialized views
by Van Halbert (JIRA)
[ https://issues.jboss.org/browse/TEIID-4526?page=com.atlassian.jira.plugin... ]
Van Halbert commented on TEIID-4526:
------------------------------------
There needs to be a client written (see [1]) to consume all of the events in 1 or more topics. According Randal, right now, every table maps to a separate topic, but they are considering changing that to optionally write all changes in each database to a single topic. This would be easier and require less changes as tables are added/removed, and it would ensure that all change events for a single transaction are adjacent to each other. I suspect Teiid would prefer to use this approach, but it’ll be at least a few weeks before we have that.
[1] http://www.confluent.io/blog/tutorial-getting-started-with-the-new-apache...
> Integrate with Debezium for CDC for maintaining materialized views
> ------------------------------------------------------------------
>
> Key: TEIID-4526
> URL: https://issues.jboss.org/browse/TEIID-4526
> Project: Teiid
> Issue Type: Feature Request
> Components: Server
> Affects Versions: 9.2
> Reporter: Van Halbert
> Assignee: Steven Hawkins
> Priority: Critical
>
> Integrate with Debezium so that Teiid will be able to consume and react to the row-level change events and do something interesting with them, such as update the materialized view(s).
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
7 years, 8 months
[JBoss JIRA] (TEIID-2578) add/remove schema elements
by Ramesh Reddy (JIRA)
[ https://issues.jboss.org/browse/TEIID-2578?page=com.atlassian.jira.plugin... ]
Ramesh Reddy commented on TEIID-2578:
-------------------------------------
On second that I do not think we wnat -vdb.ddl based deployment at all, there not much value at all in that.
>> inverted the role of DatabaseStorage, so that DatabaseStore is primary. Also combined PersistenceProxy and DatabaseStorage.
BTW, I reason behind this was DatabaseStorage is extension point for user to provide additional storages, where as the DatabaseStore is engine controlled, no user extensions here. PersistenceProxy is fine, as I mentioned before, this needs to be expanded for finer grade methods, since I was just doing file based persistence I took block methods for time being.
>> we need to address the potential consistency issues with alter to prevent the entire vdb / repository from becoming unusable.
The way I was trying do this is unless the metadata is valid, it will not make it into DatabaseStore, thus keeping it pristine. You are correct, in order to do it correctly, we need like SNAPSHOT of changes that can be overlaid on DatabaseStore, and validate. I could not think of a cleaver way to do this with out duplicated whole state of DatabaseStore. Needs a rethought here.
>>databasestorage service has some wait cycles. shouldn't it just be able to wait for services?
yeah, the data sources availability could not tie it, on server restart while loading the VDBs from repo, these will get ahead of the system start. This needs to wait until system started then it can do the loading. Then this can avoid creation of data sources if they exist. I think this can be avoided, need to take another look.
> add/remove schema elements
> --------------------------
>
> Key: TEIID-2578
> URL: https://issues.jboss.org/browse/TEIID-2578
> Project: Teiid
> Issue Type: Feature Request
> Components: Query Engine
> Reporter: Ramesh Reddy
> Assignee: Ramesh Reddy
> Fix For: 9.2
>
>
> Schemas are currently static after load. Modifications can only happen with restarts or new versions. We should allow add/drop at runtime.
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
7 years, 8 months
[JBoss JIRA] (TEIID-4519) JDG prevent execution of delete or update on child objects
by RH Bugzilla Integration (JIRA)
[ https://issues.jboss.org/browse/TEIID-4519?page=com.atlassian.jira.plugin... ]
RH Bugzilla Integration commented on TEIID-4519:
------------------------------------------------
jolee(a)redhat.com changed the Status of [bug 1385644|https://bugzilla.redhat.com/show_bug.cgi?id=1385644] from NEW to MODIFIED
> JDG prevent execution of delete or update on child objects
> ----------------------------------------------------------
>
> Key: TEIID-4519
> URL: https://issues.jboss.org/browse/TEIID-4519
> Project: Teiid
> Issue Type: Bug
> Components: Misc. Connectors
> Affects Versions: 8.12.7.6_3
> Reporter: Jan Stastny
> Assignee: Van Halbert
> Priority: Critical
> Fix For: 9.2, 8.12.7.6_3
>
>
> When using JDG and complex pojos, delete query issued against child tables end with error.
> * 1-1 relation:
> When deleting a child object, whole parent object is deleted
> * 1-n relation:
> An exception is thrown:
> {code:plain}
> 13:31:28,926 WARN [org.teiid.CONNECTOR] (Worker7_QueryProcessorQueue91) Connector worker process failed for atomic-request=rKsoeHUfbaAW.21.0.21: org.teiid.translator.TranslatorException: TEIID31111 No such accessible property/method number on class java.util.ArrayList.
> at org.teiid.translator.object.ObjectUpdateExecution.handleDelete(ObjectUpdateExecution.java:388) [translator-object-8.12.6.6_3.jar:8.12.6.6_3]
> at org.teiid.translator.object.ObjectUpdateExecution.executeUpdate(ObjectUpdateExecution.java:137) [translator-object-8.12.6.6_3.jar:8.12.6.6_3]
> at org.teiid.translator.object.ObjectUpdateExecution.execute(ObjectUpdateExecution.java:100) [translator-object-8.12.6.6_3.jar:8.12.6.6_3]
> at org.teiid.dqp.internal.datamgr.ConnectorWorkItem$1.execute(ConnectorWorkItem.java:402) [teiid-engine-8.12.6.6_3-redhat-1.jar:8.12.6.6_3-redhat-1]
> at org.teiid.dqp.internal.datamgr.ConnectorWorkItem.execute(ConnectorWorkItem.java:364) [teiid-engine-8.12.6.6_3-redhat-1.jar:8.12.6.6_3-redhat-1]
> at sun.reflect.GeneratedMethodAccessor211.invoke(Unknown Source) [:1.8.0_102]
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_102]
> at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_102]
> at org.teiid.dqp.internal.datamgr.ConnectorManager$1.invoke(ConnectorManager.java:211) [teiid-engine-8.12.6.6_3-redhat-1.jar:8.12.6.6_3-redhat-1]
> at com.sun.proxy.$Proxy81.execute(Unknown Source)
> at org.teiid.dqp.internal.process.DataTierTupleSource.getResults(DataTierTupleSource.java:306) [teiid-engine-8.12.6.6_3-redhat-1.jar:8.12.6.6_3-redhat-1]
> at org.teiid.dqp.internal.process.DataTierTupleSource$1.call(DataTierTupleSource.java:112) [teiid-engine-8.12.6.6_3-redhat-1.jar:8.12.6.6_3-redhat-1]
> at org.teiid.dqp.internal.process.DataTierTupleSource$1.call(DataTierTupleSource.java:108) [teiid-engine-8.12.6.6_3-redhat-1.jar:8.12.6.6_3-redhat-1]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_102]
> at org.teiid.dqp.internal.process.FutureWork.run(FutureWork.java:65) [teiid-engine-8.12.6.6_3-redhat-1.jar:8.12.6.6_3-redhat-1]
> at org.teiid.dqp.internal.process.DQPWorkContext.runInContext(DQPWorkContext.java:276) [teiid-engine-8.12.6.6_3-redhat-1.jar:8.12.6.6_3-redhat-1]
> at org.teiid.dqp.internal.process.ThreadReuseExecutor$RunnableWrapper.run(ThreadReuseExecutor.java:119) [teiid-engine-8.12.6.6_3-redhat-1.jar:8.12.6.6_3-redhat-1]
> at org.teiid.dqp.internal.process.ThreadReuseExecutor$3.run(ThreadReuseExecutor.java:210) [teiid-engine-8.12.6.6_3-redhat-1.jar:8.12.6.6_3-redhat-1]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_102]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_102]
> at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_102]
> Caused by: javax.script.ScriptException: TEIID31111 No such accessible property/method number on class java.util.ArrayList.
> at org.teiid.translator.object.ObjectScriptEngine$1.eval(ObjectScriptEngine.java:144) [translator-object-8.12.6.6_3.jar:8.12.6.6_3]
> at org.teiid.translator.object.ObjectUpdateExecution.handleDelete(ObjectUpdateExecution.java:372) [translator-object-8.12.6.6_3.jar:8.12.6.6_3]
> ... 20 more
> {code}
> These two cases should be handled. Given the lack of DELETE support, all the DELETE queries on child entities should throw exception instead.
> The DELETE on parent should be enabled though.
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
7 years, 8 months
[JBoss JIRA] (TEIID-2578) add/remove schema elements
by Steven Hawkins (JIRA)
[ https://issues.jboss.org/browse/TEIID-2578?page=com.atlassian.jira.plugin... ]
Steven Hawkins commented on TEIID-2578:
---------------------------------------
> and mark the imported metadata as READONLY and not allow any ALTER or persistent operations.
The side effect of supporting this style of import is that the vdb load becomes dependent upon services or we have to limit the import to only other ddl based vdbs and add logic to handle the load order.
> But if want to go towards the deployment based, we can simply provide *-vdb.ddl based deployer back into this without much hassle, but we would not support ALTERS on that.
That depends upon how far we want to go with these changes. If the effect of deployment was simply to tie the deployment to an equivalent databasestore state, then alter would be supported.
Next changes:
- inverted the role of DatabaseStorage, so that DatabaseStore is primary. Also combined PersistenceProxy and DatabaseStorage.
- implemented an undo/restore of a change if there is a metadata validation issue
- created a dummy command/plan to represent non-statement based ddl
- suppressed the cancel exception for ddl
Concerns:
- we need to address the potential consistency issues with alter to prevent the entire vdb / repository from becoming unusable.
- we may need to address the performance of metadata writes as eventually these files could grow to 10s of megabytes.
- databasestorage service has some wait cycles. shouldn't it just be able to wait for services?
- authorization is not checked for ddl operations
I think the best approach at this point is for me to push my changes to a common branch under teiid and we'll continue to iterate on things from there.
> add/remove schema elements
> --------------------------
>
> Key: TEIID-2578
> URL: https://issues.jboss.org/browse/TEIID-2578
> Project: Teiid
> Issue Type: Feature Request
> Components: Query Engine
> Reporter: Ramesh Reddy
> Assignee: Ramesh Reddy
> Fix For: 9.2
>
>
> Schemas are currently static after load. Modifications can only happen with restarts or new versions. We should allow add/drop at runtime.
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
7 years, 8 months
[JBoss JIRA] (TEIID-2578) add/remove schema elements
by Ramesh Reddy (JIRA)
[ https://issues.jboss.org/browse/TEIID-2578?page=com.atlassian.jira.plugin... ]
Ramesh Reddy commented on TEIID-2578:
-------------------------------------
>> You loose ease of use of metadata reload in the import schema case. That is something we could workaround by adding a high-level operation that drops all the objects of the schema, >>then reruns the import as part of a "restart" operation.
User has the ability to carry out this operation, it is not just tied to the restart/redeploy option. It provides a different way to update, so IMO this is OK as long as there is way to do this.
On VDB import I agree. How about something like
CREATE SCHEMA {name} IDENTIFIED BY DATABASE x VERSION y SCHEMA z;
and mark the imported metadata as READONLY and not allow any ALTER or persistent operations.
>>Not yet, they require a RuntimeVDB acquired from the VDBService, which is not associated with a DDL based VDB.
I know I worked on this, I do not think I ran through all the methods.
I do not think we can have this as complete replacement with other behavior wise, especially when there is no external concept of deployment. But if want to go towards the deployment based, we can simply provide *-vdb.ddl based deployer back into this without much hassle, but we would not support ALTERS on that.
> add/remove schema elements
> --------------------------
>
> Key: TEIID-2578
> URL: https://issues.jboss.org/browse/TEIID-2578
> Project: Teiid
> Issue Type: Feature Request
> Components: Query Engine
> Reporter: Ramesh Reddy
> Assignee: Ramesh Reddy
> Fix For: 9.2
>
>
> Schemas are currently static after load. Modifications can only happen with restarts or new versions. We should allow add/drop at runtime.
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
7 years, 8 months
[JBoss JIRA] (TEIID-2578) add/remove schema elements
by Steven Hawkins (JIRA)
[ https://issues.jboss.org/browse/TEIID-2578?page=com.atlassian.jira.plugin... ]
Steven Hawkins commented on TEIID-2578:
---------------------------------------
> With this we are some what similar to other databases.
To an extent. You loose ease of use of metadata reload in the import schema case. That is something we could workaround by adding a high-level operation that drops all the objects of the schema, then reruns the import as part of a "restart" operation.
However we are fundamentally breaking the expectations of the vdb reuse feature. The reuse here is now only at the time of the import statement - any further changes to the imported vdb require a manual process on the part of the user to incorporate into all importing vdbs. I don't see a simple workaround here since the load operation from the databasestore assumes everything is fully realized and in a valid state. The load will need some work in general as any issue with any vdb load will cause the whole load to be aborted.
> They should work
Not yet, they require a RuntimeVDB acquired from the VDBService, which is not associated with a DDL based VDB.
> I see the benefits of deployment and repository based vdbs, we need to articulate both positions in examples and documents to choose what is better for the user.
I'm a little uncomfortable with that if it means they have different capabilities/behavior.
> add/remove schema elements
> --------------------------
>
> Key: TEIID-2578
> URL: https://issues.jboss.org/browse/TEIID-2578
> Project: Teiid
> Issue Type: Feature Request
> Components: Query Engine
> Reporter: Ramesh Reddy
> Assignee: Ramesh Reddy
> Fix For: 9.2
>
>
> Schemas are currently static after load. Modifications can only happen with restarts or new versions. We should allow add/drop at runtime.
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
7 years, 8 months