[teiid-issues] [JBoss JIRA] (TEIID-4526) Integrate with Debezium for CDC for maintaining materialized views

[Steven Hawkins (JIRA)]

    [ https://issues.jboss.org/browse/TEIID-4526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13309464#comment-13309464 ] 

Steven Hawkins commented on TEIID-4526:
---------------------------------------

> the above approach of using updatematViewRow is not correct because its updating the actual row in the mat view table without applying the actual View transformation logic

The considerations here are that the updateMatViewRow is only for internal materialization, and yes it's directly tied to modifying the materialization table.  To create an appropriate change value, you would need to do a select with no cache from view and pull appropriately updated rows, then update each one.  This method largely exists because there was no other way to modify the internal materialization target other than to completely refresh and the change is easy to broadcast.

> The only way to ensure the correct result is to only trigger the update when the "key" can be identified in a change.

Yes, some "trigger" has to understand a base tables relationship to all consuming entities and propagate them accordingly.  As mentioned above this can be done with a full row fresh, or what I've talked about is updating of materialization logic to support lazier row refreshing using an invalidation flag - but it still requires determining what row(s) are affected.

> Am I over thinking this? Is there something simpler?

Unfortunately it's much more complicated.  Our problems with CDC starts with the metadata.  Since we allow for hand editing and only capture a simple quoted name in source, we first have a matching problem from any event source to our source metadata.  Then inferring values directly from the change event can be an issue as the CDC layer is broadcasting raw values, not what you would fetch over jdbc (there can be type issues, or even the consideration of masking and other affects).  From there yes we have to sufficiently understand the view structure to make an appropriate update.  That will not always be a simple changes as aggregates may need recomputed, multiple rows removed, etc.  What we talked about at the face-to-face was at least keeping some notion of how "dirty" a particular materialized view was (number of updates that affect the view vs. number of view rows) and trigger a full refresh above a given threshold.  We also need to determine if there are differing expectations for internal vs. external materialization.


> Integrate with Debezium for CDC for maintaining materialized views
> ------------------------------------------------------------------
>
>                 Key: TEIID-4526
>                 URL: https://issues.jboss.org/browse/TEIID-4526
>             Project: Teiid
>          Issue Type: Feature Request
>          Components: Server
>    Affects Versions: 9.2
>            Reporter: Van Halbert
>            Assignee: Steven Hawkins
>            Priority: Critical
>
> Integrate with Debezium so that Teiid will be able to consume and react to the row-level change events and do something interesting with them, such as update the materialized view(s).



--
This message was sent by Atlassian JIRA
(v6.4.11#64026)