[teiid-issues] [JBoss JIRA] (TEIID-2584) Add management features to materialization

[Ramesh Reddy (JIRA)]

    [ https://issues.jboss.org/browse/TEIID-2584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12798460#comment-12798460 ] 

Ramesh Reddy commented on TEIID-2584:
-------------------------------------

> I think we'll want to introduce a new system table to expose the transformation sql rather than through the properties table. We can use a sysadmin table as to not expose inadvertently expose transformation logic that may be intended to be more private and we can ensure that we aren't colliding with any existing property key.

We can add the "Teiid relational URI" name space before to avoid any name space collisions. I really do not need those properties for this feature. As I was adding the Materialization table, I added others for completeness. From the privacy aspect, should we better off tie into some role to access any system tables in general?

> With the load procedure shouldn't we return a negative status code to indicate loading/loaded rather than 0?
It is returning the number of rows loaded in that "load" call, not the status of the call.

>Should there be exception handling for inserting a status entry? Alternatively could this be done at vdb startup time?

I made the status table "valid" column as not null, since it is doing null check it will never get into "insert" if there is a entry already. But, I see your point, if user did not create table as specified then that could be issue. May be I should change it to one of the PK column. I did not choose VDB startup time to avoid adding another procedure, as I need to do "if" check.

>How would a long load or an unexpectedly failed load be handled - in the former case it seems like it's possible to have concurrent  loads and in the latter it seems like some manual process would be needed to clear the loading status.

The "load" procedure is designed such that, (see begin atomic) if any process is already "loading" the matview table based on "status" table, they simply exit the load procedure. But all schedulers at cluster nodes compete to load. Right now there is no-fairness algorithm as to who gets to load. "failed-load" is same way, once the load has failed, the scheduler should try back in ttl/4 or 1 minute again to load. I was thinking with "begin atomic" concurrent loads are not possible, but I could add version based optimistic locking. I am expecting there should not be any manual clearing, that would not be ideal.

> Should the start/shutdown hooks also be definable via the vdb.xml?
Thought about it, but was not sure if the current model of vdb.xml is good vehicle for it. Also, we need to worry about distinguishing between vdb reloads, and first time loads to expose as general purpose feature. Could be worked as separate issue.

> From above #5 - How does this work in a cluster?
All nodes try to compete to load, but first one who grabs the status gets to load, assuming the status table is common among the nodes.

> What tie-ins are you thinking of exposing for incremental updates?
How about this proposal? Add two additional metadata properties on View
{code}
  teiid_rel:ALLOW_MATVIEW_PARTIAL_LOADS = allow partial loads on the matview table
  teiid_rel:ALLOW_MATVIEW_PARTIAL_LOAD_SCRIPT = script to load partial table
{code}

Also, I will add "LoadCount" (could be double used as version for optimistic locking) to status table. Then modify the load procedure such that, when "LoadCount > 1" and when partial loads are allowed, then instead of executing 

{code}
beforeLoadScript
loadScript
afterLoadScript
{code}

will execute

{code}
partialLoadScript
{code}

Now it is up to user to provide script needed for call back into script.
                
> Add management features to materialization
> ------------------------------------------
>
>                 Key: TEIID-2584
>                 URL: https://issues.jboss.org/browse/TEIID-2584
>             Project: Teiid
>          Issue Type: Feature Request
>            Reporter: Ramesh Reddy
>            Assignee: Ramesh Reddy
>              Labels: Beta3
>             Fix For: 8.5
>
>
> Currently Teiid supports internal and external materialization features. The internal is managed completely by the Teiid query engine along with the infinispan cache. 
> External materialization is completely unmanaged and left out to the user to manage it externally. This goals for this feature are unify the materialization logic for internal and external, such that both are managed similarly irrespective of the type of materialization chosen. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira