[JBoss JIRA] (TEIID-4251) Built in support for Postgres DB as materialization target
by Ramesh Reddy (Jira)
[ https://issues.jboss.org/browse/TEIID-4251?page=com.atlassian.jira.plugin... ]
Ramesh Reddy commented on TEIID-4251:
-------------------------------------
> It will need to fully babysit that load - if the delegated pod goes down, then the load needs to be restarted.
Yes, it can effectively do that, it can keep checking for status and if failed it can resubmit the request.
> Both have a very weak notion of periodic invalidation
I thought they were based on the status table times, if yes, Operator can only issue on correct time invalidations. Basically Operator reconciler is tick at specified intervel, what you want to do it upto you.
We really do not need coordination among nodes as if you use cental PG store right? that would leave service account concerns and DDL generation concerns.
> Built in support for Postgres DB as materialization target
> ----------------------------------------------------------
>
> Key: TEIID-4251
> URL: https://issues.jboss.org/browse/TEIID-4251
> Project: Teiid
> Issue Type: Feature Request
> Components: Server
> Reporter: Ramesh Reddy
> Assignee: Steven Hawkins
> Priority: Major
> Fix For: 12.3
>
>
> If Postgres database is available along with install or assumed that it is available, then some of the materialization task can be automated, like
> - Creation of a common STATUS table
> - Creation of the materilization targets (create views on dbms)
> - On load, on undeploy and load scripts for all the materialization views
> We need to device a way this to be pluggable, such that based on success of this, we can provide additional support for other sources.
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
4 years, 10 months
[JBoss JIRA] (TEIID-4251) Built in support for Postgres DB as materialization target
by Steven Hawkins (Jira)
[ https://issues.jboss.org/browse/TEIID-4251?page=com.atlassian.jira.plugin... ]
Steven Hawkins commented on TEIID-4251:
---------------------------------------
> It does not need to do the load logic, it just needs to elect a member to delegate to.
It will need to fully babysit that load - if the delegated pod goes down, then the load needs to be restarted.
> Is that something you can make use of with "existing materialization" approach?
Very abstractly yes. Also consider that the existing logic is based upon:
internal - all jgroups coordination
external - jgroups coordination and atomic status table updates
Both have a very weak notion of periodic invalidation - just based upon a local timer at startup, which means that if the pods come up in a disparate fashion they are all making very different time checks or if a single pod goes down in the middle of a long cycle, it will be off for a while the next time it comes up.
So we'll obviously retain the status table update, but the jgroups coordination for detecting members that drop out or for all of the internal materialization coordination is effectively missing.
> as I gather fdw seems to be more work for not any improved functionality.
It adds guarantees to the the load and status logic, and reduces the amount of ddl we have to generate.
> Another option is since we are saying Teiid controls the PG instance, having a sidecar kind of container to do the same co-ordination work?
Keep in mind that anything that externally kicks off the load requires some sort of service account, which has some of the issues discussed on the other jira. However if it's just calling materialization procedures, then it's a lower level service account as it doesn't need to see/read everything.
> Built in support for Postgres DB as materialization target
> ----------------------------------------------------------
>
> Key: TEIID-4251
> URL: https://issues.jboss.org/browse/TEIID-4251
> Project: Teiid
> Issue Type: Feature Request
> Components: Server
> Reporter: Ramesh Reddy
> Assignee: Steven Hawkins
> Priority: Major
> Fix For: 12.3
>
>
> If Postgres database is available along with install or assumed that it is available, then some of the materialization task can be automated, like
> - Creation of a common STATUS table
> - Creation of the materilization targets (create views on dbms)
> - On load, on undeploy and load scripts for all the materialization views
> We need to device a way this to be pluggable, such that based on success of this, we can provide additional support for other sources.
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
4 years, 10 months
[JBoss JIRA] (TEIID-4251) Built in support for Postgres DB as materialization target
by Ramesh Reddy (Jira)
[ https://issues.jboss.org/browse/TEIID-4251?page=com.atlassian.jira.plugin... ]
Ramesh Reddy edited comment on TEIID-4251 at 6/28/19 9:17 AM:
--------------------------------------------------------------
> The second one is the thorniest issue. More than likely we'd need to centralize the load logic.
We will have an Operator, which can be used to coordinate events centrally. It does not need to do the load logic, it just needs to elect a member to delegate to. Is that something you can make use of with "existing materialization" approach? as I gather fdw seems to be more work for not any improved functionality.
Another option is since we are saying Teiid controls the PG instance, having a sidecar kind of container to do the same co-ordination work?
was (Author: rareddy):
> The second one is the thorniest issue. More than likely we'd need to centralize the load logic.
We will have an Operator, which can be used to coordinate events centrally. It does need to do the load logic, it just needs to elect a member to delegate to. Is that something you can make use of with "existing materialization" approach? as I gather fdw seems to be more work for not any improved functionality.
Another option is since we are saying Teiid controls the PG instance, having a sidecar kind of container to do the same co-ordination work?
> Built in support for Postgres DB as materialization target
> ----------------------------------------------------------
>
> Key: TEIID-4251
> URL: https://issues.jboss.org/browse/TEIID-4251
> Project: Teiid
> Issue Type: Feature Request
> Components: Server
> Reporter: Ramesh Reddy
> Assignee: Steven Hawkins
> Priority: Major
> Fix For: 12.3
>
>
> If Postgres database is available along with install or assumed that it is available, then some of the materialization task can be automated, like
> - Creation of a common STATUS table
> - Creation of the materilization targets (create views on dbms)
> - On load, on undeploy and load scripts for all the materialization views
> We need to device a way this to be pluggable, such that based on success of this, we can provide additional support for other sources.
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
4 years, 10 months
[JBoss JIRA] (TEIID-4251) Built in support for Postgres DB as materialization target
by Ramesh Reddy (Jira)
[ https://issues.jboss.org/browse/TEIID-4251?page=com.atlassian.jira.plugin... ]
Ramesh Reddy commented on TEIID-4251:
-------------------------------------
> The second one is the thorniest issue. More than likely we'd need to centralize the load logic.
We will have an Operator, which can be used to coordinate events centrally. It does need to do the load logic, it just needs to elect a member to delegate to. Is that something you can make use of with "existing materialization" approach? as I gather fdw seems to be more work for not any improved functionality.
Another option is since we are saying Teiid controls the PG instance, having a sidecar kind of container to do the same co-ordination work?
> Built in support for Postgres DB as materialization target
> ----------------------------------------------------------
>
> Key: TEIID-4251
> URL: https://issues.jboss.org/browse/TEIID-4251
> Project: Teiid
> Issue Type: Feature Request
> Components: Server
> Reporter: Ramesh Reddy
> Assignee: Steven Hawkins
> Priority: Major
> Fix For: 12.3
>
>
> If Postgres database is available along with install or assumed that it is available, then some of the materialization task can be automated, like
> - Creation of a common STATUS table
> - Creation of the materilization targets (create views on dbms)
> - On load, on undeploy and load scripts for all the materialization views
> We need to device a way this to be pluggable, such that based on success of this, we can provide additional support for other sources.
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
4 years, 10 months
[JBoss JIRA] (TEIIDSB-101) OData issues with multiple visible schema
by Steven Hawkins (Jira)
[ https://issues.jboss.org/browse/TEIIDSB-101?page=com.atlassian.jira.plugi... ]
Steven Hawkins resolved TEIIDSB-101.
------------------------------------
Resolution: Done
Yes, this should be good now with the update to 12.3.
> OData issues with multiple visible schema
> -----------------------------------------
>
> Key: TEIIDSB-101
> URL: https://issues.jboss.org/browse/TEIIDSB-101
> Project: Teiid Spring Boot
> Issue Type: Quality Risk
> Components: OData
> Reporter: Steven Hawkins
> Assignee: Steven Hawkins
> Priority: Major
> Fix For: 1.2.0
>
>
> The odata logic accommodates a subcontext specifying the schema to access. Other logic however doesn't allow for this:
> * if multiple visible schema are exposed and have a cross reference, then the metadata links will be invalid as they are created based upon the Teiid Wildfly conventions.
> * The keycloak logic exposes /$metadata as unsecured, but any of the schema/$metadata links would require authentication.
>
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
4 years, 10 months
[JBoss JIRA] (TEIIDSB-101) OData issues with multiple visible schema
by Ramesh Reddy (Jira)
[ https://issues.jboss.org/browse/TEIIDSB-101?page=com.atlassian.jira.plugi... ]
Ramesh Reddy commented on TEIIDSB-101:
--------------------------------------
[~shawkins] Can we resolve this?
> OData issues with multiple visible schema
> -----------------------------------------
>
> Key: TEIIDSB-101
> URL: https://issues.jboss.org/browse/TEIIDSB-101
> Project: Teiid Spring Boot
> Issue Type: Quality Risk
> Components: OData
> Reporter: Steven Hawkins
> Assignee: Steven Hawkins
> Priority: Major
> Fix For: 1.2.0
>
>
> The odata logic accommodates a subcontext specifying the schema to access. Other logic however doesn't allow for this:
> * if multiple visible schema are exposed and have a cross reference, then the metadata links will be invalid as they are created based upon the Teiid Wildfly conventions.
> * The keycloak logic exposes /$metadata as unsecured, but any of the schema/$metadata links would require authentication.
>
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
4 years, 10 months