[JBoss JIRA] (TEIID-2578) add/remove schema elements
by Steven Hawkins (JIRA)
[ https://issues.jboss.org/browse/TEIID-2578?page=com.atlassian.jira.plugin... ]
Steven Hawkins commented on TEIID-2578:
---------------------------------------
> What about procedures? I could not think of way to wedge those into the SQL/MED's IMPORT statement.
We don't need to have extra built-in support if it's not expected. The JDBC import properties related to procedures can still be used.
> My thinking is if the refresh of metadata is needed then the IMPORT command needs to be executed again, if the specific tables are defined in the IMPORT and target schema already has the tables then a update of the metadata is performed on on those tables, any new ones will be added.
This doesn't sound like how it should work, unless there were some specific MERGE INTO or IMPORT IF NOT EXISTS option. Otherwise I'd expect that the action of the import would need to have the tables dropped ahead of time.
> Some of the metadata loading based on the knowledge that persisted VDB always has all the metadata. This way the IMPORT failures or data source non-availability issues are handled when user is actually performing the task, not at other times like start of the server.
Which is no different that we do today as we default to caching the source metadata after the first deployment. However we allow the user to set on the model/schema if they want it to always reload and via the adminapi allow for metadata cache to be specifically invalidated as part of reload. We still should be able to support similar functionality.
> I consider the usage of NativeMetadataRepository equal to some one running the DDL ad-hoc.
What do you mean?
> Also, I am not expecting anyone to directly edit these files, but if at all these make into any SCM, then that is a possible pattern users might do.
I don't think they should not be dual purposed. Having ownership of the "repository state" makes it clear that we have full control over how the file is changed - and we aren't preserving commenting and/or including additional metadata.
> The way I have it right now is, user will only be able to edit VDB that is created through "CREATE DATABASE" construct, where I have the metadata of VDB available in the DatabaseStore object. Also when one issues an "Alter" I do re-validate to make sure that is allowed. As per session, I wired such that I flush out the state of the session, but keep the connection during the VDB modifications. I thought that covers issue, and gives that seamless continuity. Do you see issues with that?
Yes, I'm saying you cannot keep the session alive in general. Only certain actions would be allowable with in-flight queries, otherwise you'll end up with mismatches between the metadata associated with the query processing and the metadata in the session - and potentially run afoul of permissioning, such as the case I mentioned where the security domain changes.
> We don't recommend anyone editing in production anyway. When VDB is dropped or re-created using "DROP DATABASE/CREATE DATABASE" then sessions behave like they do currently. IMO, AutoFailover=true is fine but, we need to make that default for editing VDBs, that is users expect anyway when they replace the VDBs with same name and version.
I'm not sure what you mean by make it the default - something will be required on the client side to denote that they expect the reconnect behavior, whether it's the failover flag or something else related to connecting to vdb that is modifiable.
> The convert utility is the current "get-schema" method on the Admin API. It is always associated with the running server. I am basing the convert on running VDB metadata, rather than the deployment time resolution of metadata of VDB. Inline with IMPORT comment above.
Sorry I was looking at CovertVDBToDDL, which is just a test utility.
> Yes, this is an issue. IMO we can piggy back on the ALTER grant for this, once we have the connection working over the JDBC I can take a look at this.
> add/remove schema elements
> --------------------------
>
> Key: TEIID-2578
> URL: https://issues.jboss.org/browse/TEIID-2578
> Project: Teiid
> Issue Type: Feature Request
> Components: Query Engine
> Reporter: Ramesh Reddy
> Assignee: Ramesh Reddy
> Fix For: 9.1
>
>
> Schemas are currently static after load. Modifications can only happen with restarts or new versions. We should allow add/drop at runtime.
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
9 years, 6 months
[JBoss JIRA] (TEIID-4421) Teiid Translator per source/version
by Steven Hawkins (JIRA)
[ https://issues.jboss.org/browse/TEIID-4421?page=com.atlassian.jira.plugin... ]
Steven Hawkins resolved TEIID-4421.
-----------------------------------
Fix Version/s: 9.1
Resolution: Done
The change is backwards compatible, but also allow for Embedded to have a notion of override translators. The methods are as follows:
addTranslator(Class) - if the class is annotated as a Translator, then we'll be able to create new instances. Otherwise, we'll assume the old behavior and add a single instance.
addTranslator(String, ExecutionFactory) - will add a single shared instance.
addTranslator(String, String, Map<String, String>) - a new methods that adds an override translator.
So to have the per source/version behavior, your current code would need to use the addTranslator(Class) method.
Also the deprecated addTranslator(ExecutionFactory) method was removed.
> Teiid Translator per source/version
> -----------------------------------
>
> Key: TEIID-4421
> URL: https://issues.jboss.org/browse/TEIID-4421
> Project: Teiid
> Issue Type: Feature Request
> Components: Embedded
> Reporter: Mark Tawk
> Assignee: Steven Hawkins
> Fix For: 9.1
>
>
> Teiid actually uses a single translator to connect to a single source/version. However in the case of translators that auto-configure to the given version it is expected to manage creating new instances when necessary.
> Case reproduced when connecting to 2 DBs, one on SQL Server 2012 and another on SQL Server 2008. The pagination SQL translation of SQL Server 2012 does not work on 2008.
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
9 years, 6 months
[JBoss JIRA] (TEIID-4421) Teiid Translator per source/version
by Steven Hawkins (JIRA)
[ https://issues.jboss.org/browse/TEIID-4421?page=com.atlassian.jira.plugin... ]
Work on TEIID-4421 started by Steven Hawkins.
---------------------------------------------
> Teiid Translator per source/version
> -----------------------------------
>
> Key: TEIID-4421
> URL: https://issues.jboss.org/browse/TEIID-4421
> Project: Teiid
> Issue Type: Feature Request
> Components: Embedded
> Reporter: Mark Tawk
> Assignee: Steven Hawkins
>
> Teiid actually uses a single translator to connect to a single source/version. However in the case of translators that auto-configure to the given version it is expected to manage creating new instances when necessary.
> Case reproduced when connecting to 2 DBs, one on SQL Server 2012 and another on SQL Server 2008. The pagination SQL translation of SQL Server 2012 does not work on 2008.
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
9 years, 6 months
[JBoss JIRA] (TEIID-4427) Migration to Teiid 9.0.0 - Materialized views error: Expected integer, but was bigdecimal
by Steven Hawkins (JIRA)
[ https://issues.jboss.org/browse/TEIID-4427?page=com.atlassian.jira.plugin... ]
Steven Hawkins commented on TEIID-4427:
---------------------------------------
> I'm using Teiid embedded, what is the way to send you the vdb.xml?
If it can't be posted, would it be possible to just provide the parts that are relevant to the oracle source - how is the translator configured in your source and what import options/ddl is being used in the vdb.xml or ModelMetaData.
> Otherwise, for test sake, in teiid 9.0.0 i have set useIntegralTypes=true and changed the data type of the materialized column month(myfield) to Number(9) and the fetch executed successfully.
Good - that does confirm things are working as expected when the types match.
> Still need to debug in teiid 8.11.3 to see what exactly is the difference?
It would be good to know what is the difference yes.
> Migration to Teiid 9.0.0 - Materialized views error: Expected integer, but was bigdecimal
> -----------------------------------------------------------------------------------------
>
> Key: TEIID-4427
> URL: https://issues.jboss.org/browse/TEIID-4427
> Project: Teiid
> Issue Type: Bug
> Components: Query Engine
> Affects Versions: 9.0
> Reporter: Mark Tawk
> Assignee: Steven Hawkins
> Priority: Critical
> Fix For: 9.1, 9.0.4
>
>
> After migrating to Teiid 9.0.0, we are facing issues with existing views materialized on Oracle server.
> when fetching an Oracle materialized view that contains a column using date function like : "Month" or "Year" or "TIMESTAMPDIFF", we are getting the error : Expected integer, but was bigdecimal.
> If we go back to Teiid 8.11.3, the same materialized views fetch without any problem.
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
9 years, 6 months
[JBoss JIRA] (TEIID-4427) Migration to Teiid 9.0.0 - Materialized views error: Expected integer, but was bigdecimal
by Mark Tawk (JIRA)
[ https://issues.jboss.org/browse/TEIID-4427?page=com.atlassian.jira.plugin... ]
Mark Tawk commented on TEIID-4427:
----------------------------------
I'm using Teiid embedded, what is the way to send you the vdb.xml?
Otherwise, for test sake, in teiid 9.0.0 i have set useIntegralTypes=true and changed the data type of the materialized column month(myfield) to Number(9) and the fetch executed successfully.
Still need to debug in teiid 8.11.3 to see what exactly is the difference?
> Migration to Teiid 9.0.0 - Materialized views error: Expected integer, but was bigdecimal
> -----------------------------------------------------------------------------------------
>
> Key: TEIID-4427
> URL: https://issues.jboss.org/browse/TEIID-4427
> Project: Teiid
> Issue Type: Bug
> Components: Query Engine
> Affects Versions: 9.0
> Reporter: Mark Tawk
> Assignee: Steven Hawkins
> Priority: Critical
> Fix For: 9.1, 9.0.4
>
>
> After migrating to Teiid 9.0.0, we are facing issues with existing views materialized on Oracle server.
> when fetching an Oracle materialized view that contains a column using date function like : "Month" or "Year" or "TIMESTAMPDIFF", we are getting the error : Expected integer, but was bigdecimal.
> If we go back to Teiid 8.11.3, the same materialized views fetch without any problem.
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
9 years, 6 months
[JBoss JIRA] (TEIID-2578) add/remove schema elements
by Ramesh Reddy (JIRA)
[ https://issues.jboss.org/browse/TEIID-2578?page=com.atlassian.jira.plugin... ]
Ramesh Reddy commented on TEIID-2578:
-------------------------------------
{quote} - It looks like we need to put some thought into general support of include/exclude tables across all translators. Just doing after the fact may require quite additional repair such dropping foreign keys. {quote}
Yes, that would be good. I can enter a JIRA for this. What about procedures? I could not think of way to wedge those into the SQL/MED's IMPORT statement.
{quote}- Correct me if I'm wrong, but the concept of the exportable DDL form and the internal repository DDL seem to be intermingled. For example we still need to be able to allow a vdb to "refresh" a particular set of imports, so we would need to give the option for export to contain the actual imported metadata or just the equivalent import statements.{quote}
My thinking is if the refresh of metadata is needed then the IMPORT command needs to be executed again, if the specific tables are defined in the IMPORT and target schema already has the tables then a update of the metadata is performed on on those tables, any new ones will be added. Some of the metadata loading based on the knowledge that persisted VDB always has all the metadata. This way the IMPORT failures or data source non-availability issues are handled when user is actually performing the task, not at other times like start of the server. I consider the usage of NativeMetadataRepository equal to some one running the DDL ad-hoc.
{quote}- The storage mechanism is only currently effective for lob style storage - do you envision wanting to shred storage into metadata tables? Also for larger vdbs it seems like we may still want the option of splitting the contents based upon schema.{quote}
Yes, that is true. Since I was starting out with file I kept this to a minimal interface. My current thinking if we go further with relational store persistence (I am also thinking Neo4J may be perfect) then this need to be expanded to finer API such that we are reacting to individual events rather then whole DB. When this is in a file, I did wanted to keep it to a single file, so that all the contents are there, that is one reason I added comments to the persisted file separating the Schema for easier finding. Also, I am not expecting anyone to directly edit these files, but if at all these make into any SCM, then that is a possible pattern users might do.
{quote}- The change to the SYS xml procedure may be better as quoting schema so that if someone was using (quoted) named parameters it would still work.{quote}
sure, I can revert that. I even had "Type" column quoted, opted later to revert that. I have to say Type keyword gave me lot of issues with testcases.
{quote}- We still have to terminate the session if the vdb is reloaded - or do a lot more work to determine if the change is allowable (such as with the existing DDL support to alter view, trigger, properties, etc.). In the simplest case we just need to have the jdbc client use the autoFailover=true connection property. But in the worst case, such as if the security-domain of the vdb is altered, then the login may no longer be valid.{quote}
The way I have it right now is, user will only be able to edit VDB that is created through "CREATE DATABASE" construct, where I have the metadata of VDB available in the DatabaseStore object. Also when one issues an "Alter" I do re-validate to make sure that is allowed. As per session, I wired such that I flush out the state of the session, but keep the connection during the VDB modifications. I thought that covers issue, and gives that seamless continuity. Do you see issues with that?
We don't recommend anyone editing in production anyway. When VDB is dropped or re-created using "DROP DATABASE/CREATE DATABASE" then sessions behave like they do currently. IMO, AutoFailover=true is fine but, we need to make that default for editing VDBs, that is users expect anyway when they replace the VDBs with same name and version.
{quote}- The convert vdb utility will only work for vdbs that contain no vdb imports nor source metadata imports, I'm assuming in those cases you expect the user to use a running server.{quote}
The convert utility is the current "get-schema" method on the Admin API. It is always associated with the running server. I am basing the convert on running VDB metadata, rather than the deployment time resolution of metadata of VDB. Inline with IMPORT comment above.
{quote}As for using this DDL via JDBC/ODBC, it doesn't seem like too much of a stretch - it would however be difficult to allow general DDL in procedures or dynamic sql for example. The simplest mechanics would be to use a connection property to determine if we are in an editing mode, then on a parsing failure we'd run the statement through the DDL logic (we'd obviously know the vdb/version context). The issue would be that create table / procedure would need to be against the "current" schema which wouldn't be saved if we simply terminated the connection. We could just make that part of the client state and implement the Connection.get/setSchema methods - as well as another SET statement override that does the same.{quote}
yes, I would leave DDL out of procedure and dynamic sql for now. Enough issues to tackle. :)
As per the JDBC/ODBC, hmm, I like it. Need to think about this little more. Reworking to make command based may give me a way to implement "undo" which I have been not happy with in current design. If I go that route, I need to put more thought into command pattern design for all DDL commands. Also, since I have been validating the whole VDB each time, that is going to be an issue. If I have the dependency tree for a single table or procedure, then I can put together "selective" validation where I do not need to write the command into store before validating, thus preserving the previous state of the VDB and undo operation. (cloning the entire Database object I am considering as expensive operation, otherwise it would have been easy to do)
{quote}In general we need to check the operations against the role system, just as we do currently for things like alter view. So we may need to have the create vdb operation convey that the vdb is in edit mode, make the user assign data roles, etc.{quote}
Yes, this is an issue. IMO we can piggy back on the ALTER grant for this, once we have the connection working over the JDBC I can take a look at this.
> add/remove schema elements
> --------------------------
>
> Key: TEIID-2578
> URL: https://issues.jboss.org/browse/TEIID-2578
> Project: Teiid
> Issue Type: Feature Request
> Components: Query Engine
> Reporter: Ramesh Reddy
> Assignee: Ramesh Reddy
> Fix For: 9.1
>
>
> Schemas are currently static after load. Modifications can only happen with restarts or new versions. We should allow add/drop at runtime.
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
9 years, 6 months