[teiid-issues] [JBoss JIRA] (TEIID-4745) Avoid loading the same file over again when using 'org.jboss.teiid.resource-adapter.file'

[Steven Hawkins (JIRA)]

     [ https://issues.jboss.org/browse/TEIID-4745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Steven Hawkins updated TEIID-4745:
----------------------------------
      Component/s: Query Engine
    Fix Version/s: 10.x


Adding into 10.x to recommend a solution related to cdc for file modifications or to investigate a more integrated approach to using hsql text tables.

> Avoid loading the same file over again when using 'org.jboss.teiid.resource-adapter.file'
> -----------------------------------------------------------------------------------------
>
>                 Key: TEIID-4745
>                 URL: https://issues.jboss.org/browse/TEIID-4745
>             Project: Teiid
>          Issue Type: Enhancement
>          Components: Query Engine
>         Environment: * WildFly 10
> * Teiid Server 9.1.2
> * CentOs 7
>            Reporter: Pedro Inácio
>            Assignee: Steven Hawkins
>             Fix For: 10.x
>
>
> When using the file resource adapter _org.jboss.teiid.resource-adapter.file_, it should be possible that the system per se, by the means of a new property in the resource-adapter, or other option in the view definition, or other configuration elsewhere, to not to unnecessary load a file that is already loaded. Theoretically it should be possible, for example, to check the file date or the file checksum, to see if the file is already loaded by _Teiid_. If it is, then there is no need to parse the file again.
> This is important due to, specially, performance improvements. If there are a lot of files to be processed, and these files are big (200 MB or more), the system will be processing over and over again the same files if they are not updated for quite some time. 
> If, for example, the user uses a Materialized View, then the view will be constantly being updated with the same data, which brings another performance penalty.
> *Note*: This could be extended to other resource-adapters with some sort of last data date for example, if the user whant to use this sort of mechanism. For example, if using a database like MySQL as source, then tthe user could define a last update table, and the system could do a check first against this table before tryng to load the data.



--
This message was sent by Atlassian JIRA
(v7.2.3#72005)