[teiid-issues] [JBoss JIRA] (TEIIDSB-211) create an s3 file source

[Steven Hawkins (Jira)]

    [ https://issues.redhat.com/browse/TEIIDSB-211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14151714#comment-14151714 ] 

Steven Hawkins commented on TEIIDSB-211:
----------------------------------------

The amazon sdk has examples here: https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/examples-s3.html
The sdk will work against other providers, such as ceph or minio: https://docs.min.io/docs/how-to-use-aws-sdk-for-java-with-minio-server.html
minio is great for testing: https://docs.min.io/docs/minio-docker-quickstart-guide.html
you can also use podman instead of docker:  podman run -p 9000:9000 minio/minio server /data
see https://podman.io/ and https://podman.io/getting-started/installation.html

What we'll want is another source that supports the FileConnection/VirtualFile interfaces.  There is additional support that can be added later, but that is the initial target.  It looks like amazon produces an sdk release nearly every week, so we shouldn't worry too much about the specific version initially.

The source will need details like, bucket, region, access key, encryption support etc. as configuration properties.  You can probably just use the property set from the current Teiid S3ExecutionFactory, but also include an endPoint property - which can default to aws.


> create an s3 file source 
> -------------------------
>
>                 Key: TEIIDSB-211
>                 URL: https://issues.redhat.com/browse/TEIIDSB-211
>             Project: Teiid Spring Boot
>          Issue Type: Feature Request
>          Components: datasource
>            Reporter: Steven Hawkins
>            Assignee: Steven Hawkins
>            Priority: Major
>             Fix For: 1.6.0
>
>
> The existing s3 support is implemented as a translator / ws source combo.  We instead need this to be just a source to allow for translators to utilize it (excel, parquet, avro).
> As an alternative to our existing support we should evaluate utilizing the sdk rather than providing our own processing logic.  More than likely this will be a quicker path to things like s3 select support.  However ceph seems to lag in s3 support (see TEIID-5935 and no s3 select support) so we'd have to compensate, make the metadata and/or the capabilities specific to the connection type.



--
This message was sent by Atlassian Jira
(v7.13.8#713008)