[JBoss JIRA] (TEIIDSB-211) create an s3 file source
by Steven Hawkins (Jira)
[ https://issues.redhat.com/browse/TEIIDSB-211?page=com.atlassian.jira.plug... ]
Steven Hawkins commented on TEIIDSB-211:
----------------------------------------
The amazon sdk has examples here: https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/examples-s3.html
The sdk will work against other providers, such as ceph or minio: https://docs.min.io/docs/how-to-use-aws-sdk-for-java-with-minio-server.html
minio is great for testing: https://docs.min.io/docs/minio-docker-quickstart-guide.html
you can also use podman instead of docker: podman run -p 9000:9000 minio/minio server /data
see https://podman.io/ and https://podman.io/getting-started/installation.html
What we'll want is another source that supports the FileConnection/VirtualFile interfaces. There is additional support that can be added later, but that is the initial target. It looks like amazon produces an sdk release nearly every week, so we shouldn't worry too much about the specific version initially.
The source will need details like, bucket, region, access key, encryption support etc. as configuration properties. You can probably just use the property set from the current Teiid S3ExecutionFactory, but also include an endPoint property - which can default to aws.
> create an s3 file source
> -------------------------
>
> Key: TEIIDSB-211
> URL: https://issues.redhat.com/browse/TEIIDSB-211
> Project: Teiid Spring Boot
> Issue Type: Feature Request
> Components: datasource
> Reporter: Steven Hawkins
> Assignee: Steven Hawkins
> Priority: Major
> Fix For: 1.6.0
>
>
> The existing s3 support is implemented as a translator / ws source combo. We instead need this to be just a source to allow for translators to utilize it (excel, parquet, avro).
> As an alternative to our existing support we should evaluate utilizing the sdk rather than providing our own processing logic. More than likely this will be a quicker path to things like s3 select support. However ceph seems to lag in s3 support (see TEIID-5935 and no s3 select support) so we'd have to compensate, make the metadata and/or the capabilities specific to the connection type.
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
5 years, 9 months
[JBoss JIRA] (TEIIDSB-211) create an s3 file source
by Steven Hawkins (Jira)
[ https://issues.redhat.com/browse/TEIIDSB-211?page=com.atlassian.jira.plug... ]
Steven Hawkins reassigned TEIIDSB-211:
--------------------------------------
Assignee: Aditya Manglam Sharma (was: Steven Hawkins)
> create an s3 file source
> -------------------------
>
> Key: TEIIDSB-211
> URL: https://issues.redhat.com/browse/TEIIDSB-211
> Project: Teiid Spring Boot
> Issue Type: Feature Request
> Components: datasource
> Reporter: Steven Hawkins
> Assignee: Aditya Manglam Sharma
> Priority: Major
> Fix For: 1.6.0
>
>
> The existing s3 support is implemented as a translator / ws source combo. We instead need this to be just a source to allow for translators to utilize it (excel, parquet, avro).
> As an alternative to our existing support we should evaluate utilizing the sdk rather than providing our own processing logic. More than likely this will be a quicker path to things like s3 select support. However ceph seems to lag in s3 support (see TEIID-5935 and no s3 select support) so we'd have to compensate, make the metadata and/or the capabilities specific to the connection type.
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
5 years, 9 months
[JBoss JIRA] (TEIID-5972) create an s3 file source
by Steven Hawkins (Jira)
Steven Hawkins created TEIID-5972:
-------------------------------------
Summary: create an s3 file source
Key: TEIID-5972
URL: https://issues.redhat.com/browse/TEIID-5972
Project: Teiid
Issue Type: Feature Request
Components: Misc. Connectors
Reporter: Steven Hawkins
Assignee: Steven Hawkins
Fix For: 15.0
The existing s3 support is implemented as a translator / ws source combo. We instead need this to be just a source to allow for translators to utilize it (excel, parquet, avro).
As an alternative to our existing support we should evaluate utilizing the sdk rather than providing our own processing logic. More than likely this will be a quicker path to things like s3 select support. However ceph seems to lag in s3 support (see TEIID-5935 and no s3 select support) so we'd have to compensate, make the metadata and/or the capabilities specific to the connection type.
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
5 years, 9 months
[JBoss JIRA] (TEIIDSB-211) create an s3 file source
by Steven Hawkins (Jira)
[ https://issues.redhat.com/browse/TEIIDSB-211?page=com.atlassian.jira.plug... ]
Steven Hawkins moved TEIID-5972 to TEIIDSB-211:
-----------------------------------------------
Project: Teiid Spring Boot (was: Teiid)
Key: TEIIDSB-211 (was: TEIID-5972)
Component/s: (was: Misc. Connectors)
Fix Version/s: 1.6.0
(was: 15.0)
> create an s3 file source
> -------------------------
>
> Key: TEIIDSB-211
> URL: https://issues.redhat.com/browse/TEIIDSB-211
> Project: Teiid Spring Boot
> Issue Type: Feature Request
> Reporter: Steven Hawkins
> Assignee: Steven Hawkins
> Priority: Major
> Fix For: 1.6.0
>
>
> The existing s3 support is implemented as a translator / ws source combo. We instead need this to be just a source to allow for translators to utilize it (excel, parquet, avro).
> As an alternative to our existing support we should evaluate utilizing the sdk rather than providing our own processing logic. More than likely this will be a quicker path to things like s3 select support. However ceph seems to lag in s3 support (see TEIID-5935 and no s3 select support) so we'd have to compensate, make the metadata and/or the capabilities specific to the connection type.
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
5 years, 9 months
[JBoss JIRA] (TEIIDSB-211) create an s3 file source
by Steven Hawkins (Jira)
[ https://issues.redhat.com/browse/TEIIDSB-211?page=com.atlassian.jira.plug... ]
Steven Hawkins updated TEIIDSB-211:
-----------------------------------
Component/s: datasource
> create an s3 file source
> -------------------------
>
> Key: TEIIDSB-211
> URL: https://issues.redhat.com/browse/TEIIDSB-211
> Project: Teiid Spring Boot
> Issue Type: Feature Request
> Components: datasource
> Reporter: Steven Hawkins
> Assignee: Steven Hawkins
> Priority: Major
> Fix For: 1.6.0
>
>
> The existing s3 support is implemented as a translator / ws source combo. We instead need this to be just a source to allow for translators to utilize it (excel, parquet, avro).
> As an alternative to our existing support we should evaluate utilizing the sdk rather than providing our own processing logic. More than likely this will be a quicker path to things like s3 select support. However ceph seems to lag in s3 support (see TEIID-5935 and no s3 select support) so we'd have to compensate, make the metadata and/or the capabilities specific to the connection type.
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
5 years, 9 months
[JBoss JIRA] (TEIID-5962) Teiid locks many threads when Salesforce is disconnected
by Steven Hawkins (Jira)
[ https://issues.redhat.com/browse/TEIID-5962?page=com.atlassian.jira.plugi... ]
Steven Hawkins resolved TEIID-5962.
-----------------------------------
Resolution: Partially Completed
This needs recaptured as a teiid spring boot issue.
> Teiid locks many threads when Salesforce is disconnected
> --------------------------------------------------------
>
> Key: TEIID-5962
> URL: https://issues.redhat.com/browse/TEIID-5962
> Project: Teiid
> Issue Type: Bug
> Components: Salesforce Connector
> Affects Versions: 12.3.1
> Reporter: Renat Eskenin
> Assignee: Steven Hawkins
> Priority: Major
> Attachments: Снимок экрана от 2020-05-31 11-09-10.png
>
> Original Estimate: 1 hour
> Time Spent: 45 minutes
> Remaining Estimate: 15 minutes
>
> We had two incidents in our experimental service with teiid spring boot app. When Salesforce is down to 20min and service with teiid trying to call SQL from Salesforce we have many threads in tomcat server (in spring boot app). Then when 200 threads is activated service do not reply to requests because is "max threads" in tomcat of service.
> Why teiid do not send response as "Salesforce is down - error" or connection timeout?
> Micro schema:
> HTTP Requests to REST api -> Teiid spring boot app -> {broken 20min}Salesforce
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
5 years, 9 months
[JBoss JIRA] (TEIID-5971) Hidden connection config
by Steven Hawkins (Jira)
[ https://issues.redhat.com/browse/TEIID-5971?page=com.atlassian.jira.plugi... ]
Steven Hawkins resolved TEIID-5971.
-----------------------------------
Resolution: Out of Date
This has already been addressed. Also please start logging issues under teiid spring boot.
> Hidden connection config
> ------------------------
>
> Key: TEIID-5971
> URL: https://issues.redhat.com/browse/TEIID-5971
> Project: Teiid
> Issue Type: Bug
> Components: Salesforce Connector
> Affects Versions: 14.0
> Reporter: Renat Eskenin
> Assignee: Steven Hawkins
> Priority: Major
>
> In
> SalesforceConnectionImpl:31
> private SalesforceConnectorConfig config;
> In
> BaseSalesforceConnection:77
> protected ConnectorConfig config;
> When call
> BaseSalesforceConnection:82
> we get NPE because
> protected ConnectorConfig config == null;
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
5 years, 9 months
[JBoss JIRA] (TEIID-5971) Hidden connection config
by Renat Eskenin (Jira)
[ https://issues.redhat.com/browse/TEIID-5971?page=com.atlassian.jira.plugi... ]
Renat Eskenin updated TEIID-5971:
---------------------------------
Description:
In
SalesforceConnectionImpl:31
private SalesforceConnectorConfig config;
In
BaseSalesforceConnection:77
protected ConnectorConfig config;
When call
BaseSalesforceConnection:82
we get NPE because
protected ConnectorConfig config == null;
was:
In
SalesforceConnectionImpl:31
private SalesforceConnectorConfig config;
In
BaseSalesforceConnection:77
protected ConnectorConfig config;
When call
BaseSalesforceConnection:82
we get NPE because
protected ConnectorConfig config == null;
> Hidden connection config
> ------------------------
>
> Key: TEIID-5971
> URL: https://issues.redhat.com/browse/TEIID-5971
> Project: Teiid
> Issue Type: Bug
> Components: Salesforce Connector
> Affects Versions: 14.0
> Reporter: Renat Eskenin
> Assignee: Steven Hawkins
> Priority: Major
>
> In
> SalesforceConnectionImpl:31
> private SalesforceConnectorConfig config;
> In
> BaseSalesforceConnection:77
> protected ConnectorConfig config;
> When call
> BaseSalesforceConnection:82
> we get NPE because
> protected ConnectorConfig config == null;
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
5 years, 9 months
[JBoss JIRA] (TEIID-5971) Hidden connection config
by Renat Eskenin (Jira)
[ https://issues.redhat.com/browse/TEIID-5971?page=com.atlassian.jira.plugi... ]
Renat Eskenin updated TEIID-5971:
---------------------------------
Summary: Hidden connection config (was: Hided connection config)
> Hidden connection config
> ------------------------
>
> Key: TEIID-5971
> URL: https://issues.redhat.com/browse/TEIID-5971
> Project: Teiid
> Issue Type: Bug
> Components: Salesforce Connector
> Affects Versions: 14.0
> Reporter: Renat Eskenin
> Assignee: Steven Hawkins
> Priority: Major
>
> In
> SalesforceConnectionImpl:31
> private SalesforceConnectorConfig config;
> In
> BaseSalesforceConnection:77
> protected ConnectorConfig config;
> When call
> BaseSalesforceConnection:82
> we get NPE because
> protected ConnectorConfig config == null;
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
5 years, 9 months
[JBoss JIRA] (TEIID-5971) Hided connection config
by Renat Eskenin (Jira)
[ https://issues.redhat.com/browse/TEIID-5971?page=com.atlassian.jira.plugi... ]
Renat Eskenin updated TEIID-5971:
---------------------------------
Summary: Hided connection config (was: Shadowed connection config)
> Hided connection config
> -----------------------
>
> Key: TEIID-5971
> URL: https://issues.redhat.com/browse/TEIID-5971
> Project: Teiid
> Issue Type: Bug
> Components: Salesforce Connector
> Affects Versions: 14.0
> Reporter: Renat Eskenin
> Assignee: Steven Hawkins
> Priority: Major
>
> In
> SalesforceConnectionImpl:31
> private SalesforceConnectorConfig config;
> In
> BaseSalesforceConnection:77
> protected ConnectorConfig config;
> When call
> BaseSalesforceConnection:82
> we get NPE because
> protected ConnectorConfig config == null;
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
5 years, 9 months