[teiid-issues] [JBoss JIRA] (TEIID-5927) S3 translator can not support minIO endpoint

Wed Apr 8 17:06:55 EDT 2020

    [ https://issues.redhat.com/browse/TEIID-5927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14027832#comment-14027832 ] 

Steven Hawkins commented on TEIID-5927:
---------------------------------------

> We were already very deep with CXF stuff, so did not want to bring Amazon specified ones

The biggest downsides that I see are issues similar to the salesforce dependencies.  There is much more constrained metadata here, but they seem to release a new version twice a week... https://mvnrepository.com/artifact/com.amazonaws/aws-java-sdk

> I did not read up on that "S3 Select" stuff, not sure how much of it is portable across (I do not think it will be)

Can you clarify where that is coming into play?

> Not sure about authentication either.

Both of example links show using the sdk supported authentication with their respective implementations, so it can work - but I'd need to look at things more to see what superset of options may be needed.

> S3 translator can not support minIO endpoint
> --------------------------------------------
>
>                 Key: TEIID-5927
>                 URL: https://issues.redhat.com/browse/TEIID-5927
>             Project: Teiid
>          Issue Type: Feature Request
>          Components: Misc. Connectors
>            Reporter: Rui Wang
>            Assignee: Steven Hawkins
>            Priority: Major
>             Fix For: 14.0
>
>
> When I try to config my on own minIO storage to teiied RA, get following error, seems it tied directly with amazon endpoints, could you pls takes an enhancement to make it generic so it can be used with minio stores. Thanks a lot. 
> <resource-adapter id="webservice">
>                     <module slot="main" id="org.jboss.teiid.resource-adapter.webservice"/>
>                     <connection-definitions>
>                         <connection-definition class-name="org.teiid.resource.adapter.ws.WSManagedConnectionFactory" jndi-name="java:/wsDS" enabled="true" use-java-context="true" pool-name="wsDS">
>                             <config-property name="EndPoint">
>                                 http://myMinioIp:9091/minio/bucket1/
>                             </config-property>
>                         </connection-definition>
>                     </connection-definitions>
>  </resource-adapter>
>  
>  
> Exception in thread "main" org.teiid.jdbc.TeiidSQLException: TEIID30504 Remote org.teiid.core.TeiidProcessingException: TEIID30504 web-connector: Error reading the file https://s3.amazonaws.com/bucket1/price.csv; The response code returned 403. Details: <?xml version="1.0" encoding="UTF-8"?>
> <Error><Code>InvalidAccessKeyId</Code><Message>The AWS Access Key Id you provided does not exist in our records.</Message><AWSAccessKeyId>minioadmin</AWSAccessKeyId><RequestId>EB7421ECD8EF9B88</RequestId><HostId>m3fHokjXC5shrV+y+2vIHl7KNcr46RvoKRXuZ8TnU4fiZyi1XG6rT9DiTu6yRZfg328QGcr2w+A=</HostId></Error>
>     at org.teiid.jdbc.TeiidSQLException.create(TeiidSQLException.java:131)
>     at org.teiid.jdbc.TeiidSQLException.create(TeiidSQLException.java:67)
>     at org.teiid.jdbc.StatementImpl.postReceiveResults(StatementImpl.java:783)
>     at org.teiid.jdbc.StatementImpl.access$100(StatementImpl.java:62)
>     at org.teiid.jdbc.StatementImpl$2.onCompletion(StatementImpl.java:581)
>     at org.teiid.client.util.ResultsFuture.done(ResultsFuture.java:131)
>     at org.teiid.client.util.ResultsFuture.access$200(ResultsFuture.java:36)
>     at org.teiid.client.util.ResultsFuture$1.receiveResults(ResultsFuture.java:75)
>     at org.teiid.net.socket.SocketServerInstanceImpl.receivedMessage(SocketServerInstanceImpl.java:281)
>     at org.teiid.net.socket.SocketServerInstanceImpl.read(SocketServerInstanceImpl.java:333)
>     at org.teiid.net.socket.SocketServerInstanceImpl$RemoteInvocationHandler$1.get(SocketServerInstanceImpl.java:423)
>     at org.teiid.jdbc.StatementImpl.executeSql(StatementImpl.java:590)
>     at org.teiid.jdbc.StatementImpl.executeSql(StatementImpl.java:420)
>     at org.teiid.jdbc.StatementImpl.executeQuery(StatementImpl.java:351)
>     at com.ibm.vdb.TeiidFile.showVdbInfo(TeiidFile.java:181)
>     at com.ibm.vdb.TeiidFile.main(TeiidFile.java:205)
> Caused by: org.teiid.core.TeiidProcessingException: TEIID30504 Remote org.teiid.core.TeiidProcessingException: TEIID30504 web-connector: Error reading the file https://s3.amazonaws.com/bucket1/price.csv; The response code returned 403. Details: <?xml version="1.0" encoding="UTF-8"?>
> <Error><Code>InvalidAccessKeyId</Code><Message>The AWS Access Key Id you provided does not exist in our records.</Message><AWSAccessKeyId>minioadmin</AWSAccessKeyId><RequestId>EB7421ECD8EF9B88</RequestId><HostId>m3fHokjXC5shrV+y+2vIHl7KNcr46RvoKRXuZ8TnU4fiZyi1XG6rT9DiTu6yRZfg328QGcr2w+A=</HostId></Error>
>     at org.teiid.dqp.internal.process.DataTierTupleSource.exceptionOccurred(DataTierTupleSource.java:397)
>     at org.teiid.dqp.internal.process.DataTierTupleSource.nextTuple(DataTierTupleSource.java:157)
>     at org.teiid.query.processor.relational.AccessNode.nextBatchDirect(AccessNode.java:401)
>     at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:277)
>     at org.teiid.query.processor.relational.ProjectNode.nextBatchDirect(ProjectNode.java:146)
>     at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:277)
>     at org.teiid.query.processor.BatchCollector.collectTuples(BatchCollector.java:160)
>     at org.teiid.query.processor.relational.SourceState.prefetch(SourceState.java:207)
>     at org.teiid.query.processor.relational.SourceState.rowCountLE(SourceState.java:154)
>     at org.teiid.query.processor.relational.JoinNode.nextBatchDirectInternal(JoinNode.java:252)
>     at org.teiid.query.processor.relational.JoinNode.nextBatchDirect(JoinNode.java:195)
>     at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:277)
>     at org.teiid.query.processor.relational.ProjectNode.nextBatchDirect(ProjectNode.java:146)
>     at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:277)
>     at org.teiid.query.processor.relational.RelationalPlan.nextBatch(RelationalPlan.java:139)
>     at org.teiid.query.processor.QueryProcessor.nextBatchDirect(QueryProcessor.java:147)
>     at org.teiid.query.processor.QueryProcessor.nextBatch(QueryProcessor.java:110)
>     at org.teiid.query.processor.BatchCollector.collectTuples(BatchCollector.java:160)
>     at org.teiid.query.processor.BatchCollector.collectTuples(BatchCollector.java:142)
>     at org.teiid.dqp.internal.process.RequestWorkItem.processMore(RequestWorkItem.java:492)
>     at org.teiid.dqp.internal.process.RequestWorkItem.process(RequestWorkItem.java:362)
>     at org.teiid.dqp.internal.process.AbstractWorkItem.run(AbstractWorkItem.java:43)
>     at org.teiid.dqp.internal.process.RequestWorkItem.run(RequestWorkItem.java:285)
>     at org.teiid.dqp.internal.process.DQPWorkContext.runInContext(DQPWorkContext.java:281)
>     at org.teiid.dqp.internal.process.ThreadReuseExecutor$RunnableWrapper.run(ThreadReuseExecutor.java:124)
>     at org.teiid.dqp.internal.process.ThreadReuseExecutor$2.run(ThreadReuseExecutor.java:212)
>     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>     at java.lang.Thread.run(Thread.java:748)
> Caused by: org.teiid.core.TeiidException: Remote org.teiid.translator.TranslatorException: Error reading the file https://s3.amazonaws.com/bucket1/price.csv; The response code returned 403. Details: <?xml version="1.0" encoding="UTF-8"?>
> <Error><Code>InvalidAccessKeyId</Code><Message>The AWS Access Key Id you provided does not exist in our records.</Message><AWSAccessKeyId>minioadmin</AWSAccessKeyId><RequestId>EB7421ECD8EF9B88</RequestId><HostId>m3fHokjXC5shrV+y+2vIHl7KNcr46RvoKRXuZ8TnU4fiZyi1XG6rT9DiTu6yRZfg328QGcr2w+A=</HostId></Error>
>     at org.teiid.translator.amazon.s3.S3ProcedureExecution.execute(S3ProcedureExecution.java:114)
>     at org.teiid.dqp.internal.datamgr.ConnectorWorkItem.execute(ConnectorWorkItem.java:402)
>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>     at java.lang.reflect.Method.invoke(Method.java:498)
>     at org.teiid.dqp.internal.datamgr.ConnectorManager$1.invoke(ConnectorManager.java:228)
>     at com.sun.proxy.$Proxy43.execute(Unknown Source)
>     at org.teiid.dqp.internal.process.DataTierTupleSource.getResults(DataTierTupleSource.java:302)
>     at org.teiid.dqp.internal.process.DataTierTupleSource$1.call(DataTierTupleSource.java:108)
>     at org.teiid.dqp.internal.process.DataTierTupleSource$1.call(DataTierTupleSource.java:104)
>     at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>     at org.teiid.dqp.internal.process.FutureWork.run(FutureWork.java:59)
>     ... 6 more

--
This message was sent by Atlassian Jira
(v7.13.8#713008)