[teiid-issues] [JBoss JIRA] (TEIID-5927) S3 translator can not support minIO endpoint

Wed Apr 15 08:42:14 EDT 2020

    [ https://issues.redhat.com/browse/TEIID-5927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14036414#comment-14036414 ] 

Steven Hawkins commented on TEIID-5927:
---------------------------------------

The changes for minio now seem to work.  

[~rareddy] There is a lot of connectivity details in translator properties - region, access key, and secret key - which are also fully exposed by the procedures.  For the paradigm of having a split between translator and source, this makes environmental migration difficult.  For an explicit vdb with Teiid Spring Boot / crds we can still have an issue as TEIIDSB-136 only applies to implicit vdbs - that is you currently cannot set a translator property anything other than ddl.  If we go down the path of TEIID-5936, then we'll create an s3 source which will better encapsulate things.  We may also consider if / how we want users to update execution properties via a crd / teiid spring boot.

> S3 translator can not support minIO endpoint
> --------------------------------------------
>
>                 Key: TEIID-5927
>                 URL: https://issues.redhat.com/browse/TEIID-5927
>             Project: Teiid
>          Issue Type: Feature Request
>          Components: Misc. Connectors
>            Reporter: Rui Wang
>            Assignee: Steven Hawkins
>            Priority: Major
>             Fix For: 14.0
>
>   Original Estimate: 6 hours
>  Remaining Estimate: 6 hours
>
> When I try to config my on own minIO storage to teiied RA, get following error, seems it tied directly with amazon endpoints, could you pls takes an enhancement to make it generic so it can be used with minio stores. Thanks a lot. 
> <resource-adapter id="webservice">
>                     <module slot="main" id="org.jboss.teiid.resource-adapter.webservice"/>
>                     <connection-definitions>
>                         <connection-definition class-name="org.teiid.resource.adapter.ws.WSManagedConnectionFactory" jndi-name="java:/wsDS" enabled="true" use-java-context="true" pool-name="wsDS">
>                             <config-property name="EndPoint">
>                                 http://myMinioIp:9091/minio/bucket1/
>                             </config-property>
>                         </connection-definition>
>                     </connection-definitions>
>  </resource-adapter>
>  
>  
> Exception in thread "main" org.teiid.jdbc.TeiidSQLException: TEIID30504 Remote org.teiid.core.TeiidProcessingException: TEIID30504 web-connector: Error reading the file https://s3.amazonaws.com/bucket1/price.csv; The response code returned 403. Details: <?xml version="1.0" encoding="UTF-8"?>
> <Error><Code>InvalidAccessKeyId</Code><Message>The AWS Access Key Id you provided does not exist in our records.</Message><AWSAccessKeyId>minioadmin</AWSAccessKeyId><RequestId>EB7421ECD8EF9B88</RequestId><HostId>m3fHokjXC5shrV+y+2vIHl7KNcr46RvoKRXuZ8TnU4fiZyi1XG6rT9DiTu6yRZfg328QGcr2w+A=</HostId></Error>
>     at org.teiid.jdbc.TeiidSQLException.create(TeiidSQLException.java:131)
>     at org.teiid.jdbc.TeiidSQLException.create(TeiidSQLException.java:67)
>     at org.teiid.jdbc.StatementImpl.postReceiveResults(StatementImpl.java:783)
>     at org.teiid.jdbc.StatementImpl.access$100(StatementImpl.java:62)
>     at org.teiid.jdbc.StatementImpl$2.onCompletion(StatementImpl.java:581)
>     at org.teiid.client.util.ResultsFuture.done(ResultsFuture.java:131)
>     at org.teiid.client.util.ResultsFuture.access$200(ResultsFuture.java:36)
>     at org.teiid.client.util.ResultsFuture$1.receiveResults(ResultsFuture.java:75)
>     at org.teiid.net.socket.SocketServerInstanceImpl.receivedMessage(SocketServerInstanceImpl.java:281)
>     at org.teiid.net.socket.SocketServerInstanceImpl.read(SocketServerInstanceImpl.java:333)
>     at org.teiid.net.socket.SocketServerInstanceImpl$RemoteInvocationHandler$1.get(SocketServerInstanceImpl.java:423)
>     at org.teiid.jdbc.StatementImpl.executeSql(StatementImpl.java:590)
>     at org.teiid.jdbc.StatementImpl.executeSql(StatementImpl.java:420)
>     at org.teiid.jdbc.StatementImpl.executeQuery(StatementImpl.java:351)
>     at com.ibm.vdb.TeiidFile.showVdbInfo(TeiidFile.java:181)
>     at com.ibm.vdb.TeiidFile.main(TeiidFile.java:205)
> Caused by: org.teiid.core.TeiidProcessingException: TEIID30504 Remote org.teiid.core.TeiidProcessingException: TEIID30504 web-connector: Error reading the file https://s3.amazonaws.com/bucket1/price.csv; The response code returned 403. Details: <?xml version="1.0" encoding="UTF-8"?>
> <Error><Code>InvalidAccessKeyId</Code><Message>The AWS Access Key Id you provided does not exist in our records.</Message><AWSAccessKeyId>minioadmin</AWSAccessKeyId><RequestId>EB7421ECD8EF9B88</RequestId><HostId>m3fHokjXC5shrV+y+2vIHl7KNcr46RvoKRXuZ8TnU4fiZyi1XG6rT9DiTu6yRZfg328QGcr2w+A=</HostId></Error>
>     at org.teiid.dqp.internal.process.DataTierTupleSource.exceptionOccurred(DataTierTupleSource.java:397)
>     at org.teiid.dqp.internal.process.DataTierTupleSource.nextTuple(DataTierTupleSource.java:157)
>     at org.teiid.query.processor.relational.AccessNode.nextBatchDirect(AccessNode.java:401)
>     at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:277)
>     at org.teiid.query.processor.relational.ProjectNode.nextBatchDirect(ProjectNode.java:146)
>     at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:277)
>     at org.teiid.query.processor.BatchCollector.collectTuples(BatchCollector.java:160)
>     at org.teiid.query.processor.relational.SourceState.prefetch(SourceState.java:207)
>     at org.teiid.query.processor.relational.SourceState.rowCountLE(SourceState.java:154)
>     at org.teiid.query.processor.relational.JoinNode.nextBatchDirectInternal(JoinNode.java:252)
>     at org.teiid.query.processor.relational.JoinNode.nextBatchDirect(JoinNode.java:195)
>     at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:277)
>     at org.teiid.query.processor.relational.ProjectNode.nextBatchDirect(ProjectNode.java:146)
>     at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:277)
>     at org.teiid.query.processor.relational.RelationalPlan.nextBatch(RelationalPlan.java:139)
>     at org.teiid.query.processor.QueryProcessor.nextBatchDirect(QueryProcessor.java:147)
>     at org.teiid.query.processor.QueryProcessor.nextBatch(QueryProcessor.java:110)
>     at org.teiid.query.processor.BatchCollector.collectTuples(BatchCollector.java:160)
>     at org.teiid.query.processor.BatchCollector.collectTuples(BatchCollector.java:142)
>     at org.teiid.dqp.internal.process.RequestWorkItem.processMore(RequestWorkItem.java:492)
>     at org.teiid.dqp.internal.process.RequestWorkItem.process(RequestWorkItem.java:362)
>     at org.teiid.dqp.internal.process.AbstractWorkItem.run(AbstractWorkItem.java:43)
>     at org.teiid.dqp.internal.process.RequestWorkItem.run(RequestWorkItem.java:285)
>     at org.teiid.dqp.internal.process.DQPWorkContext.runInContext(DQPWorkContext.java:281)
>     at org.teiid.dqp.internal.process.ThreadReuseExecutor$RunnableWrapper.run(ThreadReuseExecutor.java:124)
>     at org.teiid.dqp.internal.process.ThreadReuseExecutor$2.run(ThreadReuseExecutor.java:212)
>     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>     at java.lang.Thread.run(Thread.java:748)
> Caused by: org.teiid.core.TeiidException: Remote org.teiid.translator.TranslatorException: Error reading the file https://s3.amazonaws.com/bucket1/price.csv; The response code returned 403. Details: <?xml version="1.0" encoding="UTF-8"?>
> <Error><Code>InvalidAccessKeyId</Code><Message>The AWS Access Key Id you provided does not exist in our records.</Message><AWSAccessKeyId>minioadmin</AWSAccessKeyId><RequestId>EB7421ECD8EF9B88</RequestId><HostId>m3fHokjXC5shrV+y+2vIHl7KNcr46RvoKRXuZ8TnU4fiZyi1XG6rT9DiTu6yRZfg328QGcr2w+A=</HostId></Error>
>     at org.teiid.translator.amazon.s3.S3ProcedureExecution.execute(S3ProcedureExecution.java:114)
>     at org.teiid.dqp.internal.datamgr.ConnectorWorkItem.execute(ConnectorWorkItem.java:402)
>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>     at java.lang.reflect.Method.invoke(Method.java:498)
>     at org.teiid.dqp.internal.datamgr.ConnectorManager$1.invoke(ConnectorManager.java:228)
>     at com.sun.proxy.$Proxy43.execute(Unknown Source)
>     at org.teiid.dqp.internal.process.DataTierTupleSource.getResults(DataTierTupleSource.java:302)
>     at org.teiid.dqp.internal.process.DataTierTupleSource$1.call(DataTierTupleSource.java:108)
>     at org.teiid.dqp.internal.process.DataTierTupleSource$1.call(DataTierTupleSource.java:104)
>     at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>     at org.teiid.dqp.internal.process.FutureWork.run(FutureWork.java:59)
>     ... 6 more

--
This message was sent by Atlassian Jira
(v7.13.8#713008)