[JBoss JIRA] (TEIID-5935) S3 support for ceph
by Steven Hawkins (Jira)
[ https://issues.redhat.com/browse/TEIID-5935?page=com.atlassian.jira.plugi... ]
Steven Hawkins updated TEIID-5935:
----------------------------------
Original Estimate: 4 hours
Remaining Estimate: 4 hours
> S3 support for ceph
> -------------------
>
> Key: TEIID-5935
> URL: https://issues.redhat.com/browse/TEIID-5935
> Project: Teiid
> Issue Type: Feature Request
> Components: Misc. Connectors
> Reporter: Steven Hawkins
> Assignee: Steven Hawkins
> Priority: Major
> Fix For: 14.0, 8.0-tp1-13.1.x
>
> Original Estimate: 4 hours
> Remaining Estimate: 4 hours
>
> We need to validate ceph support against the changes in TEIID-5927. The most immediate issue appears to be that ceph does not support the v2 listing of bucket contents and would need to use an older list method.
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
5 years, 11 months
[JBoss JIRA] (TEIID-5935) S3 support for ceph
by Steven Hawkins (Jira)
[ https://issues.redhat.com/browse/TEIID-5935?page=com.atlassian.jira.plugi... ]
Steven Hawkins resolved TEIID-5935.
-----------------------------------
Fix Version/s: 8.0-tp1-13.1.x
Resolution: Done
> S3 support for ceph
> -------------------
>
> Key: TEIID-5935
> URL: https://issues.redhat.com/browse/TEIID-5935
> Project: Teiid
> Issue Type: Feature Request
> Components: Misc. Connectors
> Reporter: Steven Hawkins
> Assignee: Steven Hawkins
> Priority: Major
> Fix For: 14.0, 8.0-tp1-13.1.x
>
>
> We need to validate ceph support against the changes in TEIID-5927. The most immediate issue appears to be that ceph does not support the v2 listing of bucket contents and would need to use an older list method.
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
5 years, 11 months
[JBoss JIRA] (TEIID-5935) S3 support for ceph
by Steven Hawkins (Jira)
[ https://issues.redhat.com/browse/TEIID-5935?page=com.atlassian.jira.plugi... ]
Steven Hawkins commented on TEIID-5935:
---------------------------------------
Added a listv1 procedure and modified the bucket view to use the listv1 procedure - which is supported by all s3 providers. The usage of v2 with the bucket was incomplete because it exposed the continuation token, but a user could not do anything meaningful with that, so the bucket view is still functionally the same. Any additional efforts will be directed instead to TEIID-5936
> S3 support for ceph
> -------------------
>
> Key: TEIID-5935
> URL: https://issues.redhat.com/browse/TEIID-5935
> Project: Teiid
> Issue Type: Feature Request
> Components: Misc. Connectors
> Reporter: Steven Hawkins
> Assignee: Steven Hawkins
> Priority: Major
> Fix For: 14.0
>
>
> We need to validate ceph support against the changes in TEIID-5927. The most immediate issue appears to be that ceph does not support the v2 listing of bucket contents and would need to use an older list method.
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
5 years, 11 months
[JBoss JIRA] (TEIIDSB-170) Automate materialization to JDG
by Ramesh Reddy (Jira)
[ https://issues.redhat.com/browse/TEIIDSB-170?page=com.atlassian.jira.plug... ]
Ramesh Reddy commented on TEIIDSB-170:
--------------------------------------
>introduce a fuller parser to the operator. Problems still exist as you are doing a lot of contextual parsing here - assumptions about what schema we're creating >things under for example. If we add support for create view schema.name - then that would break this.
Currently, with Regex I did try to capture the context, but yes a full parser of these multiple statements is probably a better way to do this.
>Move the crd creation to Teiid Spring Boot where we are currently implicitly creating the caches. If we don't do this, we'll need to set the translator to not create >if a cache does not exist.
This is more to do with deletion, so IMO this does not help
> Use an explicit cr construct:
Do not like this, IMO the regex has a better chance than this.
> by reusing the MATERIALIZATION_TARGET property
This will work actually, this directly represents the cache name. I am ok with this. This is the most simple one.
> Maybe a table of who creates/destroys the secret, the cluster, and the caches.
| Secret Name | Owner (who creates cluster) | "Create" Key in Secret | On VDB delete | Cluster Shared |
| teiid-cache-store | User | NO|caches removed (not currently) | Yes, across all vdbs and their versions |
| {{vdb}}-cache-store| User | NO| caches removed (not currently) | No, only for the given versions of the VDB |
|{{vdb}}-cache-store| Operator | Yes | Operator removes the cluster | No, only for the given versions of the VDB |
| none | Operator |Yes| cluster removed | No (only for the given versions of the VDB |
* in all instances the Infinispan Operator is available, if not available then feature is turned off and Operator will not generate the materialized model
> Automate materialization to JDG
> -------------------------------
>
> Key: TEIIDSB-170
> URL: https://issues.redhat.com/browse/TEIIDSB-170
> Project: Teiid Spring Boot
> Issue Type: Enhancement
> Components: OpenShift
> Reporter: Steven Hawkins
> Assignee: Ramesh Reddy
> Priority: Major
> Fix For: 1.5.0
>
> Original Estimate: 1 week
> Time Spent: 1 week, 3 days, 2 hours
> Remaining Estimate: 0 minutes
>
> Create an internal materialization replacement needs that is turnkey materialization to JDG (little to no user setup required)
> - the operator may create the infinispan cluster if needed
> - the status table and internal representation of the materialization target would be setup automatically
> For the user this would be as simple marking a view as materialized and then it would be populated in jdg upon deployment. They would not have any concerns with cache naming, status tables, etc.
> For simplicity the initial version would make a similar assumption to the current internal logic - it is for only a specific vdb. If the vdb cr is modified, then it's expected that the cache would be recreated.
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
5 years, 11 months
[JBoss JIRA] (TEIIDSB-170) Automate materialization to JDG
by Ramesh Reddy (Jira)
[ https://issues.redhat.com/browse/TEIIDSB-170?focusedWorklogId=12450953&pa... ]
Ramesh Reddy logged work on TEIIDSB-170:
----------------------------------------
Author: Ramesh Reddy
Created on: 04/May/20 4:15 PM
Start Date: 04/May/20 4:14 PM
Worklog Time Spent: 2 days
Work Description: more work towards reducing the conflict in names and changing the operator to start independent of the materialization tags to use in the resultset caching
Issue Time Tracking
-------------------
Time Spent: 1 week, 3 days, 2 hours (was: 1 week, 1 day, 2 hours)
Worklog Id: (was: 12450953)
> Automate materialization to JDG
> -------------------------------
>
> Key: TEIIDSB-170
> URL: https://issues.redhat.com/browse/TEIIDSB-170
> Project: Teiid Spring Boot
> Issue Type: Enhancement
> Components: OpenShift
> Reporter: Steven Hawkins
> Assignee: Ramesh Reddy
> Priority: Major
> Fix For: 1.5.0
>
> Original Estimate: 1 week
> Time Spent: 1 week, 3 days, 2 hours
> Remaining Estimate: 0 minutes
>
> Create an internal materialization replacement needs that is turnkey materialization to JDG (little to no user setup required)
> - the operator may create the infinispan cluster if needed
> - the status table and internal representation of the materialization target would be setup automatically
> For the user this would be as simple marking a view as materialized and then it would be populated in jdg upon deployment. They would not have any concerns with cache naming, status tables, etc.
> For simplicity the initial version would make a similar assumption to the current internal logic - it is for only a specific vdb. If the vdb cr is modified, then it's expected that the cache would be recreated.
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
5 years, 11 months
[JBoss JIRA] (TEIID-5935) S3 support for ceph
by Steven Hawkins (Jira)
[ https://issues.redhat.com/browse/TEIID-5935?page=com.atlassian.jira.plugi... ]
Steven Hawkins commented on TEIID-5935:
---------------------------------------
Using a local crc instance, rook didn't seem to work for me. No osd pods were ever created. From https://rook.io/docs/rook/v1.3/ceph-prerequisites.html it seems like I'm not providing the suitable storage. Perhaps I need an example of creating a PV with storage class block.
I also attempted to create using rooted podman (rootless failed) with what's done in travis https://github.com/ceph/ceph-container/blob/6340c096ab7f71832625617e069e9... for podman instead of docker. This inculded fs setup using https://github.com/ceph/ceph-container/blob/master/travis-builds/prepare_... However the container would crash not long after starting.
ceph-adm gets further: https://docs.ceph.com/docs/master/cephadm/install/
However running "./cephadm install" fails:
{code}
[shawkins@localhost ceph]$ sudo ./cephadm install
INFO:cephadm:Installing packages ['cephadm']...
INFO:cephadm:Non-zero exit code 1 from yum install -y cephadm
INFO:cephadm:yum:stdout Ceph x86_64 419 B/s | 162 B 00:00
INFO:cephadm:yum:stderr Errors during downloading metadata for repository 'Ceph':
INFO:cephadm:yum:stderr - Status code: 404 for https://download.ceph.com/rpm-octopus/fc31/x86_64/repodata/repomd.xml (IP: 158.69.68.124)
INFO:cephadm:yum:stderr Error: Failed to download metadata for repo 'Ceph': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried
Traceback (most recent call last):
File "./cephadm", line 4579, in <module>
r = args.func()
File "./cephadm", line 4096, in command_install
pkg.install(args.packages)
File "./cephadm", line 3956, in install
call_throws([self.tool, 'install', '-y'] + ls)
File "./cephadm", line 837, in call_throws
raise RuntimeError('Failed command: %s' % ' '.join(command))
RuntimeError: Failed command: yum install -y cephadm
{code}
It is looking for fedora packages, but on the server there is only el7/8.
Ignoring this, I could bring ceph up after making it happy with my hostname for the mon-ip.
However I don't see a way to enable the s3 gateway as ceph-deploy (https://docs.ceph.com/docs/mimic/install/install-ceph-gateway/) is not available on my machine (probably because cephadm install failed) nor in the ceph cli.
> S3 support for ceph
> -------------------
>
> Key: TEIID-5935
> URL: https://issues.redhat.com/browse/TEIID-5935
> Project: Teiid
> Issue Type: Feature Request
> Components: Misc. Connectors
> Reporter: Steven Hawkins
> Assignee: Steven Hawkins
> Priority: Major
> Fix For: 14.0
>
>
> We need to validate ceph support against the changes in TEIID-5927. The most immediate issue appears to be that ceph does not support the v2 listing of bucket contents and would need to use an older list method.
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
5 years, 11 months
[JBoss JIRA] (TEIIDSB-170) Automate materialization to JDG
by Steven Hawkins (Jira)
[ https://issues.redhat.com/browse/TEIIDSB-170?page=com.atlassian.jira.plug... ]
Steven Hawkins commented on TEIIDSB-170:
----------------------------------------
Continued from the pr about updating the regex for parsing comments and new lines:
It's not really about a single comment, it's the complexity that it introduces to these regex expressions - anywhere you can have whitespace you can have a comment - and multi-line comments I believe would be pretty messy to capture.
> need the FQN of the view names to manage the cache when the Infinispan moves to support Cache as CR, that is is the reason for additional parsing. Otherwise, there are no other good options.
Options:
* introduce a fuller parser to the operator. Problems still exist as you are doing a lot of contextual parsing here - assumptions about what schema we're creating things under for example. If we add support for create view schema.name - then that would break this.
* Move the crd creation to Teiid Spring Boot where we are currently implicitly creating the caches. If we don't do this, we'll need to set the translator to not create if a cache does not exist.
* Use an explicit cr construct:
{code}
datagrid:
- viewname1: basecachename
- viewname2 ...
{code}
where the view name would need to be fully qualified and base cache name would be manipulated to account for multiple deployments.
* Or require something similar in DDL - by reusing the MATERIALIZATION_TARGET property (as an unqualfied name) or a new property that indicates the cache name to use. That simplifies the parsing to just that property key/value. This would require some additional logic in the codegen plugin more than likely.
* Have the operator run a Java "vdb service" pod that provides a rest interface for getting vdb information from submitted ddl.
A general thought is that we'll need to default to transactional caches.
I'd also like to have it spelled out the various creation / deletion scenarios we are proposing to support. It looks like so far we have:
teiid-cache-store indicates a cache store to be used by all dv in the namespace
vdb-cache-store indicates a cache store to be used by all deployments of the given vdb
???
Maybe a table of who creates/destroys the secret, the cluster, and the caches.
> Automate materialization to JDG
> -------------------------------
>
> Key: TEIIDSB-170
> URL: https://issues.redhat.com/browse/TEIIDSB-170
> Project: Teiid Spring Boot
> Issue Type: Enhancement
> Components: OpenShift
> Reporter: Steven Hawkins
> Assignee: Ramesh Reddy
> Priority: Major
> Fix For: 1.5.0
>
> Original Estimate: 1 week
> Time Spent: 1 week, 1 day, 2 hours
> Remaining Estimate: 0 minutes
>
> Create an internal materialization replacement needs that is turnkey materialization to JDG (little to no user setup required)
> - the operator may create the infinispan cluster if needed
> - the status table and internal representation of the materialization target would be setup automatically
> For the user this would be as simple marking a view as materialized and then it would be populated in jdg upon deployment. They would not have any concerns with cache naming, status tables, etc.
> For simplicity the initial version would make a similar assumption to the current internal logic - it is for only a specific vdb. If the vdb cr is modified, then it's expected that the cache would be recreated.
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
5 years, 11 months
[JBoss JIRA] (TEIIDSB-170) Automate materialization to JDG
by Ramesh Reddy (Jira)
[ https://issues.redhat.com/browse/TEIIDSB-170?page=com.atlassian.jira.plug... ]
Ramesh Reddy commented on TEIIDSB-170:
--------------------------------------
However, I do see that in rollup deployments, the cache gets new names without collisions but since it reuses the same Infinispan cluster, it does not clear the previous entries. This is an issue can be only solved when Infinispan supports the CR based cache creation and deletion.
> Automate materialization to JDG
> -------------------------------
>
> Key: TEIIDSB-170
> URL: https://issues.redhat.com/browse/TEIIDSB-170
> Project: Teiid Spring Boot
> Issue Type: Enhancement
> Components: OpenShift
> Reporter: Steven Hawkins
> Assignee: Ramesh Reddy
> Priority: Major
> Fix For: 1.5.0
>
> Original Estimate: 1 week
> Time Spent: 1 week, 1 day, 2 hours
> Remaining Estimate: 0 minutes
>
> Create an internal materialization replacement needs that is turnkey materialization to JDG (little to no user setup required)
> - the operator may create the infinispan cluster if needed
> - the status table and internal representation of the materialization target would be setup automatically
> For the user this would be as simple marking a view as materialized and then it would be populated in jdg upon deployment. They would not have any concerns with cache naming, status tables, etc.
> For simplicity the initial version would make a similar assumption to the current internal logic - it is for only a specific vdb. If the vdb cr is modified, then it's expected that the cache would be recreated.
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
5 years, 11 months