[JBoss JIRA] (TEIID-2584) Add management features to materialization
by Ramesh Reddy (JIRA)
[ https://issues.jboss.org/browse/TEIID-2584?page=com.atlassian.jira.plugin... ]
Ramesh Reddy commented on TEIID-2584:
-------------------------------------
The EJB3 timer in AS7 seems like utility as simple as java.util.Timer, which is fine to use for Teiid purposes. I will just use Teiid's own java.util.Timer and use the "teiid-async" thread pool that is configured to execute the jobs.
> Add management features to materialization
> ------------------------------------------
>
> Key: TEIID-2584
> URL: https://issues.jboss.org/browse/TEIID-2584
> Project: Teiid
> Issue Type: Feature Request
> Reporter: Ramesh Reddy
> Assignee: Ramesh Reddy
> Fix For: 8.5
>
>
> Currently Teiid supports internal and external materialization features. The internal is managed completely by the Teiid query engine along with the infinispan cache.
> External materialization is completely unmanaged and left out to the user to manage it externally. This goals for this feature are unify the materialization logic for internal and external, such that both are managed similarly irrespective of the type of materialization chosen.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 4 months
[JBoss JIRA] (TEIID-2626) If a column type is not set, null values will be written into persisted batches
by Steven Hawkins (JIRA)
[ https://issues.jboss.org/browse/TEIID-2626?page=com.atlassian.jira.plugin... ]
Steven Hawkins resolved TEIID-2626.
-----------------------------------
Resolution: Done
Switched the tests to run with the weak reference cache disabled (to minimize environmental variation at a small performance cost) and added an assertion when checking the column types in the buffermanager.
> If a column type is not set, null values will be written into persisted batches
> -------------------------------------------------------------------------------
>
> Key: TEIID-2626
> URL: https://issues.jboss.org/browse/TEIID-2626
> Project: Teiid
> Issue Type: Bug
> Components: Query Engine
> Affects Versions: 7.5
> Reporter: Steven Hawkins
> Assignee: Steven Hawkins
> Priority: Blocker
> Fix For: 8.4.1, 8.5
>
>
> The buffer manager will choose the null serializer if a column type is not set. This is not expected to happen, but test code can be sloppy which isn't evident if the weak reference cache is working effectively.
> However there is one place in the code that is effected, which is the internally generated partitionId column on any window function expression with a non-empty over clause. This means that on systems with limited memory and or poor performing weak references no results for the window function may be returned.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 4 months
[JBoss JIRA] (TEIID-2626) If a column type is not set, null values will be written into persisted batches
by Steven Hawkins (JIRA)
Steven Hawkins created TEIID-2626:
-------------------------------------
Summary: If a column type is not set, null values will be written into persisted batches
Key: TEIID-2626
URL: https://issues.jboss.org/browse/TEIID-2626
Project: Teiid
Issue Type: Bug
Components: Query Engine
Affects Versions: 7.5
Reporter: Steven Hawkins
Assignee: Steven Hawkins
Fix For: 8.4.1, 8.5
The buffer manager will choose the null serializer if a column type is not set. This is not expected to happen, but test code can be sloppy which isn't evident if the weak reference cache is working effectively.
However there is one place in the code that is effected, which is the internally generated partitionId column on any window function expression with a non-empty over clause. This means that on systems with limited memory and or poor performing weak references no results for the window function may be returned.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 4 months
[JBoss JIRA] (TEIID-2626) If a column type is not set, null values will be written into persisted batches
by Steven Hawkins (JIRA)
[ https://issues.jboss.org/browse/TEIID-2626?page=com.atlassian.jira.plugin... ]
Steven Hawkins updated TEIID-2626:
----------------------------------
Priority: Blocker (was: Major)
> If a column type is not set, null values will be written into persisted batches
> -------------------------------------------------------------------------------
>
> Key: TEIID-2626
> URL: https://issues.jboss.org/browse/TEIID-2626
> Project: Teiid
> Issue Type: Bug
> Components: Query Engine
> Affects Versions: 7.5
> Reporter: Steven Hawkins
> Assignee: Steven Hawkins
> Priority: Blocker
> Fix For: 8.4.1, 8.5
>
>
> The buffer manager will choose the null serializer if a column type is not set. This is not expected to happen, but test code can be sloppy which isn't evident if the weak reference cache is working effectively.
> However there is one place in the code that is effected, which is the internally generated partitionId column on any window function expression with a non-empty over clause. This means that on systems with limited memory and or poor performing weak references no results for the window function may be returned.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 4 months
[JBoss JIRA] (TEIID-2584) Add management features to materialization
by Steven Hawkins (JIRA)
[ https://issues.jboss.org/browse/TEIID-2584?page=com.atlassian.jira.plugin... ]
Steven Hawkins commented on TEIID-2584:
---------------------------------------
> I thought nested transactions were not supported in Teiid procedure language, are they?
They could be but we don't consider them as currently supported.
> If you are saying two separate requests
I'm confused on what you mean by requests (user request?). I'm saying that within the scope of a load procedure if you use a transaction specifically around the test for loading:
{code}
BEGIN
...
declare boolean shouldLoad;
BEGIN ATOMIC
update status set (isLoading = true) WHERE ... and isLoading = false;
shouldLoad = (ROWCOUNT = 1);
END
if (shouldLoad)
...
END
{code}
Then regardless of the isolation level if shouldLoad is true, then you would proceed. This of course glosses over any of the complexity with failed loads or initially populating a status entry.
As you are getting though if this is being run in the scope of a higher level transaction, then there will be an issue.
> but right now with this JIRA
No I think that is what you have to do. Adding the consideration of implicit loading strategies is cumbersome.
> this enables Teiid to keep the cache near close to fresh
Not exactly - it provides a more centralized coordination mechanism and more easily allows for hard time based refreshing (refresh all at midnight, etc.). However the real trick with freshness is based upon change events and incremental updates.
> Add management features to materialization
> ------------------------------------------
>
> Key: TEIID-2584
> URL: https://issues.jboss.org/browse/TEIID-2584
> Project: Teiid
> Issue Type: Feature Request
> Reporter: Ramesh Reddy
> Assignee: Ramesh Reddy
> Fix For: 8.5
>
>
> Currently Teiid supports internal and external materialization features. The internal is managed completely by the Teiid query engine along with the infinispan cache.
> External materialization is completely unmanaged and left out to the user to manage it externally. This goals for this feature are unify the materialization logic for internal and external, such that both are managed similarly irrespective of the type of materialization chosen.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 4 months
[JBoss JIRA] (TEIID-2584) Add management features to materialization
by Ramesh Reddy (JIRA)
[ https://issues.jboss.org/browse/TEIID-2584?page=com.atlassian.jira.plugin... ]
Ramesh Reddy commented on TEIID-2584:
-------------------------------------
hmm.. I thought nested transactions were not supported in Teiid procedure language, are they? If you are saying two separate requests, then I could not see a good way to spin up an additional request based on the request for original materialization view selection request.
About the timer, yes that was my intention to make management of materialization independent of the processing logic both for external and internal matviews, but right now with this JIRA, I am only targeting the external. Also, this enables Teiid to keep the cache near close to fresh, and mostly loading of caches becomes an asynchronous activity. Cache hint can still drive the refresh times. As per the choice of Timer service, I will take look at the EJB Timer service in AS7.
> Add management features to materialization
> ------------------------------------------
>
> Key: TEIID-2584
> URL: https://issues.jboss.org/browse/TEIID-2584
> Project: Teiid
> Issue Type: Feature Request
> Reporter: Ramesh Reddy
> Assignee: Ramesh Reddy
> Fix For: 8.5
>
>
> Currently Teiid supports internal and external materialization features. The internal is managed completely by the Teiid query engine along with the infinispan cache.
> External materialization is completely unmanaged and left out to the user to manage it externally. This goals for this feature are unify the materialization logic for internal and external, such that both are managed similarly irrespective of the type of materialization chosen.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 4 months
[JBoss JIRA] (TEIID-2584) Add management features to materialization
by Steven Hawkins (JIRA)
[ https://issues.jboss.org/browse/TEIID-2584?page=com.atlassian.jira.plugin... ]
Steven Hawkins commented on TEIID-2584:
---------------------------------------
> Can you suggest any alternatives, how I may be able to fix this?
You would make a more granular transaction around a an isloading status flag. Whatever process can read the value as false and write it as true and commit in a txn block should do the writing.
> How about using a job scheduler like http://quartz-scheduler.org/ for this purpose of keeping the caches updated, that will also fix the above issue?
We've tossed quartz around before, but I can't say off hand if it's a good fit. There are also ejb timers in as 7. I think what you are really getting at is removing the maintenance concern from the processing logic. I think that is the right approach with external materialization (internal too if it's re-targeted at something that has a longer lifespan). The processing logic really should just be concerned if the materialization table is in a readable state and have a known outcome if it's not (wait, error, ignore and read anyway, etc.).
> Add management features to materialization
> ------------------------------------------
>
> Key: TEIID-2584
> URL: https://issues.jboss.org/browse/TEIID-2584
> Project: Teiid
> Issue Type: Feature Request
> Reporter: Ramesh Reddy
> Assignee: Ramesh Reddy
> Fix For: 8.5
>
>
> Currently Teiid supports internal and external materialization features. The internal is managed completely by the Teiid query engine along with the infinispan cache.
> External materialization is completely unmanaged and left out to the user to manage it externally. This goals for this feature are unify the materialization logic for internal and external, such that both are managed similarly irrespective of the type of materialization chosen.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 4 months
[JBoss JIRA] (TEIID-2584) Add management features to materialization
by Ramesh Reddy (JIRA)
[ https://issues.jboss.org/browse/TEIID-2584?page=com.atlassian.jira.plugin... ]
Ramesh Reddy commented on TEIID-2584:
-------------------------------------
How about using a job scheduler like http://quartz-scheduler.org/ for this purpose of keeping the caches updated, that will also fix the above issue?
> Add management features to materialization
> ------------------------------------------
>
> Key: TEIID-2584
> URL: https://issues.jboss.org/browse/TEIID-2584
> Project: Teiid
> Issue Type: Feature Request
> Reporter: Ramesh Reddy
> Assignee: Ramesh Reddy
> Fix For: 8.5
>
>
> Currently Teiid supports internal and external materialization features. The internal is managed completely by the Teiid query engine along with the infinispan cache.
> External materialization is completely unmanaged and left out to the user to manage it externally. This goals for this feature are unify the materialization logic for internal and external, such that both are managed similarly irrespective of the type of materialization chosen.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 4 months
[JBoss JIRA] (TEIID-2625) Part of join criteria is getting ignored
by Steven Hawkins (JIRA)
[ https://issues.jboss.org/browse/TEIID-2625?page=com.atlassian.jira.plugin... ]
Steven Hawkins commented on TEIID-2625:
---------------------------------------
You'll need to try this on a more recent release 8.4 or 8.3. Without looking too much I would say that it's probably TEIID-2190.
> Part of join criteria is getting ignored
> ----------------------------------------
>
> Key: TEIID-2625
> URL: https://issues.jboss.org/browse/TEIID-2625
> Project: Teiid
> Issue Type: Bug
> Components: Query Engine
> Affects Versions: 7.7.1
> Environment: z/OS JZOS
> Reporter: Jeff Hayes
> Assignee: Steven Hawkins
> Attachments: 7.7_Teiid_SHOWPLAN
>
>
> The second part of the join criteria (and CHORUS_J0.userid = CHORUS_B.evtuserid) appears to be getting ignored. The result has same results as when only the first criteria (CHORUS_J0.sysid = cx2.sysid) is provided.
> Same query works are 8.1 with query plan listing both criteria in joinNode however for 7.7 only shows first criteria.
>
> Please see attached query plan.
> One thing to note is that the same query on 7.7 works when the databases
> are on DB2 (using DB2 translator) but not when they are on Datacom (different translator). Is this possibly a translator issue maybe not being able to support multiple join criteria?
> Any advice would be appreciated as this is affected a customer installation.
> Thanks!
> Jeff
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 4 months
[JBoss JIRA] (TEIID-2625) Part of join criteria is getting ignored
by Jeff Hayes (JIRA)
[ https://issues.jboss.org/browse/TEIID-2625?page=com.atlassian.jira.plugin... ]
Jeff Hayes updated TEIID-2625:
------------------------------
Description:
The second part of the join criteria (and CHORUS_J0.userid = CHORUS_B.evtuserid) appears to be getting ignored. The result has same results as when only the first criteria (CHORUS_J0.sysid = cx2.sysid) is provided.
Same query works are 8.1 with query plan listing both criteria in joinNode however for 7.7 only shows first criteria.
Please see attached query plan.
One thing to note is that the same query on 7.7 works when the databases
are on DB2 (using DB2 translator) but not when they are on Datacom (different translator). Is this possibly a translator issue maybe not being able to support multiple join criteria?
Any advice would be appreciated as this is affected a customer installation.
Thanks!
Jeff
was:
The second part of the join criteria (and CHORUS_J0.userid = CHORUS_B.evtuserid) appears to be getting ignored. The result has same results as when only the first criteria (CHORUS_J0.sysid = cx2.sysid) is provided.
Same query works are 8.1 with query plan listing both criteria in joinNode however for 7.7 only shows first criteria.
Please see attached query plan.
Is there a fix for this or workaround for the query?
Thanks!
Jeff
> Part of join criteria is getting ignored
> ----------------------------------------
>
> Key: TEIID-2625
> URL: https://issues.jboss.org/browse/TEIID-2625
> Project: Teiid
> Issue Type: Bug
> Components: Query Engine
> Affects Versions: 7.7.1
> Environment: z/OS JZOS
> Reporter: Jeff Hayes
> Assignee: Steven Hawkins
> Attachments: 7.7_Teiid_SHOWPLAN
>
>
> The second part of the join criteria (and CHORUS_J0.userid = CHORUS_B.evtuserid) appears to be getting ignored. The result has same results as when only the first criteria (CHORUS_J0.sysid = cx2.sysid) is provided.
> Same query works are 8.1 with query plan listing both criteria in joinNode however for 7.7 only shows first criteria.
>
> Please see attached query plan.
> One thing to note is that the same query on 7.7 works when the databases
> are on DB2 (using DB2 translator) but not when they are on Datacom (different translator). Is this possibly a translator issue maybe not being able to support multiple join criteria?
> Any advice would be appreciated as this is affected a customer installation.
> Thanks!
> Jeff
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 4 months