[JBoss JIRA] (TEIID-3275) remaining work should always be canceled
by Steven Hawkins (JIRA)
Steven Hawkins created TEIID-3275:
-------------------------------------
Summary: remaining work should always be canceled
Key: TEIID-3275
URL: https://issues.jboss.org/browse/TEIID-3275
Project: Teiid
Issue Type: Quality Risk
Components: JDBC Connector, Query Engine
Affects Versions: 7.0
Reporter: Steven Hawkins
Assignee: Steven Hawkins
Fix For: 8.10
When a result is partially read we typically send a cancel to source queries that remain open for faster cleanup. Depending upon the plan and the usage of max rows this may not happen in all circumstances. We should ensure that cancel is always called on partially read results.
--
This message was sent by Atlassian JIRA
(v6.3.11#6341)
10 years, 2 months
[JBoss JIRA] (TEIID-3268) Dependent values can exceed source settings
by Steven Hawkins (JIRA)
[ https://issues.jboss.org/browse/TEIID-3268?page=com.atlassian.jira.plugin... ]
Steven Hawkins resolved TEIID-3268.
-----------------------------------
Resolution: Done
Ensured that we won't exceed the max settings when there are multiple dependent predicates that do not divide the max evenly.
> Dependent values can exceed source settings
> -------------------------------------------
>
> Key: TEIID-3268
> URL: https://issues.jboss.org/browse/TEIID-3268
> Project: Teiid
> Issue Type: Bug
> Components: Query Engine
> Affects Versions: 8.4
> Reporter: Steven Hawkins
> Assignee: Steven Hawkins
> Fix For: 8.10
>
>
> If the number of dependent predicates does not evenly divide the number of needed predicates when there are more than 1 dependent expression, then we can end up putting more values in the source sql than intended.
--
This message was sent by Atlassian JIRA
(v6.3.11#6341)
10 years, 2 months
[JBoss JIRA] (TEIID-3274) Change archetypes to create independent projects
by Van Halbert (JIRA)
Van Halbert created TEIID-3274:
----------------------------------
Summary: Change archetypes to create independent projects
Key: TEIID-3274
URL: https://issues.jboss.org/browse/TEIID-3274
Project: Teiid
Issue Type: Enhancement
Components: Tools
Affects Versions: 8.7.1
Reporter: Van Halbert
Assignee: Van Halbert
The current archetypes were developed with the mindset of generating projects that are part of the Teiid build. However, they need to be changed so that they are generating projects that are independent, that can stand on their own without requiring Teiid jar's.
--
This message was sent by Atlassian JIRA
(v6.3.11#6341)
10 years, 2 months
[JBoss JIRA] (TEIID-3272) JDBC Translator | Segregation of batch execution and commit mode
by Steven Hawkins (JIRA)
[ https://issues.jboss.org/browse/TEIID-3272?page=com.atlassian.jira.plugin... ]
Steven Hawkins edited comment on TEIID-3272 at 1/5/15 9:53 AM:
---------------------------------------------------------------
> 1. If I have millions of rows being updated, how could I send such a huge update request in one one batch command, expecting executeTranslatedCommand to break it down for me. Since there would be a limit on what we can hold in memory, we would be restricted on this front.
If the update were that large, then the only current mechanism to break it up would be a global transaction. The current paradigm for an execution makes it quite difficult for this to be done with just a source local transaction.
> 2. Secondly, since we are setting the autocommit property for the connection to false, wouldnt it be better, if the control for Commit , is exposed to the caller. In this case, if we have the control over the "autocommit" property for the connection, we could send the batches as we desire and yet control the commit as per requirement.
>From the caller's connection perspective they are in one of:
a global txn (auto commit is implied to be false)
auto commit = false
auto commit = true
In the first two cases Teiid will coordinate the client work under a global transaction - in which case the translator level auto commit behavior does not apply.
Only in auto commit = true does Teiid have more room to determine the transaction handling. At a high level we'll first consult the autoCommitTxn property if it is true we'll start a global transaction. If it is auto, then we'll inspect the plan to see if a transaction appears to be needed to coordinate the work. If no transaction is started, then we'll get to the translator layer where we can choose how to handle the source connection auto commit. The logic assumes then the logical operation should be atomic and will start/commit a local transaction as needed.
was (Author: shawkins):
> 1. If I have millions of rows being updated, how could I send such a huge update request in one one batch command, expecting executeTranslatedCommand to break it down for me. Since there would be a limit on what we can hold in memory, we would be restricted on this front.
If the update were that large, then the only current mechanism to break it up would be a global transaction. The current paradigm for an execution makes it quite difficult for this to be done with just a local transaction.
> 2. Secondly, since we are setting the autocommit property for the connection to false, wouldnt it be better, if the control for Commit , is exposed to the caller. In this case, if we have the control over the "autocommit" property for the connection, we could send the batches as we desire and yet control the commit as per requirement.
>From the caller's connection perspective they are in one of:
a global txn (auto commit is implied to be false)
auto commit = false
auto commit = true
In the first two cases Teiid will coordinate the client work under a global transaction - in which case the translator level auto commit behavior does not apply.
Only in auto commit = true does Teiid have more room to determine the transaction handling. At a high level we'll first consult the autoCommitTxn property if it is true we'll start a global transaction. If it is auto, then we'll inspect the plan to see if a transaction appears to be needed to coordinate the work. If no transaction is started, then we'll get to the translator layer where we can choose how to handle the source connection auto commit. The logic assumes then the logical operation should be atomic and will start/commit a local transaction as needed.
> JDBC Translator | Segregation of batch execution and commit mode
> -----------------------------------------------------------------
>
> Key: TEIID-3272
> URL: https://issues.jboss.org/browse/TEIID-3272
> Project: Teiid
> Issue Type: Enhancement
> Components: JDBC Connector
> Reporter: Shiveeta Mattoo
> Assignee: Steven Hawkins
> Labels: Batch-Commit, JDBC-Transator
> Attachments: batch1.png
>
>
> In JDBC Translator, the batch execution and commit are coupled together. Hence, we after every batch execution, a commit is executed.
> This enhancement is for providing an ability to support the commit after a configurable 'x' no. of batches has completed, instead of an automatic commit after every batch.
> The no of batches after which a commit is desired should be a configurable option.
>
--
This message was sent by Atlassian JIRA
(v6.3.11#6341)
10 years, 2 months
[JBoss JIRA] (TEIID-3272) JDBC Translator | Segregation of batch execution and commit mode
by Steven Hawkins (JIRA)
[ https://issues.jboss.org/browse/TEIID-3272?page=com.atlassian.jira.plugin... ]
Steven Hawkins commented on TEIID-3272:
---------------------------------------
> 1. If I have millions of rows being updated, how could I send such a huge update request in one one batch command, expecting executeTranslatedCommand to break it down for me. Since there would be a limit on what we can hold in memory, we would be restricted on this front.
If the update were that large, then the only current mechanism to break it up would be a global transaction. The current paradigm for an execution makes it quite difficult for this to be done with just a local transaction.
> 2. Secondly, since we are setting the autocommit property for the connection to false, wouldnt it be better, if the control for Commit , is exposed to the caller. In this case, if we have the control over the "autocommit" property for the connection, we could send the batches as we desire and yet control the commit as per requirement.
>From the caller's connection perspective they are in one of:
a global txn (auto commit is implied to be false)
auto commit = false
auto commit = true
In the first two cases Teiid will coordinate the client work under a global transaction - in which case the translator level auto commit behavior does not apply.
Only in auto commit = true does Teiid have more room to determine the transaction handling. At a high level we'll first consult the autoCommitTxn property if it is true we'll start a global transaction. If it is auto, then we'll inspect the plan to see if a transaction appears to be needed to coordinate the work. If no transaction is started, then we'll get to the translator layer where we can choose how to handle the source connection auto commit. The logic assumes then the logical operation should be atomic and will start/commit a local transaction as needed.
> JDBC Translator | Segregation of batch execution and commit mode
> -----------------------------------------------------------------
>
> Key: TEIID-3272
> URL: https://issues.jboss.org/browse/TEIID-3272
> Project: Teiid
> Issue Type: Enhancement
> Components: JDBC Connector
> Reporter: Shiveeta Mattoo
> Assignee: Steven Hawkins
> Labels: Batch-Commit, JDBC-Transator
> Attachments: batch1.png
>
>
> In JDBC Translator, the batch execution and commit are coupled together. Hence, we after every batch execution, a commit is executed.
> This enhancement is for providing an ability to support the commit after a configurable 'x' no. of batches has completed, instead of an automatic commit after every batch.
> The no of batches after which a commit is desired should be a configurable option.
>
--
This message was sent by Atlassian JIRA
(v6.3.11#6341)
10 years, 2 months
[JBoss JIRA] (TEIID-3272) JDBC Translator | Segregation of batch execution and commit mode
by Shiveeta Mattoo (JIRA)
[ https://issues.jboss.org/browse/TEIID-3272?page=com.atlassian.jira.plugin... ]
Shiveeta Mattoo commented on TEIID-3272:
----------------------------------------
Thanks Steven. I understand now, that there's a gap in the way the batches are being sent vs how they are expected to execute by the executeTransalatedCommand method.
However my request still remains due to following reasons -
1. If I have millions of rows being updated, how could I send such a huge update request in one one batch command, expecting executeTranslatedCommand to break it down for me. Since there would be a limit on what we can hold in memory, we would be restricted on this front.
2. Secondly, since we are setting the autocommit property for the connection to false, wouldnt it be better, if the control for Commit , is exposed to the caller. In this case, if we have the control over the "autocommit" property for the connection, we could send the batches as we desire and yet control the commit as per requirement.
> JDBC Translator | Segregation of batch execution and commit mode
> -----------------------------------------------------------------
>
> Key: TEIID-3272
> URL: https://issues.jboss.org/browse/TEIID-3272
> Project: Teiid
> Issue Type: Enhancement
> Components: JDBC Connector
> Reporter: Shiveeta Mattoo
> Assignee: Steven Hawkins
> Labels: Batch-Commit, JDBC-Transator
> Attachments: batch1.png
>
>
> In JDBC Translator, the batch execution and commit are coupled together. Hence, we after every batch execution, a commit is executed.
> This enhancement is for providing an ability to support the commit after a configurable 'x' no. of batches has completed, instead of an automatic commit after every batch.
> The no of batches after which a commit is desired should be a configurable option.
>
--
This message was sent by Atlassian JIRA
(v6.3.11#6341)
10 years, 2 months
[JBoss JIRA] (TEIID-3272) JDBC Translator | Segregation of batch execution and commit mode
by Steven Hawkins (JIRA)
[ https://issues.jboss.org/browse/TEIID-3272?page=com.atlassian.jira.plugin... ]
Steven Hawkins commented on TEIID-3272:
---------------------------------------
> The executed method public void execute() throws TranslatorException, recevies each batch one by one, and delegates processing to
executeTranslatedCommand for each batch which is getting committed by restoreAutoCommit.
That means that your batches are being broken up at a higher level. Each call to execute represents a single logical source query operation. If there were a large bulk insert being executed, then executeTranslatedCommand would further break that up into batches as determined by the max prepared insert batch size. The only way for an operation to span multiple executes is with a global transaction.
What does your user query look like? How many updates are you trying to process, etc.?
> JDBC Translator | Segregation of batch execution and commit mode
> -----------------------------------------------------------------
>
> Key: TEIID-3272
> URL: https://issues.jboss.org/browse/TEIID-3272
> Project: Teiid
> Issue Type: Enhancement
> Components: JDBC Connector
> Reporter: Shiveeta Mattoo
> Assignee: Steven Hawkins
> Labels: Batch-Commit, JDBC-Transator
> Attachments: batch1.png
>
>
> In JDBC Translator, the batch execution and commit are coupled together. Hence, we after every batch execution, a commit is executed.
> This enhancement is for providing an ability to support the commit after a configurable 'x' no. of batches has completed, instead of an automatic commit after every batch.
> The no of batches after which a commit is desired should be a configurable option.
>
--
This message was sent by Atlassian JIRA
(v6.3.11#6341)
10 years, 2 months
[JBoss JIRA] (TEIID-3272) JDBC Translator | Segregation of batch execution and commit mode
by Shiveeta Mattoo (JIRA)
[ https://issues.jboss.org/browse/TEIID-3272?page=com.atlassian.jira.plugin... ]
Shiveeta Mattoo edited comment on TEIID-3272 at 1/5/15 8:46 AM:
----------------------------------------------------------------
The whole intent of the configurable batch commit size is to ensure that although we are capable of using the performance benefit offered by batching the SQL statements, we are able to control the atomicity by configuring the "batch count to commit".
Few scenarios around expected behaviour :
1. Batch size = 100, Batch count to commit = 2, Input record count = 500
If Batch # 3 has any error, then Batch#1,2 will be committed and whole of Batch # 3 will be rolled back, and no further batches processed.
2. Batch size = 100, Batch count to commit = 5, Input record count = 500
If any failure is last batch # 5, none of the records are committed, as all 500 records are logically one atomic transation
3.Batch size = 100, Batch count to commit = 2, Input record count = 500
Provide user an additional -- continueOnError option. If this option is enabled, we will do a Best Effort processing , So only error records are skipped in this case, and everything else is committed.
The actual implementation of this feature would depend on what features the underlying database / source provides for error handling.
was (Author: shiveeta.mattoo):
The whole intent of the configurable batch commit size is to ensure that although we are capable of using the performance benefit offered by batching the SQL statements, we are able to control the atomicity by configuring the "batch count to commit".
Few scenarios around expected behaviour :
1. Batch size = 100, Batch count to commit = 2, Input record count = 500
If Batch # 3 has any error, then Batch#1,2 will be committed and whole of Batch # 3 will be rolled back, and no further batches processed.
2. Batch size = 100, Batch count to commit = 5, Input record count = 500
If any failure is last batch # 5, none of the records are committed, as all 500 records are logically one atomic transation
3.Batch size = 100, Batch count to commit = 2, Input record count = 500
Provide user an additional -- continueOnError option. If this option is enabled we will process the erroneous batch till the erroneous record is reached :
If Batch#3 has any error, a Savepoint is set at end of Batch#2 and further records pf batch# 3 are executed and committed one by one till the erroneous record is reached and then processing continues after the failed batch.
So only error records can be skipped in this case
> JDBC Translator | Segregation of batch execution and commit mode
> -----------------------------------------------------------------
>
> Key: TEIID-3272
> URL: https://issues.jboss.org/browse/TEIID-3272
> Project: Teiid
> Issue Type: Enhancement
> Components: JDBC Connector
> Reporter: Shiveeta Mattoo
> Assignee: Steven Hawkins
> Labels: Batch-Commit, JDBC-Transator
> Attachments: batch1.png
>
>
> In JDBC Translator, the batch execution and commit are coupled together. Hence, we after every batch execution, a commit is executed.
> This enhancement is for providing an ability to support the commit after a configurable 'x' no. of batches has completed, instead of an automatic commit after every batch.
> The no of batches after which a commit is desired should be a configurable option.
>
--
This message was sent by Atlassian JIRA
(v6.3.11#6341)
10 years, 2 months
[JBoss JIRA] (TEIID-3272) JDBC Translator | Segregation of batch execution and commit mode
by Shiveeta Mattoo (JIRA)
[ https://issues.jboss.org/browse/TEIID-3272?page=com.atlassian.jira.plugin... ]
Shiveeta Mattoo edited comment on TEIID-3272 at 1/5/15 8:40 AM:
----------------------------------------------------------------
On debugging the code, I am seeing that the input to : private int[] executeTranslatedCommand(TranslatedCommand translatedComm)
is a single batch of multiple records. Is that an incorrect expectation for the method?
The executed method public void execute() throws TranslatorException, recevies each batch one by one, and delegates processing to
executeTranslatedCommand for each batch which is getting committed by restoreAutoCommit.
As seem in attached screenshot, the batch of 2 records (201,202) is processed by the method in one turn.
was (Author: shiveeta.mattoo):
On debugging the code, I am seeing that the input to : private int[] executeTranslatedCommand(TranslatedCommand translatedComm)
is a single batch of multiple records. Is that an incorrect expectation for the method?
The executed method public void execute() throws TranslatorException, recevies each batch one by one, and delegates processing to
executeTranslatedCommand for each batch which is getting committed by restoreAutoCommit.
> JDBC Translator | Segregation of batch execution and commit mode
> -----------------------------------------------------------------
>
> Key: TEIID-3272
> URL: https://issues.jboss.org/browse/TEIID-3272
> Project: Teiid
> Issue Type: Enhancement
> Components: JDBC Connector
> Reporter: Shiveeta Mattoo
> Assignee: Steven Hawkins
> Labels: Batch-Commit, JDBC-Transator
> Attachments: batch1.png
>
>
> In JDBC Translator, the batch execution and commit are coupled together. Hence, we after every batch execution, a commit is executed.
> This enhancement is for providing an ability to support the commit after a configurable 'x' no. of batches has completed, instead of an automatic commit after every batch.
> The no of batches after which a commit is desired should be a configurable option.
>
--
This message was sent by Atlassian JIRA
(v6.3.11#6341)
10 years, 2 months
[JBoss JIRA] (TEIID-3272) JDBC Translator | Segregation of batch execution and commit mode
by Shiveeta Mattoo (JIRA)
[ https://issues.jboss.org/browse/TEIID-3272?page=com.atlassian.jira.plugin... ]
Shiveeta Mattoo edited comment on TEIID-3272 at 1/5/15 8:41 AM:
----------------------------------------------------------------
On debugging the code, I am seeing that the input to : private int[] executeTranslatedCommand(TranslatedCommand translatedComm)
is a single batch of multiple records. Is that an incorrect expectation for the method?
The executed method public void execute() throws TranslatorException, recevies each batch one by one, and delegates processing to
executeTranslatedCommand for each batch which is getting committed by restoreAutoCommit.
As seem in attached screenshot, the batch of 2 records (201,202) is processed by the method in one turn.
We are seeking feature to commit after many such batches have completed execution
was (Author: shiveeta.mattoo):
On debugging the code, I am seeing that the input to : private int[] executeTranslatedCommand(TranslatedCommand translatedComm)
is a single batch of multiple records. Is that an incorrect expectation for the method?
The executed method public void execute() throws TranslatorException, recevies each batch one by one, and delegates processing to
executeTranslatedCommand for each batch which is getting committed by restoreAutoCommit.
As seem in attached screenshot, the batch of 2 records (201,202) is processed by the method in one turn.
> JDBC Translator | Segregation of batch execution and commit mode
> -----------------------------------------------------------------
>
> Key: TEIID-3272
> URL: https://issues.jboss.org/browse/TEIID-3272
> Project: Teiid
> Issue Type: Enhancement
> Components: JDBC Connector
> Reporter: Shiveeta Mattoo
> Assignee: Steven Hawkins
> Labels: Batch-Commit, JDBC-Transator
> Attachments: batch1.png
>
>
> In JDBC Translator, the batch execution and commit are coupled together. Hence, we after every batch execution, a commit is executed.
> This enhancement is for providing an ability to support the commit after a configurable 'x' no. of batches has completed, instead of an automatic commit after every batch.
> The no of batches after which a commit is desired should be a configurable option.
>
--
This message was sent by Atlassian JIRA
(v6.3.11#6341)
10 years, 2 months