[JBoss JIRA] (TEIID-4780) Restart during materialization load, does not update Materialized views anymore
by Ramesh Reddy (JIRA)
Ramesh Reddy created TEIID-4780:
-----------------------------------
Summary: Restart during materialization load, does not update Materialized views anymore
Key: TEIID-4780
URL: https://issues.jboss.org/browse/TEIID-4780
Project: Teiid
Issue Type: Sub-task
Components: Common
Affects Versions: 9.1.2
Environment: * Wildfly 10
* MySQL 5.6.32
* Teiid Server 9.1.2
Reporter: Pedro Inácio
Assignee: Ramesh Reddy
Fix For: 9.3
Attachments: StatusTableAfterStart.csv, StatusTableAfterStop.csv
Under some circumstances, start and stop the server several times at different points, some Materialized Views are not being updated by the server.
It is observable that in status table some views are no longer updated. In server log are being printed lots of INFO lines.
Waited more than 15 minutes and the server was not capable of recovering.
See the _StatusTableAfterStop.csv_ file to observe the status table content after stopping the server.
See the _StatusTableAfterStart.csv_ file to observe the status table content after approximately 15 minutes after restart.
Check _server.log_ for the Teiid Server log.
See that hundreds of lines similar to the following occur
{panel:title=Server.log}
2017-01-31 13:06:34,597 INFO [org.teiid.COMMAND_LOG] (Teiid Timer0) JfGY9cInAbrb START USER COMMAND: startTime=2017-01-31 13:06:34.597 requestID=JfGY9cInAbrb.0 txID=null sessionID=JfGY9cInAbrb applicationName=internal principal=embedded-async vdbName=CountryServiceListVDB vdbVersion=1 sql=execute SYSADMIN.matViewStatus('NumberingPlan', 'numbering_plan')
{panel}
The server cannot continue to load materialized views.
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
9 years, 1 month
[JBoss JIRA] (TEIID-4775) External materialization, after transaction timeout, LoadState continues to be LOADING
by Ramesh Reddy (JIRA)
[ https://issues.jboss.org/browse/TEIID-4775?page=com.atlassian.jira.plugin... ]
Ramesh Reddy updated TEIID-4775:
--------------------------------
Summary: External materialization, after transaction timeout, LoadState continues to be LOADING (was: When using external materialization, after transaction timeout, LoadState continues to be LOADING)
> External materialization, after transaction timeout, LoadState continues to be LOADING
> --------------------------------------------------------------------------------------
>
> Key: TEIID-4775
> URL: https://issues.jboss.org/browse/TEIID-4775
> Project: Teiid
> Issue Type: Sub-task
> Components: Server
> Affects Versions: 8.7
> Environment: * CentOs 7
> * Teiid 9.1.2
> * Wildfly 10
> Reporter: Pedro Inácio
> Assignee: Ramesh Reddy
> Fix For: 9.3
>
>
> For some unknown reason, sometimes a materialized view takes more time to finish the load/update than the one defined in transaction timeout (300 seconds is the default value, and we have configured 600 seconds to try to solve the problem).
> The transaction reaper will kill the update transaction. The _LoadState_ in _status_ table will be *LOADING*.
> Any subsequent try to update the view will fail, since the system reads the state as being *LOADING* even if the system is not currently updating it.
> Example log next.
> The update took too long (more than 10 minutes [defined transaction timeout]):
> {code:batch}
> 2017-02-17 16:10:44,487 INFO [org.teiid.COMMAND_LOG] (NIO6) gZueFR0aNjgi START USER COMMAND: startTime=2017-02-17 16:10:44.487 requestID=gZueFR0aNjgi.11 txID=null sessionID=gZueFR0aNjgi applicationName=JDBC principal=manel vdbName=CountryServiceListVDB vdbVersion=1 sql={call SYSADMIN.loadMatView(?, ?)}
> 2017-02-17 16:20:44,528 WARN [com.arjuna.ats.arjuna] (Transaction Reaper) ARJUNA012117: TransactionReaper::check timeout for TX 0:ffff0a080174:21ddbdf0:58a71a32:cb in state RUN
> 2017-02-17 16:20:44,564 WARN [com.arjuna.ats.arjuna] (Transaction Reaper Worker 0) ARJUNA012121: TransactionReaper::doCancellations worker Thread[Transaction Reaper Worker 0,5,main] successfully canceled TX 0:ffff0a080174:21ddbdf0:58a71a32:cb
> {code}
> Next I tried to update the view:
> {code:batch}
> 2017-02-17 16:29:21,272 INFO [org.teiid.COMMAND_LOG] (NIO2) E1xyKpChGKvC START USER COMMAND: startTime=2017-02-17 16:29:21.272 requestID=E1xyKpChGKvC.7 txID=null sessionID=E1xyKpChGKvC applicationName=JDBC principal=manel vdbName=CountryServiceListVDB vdbVersion=1 sql=EXECUTE SYSADMIN.loadMatView('CountryServiceListCalculatedRaw','country_service_list_calculated_raw')
> 2017-02-17 16:29:21,391 INFO [org.teiid.COMMAND_LOG] (Worker25_QueryProcessorQueue18136) E1xyKpChGKvC END USER COMMAND: endTime=2017-02-17 16:29:21.391 requestID=E1xyKpChGKvC.7 txID=null sessionID=E1xyKpChGKvC principal=manel vdbName=CountryServiceListVDB vdbVersion=1 finalRowCount=1
> {code}
> And nothing happened.
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
9 years, 1 month
[JBoss JIRA] (TEIID-4758) Permanent materialization load failure is when JDG source goes down
by Ramesh Reddy (JIRA)
[ https://issues.jboss.org/browse/TEIID-4758?page=com.atlassian.jira.plugin... ]
Ramesh Reddy updated TEIID-4758:
--------------------------------
Summary: Permanent materialization load failure is when JDG source goes down (was: Permanent materialization load failure is when target source goes down)
> Permanent materialization load failure is when JDG source goes down
> -------------------------------------------------------------------
>
> Key: TEIID-4758
> URL: https://issues.jboss.org/browse/TEIID-4758
> Project: Teiid
> Issue Type: Sub-task
> Components: Server
> Affects Versions: 8.12
> Reporter: Ramesh Reddy
> Assignee: Van Halbert
> Fix For: 9.3
>
>
> During the external materialization load, if the target cache database goes offline, the materialization job stops, but the {{Status}} table is left in {{LOADING}} state, which will never recover when the target cache database comes back up again.
> This situation is observed when JDG is used in OpenShift along with JDV However, behavior can occur in standalone situations too. The system should resilient and must recover in this situation.
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
9 years, 1 month
[JBoss JIRA] (TEIID-4758) Permanent materialization load failure is when target source goes down
by Ramesh Reddy (JIRA)
[ https://issues.jboss.org/browse/TEIID-4758?page=com.atlassian.jira.plugin... ]
Ramesh Reddy reassigned TEIID-4758:
-----------------------------------
Assignee: Van Halbert (was: Ramesh Reddy)
> Permanent materialization load failure is when target source goes down
> ----------------------------------------------------------------------
>
> Key: TEIID-4758
> URL: https://issues.jboss.org/browse/TEIID-4758
> Project: Teiid
> Issue Type: Sub-task
> Components: Server
> Affects Versions: 8.12
> Reporter: Ramesh Reddy
> Assignee: Van Halbert
> Fix For: 9.3
>
>
> During the external materialization load, if the target cache database goes offline, the materialization job stops, but the {{Status}} table is left in {{LOADING}} state, which will never recover when the target cache database comes back up again.
> This situation is observed when JDG is used in OpenShift along with JDV However, behavior can occur in standalone situations too. The system should resilient and must recover in this situation.
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
9 years, 1 month
[JBoss JIRA] (TEIID-4758) Permanent materialization load failure is when target source goes down
by Ramesh Reddy (JIRA)
[ https://issues.jboss.org/browse/TEIID-4758?page=com.atlassian.jira.plugin... ]
Ramesh Reddy commented on TEIID-4758:
-------------------------------------
This turns out to be close enough to TEIID-4775, for the fix for the server side. However this error was logged against JDG, here the issue is JDG throws a RuntimeException instead of throwing a TranslatorException as it should when a connection is not found.
So, I am going, to convert this JIRA to fix the JDG issue, assigning to [~rhn-engineering-vhalbert]
> Permanent materialization load failure is when target source goes down
> ----------------------------------------------------------------------
>
> Key: TEIID-4758
> URL: https://issues.jboss.org/browse/TEIID-4758
> Project: Teiid
> Issue Type: Sub-task
> Components: Server
> Affects Versions: 8.12
> Reporter: Ramesh Reddy
> Assignee: Ramesh Reddy
> Fix For: 9.3
>
>
> During the external materialization load, if the target cache database goes offline, the materialization job stops, but the {{Status}} table is left in {{LOADING}} state, which will never recover when the target cache database comes back up again.
> This situation is observed when JDG is used in OpenShift along with JDV However, behavior can occur in standalone situations too. The system should resilient and must recover in this situation.
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
9 years, 1 month
[JBoss JIRA] (TEIID-4734) Restart during materialization load, does not update Materialized views anymore
by Ramesh Reddy (JIRA)
[ https://issues.jboss.org/browse/TEIID-4734?page=com.atlassian.jira.plugin... ]
Ramesh Reddy updated TEIID-4734:
--------------------------------
Summary: Restart during materialization load, does not update Materialized views anymore (was: Under some circumstances after restart Teiid Server does not update Materialized views anymore)
> Restart during materialization load, does not update Materialized views anymore
> -------------------------------------------------------------------------------
>
> Key: TEIID-4734
> URL: https://issues.jboss.org/browse/TEIID-4734
> Project: Teiid
> Issue Type: Sub-task
> Components: Common
> Affects Versions: 9.1.2
> Environment: * Wildfly 10
> * MySQL 5.6.32
> * Teiid Server 9.1.2
> Reporter: Pedro Inácio
> Assignee: Ramesh Reddy
> Fix For: 9.3
>
> Attachments: StatusTableAfterStart.csv, StatusTableAfterStop.csv
>
>
> Under some circumstances, start and stop the server several times at different points, some Materialized Views are not being updated by the server.
> It is observable that in status table some views are no longer updated. In server log are being printed lots of INFO lines.
> Waited more than 15 minutes and the server was not capable of recovering.
> See the _StatusTableAfterStop.csv_ file to observe the status table content after stopping the server.
> See the _StatusTableAfterStart.csv_ file to observe the status table content after approximately 15 minutes after restart.
> Check _server.log_ for the Teiid Server log.
> See that hundreds of lines similar to the following occur
> {panel:title=Server.log}
> 2017-01-31 13:06:34,597 INFO [org.teiid.COMMAND_LOG] (Teiid Timer0) JfGY9cInAbrb START USER COMMAND: startTime=2017-01-31 13:06:34.597 requestID=JfGY9cInAbrb.0 txID=null sessionID=JfGY9cInAbrb applicationName=internal principal=embedded-async vdbName=CountryServiceListVDB vdbVersion=1 sql=execute SYSADMIN.matViewStatus('NumberingPlan', 'numbering_plan')
> {panel}
> The server cannot continue to load materialized views.
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
9 years, 1 month
[JBoss JIRA] (TEIID-4775) When using external materialization, after transaction timeout, LoadState continues to be LOADING
by Ramesh Reddy (JIRA)
[ https://issues.jboss.org/browse/TEIID-4775?page=com.atlassian.jira.plugin... ]
Ramesh Reddy commented on TEIID-4775:
-------------------------------------
The situation can also occur if any of the resource-adapter, translator or data source throw a runtime exception (transaction rollback exception is also runtime exception) while processing a procedure block. In this case, the exception is raised above the exception block in the procedure and never executes any exception handling defined in the procedure.
Per [~shawkins] I think the rationale was that it is unexpected - but it does logically apply to the block, so it should be handled - it's just that it will mask the real exception
So, the solution is to let the procedure handle the all exceptions, but they can choose to log the message to keep track of the original exception instead of swallowing it.
> When using external materialization, after transaction timeout, LoadState continues to be LOADING
> -------------------------------------------------------------------------------------------------
>
> Key: TEIID-4775
> URL: https://issues.jboss.org/browse/TEIID-4775
> Project: Teiid
> Issue Type: Sub-task
> Components: Server
> Affects Versions: 8.7
> Environment: * CentOs 7
> * Teiid 9.1.2
> * Wildfly 10
> Reporter: Pedro Inácio
> Assignee: Ramesh Reddy
> Fix For: 9.3
>
>
> For some unknown reason, sometimes a materialized view takes more time to finish the load/update than the one defined in transaction timeout (300 seconds is the default value, and we have configured 600 seconds to try to solve the problem).
> The transaction reaper will kill the update transaction. The _LoadState_ in _status_ table will be *LOADING*.
> Any subsequent try to update the view will fail, since the system reads the state as being *LOADING* even if the system is not currently updating it.
> Example log next.
> The update took too long (more than 10 minutes [defined transaction timeout]):
> {code:batch}
> 2017-02-17 16:10:44,487 INFO [org.teiid.COMMAND_LOG] (NIO6) gZueFR0aNjgi START USER COMMAND: startTime=2017-02-17 16:10:44.487 requestID=gZueFR0aNjgi.11 txID=null sessionID=gZueFR0aNjgi applicationName=JDBC principal=manel vdbName=CountryServiceListVDB vdbVersion=1 sql={call SYSADMIN.loadMatView(?, ?)}
> 2017-02-17 16:20:44,528 WARN [com.arjuna.ats.arjuna] (Transaction Reaper) ARJUNA012117: TransactionReaper::check timeout for TX 0:ffff0a080174:21ddbdf0:58a71a32:cb in state RUN
> 2017-02-17 16:20:44,564 WARN [com.arjuna.ats.arjuna] (Transaction Reaper Worker 0) ARJUNA012121: TransactionReaper::doCancellations worker Thread[Transaction Reaper Worker 0,5,main] successfully canceled TX 0:ffff0a080174:21ddbdf0:58a71a32:cb
> {code}
> Next I tried to update the view:
> {code:batch}
> 2017-02-17 16:29:21,272 INFO [org.teiid.COMMAND_LOG] (NIO2) E1xyKpChGKvC START USER COMMAND: startTime=2017-02-17 16:29:21.272 requestID=E1xyKpChGKvC.7 txID=null sessionID=E1xyKpChGKvC applicationName=JDBC principal=manel vdbName=CountryServiceListVDB vdbVersion=1 sql=EXECUTE SYSADMIN.loadMatView('CountryServiceListCalculatedRaw','country_service_list_calculated_raw')
> 2017-02-17 16:29:21,391 INFO [org.teiid.COMMAND_LOG] (Worker25_QueryProcessorQueue18136) E1xyKpChGKvC END USER COMMAND: endTime=2017-02-17 16:29:21.391 requestID=E1xyKpChGKvC.7 txID=null sessionID=E1xyKpChGKvC principal=manel vdbName=CountryServiceListVDB vdbVersion=1 finalRowCount=1
> {code}
> And nothing happened.
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
9 years, 1 month
[JBoss JIRA] (TEIID-4734) Under some circumstances after restart Teiid Server does not update Materialized views anymore
by Ramesh Reddy (JIRA)
[ https://issues.jboss.org/browse/TEIID-4734?page=com.atlassian.jira.plugin... ]
Ramesh Reddy updated TEIID-4734:
--------------------------------
Parent: TEIID-4779
Issue Type: Sub-task (was: Bug)
Steps to Reproduce: (was: # Define some materialized view
# Start and stop the server several times at different points (while loading the VDB, while running normally, etc)
# At some point the server no longer is capable of updating some materialized views)
Workaround Description: (was: # Stop the server
# Truncate all tables in database (related with Materialization)
# Start the server)
Workaround: (was: Workaround Exists)
> Under some circumstances after restart Teiid Server does not update Materialized views anymore
> ----------------------------------------------------------------------------------------------
>
> Key: TEIID-4734
> URL: https://issues.jboss.org/browse/TEIID-4734
> Project: Teiid
> Issue Type: Sub-task
> Components: Common
> Affects Versions: 9.1.2
> Environment: * Wildfly 10
> * MySQL 5.6.32
> * Teiid Server 9.1.2
> Reporter: Pedro Inácio
> Assignee: Ramesh Reddy
> Fix For: 9.3
>
> Attachments: StatusTableAfterStart.csv, StatusTableAfterStop.csv
>
>
> Under some circumstances, start and stop the server several times at different points, some Materialized Views are not being updated by the server.
> It is observable that in status table some views are no longer updated. In server log are being printed lots of INFO lines.
> Waited more than 15 minutes and the server was not capable of recovering.
> See the _StatusTableAfterStop.csv_ file to observe the status table content after stopping the server.
> See the _StatusTableAfterStart.csv_ file to observe the status table content after approximately 15 minutes after restart.
> Check _server.log_ for the Teiid Server log.
> See that hundreds of lines similar to the following occur
> {panel:title=Server.log}
> 2017-01-31 13:06:34,597 INFO [org.teiid.COMMAND_LOG] (Teiid Timer0) JfGY9cInAbrb START USER COMMAND: startTime=2017-01-31 13:06:34.597 requestID=JfGY9cInAbrb.0 txID=null sessionID=JfGY9cInAbrb applicationName=internal principal=embedded-async vdbName=CountryServiceListVDB vdbVersion=1 sql=execute SYSADMIN.matViewStatus('NumberingPlan', 'numbering_plan')
> {panel}
> The server cannot continue to load materialized views.
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
9 years, 1 month
[JBoss JIRA] (TEIID-4768) Clarify Caching guide using external materializaion
by Ramesh Reddy (JIRA)
[ https://issues.jboss.org/browse/TEIID-4768?page=com.atlassian.jira.plugin... ]
Ramesh Reddy updated TEIID-4768:
--------------------------------
Parent: TEIID-4779
Issue Type: Sub-task (was: Enhancement)
> Clarify Caching guide using external materializaion
> ---------------------------------------------------
>
> Key: TEIID-4768
> URL: https://issues.jboss.org/browse/TEIID-4768
> Project: Teiid
> Issue Type: Sub-task
> Components: Documentation
> Affects Versions: 9.3
> Reporter: Van Halbert
> Assignee: Ramesh Reddy
> Priority: Critical
>
> The caching guide section that relates to external materialization needs to add the following information for clarity:
> - the TTL needs to be set at a minimum of 5 minutes (300000) so in cases where the status gets hung at "LOADING" (i.e., the target cache goes down during materialization), the materialization process will recover and restart the loading.
> - add a status table "Status" matrix:
> LOADED - means the cache was loaded
> LOADING - means cache is in the process of loading
> FAILED_LOAD - means an error occurred during the load process
> ...
> and these should correlate to the section that describes the process in detail.
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
9 years, 1 month