[JBoss JIRA] (SWSQE-124) Fix or replace the CNCF CI OpenShift Cluster
by Gary Brown (JIRA)
[ https://issues.jboss.org/browse/SWSQE-124?page=com.atlassian.jira.plugin.... ]
Gary Brown commented on SWSQE-124:
----------------------------------
Hopefully will be possible to resolve the problem - as this could potentially occur on a customer site, so would be good to fix while reproducible.
> Fix or replace the CNCF CI OpenShift Cluster
> --------------------------------------------
>
> Key: SWSQE-124
> URL: https://issues.jboss.org/browse/SWSQE-124
> Project: Kiali QE
> Issue Type: QE Task
> Reporter: Kevin Earls
> Assignee: Matt Mahoney
>
> As we've discussed in email, the CNCF CI OpenShift cluster is currently in a state where deleting some resources is impossible, which makes it impossible to run Jenkins tests. I first saw this when a test tried to delete ElasticSearch. Then after a cluster restart I saw the same problem with Cassandra.
> We either need to figure out how to resolve the problem, or replace the cluster.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
8 years, 3 months
[JBoss JIRA] (WFLY-10053) TM is not forward compatible
by Flavia Rainone (JIRA)
[ https://issues.jboss.org/browse/WFLY-10053?page=com.atlassian.jira.plugin... ]
Flavia Rainone commented on WFLY-10053:
---------------------------------------
[~ochaloup] Don't you think that the error above is the one that happens before the second server is restarted? I.e., the error above is just the test scenario, it is the first attempt to commit, the one that will go wrong because the server drops. What should happen after the wildfly server is restarted? Should an attempt to recover the node be sent? Any pointers to the arjuna code of where this happens will be useful, because I think that this is where I need to investigate it.
[~tomjenkinson] Doesn't it look like something that has been fixed already, but only on the new versions of the server? I think the problem is related to the older versions of the code, look at the following snippet:
"The issue is valid for:
6.4.x -> WF12
7.0.X -> WF12
The issue is not valid for:
WF12 -> 6.4.x
WF12-> 7.0.x
7.1.0 -> WF 12"
To me, it means that when server 1 is WFLY12 or EAP 7.1.0, it is capable to recover the transaction commit after the second server is restarted, regardless of the version of the second server. On the other hand, previous server versions, namely 6.4.x and 7.0.x, are not capable of realizing that the second server is restarted and finding it/recovering the missing resource. I wonder what is the output if we have something like 7.0.x -> 7.0.x.
I'll try to run this as soon as I get access to the test code (I don't have a login to gitlab repo).
> TM is not forward compatible
> -----------------------------
>
> Key: WFLY-10053
> URL: https://issues.jboss.org/browse/WFLY-10053
> Project: WildFly
> Issue Type: Bug
> Components: Transactions
> Affects Versions: 12.0.0.Final
> Reporter: Ivan Straka
> Assignee: Tom Jenkinson
> Attachments: EAP7.0.9.log, WF12.log
>
>
> TM is unable to recover transaction after one of the nodes crash during commit phase of a resource.
> Scenario:
> # EAP 7.0.9 enlists dummy XA resource, update db value (XADatasource) and call WildFly 12
> # WildFly 12 enlists dummy XA resource, update other db value
> # At entry of commit phase of dummy XA resource on WildFly12 byteman crash the server
> # WildFly 12 is started
> # Recovery process should recover the transaction
> Following exception is seen in EAP 7.0.9 log
> {code:java}
> 2018-03-20 12:42:35,171 WARN [com.arjuna.ats.jta] (Periodic Recovery) ARJUNA016036: commit on < formatId=131077, gtrid_length=32, bqual_length=36, tx_uid=0:ffff7f000001:643452d5:5ab0f37d:f, node_name=eap1, branch_uid=0:ffff7f000001:643452d5:5ab0f37d:1e, subordinatenodename=null, eis_name=unknown eis name > (RecoveryOnlySerializedEJBXAResource{ejbReceiverNodeName='eap2'}) failed with exception $XAException.XA_RETRY: javax.transaction.xa.XAException
> at org.jboss.ejb.client.RecoveryOnlySerializedEJBXAResource.commit(RecoveryOnlySerializedEJBXAResource.java:56)
> at com.arjuna.ats.internal.jta.resources.arjunacore.XAResourceRecord.topLevelCommit(XAResourceRecord.java:477)
> at com.arjuna.ats.arjuna.coordinator.BasicAction.doCommit(BasicAction.java:2869)
> at com.arjuna.ats.arjuna.coordinator.BasicAction.doCommit(BasicAction.java:2785)
> at com.arjuna.ats.arjuna.coordinator.BasicAction.phase2Commit(BasicAction.java:1853)
> at com.arjuna.ats.arjuna.recovery.RecoverAtomicAction.replayPhase2(RecoverAtomicAction.java:71)
> at com.arjuna.ats.internal.arjuna.recovery.AtomicActionRecoveryModule.doRecoverTransaction(AtomicActionRecoveryModule.java:152)
> at com.arjuna.ats.internal.arjuna.recovery.AtomicActionRecoveryModule.processTransactionsStatus(AtomicActionRecoveryModule.java:253)
> at com.arjuna.ats.internal.arjuna.recovery.AtomicActionRecoveryModule.periodicWorkSecondPass(AtomicActionRecoveryModule.java:109)
> at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.doWorkInternal(PeriodicRecovery.java:811)
> at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.run(PeriodicRecovery.java:377)
> {code}
> Result is that resources on EAP7.0.9 are commited and resources on WF12 are not:
> {code:java}
> [INFO] there is (are) 1 transaction(s) in tx obj store /home/istraka/manu-units/tests-transactions-propagation/manu-unit-eap-transactions-propagation/out/EAP2_7.2.0/workspace/wildfly-12.0.0.Final/standalone/data/tx-object-store
> [INFO] Transaction
> Xid:< formatId=131077, gtrid_length=32, bqual_length=40, tx_uid=0:ffff7f000001:-174ad8d9:5ab0fb3b:f, node_name=eap1, branch_uid=0:ffff7f000001:-174ad8d9:5ab0fb3b:1e, subordinatenodename=eap2, eis_name=0 >
> Type:StateManager/BasicAction/TwoPhaseCoordinator/AtomicAction/SubordinateAtomicAction/JCA
> ParentNodeName:eap1
> Participant:false
> CreationTime:Tue, 20 Mar 2018 13:15:18 +0100
> AgeInSeconds:269
> Id:0:ffff7f000001:39e6900e:5ab0fb4a:13
> Participant
> HeuristicValue:-1
> Status:PREPARED
> JndiName:java:/TestXAResource
> GlobalTransactionId:[B@5753c786
> HeuristicStatus:UNKNOWN
> NodeName:null
> Timeout:0
> FormatId:-1
> BranchQualifier:[B@1874640
> Type:/StateManager/AbstractRecord/XAResourceRecord
> Participant:true
> EisProductVersion:EAP Test
> ClassName:com.arjuna.ats.internal.jta.resources.arjunacore.XAResourceRecord
> EisProductName:Crash Recovery Test
> Id:0:ffff7f000001:39e6900e:5ab0fb4a:1d
> Participant
> HeuristicValue:-1
> Status:PREPARED
> JndiName:java:jboss/eap2-ds-jndi
> GlobalTransactionId:[B@4ae75d2d
> HeuristicStatus:UNKNOWN
> NodeName:null
> Timeout:0
> FormatId:-1
> BranchQualifier:[B@5eaa38ce
> Type:/StateManager/AbstractRecord/XAResourceRecord
> Participant:true
> EisProductVersion:9.3.15
> ClassName:com.arjuna.ats.internal.jta.resources.arjunacore.XAResourceRecord
> EisProductName:PostgreSQL
> Id:0:ffff7f000001:39e6900e:5ab0fb4a:23
> {code}
> The issue is valid for:
> * 6.4.x -> WF12
> * 7.0.X -> WF12
> The issue is *not* valid for:
> * WF12 -> 6.4.x
> * WF12-> 7.0.x
> * 7.1.0 -> WF 12
> If you have any question (about testsuite, scenario,...) feel free to ask.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
8 years, 3 months
[JBoss JIRA] (WFLY-10053) TM is not forward compatible
by Flavia Rainone (JIRA)
[ https://issues.jboss.org/browse/WFLY-10053?page=com.atlassian.jira.plugin... ]
Flavia Rainone commented on WFLY-10053:
---------------------------------------
[~tomjenkinson][~ochaloup] I am investigating this issue. [~istraka] I'll need to reproduce locally the bug, I'll send you an e-mail if I have any difficulties running those tests.
> TM is not forward compatible
> -----------------------------
>
> Key: WFLY-10053
> URL: https://issues.jboss.org/browse/WFLY-10053
> Project: WildFly
> Issue Type: Bug
> Components: Transactions
> Affects Versions: 12.0.0.Final
> Reporter: Ivan Straka
> Assignee: Tom Jenkinson
> Attachments: EAP7.0.9.log, WF12.log
>
>
> TM is unable to recover transaction after one of the nodes crash during commit phase of a resource.
> Scenario:
> # EAP 7.0.9 enlists dummy XA resource, update db value (XADatasource) and call WildFly 12
> # WildFly 12 enlists dummy XA resource, update other db value
> # At entry of commit phase of dummy XA resource on WildFly12 byteman crash the server
> # WildFly 12 is started
> # Recovery process should recover the transaction
> Following exception is seen in EAP 7.0.9 log
> {code:java}
> 2018-03-20 12:42:35,171 WARN [com.arjuna.ats.jta] (Periodic Recovery) ARJUNA016036: commit on < formatId=131077, gtrid_length=32, bqual_length=36, tx_uid=0:ffff7f000001:643452d5:5ab0f37d:f, node_name=eap1, branch_uid=0:ffff7f000001:643452d5:5ab0f37d:1e, subordinatenodename=null, eis_name=unknown eis name > (RecoveryOnlySerializedEJBXAResource{ejbReceiverNodeName='eap2'}) failed with exception $XAException.XA_RETRY: javax.transaction.xa.XAException
> at org.jboss.ejb.client.RecoveryOnlySerializedEJBXAResource.commit(RecoveryOnlySerializedEJBXAResource.java:56)
> at com.arjuna.ats.internal.jta.resources.arjunacore.XAResourceRecord.topLevelCommit(XAResourceRecord.java:477)
> at com.arjuna.ats.arjuna.coordinator.BasicAction.doCommit(BasicAction.java:2869)
> at com.arjuna.ats.arjuna.coordinator.BasicAction.doCommit(BasicAction.java:2785)
> at com.arjuna.ats.arjuna.coordinator.BasicAction.phase2Commit(BasicAction.java:1853)
> at com.arjuna.ats.arjuna.recovery.RecoverAtomicAction.replayPhase2(RecoverAtomicAction.java:71)
> at com.arjuna.ats.internal.arjuna.recovery.AtomicActionRecoveryModule.doRecoverTransaction(AtomicActionRecoveryModule.java:152)
> at com.arjuna.ats.internal.arjuna.recovery.AtomicActionRecoveryModule.processTransactionsStatus(AtomicActionRecoveryModule.java:253)
> at com.arjuna.ats.internal.arjuna.recovery.AtomicActionRecoveryModule.periodicWorkSecondPass(AtomicActionRecoveryModule.java:109)
> at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.doWorkInternal(PeriodicRecovery.java:811)
> at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.run(PeriodicRecovery.java:377)
> {code}
> Result is that resources on EAP7.0.9 are commited and resources on WF12 are not:
> {code:java}
> [INFO] there is (are) 1 transaction(s) in tx obj store /home/istraka/manu-units/tests-transactions-propagation/manu-unit-eap-transactions-propagation/out/EAP2_7.2.0/workspace/wildfly-12.0.0.Final/standalone/data/tx-object-store
> [INFO] Transaction
> Xid:< formatId=131077, gtrid_length=32, bqual_length=40, tx_uid=0:ffff7f000001:-174ad8d9:5ab0fb3b:f, node_name=eap1, branch_uid=0:ffff7f000001:-174ad8d9:5ab0fb3b:1e, subordinatenodename=eap2, eis_name=0 >
> Type:StateManager/BasicAction/TwoPhaseCoordinator/AtomicAction/SubordinateAtomicAction/JCA
> ParentNodeName:eap1
> Participant:false
> CreationTime:Tue, 20 Mar 2018 13:15:18 +0100
> AgeInSeconds:269
> Id:0:ffff7f000001:39e6900e:5ab0fb4a:13
> Participant
> HeuristicValue:-1
> Status:PREPARED
> JndiName:java:/TestXAResource
> GlobalTransactionId:[B@5753c786
> HeuristicStatus:UNKNOWN
> NodeName:null
> Timeout:0
> FormatId:-1
> BranchQualifier:[B@1874640
> Type:/StateManager/AbstractRecord/XAResourceRecord
> Participant:true
> EisProductVersion:EAP Test
> ClassName:com.arjuna.ats.internal.jta.resources.arjunacore.XAResourceRecord
> EisProductName:Crash Recovery Test
> Id:0:ffff7f000001:39e6900e:5ab0fb4a:1d
> Participant
> HeuristicValue:-1
> Status:PREPARED
> JndiName:java:jboss/eap2-ds-jndi
> GlobalTransactionId:[B@4ae75d2d
> HeuristicStatus:UNKNOWN
> NodeName:null
> Timeout:0
> FormatId:-1
> BranchQualifier:[B@5eaa38ce
> Type:/StateManager/AbstractRecord/XAResourceRecord
> Participant:true
> EisProductVersion:9.3.15
> ClassName:com.arjuna.ats.internal.jta.resources.arjunacore.XAResourceRecord
> EisProductName:PostgreSQL
> Id:0:ffff7f000001:39e6900e:5ab0fb4a:23
> {code}
> The issue is valid for:
> * 6.4.x -> WF12
> * 7.0.X -> WF12
> The issue is *not* valid for:
> * WF12 -> 6.4.x
> * WF12-> 7.0.x
> * 7.1.0 -> WF 12
> If you have any question (about testsuite, scenario,...) feel free to ask.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
8 years, 3 months
[JBoss JIRA] (JBJCA-1371) Deadlock for threads executing org.jboss.jca.adapters.jdbc.xa.XAManagedConnectionFactory.isEqual
by Flavia Rainone (JIRA)
[ https://issues.jboss.org/browse/JBJCA-1371?page=com.atlassian.jira.plugin... ]
Flavia Rainone updated JBJCA-1371:
----------------------------------
Git Pull Request: https://github.com/ironjacamar/ironjacamar/pull/657
> Deadlock for threads executing org.jboss.jca.adapters.jdbc.xa.XAManagedConnectionFactory.isEqual
> ------------------------------------------------------------------------------------------------
>
> Key: JBJCA-1371
> URL: https://issues.jboss.org/browse/JBJCA-1371
> Project: IronJacamar
> Issue Type: Bug
> Components: Core
> Reporter: Flavia Rainone
> Assignee: Flavia Rainone
>
> Two or more threads hold one lock on an instance of java.util.Collections$SynchronizedMap and attempt to obtain a lock on a different instance (e.g. locks on 0x00000001d0d08bd8 and 0x00000001cea22628 in the example below)
> "thread-one" ...
> java.lang.Thread.State: BLOCKED (on object monitor)
> at java.util.Collections$SynchronizedMap.get(Collections.java:2584)
> - waiting to lock <0x00000001cea22628> (a java.util.Collections$SynchronizedMap)
> at java.util.AbstractMap.equals(AbstractMap.java:495)
> at java.util.Collections$SynchronizedMap.equals(Collections.java:2631)
> - locked <0x00000001d0d08bd8> (a java.util.Collections$SynchronizedMap)
> at org.jboss.jca.adapters.jdbc.xa.XAManagedConnectionFactory.isEqual(XAManagedConnectionFactory.java:563)
> - locked <0x00000001d0d08bd8> (a java.util.Collections$SynchronizedMap)
> at org.jboss.jca.adapters.jdbc.xa.XAManagedConnectionFactory.equals(XAManagedConnectionFactory.java:598)
> at org.jboss.jca.adapters.jdbc.BaseWrapperManagedConnectionFactory$SubjectActions.doCheck(BaseWrapperManagedConnectionFactory.java:1397)
> at org.jboss.jca.adapters.jdbc.BaseWrapperManagedConnectionFactory$SubjectActions.addMatchingProperties(BaseWrapperManagedConnectionFactory.java:1508)
> at org.jboss.jca.adapters.jdbc.BaseWrapperManagedConnectionFactory.getConnectionProperties(BaseWrapperManagedConnectionFactory.java:1135)
> at org.jboss.jca.adapters.jdbc.xa.XAManagedConnectionFactory.matchManagedConnections(XAManagedConnectionFactory.java:533)
> ...
> at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:138)
> at org.jboss.as.connector.subsystems.datasources.WildFlyDataSource.getConnection(WildFlyDataSource.java:66)
> ...
> "thread-two" ...
> java.lang.Thread.State: BLOCKED (on object monitor)
> at java.util.Collections$SynchronizedMap.size(Collections.java:2572)
> - waiting to lock <0x00000001d0d08bd8> (a java.util.Collections$SynchronizedMap)
> at java.util.AbstractMap.equals(AbstractMap.java:482)
> at java.util.Collections$SynchronizedMap.equals(Collections.java:2631)
> - locked <0x00000001cea22628> (a java.util.Collections$SynchronizedMap)
> at org.jboss.jca.adapters.jdbc.xa.XAManagedConnectionFactory.isEqual(XAManagedConnectionFactory.java:563)
> - locked <0x00000001cea22628> (a java.util.Collections$SynchronizedMap)
> at org.jboss.jca.adapters.jdbc.xa.XAManagedConnectionFactory.equals(XAManagedConnectionFactory.java:598)
> ... [same stack as thread-one above] ...
> at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:138)
> at org.jboss.as.connector.subsystems.datasources.WildFlyDataSource.getConnection(WildFlyDataSource.java:66)
> ...
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
8 years, 3 months
[JBoss JIRA] (WFLY-10182) Unstable work of the persistent EJB timers with Oracle DB. ORA-08177: can't serialize access for this transaction
by Maxim Karavaev (JIRA)
[ https://issues.jboss.org/browse/WFLY-10182?page=com.atlassian.jira.plugin... ]
Maxim Karavaev updated WFLY-10182:
----------------------------------
Workaround Description: Patch attached
> Unstable work of the persistent EJB timers with Oracle DB. ORA-08177: can't serialize access for this transaction
> -----------------------------------------------------------------------------------------------------------------
>
> Key: WFLY-10182
> URL: https://issues.jboss.org/browse/WFLY-10182
> Project: WildFly
> Issue Type: Bug
> Components: EJB
> Affects Versions: 10.1.0.Final, 11.0.0.Final, 12.0.0.Final
> Environment: WildFly cluster in domain mode.
> Oracle 12c DB
> Reporter: Maxim Karavaev
> Attachments: Stable_persistence_logic_of_EJB_Timers.patch
>
>
> The main signal is an exception in logs:
> {noformat}
> ORA-08177: can't serialize access for this transaction
> {noformat}
> When one cluster node is halted (the one on the witch "Timer" is actually working) then the other nodes are still can't catch up this "timer" from the DB due to the mentioned exception. Even worth scenario - when all nodes work perfectly, but occasionally all nodes start to decline, that timer is in the state IN_TIMEOUT and it really is due to the DB state, just because the last timer's invocation did not manage to commit final state after processing due to the same reason (SQL exception).
> So, why Oracle DB is so special? That's because it never manages transactions by themselves. It's full responsibility of the client.
> Looking into the code of the class
> {noformat}
> org.jboss.as.ejb3.timerservice.persistence.database.DatabaseTimerPersistence
> {noformat}
> I didn't find transaction management, except one method, and even in it, I found, that it probably doesn't manage transaction as desired, due to the lack of resource participation.
> After I've made some changes to enable transactions for that functionality I have managed to stable EJB Timer logic on the server. Let's have a look at the attached patch (it's for 11 version, but it also works for 12).
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
8 years, 3 months
[JBoss JIRA] (WFLY-10182) Unstable work of the persistent EJB timers with Oracle DB. ORA-08177: can't serialize access for this transaction
by Maxim Karavaev (JIRA)
[ https://issues.jboss.org/browse/WFLY-10182?page=com.atlassian.jira.plugin... ]
Maxim Karavaev updated WFLY-10182:
----------------------------------
Description:
The main signal is an exception in logs:
{noformat}
ORA-08177: can't serialize access for this transaction
{noformat}
When one cluster node is halted (the one on the witch "Timer" is actually working) then the other nodes are still can't catch up this "timer" from the DB due to the mentioned exception. Even worth scenario - when all nodes work perfectly, but occasionally all nodes start to decline, that timer is in the state IN_TIMEOUT and it really is due to the DB state, just because the last timer's invocation did not manage to commit final state after processing due to the same reason (SQL exception).
So, why Oracle DB is so special? That's because it never manages transactions by themselves. It's full responsibility of the client.
Looking into the code of the class
{noformat}
org.jboss.as.ejb3.timerservice.persistence.database.DatabaseTimerPersistence
{noformat}
I didn't find transaction management, except one method, and even in it, I found, that it probably doesn't manage transaction as desired, due to the lack of resource participation.
After I've made some changes to enable transactions for that functionality I have managed to stable EJB Timer logic on the server. Let's have a look at the attached patch (it's for 11 version, but it also works for 12).
was:
The main signal is an exception in logs:
{noformat}
ORA-08177: can't serialize access for this transaction
{noformat}
When one cluster node is halted (the one on the witch "Timer" is actually working) then the other nodes are still can't catch up this "timer" from the DB due to the mentioned exception. Even worth scenario - when all nodes work perfectly, but occasionally all nodes start to decline, that timer is in the state IN_TIMEOUT and it really is due to the DB state, just because the last timer's invocation did not manage to commit final state after processing due to the same reason (SQL exception).
So, why Oracle DB is so special? That's because it never manages transactions by themselves. It's full responsibility of the client.
Looking into the code of the class
{noformat}
org.jboss.as.ejb3.timerservice.persistence.database.DatabaseTimerPersistence
{noformat}
I didn't find transaction management, except one method, and even in it, I found, that it probably doesn't manage transaction as desired, due to the lack of resource participation.
After I've made some changes to enable transactions for that functionality I have managed to stable EJB Timer logic on the server. Let's have a look in attached patch.
> Unstable work of the persistent EJB timers with Oracle DB. ORA-08177: can't serialize access for this transaction
> -----------------------------------------------------------------------------------------------------------------
>
> Key: WFLY-10182
> URL: https://issues.jboss.org/browse/WFLY-10182
> Project: WildFly
> Issue Type: Bug
> Components: EJB
> Affects Versions: 10.1.0.Final, 11.0.0.Final, 12.0.0.Final
> Environment: WildFly cluster in domain mode.
> Oracle 12c DB
> Reporter: Maxim Karavaev
> Attachments: Stable_persistence_logic_of_EJB_Timers.patch
>
>
> The main signal is an exception in logs:
> {noformat}
> ORA-08177: can't serialize access for this transaction
> {noformat}
> When one cluster node is halted (the one on the witch "Timer" is actually working) then the other nodes are still can't catch up this "timer" from the DB due to the mentioned exception. Even worth scenario - when all nodes work perfectly, but occasionally all nodes start to decline, that timer is in the state IN_TIMEOUT and it really is due to the DB state, just because the last timer's invocation did not manage to commit final state after processing due to the same reason (SQL exception).
> So, why Oracle DB is so special? That's because it never manages transactions by themselves. It's full responsibility of the client.
> Looking into the code of the class
> {noformat}
> org.jboss.as.ejb3.timerservice.persistence.database.DatabaseTimerPersistence
> {noformat}
> I didn't find transaction management, except one method, and even in it, I found, that it probably doesn't manage transaction as desired, due to the lack of resource participation.
> After I've made some changes to enable transactions for that functionality I have managed to stable EJB Timer logic on the server. Let's have a look at the attached patch (it's for 11 version, but it also works for 12).
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
8 years, 3 months