[aerogear-dev] Upgrading process for the OpenShift AeroGear UPS cartridge

[Andrea Vibelli]

Hello all, 

I just wanted to let you know that Farah is working on AGPUSH-504, a snapshot/restore approach for upgrading the downloadable cartridge. 
The overall idea is to take a snapshot of the database (and other configuration) for an older existing cartridge instance, delete this old instance, and then create a new instance based on the latest cartridge version and restore the new instance's database (and other configuration) to the state in the snapshot. The new instance must have the same name as the old instance. The main thing that she needed to determine was what needed to be included in the snapshot that's not already part of the MySQL database. The SimplePush token key was one of these things (we generate this value when a cartridge instance is created and it is used by the server for encryption and decryption of endpoint URLs). The standalone.xml file also needed to be backed up just in case the user made any changes to it. (Creating a backup of this file allows users to manually merge back any changes they may have made.)

The snapshot/restore action hooks she has created can be found in [1].

The steps she has been using to try upgrading from an 0.8.1 instance are listed below. It would be great if you could try out these steps as well. There's one important caveat though: because an 0.8.1 instance won't have the necessary snapshot hook, you'll need to manually perform some commands to save the necessary data (step 3 below). If we include these snapshot/restore action hooks in the 0.10.0 cartridge release, users will be able to make use of these hooks for future upgrades, e.g., for upgrading from 0.10.0 to 0.11.0.
  
1) Create an 0.8.1 cartridge instance
rhc create-app mypush "https://cartreflect-claytondev.rhcloud.com/reflect?github=aerogear/openshift-origin-cartridge-aerogear-push&commit=8aada257de52157a837db23a9f7b1a8574fdffab" mysql-5.1

2) Log into the admin console, change password, create push applications, create variants, send push notifications, etc.

3) ssh into your instance and perform these commands (this step manually captures needed data)
echo "$OPENSHIFT_AEROGEAR_PUSH_TOKEN_KEY" > $OPENSHIFT_DATA_DIR/aerogear_push_token_key
cp $OPENSHIFT_AEROGEAR_PUSH_DIR/standalone/configuration/standalone.xml $OPENSHIFT_DATA_DIR/standalone.snapshot.xml

4) Create a snapshot of the 0.8.1 instance: rhc snapshot save mypush

5) Delete the 0.8.1 instance: rhc app delete mypush

6) Delete the local directory on the local machine (from step 1, you should have a directory named 'mypush')

7) Create a new cartridge instance (this uses her fork of the 0.9.0 cartridge repository that contains the snapshot/restore action hooks)
rhc create-app mypush "https://cartreflect-claytondev.rhcloud.com/reflect?github=fjuma/openshift-origin-cartridge-aerogear-push&commit=956a4fc09edc93fe76394b86b265ceb1d238ccaf" mysql-5.1

8) Restore the instance: rhc snapshot restore mypush -f /PATH/TO/mypush.tar.gz

9) You should now be able to log into the admin console with the same password you previously set, see existing push applications, variants, send push notifications, etc.

One disadvantage of this snapshot/restore approach is the extra work for the user. It would be nice to have a single command that can do the snapshot, delete, create, and restore all at once. She is going to start looking into this "wrappering" functionality.


Why am I writing about this?

I have made some test about the above steps. At a first glance, it seemed that something was broken between steps 6 and 7.
Doing various tests, what I saw in the logs (server.log) were different errors:

2014/01/26 10:19:18,061 ERROR [org.jboss.as] (MSC service thread 1-3) JBAS015875: JBoss AS 7.1.1.Final "Brontes" started (with errors) in 296088ms - Started 349 of 524 services (52 services failed or missing dependencies, 119 services are passive or on-demand)
2014/01/26 10:19:18,344 INFO  [org.jboss.as.server] (DeploymentScanner-threads - 2) JBAS015870: Deploy of deployment "ROOT.war" was rolled back with failure message {"JBAS014771: Services with missing/unavailable dependencies" => ["jboss.persistenceunit.\"ROOT.war#unifiedpush-default\"jboss.naming.context.java.jboss.datasources.PushEEDSMissing[jboss.persistenceunit.\"ROOT.war#unifiedpush-default\"jboss.naming.context.java.jboss.datasources.PushEEDS]","jboss.persistenceunit.\"ROOT.war#picketlink-default\"jboss.naming.context.java.jboss.datasources.PushEEDSMissing[jboss.persistenceunit.\"ROOT.war#picketlink-default\"jboss.naming.context.java.jboss.datasources.PushEEDS]"]}
2014/01/26 10:19:19,390 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-2) JBAS015877: Stopped deployment ROOT.war in 1125ms
2014/01/26 10:19:19,393 INFO  [org.jboss.as.controller] (DeploymentScanner-threads - 2) JBAS014774: Service status report
JBAS014775:    New missing/unsatisfied dependencies:
      service jboss.naming.context.java.jboss.datasources.PushEEDS (missing) dependents: [service jboss.persistenceunit."ROOT.war#unifiedpush-default", service jboss.persistenceunit."ROOT.war#picketlink-default"]


2014/01/26 10:19:19,440 ERROR [org.jboss.as.server.deployment.scanner] (DeploymentScanner-threads - 1) {"JBAS014653: Composite operation failed and was rolled back. Steps that failed:" => {"Operation step-2" => {"JBAS014771: Services with missing/unavailable dependencies" => ["jboss.persistenceunit.\"ROOT.war#unifiedpush-default\"jboss.naming.context.java.jboss.datasources.PushEEDSMissing[jboss.persistenceunit.\"ROOT.war#unifiedpush-default\"jboss.naming.context.java.jboss.datasources.PushEEDS]","jboss.persistenceunit.\"ROOT.war#picketlink-default\"jboss.naming.context.java.jboss.datasources.PushEEDSMissing[jboss.persistenceunit.\"ROOT.war#picketlink-default\"jboss.naming.context.java.jboss.datasources.PushEEDS]"]}}}


OR


2014/01/26 05:41:46,522 INFO  [org.jboss.as.connector] (MSC service thread 1-1) JBAS010408: Starting JCA Subsystem (JBoss IronJacamar 1.0.9.Final)
2014/01/26 05:41:46,922 ERROR [org.jboss.as.controller.management-operation] (ServerService Thread Pool -- 62) Operation ("enable") failed - address: ([
    ("subsystem" => "datasources"),
    ("data-source" => "PostgreSQLDS")
]) - failure description: "JBAS014802: Cannot resolve expression 'expression \"jdbcostgresql://${env.OPENSHIFT_POSTGRESQL_DB_HOST}:${env.OPENSHIFT_POSTGRESQL_DB_PORT}/mypush13\"' -- java.lang.IllegalStateException: Failed to resolve expression: ${env.OPENSHIFT_POSTGRESQL_DB_HOST}"


OR 


no errors in the logs, but in the deployments directory I found 'ROOT.war.dodeploy' and 'ROOT.war.isdeploying' files, but no 'ROOT.war.deployed' file.


Those error seemed to me as random and too strange (by the way, the creation of the cartridge is always correct, I never had problems if I created a new instance with a never used before name).

To investigate further, I then tried to do the same steps with a completely different application: 'Kitchensink on OpenShift', and I had the same error! After the deletion and the recreation of a cartridge with a same name, I encountered: 'The requested URL /app was not found on this server.'
The steps I have done are:
1) rhc app create -a kitchensink -t jbossas-7 --from-code git://github.com/openshift/kitchensink-example.git
2) rhc snapshot save kitchensink
3) rhc app delete kitchensink
4) rm -rf kitchensinkhtml5/
5) rhc app create -a kitchensink -t jbossas-7 --from-code git://github.com/openshift/kitchensink-example.git
6) Access to the application.

So the good news is that the error I encountered is not related to this specific AeroGear cartridges, but the bad news is that is something more general about OpenShift.
We need to investigate further and find out if there is a way to "flush" the deletion, it seems that there is something appended or cached that prevent the recreation of a cartridge with a same name. 

We are currently doing some more tests on these process.