[aerogear-dev] Upgrading process for the OpenShift AeroGear UPS cartridge

Mon Jan 27 13:07:31 EST 2014

> From: "Andrea Vibelli" <avibelli at redhat.com>
> To: aerogear-dev at lists.jboss.org
> Sent: Monday, January 27, 2014 12:27:04 PM
> Subject: [aerogear-dev] Upgrading process for the OpenShift AeroGear UPS	cartridge
> 
> Hello all,
> 
> I just wanted to let you know that Farah is working on AGPUSH-504, a
> snapshot/restore approach for upgrading the downloadable cartridge.
> The overall idea is to take a snapshot of the database (and other
> configuration) for an older existing cartridge instance, delete this old
> instance, and then create a new instance based on the latest cartridge
> version and restore the new instance's database (and other configuration) to
> the state in the snapshot. The new instance must have the same name as the
> old instance. The main thing that she needed to determine was what needed to
> be included in the snapshot that's not already part of the MySQL database.
> The SimplePush token key was one of these things (we generate this value
> when a cartridge instance is created and it is used by the server for
> encryption and decryption of endpoint URLs). The standalone.xml file also
> needed to be backed up just in case the user made any changes to it.
> (Creating a backup of this file allows users to manually merge back any
> changes they may have made.)
> 
> The snapshot/restore action hooks she has created can be found in [1].
> 
> The steps she has been using to try upgrading from an 0.8.1 instance are
> listed below. It would be great if you could try out these steps as well.
> There's one important caveat though: because an 0.8.1 instance won't have
> the necessary snapshot hook, you'll need to manually perform some commands
> to save the necessary data (step 3 below). If we include these
> snapshot/restore action hooks in the 0.10.0 cartridge release, users will be
> able to make use of these hooks for future upgrades, e.g., for upgrading
> from 0.10.0 to 0.11.0.
>   
> 1) Create an 0.8.1 cartridge instance
> rhc create-app mypush
> "https://cartreflect-claytondev.rhcloud.com/reflect?github=aerogear/openshift-origin-cartridge-aerogear-push&commit=8aada257de52157a837db23a9f7b1a8574fdffab"
> mysql-5.1
> 
> 2) Log into the admin console, change password, create push applications,
> create variants, send push notifications, etc.
> 
> 3) ssh into your instance and perform these commands (this step manually
> captures needed data)
> echo "$OPENSHIFT_AEROGEAR_PUSH_TOKEN_KEY" >
> $OPENSHIFT_DATA_DIR/aerogear_push_token_key
> cp $OPENSHIFT_AEROGEAR_PUSH_DIR/standalone/configuration/standalone.xml
> $OPENSHIFT_DATA_DIR/standalone.snapshot.xml
> 
> 4) Create a snapshot of the 0.8.1 instance: rhc snapshot save mypush
> 
> 5) Delete the 0.8.1 instance: rhc app delete mypush
> 
> 6) Delete the local directory on the local machine (from step 1, you should
> have a directory named 'mypush')
> 
> 7) Create a new cartridge instance (this uses her fork of the 0.9.0 cartridge
> repository that contains the snapshot/restore action hooks)
> rhc create-app mypush
> "https://cartreflect-claytondev.rhcloud.com/reflect?github=fjuma/openshift-origin-cartridge-aerogear-push&commit=956a4fc09edc93fe76394b86b265ceb1d238ccaf"
> mysql-5.1
> 
> 8) Restore the instance: rhc snapshot restore mypush -f
> /PATH/TO/mypush.tar.gz
> 
> 9) You should now be able to log into the admin console with the same
> password you previously set, see existing push applications, variants, send
> push notifications, etc.
> 
> One disadvantage of this snapshot/restore approach is the extra work for the
> user. It would be nice to have a single command that can do the snapshot,
> delete, create, and restore all at once. She is going to start looking into
> this "wrappering" functionality.
> 
> 
> Why am I writing about this?
> 
> I have made some test about the above steps. At a first glance, it seemed
> that something was broken between steps 6 and 7.
> Doing various tests, what I saw in the logs (server.log) were different
> errors:
> 
> 2014/01/26 10:19:18,061 ERROR [org.jboss.as] (MSC service thread 1-3)
> JBAS015875: JBoss AS 7.1.1.Final "Brontes" started (with errors) in 296088ms
> - Started 349 of 524 services (52 services failed or missing dependencies,
> 119 services are passive or on-demand)
> 2014/01/26 10:19:18,344 INFO  [org.jboss.as.server]
> (DeploymentScanner-threads - 2) JBAS015870: Deploy of deployment "ROOT.war"
> was rolled back with failure message {"JBAS014771: Services with
> missing/unavailable dependencies" =>
> ["jboss.persistenceunit.\"ROOT.war#unifiedpush-default\"jboss.naming.context.java.jboss.datasources.PushEEDSMissing[jboss.persistenceunit.\"ROOT.war#unifiedpush-default\"jboss.naming.context.java.jboss.datasources.PushEEDS]","jboss.persistenceunit.\"ROOT.war#picketlink-default\"jboss.naming.context.java.jboss.datasources.PushEEDSMissing[jboss.persistenceunit.\"ROOT.war#picketlink-default\"jboss.naming.context.java.jboss.datasources.PushEEDS]"]}
> 2014/01/26 10:19:19,390 INFO  [org.jboss.as.server.deployment] (MSC service
> thread 1-2) JBAS015877: Stopped deployment ROOT.war in 1125ms
> 2014/01/26 10:19:19,393 INFO  [org.jboss.as.controller]
> (DeploymentScanner-threads - 2) JBAS014774: Service status report
> JBAS014775:    New missing/unsatisfied dependencies:
>       service jboss.naming.context.java.jboss.datasources.PushEEDS (missing)
>       dependents: [service
>       jboss.persistenceunit."ROOT.war#unifiedpush-default", service
>       jboss.persistenceunit."ROOT.war#picketlink-default"]
> 
> 
> 2014/01/26 10:19:19,440 ERROR [org.jboss.as.server.deployment.scanner]
> (DeploymentScanner-threads - 1) {"JBAS014653: Composite operation failed and
> was rolled back. Steps that failed:" => {"Operation step-2" => {"JBAS014771:
> Services with missing/unavailable dependencies" =>
> ["jboss.persistenceunit.\"ROOT.war#unifiedpush-default\"jboss.naming.context.java.jboss.datasources.PushEEDSMissing[jboss.persistenceunit.\"ROOT.war#unifiedpush-default\"jboss.naming.context.java.jboss.datasources.PushEEDS]","jboss.persistenceunit.\"ROOT.war#picketlink-default\"jboss.naming.context.java.jboss.datasources.PushEEDSMissing[jboss.persistenceunit.\"ROOT.war#picketlink-default\"jboss.naming.context.java.jboss.datasources.PushEEDS]"]}}}

It looks like the MySQL cartridge either wasn't added or wasn't able to start successfully. The following command can be used to check the status of the MySQL cartridge:

rhc cartridge status -a APP -c mysql

> OR
> 
> 
> 2014/01/26 05:41:46,522 INFO  [org.jboss.as.connector] (MSC service thread
> 1-1) JBAS010408: Starting JCA Subsystem (JBoss IronJacamar 1.0.9.Final)
> 2014/01/26 05:41:46,922 ERROR [org.jboss.as.controller.management-operation]
> (ServerService Thread Pool -- 62) Operation ("enable") failed - address: ([
>     ("subsystem" => "datasources"),
>     ("data-source" => "PostgreSQLDS")
> ]) - failure description: "JBAS014802: Cannot resolve expression 'expression
> \"jdbcostgresql://${env.OPENSHIFT_POSTGRESQL_DB_HOST}:${env.OPENSHIFT_POSTGRESQL_DB_PORT}/mypush13\"'
> -- java.lang.IllegalStateException: Failed to resolve expression:
> ${env.OPENSHIFT_POSTGRESQL_DB_HOST}"
> 
> 
> OR
> 
> 
> no errors in the logs, but in the deployments directory I found
> 'ROOT.war.dodeploy' and 'ROOT.war.isdeploying' files, but no
> 'ROOT.war.deployed' file.
> 
> 
> Those error seemed to me as random and too strange (by the way, the creation
> of the cartridge is always correct, I never had problems if I created a new
> instance with a never used before name).
> 
> To investigate further, I then tried to do the same steps with a completely
> different application: 'Kitchensink on OpenShift', and I had the same error!
> After the deletion and the recreation of a cartridge with a same name, I
> encountered: 'The requested URL /app was not found on this server.'
> The steps I have done are:
> 1) rhc app create -a kitchensink -t jbossas-7 --from-code
> git://github.com/openshift/kitchensink-example.git
> 2) rhc snapshot save kitchensink
> 3) rhc app delete kitchensink
> 4) rm -rf kitchensinkhtml5/
> 5) rhc app create -a kitchensink -t jbossas-7 --from-code
> git://github.com/openshift/kitchensink-example.git
> 6) Access to the application.
> 
> So the good news is that the error I encountered is not related to this
> specific AeroGear cartridges, but the bad news is that is something more
> general about OpenShift.
> We need to investigate further and find out if there is a way to "flush" the
> deletion, it seems that there is something appended or cached that prevent
> the recreation of a cartridge with a same name.

It's good that the issue you're running into isn't specific to the AeroGear cartridge. I've tried quite a few times today but I'm not able to reproduce the issue with the AeroGear cartridge or with the kitchensink application using the above steps. I suspect that there is indeed an OpenShift timing issue here. Let's check with the OpenShift team on this.

Farah

> We are currently doing some more tests on these process.
> 
> 
> 
> _______________________________________________
> aerogear-dev mailing list
> aerogear-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/aerogear-dev
> 