[jboss-as7-dev] problems with deployment overlays and linking

Tue Sep 4 10:10:37 EDT 2012

Yes, model-wide.

If the tool could retry one-time in case of an optimistic failure, that 
would handle it almost always. The model isn't really meant for high 
volume concurrent updates. Even the case of doing a read and then a 
write without human think time in the middle and then getting a failure 
should be extremely rare. Still, your point is valid.

On 9/4/12 9:04 AM, Alexey Loubyansky wrote:
> I guess, it's model-wide hash, not specific to a path? It might not be
> effective if there was a totally unrelated change to rollback a request.
>
> On 09/04/2012 04:01 PM, Brian Stansberry wrote:
>> As part of our solution for audit logging, we're planning on creating a
>> hash of the model, which would go into the audit log. In the domain
>> case, hashes, one of the domain wide model and then each host has a hash
>> of its host-specific stuff. This *may* prove useful in providing a kind
>> of optimistic locking feature similar to the way Hibernate uses a
>> version field. But I haven't thought about it carefully.
>>
>> On 9/4/12 7:27 AM, Alexey Loubyansky wrote:
>>> That'll help, thanks.
>>>
>>> There is one other issue I can mention (which is not as critical as the
>>> original) which makes linking a bit more complicated in the domain mode.
>>> Standalone works fine.
>>>
>>> The linking in the domain mode is done as
>>>
>>> server-group=sg_name/deployment-overlay=overlay_name/deployment=app.war
>>>
>>> deployment-overlay=overlay_name has to be created explicitly with its
>>> own 'add' (since we don't implicitly create non-existing parent nodes).
>>> So, every time a link is added, we have to perform a check whether the
>>> server-group=sg_name/deployment-overlay=overlay_name part exists. If
>>> not, create it. Then proceed with the actual linking.
>>>
>>> It's doable, of course. But from the tools' side, to compose the final
>>> composite op, it will need to perform some intermediary ops. Of course,
>>> adding the non existing parent can and should be put into the same final
>>> composite op for efficiency.
>>>
>>> This is not the only command which behaves this way. For these commands,
>>> it means:
>>> - the final composite request may differ from one execution to another
>>> depending on the state of the model (not only argument values but actual
>>> requested actions);
>>> - the model checks (or other necessary intermediate ops) are performed
>>> not at the same time with the actual final request (i.e. kind of in a
>>> different transaction) which doesn't guarantee the same state when the
>>> actual request is executed;
>>> - the previous one may affect run-time batches, since every command
>>> added to a batch is translated into a DMR request immediately, not right
>>> before the batch is actually run (which would be safer and which I
>>> should do actually).
>>>
>>> Alexey
>>>
>>>
>>> On 09/04/2012 12:40 AM, Stuart Douglas wrote:
>>>> I will fix this to make it work like the deploy operation. Not sure
>>>> if I
>>>> will have time to get to it today.
>>>>
>>>> Stuart
>>>>> Alexey Loubyansky <mailto:alexey.loubyansky at redhat.com>
>>>>> 3 September 2012 10:39 PM
>>>>> Sorry for the late reply, I'll cherry-pick it, ok.
>>>>>
>>>>> One issue that I mentioned and still there is that you can't add a new
>>>>> overlay with content in a single composite operation (first the
>>>>> content
>>>>> should be uploaded then its references can be used in the overlay).
>>>>> So,
>>>>> this command still can't be executed as part of a batch.
>>>>>
>>>>> Alexey
>>>>> _______________________________________________
>>>>> jboss-as7-dev mailing list
>>>>> jboss-as7-dev at lists.jboss.org
>>>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>>>>> Brian Stansberry <mailto:brian.stansberry at redhat.com>
>>>>> 1 September 2012 2:28 AM
>>>>> Do you see value in cherry-picking his patch and adjusting the command
>>>>> first?  The issues you find implementing the command are what led to
>>>>> these changes, so it'd be nice to know your commands work before
>>>>> merging his stuff.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Alexey Loubyansky <mailto:olubyans at redhat.com>
>>>>> 31 August 2012 8:03 PM
>>>>> Great. Once it's merged I'll adjust the deployment-overlay command and
>>>>> submit a pull request.
>>>>>
>>>>> Thanks,
>>>>> Alexey
>>>>>
>>>>>
>>>>> Stuart Douglas <mailto:stuart.w.douglas at gmail.com>
>>>>> 31 August 2012 1:29 PM
>>>>>
>>>>> I have done up an initial implementation of this here
>>>>> https://github.com/jbossas/jboss-as/pull/2989 .
>>>>>
>>>>>
>>>>> Stuart
>>>>>
>>>>>
>>>>>
>>>>> Brian Stansberry <mailto:brian.stansberry at redhat.com>
>>>>> 24 August 2012 2:08 AM
>>>>> On 8/23/12 3:39 AM, Emanuel Muckenhuber wrote:
>>>>>> On 08/23/2012 01:11 AM, Brian Stansberry wrote:
>>>>>>> Sorry for the slow responses on this thread.
>>>>>>>
>>>>>>> I think we should go with Stuart's approach, particularly if
>>>>>>> Alexey is
>>>>>>> comfortable that the CLI can support high level commands associated
>>>>>>> with it.
>>>>>>>
>>>>>> Yeah, i am fine then as well - nevertheless some comments inline :)
>>>>>>
>>>>>
>>>>> Me too. ;)
>>>>>
>>>>>>> On 8/22/12 8:22 AM, Emanuel Muckenhuber wrote:
>>>>>>>> My main concern is regarding the domain mode, since the additional
>>>>>>>> linking does not seem to be necessary to me and just additional
>>>>>>>> steps
>>>>>>>> mgmt clients have to take care of.
>>>>>>>>
>>>>>>>> Whereas the deployment-scanner is a single client for standalone,
>>>>>>>> which
>>>>>>>> is under our control. So i wanted to see if we there is a way we
>>>>>>>> can
>>>>>>>> make this work which also satisfies your use cases.
>>>>>>>>
>>>>>>>> I have two things in mind which i'll try to explain quickly:
>>>>>>>>
>>>>>>>> 1) you still link the overlays as part of the <deployment />,
>>>>>>>> however
>>>>>>>> the deploment-scanner is aware of that and won't remove
>>>>>>>> deployments with
>>>>>>>> overlays until you use a specific invalidation marker.
>>>>>>>>
>>>>>>> This requires that the overlays be controlled by the scanner as
>>>>>>> well.
>>>>>>> Why? Because we don't persist the model resource for
>>>>>>> scanner-controlled
>>>>>>> deployments. So if a user linked a non-scanner-controlled overlay
>>>>>>> to a
>>>>>>> scanner controlled deployment (which seems quite rational if the
>>>>>>> overlay
>>>>>>> involves some environmental settings that never change), the link
>>>>>>> information will be lost on server reboot.
>>>>>>>
>>>>>> I know, but that's an ancient limitation. I guess we could just
>>>>>> start to
>>>>>> persist stuff.
>>>>>>
>>>>>
>>>>> We'd have to persist some association between a deployment and a
>>>>> scanner
>>>>> (e.g. scanner="default" on the deployment element) and then use
>>>>> that to
>>>>> indicate that the server should not auto-deploy on boot but should
>>>>> defer
>>>>> to the scanner to tell it what to do. And then alter the scanner to
>>>>> add
>>>>> logic for it to reconcile the state it sees in the scan directory vs
>>>>> what the model shows.
>>>>>
>>>>> Doable, but a fair bit of work.
>>>>>
>>>>> I considered this before going with the "don't persist" approach. I
>>>>> don't recall though if there was some fundamental problem with it that
>>>>> I'm not remembering, or if it was solely that "don't persist" was
>>>>> doable
>>>>> within the time constraints.
>>>>>
>>>>>>>> 2) use overlay directories - this is more the file based approach,
>>>>>>>> Where
>>>>>>>> we just pick the "deployment.name + suffix" for the overlay. So
>>>>>>>> there
>>>>>>>> would be an automatic overlay/linking done by the
>>>>>>>> deployment-scanner.
>>>>>>>>
>>>>>>> Same issue as 1).
>>>>>>>
>>>>>> Not really. It can automatically recreate the overlays therefore it
>>>>>> does
>>>>>> not require persistent information.
>>>>>>
>>>>>
>>>>> It can only recreate the association for overlays that it scans, not
>>>>> ones that were added via CLI/console.
>>>>>
>>>>>>>> 2.1) the overlay directory could be just a link to a virtual
>>>>>>>> file (a
>>>>>>>> shared <deployment-overlay />) [1] - so in this case the overlays
>>>>>>>> are
>>>>>>>> managed by the model and are not linked to the deployment
>>>>>>>> lifecycle.
>>>>>>>>
>>>>>>> I confess I don't understand this. In particular, how does it
>>>>>>> fundamentally differ from what Stuart proposes in terms of how to
>>>>>>> organize the data?
>>>>>>>
>>>>>> Well that is the point. It's basically the same, just that it removes
>>>>>> the requirement for the additional linking in the domain.
>>>>>>
>>>>>
>>>>> Ok, then I still don't understand it. If it solves a problem I'd
>>>>> like to
>>>>> understand. Can you explain further?
>>>>>
>>>>>>>> On 08/22/2012 01:03 AM, Stuart Douglas wrote:
>>>>>>>>> So what are we going to do here? Personally I much prefer the
>>>>>>>>> idea of having this stored in the model and linking it to a
>>>>>>>>> deployment, so the overlay lifecycle is not tied to that of the
>>>>>>>>> deployment.
>>>>>>>>>
>>>>>>>>> Another idea I had is that we could do something like define the
>>>>>>>>> overlays at domain level, but then define the links at server
>>>>>>>>> group or server level using the same name. basically something
>>>>>>>>> like:
>>>>>>>>>
>>>>>>>>> /deployment-overlay=myOverlay/content=WEB-INF/web.xml
>>>>>>>>> /deployment-overlay=myOverlay/content=WEB-INF/ejb-jar.xml
>>>>>>>>>
>>>>>>>>> /server-group=server-group1/deployment-overlay=myOverlay/deployment=test.war
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> /server-group=server-group1/deployment-overlay=myOverlay/deployment=*.war
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> For standalone they would just be combined, similar to what I had
>>>>>>>>> below:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> /deployment-overlay=myOverlay/content=WEB-INF/web.xml
>>>>>>>>> /deployment-overlay=myOverlay/content=WEB-INF/ejb-jar.xml
>>>>>>>>> /deployment-overlay=myOverlay/deployment=test.war
>>>>>>>>> /deployment-overlay=myOverlay/deployment=*.war
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Stuart
>>>>>>>>>
>>>>>>>>> On 17/08/2012, at 11:03 PM, Emanuel
>>>>>>>>> Muckenhuber<emuckenh at redhat.com>  wrote:
>>>>>>>>>
>>>>>>>>>> On 08/17/2012 02:52 AM, Stuart Douglas wrote:
>>>>>>>>>>> The big problem with this is that it does not work with the
>>>>>>>>>>> deployment scanner.
>>>>>>>>>>>
>>>>>>>>>> Yeah, the deployment-scanner is always a story for itself.
>>>>>>>>>> However i think we could actually make it persist information
>>>>>>>>>> and be aware of overlays.
>>>>>>>>>>
>>>>>>>>>> Having overlays as part of the deployment means the lifecycle is
>>>>>>>>>> managed together, so the deployment-scanner could define it's
>>>>>>>>>> own invalidation policy. May it be based on whether the content
>>>>>>>>>> got removed or we have a specific marker for that.
>>>>>>>>>>
>>>>>>>>>> We could also think of providing a file-based overlay solution,
>>>>>>>>>> managed by the deployment scanner only. In the end people use
>>>>>>>>>> the deployment-scanner for a reason, so perhaps not having to
>>>>>>>>>> worry about the model or "content" folder could be interesting.
>>>>>>>>>> This would not even need to access persistent information
>>>>>>>>>> though. Anyway just some ideas on the side.
>>>>>>>>>>
>>>>>>>>>>> It also has to be specified on every re-deploy.
>>>>>>>>>> When you do a "remove" and "add" then yes. We do have specific
>>>>>>>>>> operations to replace the content and redeploy the deployment
>>>>>>>>>> without removing the information from the model.
>>>>>>>>>>
>>>>>>>>>>> I think the simplest solution here
>>>>>>>>>>> is to just move the links under the deployment-overlays element
>>>>>>>>>>> and loose the
>>>>>>>>>>> flexibility to link at different levels. e.g:
>>>>>>>>>>>
>>>>>>>>>>> /deployment-overlay=myOverlay/deployment=test.war
>>>>>>>>>>> /deployment-overlay=myOverlay/deployment=*.jar
>>>>>>>>>>>
>>>>>>>>>> Yes, i was not really going for the simplest solution for now -
>>>>>>>>>> this would most likely would be it :)
>>>>>>>> _______________________________________________
>>>>>>>> jboss-as7-dev mailing list
>>>>>>>> jboss-as7-dev at lists.jboss.org
>>>>>>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>>>>>>>>
>>>>>> _______________________________________________
>>>>>> jboss-as7-dev mailing list
>>>>>> jboss-as7-dev at lists.jboss.org
>>>>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>>>>>>
>>>>>
>>>>>
>>
>>

-- 
Brian Stansberry
Principal Software Engineer
JBoss by Red Hat