Hi,<br><br>On start the root node is created with an empty map. I've changed it to be created with a sql null value rather than an empty map.<br>This way we'll stay consistent with the nodes added indirectly(as they are parents of nodes that are specifically added).
<br>I also hope this will solve extending JDBCacheLoader problem, as I think that for deserializaion the TransformingJDBCCacheLoader knows how to handle DB nulls.<br><br>Cheers,<br>Mircea<br><br><div><span class="gmail_quote">
On 3/4/07, <b class="gmail_sendername">Galder Zamarreno</b> <<a href="mailto:galder.zamarreno@redhat.com" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">
galder.zamarreno@redhat.com
</a>> wrote:</span><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">I've got this working, including some basic unit tests and manual
<br>
examples to transform entired cache stores from 1.x data to 2.x. These<br>last manual examples include source code, 1.x cache stores (file and<br>jdbc derby db) and sample cache configurations.<br><br>Apart from the MV issue referred earlier, I have realised that
<br>TransformingJDBCCacheLoader will have to extend JDBCCacheLoaderOld<br>instead of JDBCCacheLoader.<br><br>The reason is because when JDBCCacheLoader starts, if the root does not<br>exist, it'll create it, which in the TransformingJDBCCacheLoader will
<br>mean creating it in 2.x format.<br><br>This wouldn't be a problem if the root node didn't need querying again,<br>but if customers want to migrate their data, they will start looping<br>from the node (cache.getRoot
()) and the first thing they'll get its<br>children. This results in trying to load the root node from the cache<br>store which breaks, as we're reading from db in 1.x format.<br><br>Remember that the TransformingJDBCCacheLoader reads in
1.x format and<br>stores in 2.x format.<br><br>This has a very easy resolution which is extending JDBCCacheLoaderOld.<br>After that, it works like a treat :).<br><br>Manik, assuming you're happy with the original idea, would extending
<br>JDBCCacheLoaderOld for this one off cache loader be ok with you?<br><br>Galder Zamarreno wrote:<br>> I haven't touched this issue for a couple of weeks and over the last<br>> couple of days I had the chance to get back into it.
<br>><br>> After discussions with Brian we came up with a different approach for this.<br>><br>> My previous approach (don't need to read all below) relied on<br>> introducing legacy code into the main source code that would be able to
<br>> read 1.x serialization. As I started doing it, I realised that it would<br>> need a lot of changes and it would clutter the 2.0 codebase.<br>><br>> Instead, with the help of Brian, we came up with a different idea, which
<br>> is creating two one-off cache loaders, TransformingJDBCCacheLoader and<br>> TransformingFileCacheLoader. They extend the existing cache loaders, but<br>> they differentiate by unmarshall stuff in the 1.x way.
<br>><br>> This way, we have cache loaders that can read in 1.x way and write in<br>> 2.x way. Now, a customer just needs to write a program that uses a cache<br>> configured to use any of these two cache loaders above and all it has to
<br>> do is loop through the tree reading all nodes and putting them back, and<br>> voila! you have your data store format changed (I'll be writing an<br>> example of this).<br>><br>> It's a pretty clean solution to transforming data without making changes
<br>> to main o.j.cache tree.<br>><br>> But, there's always a but :), 1.4.x used<br>> org.jboss.invocation.MarshalledValue so there's no way of getting around<br>> the need of having this class to do this. This is because
<br>> JDBCCacheLoader stored instances of MarshalledValue, so even the MV<br>> class in AOP would not work cos it's a different package (it'd result in<br>> CCE)<br>><br>> One thing Brian suggested is that these two cache loaders and
<br>> jboss-minimal are kept in a separate dir structure to the main one and<br>> when we distribute, we provide an extra jar containing these that can be<br>> used to transform data and that's it. After that, you get rid of it, you
<br>> go back to the standard libraries.<br>><br>> It's pretty hard to find a neater way of dealing with this but the<br>> benefits are worth it, customer's data stays alive!<br>><br>> Manik and the rest, thoughts?
<br>><br>> Galder Zamarreno wrote:<br>> > Manik Surtani wrote:<br>> >> On 5 Feb 2007, at 19:57, Galder Zamarreno wrote:<br>> >><br>> >>> Quick (but a bit lengthy :( ) update on this:
<br>> >>><br>> >>> - I've created a new Marshaller called Legacy1xMarshaller (anyone's<br>> >>> got a better name?) which extends o.j.c.m.AsbtractMarshaller that<br>> >>> would do the job of marshalling stuff in the
1.x fashion. This is to<br>> >>> be used by JDBCCacheLoader and FileCacheLoader if configured to use<br>> >>> 1.x marshalling. This has the benefit that the code in these cache<br>> >>> loaders only have to do getMarshaller().whatever... , making it very
<br>> >>> simple to switch from VAM to Legacy Marshaller.<br>> >><br>> >> I presume the VAM would transparently flip between marshallers, based<br>> >> on the version short at the head of the stream?
<br>> ><br>> > The problem is that 1.x marshalling for cache loaders did not have<br>> > version numbers at the start, it was plain java serialization. Can you<br>> > expect VAM to detect that? That's why I thought of a Marshaller instance
<br>> > in AbstractCacheLoader that would either use VAM or the Legacy one. We<br>> > could however assume that if VAM does not find version number, it tries<br>> > to use Legacy one.<br>> ><br>
> > As you said later in the email, it seems like 1.4.x dealt with this<br>> > similar situation. I'll look at it.<br>> ><br>> >><br>> >>><br>> >>> - In order to do this, I need to add a new method to
<br>> >>> o.j.c.m.Marshaller called objectToStream(OutputStream). The reason<br>> >>> for doing is so that FileCacheLoader just needs to call<br>> >>> getMarshaller().objectToStream() when it's trying to store data. This
<br>> >>> will avoid having an if statement in storeAttributes() checking which<br>> >>> Marshaller is used, and calling objectToObjectStream with the<br>> >>> corresponding ObjectOutpuStream.
<br>> >><br>> >> Again, isn't this already in the VAM?<br>> ><br>> > Not for OutputStream. You have objectToObjectStream(Object obj,<br>> > ObjectOutputStream out) and objectFromStream(InputStream is), but not
<br>> > objectToStream for OutputStreams such as FileOutputStream.<br>> ><br>> >><br>> >>><br>> >>> - The decision maker for which Marshaller to use is to be done in<br>> >>> AbstractCacheLoader which will store the Marshaller used by
<br>> >>> CacheLoader. getMarshaller() would decide upon configuration, which<br>> >>> Marshaller to use, whether the default cache.getMarshaller() which is<br>> >>> VAM or the legacy one, making it quite clean to switch from to
<br>> another.<br>> >><br>> >> Look at the VAM in the 1.4.x tree - it deals with "legacy support" to<br>> >> deal with JBC 1.2.x and 1.3.x for RPC calls. (removed in 2.x since<br>
> >> the legacy support was no longer needed). Could easily be<br>> >> re-introduced if needed to supportr legacy marshalling for CLs.<br>> ><br>> > Ok, i'll definitely have a look at that.
<br>> ><br>> >><br>> >>><br>> >>> - Configuration wise, I created Legacy1xMarshallingCacheLoaderConfig<br>> >>> (I couldn't come up with a better name!) which extends
<br>> >>> IndividualCacheLoaderConfig. JDBCCacheLoaderConfig and<br>> >>> FileCacheLoaderConfig will extend<br>> >>> Legacy1xMarshallingCacheLoaderConfig instead.<br>> >><br>
> >> Could drop the 1x in the name, I suppose? :-)<br>> ><br>> > No probs :)<br>> ><br>> >><br>> >>><br>> >>> - Inside Legacy1xMarshallingCacheLoaderConfig, I search for
<br>> >>> cache.loader.marshalling.1.x (name again!) boolean property in the<br>> >>> <properties> section. If true, it uses legacy marshalling, and if<br>> >>> false, which is default value, VAM.
<br>> >>><br>> >>> - I have extended CacheLoaderTestsBase to create<br>> >>> FileCacheLoaderLegacyMarshallingTest which tests the FileCacheLoader<br>> >>> with legacy marshalling. I'll be doing the same for JDBCCacheLoader.
<br>> >>><br>> >>> - Finally and one of the most important aspects, previous marhalling<br>> >>> relies on these classes:<br>> >>><br>> >>> org.jboss.invocation.MarshalledValue
;<br>> >>> org.jboss.invocation.MarshalledValueInputStream;<br>> >>><br>> >>> Which used to be located in jboss-minimal.jar in 1.x. There's v<br>> >>> similar classes in AOP but not the same, so I'm gonna be creating a
<br>> >>> legacy directory in lib with this library. To avoid compile time<br>> >>> dependency, Legacy1xMarshaller will be instantiated via reflection,<br>> >>> so only people who actually use this will need this library. The
<br>> >>> library has no conflicts with existing 2.x libraries.<br>> >><br>> >> Look at the jboss-common-core jar and particularly JBCOMMON-8 in JIRA.<br>> >><br>> ><br>> > So, did you test whether you could read data written with
<br>> > JDBCCacheLoader wiht MV classes with a JDBCCacheLoader not using MV<br>> > classes? That's one of the tests I wanted to do to see whether this<br>> > classes were necessary.<br>> >
<br>
> > jboss-common-core.jar contains MarshalledValueOutputStream and<br>> > MarshalledValueInputStream so that wouldn't be a problem for FCL.<br>> > JDBCCacheLoader on the contrary, wrapped the node in MarshalledValue and
<br>> > the wrote it as an ObjectOutputStream. I'll look at the commons code to<br>> > see whether it's the same which I guess might be.<br>> ><br>> > There's a MarshalledValue in aop libraries but quick glance at the code
<br>> > showed that it's slightly different.<br>> ><br>> >>><br>> >>> The last problem is that these two classes access<br>> >>> org.jboss.logging.Logger that used to be in
jboss-common.jar. Now<br>> >>> this jar certainly classes with jboss-common-core.jar in 2.x, so<br>> >>> what's I've done is get jboss-logging-spi.jar <a href="http://2.0.2.GA" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">
2.0.2.GA</a>
and put it in<br>> >>> the legacy directory.<br>> >>><br>> >>> So, we end up having two legacy libraries in lib/legacy but they're<br>> >>> only needed at runtime if using
1.x marhalling. I guess it's the<br>> >>> price to pay to make customer's life a bit easier.<br>> >>><br>> >><br>> >> Trying to avoid a legacy jar dir ... like I said, see if the MV and
<br>> >> MVIS can be in jboss-common-core (without JBoss Logging deps!)<br>> ><br>> > Yeah defo, we wanna avoid any legacy jars.<br>> ><br>> >><br>> >>> The other alternative would be for
1.x marshaller not to use this<br>> >>> org.jboss.invocation.* classes and just write to Object streams but I<br>> >>> think these classes have an impact in the format of the marshalled<br>> >>> data. Brian, do you know a bit more about the role of these classes?
<br>> >>><br>> >>> A bit more complicated than initially expected but I can't see any<br>> >>> easier way of providing backwards compatibility. Hopefully we should<br>> >>> be able to phase it out asap,
3.x? :)<br>> >>><br>> >>> What this has shown as well is how different CacheLoaders marshalled<br>> >>> things in a slightly different way which makes having a common<br>> >>> framework for this even more necessary,
i.e. VAM. :D<br>> >>><br>> >>> Hope you're not snoring by now ;)<br>> >>><br>> >>> If you have better ideas for the naming I used, speak up :)<br>> >>>
<br>
> >>> Galder Zamarre�o<br>> >>> Sr. Software Maintenance Engineer<br>> >>> JBoss, a division of Red Hat<br>> >>><br>> >>> -----Original Message-----<br>> >>> From:
<a href="mailto:jbosscache-dev-bounces@lists.jboss.org" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">jbosscache-dev-bounces@lists.jboss.org</a><br>> >>> [mailto:<a href="mailto:jbosscache-dev-bounces@lists.jboss.org" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">
jbosscache-dev-bounces@lists.jboss.org
</a>] On Behalf Of Galder<br>> >>> Zamarreno<br>> >>> Sent: 31 January 2007 01:01<br>> >>> To: Manik Surtani<br>> >>> Cc: <a href="mailto:jbosscache-dev@lists.jboss.org" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">
jbosscache-dev@lists.jboss.org</a><br>> >>> Subject: RE: [jbosscache-dev] migrating data stored in 1.x format to<br>> >>> VAM format<br>> >>><br>> >>> +1, VAM should be the default.
<br>> >>><br>> >>> Only people who are resilient to change their existing stores to VAM<br>> >>> should use the 1.x option, which would need explicitly definition.<br>> >>>
<br>> >>> Galder Zamarre�o<br>> >>> Sr. Software Maintenance Engineer<br>> >>> JBoss, a division of Red Hat<br>> >>><br>> >>><br>> >>> -----Original Message-----
<br>> >>> From: Manik Surtani [mailto:<a href="mailto:manik@jboss.org" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">manik@jboss.org</a>]<br>> >>> Sent: 30 January 2007 22:55
<br>> >>> To: Galder Zamarreno<br>> >>> Cc:
<a href="mailto:jbosscache-dev@lists.jboss.org" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">jbosscache-dev@lists.jboss.org</a><br>> >>> Subject: Re: [jbosscache-dev] migrating data stored in
1.x format to<br>> >>> VAM format<br>> >>>
<br>> >>> I see what you mean, although I would like the default to be to use<br>> >>> the VAM.<br>> >>><br>> >>> --<br>> >>> Manik Surtani<br>> >>>
<br>> >>> Lead, JBoss Cache<br>> >>> JBoss, a division of Red Hat<br>> >>><br>> >>> Email: <a href="mailto:manik@jboss.org" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">
manik@jboss.org</a><br>> >>> Telephone: +44 7786 702 706
<br>> >>> MSN: <a href="mailto:manik@surtani.org" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">manik@surtani.org</a><br>> >>> Yahoo/AIM/Skype: maniksurtani<br>> >>>
<br>> >>><br>> >>><br>> >>> On 30 Jan 2007, at 20:45, Galder Zamarreno wrote:
<br>> >>><br>> >>>> Actually, the more I think about this, the less I like the idea of<br>> >>>> switching the marshalling from 1.x to 2.x at the CacheLoaders<br>> >>>> level, or at least forcing them to do so.
<br>> >>>><br>> >>>> Customers that want to use JBossCache 2.x might be reluctant to<br>> >>>> migrate their data from one format to the other. I can see how an<br>> >>>> existing customer might think this is a proper pain in the ass,
<br>> >>>> independent of the benefits, and might reduce adoption among them.<br>> >>>><br>> >>>> We want to remove barriers upgrading, but at the same time, we want<br>> >>>> new customer to use new marshalling, so I'd actually implement the
<br>> >>>> possibility to use 1.x marshalling which is plan java serialization<br>> >>>> at the CacheLoader level. This could easily achieved adding a<br>> >>>> property to the <properties> section.
<br>> >>>><br>> >>>> Just note that this does not apply to the marshalling done at<br>> >>>> replication level as there's no hard data that needs migrating.<br>> >>>>
<br>> >>>> Galder Zamarre�o<br>> >>>> Sr. Software Maintenance Engineer<br>> >>>> JBoss, a division of Red Hat<br>> >>>><br>> >>>> -----Original Message-----
<br>> >>>> From: <a href="mailto:jbosscache-dev-bounces@lists.jboss.org" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">jbosscache-dev-bounces@lists.jboss.org</a> [mailto:<a href="mailto:jbosscache-dev-" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">
jbosscache-dev-</a><br>> >>>> <a href="mailto:bounces@lists.jboss.org" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">
bounces@lists.jboss.org</a>] On Behalf Of Galder Zamarreno<br>> >>>> Sent: 25 January 2007 13:07<br>> >>>> To: <a href="mailto:jbosscache-dev@lists.jboss.org" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">
jbosscache-dev@lists.jboss.org
</a><br>> >>>> Subject: [jbosscache-dev] migrating data stored in 1.x format to<br>> >>>> VAM format<br>> >>>><br>> >>>> Hi all,<br>> >>>><br>
> >>>> I'm deferring <a href="http://jira.jboss.com/jira/browse/JBCACHE-879" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">http://jira.jboss.com/jira/browse/JBCACHE-879</a> to
<br>> >>>> BETA2 because I still need to write this: <a href="http://jira.jboss.com/" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">
http://jira.jboss.com/</a><br>> >>>> jira/browse/JBCACHE-882<br>> >>>><br>> >>>> The reason I'm deferring it is because I can't see a<br>> >>>> straightforward way of doing such thing right now. Ideally, you
<br>> >>>> should be able run a 1.x version (cache1) and a 2.x version<br>> >>>> (cache2) of JBC in the same VM so that you can do a loop of<br>> >>>> cache1.get() and call
cache2.put(). However, I have doubts that<br>> >>>> that this approach will be free of class loading issues. What do<br>> >>>> you think?<br>> >>>><br>> >>>> I was wondering whether Region based could help here, but I can't
<br>> >>>> see right now how this could be done.<br>> >>>><br>> >>>> Something I had in mind is having the capability of to start a<br>> >>>> cache with either
1.x marshalling or VAM marshalling, but oriented<br>> >>>> at being used only at the cache loader level. It wouldn't make much<br>> >>>> sense for replication because there's no hard data there.
<br>> >>>><br>> >>>><br>> >>>> I thought that you could start two instances of cache 2.x, first<br>> >>>> with 1.x. marshalling and the other one with VAM both pointing to
<br>> >>>> different JDBCCacheLoader stores. You could then get from the first<br>> >>>> using normal mmarshalling and put in the second one which has VAM<br>> >>>> marshalling active, what do you think?
<br>> >>>><br>> >>>> If you like the approach, I should be have it ready by BETA2.<br>> >>>><br>> >>>> This last approach looks simpler to me, what do you think?
<br>> >>>><br>> >>>> Galder Zamarre�o<br>> >>>> Sr. Software Maintenance Engineer<br>> >>>> JBoss, a division of Red Hat<br>> >>>><br>> >>>>
<br>> >>>><br>> >>>> _______________________________________________<br>> >>>> jbosscache-dev mailing list<br>> >>>> <a href="mailto:jbosscache-dev@lists.jboss.org" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">
jbosscache-dev@lists.jboss.org</a><br>> >>>> <a href="https://lists.jboss.org/mailman/listinfo/jbosscache-dev" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">https://lists.jboss.org/mailman/listinfo/jbosscache-dev
</a><br>> >>>><br>> >>>> _______________________________________________
<br>> >>>> jbosscache-dev mailing list<br>> >>>> <a href="mailto:jbosscache-dev@lists.jboss.org" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">jbosscache-dev@lists.jboss.org
</a><br>> >>>> <a href="https://lists.jboss.org/mailman/listinfo/jbosscache-dev" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">
https://lists.jboss.org/mailman/listinfo/jbosscache-dev</a><br>> >>><br>> >>><br>> >>><br>> >>> _______________________________________________<br>> >>> jbosscache-dev mailing list
<br>> >>> <a href="mailto:jbosscache-dev@lists.jboss.org" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">jbosscache-dev@lists.jboss.org</a><br>> >>> <a href="https://lists.jboss.org/mailman/listinfo/jbosscache-dev" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">
https://lists.jboss.org/mailman/listinfo/jbosscache-dev
</a><br>> >><br>> ><br>> ><br>><br><br>--<br>Galder Zamarre�o<br>Sr. Software Maintenance Engineer<br>JBoss, a division of Red Hat<br><br>_______________________________________________<br>jbosscache-dev mailing list
<br><a href="mailto:jbosscache-dev@lists.jboss.org" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">jbosscache-dev@lists.jboss.org</a><br><a href="https://lists.jboss.org/mailman/listinfo/jbosscache-dev" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">
https://lists.jboss.org/mailman/listinfo/jbosscache-dev</a><br></blockquote>
</div><br>