[JBoss JIRA] (ISPN-3318) Migrate data from one cache store to another
by Galder Zamarreño (JIRA)
[ https://issues.jboss.org/browse/ISPN-3318?page=com.atlassian.jira.plugin.... ]
Galder Zamarreño commented on ISPN-3318:
----------------------------------------
Interesting idea from Randall:
{quote}Let's look at this from a user's perspective. Is it possible that the new file store **replaces** the old one, so that the user doesn't (really) need to know the difference?
How similar are the configuration properties of each? Might it be possible to:
rename and deprecate the existing one (e.g., "org.infinispan.loaders.file.LegacyFileCacheStore"), changing it to 'package' or 'protected' (iff the new one is stable enough that the old one should definitely not be used anymore),
rename the new one to "org.infinispan.loaders.file.FileCacheStore" (so that existing configurations work without change),
have some code in the new one detect the old file structure and automatically run a migration to convert from the old file structure to the new format (and possibly move the old files into a subdirectory)
If this is possible, this is by far the best option for users since it provides a simple (perhaps even transparent) migration path with little if any changes required to configurations, but at the same time it allows the developers to get what they want: a new FCS that replaces the old one (which goes away).{quote}
> Migrate data from one cache store to another
> --------------------------------------------
>
> Key: ISPN-3318
> URL: https://issues.jboss.org/browse/ISPN-3318
> Project: Infinispan
> Issue Type: Task
> Components: Loaders and Stores
> Reporter: Galder Zamarreño
> Assignee: Galder Zamarreño
> Fix For: 6.0.0.Final
>
>
> Find a generic way to transfer data from one cache store to another, which could involve different Infinispan versions. This is handy to migrate file cache store based users to single file cache store (ISPN-2806).
> Ideally, this should be added as a recipe for rolling upgrades.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years, 9 months
[JBoss JIRA] (ISPN-761) Cache.keySet(), entrySet(), values(), size() ignore contents of cache loader
by William Burns (JIRA)
[ https://issues.jboss.org/browse/ISPN-761?page=com.atlassian.jira.plugin.s... ]
William Burns commented on ISPN-761:
------------------------------------
Continuation from previous comment
{quote}
* mmarkus has quit (Quit: Leaving.)
<wburns> sannegrinovero: actually since we are on size now - the problem with just returning a count I have found is the Flag can mess with that optimization
<sannegrinovero> wburns: ? how can the Flag mess with a SELECT COUNT(*) ??
<wburns> sannegrinovero: passivation enabled you would add memory store and cache store size
<wburns> sannegrinovero: no passivation you would just add the cache store size
<wburns> sannegrinovero: however the no passivation if you use the flag the cache store being a superset is no longer true
<sannegrinovero> wburns: right.depending if it's passivation or not the math to be used is different, but not hard.
<sannegrinovero> (either you add the size of the container or you don't)
<wburns> sannegrinovero: that isn't true when the flag is used
<sannegrinovero> BTW this will never be an exact value, that's not possible. This is needed for statistical purposes only.
* pilhuhn has quit (Quit: Computer has gone to sleep.)
<sannegrinovero> wburns: which Flag are you talking about?
<wburns> sannegrinovero: SKIP_CACHE_STORE on a put operation for a new key
<wburns> sannegrinovero: the exact value portion is the discussion I was looking for, do we care about this case with the Flag
<wburns> sannegrinovero: cause the only way to be 100% correct is to get all the keys from the memory store and the cache store and return the size of the union
<wburns> in the no passivation one
<wburns> sannegrinovero: however if we ignore the flag usage then we can just return the size of the cache store
<sannegrinovero> wburns: well no we don't care
* mmarkus (~Adium@redhat/jboss/mmarkus) has joined #infinispan
* ChanServ gives channel operator status to mmarkus
<mmarkus> sannegrinovero: my connection dropped
<mmarkus> sannegrinovero: wburns: back now
<wburns> mmarkus: whoops you missed the good parts :P
<wburns> sannegrinovero: okay that was mmarkus feelings as well
<sannegrinovero> mmarkus: np I think the interesting discussion finished. I'll add it all to JIRA ok? So we don't lose this again as I'm sure it was discussed on the mailing list already.
<wburns> mmarkus and I I meant
<sannegrinovero> +1
* Alkpone has quit (Quit: Leaving.)
<sannegrinovero> if you use flags you need to deal with the consequences ;-)
<wburns> sannegrinovero: :-)
* tsykora__ has quit (Ping timeout: 264 seconds)
<mmarkus> sannegrinovero: wburns: thinking more about it I think sannegrinovero has a good point regarding keySet() and entries()
<mmarkus> for Cache.keySet() we want to return a Set that only allows invoking Set.iterator on it
<wburns> mmarkus: I agree with his points, but which that would affect 761 right now?
<mmarkus> and this iterator would only load data lazily
<mmarkus> sannegrinovero: ^^ I guess that's what you have in mind
<wburns> mmarkus: so basically like the Streams from Java 8?
<sannegrinovero> mmarkus: yea. I think we should work on this in tandem with ISPN-3290 : you have the opportunity now to add/adjust some methods there and should just do it.
<jbossbot> jira [ISPN-3290] Redesign CacheStore API [Open (Unresolved) Task, Critical, Mircea Markus] https://issues.jboss.org/browse/ISPN-3290
<sannegrinovero> mmarkus: TBH I don't think ISPN-761 is as important as Map/Reduce .. but it's the same problem so we'd better consider them both as "customers" for the above issue.
<jbossbot> jira [ISPN-761] Cache.keySet(),entrySet(),values(),size() ignore contents of cache loader [Pull Request Sent (Unresolved) Bug, Critical, William Burns] https://issues.jboss.org/browse/ISPN-761
<mmarkus> sannegrinovero: so we'd need an interable cache store
* rachmatowicz has quit (Quit: Ex-Chat)
<sannegrinovero> mmarkus: there is another important customer actually: CacheStore performance: when using DIST and starting up a new node you need to load all entries, but only _owned_ entities are of interest.
* rachmatowicz (~nrla(a)modemcable117.232-202-24.mc.videotron.ca) has joined #infinispan
<sannegrinovero> mmarkus: not sure if we had Iterable in mind. We definitely discussed this on ML and had several good ideas. need to find that.
<sannegrinovero> mmarkus: but agreed it definitely is something like that, conceptually.
<mmarkus> sannegrinovero: right
<mmarkus> sannegrinovero: not sure we'd be able to make all the stores "iterable" though
<mmarkus> sannegrinovero: I mean the underlying implementation to allow that
<wburns> mmarkus: should we do something inline with Java 8?
* hagarth has quit (Ping timeout: 276 seconds)
<sannegrinovero> mmarkus: that's why I'm starting to buy in your idea to move some CacheStore(s) out of the repository :P
<mmarkus> wburns: I think that would make sense
<mmarkus> sannegrinovero: well for those that are not iterable we can still stick to the existing approach
* jbossbot has quit (Read error: Operation timed out)
<sannegrinovero> mmarkus: some implementations might not be able to implement all optimisations (needs to be optional) and also we might want to update some CacheStores after the Infinispan 6.0 deadlines.. relax the release dates for non product components.
<sannegrinovero> mmarkus: +100
<sannegrinovero> mmarkus: but also don't bother updating all of them at the same time.
* pmuir (~pmuir@redhat/jboss/pmuir) has joined #infinispan
<pmuir> galderz: ping
<mmarkus> sannegrinovero: indeed
<mmarkus> then back to the map.entries and map.keys problem
* vchepeli has quit (Quit: Leaving.)
<wburns> mmarkus: sannegrinovero: why couldn't the non interable ones basically just be wrapped with an iterable interface and they are just documented as possible OOM stores :P
<sannegrinovero> mmarkus: I guess a good plan would be to start moving out the stores from the repo, so that then you can experiment with SPI changes without too much update overhead.
<mmarkus> wburns: read my mind :-)
<sannegrinovero> wburns: +1 not all cachestores are fit for this. Ideally we'd make a table on the doc listing which ones support certain "smarter" features and which ones are more limited.
<wburns> mmarkus: sannegrinovero: and actually we can do that very easily when moving to Java 8 when the stream methods are added to collections and they do all the hard work for us :)
<wburns> mmarkus: sannegrinovero: too bad we won't be on Java 8 until who knows when, haha
<sannegrinovero> wburns: right! but that will take some time. Hopefully quick enough for the various cachestores to update their APIs as well so that we can try and find similarities.
<sannegrinovero> mmarkus: wburns: cool discussions :) missed you guys.
<mmarkus> wburns: sannegrinovero: okay then. So the current implementation Will is working on in the scope of ISPN-3290 will make sense in the future as well, for the CS that are not iterable
<wburns> mmarkus: back into reality: what do want for tomorrow? :P
<mmarkus> once we add iterable cache stores, we'll be able to enhance the implementation for these stores to be much more memory efficient
<sannegrinovero> mmarkus: ok good point, it's not worthless code.
<mmarkus> wburns: +1 to move on with your impl for ISPN-761 and then extend it in the scope of ISPN-3290
<sannegrinovero> wburns: I think you should explore how the CacheStore SPI should look like. You need to study both your needs (Infinispan Core) and the implementation options you have (I'd start from the CacheStores we support)
<mmarkus> wburns: move on == document limitations and integrate it
{quote}
> Cache.keySet(),entrySet(),values(),size() ignore contents of cache loader
> -------------------------------------------------------------------------
>
> Key: ISPN-761
> URL: https://issues.jboss.org/browse/ISPN-761
> Project: Infinispan
> Issue Type: Bug
> Components: Loaders and Stores
> Affects Versions: 4.2.0.BETA1
> Reporter: Paul Ferraro
> Assignee: William Burns
> Priority: Critical
> Labels: onboard, potential_performance_hit
> Fix For: 6.0.0.Alpha1, 6.0.0.Final
>
>
> Passivated cache entries are not represented in values returned by the keySet(), entrySet(), values(), and size() Cache methods. This results in inconsistent behavior, since it is possible that a given cache key may not be contained in keySet() even though Cache.get(...) would return a non-null value if the entry was previously passivated.
> I think CacheLoaderInterceptor needs to implement the following methods:
> Object visitSizeCommand(InvocationContext ctx, SizeCommand command) throws Throwable
> Object visitValuesCommand(InvocationContext ctx, ValuesCommand command) throws Throwable
> Object visitEntrySetCommand(InvocationContext ctx, EntrySetCommand command) throws Throwable
> Object visitKeySetCommand(InvocationContext ctx, KeySetCommand command) throws Throwable
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years, 9 months
[JBoss JIRA] (ISPN-3318) Migrate data from one cache store to another
by Galder Zamarreño (JIRA)
Galder Zamarreño created ISPN-3318:
--------------------------------------
Summary: Migrate data from one cache store to another
Key: ISPN-3318
URL: https://issues.jboss.org/browse/ISPN-3318
Project: Infinispan
Issue Type: Task
Components: Loaders and Stores
Reporter: Galder Zamarreño
Assignee: Galder Zamarreño
Fix For: 6.0.0.Final
Find a generic way to transfer data from one cache store to another, which could involve different Infinispan versions. This is handy to migrate file cache store based users to single file cache store (ISPN-2806).
Ideally, this should be added as a recipe for rolling upgrades.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years, 9 months
[JBoss JIRA] (ISPN-761) Cache.keySet(), entrySet(), values(), size() ignore contents of cache loader
by Sanne Grinovero (JIRA)
[ https://issues.jboss.org/browse/ISPN-761?page=com.atlassian.jira.plugin.s... ]
Sanne Grinovero commented on ISPN-761:
--------------------------------------
We discussed this on IRC:
{quote}
wburns: mmarkus, anistor, dberindei, galderz, pruivo, ttarant: sannegrinovero: was looking for input on ISPN-761
wburns: mmarkus brought up a point about non passivated caches as well - and this led me to find an issue with how size would work currently
wburns: basically you have a non passivated cache that has a user use the SKIP_CACHE_STORE flag etc. could cause inconsistencies with size method as the cache store is no longer a superset of the memory store
sannegrinovero: wburns: IMHO #size() should be deleted.
wburns: which would require reading all keys in memory and in the cache store to correctly determine size
sannegrinovero: wburns: that might not happen in the near future, but just saying that I wouldn't spend too much time on it.
wburns: sannegrinovero: 1 thought we had was not to worry about this edge case since it is rarely used Flag outside of internal infinispan
sannegrinovero: wburns: no, size() alreasy is documented that it only returns the size in the (local) data container, so I don't think it should load from the CacheStore at all.
wburns: sannegrinovero: the Jira 761 is to change that and as galderz pointed out I updated the javadoc on Cache
sannegrinovero: wburns: SKIP_CACHE_STORE? no it's super useful for client apps as well. Quite critical actually.
wburns: for write operations for a new key that is
sannegrinovero: wburns: what's the reason to change that? -1 from me.
wburns: sannegrinovero: it is what people want I guess :/
sannegrinovero: wburns: we can't listen to all, it's primarily important that the API stays consistent and clear.
mmarkus: I think size should also include the cache store
sannegrinovero: wburns: did you read the comments on the issue? It was moved to 6.0 so that it could be implemented because, as Manik states there, it requires changes in the CacheStore SPI to be implemented afficiently.
sannegrinovero: efficiently.
sannegrinovero: mmarkus: ^
mmarkus: if people want to ignore it they can always skip it for persistence
mmarkus: sannegrinovero: I'm not aware of any use cases that skip store *writes* with flags, any example?
sannegrinovero: mmarkus: these methods are unreliable anyway as they return just the view of the single node, people shouldn't use them at all or just in statistics (knowing the limitations)
sannegrinovero: mmarkus: there are dozens of examples of that in the Lucene Directory module.
sannegrinovero: mmarkus: I made the flag ;)
wburns: sannegrinovero: so you are the culprit :P jk
mmarkus: sannegrinovero: well directory module is kindof internal though :-)
sannegrinovero: mmarkus: true, but it also is an example of an application dealing with the Infinispan API.
wburns: sannegrinovero: but yes in regards to efficiency I would agree - I have added comments to the design doc to address those deficiencies as well
mmarkus: sannegrinovero: wburns: and agreed that involving cache stores makes more sense for local caches
sannegrinovero:wburns: mmarkus: my vote would go to throw an "UnsupportedOperationException" if we can't remove the ConcurrentMap API ;-)
mmarkus: sannegrinovero: on size() & co?
sannegrinovero: wburns: mmarkus: yes. unless you prefer to "throw OutOfMemoryException() " ... :D
mmarkus: sannegrinovero: −1. I think they make sense for local caches in particular
sannegrinovero: mmarkus: makes sense for local caches without CacheStore. IFF I need a CacheStore, and I am not loading it all in memory anyway, it likely is because there is not enough memory.
mmarkus: sannegrinovero: you might want to get all the keys in the system for certain scenarios and currently there's no way to do that
mmarkus: sannegrinovero: the way the currently are these methods are broken as they won't return that. Users have been complaining about it for quite a while.
sannegrinovero: mmarkus: I know, and we discussed this many times on the mailing list. As I said to wburns, this issue was moved to 6.0 so that we would fix the CacheStore SPI to allow for better implementation of size() or iterate()
mmarkus: sannegrinovero: also if they won't the old functionality they can use the SKIP_STORE flag ;)
sannegrinovero: mmarkus: also Map/Reduce is WORTHLESS until you fix that SPI method.
sannegrinovero: mmarkus: as it has the same problem: needs to iterate (map) all entries, but they just don't fit in memory.
wburns: sannegrinovero: I wasn't around when that discussion took place... is that for a streaming SPI or something?
sannegrinovero: mmarkus: I'm not against implementing it, no doubt that it could be useful. It's the implementation which is simply not correct.
sannegrinovero: wburns: you are familiar with the Map/Reduce pattern?
mmarkus: sannegrinovero: +1 to implement them more efficiently, but and the required optimisations were tracked in ISPN-3290
wburns: sannegrinovero: yeah
sannegrinovero: mmarkus: fixed the dependency for you on JIRA.
mmarkus: sannegrinovero: you have a point but it's not a hard dependecy
sannegrinovero: mmarkus: it's not a hard dependency if you document those methods with "throws OOException most of the times"
sannegrinovero: mmarkus: BTW why would you want to fix the issue twice? this is doubling the work if wburns has to fix it now and then fix it again to use the new SPI
mmarkus sannegrinovero: the effort duplication should be small though
sannegrinovero: mmarkus: your call. I just hope nobody will use that method.
wburns: sannegrinovero: what was the part about Map/Reduce? is it basically that cache store entries aren't used - and we want to add a streaming API to go through them by chunks instead of all the contents?
sannegrinovero: wburns: CacheStores are actually used, but as it is today to run a Map/Reduce job it is loading all entries in memory to run the first Map part. Which is pretty much defeating the purpose of the pattern.
wburns: sannegrinovero: guessing we have optimizations for shared at least :) - but basically is just to stream through the cache store contents
sannegrinovero: wburns: to take advantage of the pattern, it should pass the mapping definition to the store engine to have it narrow down execution (ideally) or (more practically) to run it on an iterator from the CacheStore itself to not run in OOM at least.
sannegrinovero: wburns: no optimisations whatsoever :(
mmarkus: sannegrinovero: I imagine it's the same way you're thinking to have keySet() and entries() implemented as well
sannegrinovero: wburns: and that's another point for the SPI needing improvements. At least the option to load only the entries owned by the current node should be added.
mmarkus: sannegrinovero: use keySet().iterator() to iterate over the set of keys
mmarkus: sannegrinovero: ^^?
wburns: sannegrinovero: ouch
sannegrinovero: mmarkus: exactly. But think about size() to keep it simple.. just add a size() method on the cacheStore first, then the JDBC one (for example) could run a simple COUNT instead of attempting to load 100GB in memory
mmarkus: sannegrinovero: yep, that's noted
{quote}
> Cache.keySet(),entrySet(),values(),size() ignore contents of cache loader
> -------------------------------------------------------------------------
>
> Key: ISPN-761
> URL: https://issues.jboss.org/browse/ISPN-761
> Project: Infinispan
> Issue Type: Bug
> Components: Loaders and Stores
> Affects Versions: 4.2.0.BETA1
> Reporter: Paul Ferraro
> Assignee: William Burns
> Priority: Critical
> Labels: onboard, potential_performance_hit
> Fix For: 6.0.0.Alpha1, 6.0.0.Final
>
>
> Passivated cache entries are not represented in values returned by the keySet(), entrySet(), values(), and size() Cache methods. This results in inconsistent behavior, since it is possible that a given cache key may not be contained in keySet() even though Cache.get(...) would return a non-null value if the entry was previously passivated.
> I think CacheLoaderInterceptor needs to implement the following methods:
> Object visitSizeCommand(InvocationContext ctx, SizeCommand command) throws Throwable
> Object visitValuesCommand(InvocationContext ctx, ValuesCommand command) throws Throwable
> Object visitEntrySetCommand(InvocationContext ctx, EntrySetCommand command) throws Throwable
> Object visitKeySetCommand(InvocationContext ctx, KeySetCommand command) throws Throwable
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years, 9 months
[JBoss JIRA] (ISPN-3317) XSD missing storeAsBinary.defensive entry
by Galder Zamarreño (JIRA)
Galder Zamarreño created ISPN-3317:
--------------------------------------
Summary: XSD missing storeAsBinary.defensive entry
Key: ISPN-3317
URL: https://issues.jboss.org/browse/ISPN-3317
Project: Infinispan
Issue Type: Bug
Components: Configuration
Affects Versions: 5.3.0.Final
Reporter: Galder Zamarreño
Assignee: Galder Zamarreño
Fix For: 5.3.1.Final, 6.0.0.Alpha1, 6.0.0.Final
Defensive copying configuration attribute missing from XSD.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years, 9 months