In abtract I am load testing a Hibernate/JBC3 web application and am seeing that about 7% of the time inserting a child entity into a previously-empty collection results in update (INSERT) into DB but local cache remains unchanged: -collection Node is still empty -there is no child Entity node This does not appear to be a race condition, as from that point on in any TX parent.getChildren() will return an empty collection from cache (even though DB state is non-empty). This seems vaguely similar to https://jira.jboss.org/jira/browse/JBCACHE-1481 but that issue's marked resolved. In detail App Config Single Tomcat 6 NIO Hibernate 3.3.1.GA JBossCache 3.0.3.GA - Local Cache only; using default Locking (MVCC); READ_COMMITTED isolation BTM JTA transaction manager Load Test JMeter load test where each of userCount users loop over the following scenario of HTTP requests: for(I iterations) | { | login(); | thinkTime //sleep; | getRestrictedData(); //requires being logged in | thinkTime //sleep; | logout(); | thinkTime //sleep; | } | App Pseudo Code Domain Model -Transactional entity caches User, UserSession; collection cache User.sessions | class User{ | private long id; | private Set<UserSession> sessions; //1-to-[0,1] modeled as one-to-many with unique constraint | | public boolean isLoggedIn() | { | return !getSessions().isEmpty(); | } | | public void addSession(UserSession us) | { | us.setUser(this); | sessions.add(us); | } | | public Set<UserSessions> getSessions() | { | return sessions; | } | } | | | | class UserSession | { | private User user; | | void setUser(User usr) | { | this.user = usr; | } | | } Service Methods //insert new UserSession if credentials check out | //throw exception otherwise | public void login(id, credentials) | { | User usr = dao.getUser(id); | if(credentials == bad) | { | throw new AuthenticationException(); | } | usr.add(new UserSession()); | } | | //return data only if a userSession is present | //throw exception otherwise | public String getRestrictedData(id) | { | User usr = dao.getUser(id); | if(! usr.isLoggedIn()) | { | throw new AuthorizationException("not logged in"); | } | return data; | } | | public void logout() | { | //clear userSessions | } | } | Problem Observed When userCount==1, no requests fail, regardless of thinkTime (varied latter between 0 and 2000ms) | When userCount is > 30, about 10-20% of users eventually (and most of the time on the very 1st iteration), fail like so: | -login() //success | -getRestrictedData() //fail: user not logged in | | My load test terminates on 1st failure. Inspecting states of DB and JBoss Cache (via JMX) reveal: | -DB does have record of UserSession in question | -In Collection cache: User.sessions Node contains an empty PersistentSet | -In entity cache: User node references above (empty) collection; no node for UserSession that is in DB; | -hibernateVersion of User (parent) is updated in DB but not in Cache node | | Musings | | Though I am unable to repro this problem with 1 user, it is not possible for user A to affect state of user B in my app. So I am not quite clear on what's going on here. Since I don't (yet) understand the internals of MVCC (and jbc in general), I'd appreciate any thoughts and answers to the following questions that'll help me narrow down the problem: | | 1. In MVCC, how many threads participate in login() method? Is it the case that one thread does the read and writes a versioned update, but another thread merges the updated data *after* TX commit? | | 2. A somewhat dated http://www.jboss.org/community/docs/DOC-10266 wiki says there exists a faulty assumption about consistent ordering of synchronizations in Hibernate/JBC (problem #5). Has that been addressed? If not, could that be causing given problem? | | 3. Same wiki's problem #7: | anonymous wrote : Hibernate should either retry or JBossCache should fail silently on this. (Nikita: not sure what 'this' refers to) As long as the data is stored in the db correctly, failure putting the data in the cache could fail silently. Next time someone requests the entity, it'd be retrieved from db and put in the cache. | | This paragraph seems to imply that it is normal that cache is not updated on insert of new UserSession during login(). However, it also expects subsequent User.getSessions() to result in a DB read (and a put into cache). But what I am seeing that User.getSessions() in getRestrictedData() results in a cache hit on a (stale) node containing empty set user.sessions. Should addSession() in login() result in User.sessions collection node being invalidated (to be populated during subsequent reads) or being updated with a new reference to UserSession node? Similarly, should addSession() result in put into UserSession entity cache, or is that put expected to occur only on subsequent read? | | 4. Finally, turning up debugging shows multitude of these 2 types of messages. These are emitted seemingly in failing and succeeding requests: | | anonymous wrote : [3/2/09 16:27:38:268] DEBUG [d585d9d4-cc59-4b35-b717-d26c97db8f89,ltester-100000] org.jboss.cache.interceptors.InvocationContextInterceptor - FAIL_SILENTLY Option is present - suspending any ongoing transaction. | | [3/2/09 16:27:38:269] DEBUG [76225fe7-1541-4960-9c2d-cdbe1330f5c5,ltester-100001] org.jboss.cache.invocation.CacheInvocationDelegate - putForExternalRead() called with Fqn /com/doppelganger/domain/User/userSessions/COLL/com.doppelganger.domain.User.userSessions#183 and this node already exists. This method is hence a no op. | | | | Does the latter say that a collections cache node will not be updated because one already exists. What if existing node contains an empty collection but the candidate node contains a non-empty collection? View the original post : http://www.jboss.org/index.html?module=bb&op=viewtopic&p=4214380#... Reply to the post : http://www.jboss.org/index.html?module=bb&op=posting&mode=reply&a...