12:54 a.m.
In abtract
I am load testing a Hibernate/JBC3 web application and am seeing that about 7% of the time
inserting a child entity into a previously-empty collection results in update (INSERT)
into DB but local cache remains unchanged:
-collection Node is still empty
-there is no child Entity node
This does not appear to be a race condition, as from that point on in any TX
parent.getChildren() will return an empty collection from cache (even though DB state is
non-empty).
This seems vaguely similar to https://jira.jboss.org/jira/browse/JBCACHE-1481 but that
issue's marked resolved.
In detail
App Config
Single Tomcat 6 NIO
Hibernate 3.3.1.GA
JBossCache 3.0.3.GA - Local Cache only; using default Locking (MVCC); READ_COMMITTED
isolation
BTM JTA transaction manager
Load Test
JMeter load test where each of userCount users loop over the following scenario of HTTP
requests:
for(I iterations)
| {
| login();
| thinkTime //sleep;
| getRestrictedData(); //requires being logged in
| thinkTime //sleep;
| logout();
| thinkTime //sleep;
| }
|
App Pseudo Code
Domain Model
-Transactional entity caches User, UserSession; collection cache User.sessions
| class User{
| private long id;
| private Set<UserSession> sessions; //1-to-[0,1] modeled as one-to-many with
unique constraint
|
| public boolean isLoggedIn()
| {
| return !getSessions().isEmpty();
| }
|
| public void addSession(UserSession us)
| {
| us.setUser(this);
| sessions.add(us);
| }
|
| public Set<UserSessions> getSessions()
| {
| return sessions;
| }
| }
|
|
|
| class UserSession
| {
| private User user;
|
| void setUser(User usr)
| {
| this.user = usr;
| }
|
| }
Service Methods
//insert new UserSession if credentials check out
| //throw exception otherwise
| public void login(id, credentials)
| {
| User usr = dao.getUser(id);
| if(credentials == bad)
| {
| throw new AuthenticationException();
| }
| usr.add(new UserSession());
| }
|
| //return data only if a userSession is present
| //throw exception otherwise
| public String getRestrictedData(id)
| {
| User usr = dao.getUser(id);
| if(! usr.isLoggedIn())
| {
| throw new AuthorizationException("not logged in");
| }
| return data;
| }
|
| public void logout()
| {
| //clear userSessions
| }
| }
|
Problem Observed
When userCount==1, no requests fail, regardless of thinkTime (varied latter between 0 and
2000ms)
| When userCount is > 30, about 10-20% of users eventually (and most of the time on
the very 1st iteration), fail like so:
| -login() //success
| -getRestrictedData() //fail: user not logged in
|
| My load test terminates on 1st failure. Inspecting states of DB and JBoss Cache (via
JMX) reveal:
| -DB does have record of UserSession in question
| -In Collection cache: User.sessions Node contains an empty PersistentSet
| -In entity cache: User node references above (empty) collection; no node for
UserSession that is in DB;
| -hibernateVersion of User (parent) is updated in DB but not in Cache node
|
| Musings
|
| Though I am unable to repro this problem with 1 user, it is not possible for user A to
affect state of user B in my app. So I am not quite clear on what's going on here.
Since I don't (yet) understand the internals of MVCC (and jbc in general), I'd
appreciate any thoughts and answers to the following questions that'll help me narrow
down the problem:
|
| 1. In MVCC, how many threads participate in login() method? Is it the case that one
thread does the read and writes a versioned update, but another thread merges the updated
data *after* TX commit?
|
| 2. A somewhat dated http://www.jboss.org/community/docs/DOC-10266 wiki says there
exists a faulty assumption about consistent ordering of synchronizations in Hibernate/JBC
(problem #5). Has that been addressed? If not, could that be causing given problem?
|
| 3. Same wiki's problem #7:
| anonymous wrote : Hibernate should either retry or JBossCache should fail silently on
this. (Nikita: not sure what 'this' refers to) As long as the data is stored in
the db correctly, failure putting the data in the cache could fail silently. Next time
someone requests the entity, it'd be retrieved from db and put in the cache.
|
| This paragraph seems to imply that it is normal that cache is not updated on insert of
new UserSession during login(). However, it also expects subsequent User.getSessions() to
result in a DB read (and a put into cache). But what I am seeing that User.getSessions()
in getRestrictedData() results in a cache hit on a (stale) node containing empty set
user.sessions. Should addSession() in login() result in User.sessions collection node
being invalidated (to be populated during subsequent reads) or being updated with a new
reference to UserSession node? Similarly, should addSession() result in put into
UserSession entity cache, or is that put expected to occur only on subsequent read?
|
| 4. Finally, turning up debugging shows multitude of these 2 types of messages. These
are emitted seemingly in failing and succeeding requests:
|
| anonymous wrote : [3/2/09 16:27:38:268] DEBUG
[d585d9d4-cc59-4b35-b717-d26c97db8f89,ltester-100000]
org.jboss.cache.interceptors.InvocationContextInterceptor - FAIL_SILENTLY Option is
present - suspending any ongoing transaction.
| | [3/2/09 16:27:38:269] DEBUG [76225fe7-1541-4960-9c2d-cdbe1330f5c5,ltester-100001]
org.jboss.cache.invocation.CacheInvocationDelegate - putForExternalRead() called with Fqn
/com/doppelganger/domain/User/userSessions/COLL/com.doppelganger.domain.User.userSessions#183
and this node already exists. This method is hence a no op.
| |
|
| Does the latter say that a collections cache node will not be updated because one
already exists. What if existing node contains an empty collection but the candidate node
contains a non-empty collection?
View the original post :
http://www.jboss.org/index.html?module=bb&op=viewtopic&p=4214380#...
Reply to the post :
http://www.jboss.org/index.html?module=bb&op=posting&mode=reply&a...