4:45 a.m.
On 5 Jan 2011, at 17:19, Jonathan Halliday wrote:
On 01/05/2011 03:42 PM, Mircea Markus wrote:
>
> On 5 Jan 2011, at 14:51, Mircea Markus wrote:
>
>> FYI, a discussion I have with Jonathan around recovery support from TM
>>
>> On 5 Jan 2011, at 14:43, Jonathan Halliday wrote:
>>> On 01/05/2011 02:18 PM, Mircea Markus wrote:
>>>
>>>> I don't know how the TM recovery process picks up the XAResource
instance on which to call XAResource.recover, but I imagine it expects this method to
return all the prepared(or heuristic completed) transactions from the _whole transaction
branch_, i.e. from the entire cluster.
>>>
>>> all from the logical RM, which you happen to implement as a cluster, yes.
>>>
>>>> I'm asking this because right now there's no way for a node to
know all the prepared transaction in the entire cluster. This is doable but would involve
an broadcast to query the cluster, which might be costly (time and bandwidth).
>>>
>>> right. not to mention it should, strictly speaking, block or fail if any node
is unreachable, which kinda sucks from an availability perspective.
> So if a node does not respond to the broadcast, it is incorrect to return the
prepared transactions received from the other nodes? (is this because the TM expects to
receive some tx that it knows for sure to be prepared?) Or would a "best effort"
be "good enough"? (e.g. I broadcast the query and return all the results
received in 1 sec)
hmm, interesting question.
Keep in mind that the XA spec dates from a time when a typical large clustered RM was 2-3
oracle nodes on the same LAN segment. It simply isn't geared to a world where the
number of nodes is so large and widely distributed that the probability of *all* of them
being available simultaneously is pretty slim. Likewise the number of transaction managers
connected to a resource was assumed to be small, often 1, rather than the large N we see
on modern clusters / clouds. As a result, the spec either fails to give guidance on some
issues because they weren't significant at the time it was written, or
implies/mandates behaviour that is counter productive in modern environments.
Thus IMO some compromises are necessary to make XA usable in the real world, especially
at scale. To further complicate matters, these are split across RM and TM, with different
vendors having different views on the subject. My advice is geared to the way JBossTS
drives XA recovery - other TMs may behave differently and make greater or lesser
assumptions about compliance with the letter of the spec. As a result you may find that
making your RM work with multiple vendor's TMs requires a) configuration options and
b) a lot of painful testing. Likewise JBossTS contains code paths and config options
geared to dealing with bugs or non-compliant behaviour in various vendor's RMs.
Now, on to the specific question: The list returned should, strictly speaking, be
complete. There are two problems with that. First, you have to be able to reach all your
cluster nodes to build a complete list which, as previously mentioned, is pretty unlikely
in a sufficiently large cluster. Your practical strategies are thus as you say: either a)
throw an XAException(XAER_RMFAIL) if any node is unreachable within a reasonable timeout
and accept that this may mean an unnecessary delay in recovering the subset of tx that are
known or b) return a partial list on a best effort basis. The latter approach allows the
transaction manager to deal with at least some of the in-doubt tx, which may in turn mean
releasing resources/locks in the RM. In general I'd favour that option as having
higher practical value in terms of allowing the best possible level of service to be
maintained in the face of ongoing failures.
JBossTS will rescan every N minutes (2 by default) and thus you can simply include any
newly discovered in-doubt tx as they become known due to e.g. partitioned nodes rejoining
the cluster, and the TM will deal with them when they are first seen. Note however that
some TMs assume that if they scan an RM and that RM does not subsequently crash, no new
in-doubt transactions will occur except from heuristics. Let's gloss over how they can
even detect a crash/recover of the RM if the driver masks it with failover or the event
happens during a period when the TM makes no call on the driver. Such a TM will perform a
recovery scan once at TM startup and not repeat. In such case you may have in-doubt tx
from nodes unavailable at that crucial time subsequently sitting around for a prolonged
period, tying up precious resources and potentially blocking subsequent updates. Most RM
vendors provide some kind of management capability for admins to view and manually force
completion of in-doubt tx. command line tool, jmx, web gui, whatever, just so long as it
exists.
When a node crashes all the transactions that node owns (i.e. tx which were
originated on that node and XAResource instance residing on that node) automatically
rollback, so that no resources (locks mainly) are held. The only thing we need to make
sure though is that the given transaction ids (the one that heuristically rollback) are
returned by theXAResource.recover method - doable in the same way we handle prepares. I
imagine that we'll have to keep these XIDs until XAResource.forget(XID) is called, am
I right? Is it common/possible for people to use TM _without_ recovery? If so, this
"held heuristic completed TX" functionality should be configurable
(enabled/disabled) in order to avoid memory leaks (no recovery means .forget never gets
called)
Another interesting issue is what constitutes an 'in-doubt'
tx. Pretty much all RMs will include heuristically completed tx in the recovery list. Some
will include tx branches that have prepared but not yet committed or rolled back. Some
will include such only if they have been in the prepared state for greater than some
threshold length of time (a few seconds i.e. a couple of order of magnitude longer than a
tx would normally be expected to hold that state). There is also the question of when a tx
should be removed from the list. The wording of the spec
'Two consecutive invocation of [recover] that starts from the beginning of the list
must return the same list
of transaction branches unless one of the following takes place:
- the transaction manager invokes the commit, forget, prepare, or rollback method
for that resource
manager, between the two consecutive invocation of the recovery scan
...'
seems to imply a single transaction manager.
doesn't this also imply that the
prepare-treshold isn't the spec's way? I.e. even though TM doesn't call any
method on the RM , the RM returns a new XID in the result of XAResource.recover when the
threshold is reached.
In cases where more than one TM is connected to the RM, the list
clearly can't be considered stable between calls, as that would require prolonged
blocking. Thus a reasonable TM should not expect a stable list. However, it's less
clear how it should react to items that appear and disappear arbitrarily over consecutive
calls to recover.
In the case of JBossTS, it assumes the RM may include tx branches that are actually
proceeding normally in the results of a recovery scan. It thus caches the list, sleeps for
an interval considered long enough for any normal prepared tx to complete, then rescans
and compares the results. Any tx appearing in both scans is considered genuinely in need
of recovery. If an RM includes in its recover results a normally proceeding tx branch and
the TM does not perform such a backoff, it may, in a clustered environment, rollback a tx
branch that another TM will try to commit a split second later, thus resulting in an
unnecessary heuristic outcome. The worst scenario I've ever seen was a certain large
DB vendor who considered tx to be in-doubt not just from the time they were prepared, but
from the moment they were started. ouch.
Naturally most well behaved TMs will have some sense of tx branch ownership and not
recover unrecognised tx, but it's one of those delightful gray areas where you may
benefit from being a little paranoid.
So each TM would only care to recover the
transactions it manages? Sort of makes sense.
Which, as it happens, gives you a perfect excuse to return a partial
list of tx - in terms of the way TM recovery works there is nothing to distinguish a tx
that's missing due to node outage from one that is missing due to not yet having
reached the in-doubt time threshold.
Of course it does not end with the recovery scan. Let's take the case where a
specific tx is identified as in-doubt and returned from the recover call. The TM may then
call e.g. commit on it. Processing that commit may involve the driver/server talking to
multiple cluster nodes, some of which may not respond. Indeed this is the case during a
normal commit too, not just one resulting from a recovery scan. You need to think very
carefully about what the result of a failed call should be. A lot of bugs we've seen
result from resource managers using incorrect XAException codes or transaction managers
misinterpreting them. Be aware of the semantics of specific error codes and be as concise
as possible when throwing errors.
Good to know! Something to discuss/review on our
next meeting.
> I would expect a single (periodic) XAResource.recover call per cluster(assuming the
cluster is the RM) / transaction manager. Am I wrong?
Only in so far as you are assuming one logical TM per cluster, with cluster here meaning
JBossAS cluster as distinct from a possible separate infinispan RM cluster.
In practice although a group of JBossAS servers may be clustering e.g. web sessions or
EJBs, they don't cluster TMs. Each JBossAS instance is a logically separate and
independently functioning TM and each is responsible for running its own recovery. Note
that there ways of clustering the JTS version of the TM, but we don't advertise or
support them and as a result to the best of my knowledge no end user actually runs with
such configuration. Likewise for running a single out of process recovery subsystem which
is shared by multiple TMs/JBossAS nodes.
>>> you mean a logically separate resource manager, yes. You are basically
talking about not doing interposition in the driver/server but rather relying on the
transaction manager to handle multiple resources. It may make your implementation simpler
but probably less performant on the critical path (transaction commit) vs. recovery.
> I don't know exactly how many RPC's happen in this approach, with TM handling
multiple resources. I imagine the TM would do a XAResource.prepare for each of the nodes
involved. In this XAResource.prepare call I would have to implement the logic of going
remotely to each involved node. Then the same for XAResource.commit. Is that so? (If so
then this is pretty much what we already do in ISPN when it comes to commit/rollback).
> On of the advantages of allowing TM to handle each individual node is that we can
benefit from some nice TM features like read-only optimisation or 1PC for single
participants (these are to be implemented anyway in ISPN).
Although the RPC topology can matter, particularly where WAN hops are involved, the
critical difference is actually where the logging responsibility lies.
If the TM has more than one XAResource enlisted, it has to write to disk between the
prepare and commit phases. So where it has got e.g. two infinispan nodes as separate
XAResources, the tx is bottlenecked on disk I/O even though the RM is entirely in (network
distributed) RAM.
Right! I was looking in the wrong place.
Where the clustered RM presents itself as a single resource, the TM won't necessarily
log, due to automatic one phase commit optimisation.
However...
In that case your driver/server has to guarantee correct behaviour in the event of node
failure during the commit call. In other words, it has to do its own persistence if the
state updates span more than one cluster node. In the case of infinispan that's likely
to be via. additional RPCs for replication rather than by a disk write as in most database
cluster RMs.
In short, you're trading disk I/O in the TM against additional network I/O in the
clustered RM. All else being equal I think having the RM do the work will perform better
at scale, but that's just a guess.
+1
You more or less need the code for that either way, as even where a
single infinispan node is a logically separate RM, it still needs to talk to its cluster
mates (or the disk I guess) for persistence or it can't guarantee durability of the
tx.
Thanks Jonathan!
Jonathan.
--
------------------------------------------------------------
Registered Address: Red Hat UK Ltd, Amberley Place, 107-111 Peascod Street, Windsor,
Berkshire, SI4 1TE, United Kingdom.
Registered in UK and Wales under Company Registration No. 3798903 Directors: Michael
Cunningham (USA), Charlie Peters (USA), Matt Parsons (USA) and Brendan Lane (Ireland)
5:48 a.m.
New subject: XAResource.isSameRM
On 01/06/2011 10:45 AM, Mircea Markus wrote:
When a node crashes all the transactions that node owns (i.e. tx
which were originated on that node and XAResource instance residing on that node)
automatically rollback, so that no resources (locks mainly) are held. The only thing we
need to make sure though is that the given transaction ids (the one that heuristically
rollback) are returned by theXAResource.recover method - doable in the same way we handle
prepares. I imagine that we'll have to keep these XIDs until XAResource.forget(XID) is
called, am I right?
I was under the impression a node does not own a tx. It may
own a *branch* of that tx. Take the case where node
JBossAS-1 starts a tx, propagates it to node JBossAS-2 and
both JBossAS-1 and JBossAS-2 then simultaneously contact
different nodes of the infinispan cluster in the scope of
that tx. Each node would see a different branch of the same
global tx. You presumably don't want to have to sync the
ownership for each new tx across the cluster? The only way
you could tie an entire tx to a single infinispan node is
e.g. consistent hashing to force the decision of which node
the driver connects to based on the tx context.
An XAResource does not reside in an infinispan node
(although there may be something equivalent holding tx state
on the server side) - it's a client/driver side construct.
Given that the driver does transparent failover / load
balancing and such, the XAResource can't be said to belong
to a specific infinispan node unless you throw away some of
the clustering availability advantages. It's really a
question of where you are going to put the clustering
intelligence - in a smart client side driver or in a server
side component that acts as a kind of routing proxy.
With your 'rollback tx branch on node crash' model you are
failing to provide ACID semantics for the cluster as a
whole. You can abort them before the prepare stage, but post
prepare it's not an option as you'll piss off clients who
expect the cluster to behave correctly as long as a majority
of its nodes survive. I'm not saying it's flat out wrong,
just that it needs to be very clearly documented in order to
avoid getting whined at by disgruntled users. My
understanding is you're pitching infinispan not as a
volatile cache, but an in-memory data grid. In that model
the node does not own the tx, the cluster owns the tx and is
responsible for masking node failures from the client.
Rollback of prepared tx on node failure is therefore not an
option - some part of the tx state may already have been
committed by surviving nodes and you'll get inconsistencies.
You need to replicate enough information to avoid that,
otherwise the client app is going to have to explicitly
provide logic to do the reconciliation, which sucks.
Is it common/possible for people to use TM _without_ recovery? If so,
this "held heuristic completed TX" functionality should be configurable
(enabled/disabled) in order to avoid memory leaks (no recovery means .forget never gets
called)
It is not common. That said, JBossTS has for similar reasons
got a 'give up after N hours' config option which will
eventually abandon tx that have not recovered. It's off
(i.e. never give up) by default but a small number of users
find it handy. Most just use the admin tooling to manually
clean up the small number of unrecoverable situations - it's
safer in most cases.
> Another interesting issue is what constitutes an
'in-doubt' tx. Pretty much all RMs will include heuristically completed tx in the
recovery list. Some will include tx branches that have prepared but not yet committed or
rolled back. Some will include such only if they have been in the prepared state for
greater than some threshold length of time (a few seconds i.e. a couple of order of
magnitude longer than a tx would normally be expected to hold that state). There is also
the question of when a tx should be removed from the list. The wording of the spec
>
> 'Two consecutive invocation of [recover] that starts from the beginning of the
list must return the same list
> of transaction branches unless one of the following takes place:
> - the transaction manager invokes the commit, forget, prepare, or rollback
method for that resource
> manager, between the two consecutive invocation of the recovery scan
> ...'
>
> seems to imply a single transaction manager.
doesn't this also imply that the prepare-treshold isn't the
spec's way? I.e. even though TM doesn't call any method on the RM , the RM returns
a new XID in the result of XAResource.recover when the threshold is reached.
The definition of what constitutes an 'in-doubt' tx for
purposes of inclusion in the recovery list is not well
defined by the spec.
In a world where there is only one TM driving the RM and
that TM performs recovery before starting up and running new
tx, no new tx will be added to the recovery list. In the
real world new ones are being added continuously as the
system is always under load.
The concern is more around the stability of the list with
respect to not *removing* things except in response to TM
activity. i.e. if a node goes away, should you return a
cached snapshot of its tx for list stability, or exclude
them? both options carry risks.
Jonathan.
--
------------------------------------------------------------
Registered Address: Red Hat UK Ltd, Amberley Place, 107-111
Peascod Street, Windsor, Berkshire, SI4 1TE, United Kingdom.
Registered in UK and Wales under Company Registration No.
3798903 Directors: Michael Cunningham (USA), Charlie Peters
(USA), Matt Parsons (USA) and Brendan Lane (Ireland)
8:29 a.m.
New subject: XAResource.isSameRM
On 6 Jan 2011, at 11:48, Jonathan Halliday wrote:
On 01/06/2011 10:45 AM, Mircea Markus wrote:
> When a node crashes all the transactions that node owns (i.e. tx which were
originated on that node and XAResource instance residing on that node) automatically
rollback, so that no resources (locks mainly) are held. The only thing we need to make
sure though is that the given transaction ids (the one that heuristically rollback) are
returned by theXAResource.recover method - doable in the same way we handle prepares. I
imagine that we'll have to keep these XIDs until XAResource.forget(XID) is called, am
I right?
I was under the impression a node does not own a tx. It may own a *branch* of that tx.
Take the case where node JBossAS-1 starts a tx, propagates it to node JBossAS-2 and both
JBossAS-1 and JBossAS-2 then simultaneously contact different nodes of the infinispan
cluster in the scope of that tx. Each node would see a different branch of the same global
tx. You presumably don't want to have to sync the ownership for each new tx across the
cluster? The only way you could tie an entire tx to a single infinispan node is e.g.
consistent hashing to force the decision of which node the driver connects to based on the
tx context.
you are absolutely right, I don't want to sync ownership across the
cluster.
An XAResource does not reside in an infinispan node (although there may be something
equivalent holding tx state on the server side) - it's a client/driver side construct.
I see.
At the moment the *only way to transactionally* access a node is by collocating the client
and the server in the same VM. Hence the XAResource residing in the ISPN node's side.
ISPN 5.0 will take things further by allowing client/server access through
Hotrod:http://community.jboss.org/wiki/TransactionsOverHotRod
Given that the driver does transparent failover / load balancing and
such, the XAResource can't be said to belong to a specific infinispan node unless you
throw away some of the clustering availability advantages. It's really a question of
where you are going to put the clustering intelligence - in a smart client side driver or
in a server side component that acts as a kind of routing proxy.
At the moment if a
node that is touched by a transaction/branch crashes before transaction commits then all
the acquired resources are releases and the transaction is marked for rollback.
With your 'rollback tx branch on node crash' model you are failing to provide
ACID semantics for the cluster as a whole. You can abort them before the prepare stage,
but post prepare it's not an option as you'll piss off clients who expect the
cluster to behave correctly as long as a majority of its nodes survive. I'm not saying
it's flat out wrong, just that it needs to be very clearly documented in order to
avoid getting whined at by disgruntled users. My understanding is you're pitching
infinispan not as a volatile cache, but an in-memory data grid. In that model the node
does not own the tx, the cluster owns the tx and is responsible for masking node failures
from the client. Rollback of prepared tx on node failure is therefore not an option - some
part of the tx state may already have been committed by surviving nodes and you'll get
inconsistencies. You need to replicate enough information to avoid that, otherwise the
client app is going to have to explicitly provide logic to do the reconciliation, which
sucks.
Good point. At the moment (i.e. client and the node in same VM) this is not
an issue because if the node crashes you can safely assume that the client is crashed as
well. Something to be considered though, with remote TX over hotrod in mind.
> Is it common/possible for people to use TM _without_ recovery? If so, this
"held heuristic completed TX" functionality should be configurable
(enabled/disabled) in order to avoid memory leaks (no recovery means .forget never gets
called)
It is not common. That said, JBossTS has for similar reasons got a 'give up after N
hours' config option which will eventually abandon tx that have not recovered.
It's off (i.e. never give up) by default but a small number of users find it handy.
Most just use the admin tooling to manually clean up the small number of unrecoverable
situations - it's safer in most cases.
Right, JMX might come handy for that.
>> Another interesting issue is what constitutes an 'in-doubt' tx. Pretty
much all RMs will include heuristically completed tx in the recovery list. Some will
include tx branches that have prepared but not yet committed or rolled back. Some will
include such only if they have been in the prepared state for greater than some threshold
length of time (a few seconds i.e. a couple of order of magnitude longer than a tx would
normally be expected to hold that state). There is also the question of when a tx should
be removed from the list. The wording of the spec
>>
>> 'Two consecutive invocation of [recover] that starts from the beginning of
the list must return the same list
>> of transaction branches unless one of the following takes place:
>> - the transaction manager invokes the commit, forget, prepare, or rollback
method for that resource
>> manager, between the two consecutive invocation of the recovery scan
>> ...'
>>
>> seems to imply a single transaction manager.
> doesn't this also imply that the prepare-treshold isn't the spec's way?
I.e. even though TM doesn't call any method on the RM , the RM returns a new XID in
the result of XAResource.recover when the threshold is reached.
The definition of what constitutes an 'in-doubt' tx for purposes of inclusion in
the recovery list is not well defined by the spec.
In a world where there is only one TM driving the RM and that TM performs recovery before
starting up and running new tx, no new tx will be added to the recovery list. In the real
world new ones are being added continuously as the system is always under load.
The concern is more around the stability of the list with respect to not *removing*
things except in response to TM activity. i.e. if a node goes away, should you return a
cached snapshot of its tx for list stability, or exclude them? both options carry risks.
Thanks for the clarification.
Jonathan.
--
------------------------------------------------------------
Registered Address: Red Hat UK Ltd, Amberley Place, 107-111 Peascod Street, Windsor,
Berkshire, SI4 1TE, United Kingdom.
Registered in UK and Wales under Company Registration No. 3798903 Directors: Michael
Cunningham (USA), Charlie Peters (USA), Matt Parsons (USA) and Brendan Lane (Ireland)
8:45 a.m.
New subject: XAResource.isSameRM
On 01/06/2011 02:29 PM, Mircea Markus wrote:
At the moment the *only way to transactionally* access a
node is by collocating the client and the server in the same
VM.
So the scope of the transaction is limited to data residing
in that local node? What if I want a single transaction to
span the local node and data in a remote node? Or what about
where the data is all in the local node, but the node has a
remote replica for fault tolerance?
client and the node in same VM ... if the node crashes you can
safely assume that the client is crashed as well.
No, you can't. The client has persistent state on disk
courtesy of the transaction manager log and will restore
that after the crash. It's not volatile. The transaction
updates may also have spanned out to other systems. Your
model only works for infinispan in single node volatile
store configurations. As soon as you cluster or spill to
persistent backing store you are in trouble. That's better
than nothing, but pretty limiting for users. I'm not really
clear what advantage it has over implementing
Synchronization rather than XAResource. Why are you
bothering with 2PC if you're not going to actually provide
ACID semantics?
Jonathan.
--
------------------------------------------------------------
Registered Address: Red Hat UK Ltd, Amberley Place, 107-111
Peascod Street, Windsor, Berkshire, SI4 1TE, United Kingdom.
Registered in UK and Wales under Company Registration No.
3798903 Directors: Michael Cunningham (USA), Charlie Peters
(USA), Matt Parsons (USA) and Brendan Lane (Ireland)
11:43 a.m.
New subject: XAResource.isSameRM
On 6 Jan 2011, at 14:45, Jonathan Halliday wrote:
On 01/06/2011 02:29 PM, Mircea Markus wrote:
> At the moment the *only way to transactionally* access a
> node is by collocating the client and the server in the same
> VM.
So the scope of the transaction is limited to data residing in that local node? What if I
want a single transaction to span the local node and data in a remote node?
that's possible. It's just that you have to always interact with the local
node that will acquire remote locks remotely on behalf of your transaction.
Or what about where the data is all in the local node, but the node
has a remote replica for fault tolerance?
The same scenario as above. Attached is a
diagram depicting how ISPN handle tx under the hood(TM, User and NodeA are in the same
VM)
> client and the node in same VM ... if the node crashes you can
> safely assume that the client is crashed as well.
No, you can't. The client has persistent state on disk courtesy of the transaction
manager log and will restore that after the crash. It's not volatile. The transaction
updates may also have spanned out to other systems.
Wouldn't that be solved if the XID for the heuristic rollback transaction is returned
by the RM(ISPN cluster) in the XAResource.recover method?
Might be wrong but I see two possibilities:
a) node goes down before TM issued prepare
- when TM resurrects and calls XAResource.recover it receives the given XID, realises
that there's an heuristic decision (because it didn't call prepare) and take some
action (rollbacks other participants, notify sys admin?).
b) node goes down after TM issues prepare
- when TM issues a commit it receives an XAException (perhaps XA_HEURRB) and again
it is aware of the heuristic outcome
Your model only works for infinispan in single node volatile store
configurations. As soon as you cluster or spill to persistent backing store you are in
trouble. That's better than nothing, but pretty limiting for users. I'm not really
clear what advantage it has over implementing Synchronization rather than XAResource. Why
are you bothering with 2PC if you're not going to actually provide ACID semantics?
The plan is to support ACID semantics.
Jonathan.
--
------------------------------------------------------------
Registered Address: Red Hat UK Ltd, Amberley Place, 107-111 Peascod Street, Windsor,
Berkshire, SI4 1TE, United Kingdom.
Registered in UK and Wales under Company Registration No. 3798903 Directors: Michael
Cunningham (USA), Charlie Peters (USA), Matt Parsons (USA) and Brendan Lane (Ireland)
1:01 p.m.
New subject: XAResource.isSameRM
On 01/06/2011 05:43 PM, Mircea Markus wrote:
On 6 Jan 2011, at 14:45, Jonathan Halliday wrote:
> On 01/06/2011 02:29 PM, Mircea Markus wrote:
>
>> At the moment the *only way to transactionally* access a
>> node is by collocating the client and the server in the same
>> VM.
>
> So the scope of the transaction is limited to data
> residing in that local node? What if I want a single
> transaction to span the local node and data in a remote node?
that's possible. It's just that you have to always interact
with the local node that will acquire remote locks remotely
on behalf of your transaction.
ok, so the cluster intelligence is in the local node rather
than the client, not that there is any significant
distinction for now as they are co-located.
a) node goes down before TM issued prepare
- when TM resurrects and calls XAResource.recover it
receives the given XID, realises that there's an heuristic
decision (because it didn't call prepare) and take some
action (rollbacks other participants, notify sys admin?).
That's not a heuristic decision. A RM is perfectly entitled
to throw away any tx state up until prepare. Under the
presumed abort doctrine it simply throws an error from
prepare and the tx aborts cleanly. Recovery is not involved
- it applies only to tx that have reached prepare.
b) node goes down after TM issues prepare
- when TM issues a commit it receives an XAException
(perhaps XA_HEURRB) and again it is aware of the heuristic
outcome
Returning cleanly from a prepare is a promise by the RM to
successfully apply any subsequent commit. You're not in a
position to make such a promise unless your state is fault
tolerant, as a node crash would otherwise leave you with
inconsistent state.
It's not as simple as saying you'd rollback - what if you
prepare, get told to commit, apply remote changes (step
4.2.1), then crash before applying local changes (4.2.2)?
You can't report that as a rollback - you applied some of
the updates. You have to include the tx in the recovery list
as heuristic hazard (unless NodeA will transparently
repopulate with the committed data, in which case you can
mask the failure or report heuristic commit), but how to
even detect the heuristic at recovery time? NodeA has no
persistent record of the tx and NodeB thinks it completed
cleanly and has cleaned up its tx record to avoid leaking.
Where is the data that tells you you've got a problem?
Or have a more sophisticated scenario where there is an
additional NodeC, thus requiring multiple 'apply remote
changes' calls. Are those atomic across the cluster? If
there is a possibility that NodeB will apply the update but
NodeC won't, or NodeA will crash after issuing a call to B
but before C, you can wind up with inconsistent state in the
surviving B and C. Alternatively, what if A survives but C
crashes whilst applying changes that B has already
sucessfully applied? That's not necessarily a recovery
situation as far as the TM is concerned, but it may be from
your perspective as you'll need to detect and (ideally) fix
or (as a last resort) report the inconsistent data.
A lot of your behaviour is going to depend on what it means
for a node to recover after a crash. If it simply comes up
empty and expects to be repopulated from an external source,
as with a normal cache, then your relation to the XAResource
of that external source is critical. On the other hand if
your cluster node is itself fault tolerant through
replication, then you need to think carefully about how the
RM functionality ties into that replication - basically the
tx state information is not local to the node where the
XAResource resides, but must be replicated in the same
manner as the other data in that node and that replication
must be synchronous at certain state transitions in the tx
lifecycle - it's logging recovery information through RPC
rather than disk write. Really interesting things are going
to happen if a single transaction spans data that is a mix
of cache copy of data stored persistently in an XA database
and data for which infinispan is the definitive, fault
tolerant repository.
To make a cluster appear to the outside world as a single
logical entity for transaction purposes, you're pretty much
going to wind up doing interposition. That means you're
implementing not only an RM but substantial chunks of what
amounts to a TM too. Have fun.
Jonathan.
--
------------------------------------------------------------
Registered Address: Red Hat UK Ltd, Amberley Place, 107-111
Peascod Street, Windsor, Berkshire, SI4 1TE, United Kingdom.
Registered in UK and Wales under Company Registration No.
3798903 Directors: Michael Cunningham (USA), Charlie Peters
(USA), Matt Parsons (USA) and Brendan Lane (Ireland)
5:51 a.m.
New subject: XAResource.isSameRM
+1
On 6 Jan 2011, at 19:01, Jonathan Halliday wrote:
On 01/06/2011 05:43 PM, Mircea Markus wrote:
>
> On 6 Jan 2011, at 14:45, Jonathan Halliday wrote:
>
>> On 01/06/2011 02:29 PM, Mircea Markus wrote:
>>
>>> At the moment the *only way to transactionally* access a
>>> node is by collocating the client and the server in the same
>>> VM.
>>
>> So the scope of the transaction is limited to data
>> residing in that local node? What if I want a single
>> transaction to span the local node and data in a remote node?
> that's possible. It's just that you have to always interact
> with the local node that will acquire remote locks remotely
> on behalf of your transaction.
ok, so the cluster intelligence is in the local node rather
than the client, not that there is any significant
distinction for now as they are co-located.
> a) node goes down before TM issued prepare
> - when TM resurrects and calls XAResource.recover it
> receives the given XID, realises that there's an heuristic
> decision (because it didn't call prepare) and take some
> action (rollbacks other participants, notify sys admin?).
That's not a heuristic decision. A RM is perfectly entitled
to throw away any tx state up until prepare. Under the
presumed abort doctrine it simply throws an error from
prepare and the tx aborts cleanly. Recovery is not involved
- it applies only to tx that have reached prepare.
> b) node goes down after TM issues prepare
> - when TM issues a commit it receives an XAException
> (perhaps XA_HEURRB) and again it is aware of the heuristic
> outcome
Returning cleanly from a prepare is a promise by the RM to
successfully apply any subsequent commit. You're not in a
position to make such a promise unless your state is fault
tolerant, as a node crash would otherwise leave you with
inconsistent state.
It's not as simple as saying you'd rollback - what if you
prepare, get told to commit, apply remote changes (step
4.2.1), then crash before applying local changes (4.2.2)?
You can't report that as a rollback - you applied some of
the updates. You have to include the tx in the recovery list
as heuristic hazard (unless NodeA will transparently
repopulate with the committed data, in which case you can
mask the failure or report heuristic commit), but how to
even detect the heuristic at recovery time? NodeA has no
persistent record of the tx and NodeB thinks it completed
cleanly and has cleaned up its tx record to avoid leaking.
Where is the data that tells you you've got a problem?
Or have a more sophisticated scenario where there is an
additional NodeC, thus requiring multiple 'apply remote
changes' calls. Are those atomic across the cluster? If
there is a possibility that NodeB will apply the update but
NodeC won't, or NodeA will crash after issuing a call to B
but before C, you can wind up with inconsistent state in the
surviving B and C. Alternatively, what if A survives but C
crashes whilst applying changes that B has already
sucessfully applied? That's not necessarily a recovery
situation as far as the TM is concerned, but it may be from
your perspective as you'll need to detect and (ideally) fix
or (as a last resort) report the inconsistent data.
A lot of your behaviour is going to depend on what it means
for a node to recover after a crash. If it simply comes up
empty and expects to be repopulated from an external source,
as with a normal cache, then your relation to the XAResource
of that external source is critical. On the other hand if
your cluster node is itself fault tolerant through
replication, then you need to think carefully about how the
RM functionality ties into that replication - basically the
tx state information is not local to the node where the
XAResource resides, but must be replicated in the same
manner as the other data in that node and that replication
must be synchronous at certain state transitions in the tx
lifecycle - it's logging recovery information through RPC
rather than disk write. Really interesting things are going
to happen if a single transaction spans data that is a mix
of cache copy of data stored persistently in an XA database
and data for which infinispan is the definitive, fault
tolerant repository.
To make a cluster appear to the outside world as a single
logical entity for transaction purposes, you're pretty much
going to wind up doing interposition. That means you're
implementing not only an RM but substantial chunks of what
amounts to a TM too. Have fun.
Jonathan.
--
------------------------------------------------------------
Registered Address: Red Hat UK Ltd, Amberley Place, 107-111
Peascod Street, Windsor, Berkshire, SI4 1TE, United Kingdom.
Registered in UK and Wales under Company Registration No.
3798903 Directors: Michael Cunningham (USA), Charlie Peters
(USA), Matt Parsons (USA) and Brendan Lane (Ireland)
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
---
Mark Little
mlittle(a)redhat.com
JBoss, by Red Hat
Registered Address: Red Hat UK Ltd, Amberley Place, 107-111 Peascod Street, Windsor,
Berkshire, SI4 1TE, United Kingdom.
Registered in UK and Wales under Company Registration No. 3798903 Directors: Michael
Cunningham (USA), Charlie Peters (USA), Matt Parsons (USA) and Brendan Lane (Ireland).
6:33 a.m.
New subject: XAResource.isSameRM
On 6 Jan 2011, at 11:48, Jonathan Halliday wrote:
> Is it common/possible for people to use TM _without_ recovery? If so, this
"held heuristic completed TX" functionality should be configurable
(enabled/disabled) in order to avoid memory leaks (no recovery means .forget never gets
called)
It is not common. That said, JBossTS has for similar reasons
got a 'give up after N hours' config option which will
eventually abandon tx that have not recovered. It's off
(i.e. never give up) by default but a small number of users
find it handy. Most just use the admin tooling to manually
clean up the small number of unrecoverable situations - it's
safer in most cases.
That option is there because sometimes machines don't recover. The assumption behind
transactions is that failed machines do eventually recover, but the realities are
sometimes slightly different. Or sys admins decide to recover a machine's state on a
different machine, with a different IP address, and expect things (e.g., the TM) to be
omniscient and figure it all out ;-)
Mark.
---
Mark Little
mlittle(a)redhat.com
JBoss, by Red Hat
Registered Address: Red Hat UK Ltd, Amberley Place, 107-111 Peascod Street, Windsor,
Berkshire, SI4 1TE, United Kingdom.
Registered in UK and Wales under Company Registration No. 3798903 Directors: Michael
Cunningham (USA), Charlie Peters (USA), Matt Parsons (USA) and Brendan Lane (Ireland).
6:29 a.m.
New subject: XAResource.isSameRM
On 6 Jan 2011, at 10:45, Mircea Markus wrote:
On 5 Jan 2011, at 17:19, Jonathan Halliday wrote:
> On 01/05/2011 03:42 PM, Mircea Markus wrote:
>>
>> On 5 Jan 2011, at 14:51, Mircea Markus wrote:
>>
>>> FYI, a discussion I have with Jonathan around recovery support from TM
>>>
>>> On 5 Jan 2011, at 14:43, Jonathan Halliday wrote:
>>>> On 01/05/2011 02:18 PM, Mircea Markus wrote:
>>>>
>>>>> I don't know how the TM recovery process picks up the XAResource
instance on which to call XAResource.recover, but I imagine it expects this method to
return all the prepared(or heuristic completed) transactions from the _whole transaction
branch_, i.e. from the entire cluster.
>>>>
>>>> all from the logical RM, which you happen to implement as a cluster,
yes.
>>>>
>>>>> I'm asking this because right now there's no way for a node
to know all the prepared transaction in the entire cluster. This is doable but would
involve an broadcast to query the cluster, which might be costly (time and bandwidth).
>>>>
>>>> right. not to mention it should, strictly speaking, block or fail if any
node is unreachable, which kinda sucks from an availability perspective.
>> So if a node does not respond to the broadcast, it is incorrect to return the
prepared transactions received from the other nodes? (is this because the TM expects to
receive some tx that it knows for sure to be prepared?) Or would a "best effort"
be "good enough"? (e.g. I broadcast the query and return all the results
received in 1 sec)
>
> hmm, interesting question.
>
> Keep in mind that the XA spec dates from a time when a typical large clustered RM was
2-3 oracle nodes on the same LAN segment. It simply isn't geared to a world where the
number of nodes is so large and widely distributed that the probability of *all* of them
being available simultaneously is pretty slim. Likewise the number of transaction managers
connected to a resource was assumed to be small, often 1, rather than the large N we see
on modern clusters / clouds. As a result, the spec either fails to give guidance on some
issues because they weren't significant at the time it was written, or
implies/mandates behaviour that is counter productive in modern environments.
>
> Thus IMO some compromises are necessary to make XA usable in the real world,
especially at scale. To further complicate matters, these are split across RM and TM, with
different vendors having different views on the subject. My advice is geared to the way
JBossTS drives XA recovery - other TMs may behave differently and make greater or lesser
assumptions about compliance with the letter of the spec. As a result you may find that
making your RM work with multiple vendor's TMs requires a) configuration options and
b) a lot of painful testing. Likewise JBossTS contains code paths and config options
geared to dealing with bugs or non-compliant behaviour in various vendor's RMs.
>
> Now, on to the specific question: The list returned should, strictly speaking, be
complete. There are two problems with that. First, you have to be able to reach all your
cluster nodes to build a complete list which, as previously mentioned, is pretty unlikely
in a sufficiently large cluster. Your practical strategies are thus as you say: either a)
throw an XAException(XAER_RMFAIL) if any node is unreachable within a reasonable timeout
and accept that this may mean an unnecessary delay in recovering the subset of tx that are
known or b) return a partial list on a best effort basis. The latter approach allows the
transaction manager to deal with at least some of the in-doubt tx, which may in turn mean
releasing resources/locks in the RM. In general I'd favour that option as having
higher practical value in terms of allowing the best possible level of service to be
maintained in the face of ongoing failures.
+1
In fact on some mainframe implementations of CICS, for example, it behaves exactly like
this.
>
> JBossTS will rescan every N minutes (2 by default) and thus you can simply include
any newly discovered in-doubt tx as they become known due to e.g. partitioned nodes
rejoining the cluster, and the TM will deal with them when they are first seen. Note
however that some TMs assume that if they scan an RM and that RM does not subsequently
crash, no new in-doubt transactions will occur except from heuristics. Let's gloss
over how they can even detect a crash/recover of the RM if the driver masks it with
failover or the event happens during a period when the TM makes no call on the driver.
Such a TM will perform a recovery scan once at TM startup and not repeat. In such case you
may have in-doubt tx from nodes unavailable at that crucial time subsequently sitting
around for a prolonged period, tying up precious resources and potentially blocking
subsequent updates. Most RM vendors provide some kind of management capability for admins
to view and manually force completion of in!
-doubt tx. command line tool, jmx, web gui, whatever, just so long as it exists.
When a node crashes all the transactions that node owns (i.e. tx which were originated on
that node and XAResource instance residing on that node) automatically rollback, so that
no resources (locks mainly) are held. The only thing we need to make sure though is that
the given transaction ids (the one that heuristically rollback) are returned by
theXAResource.recover method - doable in the same way we handle prepares. I imagine that
we'll have to keep these XIDs until XAResource.forget(XID) is called, am I right? Is
it common/possible for people to use TM _without_ recovery? If so, this "held
heuristic completed TX" functionality should be configurable (enabled/disabled) in
order to avoid memory leaks (no recovery means .forget never gets called)
If you're using a transaction manager then use it all. Don't futz about and just
use this bit or that bit and still say you're using transactions ;-)
Mark.
---
Mark Little
mlittle(a)redhat.com
JBoss, by Red Hat
Registered Address: Red Hat UK Ltd, Amberley Place, 107-111 Peascod Street, Windsor,
Berkshire, SI4 1TE, United Kingdom.
Registered in UK and Wales under Company Registration No. 3798903 Directors: Michael
Cunningham (USA), Charlie Peters (USA), Matt Parsons (USA) and Brendan Lane (Ireland).
4883
days inactive
4884
days old
infinispan-dev@lists.jboss.org
8 comments
3 participants
participants (3)
-
Jonathan Halliday -
Mark Little -
Mircea Markus