Design notes for work on https://issues.jboss.org/browse/AS7-444.
My current development topic branch for this is https://github.com/bstansberry/jboss-as/tree/AS7-444. (Note that this is a personal branch; I make no guarantees not to change its commit history.) This branch is based on work previously done by John Bailey.
Design notes:
1) Processing location in OperationContext
a) when done
b) when decision to not proceed is made in OperationContext
2) Log record format:
a) date
b) read-only?
c) boot?
d) (if boot) version (as string???)
e) uuid
f) model hash length
g) model hash
h) overall hash length
i) overall hash
j) result action
k) operation
l) userid
m) length of a->j (scan from end to last boot during validation; alternative is to write the length first and scan from beginning to last boot)
3) Index record format:
a) overall hash length
b) overall hash
c) model hash length
d) model hash
4) Mgmt operations
a) truncate (back to last boot)
i) don't discard previous; just write under a different file name
b) read log (params -- from (default 0); batch-size (default 20; -1 means all))
c) validate log
d) config changes
5) Validation:
a) compare persisted model hash to current model
b) scan back through log to last config-file-modified boot or version-change boot. We stop at version-change boot to avoid spurious validation problems resulting from changes in model construction across versions
c) recreate by re-running ops
-- validation problems
a) need to verify model hash at each stage -- special controller that ignores all runtime ops
i) problem -- how to recreate domain ops involving remote slaves that don't exist?
ii) this "special controller" could end up duplicating a lot of logic just for validation
b) reads and runtime ops cannot be verified against model; it can only be confirmed that their hash is consistent with subsequent hashes in the log. Tamper resistant as modification would require modifying all subsequent log entries, but not tamper proof as this could be done.
6) Domain mode
a) tracking requests from master to slaves
apply a uuid to requests
-- as a header
include uuid in log record
b) detecting host.xml config changes at boot
problem -- can't detect host.xml config changes at boot, as model changes may be due to domain.xml stuff coming from the master
so, independent domain and host logging
but, what about non-model affecting ops? log twice?
c) domain logging on slaves? only if --backup-dc is set?
d) remote-slave-only logging on master?
yes, to maintain a complete record of what was done
7) Non-model-affecting ops
a) re-calculating the model hash for all such ops is too expensive
b) store the hash(es) in the ModelController along with the model
c) provide the hash along with the model to OperationContext
d) persisters recalc hashes and return them
8) Config
a) <audit-log> element
b) part of <management> section
c) resource is /core-service=audit-log
d) attributes
i) path
ii) relative-to
iii) log-read-only?
iv) daily-rolling??? (hassle; tempting to defer to later release but it's probably necessary immediately)
9) Other issues
a) disable logging overhead for dev scenarios
i) not sure how to disable it during initial boot (until whatever config element that turns it on/off is read) -- probably maintain log data in memory and only write on last boot op
b) parallel boot handlers should not log
c) fluctuating model hashes
i) if op took the controller lock, that lock ensures proper ordering of model hashes
ii) if op does not take controller lock (reads) earlier model hashes can intermix with later -- validation needs to account for this