[jboss-dev-forums] [Design of Messaging on JBoss (Messaging/JBoss)] - Re: Journal Cleanup and Journal Compactor

[clebert.suconic@jboss.com]

Before jumping into my proposed solution, let me just make a quick note about what's stored on the journal.

// A list of dataFiles (used files)
private final Queue dataFiles;

//A list of freeFiles
private final Queue freeFiles = new ConcurrentLinkedQueue();

//A list of freeFiles but already opened (for fast move forward on the journal)
private final BlockingQueue openedFiles;

//A list of Adds and updates for each recordID
// This is being renamed to recordsMap BTW
private final ConcurrentMap<Long, PosFiles> posFilesMap;


now the compacting would be:

Startup

exclusiveLockOnJournal (for a very short time, this is required to take a valid snapshot before compacting starts)
{
- Disallow reclaiming while compacting is being done
- set some flag such as compacting = true
- Take a snapshot of dataFiles, posFilesMap and pending transactions.
- I believe 
}


- for each dataFile on the snapshot
   - append valid records (based on the snapshot) to a new temporary datafile.  if the temporary datafile is full, open a new one
   - as records are appended, calculate the new posFilesMap


- As soon as the compacting is done, I need to rename temporary files (using the process you originaly described.. with a small mark file)

- I need also to update the posfilesMap.

I will take the list of updates and deletes tha happened while compacting was working, and replay them on the new posfilesMap (in a fast operation).


This is because at this point, I wouldn't have any information about deletes and updates.

- When a delete happen, you only have a neg added to DataFile, and I wouldn't know how to replay the information.

- For updates, I only have a list of what files took an update (inside PosFiles). You could have two updates on a same file, and each updates was sent to a different file

So far I need to compute that information, as I don't have it anywhere.

View the original post : http://www.jboss.org/index.html?module=bb&op=viewtopic&p=4240085#4240085

Reply to the post : http://www.jboss.org/index.html?module=bb&op=posting&mode=reply&p=4240085