[jboss-dev-forums] [Design of Messaging on JBoss (Messaging/JBoss)] - Re: Journal Cleanup and Journal Compactor

[clebert.suconic@jboss.com]

Full compacting won't eliminate the need for cleanup.

The previous post explained how to replicate the issue...

And here is a short example of what it happens...



You have these files:
F1 : SA1, SA2, SA3... SA100... A1
F2: A2, D1
F3: A3, D2
F4: A4, D3
...
Fn: An, D(n-1), A(n+1)


* A = Append, D = Delete, U=Update, F = File
* SA = Surviving Add (Nothing will ever delete those records)




On the example above, you have 100 records on F1, that will never be deleted.


As soon as you compact this, you will have:

New-F1: SA1, SA2, SA3... SA100. A(n+1)

That is... the live records from FN were compacted into New-F1 also.



After compacting is done, we delete that record that was on FN, and we add a new one:

New-F1: SA1, SA2, SA3... SA100.  A(n+1)
New-F2: D(n+1) A(n+2)


At this point there is a link between New-F1 and New-F2. 

Conclusion: once you have live records: compacting will aways be executed time to time, what will have consequences on performance.

We should have compacting being executed ocasionally.. not every 5 minutes. We need to cleanup those links somehow when they happen.


On the tests I did when I left 100 surviving records on the journal, compacting was called at least 3 times while sending/consuming 100K messages.

Tim said on the meeting today.. let compacting to happen every 30 minutes.. but that wouldn't work, as nothing will be reclaimed for 30 minutes (you could eat up all your disk space in production because of this.. as bad as it is).


Possible solution is to cleanup New-F1 only (eliminating A(n+1) phisically from New-F1 will eliminate the link between New-F1 and New-F2). I could use the Compacting code for that, and that would work as long as we start from the beggining of the list.


I don't intend to do any deep technical discussions around the cleanup implementation now, since I need to finish compacting first. So far I only want to point the issue that we really need the cleanup also.

View the original post : http://www.jboss.org/index.html?module=bb&op=viewtopic&p=4241546#4241546

Reply to the post : http://www.jboss.org/index.html?module=bb&op=posting&mode=reply&p=4241546