[jboss-jira] [JBoss JIRA] Commented: (JBMESSAGING-1678) OME on the Journal when processing too many files

[Clebert Suconic (JIRA)]

    [ https://jira.jboss.org/jira/browse/JBMESSAGING-1678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12475298#action_12475298 ] 

Clebert Suconic commented on JBMESSAGING-1678:
----------------------------------------------

This is because the method load has two lists:

      final Set<Long> recordsToDelete = new HashSet<Long>();
      final List<RecordInfo> records = new ArrayList<RecordInfo>();


The method load will add to both recordsToDelete and records.  At the end of the method, it will only consider the records that were not deleted:

      for (RecordInfo record : records)
      {
         if (!recordsToDelete.contains(record.id))
         {
            committedRecords.add(record);
         }
      }



But In a scenario where compacting is not working, records and recordsToDelete will grow too large, up to the point it won't fit on the memory in a single load.


We could just make compacting mandatory, or change the loading method to use some sort of Map. It's a bit of a challange here as the journal needs to keep the adds and update records in order.

> OME on the Journal when processing too many files
> -------------------------------------------------
>
>                 Key: JBMESSAGING-1678
>                 URL: https://jira.jboss.org/jira/browse/JBMESSAGING-1678
>             Project: JBoss Messaging
>          Issue Type: Bug
>    Affects Versions: 2.0.0 Beta1, 2.0.0.beta2, 2.0.0.beta3
>            Reporter: Clebert Suconic
>            Assignee: Clebert Suconic
>             Fix For: 2.0.0
>
>
> Set one producer.. sending one message / second (about 200 messages)
> Set another producer/consumer (in another destination) sending as fast as it can. (about 500K messages).
> At the end you will have 200 files... but only 200 messages (one valid on each file).
> This will lead to an OME during reload.
> On trunk, you need to deactivate compacting to replicate this.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira