Hrushikesh Mahapatro (
https://hibernate.atlassian.net/secure/ViewProfile.jspa?accountId=6097b6b...
) *created* an issue
Hibernate ORM (
https://hibernate.atlassian.net/browse/HHH?atlOrigin=eyJpIjoiMjhiOWE4OWQ2...
) / Bug (
https://hibernate.atlassian.net/browse/HHH-16446?atlOrigin=eyJpIjoiMjhiOW...
) HHH-16446 (
https://hibernate.atlassian.net/browse/HHH-16446?atlOrigin=eyJpIjoiMjhiOW...
) org.hibernate.engine.spi.EntityKey consumes a lot of memory (
https://hibernate.atlassian.net/browse/HHH-16446?atlOrigin=eyJpIjoiMjhiOW...
)
Issue Type: Bug Affects Versions: 6.2.0 Assignee: Unassigned Components: hibernate-core,
hibernate-orm-modules Created: 06/Apr/2023 08:11 AM Environment: Spring boot 3.0.5
Hibernate 6.2.0 Spring Data JPA 3.x Labels: api core Priority: Blocker Reporter:
Hrushikesh Mahapatro (
https://hibernate.atlassian.net/secure/ViewProfile.jspa?accountId=6097b6b...
)
1
We have just migrated our application to Springboot 3.x, Spring Data JPA 3.1.x &
Hibernate 6.1.7 Final.
Immediately we observed heap memory spike as we deployed same code without any significant
changes.
Context: We're trying to fetch a stream of objects from SQL db, and write straight
onto a csv file in azure blob storage. As data volume is large 13+GB, we're setting
few query hints to fetch 500 objects at a time, disable cacheing also read committed.
We're flushing/Clearing both OpenCSV writer and entity manager every 1500 records.
This was working all good, until we did this migration.
Few code snippets & Profiler Analysis
@Transactional(readOnly = true, isolation = Isolation.READ_COMMITTED)
public void generateFile() {
try (Stream<MyEntity> supplier = myRepo.getDataStream()) {
writeOntoFile(ApplicationConstants.REPORT_DIRECTORY, supplier);
} catch (IOException ex) {
log.error("Error occurred while generating file", ex);
}
}
private <T extends HasId<?>> void writeOntoFile(final String reportDirectory,
Stream<T> dataStreamSupplier)
throws IOException {
final String blobName = blobName(reportDirectory);
try (ICSVWriter writer =
blobStorageService.getCsvWriterBlobOutputStream(storageContainer, blobName)) {
log.info("Writing to {} using beantoCsv mapping", reportDirectory);
StatefulBeanToCsv<T> beanToCsv = new
StatefulBeanToCsvBuilder<T>(writer).build();
final int[] flushCounter = new int[1];
dataStreamSupplier.forEach(dataRecord -> {
writeCSV(reportDirectory, dataRecord, beanToCsv);
flushCounter[0]++;
if (flushCounter[0] % CLEAR_INTERVAL == 0) {
entityManager.clear();
entityManagerFactory.getCache().evict(MyEntity.class);
try {
writer.flush();
} catch (IOException ioe) {
log.error("Unable to flush the writing stream");
throw new RuntimeException(ioe);
}
}
});
log.info("Data written in With Azure Blob file {}", blobName);
} catch (RuntimeException re) {
log.error("Error occurred while processing report {}",
reportDirectory, re);
throw re;
}
Repository Class
@Query("select e from MyEntity e JOIN FETCH e.uuid JOIN FETCH e.sourceTable")
@QueryHints(value = {
@QueryHint(name = HINT_FETCH_SIZE, value =
ApplicationConstants.DATA_FETCH_SIZE),
@QueryHint(name = HINT_CACHEABLE, value = "false"),
@QueryHint(name = HINT_READONLY, value = "true")
})
Stream<MyEntity> getDataStream();
Profiler Log: I tried few things such as entityManagerFactory.getCache.evictAll() seems
doesn't work.
We're looking to clear these EntityKey objects from Heap. We tried pagination but
it's too slow to export such a large volume data.
(
https://hibernate.atlassian.net/browse/HHH-16446#add-comment?atlOrigin=ey...
) Add Comment (
https://hibernate.atlassian.net/browse/HHH-16446#add-comment?atlOrigin=ey...
)
Get Jira notifications on your phone! Download the Jira Cloud app for Android (
https://play.google.com/store/apps/details?id=com.atlassian.android.jira....
) or iOS (
https://itunes.apple.com/app/apple-store/id1006972087?pt=696495&ct=Em...
) This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100221- sha1:8f8083a )