1
We have just migrated our application to Springboot 3.x, Spring Data JPA 3.1.x & Hibernate 6.1.7 Final.
Immediately we observed heap memory spike as we deployed same code without any significant changes.
Context: We're trying to fetch a stream of objects from SQL db, and write straight onto a csv file in azure blob storage. As data volume is large 13+GB, we're setting few query hints to fetch 500 objects at a time, disable cacheing also read committed.
We're flushing/Clearing both OpenCSV writer and entity manager every 1500 records. This was working all good, until we did this migration.
Few code snippets & Profiler Analysis
{noformat}@Transactional(readOnly = true, isolation = Isolation.READ_COMMITTED)
public void generateFile() {
try (Stream<MyEntity> supplier = myRepo.getDataStream()) {
writeOntoFile(ApplicationConstants.REPORT_DIRECTORY, supplier);
} catch (IOException ex) {
log.error("Error occurred while generating file", ex);
}
}
{noformat}
{noformat}private <T extends HasId<?>> void writeOntoFile(final String reportDirectory,
Stream<T> dataStreamSupplier) throws IOException {
final String blobName = blobName(reportDirectory);
try (ICSVWriter writer = blobStorageService.getCsvWriterBlobOutputStream(storageContainer, blobName)) {
log.info("Writing to {} using beantoCsv mapping", reportDirectory);
StatefulBeanToCsv<T> beanToCsv = new StatefulBeanToCsvBuilder<T>(writer).build();
final int[] flushCounter = new int[1];
dataStreamSupplier.forEach(dataRecord -> {
writeCSV(reportDirectory, dataRecord, beanToCsv);
flushCounter[0]++;
if (flushCounter[0] % CLEAR_INTERVAL == 0) {
entityManager.clear();
entityManagerFactory.getCache().evict(MyEntity.class);
try {
writer.flush();
} catch (IOException ioe) {
log.error("Unable to flush the writing stream");
throw new RuntimeException(ioe);
}
}
});
log.info("Data written in With Azure Blob file {}", blobName);
} catch (RuntimeException re) {
log.error("Error occurred while processing report {}", reportDirectory, re);
throw re;
}
{noformat}
Repository Class
{noformat} @Query("select e from MyEntity e JOIN FETCH e.uuid JOIN FETCH e.sourceTable")
@QueryHints(value = {
@QueryHint(name = HINT_FETCH_SIZE, value = ApplicationConstants.DATA_FETCH_SIZE),
@QueryHint(name = HINT_CACHEABLE, value = "false"),
@QueryHint(name = HINT_READONLY, value = "true")
})
Stream<MyEntity> getDataStream();
{noformat}
Profiler Log: I tried few things such as entityManagerFactory.getCache.evictAll() seems doesn't work.
!https://i.stack.imgur.com/vSRQL.png|alt="enter image description here"!
We're looking to clear these EntityKey objects from Heap. We tried pagination but it's too slow to export such a large volume data. |
|
