[JIRA] (HHH-16446) org.hibernate.engine.spi.EntityKey consumes a lot of memory
by Hrushikesh Mahapatro (JIRA)
Hrushikesh Mahapatro ( https://hibernate.atlassian.net/secure/ViewProfile.jspa?accountId=6097b6b... ) *created* an issue
Hibernate ORM ( https://hibernate.atlassian.net/browse/HHH?atlOrigin=eyJpIjoiMjhiOWE4OWQ2... ) / Bug ( https://hibernate.atlassian.net/browse/HHH-16446?atlOrigin=eyJpIjoiMjhiOW... ) HHH-16446 ( https://hibernate.atlassian.net/browse/HHH-16446?atlOrigin=eyJpIjoiMjhiOW... ) org.hibernate.engine.spi.EntityKey consumes a lot of memory ( https://hibernate.atlassian.net/browse/HHH-16446?atlOrigin=eyJpIjoiMjhiOW... )
Issue Type: Bug Affects Versions: 6.2.0 Assignee: Unassigned Components: hibernate-core, hibernate-orm-modules Created: 06/Apr/2023 08:11 AM Environment: Spring boot 3.0.5 Hibernate 6.2.0 Spring Data JPA 3.x Labels: api core Priority: Blocker Reporter: Hrushikesh Mahapatro ( https://hibernate.atlassian.net/secure/ViewProfile.jspa?accountId=6097b6b... )
1
We have just migrated our application to Springboot 3.x, Spring Data JPA 3.1.x & Hibernate 6.1.7 Final.
Immediately we observed heap memory spike as we deployed same code without any significant changes.
Context: We're trying to fetch a stream of objects from SQL db, and write straight onto a csv file in azure blob storage. As data volume is large 13+GB, we're setting few query hints to fetch 500 objects at a time, disable cacheing also read committed.
We're flushing/Clearing both OpenCSV writer and entity manager every 1500 records. This was working all good, until we did this migration.
Few code snippets & Profiler Analysis
@Transactional(readOnly = true, isolation = Isolation.READ_COMMITTED)
public void generateFile() {
try (Stream<MyEntity> supplier = myRepo.getDataStream()) {
writeOntoFile(ApplicationConstants.REPORT_DIRECTORY, supplier);
} catch (IOException ex) {
log.error("Error occurred while generating file", ex);
}
}
private <T extends HasId<?>> void writeOntoFile(final String reportDirectory,
Stream<T> dataStreamSupplier) throws IOException {
final String blobName = blobName(reportDirectory);
try (ICSVWriter writer = blobStorageService.getCsvWriterBlobOutputStream(storageContainer, blobName)) {
log.info("Writing to {} using beantoCsv mapping", reportDirectory);
StatefulBeanToCsv<T> beanToCsv = new StatefulBeanToCsvBuilder<T>(writer).build();
final int[] flushCounter = new int[1];
dataStreamSupplier.forEach(dataRecord -> {
writeCSV(reportDirectory, dataRecord, beanToCsv);
flushCounter[0]++;
if (flushCounter[0] % CLEAR_INTERVAL == 0) {
entityManager.clear();
entityManagerFactory.getCache().evict(MyEntity.class);
try {
writer.flush();
} catch (IOException ioe) {
log.error("Unable to flush the writing stream");
throw new RuntimeException(ioe);
}
}
});
log.info("Data written in With Azure Blob file {}", blobName);
} catch (RuntimeException re) {
log.error("Error occurred while processing report {}", reportDirectory, re);
throw re;
}
Repository Class
@Query("select e from MyEntity e JOIN FETCH e.uuid JOIN FETCH e.sourceTable")
@QueryHints(value = {
@QueryHint(name = HINT_FETCH_SIZE, value = ApplicationConstants.DATA_FETCH_SIZE),
@QueryHint(name = HINT_CACHEABLE, value = "false"),
@QueryHint(name = HINT_READONLY, value = "true")
})
Stream<MyEntity> getDataStream();
Profiler Log: I tried few things such as entityManagerFactory.getCache.evictAll() seems doesn't work.
We're looking to clear these EntityKey objects from Heap. We tried pagination but it's too slow to export such a large volume data.
( https://hibernate.atlassian.net/browse/HHH-16446#add-comment?atlOrigin=ey... ) Add Comment ( https://hibernate.atlassian.net/browse/HHH-16446#add-comment?atlOrigin=ey... )
Get Jira notifications on your phone! Download the Jira Cloud app for Android ( https://play.google.com/store/apps/details?id=com.atlassian.android.jira.... ) or iOS ( https://itunes.apple.com/app/apple-store/id1006972087?pt=696495&ct=EmailN... ) This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100221- sha1:8f8083a )
3 years
[JIRA] (HSEARCH-4574) Explicit binding of constructor parameters for @ProjectionConstructor
by Yoann Rodière (JIRA)
Yoann Rodière ( https://hibernate.atlassian.net/secure/ViewProfile.jspa?accountId=557058%... ) *updated* an issue
Hibernate Search ( https://hibernate.atlassian.net/browse/HSEARCH?atlOrigin=eyJpIjoiZDlkMzFl... ) / New Feature ( https://hibernate.atlassian.net/browse/HSEARCH-4574?atlOrigin=eyJpIjoiZDl... ) HSEARCH-4574 ( https://hibernate.atlassian.net/browse/HSEARCH-4574?atlOrigin=eyJpIjoiZDl... ) Explicit binding of constructor parameters for @ProjectionConstructor ( https://hibernate.atlassian.net/browse/HSEARCH-4574?atlOrigin=eyJpIjoiZDl... )
Change By: Yoann Rodière ( https://hibernate.atlassian.net/secure/ViewProfile.jspa?accountId=557058%... )
Follows up on [https://hibernate.atlassian.net/browse/HSEARCH-3927|https://hibernate.atl...].
Use case: define a parameter of a projection constructor as a specific inner projection either something completely different from field projections such as an identifier projection, or more precise configuration of field value/object projections.
For example:
{code:java}@ProjectionConstructor
record MyProjection(@IdProjection String id, @FieldProjection(convert = ValueConvert.NO) String myEnum){code}
Currently, all constructor parameters are bound implicitly, which limits users to field projections or object projections. There is no ID projection in particular, which prevents from fully loading an entity through its projection constructor, since it can’t project on the ID. This also means {{-parameters}} _must_ be passed to the compiler when compiling projection constructors.
Core set of annotations (must be implemented):
* {{@IdProjection}}
* {{@FieldProjection}}
* {{@ObjectProjection}}
** we might want a {{includeDepth}} in here ,
(=> moved to [https://hibernate.atlassian.net/browse/HSEARCH-4725|https://hibernate.atl...] )
* {{@ProjectionBinding(binder = ...)}} (for custom projection annotations)
Less useful/more challenging annotations that could be split into another ticket:
* {{@ScoreProjection}}
* {{@DistanceProjection}} (=> moved to [https://hibernate.atlassian.net/browse/HSEARCH-4807|https://hibernate.atl...] )
* {{@EntityProjection}}
* {{@EntityReferenceProjection}}
* {{@DocumentReferenceProjection}}
* Others, see all projections available.
Some things to test in particular:
* There should be no error if {{-parameters}} is not used when compiling, as long as all constructor parameters of a projection constructor have a {{@*Projection}} annotation (and that projection specifies the field path, if relevant).
* There should be an error if {{-parameters}} is not used when compiling, and all constructor parameters of a projection constructor have a {{@*Projection}} annotation, but one of them is annotated with {{@FieldProjection}} or {{@ObjectProjection}} and does not specify the field path.
* There should be an error when using two {{@*Projection}} annotations on the same constructor parameter.
* And more (the list above is not exhaustive, obviously).
( https://hibernate.atlassian.net/browse/HSEARCH-4574#add-comment?atlOrigin... ) Add Comment ( https://hibernate.atlassian.net/browse/HSEARCH-4574#add-comment?atlOrigin... )
Get Jira notifications on your phone! Download the Jira Cloud app for Android ( https://play.google.com/store/apps/details?id=com.atlassian.android.jira.... ) or iOS ( https://itunes.apple.com/app/apple-store/id1006972087?pt=696495&ct=EmailN... ) This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100221- sha1:8f8083a )
3 years
[JIRA] (HHH-16445) avg() no longer works with enums
by Stéphane Nicoll (JIRA)
Stéphane Nicoll ( https://hibernate.atlassian.net/secure/ViewProfile.jspa?accountId=557058%... ) *created* an issue
Hibernate ORM ( https://hibernate.atlassian.net/browse/HHH?atlOrigin=eyJpIjoiZGYxNjA3Nzc2... ) / Bug ( https://hibernate.atlassian.net/browse/HHH-16445?atlOrigin=eyJpIjoiZGYxNj... ) HHH-16445 ( https://hibernate.atlassian.net/browse/HHH-16445?atlOrigin=eyJpIjoiZGYxNj... ) avg() no longer works with enums ( https://hibernate.atlassian.net/browse/HHH-16445?atlOrigin=eyJpIjoiZGYxNj... )
Issue Type: Bug Affects Versions: 6.2.0 Assignee: Unassigned Components: query-criteria Created: 06/Apr/2023 07:24 AM Priority: Major Reporter: Stéphane Nicoll ( https://hibernate.atlassian.net/secure/ViewProfile.jspa?accountId=557058%... )
We have a very basic smoke test that uses an enum to represent a rating of a review. We use the avg function to compute an average of the reviews.
Upon upgrading to Hibernate 6.2, this now fails as follows:
Caused by: org.hibernate.QueryException: Parameter 1 of function avg() has type NUMERIC, but argument is of type smoketest.data.jpa.domain.Rating
at org.hibernate.query.sqm.produce.function.ArgumentTypesValidator.throwError(ArgumentTypesValidator.java:248) ~[hibernate-core-6.2.0.Final.jar:6.2.0.Final]
at org.hibernate.query.sqm.produce.function.ArgumentTypesValidator.checkType(ArgumentTypesValidator.java:213) ~[hibernate-core-6.2.0.Final.jar:6.2.0.Final]
at org.hibernate.query.sqm.produce.function.ArgumentTypesValidator.validate(ArgumentTypesValidator.java:97) ~[hibernate-core-6.2.0.Final.jar:6.2.0.Final]
at org.hibernate.query.sqm.function.AbstractSqmFunctionDescriptor.generateAggregateSqmExpression(AbstractSqmFunctionDescriptor.java:121) ~[hibernate-core-6.2.0.Final.jar:6.2.0.Final]
at org.hibernate.query.hql.internal.SemanticQueryBuilder.visitGenericFunction(SemanticQueryBuilder.java:3984) ~[hibernate-core-6.2.0.Final.jar:6.2.0.Final]
at org.hibernate.grammars.hql.HqlParser$GenericFunctionContext.accept(HqlParser.java:11403) ~[hibernate-core-6.2.0.Final.jar:6.2.0.Final]
at org.antlr.v4.runtime.tree.AbstractParseTreeVisitor.visitChildren(AbstractParseTreeVisitor.java:46) ~[antlr4-runtime-4.10.1.jar:4.10.1]
at org.hibernate.grammars.hql.HqlParserBaseVisitor.visitFunction(HqlParserBaseVisitor.java:1217) ~[hibernate-core-6.2.0.Final.jar:6.2.0.Final]
at org.hibernate.grammars.hql.HqlParser$FunctionContext.accept(HqlParser.java:11171) ~[hibernate-core-6.2.0.Final.jar:6.2.0.Final]
at org.hibernate.query.hql.internal.SemanticQueryBuilder.visitFunctionExpression(SemanticQueryBuilder.java:1732) ~[hibernate-core-6.2.0.Final.jar:6.2.0.Final]
at org.hibernate.query.hql.internal.SemanticQueryBuilder.visitFunctionExpression(SemanticQueryBuilder.java:253) ~[hibernate-core-6.2.0.Final.jar:6.2.0.Final]
at org.hibernate.grammars.hql.HqlParser$FunctionExpressionContext.accept(HqlParser.java:7476) ~[hibernate-core-6.2.0.Final.jar:6.2.0.Final]
at org.antlr.v4.runtime.tree.AbstractParseTreeVisitor.visitChildren(AbstractParseTreeVisitor.java:46) ~[antlr4-runtime-4.10.1.jar:4.10.1]
at org.hibernate.grammars.hql.HqlParserBaseVisitor.visitBarePrimaryExpression(HqlParserBaseVisitor.java:720) ~[hibernate-core-6.2.0.Final.jar:6.2.0.Final]
at org.hibernate.grammars.hql.HqlParser$BarePrimaryExpressionContext.accept(HqlParser.java:7064) ~[hibernate-core-6.2.0.Final.jar:6.2.0.Final]
at org.antlr.v4.runtime.tree.AbstractParseTreeVisitor.visitChildren(AbstractParseTreeVisitor.java:46) ~[antlr4-runtime-4.10.1.jar:4.10.1]
at org.hibernate.grammars.hql.HqlParserBaseVisitor.visitExpressionOrPredicate(HqlParserBaseVisitor.java:860) ~[hibernate-core-6.2.0.Final.jar:6.2.0.Final]
at org.hibernate.grammars.hql.HqlParser$ExpressionOrPredicateContext.accept(HqlParser.java:7813) ~[hibernate-core-6.2.0.Final.jar:6.2.0.Final]
at org.hibernate.query.hql.internal.SemanticQueryBuilder.visitSelectableNode(SemanticQueryBuilder.java:1278) ~[hibernate-core-6.2.0.Final.jar:6.2.0.Final]
at org.hibernate.query.hql.internal.SemanticQueryBuilder.visitSelection(SemanticQueryBuilder.java:1252) ~[hibernate-core-6.2.0.Final.jar:6.2.0.Final]
at org.hibernate.query.hql.internal.SemanticQueryBuilder.visitSelectClause(SemanticQueryBuilder.java:1235) ~[hibernate-core-6.2.0.Final.jar:6.2.0.Final]
at org.hibernate.query.hql.internal.SemanticQueryBuilder.visitQuery(SemanticQueryBuilder.java:1155) ~[hibernate-core-6.2.0.Final.jar:6.2.0.Final]
at org.hibernate.query.hql.internal.SemanticQueryBuilder.visitQuerySpecExpression(SemanticQueryBuilder.java:937) ~[hibernate-core-6.2.0.Final.jar:6.2.0.Final]
at org.hibernate.query.hql.internal.SemanticQueryBuilder.visitQuerySpecExpression(SemanticQueryBuilder.java:253) ~[hibernate-core-6.2.0.Final.jar:6.2.0.Final]
at org.hibernate.grammars.hql.HqlParser$QuerySpecExpressionContext.accept(HqlParser.java:1818) ~[hibernate-core-6.2.0.Final.jar:6.2.0.Final]
at org.hibernate.query.hql.internal.SemanticQueryBuilder.visitSimpleQueryGroup(SemanticQueryBuilder.java:931) ~[hibernate-core-6.2.0.Final.jar:6.2.0.Final]
at org.hibernate.query.hql.internal.SemanticQueryBuilder.visitSimpleQueryGroup(SemanticQueryBuilder.java:253) ~[hibernate-core-6.2.0.Final.jar:6.2.0.Final]
at org.hibernate.grammars.hql.HqlParser$SimpleQueryGroupContext.accept(HqlParser.java:1711) ~[hibernate-core-6.2.0.Final.jar:6.2.0.Final]
at org.hibernate.query.hql.internal.SemanticQueryBuilder.visitSelectStatement(SemanticQueryBuilder.java:418) ~[hibernate-core-6.2.0.Final.jar:6.2.0.Final]
at org.hibernate.query.hql.internal.SemanticQueryBuilder.visitStatement(SemanticQueryBuilder.java:377) ~[hibernate-core-6.2.0.Final.jar:6.2.0.Final]
at org.hibernate.query.hql.internal.SemanticQueryBuilder.buildSemanticModel(SemanticQueryBuilder.java:295) ~[hibernate-core-6.2.0.Final.jar:6.2.0.Final]
at org.hibernate.query.hql.internal.StandardHqlTranslator.translate(StandardHqlTranslator.java:81) ~[hibernate-core-6.2.0.Final.jar:6.2.0.Final]
at org.hibernate.internal.AbstractSharedSessionContract.lambda$interpretHql$2(AbstractSharedSessionContract.java:744) ~[hibernate-core-6.2.0.Final.jar:6.2.0.Final]
at org.hibernate.query.internal.QueryInterpretationCacheStandardImpl.createHqlInterpretation(QueryInterpretationCacheStandardImpl.java:141) ~[hibernate-core-6.2.0.Final.jar:6.2.0.Final]
at org.hibernate.query.internal.QueryInterpretationCacheStandardImpl.resolveHqlInterpretation(QueryInterpretationCacheStandardImpl.java:128) ~[hibernate-core-6.2.0.Final.jar:6.2.0.Final]
at org.hibernate.internal.AbstractSharedSessionContract.interpretHql(AbstractSharedSessionContract.java:741) ~[hibernate-core-6.2.0.Final.jar:6.2.0.Final]
at org.hibernate.internal.AbstractSharedSessionContract.createQuery(AbstractSharedSessionContract.java:786) ~[hibernate-core-6.2.0.Final.jar:6.2.0.Final]
... 49 common frames omitted
This fails as the type code is -101977 which seems to be type when an enum has an unknown jdbc type.
I can try to build a sample if you can’t easily figure out what happened between CR1 and CR2. Thanks
( https://hibernate.atlassian.net/browse/HHH-16445#add-comment?atlOrigin=ey... ) Add Comment ( https://hibernate.atlassian.net/browse/HHH-16445#add-comment?atlOrigin=ey... )
Get Jira notifications on your phone! Download the Jira Cloud app for Android ( https://play.google.com/store/apps/details?id=com.atlassian.android.jira.... ) or iOS ( https://itunes.apple.com/app/apple-store/id1006972087?pt=696495&ct=EmailN... ) This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100221- sha1:8f8083a )
3 years