[JIRA] (HHH-13922) When an audited column uses a columnDefinition, schema validation may fail.
by Chris Cranford (JIRA)
Chris Cranford ( https://hibernate.atlassian.net/secure/ViewProfile.jspa?accountId=557058%... ) *created* an issue
Hibernate ORM ( https://hibernate.atlassian.net/browse/HHH?atlOrigin=eyJpIjoiMTY4ODc3MmZm... ) / Bug ( https://hibernate.atlassian.net/browse/HHH-13922?atlOrigin=eyJpIjoiMTY4OD... ) HHH-13922 ( https://hibernate.atlassian.net/browse/HHH-13922?atlOrigin=eyJpIjoiMTY4OD... ) When an audited column uses a columnDefinition, schema validation may fail. ( https://hibernate.atlassian.net/browse/HHH-13922?atlOrigin=eyJpIjoiMTY4OD... )
Issue Type: Bug Affects Versions: 5.4.11, 5.4.12, 5.4.13 Assignee: Chris Cranford ( https://hibernate.atlassian.net/secure/ViewProfile.jspa?accountId=557058%... ) Components: hibernate-envers Created: 01/Apr/2020 11:40 AM Priority: Major Reporter: Chris Cranford ( https://hibernate.atlassian.net/secure/ViewProfile.jspa?accountId=557058%... )
As mentioned on SO https://stackoverflow.com/questions/60931264/hibernate-envers-org-hiberna... , the following exception:
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'entityManagerFactory' defined in class path resource [org/springframework/boot/autoconfigure/orm/jpa/HibernateJpaConfiguration.class]: Invocation of init method failed; nested exception is javax.persistence.PersistenceException: [PersistenceUnit: default ] Unable to build Hibernate SessionFactory; nested exception is org.hibernate.tool.schema.spi.SchemaManagementException: Schema-validation: wrong column type encountered in column [password] in table [users_aud]; found [binary (Types#BINARY)], but expecting [varchar(255) (Types#VARCHAR)]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1796) ~[spring-beans-5.2.5.RELEASE.jar:5.2.5.RELEASE]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:595) ~[spring-beans-5.2.5.RELEASE.jar:5.2.5.RELEASE]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:517) ~[spring-beans-5.2.5.RELEASE.jar:5.2.5.RELEASE]
at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:323) ~[spring-beans-5.2.5.RELEASE.jar:5.2.5.RELEASE]
at org.springframework.beans.factory.support.AbstractBeanFactory$$Lambda$276/0000000000000000.getObject(Unknown Source) ~[na:na]
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222) ~[spring-beans-5.2.5.RELEASE.jar:5.2.5.RELEASE]
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:321) ~[spring-beans-5.2.5.RELEASE.jar:5.2.5.RELEASE]
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202) ~[spring-beans-5.2.5.RELEASE.jar:5.2.5.RELEASE]
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:330) ~[spring-beans-5.2.5.RELEASE.jar:5.2.5.RELEASE]
... 117 common frames omitted
Caused by: javax.persistence.PersistenceException: [PersistenceUnit: default ] Unable to build Hibernate SessionFactory; nested exception is org.hibernate.tool.schema.spi.SchemaManagementException: Schema-validation: wrong column type encountered in column [password] in table [users_aud]; found [binary (Types#BINARY)], but expecting [varchar(255) (Types#VARCHAR)]
at org.springframework.orm.jpa.AbstractEntityManagerFactoryBean.buildNativeEntityManagerFactory(AbstractEntityManagerFactoryBean.java:403) ~[spring-orm-5.2.5.RELEASE.jar:5.2.5.RELEASE]
at org.springframework.orm.jpa.AbstractEntityManagerFactoryBean.afterPropertiesSet(AbstractEntityManagerFactoryBean.java:378) ~[spring-orm-5.2.5.RELEASE.jar:5.2.5.RELEASE]
at org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean.afterPropertiesSet(LocalContainerEntityManagerFactoryBean.java:341) ~[spring-orm-5.2.5.RELEASE.jar:5.2.5.RELEASE]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1855) ~[spring-beans-5.2.5.RELEASE.jar:5.2.5.RELEASE]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1792) ~[spring-beans-5.2.5.RELEASE.jar:5.2.5.RELEASE]
... 125 common frames omitted
Caused by: org.hibernate.tool.schema.spi.SchemaManagementException: Schema-validation: wrong column type encountered in column [password] in table [users_aud]; found [binary (Types#BINARY)], but expecting [varchar(255) (Types#VARCHAR)]
at org.hibernate.tool.schema.internal.AbstractSchemaValidator.validateColumnType(AbstractSchemaValidator.java:159) ~[hibernate-core-5.4.12.Final.jar:5.4.12.Final]
at org.hibernate.tool.schema.internal.AbstractSchemaValidator.validateTable(AbstractSchemaValidator.java:143) ~[hibernate-core-5.4.12.Final.jar:5.4.12.Final]
at org.hibernate.tool.schema.internal.GroupedSchemaValidatorImpl.validateTables(GroupedSchemaValidatorImpl.java:42) ~[hibernate-core-5.4.12.Final.jar:5.4.12.Final]
at org.hibernate.tool.schema.internal.AbstractSchemaValidator.performValidation(AbstractSchemaValidator.java:89) ~[hibernate-core-5.4.12.Final.jar:5.4.12.Final]
at org.hibernate.tool.schema.internal.AbstractSchemaValidator.doValidation(AbstractSchemaValidator.java:68) ~[hibernate-core-5.4.12.Final.jar:5.4.12.Final]
at org.hibernate.tool.schema.spi.SchemaManagementToolCoordinator.performDatabaseAction(SchemaManagementToolCoordinator.java:192) ~[hibernate-core-5.4.12.Final.jar:5.4.12.Final]
at org.hibernate.tool.schema.spi.SchemaManagementToolCoordinator.process(SchemaManagementToolCoordinator.java:73) ~[hibernate-core-5.4.12.Final.jar:5.4.12.Final]
at org.hibernate.internal.SessionFactoryImpl.<init>(SessionFactoryImpl.java:314) ~[hibernate-core-5.4.12.Final.jar:5.4.12.Final]
at org.hibernate.boot.internal.SessionFactoryBuilderImpl.build(SessionFactoryBuilderImpl.java:468) ~[hibernate-core-5.4.12.Final.jar:5.4.12.Final]
at org.hibernate.jpa.boot.internal.EntityManagerFactoryBuilderImpl.build(EntityManagerFactoryBuilderImpl.java:1237) ~[hibernate-core-5.4.12.Final.jar:5.4.12.Final]
at org.springframework.orm.jpa.vendor.SpringHibernateJpaPersistenceProvider.createContainerEntityManagerFactory(SpringHibernateJpaPersistenceProvider.java:58) ~[spring-orm-5.2.5.RELEASE.jar:5.2.5.RELEASE]
at org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean.createNativeEntityManagerFactory(LocalContainerEntityManagerFactoryBean.java:365) ~[spring-orm-5.2.5.RELEASE.jar:5.2.5.RELEASE]
at org.springframework.orm.jpa.AbstractEntityManagerFactoryBean.buildNativeEntityManagerFactory(AbstractEntityManagerFactoryBean.java:391) ~[spring-orm-5.2.5.RELEASE.jar:5.2.5.RELEASE]
... 129 common frames omitted
occurs for an entity using the following mapping:
@Entity
@Audited
@Inheritance(strategy = InheritanceType.JOINED)
@NoArgsConstructor
@Getter
@Table(name = "users" )
public abstract class User extends HasId implements UserDetails {
@Column(columnDefinition = "BINARY(60)" , nullable = false )
private String password;
}
The entity database model is currently:
create table users
(
password binary(60) not null ,
) engine = InnoDB;
create table users_aud
(
id bigint not null ,
rev integer not null ,
revtype tinyint,
revend integer,
password binary(60),
password_mod bit,
primary key (id, rev)
) engine = InnoDB;
This appears it could be related to https://hibernate.atlassian.net/browse/HHH-10844.
( https://hibernate.atlassian.net/browse/HHH-13922#add-comment?atlOrigin=ey... ) Add Comment ( https://hibernate.atlassian.net/browse/HHH-13922#add-comment?atlOrigin=ey... )
Get Jira notifications on your phone! Download the Jira Cloud app for Android ( https://play.google.com/store/apps/details?id=com.atlassian.android.jira.... ) or iOS ( https://itunes.apple.com/app/apple-store/id1006972087?pt=696495&ct=EmailN... ) This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100123- sha1:26275c0 )
4 years, 8 months
[JIRA] (HSEARCH-3878) Maximize utilization of database connections during mass indexing
by Yoann Rodière (JIRA)
Yoann Rodière ( https://hibernate.atlassian.net/secure/ViewProfile.jspa?accountId=557058%... ) *created* an issue
Hibernate Search ( https://hibernate.atlassian.net/browse/HSEARCH?atlOrigin=eyJpIjoiMmJjZGFm... ) / Improvement ( https://hibernate.atlassian.net/browse/HSEARCH-3878?atlOrigin=eyJpIjoiMmJ... ) HSEARCH-3878 ( https://hibernate.atlassian.net/browse/HSEARCH-3878?atlOrigin=eyJpIjoiMmJ... ) Maximize utilization of database connections during mass indexing ( https://hibernate.atlassian.net/browse/HSEARCH-3878?atlOrigin=eyJpIjoiMmJ... )
Issue Type: Improvement Assignee: Unassigned Attachments: queueCount10_bulkSize250.svg Components: mapper-orm Created: 01/Apr/2020 08:19 AM Fix Versions: 6.0.0.Beta-backlog-low-priority Priority: Major Reporter: Yoann Rodière ( https://hibernate.atlassian.net/secure/ViewProfile.jspa?accountId=557058%... )
The mass indexer threads responsible for loading entities from the database currently have a loop that looks like this:
* load X entities from the database
* submit indexing requests for each entity
* wait for indexing to finish for these entities
* repeat
The "wait" step means that, while the backend is busy indexing, the thread will not load anything from the database. This effectively means that it will hold a reference to a database connection for nothing during that time.
This means that indexing doesn't execute in parallel of database loading, and as a result the execution time of mass indexing is probably close to the *sum* of the time spent loading entities and the time spent indexing. Ideally, we'd want those two operations to happen in parallel, so that the execution time of mass indexing is close to the *maximum* instead of the sum.
You can see the effect quite clearly in queueCount10_bulkSize250.svg ( https://hibernate.atlassian.net/secure/attachment/49646/49646_queueCount1... ) , which is a gantt chart of the indexing tasks in an Elasticsearch backend during mass indexing. From time to time, almost all executors are idle, because entities have not been submitted yet... If entity loading happened in parallel, the indexing executes would be less likely to stay idle.
One solution to this problem would be to move to a loop like this:
* load X entities from the database (load #1)
* submit indexing requests for each entity
* load X entities from the database (load #2)
* submit indexing requests for each entity
* wait for indexing to finish for load #1
* load X entities from the database (load #3)
* submit indexing requests for each entity
* wait for indexing to finish for load #2
* load X entities from the database (load #4)
* submit indexing requests for each entity
* wait for indexing to finish for load #3
* ...
This should greatly reduce the amount of waiting in the loading thread, since we will be loading the next batch of entities while the previous batch is being indexed. And this preserves the characteristics of the previous algorithm: if database loading is much faster than indexing (who knows...), we won't flood indexing queues with indexing requests, and will only ever have at most two batches of entities pending at any given time.
This would probably be rather easy to implement, since the completion of indexing is modelled by CompletableFutures : we just have to store the future of the last batch somewhere, and wait for it to finish.
( https://hibernate.atlassian.net/browse/HSEARCH-3878#add-comment?atlOrigin... ) Add Comment ( https://hibernate.atlassian.net/browse/HSEARCH-3878#add-comment?atlOrigin... )
Get Jira notifications on your phone! Download the Jira Cloud app for Android ( https://play.google.com/store/apps/details?id=com.atlassian.android.jira.... ) or iOS ( https://itunes.apple.com/app/apple-store/id1006972087?pt=696495&ct=EmailN... ) This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100123- sha1:26275c0 )
4 years, 8 months