[JIRA] (HSEARCH-4700) Configurable behavior on full queues for indexing operations
by Yoann Rodière (JIRA)
Yoann Rodière ( https://hibernate.atlassian.net/secure/ViewProfile.jspa?accountId=557058%... ) *created* an issue
Hibernate Search ( https://hibernate.atlassian.net/browse/HSEARCH?atlOrigin=eyJpIjoiM2JmMGY3... ) / Improvement ( https://hibernate.atlassian.net/browse/HSEARCH-4700?atlOrigin=eyJpIjoiM2J... ) HSEARCH-4700 ( https://hibernate.atlassian.net/browse/HSEARCH-4700?atlOrigin=eyJpIjoiM2J... ) Configurable behavior on full queues for indexing operations ( https://hibernate.atlassian.net/browse/HSEARCH-4700?atlOrigin=eyJpIjoiM2J... )
Issue Type: Improvement Assignee: Unassigned Components: backend-elasticsearch, backend-lucene, engine, mapper-pojo-base Created: 09/Sep/2022 08:36 AM Fix Versions: 6.2-backlog Priority: Major Reporter: Yoann Rodière ( https://hibernate.atlassian.net/secure/ViewProfile.jspa?accountId=557058%... )
See https://infinispan.zulipchat.com/#narrow/stream/118645-infinispan/topic/J... ( https://infinispan.zulipchat.com/#narrow/stream/118645-infinispan/topic/J... )
Essentially, users of methods such as:
* org.hibernate.search.mapper.pojo.work.spi.PojoIndexingPlan#executeAndReport
* org.hibernate.search.mapper.pojo.work.spi.PojoIndexer#add
* org.hibernate.search.mapper.pojo.work.spi.PojoIndexer#addOrUpdate
* …
* org.hibernate.search.mapper.pojo.work.spi.PojoIndexer#delete(org.hibernate.search.mapper.pojo.model.spi.PojoRawTypeIdentifier<?>, java.lang.Object, org.hibernate.search.mapper.pojo.route.DocumentRoutesDescriptor, org.hibernate.search.engine.backend.work.execution.DocumentCommitStrategy, org.hibernate.search.engine.backend.work.execution.DocumentRefreshStrategy)
* org.hibernate.search.mapper.pojo.work.spi.PojoScopeWorkspace#mergeSegments
* …
* org.hibernate.search.mapper.pojo.work.spi.PojoScopeWorkspace#refresh
… need to be able to specify the behavior they want to enforce back-pressure:
* Block the thread: that’s the current behavior, and it’s fine when calling these methods from blocking processes such as the mass indexer or org.hibernate.search.mapper.orm.work.impl.SearchWorkspaceImpl#mergeSegments. But it's a big no-no when calling from non-blocking processes such as Infinispan.
* Fail: that’s the preferred behavior when calling these methods from non-blocking processes, e.g. in Infinispan, or in org.hibernate.search.mapper.orm.work.impl.SearchWorkspaceImpl#mergeSegmentsAsync.
I think we should add a parameter to those methods to control that behavior, something like a OperationSubmitter. We would propagate that parameter all the way through to the AbstractWorkOrchestrator , which would take it into account as appropriate: submitter.submitToQueue(queue, element) , submitter.submitToExecutor(executor, task) , …
We would provide two built-in implementations: BLOCK and REJECTED_EXECUTION_EXCEPTION.
We would *not* allow custom implementations (at least for now); allowing things such as offloading to another thread, for example, could be very dangerous as submitting an operation sometimes needs to happen in a certain context (with certain locks held by the current thread). Also, we may need to add more methods to the OperationSubmitter interface and that would be a breaking change.
We would *not* add a OperationSubmitter argument to methods that are clearly blocking.
*Maybe we should also introduce a configuration property* (also SPI) so that Infinispan can set the default globally.
Finally, maybe we could do the same for schema management operations:
* org.hibernate.search.mapper.pojo.schema.management.spi.PojoScopeSchemaManager#createIfMissing
* …
* org.hibernate.search.mapper.pojo.schema.management.spi.PojoScopeSchemaManager#validate
( https://hibernate.atlassian.net/browse/HSEARCH-4700#add-comment?atlOrigin... ) Add Comment ( https://hibernate.atlassian.net/browse/HSEARCH-4700#add-comment?atlOrigin... )
Get Jira notifications on your phone! Download the Jira Cloud app for Android ( https://play.google.com/store/apps/details?id=com.atlassian.android.jira.... ) or iOS ( https://itunes.apple.com/app/apple-store/id1006972087?pt=696495&ct=EmailN... ) This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100206- sha1:d8a7a40 )
1 year, 7 months