[JBoss JIRA] (ISPN-7842) Declarative indexed entity mapping
by Tristan Tarrant (JIRA)
Tristan Tarrant created ISPN-7842:
-------------------------------------
Summary: Declarative indexed entity mapping
Key: ISPN-7842
URL: https://issues.jboss.org/browse/ISPN-7842
Project: Infinispan
Issue Type: Enhancement
Components: Configuration, Embedded Querying
Reporter: Tristan Tarrant
Assignee: Jakub Senko
Priority: Minor
We need a way to list annotation-less indexed entities in the infinispan XML.
The `<indexed-entities>` element schema will need to be extended as follows:
`<indexing index="ALL">
<indexed-entities>
<!-- annotated entity -->
<indexed-entity>org.infinispan.query.queries.faceting.Car</indexed-entity>
<!-- non-annotated entity -->
<indexed-entity-mapping>
<!-- the FQN of the class to index -->
<class>my.domain.model.Author</class>
<!-- optional -->
<spatial name="place" mode="HASH"/>
<!-- list of indexed properties -->
<property name="name" type="method">
<field store="true" index="true" analyze="true" norms="true" term-vector="yes" boost="0.5"/>
</property>
<property name="title" type="method">
<field store="true" index="true" analyze="true" norms="true" term-vector="yes" boost="0.5" analyzer="titleanalyzer"/>
</property>
<property type="method" name="birthdate">
<field store="true" index="true" analyze="true" norms="true" term-vector="yes" boost="0.5"/>
<date-bridge resolution="DAY"/>
</property>
<property type="method" name="city">
<spatial name="name" store="true" boost="0.5" spatial-mode="RANGE" />
</property>
</indexed-entity-mapping>
</indexed-entities>
<!-- Use our custom IndexManager; TODO autoinject by default? -->
<property name="hibernate.search.default.indexmanager">org.infinispan.query.indexmanager.InfinispanIndexManager</property>
<property name="hibernate.search.indexing_strategy">manual</property>
<!-- Use latest Lucene version -->
<property name="lucene_version">LUCENE_CURRENT</property>
</indexing>`
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
8 years, 10 months
[JBoss JIRA] (ISPN-7837) ClusteredQuery throws NPE if any local index is empty
by Gustavo Fernandes (JIRA)
[ https://issues.jboss.org/browse/ISPN-7837?page=com.atlassian.jira.plugin.... ]
Gustavo Fernandes updated ISPN-7837:
------------------------------------
Status: Pull Request Sent (was: Open)
Git Pull Request: https://github.com/infinispan/infinispan/pull/5135
> ClusteredQuery throws NPE if any local index is empty
> -----------------------------------------------------
>
> Key: ISPN-7837
> URL: https://issues.jboss.org/browse/ISPN-7837
> Project: Infinispan
> Issue Type: Bug
> Components: Embedded Querying
> Affects Versions: 9.0.0.Final
> Reporter: Gustavo Fernandes
> Assignee: Gustavo Fernandes
>
> The ClusteredQuery API expects that all local indexes are populated with documents. With the setting Index.LOCAL, it's possible to index all documents in a single node (by only executing puts in this node), leaving all the other nodes empty. Exception thrown:
> {noformat}
> Caused by: java.lang.NullPointerException
> at org.infinispan.query.clustered.commandworkers.CQCreateEagerQuery.collectKeys(CQCreateEagerQuery.java:37)
> at org.infinispan.query.clustered.commandworkers.CQCreateEagerQuery.perform(CQCreateEagerQuery.java:25)
> at org.infinispan.query.clustered.ClusteredQueryCommand.perform(ClusteredQueryCommand.java:114)
> at org.infinispan.query.clustered.ClusteredQueryCommand.invokeAsync(ClusteredQueryCommand.java:109)
> at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.invokeCommand(BasePerCacheInboundInvocationHandler.java:101)
> at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.invoke(BaseBlockingRunnable.java:99)
> at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.runAsync(BaseBlockingRunnable.java:71)
> at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.run(BaseBlockingRunnable.java:40)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
8 years, 10 months
[JBoss JIRA] (ISPN-3795) QueryInterceptor incorrectly relies on the return value of a RemoveCommand
by Gustavo Fernandes (JIRA)
[ https://issues.jboss.org/browse/ISPN-3795?page=com.atlassian.jira.plugin.... ]
Gustavo Fernandes commented on ISPN-3795:
-----------------------------------------
[~sannegrinovero] Could it be that when IGNORE_RETURN_VALUE was introduced, the overhead of indexing was orders of magnitude higher than today, given that previously it was doing 1 commit per entry? After that we add transparent batching to Hibernate Search SYNC backend, plus the async indexing backend, I reckon performance is vastly superior to do batch loading nowadays.
We could allso introduce a way to temporarily enable async indexing to do batch loading, which will always beat the sync option.
> QueryInterceptor incorrectly relies on the return value of a RemoveCommand
> --------------------------------------------------------------------------
>
> Key: ISPN-3795
> URL: https://issues.jboss.org/browse/ISPN-3795
> Project: Infinispan
> Issue Type: Bug
> Components: Embedded Querying
> Affects Versions: 6.0.0.Final
> Reporter: Dan Berindei
> Assignee: Gustavo Fernandes
>
> QueryInterceptor uses the return value from RemoveCommand/ReplaceCommand to remove the value from the index.
> But both RemoveCommand and ReplaceCommand have a variant with an expected value parameter, and this variant return a boolean value instead of the removed/replaced value. In that case, the previous value won't be removed from the index.
> QueryInterceptor should probably use the previous value from the context entries to update the index instead.
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
8 years, 10 months
[JBoss JIRA] (ISPN-7841) Add stream operations that can operate upon data exclusively
by William Burns (JIRA)
William Burns created ISPN-7841:
-----------------------------------
Summary: Add stream operations that can operate upon data exclusively
Key: ISPN-7841
URL: https://issues.jboss.org/browse/ISPN-7841
Project: Infinispan
Issue Type: Enhancement
Components: Core, Distributed Execution and Map/Reduce
Reporter: William Burns
Assignee: William Burns
Fix For: 9.1.0.Final
It would be nice to have a way to perform a stream operation where the data performed upon is done in an exclusive way. That is that the data cannot change while it is being processed upon. This is only at the level of per key. For example non tx cache could temporarily lock the entry to prevent a write. The same could occur with pessimistic transactions. Optimistic transactions are a bit special and in this case we could just allow for some retry logic to allow for the stream to reread a given key and try the stream operation again if a WriteSkewException was thrown.
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
8 years, 10 months
[JBoss JIRA] (ISPN-3795) QueryInterceptor incorrectly relies on the return value of a RemoveCommand
by Sanne Grinovero (JIRA)
[ https://issues.jboss.org/browse/ISPN-3795?page=com.atlassian.jira.plugin.... ]
Sanne Grinovero edited comment on ISPN-3795 at 5/15/17 1:24 PM:
----------------------------------------------------------------
{quote}Sanne Grinovero just curious, how bad would it be if tracked the types in the cache and every update would purge the key from all the indexes? {quote}
First off to be correct we really need to know which type is being deleted; enumerating all the types isn't enough.
>From a performance perspective, on the Lucene index a single delete (purge) operation is about one order of magnitude slower than a write, so we actually try to work hard to avoid issuing any unnecessary delete operations. Also a delete operation needs strict ordering towards other operations so there are costs in parallel scalability as well.
So issuing an unnecessary delete on all indexes is not adviseable but your suggestion is valid if we could know which index needs the one and only delete operation deterministically.
I guess if Infinispan could fetch the type rather than the actual whole value we could avoid some payload transfers, but it's still something that we'd need to be able to disable as e.g. dynamic sharding allows index selection based on some attribute of the actual value.
Given the other comments, +1 to allow Query to *ignore* _IGNORE_RETURN_VALUE_ *however* the performance of batch loading an empty grid would collapse. Please still allow some way to get that loaded fast..
was (Author: sannegrinovero):
{quote}Sanne Grinovero just curious, how bad would it be if tracked the types in the cache and every update would purge the key from all the indexes? {quote}
First off to be correct we really need to know which type is being deleted; enumerating all the types isn't enough.
>From a performance perspective, on the Lucene index a single delete (purge) operation is about one order of magnitude slower than a write, so we actually try to work hard to avoid issuing any unnecessary delete operations. Also a delete operation needs strict ordering towards other operations so there are costs in parallel scalability as well.
So issuing an unnecessary delete on all indexes is not adviseable but your suggestion is valid if we could know index needs it deterministically. I guess if Infinispan could fetch the type rather than the actual whole value we could avoid some payload transfers, but it's still something that we'd need to be able to disable as e.g. dynamic sharding allows index selection based on some attribute of the actual value.
Given the other comments, +1 to allow Query to *ignore* _IGNORE_RETURN_VALUE_ *however* the performance of batch loading an empty grid would collapse. Please still allow some way to get that loaded fast..
> QueryInterceptor incorrectly relies on the return value of a RemoveCommand
> --------------------------------------------------------------------------
>
> Key: ISPN-3795
> URL: https://issues.jboss.org/browse/ISPN-3795
> Project: Infinispan
> Issue Type: Bug
> Components: Embedded Querying
> Affects Versions: 6.0.0.Final
> Reporter: Dan Berindei
> Assignee: Gustavo Fernandes
>
> QueryInterceptor uses the return value from RemoveCommand/ReplaceCommand to remove the value from the index.
> But both RemoveCommand and ReplaceCommand have a variant with an expected value parameter, and this variant return a boolean value instead of the removed/replaced value. In that case, the previous value won't be removed from the index.
> QueryInterceptor should probably use the previous value from the context entries to update the index instead.
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
8 years, 10 months
[JBoss JIRA] (ISPN-3795) QueryInterceptor incorrectly relies on the return value of a RemoveCommand
by Sanne Grinovero (JIRA)
[ https://issues.jboss.org/browse/ISPN-3795?page=com.atlassian.jira.plugin.... ]
Sanne Grinovero edited comment on ISPN-3795 at 5/15/17 1:23 PM:
----------------------------------------------------------------
{quote}Sanne Grinovero just curious, how bad would it be if tracked the types in the cache and every update would purge the key from all the indexes? {quote}
First off to be correct we really need to know which type is being deleted; enumerating all the types isn't enough.
>From a performance perspective, on the Lucene index a single delete (purge) operation is about one order of magnitude slower than a write, so we actually try to work hard to avoid issuing any unnecessary delete operations. Also a delete operation needs strict ordering towards other operations so there are costs in parallel scalability as well.
So issuing an unnecessary delete on all indexes is not adviseable but your suggestion is valid if we could know index needs it deterministically. I guess if Infinispan could fetch the type rather than the actual whole value we could avoid some payload transfers, but it's still something that we'd need to be able to disable as e.g. dynamic sharding allows index selection based on some attribute of the actual value.
Given the other comments, +1 to allow Query to *ignore* _IGNORE_RETURN_VALUE_ *however* the performance of batch loading an empty grid would collapse. Please still allow some way to get that loaded fast..
was (Author: sannegrinovero):
{quote}Sanne Grinovero just curious, how bad would it be if tracked the types in the cache and every update would purge the key from all the indexes? {quote}
First off to be correct we really need to know which type is being deleted; enumerating all the types isn't enough.
>From a performance perspective, on the Lucene index a single delete (purge) operation is about one order of magnitude slower than a write, so we actually try to work hard to avoid issuing any unnecessary delete operations. Also a delete operation needs strict ordering towards other operations so there are costs in parallel scalability as well.
So issuing an unnecessary delete on all indexes is not adviseable but your suggestion is valid if we could know index needs it deterministically. I guess if Infinispan could fetch the type rather than the actual whole value we could avoid some payload transfers, but it's still something that we'd need to be able to disable as e.g. dynamic sharding allows index selection based on some attribute of the actual value.
Given the other comments, +1 to allow Query to *ignore*_ IGNORE_RETURN_VALUE_ *however* the performance of batch loading an empty grid would collapse. Please still allow some way to get that loaded fast..
> QueryInterceptor incorrectly relies on the return value of a RemoveCommand
> --------------------------------------------------------------------------
>
> Key: ISPN-3795
> URL: https://issues.jboss.org/browse/ISPN-3795
> Project: Infinispan
> Issue Type: Bug
> Components: Embedded Querying
> Affects Versions: 6.0.0.Final
> Reporter: Dan Berindei
> Assignee: Gustavo Fernandes
>
> QueryInterceptor uses the return value from RemoveCommand/ReplaceCommand to remove the value from the index.
> But both RemoveCommand and ReplaceCommand have a variant with an expected value parameter, and this variant return a boolean value instead of the removed/replaced value. In that case, the previous value won't be removed from the index.
> QueryInterceptor should probably use the previous value from the context entries to update the index instead.
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
8 years, 10 months
[JBoss JIRA] (ISPN-3795) QueryInterceptor incorrectly relies on the return value of a RemoveCommand
by Sanne Grinovero (JIRA)
[ https://issues.jboss.org/browse/ISPN-3795?page=com.atlassian.jira.plugin.... ]
Sanne Grinovero commented on ISPN-3795:
---------------------------------------
{quote}Sanne Grinovero just curious, how bad would it be if tracked the types in the cache and every update would purge the key from all the indexes? {quote}
First off to be correct we really need to know which type is being deleted; enumerating all the types isn't enough.
>From a performance perspective, on the Lucene index a single delete (purge) operation is about one order of magnitude slower than a write, so we actually try to work hard to avoid issuing any unnecessary delete operations. Also a delete operation needs strict ordering towards other operations so there are costs in parallel scalability as well.
So issuing an unnecessary delete on all indexes is not adviseable but your suggestion is valid if we could know index needs it deterministically. I guess if Infinispan could fetch the type rather than the actual whole value we could avoid some payload transfers, but it's still something that we'd need to be able to disable as e.g. dynamic sharding allows index selection based on some attribute of the actual value.
Given the other comments, +1 to allow Query to *ignore*_ IGNORE_RETURN_VALUE_ *however* the performance of batch loading an empty grid would collapse. Please still allow some way to get that loaded fast..
> QueryInterceptor incorrectly relies on the return value of a RemoveCommand
> --------------------------------------------------------------------------
>
> Key: ISPN-3795
> URL: https://issues.jboss.org/browse/ISPN-3795
> Project: Infinispan
> Issue Type: Bug
> Components: Embedded Querying
> Affects Versions: 6.0.0.Final
> Reporter: Dan Berindei
> Assignee: Gustavo Fernandes
>
> QueryInterceptor uses the return value from RemoveCommand/ReplaceCommand to remove the value from the index.
> But both RemoveCommand and ReplaceCommand have a variant with an expected value parameter, and this variant return a boolean value instead of the removed/replaced value. In that case, the previous value won't be removed from the index.
> QueryInterceptor should probably use the previous value from the context entries to update the index instead.
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
8 years, 10 months
[JBoss JIRA] (ISPN-5218) Add batching to the AdvancedCacheWriter interface
by Tristan Tarrant (JIRA)
[ https://issues.jboss.org/browse/ISPN-5218?page=com.atlassian.jira.plugin.... ]
Tristan Tarrant updated ISPN-5218:
----------------------------------
Description:
The AdvancedCacheWriter should be extended with a write operation which takes multiple entries so that stores which can do batching can optimize this. This would benefit:
* passivateAll
* putAll
* the async writer
Immediate users would be the JDBC stores, JPA, RocksDB.
was:
The AdvancedCacheWriter should be extended with a write operation which takes multiple entries so that stores which can do batching can optimize this. This would benefit both passivation and the async writer.
Immediate users would be the JDBC stores.
> Add batching to the AdvancedCacheWriter interface
> -------------------------------------------------
>
> Key: ISPN-5218
> URL: https://issues.jboss.org/browse/ISPN-5218
> Project: Infinispan
> Issue Type: Enhancement
> Components: Loaders and Stores
> Reporter: Tristan Tarrant
> Assignee: Ryan Emerson
> Fix For: 9.1.0.Final
>
>
> The AdvancedCacheWriter should be extended with a write operation which takes multiple entries so that stores which can do batching can optimize this. This would benefit:
> * passivateAll
> * putAll
> * the async writer
> Immediate users would be the JDBC stores, JPA, RocksDB.
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
8 years, 10 months