Hi,
On Thu, Jul 02, 2015 at 12:20:46PM +0100, Sanne Grinovero wrote:
Ideally we should provide something similar to the Dynamic Analyzer
feature but which also multiplexes an entity property into multiple
fieldnames;
for example
property "title"
-> title_en & analyzer en
-> title_de & analyzer de
The selection would work based on the Discriminator field, much like
the current Dynamic Analyzer.
That might be a possibility, even though I am not quite sure how exactly this would
look like. I would first need to dig in more into the existing code.
Do you have a more concrete idea on how this would look like?
Still, even if we were to find the bandwidth to make that, we'd
need a
deprecation path for the existing feature
Well, in the above case we are not talking about deprecation right? It would be more of
a change in behavior and use!?
> What's about the alternative to close the IndexWriter and
re-open it? Obviously this could be
> optimised, but storing the field to analyzer map together with the open IndexWriter
and only
> re-open if the mapping changes. As long as the mapping is the same the same
IndexWriter can be used.
> This way we could keep the feature with a potential performance hit for the people
who are using it.
> Still better than removing it, right? That said, what are the exact performance
impacts? Did you run
> a test?
The performace impact is huge as it would prevent you from using both
NRT and the new backend strategy to pack multiple blocks in commit
cycles;
that means the impact is in the 3 to 4 orders of magnitude in throughput.
Might be still worth testing and prototyping.
I could apply your suggestion in practice if we go for setting a
flag
in the backend to change strategy, depending if any entity is using
the Discriminator feature,
That would work for me.
but beyond that we also have the problem of
different entities sharing the same index but potentially using a
different analyzer for the same named field... I'd agree with the
Lucene developers that people should really not do it, but we support
that today.
Ok. In this case I am more inclined to enforce the same analyzer.
> How would that look like and did we not once discuss exactly the
opposite (aka letting even
> the Document be built on the master)?
We discussed to not create a Document instance on the slave, to only
serialize a custom serializable-friendly container, but that doesn't
prevent you to pre-tokenize the text on the slaves.
AFAIR we discussed to create a "master node" which doesn't need the
user classes so that would be an easy to start service w/o need for
much more than some configuration properties.. if you don't
pre-tokenize this configuration would still need the classes to read
our analyzer definitions from annotations.
Ok, that is possible as well. I think we discussed both and I was indeed
referring to the approach where the master node would do the index using
user classes and the corresponding Search metadata.
I like the solution you are referring to much better, since it also works
better with the ideas I have regarding the clustering of the index (eg with
RAFT). As you suggest, it would be beneficial if only the slaves would need
to know about the user classes.
>> master/slave clustering approach, that would have several
other
>> benefits:
>> - move the analyzer work to the slaves
>
> Why is that a benefit?
- removes the need to have the analyzer definitions on the master (see above).
Ok, in the light of the above discussed solution, you would not need the analyzers
on the master node. Not sure whether this such an important thing so.
- spreads out the CPU and memory allocations cost to each slave
node:
better scalability than have it all done on the master
Well, one could also take the point of view that the slaves should do as little as
possible
and let the master do the heavy lifting. It depends really for what you are optimizing
imo.
>> - reduce the network payloads
>
> Really, is it actually not increasing payloads?
I would expect so: a pre-filtered token sequence is usually smaller
than the source text, often by a good margin.
True, in the usual cases that is probably the case.
--Hardy