Hibernate Search connecting to multiple Elasticsearch clusters
by Sanne Grinovero
Today our experiments are assuming we're connecting to a single
Elasticsearch cluster: one hostname to configure, etc..
I think this is an acceptable limitation for version 5.6 (our first
stable milestone to support this integration) but I'm wondering if we
should document it as a temporary limitation or as an intentional
design.
I think that eventually we should allow having different entities
(indexes) to be stored on different ES clusters; this shouldn't be too
hard to manage as the codebase is already structured around this
capability of having different types in different "indexes"; so while
a single Elasticsearch cluster can manage multiple indexes (that's a
bit of a novel concept) I see no reason to not allow different indexes
to be mapped on different clusters.
= Am I missing some strong reason to not allow this?
= While this will (likely) not be supported in 5.6, should the public
API and configuration properties allow for this to be configured
per-index already? (Changing it later would be a breaking change)
See also:
- https://hibernate.atlassian.net/browse/HSEARCH-2164
Thanks,
Sanne
10 years, 1 month
Elastic search Lucene extension
by Marc Schipperheyn
I saw the elastic search blog post and while is not yet sonething on our
radar, it s certainly interesting!
I already saw that if you bypass HSearch to go native, the syntax changes.
What about Lucene extensions? For instance, we extensively use joinutil and
grouped search. Couldn't live without those.
--
Kind regards,
Marc
M.Schipperheyn
10 years, 1 month
Returned mail: Data format error
by MAILER-DAEMON
The original message was received at Sun, 6 Mar 2016 17:08:59 +0800
from lists.jboss.org [91.120.79.33]
----- The following addresses had permanent fatal errors -----
<hibernate-dev(a)lists.jboss.org>
10 years, 2 months
Changing or deprecating @AnalyzerDef
by Sanne Grinovero
I just realized that the definiton for @AnalyzerDef allows many target ypes:
@Target({ ElementType.PACKAGE, ElementType.TYPE, ElementType.FIELD,
ElementType.METHOD })
I think FIELD and METHOD might not have been intended to be there.
Should we correct this in 6 to
@Target({ ElementType.PACKAGE, ElementType.TYPE })
?
If so, since I don't see how to deprecate that I think we could start
logging a warning now when we find one "misplaced".
Alternatively we might want to fully deprecate this annotation, as it
requires direct exposure to Lucene classes.
I'm thinking we should at least change it to refer to FQCN of lucene
classes rather than class instances, but maybe someone has even better
ideas? (I don't like the typesafety of FQCN String names)
Thanks,
Sanne
10 years, 2 months
Connecting to a resource which isn't ready yet (or not ready anymore)
by Sanne Grinovero
My question here was triggered by a specific case in Hibernate Search
but it applies well to ORM's datasources, caches, and very much to OGM
as well.
When creating an index on Elasticsearch, the index is not
"instantaneously" ready.
The REST request creates the definition, but the response will only
tell if the request to create it was accepted. Elasticsearch will then
start the creation process and gradually upgrades the index status to
"yellow" and finally "green" depending on its ability to quickly
propagate the needed changes across the clusters; it might even fail
and get to a status "Red".
Our current approach is:
- send a request to define the index
- start using it
Which is probably following our traditional pattern with static
systems, but becomes a naive approach in this modern world of dynamic
services.
Our approach works *almost* fine in the singleton nodes which we're
using for testing but it's not suited for a real cluster.
Even in our integration tests, it might happen that we didn't give it
enough time to boot.
Someone else asked me recently how to make Hibernate ORM not fail to
boot when he's starting VMs containing the database and the
application in parallel or in no specific order: sometimes it would
happen that ORM would attempt to connect before the RDBMs would have
started; all he needed was to have ORM stall and wait some seconds.
Often even starting the RDBMs VM first isn't good enough, as the VM
reports "started" but the DB might be needing to finish some
maintenance tasks.. Kubernetes provides hooks to check for actual
services to being ready but people seem to expect that Hibernate could
deal with some basics too.
In that case AFAIR my suggestion was that this could be solved as an
implementation detail of the datasource / connection pool.
Back to my Elasticsearch problem: I think the short term solution
would be that we actually have to check for the index state after
having it created, and keep checking in a loop until some short
timeout expires.
-> https://hibernate.atlassian.net/browse/HSEARCH-2146
But I don't think that's the right design to pursue in the longer
term; especially as we're not dealing with the fact that the index
might "downgrade" its state at any time after we started.
I think we need to check for "cluster health" headers regularly and
monitor for acceptance for each command we send, keeping track of its
health and probably keeping a persistent queue around for the
operations which couldn't be applied yet.
Our current design will throw a SearchException to the end user, which
I don't think is practical..
History might have shown that the current approach is fine with
Hibernate ORM, but:
- requirements and expectations evolve, people might soon expect more
- I suspect that with RDBMs's this is less of a need than with the
new crop of dynamic, self healing distributed systems we're dealing
with.
Not least, I noticed that the Elasticsearch native client actually
enters the cluster as a member of it. That's similar to an Infinispan
client using zero-weight vs a REST client.. Infinispan experts will
understand there are significant different capabilities. I don't mean
to remove the JEST client approach but we might want to study more of
the consequences of that, and help me define what we will consider an
acceptable tradeoff while still using the REST client approach.
Thanks for any thoughts,
Sanne
10 years, 2 months
[Search] Travis support
by Guillaume Smet
Hi,
As mentioned in our last discussion, I explored adding Travis support to
Search.
The diff is here:
https://github.com/gsmet/hibernate-search/commit/cbd2c1fff05532974823da57...
(yes it's short but it was a long road :))
I had to raise a bit the JGroups timeout for one test and had to fix
TestRunnerStandalone
so that it effectively uses the configuration from the profile (this change
is not specific to Travis and should be committed anyway).
The result is here:
https://travis-ci.org/gsmet/hibernate-search
What I like in Travis:
- The CI configuration is code and is versioned
- Parallelization with a *lot* of resources
- The ability to build a test matrix very easily
- Complete isolation as each build is run in its own Docker container
What is less cool:
- The only way to get the logs is to ship them to AWS S3 - that said, I did
it and it's pretty straightforward as it's well integrated (I commented out
the code in the .travis.yml for now)
- It might seem less flexible as we are depending on the Travis
infrastructure (for instance, I created a ticket to ask for the support of
JDK 9: https://github.com/travis-ci/travis-ci/issues/5520) but in fact, you
can do whatever you want: see https://github.com/Blazebit/blaze-persistence
.travis.yml file for a comprehensive example
So if we move to Travis, I think the regular builds could run on Travis and
we could keep Jenkins for specific ones (deploy/release).
I'll take a look at OGM tomorrow. Travis supports out of the box most of
the NoSQL databases (https://docs.travis-ci.com/user/database-setup/) so
I'm pretty curious to see how it goes.
Thoughts?
--
Guillaume
10 years, 2 months
Fwd: Build failed in Jenkins: hibernate-orm-master-h2-infinispan8.2 #83
by Steve Ebersole
Anyone plan on looking at these frequent transient failures anytime soon?
If not, I plan on disabling this from running automatically. It's just
white noise at this point.
---------- Forwarded message ---------
From: Hibernate CI <ci(a)hibernate.org>
Date: Wed, Mar 2, 2016 at 8:22 AM
Subject: Build failed in Jenkins: hibernate-orm-master-h2-infinispan8.2 #83
To: <rvansa(a)redhat.com>, <smarlow(a)redhat.com>, <gail(a)hibernate.org>, <
galder(a)infinispan.org>, <steve(a)hibernate.org>
See <
http://ci.hibernate.org/job/hibernate-orm-master-h2-infinispan8.2/83/changes
>
Changes:
[mih_vlad] Migrate BasicType and UserType User Guide examples to unit tests
[mih_vlad] DESIGN-737 - Asciidoctor styling for Hibernate outputs.
[mih_vlad] Fix User Guide images that were missing after moving CSS files
into css
------------------------------------------
[...truncated 601 lines...]
bool TraceLoaderConstraints = false
{product rw}
bool TraceMetadataHumongousAllocation = false
{product}
bool TraceMonitorInflation = false
{product}
bool TraceParallelOldGCTasks = false
{product}
intx TraceRedefineClasses = 0
{product}
bool TraceSafepointCleanupTime = false
{product}
bool TraceSharedLookupCache = false
{product}
bool TraceSuspendWaitFailures = false
{product}
intx TrackedInitializationLimit = 50
{C2 product}
bool TransmitErrorReport = false
{product}
bool TrapBasedNullChecks = false
{pd product}
bool TrapBasedRangeChecks = false
{C2 pd product}
intx TypeProfileArgsLimit = 2
{product}
uintx TypeProfileLevel = 111
{pd product}
intx TypeProfileMajorReceiverPercent = 90
{C2 product}
intx TypeProfileParmsLimit = 2
{product}
intx TypeProfileWidth = 2
{product}
intx UnguardOnExecutionViolation = 0
{product}
bool UnlinkSymbolsALot = false
{product}
bool Use486InstrsOnly = false
{ARCH product}
bool UseAES = true
{product}
bool UseAESIntrinsics = true
{product}
intx UseAVX = 1
{ARCH product}
bool UseAdaptiveGCBoundary = false
{product}
bool UseAdaptiveGenerationSizePolicyAtMajorCollection = true
{product}
bool UseAdaptiveGenerationSizePolicyAtMinorCollection = true
{product}
bool UseAdaptiveNUMAChunkSizing = true
{product}
bool UseAdaptiveSizeDecayMajorGCCost = true
{product}
bool UseAdaptiveSizePolicy = true
{product}
bool UseAdaptiveSizePolicyFootprintGoal = true
{product}
bool UseAdaptiveSizePolicyWithSystemGC = false
{product}
bool UseAddressNop = true
{ARCH product}
bool UseAltSigs = false
{product}
bool UseAutoGCSelectPolicy = false
{product}
bool UseBMI1Instructions = true
{ARCH product}
bool UseBMI2Instructions = false
{ARCH product}
bool UseBiasedLocking = true
{product}
bool UseBimorphicInlining = true
{C2 product}
bool UseBoundThreads = true
{product}
bool UseCLMUL = true
{ARCH product}
bool UseCMSBestFit = true
{product}
bool UseCMSCollectionPassing = true
{product}
bool UseCMSCompactAtFullCollection = true
{product}
bool UseCMSInitiatingOccupancyOnly = false
{product}
bool UseCRC32Intrinsics = true
{product}
bool UseCodeCacheFlushing = true
{product}
bool UseCompiler = true
{product}
bool UseCompilerSafepoints = true
{product}
bool UseCompressedClassPointers := true
{lp64_product}
bool UseCompressedOops := true
{lp64_product}
bool UseConcMarkSweepGC = false
{product}
bool UseCondCardMark = false
{C2 product}
bool UseCountLeadingZerosInstruction = true
{ARCH product}
bool UseCountTrailingZerosInstruction = true
{ARCH product}
bool UseCounterDecay = true
{product}
bool UseDivMod = true
{C2 product}
bool UseDynamicNumberOfGCThreads = false
{product}
bool UseFPUForSpilling = false
{C2 product}
bool UseFastAccessorMethods = false
{product}
bool UseFastEmptyMethods = false
{product}
bool UseFastJNIAccessors = true
{product}
bool UseFastStosb = false
{ARCH product}
bool UseG1GC := true
{product}
bool UseGCLogFileRotation = false
{product}
bool UseGCOverheadLimit = true
{product}
bool UseGCTaskAffinity = false
{product}
bool UseHeavyMonitors = false
{product}
bool UseHugeTLBFS = false
{product}
bool UseInlineCaches = true
{product}
bool UseInterpreter = true
{product}
bool UseJumpTables = true
{C2 product}
bool UseLWPSynchronization = true
{product}
bool UseLargePages = false
{pd product}
bool UseLargePagesInMetaspace = false
{product}
bool UseLargePagesIndividualAllocation = false
{pd product}
bool UseLinuxPosixThreadCPUClocks = true
{product}
bool UseLockedTracing = false
{product}
bool UseLoopCounter = true
{product}
bool UseLoopInvariantCodeMotion = true
{C1 product}
bool UseLoopPredicate = true
{C2 product}
bool UseMathExactIntrinsics = true
{C2 product}
bool UseMaximumCompactionOnSystemGC = true
{product}
bool UseMembar = false
{pd product}
bool UseMultiplyToLenIntrinsic = true
{C2 product}
bool UseNUMA = false
{product}
bool UseNUMAInterleaving = false
{product}
bool UseNewLongLShift = true
{ARCH product}
bool UseOSErrorReporting = false
{pd product}
bool UseOldInlining = true
{C2 product}
bool UseOnStackReplacement = true
{pd product}
bool UseOnlyInlinedBimorphic = true
{C2 product}
bool UseOprofile = false
{product}
bool UseOptoBiasInlining = true
{C2 product}
bool UsePSAdaptiveSurvivorSizePolicy = true
{product}
bool UseParNewGC = false
{product}
bool UseParallelGC = false
{product}
bool UseParallelOldGC = false
{product}
bool UsePerfData = true
{product}
bool UsePopCountInstruction = true
{product}
bool UseRDPCForConstantTableBase = false
{C2 product}
bool UseRTMDeopt = false
{ARCH product}
bool UseRTMLocking = false
{ARCH product}
bool UseSHA = false
{product}
bool UseSHA1Intrinsics = false
{product}
bool UseSHA256Intrinsics = false
{product}
bool UseSHA512Intrinsics = false
{product}
bool UseSHM = false
{product}
intx UseSSE = 4
{product}
bool UseSSE42Intrinsics = true
{product}
bool UseSerialGC = false
{product}
bool UseSharedSpaces = false
{product}
bool UseSignalChaining = true
{product}
bool UseStoreImmI16 = true
{ARCH product}
bool UseStringDeduplication = false
{product}
bool UseSuperWord = true
{C2 product}
bool UseTLAB = true
{pd product}
bool UseThreadPriorities = true
{pd product}
bool UseTransparentHugePages = false
{product}
bool UseTypeProfile = true
{product}
bool UseTypeSpeculation = true
{C2 product}
bool UseUnalignedLoadStores = true
{ARCH product}
bool UseVMInterruptibleIO = false
{product}
bool UseXMMForArrayCopy = true
{product}
bool UseXmmI2D = true
{ARCH product}
bool UseXmmI2F = true
{ARCH product}
bool UseXmmLoadAndClearUpper = true
{ARCH product}
bool UseXmmRegToRegMoveAll = true
{ARCH product}
bool VMThreadHintNoPreempt = false
{product}
intx VMThreadPriority = -1
{product}
intx VMThreadStackSize = 1024
{pd product}
intx ValueMapInitialSize = 11
{C1 product}
intx ValueMapMaxLoopSize = 8
{C1 product}
intx ValueSearchLimit = 1000
{C2 product}
bool VerifyMergedCPBytecodes = true
{product}
bool VerifySharedSpaces = false
{product}
intx WorkAroundNPTLTimedWaitHang = 1
{product}
uintx YoungGenerationSizeIncrement = 20
{product}
uintx YoungGenerationSizeSupplement = 80
{product}
uintx YoungGenerationSizeSupplementDecay = 8
{product}
uintx YoungPLABSize = 4096
{product}
bool ZeroTLAB = false
{product}
intx hashCode = 5
{product}
:buildSrc:clean UP-TO-DATE
:buildSrc:compileJava UP-TO-DATE
:buildSrc:compileGroovy
:buildSrc:processResources UP-TO-DATE
:buildSrc:classes
:buildSrc:jar
:buildSrc:assemble
:buildSrc:compileTestJava UP-TO-DATE
:buildSrc:compileTestGroovy UP-TO-DATE
:buildSrc:processTestResources UP-TO-DATE
:buildSrc:testClasses UP-TO-DATE
:buildSrc:test UP-TO-DATE
:buildSrc:check UP-TO-DATE
:buildSrc:build
JAVA6_HOME setting not specified, some build features will be disabled
JAVA6_HOME setting not specified, some build features will be disabled
JAVA6_HOME setting not specified, some build features will be disabled
JAVA6_HOME setting not specified, some build features will be disabled
JAVA6_HOME setting not specified, some build features will be disabled
JAVA6_HOME setting not specified, some build features will be disabled
JAVA6_HOME setting not specified, some build features will be disabled
JAVA6_HOME setting not specified, some build features will be disabled
JAVA6_HOME setting not specified, some build features will be disabled
JAVA6_HOME setting not specified, some build features will be disabled
JAVA6_HOME setting not specified, some build features will be disabled
JAVA6_HOME setting not specified, some build features will be disabled
JAVA6_HOME setting not specified, some build features will be disabled
JAVA6_HOME setting not specified, some build features will be disabled
JAVA6_HOME setting not specified, some build features will be disabled
Overrriding Infinispan version to test vs Infinispan version: 8.2.0-SNAPSHOT
:documentation:clean
:hibernate-c3p0:clean UP-TO-DATE
:hibernate-core:clean
:hibernate-ehcache:clean UP-TO-DATE
:hibernate-enhance-maven-plugin:clean UP-TO-DATE
:hibernate-entitymanager:clean
:hibernate-envers:clean
:hibernate-gradle-plugin:clean UP-TO-DATE
:hibernate-hikaricp:clean UP-TO-DATE
:hibernate-infinispan:clean UP-TO-DATE
:hibernate-java8:clean UP-TO-DATE
:hibernate-jpamodelgen:clean UP-TO-DATE
:hibernate-osgi:clean UP-TO-DATE
:hibernate-proxool:clean UP-TO-DATE
:hibernate-spatial:clean
:hibernate-testing:clean UP-TO-DATE
:release:clean
:hibernate-core:generateGrammarSource
:hibernate-core:xjc
:hibernate-core:compileJavaNote: Some input files use or override a
deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
Starting AnimalSniffer checks using [java16-1.0.signature] against
[sourceSets.main]
:hibernate-core:processResources
:hibernate-core:classes
:hibernate-core:jar
:hibernate-infinispan:compileJavaNote: Some input files use or override a
deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
:hibernate-infinispan:processResources
:hibernate-infinispan:classes
:hibernate-testing:compileJavaNote: Some input files use or override a
deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
Starting AnimalSniffer checks using [java16-1.0.signature] against
[sourceSets.main]
:hibernate-testing:processResources
:hibernate-testing:classes
:hibernate-testing:jar
:hibernate-infinispan:compileTestJavaNote: Some input files use or override
a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
:hibernate-infinispan:processTestResources
:hibernate-infinispan:testClasses
:hibernate-infinispan:test
org.hibernate.test.cache.infinispan.entity.EntityRegionAccessStrategyTest >
testContestedPutFromLoad[JTA, REPL_SYNC,AccessType[read-write]] FAILED
java.lang.AssertionError at EntityRegionAccessStrategyTest.java:426
org.hibernate.test.cache.infinispan.entity.EntityRegionAccessStrategyTest >
testContestedPutFromLoad[JTA, REPL_SYNC,AccessType[nonstrict-read-write]]
FAILED
java.lang.AssertionError at EntityRegionAccessStrategyTest.java:426
1184 tests completed, 2 failed, 2 skipped
:hibernate-infinispan:test FAILED
:hibernate-infinispan:buildDashboard
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':hibernate-infinispan:test'.
> There were failing tests. See the report at:
file:///mnt/jenkins-workdir/workspace/hibernate-orm-master-h2-infinispan8.2/hibernate-infinispan/target/reports/tests/index.html
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or
--debug option to get more log output.
BUILD FAILED
Total time: 14 mins 30.275 secs
Build step 'Execute shell' marked build as failure
Recording test results
10 years, 2 months
Making APIs more Lambda-friendly
by Gunnar Morling
Hi,
Tinkering with result transformers, I wished I'd be able to express a
tuple transformation like this, using Java 8 Lambda-style:
session.createQuery( "SELECT foo, bar FROM Baz" )
.transformResultTuples( (tuple, aliases) -> { return tuple[0]
+ " " + tuple[1]; } )
.list();
That requires an interface with a single abstract method ("functional
interface") describing the Lambda type. The current ResultTransformer
interface has two methods, so it cannot be used as is.
I thus propose to deprecate ResultTransformer and add two new,
separate contracts: ResultRowTransformer and ResultListTransformer.
Query#setResultTransformer() would be superseded by two new methods:
transformResultTuples() and transformResultList().
Note that this change is fully compatible with previous Java versions:
One still can pass an (anonymous) implementation of the contracts when
stuck to Java 6/7. That backwards-compatible design is a really nice
trait of how Lambdas have been added to the Java language.
Lambda-friendliness is something which I think we should generally
keep in mind when designing new APIs. Maybe there also other ones
existing, that'd benefit from retrofitting.
Any thoughts?
Thanks,
--Gunnar
10 years, 2 months
