The number of environments we have to test is getting wild, and it’s taking a lot of time.

I think we should consider only testing the default environment in the main build: latest LTS JDK, latest Elasticsearch, no Opensearch, a single DB (H2 or maybe postgresql).

We would move testing of other environments to a separate Jenkins job (hibernate-search-matrix? hibernate-search-environments?).

Ideally, we’d take inspiration from ci/dependency-update/Jenkinsfile for that job, because it’s using the declarative syntax for Jenkinsfiles, and believe me that’s way easier to debug.

As to the specifics of that job:

• It would have to cover everything from the “non-default environments” part of the current main Jenkins job.
• We’d forbid concurrent builds, similarly to other jobs, but probably wouldn’t cancel running builds.
• When running against main branches (not PRs), it would have to be rate-limited (throttled), similarly to other ci/dependency-update/Jenkinsfile.
• It would probably need some form of parameters, to be able to run tests against a particular platform easily. I’d argue that using only some regexp condition would be fine. We can definitely simplify what we’re currently doing in the main Jenkins job.
• It would run against branches, but also against PRs. At the very least it should test PRs against a subset of all environments (currently I think we do Eclipse compiler, Oracle, PostgreSQL and that’s it; maybe we’d want to add the latest early-access JDK).
• Ideally we would add some code to detect when changes in a given PR have no impact on a given environment and thus skip some environments. E.g. if there are no changes in hibernate-search-backend-elasticsearch or any of its dependencies (engine, …), then there’s no need to run tests against all flavors of OpenSearch. That would allow us to expand a bit the subset of environments we test against PRs. But maybe that should be a separate Jira issue?