[JBoss JIRA] (WFCORE-2301) Mount point not found exception raised by createTempFileWithAttributes on overlayfs [JDK-8165852]
by Brian Stansberry (JIRA)
[ https://issues.jboss.org/browse/WFCORE-2301?page=com.atlassian.jira.plugi... ]
Brian Stansberry edited comment on WFCORE-2301 at 2/16/17 12:23 PM:
--------------------------------------------------------------------
I think https://github.com/wildfly/wildfly-core/pull/2188 should work around this by dealing with the failure in Files.getFileStore(Path). We call this in four places in WildFly. This fixes two, the ones involved in your report. Lucene calls it as well in some util, but catches exceptions. The fourth is a management operation handler for an operation CLI/console users can use to learn about space available on the FS (PathInfoHandler). I left that as is, since no one is required to invoke that op and if it fails it seems reasonable that the user sees that.
was (Author: brian.stansberry):
I think https://github.com/wildfly/wildfly-core/pull/2188 should work around this by dealing with the failure in Files.getFileStore(Path). We call this in four places places in WildFly. This fixes two, the ones involved in your report. Lucene calls it as well in some util, but catches exceptions. The fourth is a management operation handler for an operation CLI/console users can use to learn about space available on the FS (PathInfoHandler). I left that as is, since no one is required to invoke that op and if it fails it seems reasonable that the user sees that.
> Mount point not found exception raised by createTempFileWithAttributes on overlayfs [JDK-8165852]
> -------------------------------------------------------------------------------------------------
>
> Key: WFCORE-2301
> URL: https://issues.jboss.org/browse/WFCORE-2301
> Project: WildFly Core
> Issue Type: Bug
> Components: Domain Management
> Environment: WildFly via KeyCloak 2.5.1.Final
> {code:xml}
> <eap.version>7.0.0.Beta</eap.version>
> <jboss.as.version>7.2.0.Final</jboss.as.version>
> <wildfly.version>10.0.0.Final</wildfly.version>
> {code}
> on Docker with overlayfs or overlayfs2 as storage driver
> \# docker info | grep -i storage
> aufs: works (e.g., boot2docker, legacy minikube)
> overlay (e.g., CoreOS, current minikube): problem
> devicemapper (e.g., CentOS): works
> overlay2 (e.g., Docker for Mac): problem
> Reporter: Bjoern Stuetz
> Assignee: Brian Stansberry
>
> Mount point not found exception raised by createTempFileWithAttributes on overlayfs [JDK-8165852], i.e.,
> /opt/jboss/bin/jboss-cli.sh --file=/opt/jboss/jboss-config.cli
> inside a Docker container running on overlayfs as storage driver
> causes (full stack trace below):
> {code:java}
> java.io.IOException: Mount point not foundImage
> at sun.nio.fs.LinuxFileStore.findMountEntry(LinuxFileStore.java:91)
> {code}
> triggered by
> {code:java}
> at org.jboss.as.controller.persistence.FilePersistenceUtils.createTempFileWithAttributes(FilePersistenceUtils.java:117)
> at org.jboss.as.controller.persistence.FilePersistenceUtils.writeToTempFile(FilePersistenceUtils.java:104)
> {code}
> due to OpenJDK bug/overlayfs bug.
> We acknowledge that this is in fact an OpenJDK AND/OR overlayfs bug. However everything seems to run fine in WildFly except once the backup of the config is triggered, for example by using the cli. Hence WildFly is of limited functionality when the more and more popular overlayfs storage driver is used, and the WildFly team might be interested in providing a workaround on their side since there is no indication the OpenJDK bug will be promptly fixed. We are happy to help in any way, we are still trying to find a workaround on the Java or WildFly side; but we might need insights on why findMountEntry is invoked.
> Full Stack Trace:
> {code:java}
> java.io.IOException: Mount point not foundImage
> at sun.nio.fs.LinuxFileStore.findMountEntry(LinuxFileStore.java:91)
> at sun.nio.fs.UnixFileStore.<init>(UnixFileStore.java:65)
> at sun.nio.fs.LinuxFileStore.<init>(LinuxFileStore.java:44)
> at sun.nio.fs.LinuxFileSystemProvider.getFileStore(LinuxFileSystemProvider.java:51)
> at sun.nio.fs.LinuxFileSystemProvider.getFileStore(LinuxFileSystemProvider.java:39)
> at sun.nio.fs.UnixFileSystemProvider.getFileStore(UnixFileSystemProvider.java:368)
> at java.nio.file.Files.getFileStore(Files.java:1461)
> at org.jboss.as.controller.persistence.FilePersistenceUtils.getPosixAttributes(FilePersistenceUtils.java:129)
> at org.jboss.as.controller.persistence.FilePersistenceUtils.createTempFileWithAttributes(FilePersistenceUtils.java:117)
> at org.jboss.as.controller.persistence.FilePersistenceUtils.writeToTempFile(FilePersistenceUtils.java:104)
> at org.jboss.as.controller.persistence.ConfigurationFilePersistenceResource.doCommit(ConfigurationFilePersistenceResource.java:55)
> at org.jboss.as.controller.persistence.AbstractFilePersistenceResource.commit(AbstractFilePersistenceResource.java:58)
> at org.jboss.as.controller.ModelControllerImpl$4.commit(ModelControllerImpl.java:781)
> at org.jboss.as.controller.AbstractOperationContext.executeDoneStage(AbstractOperationContext.java:743)
> at org.jboss.as.controller.AbstractOperationContext.processStages(AbstractOperationContext.java:680)
> at org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:370)
> at org.jboss.as.controller.OperationContextImpl.executeOperation(OperationContextImpl.java:1344)
> at org.jboss.as.controller.ModelControllerImpl.internalExecute(ModelControllerImpl.java:392)
> at org.jboss.as.controller.ModelControllerImpl.execute(ModelControllerImpl.java:204)
> at org.jboss.as.controller.ModelControllerImpl$3.execute(ModelControllerImpl.java:659)
> at org.jboss.as.controller.ModelControllerImpl$3.execute(ModelControllerImpl.java:649)
> at org.jboss.as.controller.client.helpers.DelegatingModelControllerClient.execute(DelegatingModelControllerClient.java:63)
> at org.jboss.as.cli.embedded.ThreadContextsModelControllerClient.execute(ThreadContextsModelControllerClient.java:59)
> at org.jboss.as.cli.handlers.batch.BatchRunHandler.doHandle(BatchRunHandler.java:91)
> at org.jboss.as.cli.handlers.CommandHandlerWithHelp.handle(CommandHandlerWithHelp.java:88)
> at org.jboss.as.cli.impl.CommandContextImpl.handle(CommandContextImpl.java:776)
> at org.jboss.as.cli.impl.CommandContextImpl.handleSafe(CommandContextImpl.java:799)
> at org.jboss.as.cli.impl.CliLauncher.processFile(CliLauncher.java:334)
> at org.jboss.as.cli.impl.CliLauncher.main(CliLauncher.java:262)
> at org.jboss.as.cli.CommandLineMain.main(CommandLineMain.java:45)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.jboss.modules.Module.run(Module.java:329)
> at org.jboss.modules.Main.main(Main.java:507)
> {code}
> Java Bug Overview:
> https://bugs.openjdk.java.net/browse/JDK-8165852
> http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/8u...
> Wildfly Stack Overflow issue, not solved:
> https://stackoverflow.com/questions/41022393/mount-point-not-found
> Background Info:
> http://mail.openjdk.java.net/pipermail/nio-dev/2016-October/003915.html
> A) chroot environment [1]
> B) Docker container with overlay and overlay2 storage drivers [2]
> C) btrfs file system with an unmounted sub-volume [2]
> [1] https://bugs.openjdk.java.net/browse/JDK-8165323 - cannot get FileStore in chroot environment
> [2] https://bugs.openjdk.java.net/browse/JDK-8165852 - cannot get FileStore for a file in overlayfs in Docker
> Docker file system/storage driver:
> https://docs.docker.com/engine/userguide/storagedriver/selectadriver/)
> Yum yum-plugin-ovl, similar problem:
> https://github.com/CentOS/sig-cloud-instance-images/issues/15
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
9 years, 2 months
[JBoss JIRA] (WFLY-6636) WELD-001408: Unsatisfied dependencies on hot deploy of app using module-alias as dependency
by RH Bugzilla Integration (JIRA)
[ https://issues.jboss.org/browse/WFLY-6636?page=com.atlassian.jira.plugin.... ]
RH Bugzilla Integration commented on WFLY-6636:
-----------------------------------------------
Vladimir Dosoudil <dosoudil(a)redhat.com> changed the Status of [bug 1338093|https://bugzilla.redhat.com/show_bug.cgi?id=1338093] from MODIFIED to ON_QA
> WELD-001408: Unsatisfied dependencies on hot deploy of app using module-alias as dependency
> -------------------------------------------------------------------------------------------
>
> Key: WFLY-6636
> URL: https://issues.jboss.org/browse/WFLY-6636
> Project: WildFly
> Issue Type: Bug
> Components: CDI / Weld
> Affects Versions: 10.0.0.Final
> Reporter: Brad Maxwell
> Assignee: Tomas Hofman
> Attachments: test.ear
>
>
> If a sub deployment uses another sub deployment's module-alias as a dependency, it will deploy successfully, but if hot deployed, it will fail with the error below. Attached reproducer.
> {code}
> <?xml version="1.0" encoding="UTF-8"?>
> <jboss-deployment-structure xmlns="urn:jboss:deployment-structure:1.1">
> <ear-subdeployments-isolated>true</ear-subdeployments-isolated>
> <sub-deployment name="ejb1.jar">
> <module-alias name="deployment.ejb1"/>
> </sub-deployment>
> <sub-deployment name="ejb2.jar">
> <dependencies>
> <!-- works
> <module name="deployment.test.ear.ejb1.jar" slot="main"/>
> -->
> <!-- fails with WELD-001408: Unsatisfied dependencies for type TestEJB1 with qualifiers @Default on redeploy / hot deploy -->
> <module name="deployment.ejb1" slot="main"/>
> </dependencies>
> </sub-deployment>
> </jboss-deployment-structure>
> {code}
> {code}
> 10:43:42,140 ERROR [org.jboss.msc.service.fail] (MSC service thread 1-8) MSC000001: Failed to start service jboss.deployment.unit."test.ear".WeldStartService: org.jboss.msc.service.StartException in service jboss.deployment.unit."test.ear".WeldStartService: Failed to start service
> at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1904)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.jboss.weld.exceptions.DeploymentException: WELD-001408: Unsatisfied dependencies for type TestEJB1 with qualifiers @Default
> at injection point [BackedAnnotatedField] @Inject private test.TestEJB2.test1
> at test.TestEJB2.test1(TestEJB2.java:0)
> at org.jboss.weld.bootstrap.Validator.validateInjectionPointForDeploymentProblems(Validator.java:359)
> at org.jboss.weld.bootstrap.Validator.validateInjectionPoint(Validator.java:281)
> at org.jboss.weld.bootstrap.Validator.validateGeneralBean(Validator.java:134)
> at org.jboss.weld.bootstrap.Validator.validateRIBean(Validator.java:155)
> at org.jboss.weld.bootstrap.Validator.validateBean(Validator.java:518)
> at org.jboss.weld.bootstrap.ConcurrentValidator$1.doWork(ConcurrentValidator.java:68)
> at org.jboss.weld.bootstrap.ConcurrentValidator$1.doWork(ConcurrentValidator.java:66)
> at org.jboss.weld.executor.IterativeWorkerTaskFactory$1.call(IterativeWorkerTaskFactory.java:63)
> at org.jboss.weld.executor.IterativeWorkerTaskFactory$1.call(IterativeWorkerTaskFactory.java:56)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> at org.jboss.threads.JBossThread.run(JBossThread.java:320)
> 10:43:42,144 ERROR [org.jboss.as.controller.management-operation] (DeploymentScanner-threads - 2) WFLYCTL0013: Operation ("full-replace-deployment") failed - address: ([]) - failure description: {"WFLYCTL0080: Failed services" => {"jboss.deployment.unit.\"test.ear\".WeldStartService" => "org.jboss.msc.service.StartException in service jboss.deployment.unit.\"test.ear\".WeldStartService: Failed to start service
> Caused by: org.jboss.weld.exceptions.DeploymentException: WELD-001408: Unsatisfied dependencies for type TestEJB1 with qualifiers @Default
> at injection point [BackedAnnotatedField] @Inject private test.TestEJB2.test1
> at test.TestEJB2.test1(TestEJB2.java:0)
> "}}
> {code}
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
9 years, 2 months
[JBoss JIRA] (WFLY-5274) EJB IOR contains wrong port (non-SSL port) information when SSL is required
by RH Bugzilla Integration (JIRA)
[ https://issues.jboss.org/browse/WFLY-5274?page=com.atlassian.jira.plugin.... ]
RH Bugzilla Integration commented on WFLY-5274:
-----------------------------------------------
Vladimir Dosoudil <dosoudil(a)redhat.com> changed the Status of [bug 1259902|https://bugzilla.redhat.com/show_bug.cgi?id=1259902] from MODIFIED to ON_QA
> EJB IOR contains wrong port (non-SSL port) information when SSL is required
> ---------------------------------------------------------------------------
>
> Key: WFLY-5274
> URL: https://issues.jboss.org/browse/WFLY-5274
> Project: WildFly
> Issue Type: Bug
> Components: IIOP
> Reporter: Derek Horton
> Assignee: Tomasz Adamski
>
> Description of problem:
> - Configure JBoss to only allow IIOP connections over SSL
> - It is possible to do this, but the configuration is confusing (possibly a bug)
> Details of the setup/issue:
> - When enabling SSL for jacorb, it normally listens on both the non-ssl port and the ssl port
> - Setting server-requires="ServerAuth" causes the server to stop listening on non-ssl port
> - However, the IOR tells client to connect to non-ssl port ...even though its not listening on it
> String lookup = "corbaname:iiop:" + host + ":" + port +"#" + ejbLookupPath;
> // lookup the IIOP EJB
> Object iiopObj = ctx.lookup(lookup);
> // the call to the EJB will fail due to the port being wrong non-ssl vs ssl
> - The workaround is to use the following ior-setting to correct the port settings in the IOR
> /subsystem=jacorb/ior-settings=default/setting=transport-config:add(confidentiality=required)
> - Shouldn't setting "server-requires=ServerAuth" change the port info in the IOR?
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
9 years, 2 months
[JBoss JIRA] (WFLY-8141) CachedConnectionManager add operation excepts no parameters anymore
by Brian Stansberry (JIRA)
[ https://issues.jboss.org/browse/WFLY-8141?page=com.atlassian.jira.plugin.... ]
Brian Stansberry commented on WFLY-8141:
----------------------------------------
Thanks, [~simkam].
FWIW, I wrote this to [~maeste] and [~aloubyansky] a week or so ago in relation to a different problem this resource causes:
---------------------
Hi guys,
This is a follow up to the chat Alexey and I had about the CCM resource earlier today.
Basic summary is the RA subsystem parser ensures that an add op for the CCM resource is always part of the boot ops, so if the JCA subsystem has been parsed, there’s automatically a CCM resource. And the user can’t remove and readd it.
The current situation (which dates to the beginning I guess) violates the requirement that you don’t have to use xml editing to configure. You can’t start with a config with no JCA subsystem, then via the CLI add it and then configure the CCM resource. At least not without a reload in the middle.
Not having to us xml to configure has always been a requirement but with the offline CLI and the new provisioning stuff based on it, it’s an even stronger requirement now.
I see a couple options:
1)
a) Make the CCM add op public again,
b) Drop the automatic add from the parser. Parser only creates the add op if the xml configures it
c) Have JcaSubsystemAdd add a step that checks if the CCM resource exists (in case user configured it in xml or in the same composite that adds subsystem=jca) and if not that step in turn adds a step that executes the CachedConnectionManagerAdd op with the default params
d) Make CachedConnectionManagerRemove a normal remove again; removes the resource and requires reload
2) Add a parameter to the JCASubsystemAdd op. Complex object, fields are the same as the attributes in the CCM resource. Users can configure CCM when adding the JCA subsystem by setting this parameter. If present, JCASubsystemAdd takes the values in the param and adds a step that executes the CachedConnectionManagerAdd op.
--
Brian Stansberry
Manager, Senior Principal Software Engineer
JBoss by Red Hat
--------------
Since the add op was actually useful in domain mode, it seems like 1) is what we need to do. Otherwise we've introduced an incompatible API change. Removing it was based on my mistaken belief that it was a useless operation.
> CachedConnectionManager add operation excepts no parameters anymore
> -------------------------------------------------------------------
>
> Key: WFLY-8141
> URL: https://issues.jboss.org/browse/WFLY-8141
> Project: WildFly
> Issue Type: Bug
> Components: Domain Management, JCA
> Affects Versions: 11.0.0.Alpha1
> Reporter: Tomaz Cerar
> Assignee: Ingo Weiss
> Priority: Critical
>
> Fix for WFLY-2640 broke :add operation for cached-connection-manager
> scipts that do
> {noformat}
> /profile=default-web/subsystem=jca/cached-connection-manager=cached-connection-manager:add(install="true")
> {noformat}
> {noformat}
> /subsystem=jca/cached-connection-manager=cached-connection-manager:add(install="true")
> {noformat}
> now fail with
> {{Operation 'add' does not expect any property.}}
> This breaks our quickstarts
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
9 years, 2 months
[JBoss JIRA] (WFLY-3858) Infinispan cache configuration is not always applied to security-domain
by Bartosz Spyrko-Śmietanko (JIRA)
[ https://issues.jboss.org/browse/WFLY-3858?page=com.atlassian.jira.plugin.... ]
Bartosz Spyrko-Śmietanko reassigned WFLY-3858:
----------------------------------------------
Assignee: Bartosz Spyrko-Śmietanko (was: Darran Lofthouse)
> Infinispan cache configuration is not always applied to security-domain
> -----------------------------------------------------------------------
>
> Key: WFLY-3858
> URL: https://issues.jboss.org/browse/WFLY-3858
> Project: WildFly
> Issue Type: Bug
> Components: Security
> Affects Versions: 8.1.0.Final
> Reporter: Robert Tuck
> Assignee: Bartosz Spyrko-Śmietanko
> Attachments: debugger output.txt
>
>
> On Wildfly 8.1.0.Final, I have the following standalone-ha.xml:
> <subsystem xmlns="urn:jboss:domain:infinispan:2.0">
> ...
> <cache-container name="security" default-cache="auth-cache" start="EAGER">
> <transport cluster="${cluster.name}_SEC" lock-timeout="60000"/>
> <replicated-cache name="auth-cache" batching="true" mode="ASYNC">
> <eviction strategy="LRU" max-entries="10000"/>
> <expiration lifespan="60000"/>
> </replicated-cache>
> </cache-container>
> </subsystem>
> ...
> <subsystem xmlns="urn:jboss:domain:security:1.2">
> <security-domains>
> ...
> <security-domain name="OAuth-Consumer" cache-type="infinispan">
> <authentication>
> <login-module code="com.idbs.ewb.server.auth.module.OAuthConsumerLoginModule" flag="sufficient"
> module="deployment.ewb-server-ear.ear">
> <module-option name="allowedConsumerAuthFailures" value="-1"/>
> <module-option name="consumerLoginFailureTimeoutMs" value="3000"/>
> </login-module>
> </authentication>
> </security-domain>
> </security-domains>
> </subsystem>
> After startup the OAuth-Consumer security domain cache "auth-cache" is not always configured with the specified settings (~50% of the time). This can be verified by monitoring the jboss.infinispan nodes with JConsole and retrieving the cache settings, and tracking the cache hits during logins. This shows that succesful logins are cached but do not expire after the expected 60s, and that the expiration lifespan is set to -1 rather than 60000, as are eviction max entries.
> After some debugging I have narrowed down the problem to some code that gets called from inside org.jboss.as.security.plugins.JNDIBasedSecurityManagement:
> public SecurityDomainContext createSecurityDomainContext(String securityDomain, Object cacheFactory) throws Exception {
> log.debugf("Creating SDC for domain=" + securityDomain);
> AuthenticationManager am = createAuthenticationManager(securityDomain);
> // create authentication cache
> if (cacheFactory instanceof EmbeddedCacheManager) {
> EmbeddedCacheManager cacheManager = EmbeddedCacheManager.class.cast(cacheFactory);
> @SuppressWarnings("rawtypes")
> Cache cache = null;
> if (cacheManager != null) {
> // TODO override global settings with security domain specific
> ConfigurationBuilder builder = new ConfigurationBuilder();
> Configuration baseCfg = cacheManager.getCacheConfiguration("auth-cache");
> ^^^^ This call here doesn’t always return the correct configuration, baseCfg is then null.
> if (baseCfg != null) {
> builder.read(baseCfg);
> }
> cacheManager.defineConfiguration(securityDomain, builder.build());
> cache = cacheManager.getCache(securityDomain);
> }
> if (cache != null && am instanceof CacheableManager) {
> @SuppressWarnings({ "unchecked", "rawtypes" })
> CacheableManager<Map, Principal> cm = (CacheableManager<Map, Principal>) am;
> cm.setCache(cache);
> }
> } else if (cacheFactory instanceof DefaultAuthenticationCacheFactory) {
> <…>
> }
> getCacheConfiguration(String) is implemented inside org.infinispan.manager.DefaultCacheManager:
> @Override
> public Configuration getCacheConfiguration(String name) {
> Configuration configuration = configurationOverrides.get(name);
> if (configuration == null && cacheExists(name)) {
> return defaultConfiguration;
> }
> return configuration;
> }
> Seems like the condition configuration == null occurs when the cache doesn’t exist, therefore it returns null. This appears to be a race condition between this code and where it gets registered in wireAndStartCache(String).
>
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
9 years, 2 months
[JBoss JIRA] (WFLY-8143) Empty secret-value is not allowed in credential stores
by Josef Cacek (JIRA)
Josef Cacek created WFLY-8143:
---------------------------------
Summary: Empty secret-value is not allowed in credential stores
Key: WFLY-8143
URL: https://issues.jboss.org/browse/WFLY-8143
Project: WildFly
Issue Type: Bug
Components: Security
Reporter: Josef Cacek
Assignee: Darran Lofthouse
Priority: Critical
It's not possible to add an entry with empty secret-value into a credential store.
Masking the fact the password is empty is a valid scenario.
{code}
[standalone@localhost:9990 /] /subsystem=elytron/credential-store=cred-store-default/alias=emptysecret:add()
{
"outcome" => "failed",
"failure-description" => "WFLYCTL0155: 'secret-value' may not be null",
"rolled-back" => true
}
[standalone@localhost:9990 /] /subsystem=elytron/credential-store=cred-store-default/alias=emptysecret:add(secret-value="")
{
"outcome" => "failed",
"failure-description" => "WFLYCTL0113: '' is an invalid value for parameter secret-value. Values must have a minimum length of 1 characters",
"rolled-back" => true
}
{code}
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
9 years, 2 months