[JBoss JIRA] (WFCORE-2301) Mount point not found exception raised by createTempFileWithAttributes on overlayfs [JDK-8165852]
by Brian Stansberry (JIRA)
[ https://issues.jboss.org/browse/WFCORE-2301?page=com.atlassian.jira.plugi... ]
Brian Stansberry commented on WFCORE-2301:
------------------------------------------
Great; glad to hear it. We still haven't released WildFly Core 3, so this is good feedback. :)
> Mount point not found exception raised by createTempFileWithAttributes on overlayfs [JDK-8165852]
> -------------------------------------------------------------------------------------------------
>
> Key: WFCORE-2301
> URL: https://issues.jboss.org/browse/WFCORE-2301
> Project: WildFly Core
> Issue Type: Bug
> Components: Domain Management
> Environment: WildFly via KeyCloak 2.5.1.Final
> {code:xml}
> <eap.version>7.0.0.Beta</eap.version>
> <jboss.as.version>7.2.0.Final</jboss.as.version>
> <wildfly.version>10.0.0.Final</wildfly.version>
> {code}
> on Docker with overlayfs or overlayfs2 as storage driver
> \# docker info | grep -i storage
> aufs: works (e.g., boot2docker, legacy minikube)
> overlay (e.g., CoreOS, current minikube): problem
> devicemapper (e.g., CentOS): works
> overlay2 (e.g., Docker for Mac): problem
> Reporter: Bjoern Stuetz
> Assignee: Brian Stansberry
> Fix For: 3.0.0.Beta3, 2.2.1.CR2
>
>
> Mount point not found exception raised by createTempFileWithAttributes on overlayfs [JDK-8165852], i.e.,
> /opt/jboss/bin/jboss-cli.sh --file=/opt/jboss/jboss-config.cli
> inside a Docker container running on overlayfs as storage driver
> causes (full stack trace below):
> {code:java}
> java.io.IOException: Mount point not foundImage
> at sun.nio.fs.LinuxFileStore.findMountEntry(LinuxFileStore.java:91)
> {code}
> triggered by
> {code:java}
> at org.jboss.as.controller.persistence.FilePersistenceUtils.createTempFileWithAttributes(FilePersistenceUtils.java:117)
> at org.jboss.as.controller.persistence.FilePersistenceUtils.writeToTempFile(FilePersistenceUtils.java:104)
> {code}
> due to OpenJDK bug/overlayfs bug.
> We acknowledge that this is in fact an OpenJDK AND/OR overlayfs bug. However everything seems to run fine in WildFly except once the backup of the config is triggered, for example by using the cli. Hence WildFly is of limited functionality when the more and more popular overlayfs storage driver is used, and the WildFly team might be interested in providing a workaround on their side since there is no indication the OpenJDK bug will be promptly fixed. We are happy to help in any way, we are still trying to find a workaround on the Java or WildFly side; but we might need insights on why findMountEntry is invoked.
> Full Stack Trace:
> {code:java}
> java.io.IOException: Mount point not foundImage
> at sun.nio.fs.LinuxFileStore.findMountEntry(LinuxFileStore.java:91)
> at sun.nio.fs.UnixFileStore.<init>(UnixFileStore.java:65)
> at sun.nio.fs.LinuxFileStore.<init>(LinuxFileStore.java:44)
> at sun.nio.fs.LinuxFileSystemProvider.getFileStore(LinuxFileSystemProvider.java:51)
> at sun.nio.fs.LinuxFileSystemProvider.getFileStore(LinuxFileSystemProvider.java:39)
> at sun.nio.fs.UnixFileSystemProvider.getFileStore(UnixFileSystemProvider.java:368)
> at java.nio.file.Files.getFileStore(Files.java:1461)
> at org.jboss.as.controller.persistence.FilePersistenceUtils.getPosixAttributes(FilePersistenceUtils.java:129)
> at org.jboss.as.controller.persistence.FilePersistenceUtils.createTempFileWithAttributes(FilePersistenceUtils.java:117)
> at org.jboss.as.controller.persistence.FilePersistenceUtils.writeToTempFile(FilePersistenceUtils.java:104)
> at org.jboss.as.controller.persistence.ConfigurationFilePersistenceResource.doCommit(ConfigurationFilePersistenceResource.java:55)
> at org.jboss.as.controller.persistence.AbstractFilePersistenceResource.commit(AbstractFilePersistenceResource.java:58)
> at org.jboss.as.controller.ModelControllerImpl$4.commit(ModelControllerImpl.java:781)
> at org.jboss.as.controller.AbstractOperationContext.executeDoneStage(AbstractOperationContext.java:743)
> at org.jboss.as.controller.AbstractOperationContext.processStages(AbstractOperationContext.java:680)
> at org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:370)
> at org.jboss.as.controller.OperationContextImpl.executeOperation(OperationContextImpl.java:1344)
> at org.jboss.as.controller.ModelControllerImpl.internalExecute(ModelControllerImpl.java:392)
> at org.jboss.as.controller.ModelControllerImpl.execute(ModelControllerImpl.java:204)
> at org.jboss.as.controller.ModelControllerImpl$3.execute(ModelControllerImpl.java:659)
> at org.jboss.as.controller.ModelControllerImpl$3.execute(ModelControllerImpl.java:649)
> at org.jboss.as.controller.client.helpers.DelegatingModelControllerClient.execute(DelegatingModelControllerClient.java:63)
> at org.jboss.as.cli.embedded.ThreadContextsModelControllerClient.execute(ThreadContextsModelControllerClient.java:59)
> at org.jboss.as.cli.handlers.batch.BatchRunHandler.doHandle(BatchRunHandler.java:91)
> at org.jboss.as.cli.handlers.CommandHandlerWithHelp.handle(CommandHandlerWithHelp.java:88)
> at org.jboss.as.cli.impl.CommandContextImpl.handle(CommandContextImpl.java:776)
> at org.jboss.as.cli.impl.CommandContextImpl.handleSafe(CommandContextImpl.java:799)
> at org.jboss.as.cli.impl.CliLauncher.processFile(CliLauncher.java:334)
> at org.jboss.as.cli.impl.CliLauncher.main(CliLauncher.java:262)
> at org.jboss.as.cli.CommandLineMain.main(CommandLineMain.java:45)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.jboss.modules.Module.run(Module.java:329)
> at org.jboss.modules.Main.main(Main.java:507)
> {code}
> Java Bug Overview:
> https://bugs.openjdk.java.net/browse/JDK-8165852
> http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/8u...
> Wildfly Stack Overflow issue, not solved:
> https://stackoverflow.com/questions/41022393/mount-point-not-found
> Background Info:
> http://mail.openjdk.java.net/pipermail/nio-dev/2016-October/003915.html
> A) chroot environment [1]
> B) Docker container with overlay and overlay2 storage drivers [2]
> C) btrfs file system with an unmounted sub-volume [2]
> [1] https://bugs.openjdk.java.net/browse/JDK-8165323 - cannot get FileStore in chroot environment
> [2] https://bugs.openjdk.java.net/browse/JDK-8165852 - cannot get FileStore for a file in overlayfs in Docker
> Docker file system/storage driver:
> https://docs.docker.com/engine/userguide/storagedriver/selectadriver/)
> Yum yum-plugin-ovl, similar problem:
> https://github.com/CentOS/sig-cloud-instance-images/issues/15
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
8 years, 11 months
[JBoss JIRA] (WFCORE-2876) runtime-failure-causes-rollback does not seem to have effect when configured in model
by Miroslav Novak (JIRA)
[ https://issues.jboss.org/browse/WFCORE-2876?page=com.atlassian.jira.plugi... ]
Miroslav Novak commented on WFCORE-2876:
----------------------------------------
Ok, the issue was that test was deploying MDB using model controller client and not by copying MDB to deployments directory. I've updated steps to reproduce. Could you "git pull" latest changes and try the reproducer again, please? Test should fail now.
> runtime-failure-causes-rollback does not seem to have effect when configured in model
> -------------------------------------------------------------------------------------
>
> Key: WFCORE-2876
> URL: https://issues.jboss.org/browse/WFCORE-2876
> Project: WildFly Core
> Issue Type: Bug
> Components: Deployment Scanner
> Affects Versions: 3.0.0.Beta21
> Reporter: Miroslav Novak
> Assignee: ehsavoie Hugonnet
>
> This is follow up on discussion in WFCORE-1912. There is difference in behavior if deployment is deployed like:
> {code}deploy ~/tmp/mdb1.jar --unmanaged --headers={rollback-on-runtime-failure=false}{code}
> and if it's deployed by copying to deployments directory and setting {{rollback-on-runtime-failure=false}} in model:
> {code}
> /subsystem=deployment-scanner/scanner=default:write-attribute(name=runtime-failure-causes-rollback,value=false)
> {code}
> If it's deployed by the 1st way using CLI {{deploy}} command then If deployment is missing some dependencies (like connection factories, queues/topics) and those are added later then deployment is able recover from it and start working without need to reload/restart server or redeploy.
> However if it's deployed the 2nd way then deployment does not recover when missing dependencies are added. It looks like attribute {{runtime-failure-causes-rollback}} does not have the effect in this case.
> This is causing problems when Artemis is configured in colocated HA topology with replicated journal where it takes some time for Artemis to activate but deployment which depends on some connection factories and queues has already tried to deploy. We need deployment to recover from this situation automatically. Restart/Reload will not help in this case as it would end up in the same situation. Only manual redeploy will help which is a workaround.
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
8 years, 11 months
[JBoss JIRA] (WFCORE-2876) runtime-failure-causes-rollback does not seem to have effect when configured in model
by Miroslav Novak (JIRA)
[ https://issues.jboss.org/browse/WFCORE-2876?page=com.atlassian.jira.plugi... ]
Miroslav Novak updated WFCORE-2876:
-----------------------------------
Steps to Reproduce:
Steps to reproduce:
{code}
git clone git://git.app.eng.bos.redhat.com/jbossqe/eap-tests-hornetq.git
cd eap-tests-hornetq/scripts/
git checkout master
groovy -DEAP_VERSION=7.1.0.DR19 PrepareServers7.groovy
export WORKSPACE=$PWD
export JBOSS_HOME_1=$WORKSPACE/server1/jboss-eap
export JBOSS_HOME_2=$WORKSPACE/server2/jboss-eap
export JBOSS_HOME_3=$WORKSPACE/server3/jboss-eap
export JBOSS_HOME_4=$WORKSPACE/server4/jboss-eap
cd ../jboss-hornetq-testsuite/
mvn clean test -Dtest=ReplicatedColocatedClusterFailoverTestCase#testFailbackWithMdbsShutdownDeployByCopy -DfailIfNoTests=false -Deap=7x | tee log
{code}
There is 2nd test which is deploying MDB using CLI {{deploy}} command and shows that MDB is awakens on node 2 once Artemis live activates and deploys queues/connection factories:
ReplicatedColocatedClusterFailoverTestCase#testFailbackWithMdbsShutdownDeployUsingCLI
was:
Steps to reproduce:
{code}
git clone git://git.app.eng.bos.redhat.com/jbossqe/eap-tests-hornetq.git
cd eap-tests-hornetq/scripts/
git checkout master
groovy -DEAP_VERSION=7.1.0.DR19 PrepareServers7.groovy
export WORKSPACE=$PWD
export JBOSS_HOME_1=$WORKSPACE/server1/jboss-eap
export JBOSS_HOME_2=$WORKSPACE/server2/jboss-eap
export JBOSS_HOME_3=$WORKSPACE/server3/jboss-eap
export JBOSS_HOME_4=$WORKSPACE/server4/jboss-eap
cd ../jboss-hornetq-testsuite/
mvn clean test -Dtest=ReplicatedColocatedClusterFailoverTestCase#testFailbackWithMdbsShutdown -DfailIfNoTests=false -Deap=7x | tee log
{code}
There is 2nd test which is deploying MDB using CLI {{deploy}} command and shows that MDB is awakens on node 2 once Artemis live activates and deploys queues/connection factories:
ReplicatedColocatedClusterFailoverTestCase#testFailbackWithMdbsShutdownDeployUsingCLI
> runtime-failure-causes-rollback does not seem to have effect when configured in model
> -------------------------------------------------------------------------------------
>
> Key: WFCORE-2876
> URL: https://issues.jboss.org/browse/WFCORE-2876
> Project: WildFly Core
> Issue Type: Bug
> Components: Deployment Scanner
> Affects Versions: 3.0.0.Beta21
> Reporter: Miroslav Novak
> Assignee: ehsavoie Hugonnet
>
> This is follow up on discussion in WFCORE-1912. There is difference in behavior if deployment is deployed like:
> {code}deploy ~/tmp/mdb1.jar --unmanaged --headers={rollback-on-runtime-failure=false}{code}
> and if it's deployed by copying to deployments directory and setting {{rollback-on-runtime-failure=false}} in model:
> {code}
> /subsystem=deployment-scanner/scanner=default:write-attribute(name=runtime-failure-causes-rollback,value=false)
> {code}
> If it's deployed by the 1st way using CLI {{deploy}} command then If deployment is missing some dependencies (like connection factories, queues/topics) and those are added later then deployment is able recover from it and start working without need to reload/restart server or redeploy.
> However if it's deployed the 2nd way then deployment does not recover when missing dependencies are added. It looks like attribute {{runtime-failure-causes-rollback}} does not have the effect in this case.
> This is causing problems when Artemis is configured in colocated HA topology with replicated journal where it takes some time for Artemis to activate but deployment which depends on some connection factories and queues has already tried to deploy. We need deployment to recover from this situation automatically. Restart/Reload will not help in this case as it would end up in the same situation. Only manual redeploy will help which is a workaround.
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
8 years, 11 months
[JBoss JIRA] (ELY-1209) Revisit allow, forbid and selector of sasl mechanisms in Elytron subsystem and client config file
by Ondrej Lukas (JIRA)
[ https://issues.jboss.org/browse/ELY-1209?page=com.atlassian.jira.plugin.s... ]
Ondrej Lukas updated ELY-1209:
------------------------------
Affects Version/s: 1.1.0.Beta47
> Revisit allow, forbid and selector of sasl mechanisms in Elytron subsystem and client config file
> -------------------------------------------------------------------------------------------------
>
> Key: ELY-1209
> URL: https://issues.jboss.org/browse/ELY-1209
> Project: WildFly Elytron
> Issue Type: Bug
> Affects Versions: 1.1.0.Beta47
> Reporter: Ondrej Lukas
> Assignee: Darran Lofthouse
> Priority: Critical
>
> There are some topics for revising in {{allow-all-mechanisms}}, {{allow-sasl-mechanisms}}, {{forbid-sasl-mechanisms}} and {{sasl-mechanism-selector}} of Elytron subsystem and client config file.
> 1) Since selectors have been introduced in EAP 7.1.0.DR19 what is the reason for {{allow-all-mechanisms}}, {{allow-sasl-mechanisms}} and {{forbid-sasl-mechanisms}}? AFAIK they just provides the subset of configuration which can be set by {{sasl-mechanism-selector}}. It that case {{allow-all-mechanisms}}, {{allow-sasl-mechanisms}} and {{forbid-sasl-mechanisms}} can be completely removed from Elytron configuration because they just duplicates another configuration. Or they provide something which cannot be configured by selectors?
> 2) These options are mutually exclusive in Elytron subsystem, but all of them can be configured together in Elytron client configuration file. There should be added some check for mutually exclusivity of these options in Elytron client configuration file.
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
8 years, 11 months
[JBoss JIRA] (ELY-1209) Revisit allow, forbid and selector of sasl mechanisms in Elytron subsystem and client config file
by Ondrej Lukas (JIRA)
Ondrej Lukas created ELY-1209:
---------------------------------
Summary: Revisit allow, forbid and selector of sasl mechanisms in Elytron subsystem and client config file
Key: ELY-1209
URL: https://issues.jboss.org/browse/ELY-1209
Project: WildFly Elytron
Issue Type: Bug
Reporter: Ondrej Lukas
Assignee: Darran Lofthouse
Priority: Critical
There are some topics for revising in {{allow-all-mechanisms}}, {{allow-sasl-mechanisms}}, {{forbid-sasl-mechanisms}} and {{sasl-mechanism-selector}} of Elytron subsystem and client config file.
1) Since selectors have been introduced in EAP 7.1.0.DR19 what is the reason for {{allow-all-mechanisms}}, {{allow-sasl-mechanisms}} and {{forbid-sasl-mechanisms}}? AFAIK they just provides the subset of configuration which can be set by {{sasl-mechanism-selector}}. It that case {{allow-all-mechanisms}}, {{allow-sasl-mechanisms}} and {{forbid-sasl-mechanisms}} can be completely removed from Elytron configuration because they just duplicates another configuration. Or they provide something which cannot be configured by selectors?
2) These options are mutually exclusive in Elytron subsystem, but all of them can be configured together in Elytron client configuration file. There should be added some check for mutually exclusivity of these options in Elytron client configuration file.
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
8 years, 11 months