[JBoss JIRA] (ISPN-6293) Registering a clustered listener in a DIST cluster leads to SuspectException if one member is not joined yet
by Adrian Nistor (JIRA)
Adrian Nistor created ISPN-6293:
-----------------------------------
Summary: Registering a clustered listener in a DIST cluster leads to SuspectException if one member is not joined yet
Key: ISPN-6293
URL: https://issues.jboss.org/browse/ISPN-6293
Project: Infinispan
Issue Type: Bug
Affects Versions: 8.2.0.Beta2
Reporter: Adrian Nistor
Assignee: William Burns
Fix For: 8.2.0.Final
Reproducer:
{code}
import org.infinispan.configuration.cache.CacheMode;
import org.infinispan.configuration.cache.ConfigurationBuilder;
import org.infinispan.notifications.Listener;
import org.infinispan.notifications.cachelistener.annotation.CacheEntryCreated;
import org.infinispan.notifications.cachelistener.event.CacheEntryEvent;
import org.infinispan.test.MultipleCacheManagersTest;
import org.testng.annotations.Test;
@Test(groups = "functional", testName = "ClusteredListenerJoinsTest")
public class ClusteredListenerJoinsTest extends MultipleCacheManagersTest {
@Listener(clustered = true)
private static final class NoOpListener {
@CacheEntryCreated
public void handleEvent(CacheEntryEvent<Object, Object> e) {
}
}
@Override
protected void createCacheManagers() {
ConfigurationBuilder c = buildConfiguration();
createCluster(c, 3);
waitForClusterToForm();
}
private ConfigurationBuilder buildConfiguration() {
ConfigurationBuilder c = getDefaultClusteredCacheConfig(CacheMode.DIST_SYNC, false); //unreproducible with CacheMode.REPL_SYNC
c.clustering().hash().numOwners(2);
return c;
}
public void testJoins() {
cache(0).addListener(new NoOpListener());
addClusterEnabledCacheManager(buildConfiguration());
//waitForClusterToForm();
cache(0).addListener(new NoOpListener()); //fails most times with SuspectException; uncomment the line above to ensure it does not fail
}
}
{code}
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
8 years, 2 months
[JBoss JIRA] (ISPN-6293) Registering a clustered listener in a DIST cluster leads to SuspectException if one member is not joined yet
by Adrian Nistor (JIRA)
[ https://issues.jboss.org/browse/ISPN-6293?page=com.atlassian.jira.plugin.... ]
Adrian Nistor updated ISPN-6293:
--------------------------------
Status: Open (was: New)
> Registering a clustered listener in a DIST cluster leads to SuspectException if one member is not joined yet
> ------------------------------------------------------------------------------------------------------------
>
> Key: ISPN-6293
> URL: https://issues.jboss.org/browse/ISPN-6293
> Project: Infinispan
> Issue Type: Bug
> Affects Versions: 8.2.0.Beta2
> Reporter: Adrian Nistor
> Assignee: William Burns
> Fix For: 8.2.0.Final
>
>
> Reproducer:
> {code}
> import org.infinispan.configuration.cache.CacheMode;
> import org.infinispan.configuration.cache.ConfigurationBuilder;
> import org.infinispan.notifications.Listener;
> import org.infinispan.notifications.cachelistener.annotation.CacheEntryCreated;
> import org.infinispan.notifications.cachelistener.event.CacheEntryEvent;
> import org.infinispan.test.MultipleCacheManagersTest;
> import org.testng.annotations.Test;
> @Test(groups = "functional", testName = "ClusteredListenerJoinsTest")
> public class ClusteredListenerJoinsTest extends MultipleCacheManagersTest {
> @Listener(clustered = true)
> private static final class NoOpListener {
> @CacheEntryCreated
> public void handleEvent(CacheEntryEvent<Object, Object> e) {
> }
> }
> @Override
> protected void createCacheManagers() {
> ConfigurationBuilder c = buildConfiguration();
> createCluster(c, 3);
> waitForClusterToForm();
> }
> private ConfigurationBuilder buildConfiguration() {
> ConfigurationBuilder c = getDefaultClusteredCacheConfig(CacheMode.DIST_SYNC, false); //unreproducible with CacheMode.REPL_SYNC
> c.clustering().hash().numOwners(2);
> return c;
> }
> public void testJoins() {
> cache(0).addListener(new NoOpListener());
> addClusterEnabledCacheManager(buildConfiguration());
> //waitForClusterToForm();
> cache(0).addListener(new NoOpListener()); //fails most times with SuspectException; uncomment the line above to ensure it does not fail
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
8 years, 2 months
[JBoss JIRA] (ISPN-6261) CLI fails on secured caches
by Jakub Markos (JIRA)
[ https://issues.jboss.org/browse/ISPN-6261?page=com.atlassian.jira.plugin.... ]
Jakub Markos updated ISPN-6261:
-------------------------------
Description:
When running CLI on secured caches, it fails with exception bellow. This exception is quite confusing, as user has properly defined ADMIN permission on given cache. What is actually happening is that some operation, like statistics, called by CLI, iterates over all defined caches, including internal caches, and user hasn't required permission on all these caches (in this case on internal script cache)
{noformat}
00:04:23,563 ERROR [org.jboss.as.controller.management-operation] (management-handler-thread - 1) WFLYCTL0013: Operation ("read-attribute") failed - address: ([
("subsystem" => "datagrid-infinispan"),
("cache-container" => "local")
]): java.lang.SecurityException: ISPN000287: Unauthorized access: subject 'Subject with principal(s): [user@ManagementRealm, admin@ManagementRealm, admin, org.jboss.remoting3.security.UserPrincipal@36ebcb, InetAddressPrincipal <127.0.0.1/127.0.0.1>, InetAddressPrincipal <127.0.0.1/127.0.0.1>]' lacks 'ADMIN' permission
at org.infinispan.security.impl.AuthorizationHelper.checkPermission(AuthorizationHelper.java:86)
at org.infinispan.security.impl.AuthorizationManagerImpl.checkPermission(AuthorizationManagerImpl.java:37)
at org.infinispan.security.impl.SecureCacheImpl.getStats(SecureCacheImpl.java:567)
at org.infinispan.stats.impl.CacheContainerStatsImpl.calculateAverageRemoveTime(CacheContainerStatsImpl.java:131)
at org.infinispan.stats.impl.CacheContainerStatsImpl.getAverageRemoveTime(CacheContainerStatsImpl.java:121)
at org.jboss.as.clustering.infinispan.subsystem.CacheContainerMetricsHandler.executeRuntimeStep(CacheContainerMetricsHandler.java:196)
at org.jboss.as.controller.AbstractRuntimeOnlyHandler$1.execute(AbstractRuntimeOnlyHandler.java:53)
{noformat}
was:
# When running CLI on secured caches, it fails with exception bellow. This exception is quite confusing, as user has properly defined ADMIN permission on given cache. What is actually happening is that some operation, like statistics, called by CLI, iterates over all defined caches, including internal caches, and user hasn't required permission on all these caches (in this case on internal script cache)
{noformat}
00:04:23,563 ERROR [org.jboss.as.controller.management-operation] (management-handler-thread - 1) WFLYCTL0013: Operation ("read-attribute") failed - address: ([
("subsystem" => "datagrid-infinispan"),
("cache-container" => "local")
]): java.lang.SecurityException: ISPN000287: Unauthorized access: subject 'Subject with principal(s): [user@ManagementRealm, admin@ManagementRealm, admin, org.jboss.remoting3.security.UserPrincipal@36ebcb, InetAddressPrincipal <127.0.0.1/127.0.0.1>, InetAddressPrincipal <127.0.0.1/127.0.0.1>]' lacks 'ADMIN' permission
at org.infinispan.security.impl.AuthorizationHelper.checkPermission(AuthorizationHelper.java:86)
at org.infinispan.security.impl.AuthorizationManagerImpl.checkPermission(AuthorizationManagerImpl.java:37)
at org.infinispan.security.impl.SecureCacheImpl.getStats(SecureCacheImpl.java:567)
at org.infinispan.stats.impl.CacheContainerStatsImpl.calculateAverageRemoveTime(CacheContainerStatsImpl.java:131)
at org.infinispan.stats.impl.CacheContainerStatsImpl.getAverageRemoveTime(CacheContainerStatsImpl.java:121)
at org.jboss.as.clustering.infinispan.subsystem.CacheContainerMetricsHandler.executeRuntimeStep(CacheContainerMetricsHandler.java:196)
at org.jboss.as.controller.AbstractRuntimeOnlyHandler$1.execute(AbstractRuntimeOnlyHandler.java:53)
{noformat}
> CLI fails on secured caches
> ---------------------------
>
> Key: ISPN-6261
> URL: https://issues.jboss.org/browse/ISPN-6261
> Project: Infinispan
> Issue Type: Bug
> Components: CLI, Security
> Reporter: Vojtech Juranek
> Assignee: Vojtech Juranek
>
> When running CLI on secured caches, it fails with exception bellow. This exception is quite confusing, as user has properly defined ADMIN permission on given cache. What is actually happening is that some operation, like statistics, called by CLI, iterates over all defined caches, including internal caches, and user hasn't required permission on all these caches (in this case on internal script cache)
> {noformat}
> 00:04:23,563 ERROR [org.jboss.as.controller.management-operation] (management-handler-thread - 1) WFLYCTL0013: Operation ("read-attribute") failed - address: ([
> ("subsystem" => "datagrid-infinispan"),
> ("cache-container" => "local")
> ]): java.lang.SecurityException: ISPN000287: Unauthorized access: subject 'Subject with principal(s): [user@ManagementRealm, admin@ManagementRealm, admin, org.jboss.remoting3.security.UserPrincipal@36ebcb, InetAddressPrincipal <127.0.0.1/127.0.0.1>, InetAddressPrincipal <127.0.0.1/127.0.0.1>]' lacks 'ADMIN' permission
> at org.infinispan.security.impl.AuthorizationHelper.checkPermission(AuthorizationHelper.java:86)
> at org.infinispan.security.impl.AuthorizationManagerImpl.checkPermission(AuthorizationManagerImpl.java:37)
> at org.infinispan.security.impl.SecureCacheImpl.getStats(SecureCacheImpl.java:567)
> at org.infinispan.stats.impl.CacheContainerStatsImpl.calculateAverageRemoveTime(CacheContainerStatsImpl.java:131)
> at org.infinispan.stats.impl.CacheContainerStatsImpl.getAverageRemoveTime(CacheContainerStatsImpl.java:121)
> at org.jboss.as.clustering.infinispan.subsystem.CacheContainerMetricsHandler.executeRuntimeStep(CacheContainerMetricsHandler.java:196)
> at org.jboss.as.controller.AbstractRuntimeOnlyHandler$1.execute(AbstractRuntimeOnlyHandler.java:53)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
8 years, 2 months
[JBoss JIRA] (ISPN-6261) CLI fails on secured caches
by Jakub Markos (JIRA)
[ https://issues.jboss.org/browse/ISPN-6261?page=com.atlassian.jira.plugin.... ]
Jakub Markos updated ISPN-6261:
-------------------------------
Description:
# When running CLI on secured caches, it fails with exception bellow. This exception is quite confusing, as user has properly defined ADMIN permission on given cache. What is actually happening is that some operation, like statistics, called by CLI, iterates over all defined caches, including internal caches, and user hasn't required permission on all these caches (in this case on internal script cache)
{noformat}
00:04:23,563 ERROR [org.jboss.as.controller.management-operation] (management-handler-thread - 1) WFLYCTL0013: Operation ("read-attribute") failed - address: ([
("subsystem" => "datagrid-infinispan"),
("cache-container" => "local")
]): java.lang.SecurityException: ISPN000287: Unauthorized access: subject 'Subject with principal(s): [user@ManagementRealm, admin@ManagementRealm, admin, org.jboss.remoting3.security.UserPrincipal@36ebcb, InetAddressPrincipal <127.0.0.1/127.0.0.1>, InetAddressPrincipal <127.0.0.1/127.0.0.1>]' lacks 'ADMIN' permission
at org.infinispan.security.impl.AuthorizationHelper.checkPermission(AuthorizationHelper.java:86)
at org.infinispan.security.impl.AuthorizationManagerImpl.checkPermission(AuthorizationManagerImpl.java:37)
at org.infinispan.security.impl.SecureCacheImpl.getStats(SecureCacheImpl.java:567)
at org.infinispan.stats.impl.CacheContainerStatsImpl.calculateAverageRemoveTime(CacheContainerStatsImpl.java:131)
at org.infinispan.stats.impl.CacheContainerStatsImpl.getAverageRemoveTime(CacheContainerStatsImpl.java:121)
at org.jboss.as.clustering.infinispan.subsystem.CacheContainerMetricsHandler.executeRuntimeStep(CacheContainerMetricsHandler.java:196)
at org.jboss.as.controller.AbstractRuntimeOnlyHandler$1.execute(AbstractRuntimeOnlyHandler.java:53)
{noformat}
was:
When running CLI on secured caches, it fails with exception bellow. This exception is quite confusing, as user has properly defined ADMIN permission on given cache. What is actually happening is that some operation, like statistics, called by CLI, iterates over all defined caches, including internal caches, and user hasn't required permission on all these caches (in this case on internal script cache)
{noformat}
00:04:23,563 ERROR [org.jboss.as.controller.management-operation] (management-handler-thread - 1) WFLYCTL0013: Operation ("read-attribute") failed - address: ([
("subsystem" => "datagrid-infinispan"),
("cache-container" => "local")
]): java.lang.SecurityException: ISPN000287: Unauthorized access: subject 'Subject with principal(s): [user@ManagementRealm, admin@ManagementRealm, admin, org.jboss.remoting3.security.UserPrincipal@36ebcb, InetAddressPrincipal <127.0.0.1/127.0.0.1>, InetAddressPrincipal <127.0.0.1/127.0.0.1>]' lacks 'ADMIN' permission
at org.infinispan.security.impl.AuthorizationHelper.checkPermission(AuthorizationHelper.java:86)
at org.infinispan.security.impl.AuthorizationManagerImpl.checkPermission(AuthorizationManagerImpl.java:37)
at org.infinispan.security.impl.SecureCacheImpl.getStats(SecureCacheImpl.java:567)
at org.infinispan.stats.impl.CacheContainerStatsImpl.calculateAverageRemoveTime(CacheContainerStatsImpl.java:131)
at org.infinispan.stats.impl.CacheContainerStatsImpl.getAverageRemoveTime(CacheContainerStatsImpl.java:121)
at org.jboss.as.clustering.infinispan.subsystem.CacheContainerMetricsHandler.executeRuntimeStep(CacheContainerMetricsHandler.java:196)
at org.jboss.as.controller.AbstractRuntimeOnlyHandler$1.execute(AbstractRuntimeOnlyHandler.java:53)
{noformat}
> CLI fails on secured caches
> ---------------------------
>
> Key: ISPN-6261
> URL: https://issues.jboss.org/browse/ISPN-6261
> Project: Infinispan
> Issue Type: Bug
> Components: CLI, Security
> Reporter: Vojtech Juranek
> Assignee: Vojtech Juranek
>
> # When running CLI on secured caches, it fails with exception bellow. This exception is quite confusing, as user has properly defined ADMIN permission on given cache. What is actually happening is that some operation, like statistics, called by CLI, iterates over all defined caches, including internal caches, and user hasn't required permission on all these caches (in this case on internal script cache)
> {noformat}
> 00:04:23,563 ERROR [org.jboss.as.controller.management-operation] (management-handler-thread - 1) WFLYCTL0013: Operation ("read-attribute") failed - address: ([
> ("subsystem" => "datagrid-infinispan"),
> ("cache-container" => "local")
> ]): java.lang.SecurityException: ISPN000287: Unauthorized access: subject 'Subject with principal(s): [user@ManagementRealm, admin@ManagementRealm, admin, org.jboss.remoting3.security.UserPrincipal@36ebcb, InetAddressPrincipal <127.0.0.1/127.0.0.1>, InetAddressPrincipal <127.0.0.1/127.0.0.1>]' lacks 'ADMIN' permission
> at org.infinispan.security.impl.AuthorizationHelper.checkPermission(AuthorizationHelper.java:86)
> at org.infinispan.security.impl.AuthorizationManagerImpl.checkPermission(AuthorizationManagerImpl.java:37)
> at org.infinispan.security.impl.SecureCacheImpl.getStats(SecureCacheImpl.java:567)
> at org.infinispan.stats.impl.CacheContainerStatsImpl.calculateAverageRemoveTime(CacheContainerStatsImpl.java:131)
> at org.infinispan.stats.impl.CacheContainerStatsImpl.getAverageRemoveTime(CacheContainerStatsImpl.java:121)
> at org.jboss.as.clustering.infinispan.subsystem.CacheContainerMetricsHandler.executeRuntimeStep(CacheContainerMetricsHandler.java:196)
> at org.jboss.as.controller.AbstractRuntimeOnlyHandler$1.execute(AbstractRuntimeOnlyHandler.java:53)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
8 years, 2 months
[JBoss JIRA] (ISPN-6279) Refactor CacheLoader API to make it thread safe
by Dan Berindei (JIRA)
[ https://issues.jboss.org/browse/ISPN-6279?page=com.atlassian.jira.plugin.... ]
Dan Berindei commented on ISPN-6279:
------------------------------------
While I don't object to having a "standard" wrapper over {{AdvancedCacheLoader.process()}} that collects all the keys, I don't think all users of {{process()}} should be forced to keep in memory a collection of all the store's keys.
> Refactor CacheLoader API to make it thread safe
> -----------------------------------------------
>
> Key: ISPN-6279
> URL: https://issues.jboss.org/browse/ISPN-6279
> Project: Infinispan
> Issue Type: Feature Request
> Components: Loaders and Stores
> Reporter: Sebastian Łaskawiec
> Priority: Minor
> Fix For: 9.0.0.Final
>
>
> h3. Introduction
> Currently {{CacheLoader}} thread safety can be easily compromised by broken implementations. We should provide a better implementation to make it more convenient and less error prone.
> h3. Problem statement
> CacheLoader API looks like this:
> {code}
> public interface AdvancedCacheLoader<K, V> extends CacheLoader<K, V> {
> ...
> void process(KeyFilter<? super K> filter, CacheLoaderTask<K, V> task, Executor executor, boolean fetchValue, boolean fetchMetadata);
> ...
> }
> {code}
> The CacheLoader implementation controls {{CacheLoaderTask}} as well as the number of threads used for processing (specified by {{Executor}}). If one wants to collect all keys (similarly to [AdvancedCacheLoader|https://github.com/infinispan/infinispan/blob/master/...]) it is very easy to hit concurrently issues while adding them to the set (See previous JIRAs: ISPN-3938 and ISPN-6276).
> h3. Solution
> The solution should take away currently control (Executor) and keyset (the simplest idea is to add it to {{CacheLoaderTask}}) from from the implementation. One of the ideas is to create a Lambda expression for pulling records from Cache Store. Note that similar changes will need to be performed to {{CacheWriter}} (to make it similar).
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
8 years, 2 months
[JBoss JIRA] (ISPN-6279) Refactor CacheLoader API to make it thread safe
by Dan Berindei (JIRA)
[ https://issues.jboss.org/browse/ISPN-6279?page=com.atlassian.jira.plugin.... ]
Dan Berindei commented on ISPN-6279:
------------------------------------
I think forcing the cache loader to implement the iteration of the keys via a lambda would only complicate the loader implementation, with no concrete advantage for the callers.
If a caller doesn't want to deal with concurrency issues, the solution is easy: just pass in a {{WithinThreadExecutor}} instance.
> Refactor CacheLoader API to make it thread safe
> -----------------------------------------------
>
> Key: ISPN-6279
> URL: https://issues.jboss.org/browse/ISPN-6279
> Project: Infinispan
> Issue Type: Feature Request
> Components: Loaders and Stores
> Reporter: Sebastian Łaskawiec
> Priority: Minor
> Fix For: 9.0.0.Final
>
>
> h3. Introduction
> Currently {{CacheLoader}} thread safety can be easily compromised by broken implementations. We should provide a better implementation to make it more convenient and less error prone.
> h3. Problem statement
> CacheLoader API looks like this:
> {code}
> public interface AdvancedCacheLoader<K, V> extends CacheLoader<K, V> {
> ...
> void process(KeyFilter<? super K> filter, CacheLoaderTask<K, V> task, Executor executor, boolean fetchValue, boolean fetchMetadata);
> ...
> }
> {code}
> The CacheLoader implementation controls {{CacheLoaderTask}} as well as the number of threads used for processing (specified by {{Executor}}). If one wants to collect all keys (similarly to [AdvancedCacheLoader|https://github.com/infinispan/infinispan/blob/master/...]) it is very easy to hit concurrently issues while adding them to the set (See previous JIRAs: ISPN-3938 and ISPN-6276).
> h3. Solution
> The solution should take away currently control (Executor) and keyset (the simplest idea is to add it to {{CacheLoaderTask}}) from from the implementation. One of the ideas is to create a Lambda expression for pulling records from Cache Store. Note that similar changes will need to be performed to {{CacheWriter}} (to make it similar).
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
8 years, 2 months
[JBoss JIRA] (ISPN-3938) AdvancedAsyncCacheLoader.process() concurrency issues
by Dan Berindei (JIRA)
[ https://issues.jboss.org/browse/ISPN-3938?page=com.atlassian.jira.plugin.... ]
Dan Berindei commented on ISPN-3938:
------------------------------------
Or we could say ISPN-6276 is a duplicate of this issue ;)
> AdvancedAsyncCacheLoader.process() concurrency issues
> -----------------------------------------------------
>
> Key: ISPN-3938
> URL: https://issues.jboss.org/browse/ISPN-3938
> Project: Infinispan
> Issue Type: Bug
> Components: Core, Loaders and Stores
> Affects Versions: 6.0.0.Final
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Fix For: 8.2.0.CR1
>
>
> {{AdvancedAsyncCacheLoader.process()}} calls {{advancedLoader().process()}} to collect all the keys in the store, but the HashSet used to collect the keys it not thread-safe. This can cause problems, e.g. during state transfer:
> {noformat}
> WARN cheTopologyControlCommand | ISPN000071: Caught exception when handling command CacheTopologyControlCommand{cache=sessions, type=CH_UPDATE, sender=alfie-lt-46127, joinInfo=null, topologyId=3, currentCH=DefaultConsistentHash{numSegments=60, numOwners=1, members=[alfie-lt-46127]}, pendingCH=null, throwable=null, viewId=1}java.util.ConcurrentModificationException
> at java.util.HashMap$HashIterator.nextEntry(HashMap.java:926)
> at java.util.HashMap$KeyIterator.next(HashMap.java:960)
> at org.infinispan.persistence.async.AdvancedAsyncCacheLoader.process(AdvancedAsyncCacheLoader.java:80)
> at org.infinispan.persistence.manager.PersistenceManagerImpl.processOnAllStores(PersistenceManagerImpl.java:414)
> at org.infinispan.statetransfer.StateConsumerImpl.invalidateSegments(StateConsumerImpl.java:910)
> at org.infinispan.statetransfer.StateConsumerImpl.onTopologyUpdate(StateConsumerImpl.java:393)
> at org.infinispan.statetransfer.StateTransferManagerImpl.doTopologyUpdate(StateTransferManagerImpl.java:178)
> at org.infinispan.statetransfer.StateTransferManagerImpl.access$000(StateTransferManagerImpl.java:38)
> at org.infinispan.statetransfer.StateTransferManagerImpl$1.updateConsistentHash(StateTransferManagerImpl.java:100)
> at org.infinispan.topology.LocalTopologyManagerImpl.handleConsistentHashUpdate(LocalTopologyManagerImpl.java:191)
> at org.infinispan.topology.CacheTopologyControlCommand.doPerform(CacheTopologyControlCommand.java:152)
> at org.infinispan.topology.CacheTopologyControlCommand.perform(CacheTopologyControlCommand.java:124)
> at org.infinispan.topology.ClusterTopologyManagerImpl$3.run(ClusterTopologyManagerImpl.java:606)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
8 years, 2 months