[JBoss JIRA] (WFLY-9325) Reduce on-disk size and copying in the test suite
by Brian Stansberry (JIRA)
[ https://issues.jboss.org/browse/WFLY-9325?page=com.atlassian.jira.plugin.... ]
Brian Stansberry commented on WFLY-9325:
----------------------------------------
Yes, eliminating copying is a very valid goal for increasing overall CI perf.
> Reduce on-disk size and copying in the test suite
> -------------------------------------------------
>
> Key: WFLY-9325
> URL: https://issues.jboss.org/browse/WFLY-9325
> Project: WildFly
> Issue Type: Enhancement
> Components: Build System, Test Suite
> Reporter: David Lloyd
> Assignee: Tomaz Cerar
>
> The test suite takes up large amounts of disk space (over 2.7GB) during build. This causes overhead for copying and bogs down test runs and CI. Find ways to mitigate this.
> First approach is to use Maven refs instead of fixed JARs in the build process:
> {code:diff}
> diff --git a/pom.xml b/pom.xml
> index fb2b68c..5805471 100644
> --- a/pom.xml
> +++ b/pom.xml
> @@ -252,8 +252,8 @@
> <linkXRef>false</linkXRef>
>
> <server.output.dir.prefix>wildfly</server.output.dir.prefix>
> - <wildfly.build.output.dir>dist/target/${server.output.dir.prefix}-${jboss.as.release.version}</wildfly.build.output.dir>
> - <wildfly.web.build.output.dir>servlet-dist/target/${server.output.dir.prefix}-servlet-${jboss.as.release.version}</wildfly.web.build.output.dir>
> + <wildfly.build.output.dir>build/target/${server.output.dir.prefix}-${jboss.as.release.version}</wildfly.build.output.dir>
> + <wildfly.web.build.output.dir>servlet-build/target/${server.output.dir.prefix}-servlet-${jboss.as.release.version}</wildfly.web.build.output.dir>
>
> <!--
> See ChildFirstClassLoaderBuilder in model-test for the explanation of the org.jboss.model.test.cache.root and org.jboss.model.test.classpath.cache properties.
> {code}
> This saves a bit of space but copied images are still taking upwards of 55MB each, and there are very many copies; consider this example from {{testsuite/integration/iiop/target}}:
> {noformat}
> 30M jbossas
> 55M jbossas-iiop-client
> 55M jbossas-iiop-server
> 55M jbossas-iiop-ssl-client
> 55M jbossas-iiop-ssl-legacy-server
> 55M jbossas-iiop-ssl-server
> {noformat}
> Within a typical copy (after the above patch), about half of the data is in {{bin}} and half is in {{standalone}}. In {{bin}}, the dominating files are:
> * {{2.2M wildfly-elytron-tool.jar}}
> * {{7.2M client/jboss-cli-client.jar}}
> * {{17M client/jboss-client.jar}}
> In {{standalone}}, the dominator is:
> * {{25M data/activemq}} (mostly journal files but also the {{bindings}} file)
> Together these four areas cover 51.4MB of the 55MB in each IIOP server copy.
> In other areas, big users include {{standalone_xml_history}}, which can be as large as 67MB. Disabling the across the board could help.
> We're also making over 50 copies of the {{docs}} directory at 2.3M per instance.
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
8 years, 10 months
[JBoss JIRA] (JGRP-2218) New payload interface
by Dan Berindei (JIRA)
[ https://issues.jboss.org/browse/JGRP-2218?page=com.atlassian.jira.plugin.... ]
Dan Berindei commented on JGRP-2218:
------------------------------------
{quote}Hmm, I'm thinking I'd add an option to collapse all individual payloads (CompositePayload) into a single payload, but this would just mean a single byte[] array, but still a CompositePayload but all payloads pointing to the same byte[] array.{quote}
Why not go all the way and give the application a single {{ByteArrayPayload}} (or a {{NioDirectPayload}}, if the transport (e.g. {{TCP_NIO2}}) prefers to read into a direct buffer?
In Infinispan we only need to allocate a custom off-heap payload if the operation is a write and the cache is configured to use off-heap, otherwise we deserialize the command/response and discard the payload immediately. But making {{TCP_NIO2}} read into our off-heap memory directly probably requires access to some private APIs that aren't accessible in Java 9 any more, so we can't avoid the copy there.
{quote}IIRC you do implemnt Marshaller.estimatedSize(Object), don't you?{quote}
Like the name says, it's an estimate :) Also it's based on the previous objects of the same type that we serialized before, so the estimation can be wildly different from the current object's size.
> New payload interface
> ---------------------
>
> Key: JGRP-2218
> URL: https://issues.jboss.org/browse/JGRP-2218
> Project: JGroups
> Issue Type: Feature Request
> Reporter: Bela Ban
> Assignee: Bela Ban
> Fix For: 5.0
>
>
> h3. Goal
> Change payload in {{Message}} from byte[] arrays to a {{Payload}} interface which can have multiple implementations.
> h3. Reason
> Currently, having to pass a byte[] array to a message leads to unnecessary copying:
> * When an application has a ref to an NIO (direct) {{ByteBuffer}}, the bytes in the byte buffer have to be copied into a byte[] array and then set in the message
> * When the application sends around byte[] arrays, but also wants to add some additional metadata, e.g. type (1000-byte requests/responses), it needs to create a new byte[] array of (say) 1001 bytes and copy the data (1000 bytes) plus the request type (1 byte) into the new copy. Example: {{MPerf}} and {{UPerf}}
> * When an object has to be sent (e.g. in Infinispan), the object has to be marshalled into a byte[] array (first allocation) and then added to the message. With the suggested {{ObjectPayload}} (below), marshalling of the object would occur late, and it would be marshalled directly into the output stream of the bundler, eliminating the byte[] array allocation made by the application.
> h3. Design
> Instead of copying, the application creates an instance of {{Payload}} and sets the payload in {{Message}}. The {{Payload}} is then passed all the way down into the transport where it is marshalled and sent. There can be a number of payload implementations, e.g.
> * {{ByteArrayPayload}}: wraps a byte[] array with an offset and length
> * {{NioDirectPayload}}: wraps an NIO direct {{ByteBuffer}}
> * {{NioHeapPayload}}: wraps an NIO heap-based {{ByteBuffer}}
> * {{CompositePayload}}: wraps multiple Buffers. E.g. type (1 byte) and data (1000 bytes) as described above
> * {{IntPayload}}: a single integer
> * {{ObjectPayload}}: has an Object and a ClassLoader (for reading), plus a Marshaller which know how to marshal the object, this allows for objects to be passed in payloads and they're only marshalled at the end (transport).
> * {{PartialPayload}}: a ref to a {{Payload}}, with an offset and length
> The {{Payload}} interface has methods:
> * {{size()}}
> * {{writeTo(DataOutput)}}
> * {{readFrom(DataInput)}}
> * {{getInput()}}: this provides a {{DataInput}} stream for reading from the underlying payload
> and possibly also
> * {{acquire()}} and
> * {{release()}} (for ref-counting)
> * {{copy()}}
> Each payload impl has an ID and it should be possible to register new impls. A {{PayloadFactory}} maintains a mapping between IDs and impl classes.
> When marshalling a {{Payload}}, the ID is written first, followed by the payload's {{writeTo()}} method. When reading payloads, the {{PayloadFactory}} is used to create instances from IDs.
> h4. Fragmentation
> When fragmenting a buffer, the fragments are instances of {{PartialPayload}} which maintains an offset and length over an underlying payload. When marshalling a {{PartialPayload}}, only the part between offset and offset+length is written to the output stream.
> h4. Reference counting
> If we implement ref-counting, then payloads can be reused as soon as the ref-count is 0. For example, when sending a message, the payload's ref-count could be incremented by the app calling {{acquire()}}. (Assuming the message is a unicast message), {{UNICAST3}} would increment the count to 2. This is needed because {{UNICAST3}} might have to retransmit the message if it was lost on the network, and meanwhile the payload cannot be reused (changed). The app calls {{release()}} when the {{JChannel.send()}} call returns, but the payload cannot be reused until {{UNICAST3}} calls {{release()}} as well. This will happen when an {{ACK}} for the given message has been received.
> h4. Payload factory
> When a request is received, the buffer is created from the bytes received on the network, based on the ID. This should be done by asking a {{PayloadFactory}} component for a new buffer. A naive implementation might create a new buffer every time, and more sophisticated one might use a pool of payloads.
> The {{PayloadFactory}} instance could be replaced by one's own implementation; this allows for an application to control the lifecycle of payloads: thus the creation of buffers by the application and of payloads received over the network can be controlled by the same payload management impl.
> h4. Symmetry
> When sending a {{CompositePayload}} of a 500 byte {{ByteArrayPayload}} and a 1000 byte {{NioDirectPayload}}, would we want to also get the same {{CompositePayload}} consisting of 2 payloads on the receiver side, or would we want to combine the 2 payloads into one and make the 2 payloads refer to the same combined byte[] array (or NIO buffer)? Should this be made configurable?
> h4. Misc
> * Since this issue includes API changes, the version will be 5.0
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
8 years, 10 months
[JBoss JIRA] (DROOLS-1733) Enhance test coverage of FEEL built-in functions
by Tibor Zimányi (JIRA)
[ https://issues.jboss.org/browse/DROOLS-1733?page=com.atlassian.jira.plugi... ]
Tibor Zimányi updated DROOLS-1733:
----------------------------------
Description:
Majority of classes in module kie-dmn-feel in package org.kie.dmn.feel.runtime.functions can be tested by unit tests. Having such tests we can cover nearly 100% of these classes.
While coding these tests I found and fixed these bugs and problems:
* Rounding problem in EvalHelper.getBigDecimalOrNull() method - The method used .doubleValue() call which produced rounding errors.
* AnyFunction and AllFunction didn't check if the items in the list parameter are all Booleans (they must be according to the spec.)
* ConcatenateFunction didn't check for nulls in the list parameter
* EqualsFunction didn't compare BigDecimals with compareTo() method (which checks just values and ignores scale)
* MinFunction and MaxFunction didn't check if the items in the list parameter are comparable
* NumberFunction - split of one error message into two
* ReplaceFunction - NPE when having replacement param as null
* StringFunction and CodeFunction contained duplicated code
* SublistFunction - it was possible to input negative length as parameter
* ReplaceFunction now handles regexp flags
I will create a PR with new tests and fixes for mentioned things.
was:
Majority of classes in module kie-dmn-feel in package org.kie.dmn.feel.runtime.functions can be tested by unit tests. Having such tests we can cover nearly 100% of these classes.
While coding these tests I found and fixed these bugs and problems:
* Rounding problem in EvalHelper.getBigDecimalOrNull() method - The method used .doubleValue() call which produced rounding errors.
* AnyFunction and AllFunction didn't check if the items in the list parameter are all Booleans (they must be according to the spec.)
* ConcatenateFunction didn't check for nulls in the list parameter
* EqualsFunction didn't compare BigDecimals with compareTo() method (which checks just values and ignores scale)
* MinFunction and MaxFunction didn't check if the items in the list parameter are comparable
* NumberFunction - split of one error message into two
* ReplaceFunction - NPE when having replacement param as null
* StringFunction and CodeFunction contained duplicated code
* SublistFunction - it was possible to input negative length as parameter
I will create a PR with new tests and fixes for mentioned things.
> Enhance test coverage of FEEL built-in functions
> ------------------------------------------------
>
> Key: DROOLS-1733
> URL: https://issues.jboss.org/browse/DROOLS-1733
> Project: Drools
> Issue Type: Bug
> Components: dmn engine
> Affects Versions: 7.3.0.Final
> Reporter: Tibor Zimányi
> Assignee: Tibor Zimányi
> Labels: reported-by-qe
> Fix For: 7.3.1.Final
>
>
> Majority of classes in module kie-dmn-feel in package org.kie.dmn.feel.runtime.functions can be tested by unit tests. Having such tests we can cover nearly 100% of these classes.
> While coding these tests I found and fixed these bugs and problems:
> * Rounding problem in EvalHelper.getBigDecimalOrNull() method - The method used .doubleValue() call which produced rounding errors.
> * AnyFunction and AllFunction didn't check if the items in the list parameter are all Booleans (they must be according to the spec.)
> * ConcatenateFunction didn't check for nulls in the list parameter
> * EqualsFunction didn't compare BigDecimals with compareTo() method (which checks just values and ignores scale)
> * MinFunction and MaxFunction didn't check if the items in the list parameter are comparable
> * NumberFunction - split of one error message into two
> * ReplaceFunction - NPE when having replacement param as null
> * StringFunction and CodeFunction contained duplicated code
> * SublistFunction - it was possible to input negative length as parameter
> * ReplaceFunction now handles regexp flags
> I will create a PR with new tests and fixes for mentioned things.
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
8 years, 10 months
[JBoss JIRA] (DROOLS-1733) Enhance test coverage of FEEL built-in functions
by Tibor Zimányi (JIRA)
[ https://issues.jboss.org/browse/DROOLS-1733?page=com.atlassian.jira.plugi... ]
Tibor Zimányi updated DROOLS-1733:
----------------------------------
Description:
Majority of classes in module kie-dmn-feel in package org.kie.dmn.feel.runtime.functions can be tested by unit tests. Having such tests we can cover nearly 100% of these classes.
While coding these tests I found and fixed these bugs and problems:
* Rounding problem in EvalHelper.getBigDecimalOrNull() method - The method used .doubleValue() call which produced rounding errors.
* AnyFunction and AllFunction didn't check if the items in the list parameter are all Booleans (they must be according to the spec.)
* ConcatenateFunction didn't check for nulls in the list parameter
* EqualsFunction didn't compare BigDecimals with compareTo() method (which checks just values and ignores scale)
* MinFunction and MaxFunction didn't check if the items in the list parameter are comparable
* NumberFunction - split of one error message into two
* ReplaceFunction - NPE when having replacement param as null
* StringFunction and CodeFunction contained duplicated code
* SublistFunction - it was possible to input negative length as parameter
I will create a PR with new tests and fixes for mentioned things.
was:
Majority of classes in module kie-dmn-feel in package org.kie.dmn.feel.runtime.functions can be tested by unit tests. Having such tests we can cover nearly 100% of these classes.
While coding these tests I found and fixed these bugs and problems:
* Rounding problem in EvalHelper.getBigDecimalOrNull() method:
- The method used .doubleValue() call which produced rounding errors
* AnyFunction and AllFunction didn't check if the items in the list parameter are all Booleans (they must be according to the spec.)
* ConcatenateFunction didn't check for nulls in the list parameter
* EqualsFunction didn't compare BigDecimals with compareTo() method (which checks just values and ignores scale)
* MinFunction and MaxFunction didn't check if the items in the list parameter are comparable
* NumberFunction - split of one error message into two
* ReplaceFunction - NPE when having replacement param as null
* StringFunction and CodeFunction contained duplicated code
* SublistFunction - it was possible to input negative length as parameter
I will create a PR with new tests and fixes for mentioned things.
> Enhance test coverage of FEEL built-in functions
> ------------------------------------------------
>
> Key: DROOLS-1733
> URL: https://issues.jboss.org/browse/DROOLS-1733
> Project: Drools
> Issue Type: Bug
> Components: dmn engine
> Affects Versions: 7.3.0.Final
> Reporter: Tibor Zimányi
> Assignee: Tibor Zimányi
> Labels: reported-by-qe
> Fix For: 7.3.1.Final
>
>
> Majority of classes in module kie-dmn-feel in package org.kie.dmn.feel.runtime.functions can be tested by unit tests. Having such tests we can cover nearly 100% of these classes.
> While coding these tests I found and fixed these bugs and problems:
> * Rounding problem in EvalHelper.getBigDecimalOrNull() method - The method used .doubleValue() call which produced rounding errors.
> * AnyFunction and AllFunction didn't check if the items in the list parameter are all Booleans (they must be according to the spec.)
> * ConcatenateFunction didn't check for nulls in the list parameter
> * EqualsFunction didn't compare BigDecimals with compareTo() method (which checks just values and ignores scale)
> * MinFunction and MaxFunction didn't check if the items in the list parameter are comparable
> * NumberFunction - split of one error message into two
> * ReplaceFunction - NPE when having replacement param as null
> * StringFunction and CodeFunction contained duplicated code
> * SublistFunction - it was possible to input negative length as parameter
> I will create a PR with new tests and fixes for mentioned things.
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
8 years, 10 months
[JBoss JIRA] (DROOLS-1733) Enhance test coverage of FEEL built-in functions
by Tibor Zimányi (JIRA)
Tibor Zimányi created DROOLS-1733:
-------------------------------------
Summary: Enhance test coverage of FEEL built-in functions
Key: DROOLS-1733
URL: https://issues.jboss.org/browse/DROOLS-1733
Project: Drools
Issue Type: Bug
Components: dmn engine
Affects Versions: 7.3.0.Final
Reporter: Tibor Zimányi
Assignee: Tibor Zimányi
Fix For: 7.3.1.Final
Majority of classes in module kie-dmn-feel in package org.kie.dmn.feel.runtime.functions can be tested by unit tests. Having such tests we can cover nearly 100% of these classes.
While coding these tests I found and fixed these bugs and problems:
* Rounding problem in EvalHelper.getBigDecimalOrNull() method:
- The method used .doubleValue() call which produced rounding errors
* AnyFunction and AllFunction didn't check if the items in the list parameter are all Booleans (they must be according to the spec.)
* ConcatenateFunction didn't check for nulls in the list parameter
* EqualsFunction didn't compare BigDecimals with compareTo() method (which checks just values and ignores scale)
* MinFunction and MaxFunction didn't check if the items in the list parameter are comparable
* NumberFunction - split of one error message into two
* ReplaceFunction - NPE when having replacement param as null
* StringFunction and CodeFunction contained duplicated code
* SublistFunction - it was possible to input negative length as parameter
I will create a PR with new tests and fixes for mentioned things.
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
8 years, 10 months
[JBoss JIRA] (DROOLS-1728) There is always HTTP 404 Error Where I request the maven2 in workbench
by Mario Fusco (JIRA)
[ https://issues.jboss.org/browse/DROOLS-1728?page=com.atlassian.jira.plugi... ]
Mario Fusco resolved DROOLS-1728.
---------------------------------
Resolution: Rejected
LATEST is no longer supported by aether/maven ( see https://dev.eclipse.org/mhonarc/lists/aether-users/msg00529.html ) so it is no longer supported by kie-ci as well.
> There is always HTTP 404 Error Where I request the maven2 in workbench
> ----------------------------------------------------------------------
>
> Key: DROOLS-1728
> URL: https://issues.jboss.org/browse/DROOLS-1728
> Project: Drools
> Issue Type: Bug
> Components: tools
> Affects Versions: 7.3.0.Final
> Environment: CentOS 6.8
> Virtual Box 5.x
> Tomcat 8.5.20
> kie-drools-wb-7.3.0.Final-tomcat8.war
> MySQL 5.6.x
> Reporter: HuiYu Zhou
> Assignee: Mario Fusco
> Priority: Blocker
> Labels: reported-by-qe
>
> I want to try KieScanner for the remote maven repo. So I setup the workbench and create "Project1" as a sample, I also build and deploy it successfully. I can see the jar named "Project1-1.0.0.jar" with GAV "com.myteam:Project1:1.0.0" exist in “Admin” -> "Artifacts".
> So I add the repository into my development pom.xml and the jar as dependency, but the jar can't been downloaded by Maven Project.
> I also try to open the url: http://192.168.56.101:8080/kie-drools-wb/maven2, but there is always HTTP Status 404 – Not Found happened with description " The origin server did not find a current representation for the target resource or is not willing to disclose that one exists."
> The most funny thing is I can open the url: http://192.168.56.101:8080/kie-drools-wb/maven2/com/myteam/Project1/maven..., the content is shown in my webpage as following:
> This XML file does not appear to have any style information associated with it. The document tree is shown below.
> <metadata>
> <groupId>com.myteam</groupId>
> <artifactId>Project1</artifactId>
> <versioning>
> <release>1.0.0</release>
> <versions>
> <version>1.0.0</version>
> </versions>
> <lastUpdated>20170913142741</lastUpdated>
> </versioning>
> </metadata>
> As remote maven repo of workbench, why not like my private nexus maven that I can open the link even if it 's a folder?
>
> my pom.xml as following:
> <?xml version="1.0" encoding="UTF-8"?>
> <project xmlns="http://maven.apache.org/POM/4.0.0"
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
> xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
> <modelVersion>4.0.0</modelVersion>
> <groupId>drools</groupId>
> <artifactId>drools</artifactId>
> <version>1.0-SNAPSHOT</version>
> <properties>
> <drools.version>7.3.0.Final</drools.version>
> </properties>
> <repositories>
> <repository>
> <id>guvnor-m2-repo</id>
> <name>Guvnor M2 Repo</name>
> <url>http://192.168.56.101:8080/kie-drools-wb/maven2/</url>
> </repository>
> </repositories>
> <dependencies>
> <dependency>
> <groupId>com.myteam</groupId>
> <artifactId>Project1</artifactId>
> <version>1.0.0</version>
> </dependency>
> <!--drools dependencies -->
> </dependencies>
> </project>
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
8 years, 10 months
[JBoss JIRA] (DROOLS-1732) kie-maven-plugin: use correct plugin classloader
by Petr Široký (JIRA)
Petr Široký created DROOLS-1732:
-----------------------------------
Summary: kie-maven-plugin: use correct plugin classloader
Key: DROOLS-1732
URL: https://issues.jboss.org/browse/DROOLS-1732
Project: Drools
Issue Type: Enhancement
Components: tools
Affects Versions: 7.3.0.Final
Reporter: Petr Široký
Assignee: Petr Široký
{{kie-maven-plugin}} uses {{Thread.currentThread().getContextClassLoader()}} to get what it seems to be a plugin classloader (so that it has access to all the deps of the plugin). Based on a recent discussion on maven-dev list, this is not correct, and will likely break the plugin when maven 3.5.1 is released. The fix is to use {{this.getClass().getClassloader()}} to get to the plugin classloader.
See http://mail-archives.apache.org/mod_mbox/maven-dev/201709.mbox/browser for the discussion on the maven developer list.
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
8 years, 10 months
[JBoss JIRA] (ELY-1246) The elytron-1_0.xsd has elements the parser does not accept
by Ilia Vassilev (JIRA)
[ https://issues.jboss.org/browse/ELY-1246?page=com.atlassian.jira.plugin.s... ]
Ilia Vassilev reassigned ELY-1246:
----------------------------------
Assignee: Ilia Vassilev
> The elytron-1_0.xsd has elements the parser does not accept
> -----------------------------------------------------------
>
> Key: ELY-1246
> URL: https://issues.jboss.org/browse/ELY-1246
> Project: WildFly Elytron
> Issue Type: Bug
> Reporter: James Perkins
> Assignee: Ilia Vassilev
>
> There are elements in the {{elytron-1_0.xsd}} that the parser does not accept. While I didn't check them all here are at least a few that are in the XSD that the parser will reject:
> * {{allow-sasl-mechanisms}}
> * {{allow-all-sasl-mechanisms}}
> * {{forbid-sasl-mechanisms}}
> This may be all of them, but it should likely be validated that those are the only ones.
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
8 years, 10 months