Victims Java API, data, features
by Ondrej Zizka
Hi Jason,
(I'm seding 2nd mail to start a new thread, please ignore the previous one.)
I have looked closer at Víctims.
I have few questions/issues. Could you please help resolving those?
Note: I'm adding a PUBLIC mailing list, Windup developers. Feel free to
add some Victims list (is there one?)
1) Hashes are not real checksums
As someone wrote in https://github.com/victims/victims-cve-db/issues/45
the hashes used by Victims are not just SHA512 hashes of the file
content, but something else.
I'd like to be able to either find CVE's by a normal file content hash,
or create the Victims hash.
a) Is there a Java impl?
b) Could you add the plain SHA512 (or other, I'm okay with just CRC32)
hash to the data?
2) Victims Java client API
The Java API doesn't match the needs much.
From what I can see, it can
a) Sync with the server
b) Give me a list of CVE for given SHA512 hash.
What I would like to have is:
* Have some offline data distributed with our app, provide these data
* Search the database by Maven coordinates, classes,
* Get a short description of the CVE and date of appearance and
how/where it was fixed
Is there a plan for extending the Java API?
Also I guess not all these are covered in the Victims database, right?
3) Configuration
The configuration is done through system properties, that's not too
fortunate.
For instance it doesn't allow to run multiple clients at once in the
same JVM.
Could that be done through an API?
4) Data structure
The data structure of the JSON is not obvious. Is there some docs for it?
5) Data storage
The data are only stored in a database over JDBC. Could it be simply
stored in a JSON or XML file? The file is just 165 KB and not growing
too fast, so I think rather than bringing an embedded DB as a
dependency, I'd prefer to process a XML file into a HashMap or a Lucene
index and use that.
Thanks,
Ondra
On 4.4.2016 02:16, Jason Shepherd wrote:
> Hi Ondra,
>
> The architecture of Victims is such that you should never need to
> 'download' the database. The client is designed to connect to the
> central http://victi.ms API to get the latest vulnerabilities.
>
> That being said, the authors also have a 'backup' of the data in the
> form of a Github repository, [1]. In fact some members of the
> community have built a tool which just uses this repository, and does
> not use the API at all. Recently we've built a tool to rebuild the
> database from the Github repository, but it still needs some work,
> [3].
>
> [1] https://github.com/victims/victims-cve-db
> [2] https://github.com/h3xstream/maven-security-versions
> [3] https://github.com/jasinner/victims-db-builder
>
> Let me know if you need any further information.
> Regards,
> Jason Shepherd
>
> On Fri, Apr 1, 2016 at 1:38 AM, Ondrej Zizka <ozizka(a)redhat.com> wrote:
>> Great to know it goes on, last time I talked to someone (I think djorm), he
>> said the development was stagnant.
>>
>> Jason, is there a way to download a single big file with all data in the
>> database?
>>
>> Thanks,
>> Ondra
9 years, 4 months
Re: [windup-dev] [JBoss Migration Community of Practice] New message: "JBoss EAP Migration: discovering unsupported application libraries"
by Ondrej Zizka
Hi Jason,
I have looked closer at Víctims.
I have few questions/issues. Could you please help resolving those?
Note: I'm adding a PUBLIC mailing list, Windup developers. Feel free to
add some Victims list (is there one?)
1) Hashes are not real checksums
As someone wrote in https://github.com/victims/victims-cve-db/issues/45
the hashes used by Victims are not just SHA512 hashes of the file
content, but something else.
I'd like to be able to either find CVE's by a normal file content hash,
or create the Victims hash.
a) Is there a Java impl?
b) Could you add the plain SHA512 (or other, I'm okay with just CRC32)
hash to the data?
2) Victims Java client API
The Java API doesn't match the needs much.
From what I can see, it can
a) Sync with the server
b) Give me a list of CVE for given SHA512 hash.
What I would like to have is:
* Have some offline data distributed with our app, provide these data
* Search the database by Maven coordinates, classes,
* Get a short description of the CVE and date of appearance and
how/where it was fixed
Is there a plan for extending the Java API?
Also I guess not all these are covered in the Victims database, right?
3) Configuration
The configuration is done through system properties, that's not too
fortunate.
For instance it doesn't allow to run multiple clients at once in the
same JVM.
Could that be done through an API?
4) Data structure
The data structure of the JSON is not obvious. Is there some docs for it?
5) Data storage
The data are only stored in a database. Could it be simply stored in a
JSON or XML file? The file is just 165 KB and not growing too fast, so I
think rather than bringing an embedded DB as a dependency, I'd prefer to
process a XML file into a HashMap or a Lucene index and use that.
Thanks,
Ondra
On 4.4.2016 02:16, Jason Shepherd wrote:
> Hi Ondra,
>
> The architecture of Victims is such that you should never need to
> 'download' the database. The client is designed to connect to the
> central http://victi.ms API to get the latest vulnerabilities.
>
> That being said, the authors also have a 'backup' of the data in the
> form of a Github repository, [1]. In fact some members of the
> community have built a tool which just uses this repository, and does
> not use the API at all. Recently we've built a tool to rebuild the
> database from the Github repository, but it still needs some work,
> [3].
>
> [1] https://github.com/victims/victims-cve-db
> [2] https://github.com/h3xstream/maven-security-versions
> [3] https://github.com/jasinner/victims-db-builder
>
> Let me know if you need any further information.
> Regards,
> Jason Shepherd
>
> On Fri, Apr 1, 2016 at 1:38 AM, Ondrej Zizka <ozizka(a)redhat.com> wrote:
>> Great to know it goes on, last time I talked to someone (I think djorm), he
>> said the development was stagnant.
>>
>> Jason, is there a way to download a single big file with all data in the
>> database?
>>
>> Thanks,
>> Ondra
>>
>>
>>
>>
>> On 10.3.2016 03:49, Jason Shepherd wrote:
>>
>> Hello Rodney,
>>
>> The Product Security team are still maintaining that project. It's part of
>> our process to add new CVEs to the Victims Database when they are found.
>>
>> Also, if we've missed anything, you can add a library to the database
>> yourself using the web interface at https://victim.ms
>>
>> Regards,
>> Jason Shepherd
>>
>> On Thu, Mar 10, 2016 at 1:18 AM, Marek Novotny <mnovotny(a)redhat.com> wrote:
>>> Why do you think nobody picked it up?
>>> I can see there at least Stephen Milner, Jason Shepherd from RH. Are
>>> they responsible for the data? I added them to CC.
>>>
>>> Also even from non-redhatter member there is the latest commit from 4th
>>> January this year https://github.com/victims/victims-cve-db so it is not
>>> really dead ;)
>>>
>>>
>>> On 9.3.2016 16:07, Rodney Russ wrote:
>>>> So, interestingly enough, this project was started from the security
>>>> team within Red Hat. I believe the guy who started left Red Hat and no
>>>> one has picked it up since.
>>>>
>>>> On 8 Mar 2016, at 19:41, Ondrej Zizka wrote:
>>>>
>>>>> Right, I think the lack of fresh data is the issue with it. I think it
>>>>> needs someone at Red Hat adopting it and feeding with CVE's. Maybe GSS
>>>>> could take care? Try to push it higher, that's a company-wide thing.
>>>>>
>>>>> Ondra
>>>>>
>>>>>
>>>>> On 8.3.2016 00:06, Rodney Russ wrote:
>>>>>>
>>>>>> On 7 Mar 2016, at 8:35, Ondrej Zizka wrote:
>>>>>>
>>>>>>> We could use the same mechanism as with the Victims addon. Currently
>>>>>>> it's not in the distribution, though.
>>>>>> Was victims support something we wanted to move forward with? It
>>>>>> doesn't look as active as it once was.
>>>>>>
>>>>>>> Ondra
>>>>>>>
>>>>>>>
>>>>>>> On 3.3.2016 18:54, Jess Sightler wrote:
>>>>>>>> We have a condition that can add application level messages for
>>>>>>>> this. I think that would be better than a generic org.apache.axis
>>>>>>>> catchall rule. For example, I think Axis2 uses the same packages
>>>>>>>> but would not necessarily have the same issues.
>>>>>>>>
>>>>>>>> On 03/03/2016 03:38 AM, Rodney Russ wrote:
>>>>>>>>> I agree that for the specific issue of Axis discussed, a rule
>>>>>>>>> should be added to the catch-all rules. What I outlined below was
>>>>>>>>> an answer to the question posed by Benjamin:
>>>>>>>>>
>>>>>>>>> "Aside from updating Windup with a new rule to scan for Axis in
>>>>>>>>> particular, how can we discover these unsupported libraries up
>>>>>>>>> front on future applications?".
>>>>>>>>>
>>>>>>>>> -Rodney
>>>>>>>>>
>>>>>>>>> On 2 Mar 2016, at 18:05, Robb Greathouse wrote:
>>>>>>>>>
>>>>>>>>>> We should add AXIS to the blacklist. Do the packages in Axis have
>>>>>>>>>> a
>>>>>>>>>> signature (such as .axis.) that we could add to the Catch-All
>>>>>>>>>> rules?
>>>>>>>>>>
>>>>>>>>>> On Wed, Mar 2, 2016 at 4:12 PM, Rodney Russ <rruss(a)redhat.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> I believe the conclusion we came to on irc earlier today was
>>>>>>>>>>> that these
>>>>>>>>>>> types of situations are what we need the field to contribute
>>>>>>>>>>> back to the
>>>>>>>>>>> Windup project through:
>>>>>>>>>>>
>>>>>>>>>>> 1) creating rules themselves
>>>>>>>>>>> 2) creating a JIRA in the WINDUPRULE project
>>>>>>>>>>> 3) providing feedback through the link in the reports
>>>>>>>>>>>
>>>>>>>>>>> Does this seem like a reasonable approach? I'm not sure there
>>>>>>>>>>> is anything
>>>>>>>>>>> we can realistically do to identify all unsupported libraries
>>>>>>>>>>> unless
>>>>>>>>>>> someone is aware of a comprehensive list.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> -Rodney
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On 29 Feb 2016, at 0:50, Tobias Hartwig wrote:
>>>>>>>>>>>
>>>>>>>>>>> See below -
>>>>>>>>>>>> Begin forwarded message:
>>>>>>>>>>>>
>>>>>>>>>>>> From: Benjamin Meiseles <mojo-notify(a)redhat.com>
>>>>>>>>>>>>> Date: 26. Februar 2016 um 20:58:30 MEZ
>>>>>>>>>>>>> To: Tobias Hartwig <thartwig(a)redhat.com>
>>>>>>>>>>>>> Subject: [JBoss Migration Community of Practice] New message:
>>>>>>>>>>>>> "JBoss EAP
>>>>>>>>>>>>> Migration: discovering unsupported application libraries"
>>>>>>>>>>>>> Reply-To:
>>>>>>>>>>>>> jive-1245489557-6qh-2-kv7c(a)redhatinc.hosted.jivesoftware.com
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Mojo
>>>>>>>>>>>>>
>>>>>>>>>>>>> JBoss EAP Migration: discovering unsupported application
>>>>>>>>>>>>> libraries
>>>>>>>>>>>>> created by Benjamin Meiseles in JBoss Migration Community of
>>>>>>>>>>>>> Practice -
>>>>>>>>>>>>> View the full discussion
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hello all,
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On a migration project at The Hartford, Satish Irrinki, Guy
>>>>>>>>>>>>> Bianco and I
>>>>>>>>>>>>> ran into an issue in which our Support Relationship Manager
>>>>>>>>>>>>> luckily spotted
>>>>>>>>>>>>> our use of an unsupported library (Axis 1.4). We went through
>>>>>>>>>>>>> a minor
>>>>>>>>>>>>> ordeal breaking this news to the customer and working to
>>>>>>>>>>>>> maintain our
>>>>>>>>>>>>> project deadlines without putting the application at risk. We
>>>>>>>>>>>>> were under
>>>>>>>>>>>>> the impression that Windup would have flagged any unsupported
>>>>>>>>>>>>> libraries in
>>>>>>>>>>>>> the application, but there was no trace of Axis in the report,
>>>>>>>>>>>>> so this came
>>>>>>>>>>>>> as a surprise to us.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Aside from updating Windup with a new rule to scan for Axis in
>>>>>>>>>>>>> particular, how can we discover these unsupported libraries up
>>>>>>>>>>>>> front on
>>>>>>>>>>>>> future applications?
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Our initial thought is to couple our automated Windup analysis
>>>>>>>>>>>>> with a
>>>>>>>>>>>>> manual dive into the application, and ensure that all
>>>>>>>>>>>>> libraries are on the
>>>>>>>>>>>>> list of supported configurations. I am curious if anyone else
>>>>>>>>>>>>> can suggest
>>>>>>>>>>>>> alternate approaches.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Ben Meiseles
>>>>>>>>>>>>>
>>>>>>>>>>>>> Reply to this message by replying to this email, or go to the
>>>>>>>>>>>>> message on
>>>>>>>>>>>>> Mojo
>>>>>>>>>>>>> Start a new discussion in JBoss Migration Community of
>>>>>>>>>>>>> Practice by email
>>>>>>>>>>>>> or at Mojo
>>>>>>>>>>>>> Following JBoss Migration Community of Practice in these
>>>>>>>>>>>>> streams: Email
>>>>>>>>>>>>> Watches
>>>>>>>>>>>>> Put Mojo in your pocket! Get Jive for iOS or Jive for Android.
>>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Robb Greathouse
>>>>>>>>>> Middleware BU
>>>>>>>>>> 505-507-4906
>>>
>>> --
>>> Marek Novotny
>>> --
>>> Windup team member and Seam Project Lead
>>>
>>> Red Hat Czech s.r.o.
>>> Purkynova 99
>>> 612 45 Brno
>>
>>
>>
>> --
>> Regards,
>> Jason Shepherd
>> Product Security
>>
>>
>
>
9 years, 4 months
Docs for duplicated archives handling
by Ondrej Zizka
Hi Andrea,
there's a new feature - dupl archives handling.
Story 120 in Taiga
https://tree.taiga.io/project/rdruss-jboss-migration-windup-v3/us/120
PR: https://github.com/windup/windup/pull/936
There's no Jira so I am not sure where to put the documentation notes,
so at least letting you know this way.
It will need a note in the Report Index description like this:
For non-source mode (when scanning an app archives), the aggregated
issue counts and story points in tables and charts are taking duplicated
sub-archives into account, so that an application module which appears
in multiple applications or more than once in one application is only
counted once into the migration effort.
Ondra
9 years, 4 months
Fwd: [wildfly-dev] update on WildFly NoSQL prototype integration...
by Ondrej Zizka
-------- Forwarded Message --------
Subject: [wildfly-dev] update on WildFly NoSQL prototype integration...
Date: Wed, 4 May 2016 14:16:31 -0400
From: Scott Marlow <smarlow(a)redhat.com>
To: WildFly Dev List <wildfly-dev(a)lists.jboss.org>
Hi,
Below is an update on the WildFly NoSQL integration project. The goal
is for deployed applications to have access to NoSQL databases (via
Hibernate OGM or native APIs). Items 1-4, should be finished in our
first pass, with as much of the others items as we can do as well.
1. connection management will deal with obtaining NoSQL connections for
application use.
- borrow/share Hibernate OGM connection configuration setup code
- authentication integration
- support transport level security
2. CDI programming simplifications will make it easy to inject NoSQL
data into your application classes.
- https://github.com/antoinesd/javaee-nosql is initial idea
3. You will easily get a native NoSQL connection from the specified
NoSQL profile and use the native NOSQL (Cassandra/MongoDB/other) API in
your application.
4. You will also be able to easily use Hibernate OGM with the defined
NoSQL profiles (exactly how is TBD but will be awesome :-).
- Hibernate OGM static module is included.
- need to align with OGM dependencies (e.g. Hibernate ORM + other
dependencies).
- as mentioned above, OGM already has some connection setup code,
which might be good to share for WildFly + standalone NoSQL use.
- once WildFly has a common NoSQLSource (not a DataSource) that OGM
can use, OGM will be enhanced to use it.
5. How best for the WildFly NoSQL subsystem to be optional?
- Is it enough to not run the wildfly/testsuite/nosql tests by default?
- Or do we need to start a separate https://github.com/wildfly/nosql
project for the NoSQL subsystems?
6. transaction enlistment
7. compensating transactions
8. runtime application monitoring
9. How soon can we make an evaluation distribution available for use on
OpenStack/OpenShift?
- Would be great if we could do some load testing with all NoSQL
components.
- Would be great if we could enable others to also test.
10. Are there any problems with our WildFly NoSQL subsystem injecting
MongoDatabase connections via:
@Resource(lookup = "java:jboss/mongodb/test") MongoDatabase db;
- No @Resource support expected for standalone Java, TBD is whether a
runtime library can be used.
- Any problems expected on other EE application servers if this
approach becomes popular?
11. WIP topic branch is at
https://github.com/scottmarlow/wildfly/tree/nosql-dev9. Note that every
once in a while, commits are squashed and pushed to nosql-devN+1.
12. Add proper unit tests
- multi-threaded NoSQL access to show that works at all.
- use NoSQL from different EE components (e.g. JAX-RS).
- other use cases that represent how NoSQL could be used from WildFly.
Feedback/help is welcome!
Thanks,
Scott
_______________________________________________
wildfly-dev mailing list
wildfly-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/wildfly-dev
9 years, 4 months
Performance potential improvement - file pattern match, all paterns at once?
by Ondrej Zizka
Instead of scanning the files with every new rule with <filematches>, we
could collect the patterns upfront and then iterate over all files and
try all the patterns whose filename pattern matches for that file.
The idea is that the system could keep this file cached / open for all
passes.
WDYT? And aren't we already doing it? (I didn't check)
Ondra
9 years, 5 months
New fernflower fork project
by Jess Sightler
I have created a new project within Windup for maintaining a fork of
Fernflower with Java 7 compatibility and a Maven pom. Maybe we can start
releasing and keeping this up to date separately from windup core?
It is here:
https://github.com/windup/windup-fernflower
I think we can switch Windup to use this as part of 2.6.0, if noone
objects to that.
9 years, 5 months
