Re: [windup-dev] Fwd: Victims Java API, data, features

Hi Ondrej, Sorry for the late reply on this. Stephen and I have been discussing the Victims project lately and I realised I hadn't forwarded his feedback to him, so please see his replies to your feedback below. However I think we are going to refocus our efforts a bit on Victims. OWASP dependency check has become a very popular project for vulnerability tracking. It's being used by Fabric8 for vulnerability scanning in Openshift.IO at the moment. The OWASP project is willing to add Victims as a datasource, so I think we should focus our efforts on that in order to get their features, and also have some influence on the data in that tool.

One thing that the community are asking for is a change in licence for the victims-cve-db part of the project, so that's something that we'll definitely we looking at. Some members have suggested a CC BY-SA licence. What do you think of that? Read the discussion here: https://github.com/victims/victims-cve-db/issues/25

Regards, Jason Shepherd Product Security ---------- Forwarded message ---------- From: *Stephen Milner* <smilner(a)redhat.com <mailto:smilner@redhat.com>> Date: Sat, Jun 10, 2017 at 4:45 AM Subject: Re: Victims Java API, data, features To: Jason Shepherd <jshepher(a)redhat.com <mailto:jshepher@redhat.com>> Replying back to you with details. In the response please do loop my address :-) Inline ... On Fri, Jun 9, 2017 at 2:04 AM, Jason Shepherd <jshepher(a)redhat.com <mailto:jshepher@redhat.com>> wrote: > > ---------- Forwarded message ---------- > From: Ondrej Zizka <ozizka(a)redhat.com <mailto:ozizka@redhat.com>> > Date: Wed, Jun 1, 2016 at 3:09 AM > Subject: Victims Java API, data, features > To: Jason Shepherd <jshepher(a)redhat.com <mailto:jshepher@redhat.com>>, Windup-dev List > <windup-dev(a)lists.jboss.org <mailto:windup-dev@lists.jboss.org>> > > > > > > Hi Jason, > > (I'm seding 2nd mail to start a new thread, please ignore the previous one.) > > I have looked closer at Víctims. > I have few questions/issues. Could you please help resolving those? > > Note: I'm adding a PUBLIC mailing list, Windup developers. Feel free to add > some Victims list (is there one?) > > 1) Hashes are not real checksums > As someone wrote in https://github.com/victims/victims-cve-db/issues/45 <https://github.com/victims/victims-cve-db/issues/45> > the hashes used by Victims are not just SHA512 hashes of the file content, > but something else. > I'd like to be able to either find CVE's by a normal file content hash, or > create the Victims hash. That's a fair request. For some background, the reason we recreate a specific hash is that different Java implementations create different bytecode, resulting in different hashes. Our hash creator strips out implementation specific items for creating and scanning. > a) Is there a Java impl? Client side there is via https://github.com/victims/victims-lib-java <https://github.com/victims/victims-lib-java> > b) Could you add the plain SHA512 (or other, I'm okay with just CRC32) hash > to the data? We could do so. I assume this would be the SHA512 of the vulnerable jar file. > 2) Victims Java client API > > The Java API doesn't match the needs much. > From what I can see, it can > a) Sync with the server > b) Give me a list of CVE for given SHA512 hash. > > What I would like to have is: > * Have some offline data distributed with our app, provide these data > * Search the database by Maven coordinates, classes, > * Get a short description of the CVE and date of appearance and how/where it > was fixed > > Is there a plan for extending the Java API? > Also I guess not all these are covered in the Victims database, right? You're correct. There is a disconnect between the victims-cve-db and the hash database. Folks have been pretty great at submitting items to the victims-cve-db but we've gotten very little submissions for the hash db. Part of me wonders if it would be more beneficial to combine the two in the victims-cve-db. Syncing would then be a ``git pull`` rather than API call. It would also let people do PR's for data inclusion which may be more submitter friendly. Thoughts? > 3) Configuration > The configuration is done through system properties, that's not too > fortunate. > For instance it doesn't allow to run multiple clients at once in the same > JVM. > Could that be done through an API? I don't see why not. However, I think we would need some help to do that. > 4) Data structure > The data structure of the JSON is not obvious. Is there some docs for it? No, but there should be. Here is some pointers I threw together: https://gist.github.com/ashcrow/df238b1cc1e8a2f4bba94a6bb310805e <https://gist.github.com/ashcrow/df238b1cc1e8a2f4bba94a6bb310805e> > 5) Data storage > The data are only stored in a database over JDBC. Could it be simply stored > in a JSON or XML file? The file is just 165 KB and not growing too fast, so > I think rather than bringing an embedded DB as a dependency, I'd prefer to > process a XML file into a HashMap or a Lucene index and use that. I added a possible replacement at https://gist.github.com/ashcrow/df238b1cc1e8a2f4bba94a6bb310805e <https://gist.github.com/ashcrow/df238b1cc1e8a2f4bba94a6bb310805e>;. Essentially we'd move to a yaml format which is a combined version of the victims-cve-db and the hash database (which currently sits behind the api). Instead of syncing with the API one would sync via git and pull down the latest changes. PTAL and let me know what you think. > On 4.4.2016 02:16, Jason Shepherd wrote: >> >> Hi Ondra, >> >> The architecture of Victims is such that you should never need to >> 'download' the database. The client is designed to connect to the >> central http://victi.ms API to get the latest vulnerabilities. >> >> That being said, the authors also have a 'backup' of the data in the >> form of a Github repository, [1]. In fact some members of the >> community have built a tool which just uses this repository, and does >> not use the API at all. Recently we've built a tool to rebuild the >> database from the Github repository, but it still needs some work, >> [3]. >> >> [1] https://github.com/victims/victims-cve-db <https://github.com/victims/victims-cve-db> >> [2] https://github.com/h3xstream/maven-security-versions <https://github.com/h3xstream/maven-security-versions> >> [3] https://github.com/jasinner/victims-db-builder <https://github.com/jasinner/victims-db-builder> >> >> Let me know if you need any further information. >> Regards, >> Jason Shepherd >> >> On Fri, Apr 1, 2016 at 1:38 AM, Ondrej Zizka <ozizka(a)redhat.com <mailto:ozizka@redhat.com>> wrote: >>> >>> Great to know it goes on, last time I talked to someone (I think djorm), >>> he >>> said the development was stagnant. >>> >>> Jason, is there a way to download a single big file with all data in the >>> database? >>> >>> Thanks, >>> Ondra > > -- Thanks, Steve Milner Atomic | Red Hat | http://projectatomic.io/ | http://commissaire.io