From ozizka at redhat.com Mon Jun 12 03:59:26 2017 From: ozizka at redhat.com (Ondrej Zizka) Date: Mon, 12 Jun 2017 09:59:26 +0200 Subject: [windup-dev] Fwd: Victims Java API, data, features In-Reply-To: <CA+Bbw+bLi6aZvFnSG-vke-GQRD_EBhnxSuMT9q0yABwPEtf1+w@mail.gmail.com> References: <31DC2180-FE64-442E-949E-BE47C8E46A9A@redhat.com> <562B98C5-ABCA-4103-B03E-AEB1DA45E068@redhat.com> <CAJ=9Mxc2dpY4PSYfyfnn7hMJn54rW7Svk-8P=5t-Bsm0USVG1Q@mail.gmail.com> <BCF6FE9C-79F3-4025-A5BC-4A36912D7286@redhat.com> <56D87A62.8040900@redhat.com> <56DD9FAE.5080702@redhat.com> <EE092AF5-48EE-4102-981A-16E5D505052F@redhat.com> <56DF8D51.2020303@redhat.com> <DF5E9929-C7E9-4FBC-8BB0-D04359CAE670@redhat.com> <56E03EBD.8040600@redhat.com> <CA+Bbw+Z9ai4Jp7+vWVWbvsuC+nsFRzPswTkL_P+fPhrq-BzjQg@mail.gmail.com> <56FD448F.1040902@redhat.com> <CA+Bbw+YeMdi2M3p5=Wqy4tqN+zdbRX8hYiOMH=OfSGD3KvpDpg@mail.gmail.com> <574DC549.6000107@redhat.com> <CA+Bbw+YWODYPjM=9BOXv7XU+NwQuuK6i_mwuyteusiAcG81v9Q@mail.gmail.com> <CAAAj4AjrGi=JgC9sMoxAT6zLY7xy0L-kkvr54EZHxTw+YY3p-A@mail.gmail.com> <CA+Bbw+bLi6aZvFnSG-vke-GQRD_EBhnxSuMT9q0yABwPEtf1+w@mail.gmail.com> Message-ID: <bd7e42f3-e050-0a99-9150-4b45137ea1c8@redhat.com> Hi Jason, Ondrej Zizka, Red Hat Migration Toolkit On 12.6.2017 01:30, Jason Shepherd wrote: > Hi Ondrej, > > Sorry for the late reply on this. Stephen and I have been discussing > the Victims project lately and I realised I hadn't forwarded his > feedback to him, so please see his replies to your feedback below. > > However I think we are going to refocus our efforts a bit on Victims. > OWASP dependency check has become a very popular project for > vulnerability tracking. It's being used by Fabric8 for vulnerability > scanning in Openshift.IO at the moment. The OWASP project is willing > to add Victims as a datasource, so I think we should focus our efforts > on that in order to get their features, and also have some influence > on the data in that tool. Good news! > > One thing that the community are asking for is a change in licence for > the victims-cve-db part of the project, so that's something that we'll > definitely we looking at. Some members have suggested a CC BY-SA > licence. What do you think of that? Read the discussion here: > > https://github.com/victims/victims-cve-db/issues/25 I don't know much about licenses, maybe Marek will be able to tell more. I remember Windup would have a problem with the Java client lib being licensed under AGPL. Eclipse license would fit. Regarding the db part and CC BY-SA, I guess someone (Tobias?) would have to consider. Ondra > > Regards, > Jason Shepherd > Product Security > > ---------- Forwarded message ---------- > From: *Stephen Milner* <smilner at redhat.com <mailto:smilner at redhat.com>> > Date: Sat, Jun 10, 2017 at 4:45 AM > Subject: Re: Victims Java API, data, features > To: Jason Shepherd <jshepher at redhat.com <mailto:jshepher at redhat.com>> > > > Replying back to you with details. In the response please do loop my > address :-) > > Inline ... > > On Fri, Jun 9, 2017 at 2:04 AM, Jason Shepherd <jshepher at redhat.com > <mailto:jshepher at redhat.com>> wrote: > > > > ---------- Forwarded message ---------- > > From: Ondrej Zizka <ozizka at redhat.com <mailto:ozizka at redhat.com>> > > Date: Wed, Jun 1, 2016 at 3:09 AM > > Subject: Victims Java API, data, features > > To: Jason Shepherd <jshepher at redhat.com > <mailto:jshepher at redhat.com>>, Windup-dev List > > <windup-dev at lists.jboss.org <mailto:windup-dev at lists.jboss.org>> > > > > > > > > > > > > Hi Jason, > > > > (I'm seding 2nd mail to start a new thread, please ignore the > previous one.) > > > > I have looked closer at V?ctims. > > I have few questions/issues. Could you please help resolving those? > > > > Note: I'm adding a PUBLIC mailing list, Windup developers. Feel free > to add > > some Victims list (is there one?) > > > > 1) Hashes are not real checksums > > As someone wrote in > https://github.com/victims/victims-cve-db/issues/45 > <https://github.com/victims/victims-cve-db/issues/45> > > the hashes used by Victims are not just SHA512 hashes of the file > content, > > but something else. > > I'd like to be able to either find CVE's by a normal file content > hash, or > > create the Victims hash. > > That's a fair request. For some background, the reason we recreate a > specific hash > is that different Java implementations create different bytecode, > resulting in different > hashes. Our hash creator strips out implementation specific items for > creating > and scanning. > > > a) Is there a Java impl? > > Client side there is via https://github.com/victims/victims-lib-java > <https://github.com/victims/victims-lib-java> > > > b) Could you add the plain SHA512 (or other, I'm okay with just > CRC32) hash > > to the data? > > We could do so. I assume this would be the SHA512 of the vulnerable > jar file. > > > 2) Victims Java client API > > > > The Java API doesn't match the needs much. > > From what I can see, it can > > a) Sync with the server > > b) Give me a list of CVE for given SHA512 hash. > > > > What I would like to have is: > > * Have some offline data distributed with our app, provide these data > > * Search the database by Maven coordinates, classes, > > * Get a short description of the CVE and date of appearance and > how/where it > > was fixed > > > > Is there a plan for extending the Java API? > > Also I guess not all these are covered in the Victims database, right? > > You're correct. There is a disconnect between the victims-cve-db and > the hash > database. Folks have been pretty great at submitting items to the > victims-cve-db > but we've gotten very little submissions for the hash db. Part of me > wonders > if it would be more beneficial to combine the two in the > victims-cve-db. Syncing > would then be a ``git pull`` rather than API call. It would also let > people do PR's > for data inclusion which may be more submitter friendly. Thoughts? > > > 3) Configuration > > The configuration is done through system properties, that's not too > > fortunate. > > For instance it doesn't allow to run multiple clients at once in the > same > > JVM. > > Could that be done through an API? > > I don't see why not. However, I think we would need some help to do that. > > > 4) Data structure > > The data structure of the JSON is not obvious. Is there some docs > for it? > > No, but there should be. Here is some pointers I threw together: > > https://gist.github.com/ashcrow/df238b1cc1e8a2f4bba94a6bb310805e > <https://gist.github.com/ashcrow/df238b1cc1e8a2f4bba94a6bb310805e> > > > 5) Data storage > > The data are only stored in a database over JDBC. Could it be simply > stored > > in a JSON or XML file? The file is just 165 KB and not growing too > fast, so > > I think rather than bringing an embedded DB as a dependency, I'd > prefer to > > process a XML file into a HashMap or a Lucene index and use that. > > I added a possible replacement at > https://gist.github.com/ashcrow/df238b1cc1e8a2f4bba94a6bb310805e > <https://gist.github.com/ashcrow/df238b1cc1e8a2f4bba94a6bb310805e>. > Essentially we'd move to a yaml format which is a combined version of > the victims-cve-db and the hash > database (which currently sits behind the api). Instead of syncing > with the API one would sync via git > and pull down the latest changes. PTAL and let me know what you think. > > > > On 4.4.2016 02:16, Jason Shepherd wrote: > >> > >> Hi Ondra, > >> > >> The architecture of Victims is such that you should never need to > >> 'download' the database. The client is designed to connect to the > >> central http://victi.ms API to get the latest vulnerabilities. > >> > >> That being said, the authors also have a 'backup' of the data in the > >> form of a Github repository, [1]. In fact some members of the > >> community have built a tool which just uses this repository, and does > >> not use the API at all. Recently we've built a tool to rebuild the > >> database from the Github repository, but it still needs some work, > >> [3]. > >> > >> [1] https://github.com/victims/victims-cve-db > <https://github.com/victims/victims-cve-db> > >> [2] https://github.com/h3xstream/maven-security-versions > <https://github.com/h3xstream/maven-security-versions> > >> [3] https://github.com/jasinner/victims-db-builder > <https://github.com/jasinner/victims-db-builder> > >> > >> Let me know if you need any further information. > >> Regards, > >> Jason Shepherd > >> > >> On Fri, Apr 1, 2016 at 1:38 AM, Ondrej Zizka <ozizka at redhat.com > <mailto:ozizka at redhat.com>> wrote: > >>> > >>> Great to know it goes on, last time I talked to someone (I think > djorm), > >>> he > >>> said the development was stagnant. > >>> > >>> Jason, is there a way to download a single big file with all data > in the > >>> database? > >>> > >>> Thanks, > >>> Ondra > > > > > > > > -- > Thanks, > Steve Milner > > Atomic | Red Hat | http://projectatomic.io/ | http://commissaire.io > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.jboss.org/pipermail/windup-dev/attachments/20170612/ecd0ddbb/attachment.html From mnovotny at redhat.com Mon Jun 12 07:04:49 2017 From: mnovotny at redhat.com (Marek Novotny) Date: Mon, 12 Jun 2017 13:04:49 +0200 Subject: [windup-dev] Fwd: Victims Java API, data, features In-Reply-To: <bd7e42f3-e050-0a99-9150-4b45137ea1c8@redhat.com> References: <31DC2180-FE64-442E-949E-BE47C8E46A9A@redhat.com> <562B98C5-ABCA-4103-B03E-AEB1DA45E068@redhat.com> <CAJ=9Mxc2dpY4PSYfyfnn7hMJn54rW7Svk-8P=5t-Bsm0USVG1Q@mail.gmail.com> <BCF6FE9C-79F3-4025-A5BC-4A36912D7286@redhat.com> <56D87A62.8040900@redhat.com> <56DD9FAE.5080702@redhat.com> <EE092AF5-48EE-4102-981A-16E5D505052F@redhat.com> <56DF8D51.2020303@redhat.com> <DF5E9929-C7E9-4FBC-8BB0-D04359CAE670@redhat.com> <56E03EBD.8040600@redhat.com> <CA+Bbw+Z9ai4Jp7+vWVWbvsuC+nsFRzPswTkL_P+fPhrq-BzjQg@mail.gmail.com> <56FD448F.1040902@redhat.com> <CA+Bbw+YeMdi2M3p5=Wqy4tqN+zdbRX8hYiOMH=OfSGD3KvpDpg@mail.gmail.com> <574DC549.6000107@redhat.com> <CA+Bbw+YWODYPjM=9BOXv7XU+NwQuuK6i_mwuyteusiAcG81v9Q@mail.gmail.com> <CAAAj4AjrGi=JgC9sMoxAT6zLY7xy0L-kkvr54EZHxTw+YY3p-A@mail.gmail.com> <CA+Bbw+bLi6aZvFnSG-vke-GQRD_EBhnxSuMT9q0yABwPEtf1+w@mail.gmail.com> <bd7e42f3-e050-0a99-9150-4b45137ea1c8@redhat.com> Message-ID: <01ee04d6-e0f6-675d-2cfe-71627f68a4c0@redhat.com> On 12.6.2017 09:59, Ondrej Zizka wrote: > Hi Jason, > ... >> >> One thing that the community are asking for is a change in licence for the victims-cve-db part of the project, so that's >> something that we'll definitely we looking at. Some members have suggested a CC BY-SA licence. What do you think of that? Read >> the discussion here: >> >> https://github.com/victims/victims-cve-db/issues/25 > I don't know much about licenses, maybe Marek will be able to tell more. I remember Windup would have a problem with the Java > client lib being licensed under AGPL. Eclipse license would fit. I think using the victims as a library under AGPL is fine, the concerns are related to data which could be problematic, but if there is a discussion to change it to CC, that would solves all concerns I guess. BTW I read https://eclipse.org/legal/eplfaq.php#3RDPARTY and there is not listed the AGPL and it is true we are not eclipse.org project, but that list gives us some imagination how it is handled in eclipse.org (we have an eclipse plugin which uses the same data or libraries like the CLI/Web console). Cheers, > Regarding the db part and CC BY-SA, I guess someone (Tobias?) would have to consider. > > Ondra >> >> Regards, >> Jason Shepherd >> Product Security >> -- Marek Novotny -- Windup team member Red Hat Czech s.r.o. Purkynova 99 612 45 Brno