From manderse at redhat.com Wed Feb 1 07:31:02 2017 From: manderse at redhat.com (Max Rydahl Andersen) Date: Wed, 01 Feb 2017 13:31:02 +0100 Subject: [forge-dev] Multitenancy for Forge as a Service In-Reply-To: References: Message-ID: > Hey Andrew, > > Yes, this is a really important question, and it's something that I > believe > that the Forge architecture supports out-of-the-box. Every interaction > with > the UI API (the Forge commands, per say) is stored in a request-scoped > UIContext[1] object that is initialized when a command is invoked and > destroyed when the command finishes executing. That means that you can > execute multiple concurrent commands in a single Forge instance and > they > will not affect each other. Excellent. > When it comes to security, as a good practice, it is important to > restrict > the commands that the exposed REST service can execute. > That's why in the Obsidian backend[2] we only expose a REST > endpoint[3] > consuming the Forge API to execute a limited number of commands (more > specifically, the ones exposed by the obsidian-addon[4]) > > Last but not least, as the Forge Online app is just a PoC illustrating > how > Forge can work in a web environment, it does not impose restrictions > to the > supported commands, therefore it shouldn't be a web app to be used in > production. Since we need something for production I would be curious to know what you feel are missing ? > PS: It would be nice to ping the Fabric8 team for this matter as they > are > already using Forge as their project generator solution for a long > time now. In case you missed it - fabric8 team is now part of devtools and devtools are fabric8 :) We'll get James et.al. involved (they are in f2f meetings this week), but as I understood Fabric8 have a fork of the PoC adjusted to handle some more complex workflows they were exploring. Things they would like to get merged back as I understand it. That said - to outline the items we were discussing about forge as a SaaS: Main reason we need it in first place is that Forge is *massive*, takes up a lot of resources - even more than Eclipse Che; thus we are very interested in being able to run Forge without having one instance per user. In that we had some questions and concerns in addition to the ones above: A) can forge run without access to download updates etc. from maven and similar ? i.e. can we lock updates down and ensure it and its execution cannot be altered just because someone updated a plugin. (We discussed firewalling it completely to be sure :) B) Is there a way to ensure that secrets like GitHub auth keys and others are not exposed to users nor in logs ? C) Any expectations/ideas on how we would integrate obsidian generators with fabric8 ? Two options we considered: 1) call out to separate hosted obsidian-toaster and run the risk of exposing secrets on wire. 2) run obsidian-toaster or a copy of it inside fabric8 online so we don't expose any secrets. Pete and others might have more, but above are the ones I wrote down while being on bluejeans to the conversations. > I hope this helps. Looking forward to this list of requirements and > happy > to discuss about this anytime. > > [1] - > https://github.com/forge/core/tree/master/ui#uicommand-execution-lifecycle > [2] - https://github.com/obsidian-toaster/generator-backend > [3] - > https://github.com/obsidian-toaster/generator-backend/blob/master/src/main/java/org/obsidiantoaster/generator/rest/ObsidianResource.java > [4] - https://github.com/obsidian-toaster/obsidian-addon > > Best Regards, > > *George Gastaldi* > > https://onename.com/gastaldi > > On Tue, Jan 31, 2017 at 9:27 PM, Andrew Lee Rubinger > wrote: > >> Hi, all: >> >> We have some requests coming down the pike from the Developer Group >> with >> regards to running Forge as a Service (which is great!). >> >> Additionally, I'm suspecting this will soon be very important for us >> as >> well, as we continue to build out what's currently known as the >> "Obsidian >> Generator", using Forge as a backend. >> >> I'm hoping the Forge team will kindly advise as to our preparedness >> for >> running Forge in a multitenant environment. Concerns may include >> resource >> consumption (not running an instance-per-user), security, etc. Have >> we yet >> broached this subject specifically in the prototypes I've seen thus >> far >> using Forge Online? >> >> I've requested of the Developer Group a list of requirements they've >> devised; will send those along here when I have them. In the >> meantime, >> mind describing where we currently stand? >> >> S, >> ALR >> >> -- >> Red Hat Developer Group Architecture >> @ALRubinger >> _______________________________________________ >> forge-dev mailing list >> forge-dev at lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/forge-dev >> > _______________________________________________ > forge-dev mailing list > forge-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/forge-dev /max http://about.me/maxandersen From ggastald at redhat.com Wed Feb 1 07:56:36 2017 From: ggastald at redhat.com (George Gastaldi) Date: Wed, 1 Feb 2017 10:56:36 -0200 Subject: [forge-dev] Multitenancy for Forge as a Service In-Reply-To: References: Message-ID: Hi Max, Thanks for your questions, I provided my answers inline. Best Regards, *George Gastaldi* https://onename.com/gastaldi On Wed, Feb 1, 2017 at 10:31 AM, Max Rydahl Andersen wrote: > > > Hey Andrew, > > > > Yes, this is a really important question, and it's something that I > > believe > > that the Forge architecture supports out-of-the-box. Every interaction > > with > > the UI API (the Forge commands, per say) is stored in a request-scoped > > UIContext[1] object that is initialized when a command is invoked and > > destroyed when the command finishes executing. That means that you can > > execute multiple concurrent commands in a single Forge instance and > > they > > will not affect each other. > > Excellent. > > > When it comes to security, as a good practice, it is important to > > restrict > > the commands that the exposed REST service can execute. > > That's why in the Obsidian backend[2] we only expose a REST > > endpoint[3] > > consuming the Forge API to execute a limited number of commands (more > > specifically, the ones exposed by the obsidian-addon[4]) > > > > Last but not least, as the Forge Online app is just a PoC illustrating > > how > > Forge can work in a web environment, it does not impose restrictions > > to the > > supported commands, therefore it shouldn't be a web app to be used in > > production. > > Since we need something for production I would be curious to know what > you feel are missing ? > I was referring to the UI offered by Forge Online app ( https://github.com/forge/forge-service/tree/master/web). The core libraries is already being used in the Obsidian Toaster and Fabric8 (not the same binaries/packages, but the code is similar) > > > PS: It would be nice to ping the Fabric8 team for this matter as they > > are > > already using Forge as their project generator solution for a long > > time now. > > In case you missed it - fabric8 team is now part of devtools and > devtools are fabric8 :) > > We'll get James et.al. involved (they are in f2f meetings this week), > but as I understood > Fabric8 have a fork of the PoC adjusted to handle some more complex > workflows they were > exploring. Things they would like to get merged back as I understand it. > > Good, yes, that's right > That said - to outline the items we were discussing about forge as a > SaaS: > > Main reason we need it in first place is that Forge is *massive*, takes > up a lot of > resources - even more than Eclipse Che; thus we are very interested in > being able to run > Forge without having one instance per user. > > I need some numbers here regarding the "a lot of resources - even more than Eclipse Che". Is that because it creates a Weld instance per addon that depends on the furnace-cdi container? Most of the core addons depend on the furnace-simple container, which does not boot Weld. Also, Weld is no longer heavy, you know :) > In that we had some questions and concerns in addition to the ones > above: > > A) can forge run without access to download updates etc. from maven and > similar ? > i.e. can we lock updates down and ensure it and its execution cannot > be altered > just because someone updated a plugin. > (We discussed firewalling it completely to be sure :) > > No updates are downloaded automatically in Forge. There are commands to install and update addons, but if you don't expose those commands that will never happen. Forge loads the addons on a specified directory. If nothing is deployed there (by any external process), it won't change anything In addition, you can specify a settings.xml with set to ensure that the Maven API won't try to look for any remote data. B) Is there a way to ensure that secrets like GitHub auth keys and > others are > not exposed to users nor in logs ? > Yes, you can use the Configuration addon for that. There is an example encrypting data here: https://github.com/forge/core/blob/master/database-tools/impl/src/main/java/org/jboss/forge/addon/database/tools/connections/ConnectionProfileManagerImpl.java#L82 (maybe that's not a good example, but you get the idea) Basically with this addon you'll have this information read from a properties file (the contents may be encrypted or not - that's up to you). If that's not enough, we can create a Forge addon or improve any of the core ones that does what you need. > C) Any expectations/ideas on how we would integrate obsidian generators > with > fabric8 ? > Two options we considered: > 1) call out to separate hosted obsidian-toaster and run the > risk of > exposing secrets on wire. > 2) run obsidian-toaster or a copy of it inside fabric8 online so > we don't expose any secrets. > The obsidian-toaster backend consists of a Forge addon + a JAX-RS resource. You could deploy this addon in the Fabric8 Forge service and invoke the UI that Fabric8 renders dynamically based on the command's metadata (it consumes the same JSON afaik) Or you could do #2. > > Pete and others might have more, but above are the ones I wrote down > while being on bluejeans to the conversations. > > > I hope this helps. Looking forward to this list of requirements and > > happy > > to discuss about this anytime. > > > > [1] - > > https://github.com/forge/core/tree/master/ui#uicommand- > execution-lifecycle > > [2] - https://github.com/obsidian-toaster/generator-backend > > [3] - > > https://github.com/obsidian-toaster/generator-backend/ > blob/master/src/main/java/org/obsidiantoaster/generator/ > rest/ObsidianResource.java > > [4] - https://github.com/obsidian-toaster/obsidian-addon > > > > Best Regards, > > > > *George Gastaldi* > > > > https://onename.com/gastaldi > > > > On Tue, Jan 31, 2017 at 9:27 PM, Andrew Lee Rubinger > > wrote: > > > >> Hi, all: > >> > >> We have some requests coming down the pike from the Developer Group > >> with > >> regards to running Forge as a Service (which is great!). > >> > >> Additionally, I'm suspecting this will soon be very important for us > >> as > >> well, as we continue to build out what's currently known as the > >> "Obsidian > >> Generator", using Forge as a backend. > >> > >> I'm hoping the Forge team will kindly advise as to our preparedness > >> for > >> running Forge in a multitenant environment. Concerns may include > >> resource > >> consumption (not running an instance-per-user), security, etc. Have > >> we yet > >> broached this subject specifically in the prototypes I've seen thus > >> far > >> using Forge Online? > >> > >> I've requested of the Developer Group a list of requirements they've > >> devised; will send those along here when I have them. In the > >> meantime, > >> mind describing where we currently stand? > >> > >> S, > >> ALR > >> > >> -- > >> Red Hat Developer Group Architecture > >> @ALRubinger > >> _______________________________________________ > >> forge-dev mailing list > >> forge-dev at lists.jboss.org > >> https://lists.jboss.org/mailman/listinfo/forge-dev > >> > > _______________________________________________ > > forge-dev mailing list > > forge-dev at lists.jboss.org > > https://lists.jboss.org/mailman/listinfo/forge-dev > > > /max > http://about.me/maxandersen > _______________________________________________ > forge-dev mailing list > forge-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/forge-dev > From pmuir at redhat.com Sat Feb 4 09:03:34 2017 From: pmuir at redhat.com (Pete Muir) Date: Sat, 4 Feb 2017 15:03:34 +0100 Subject: [forge-dev] Multitenancy for Forge as a Service In-Reply-To: References: Message-ID: On Wednesday, 1 February 2017, Max Rydahl Andersen wrote: > > > Hey Andrew, > > > > Yes, this is a really important question, and it's something that I > > believe > > that the Forge architecture supports out-of-the-box. Every interaction > > with > > the UI API (the Forge commands, per say) is stored in a request-scoped > > UIContext[1] object that is initialized when a command is invoked and > > destroyed when the command finishes executing. That means that you can > > execute multiple concurrent commands in a single Forge instance and > > they > > will not affect each other. > > Excellent. > My understanding of the flow then is something like this: 1) The user works through one or more screens, inputting fields that describe what they want 2) The web UI creates Forge commands from this, batching them all up to send in a single request to the server 3) On receiving commands the server executes them in Forge, generates a response, and then cleans up any resources generated on the server side In other words, Forge as a Service is not stateful in anyway, so each time you send a request, Forge must rebuild the project context. If so, this sounds good, as IIRC Forge can build project contexts relatively quickly. A couple of things to follow up on in this area: 1) How confident are you with the cleanup step - principally I'm wondering if you ensure a cleanup by using some sort of temporary file system whilst the request is in progress and then removing it once the request completes? Note that our backup thought here was just to cycle the container running Forge regularly so that we don't run risk running out of disk space, so this is not overly critical. 2) Actually this is related - how are you isolating projects on the server side, particularly with disk? > > > When it comes to security, as a good practice, it is important to > > restrict > > the commands that the exposed REST service can execute. > > That's why in the Obsidian backend[2] we only expose a REST > > endpoint[3] > > consuming the Forge API to execute a limited number of commands (more > > specifically, the ones exposed by the obsidian-addon[4]) > Very sensible :-) > > > > Last but not least, as the Forge Online app is just a PoC illustrating > > how > > Forge can work in a web environment, it does not impose restrictions > > to the > > supported commands, therefore it shouldn't be a web app to be used in > > production. > We wouldn't want to use this app directly anyway... > > Since we need something for production I would be curious to know what > you feel are missing ? > > > PS: It would be nice to ping the Fabric8 team for this matter as they > > are > > already using Forge as their project generator solution for a long > > time now. > > In case you missed it - fabric8 team is now part of devtools and > devtools are fabric8 :) > > We'll get James et.al. involved (they are in f2f meetings this week), > but as I understood > Fabric8 have a fork of the PoC adjusted to handle some more complex > workflows they were > exploring. Things they would like to get merged back as I understand it. > > That said - to outline the items we were discussing about forge as a > SaaS: > > Main reason we need it in first place is that Forge is *massive*, takes > up a lot of > resources - even more than Eclipse Che; thus we are very interested in > being able to run > Forge without having one instance per user. > > In that we had some questions and concerns in addition to the ones > above: > > A) can forge run without access to download updates etc. from maven and > similar ? > i.e. can we lock updates down and ensure it and its execution cannot > be altered > just because someone updated a plugin. > (We discussed firewalling it completely to be sure :) > > B) Is there a way to ensure that secrets like GitHub auth keys and > others are > not exposed to users nor in logs ? > > C) Any expectations/ideas on how we would integrate obsidian generators > with > fabric8 ? > Two options we considered: > 1) call out to separate hosted obsidian-toaster and run the > risk of > exposing secrets on wire. > 2) run obsidian-toaster or a copy of it inside fabric8 online so > we don't expose any secrets. > > Pete and others might have more, but above are the ones I wrote down > while being on bluejeans to the conversations. > > > I hope this helps. Looking forward to this list of requirements and > > happy > > to discuss about this anytime. > > > > [1] - > > https://github.com/forge/core/tree/master/ui#uicommand- > execution-lifecycle > > [2] - https://github.com/obsidian-toaster/generator-backend > > [3] - > > https://github.com/obsidian-toaster/generator-backend/ > blob/master/src/main/java/org/obsidiantoaster/generator/ > rest/ObsidianResource.java > > [4] - https://github.com/obsidian-toaster/obsidian-addon > > > > Best Regards, > > > > *George Gastaldi* > > > > https://onename.com/gastaldi > > > > On Tue, Jan 31, 2017 at 9:27 PM, Andrew Lee Rubinger > > > wrote: > > > >> Hi, all: > >> > >> We have some requests coming down the pike from the Developer Group > >> with > >> regards to running Forge as a Service (which is great!). > >> > >> Additionally, I'm suspecting this will soon be very important for us > >> as > >> well, as we continue to build out what's currently known as the > >> "Obsidian > >> Generator", using Forge as a backend. > >> > >> I'm hoping the Forge team will kindly advise as to our preparedness > >> for > >> running Forge in a multitenant environment. Concerns may include > >> resource > >> consumption (not running an instance-per-user), security, etc. Have > >> we yet > >> broached this subject specifically in the prototypes I've seen thus > >> far > >> using Forge Online? > >> > >> I've requested of the Developer Group a list of requirements they've > >> devised; will send those along here when I have them. In the > >> meantime, > >> mind describing where we currently stand? > >> > >> S, > >> ALR > >> > >> -- > >> Red Hat Developer Group Architecture > >> @ALRubinger > >> _______________________________________________ > >> forge-dev mailing list > >> forge-dev at lists.jboss.org > >> https://lists.jboss.org/mailman/listinfo/forge-dev > >> > > _______________________________________________ > > forge-dev mailing list > > forge-dev at lists.jboss.org > > https://lists.jboss.org/mailman/listinfo/forge-dev > > > /max > http://about.me/maxandersen > _______________________________________________ > forge-dev mailing list > forge-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/forge-dev > From pmuir at redhat.com Sat Feb 4 09:03:35 2017 From: pmuir at redhat.com (Pete Muir) Date: Sat, 4 Feb 2017 15:03:35 +0100 Subject: [forge-dev] Multitenancy for Forge as a Service In-Reply-To: References: Message-ID: On Wednesday, 1 February 2017, George Gastaldi wrote: > Hi Max, > > Thanks for your questions, I provided my answers inline. > > Best Regards, > > *George Gastaldi* > > https://onename.com/gastaldi > > On Wed, Feb 1, 2017 at 10:31 AM, Max Rydahl Andersen > > wrote: > > > > > > Hey Andrew, > > > > > > Yes, this is a really important question, and it's something that I > > > believe > > > that the Forge architecture supports out-of-the-box. Every interaction > > > with > > > the UI API (the Forge commands, per say) is stored in a request-scoped > > > UIContext[1] object that is initialized when a command is invoked and > > > destroyed when the command finishes executing. That means that you can > > > execute multiple concurrent commands in a single Forge instance and > > > they > > > will not affect each other. > > > > Excellent. > > > > > When it comes to security, as a good practice, it is important to > > > restrict > > > the commands that the exposed REST service can execute. > > > That's why in the Obsidian backend[2] we only expose a REST > > > endpoint[3] > > > consuming the Forge API to execute a limited number of commands (more > > > specifically, the ones exposed by the obsidian-addon[4]) > > > > > > Last but not least, as the Forge Online app is just a PoC illustrating > > > how > > > Forge can work in a web environment, it does not impose restrictions > > > to the > > > supported commands, therefore it shouldn't be a web app to be used in > > > production. > > > > Since we need something for production I would be curious to know what > > you feel are missing ? > > > > I was referring to the UI offered by Forge Online app ( > https://github.com/forge/forge-service/tree/master/web). The core > libraries > is already being used in the Obsidian Toaster and Fabric8 (not the same > binaries/packages, but the code is similar) > > > > > > PS: It would be nice to ping the Fabric8 team for this matter as they > > > are > > > already using Forge as their project generator solution for a long > > > time now. > > > > In case you missed it - fabric8 team is now part of devtools and > > devtools are fabric8 :) > > > > We'll get James et.al. involved (they are in f2f meetings this week), > > but as I understood > > Fabric8 have a fork of the PoC adjusted to handle some more complex > > workflows they were > > exploring. Things they would like to get merged back as I understand it. > > > > Good, yes, that's right > > > > That said - to outline the items we were discussing about forge as a > > SaaS: > > > > Main reason we need it in first place is that Forge is *massive*, takes > > up a lot of > > resources - even more than Eclipse Che; thus we are very interested in > > being able to run > > Forge without having one instance per user. > > > > I need some numbers here regarding the "a lot of resources - even more > than Eclipse Che". Is that because it creates a Weld instance per addon > that depends on the furnace-cdi container? > Most of the core addons depend on the furnace-simple container, which does > not boot Weld. > Also, Weld is no longer heavy, you know :) > It's better to think about this in terms of complexity. If we have to run 1 Forge per user, then even the memory overhead of the JVM is something we want to avoid, as then for every user we have, our resource costs increase. If we can share Forge among many users, we have a much better scale factor. > > > > In that we had some questions and concerns in addition to the ones > > above: > > > > A) can forge run without access to download updates etc. from maven and > > similar ? > > i.e. can we lock updates down and ensure it and its execution cannot > > be altered > > just because someone updated a plugin. > > (We discussed firewalling it completely to be sure :) > > > > No updates are downloaded automatically in Forge. There are commands to > install and update addons, but if you don't expose those commands that will > never happen. > Forge loads the addons on a specified directory. If nothing is deployed > there (by any external process), it won't change anything > In addition, you can specify a settings.xml with set to ensure > that the Maven API won't try to look for any remote data. > Great points. > > B) Is there a way to ensure that secrets like GitHub auth keys and > > others are > > not exposed to users nor in logs ? > > > > Yes, you can use the Configuration addon for that. There is an example > encrypting data here: > https://github.com/forge/core/blob/master/database-tools/ > impl/src/main/java/org/jboss/forge/addon/database/tools/connections/ > ConnectionProfileManagerImpl.java#L82 > (maybe that's not a good example, but you get the idea) > Basically with this addon you'll have this information read from a > properties file (the contents may be encrypted or not - that's up to you). > If that's not enough, we can create a Forge addon or improve any of the > core ones that does what you need. > We would need a way to pass the github token in the request that the user sends, as we will be holding one per user. > > > > C) Any expectations/ideas on how we would integrate obsidian generators > > with > > fabric8 ? > > Two options we considered: > > 1) call out to separate hosted obsidian-toaster and run the > > risk of > > exposing secrets on wire. > > 2) run obsidian-toaster or a copy of it inside fabric8 online so > > we don't expose any secrets. > > > > The obsidian-toaster backend consists of a Forge addon + a JAX-RS resource. > You could deploy this addon in the Fabric8 Forge service and invoke the UI > that Fabric8 renders dynamically based on the command's metadata (it > consumes the same JSON afaik) > I would strongly prefer (1) as it allows us to treat Forge-really-as-a-service ;-) > > Or you could do #2. > > > > > Pete and others might have more, but above are the ones I wrote down > > while being on bluejeans to the conversations. > > > > > I hope this helps. Looking forward to this list of requirements and > > > happy > > > to discuss about this anytime. > > > > > > [1] - > > > https://github.com/forge/core/tree/master/ui#uicommand- > > execution-lifecycle > > > [2] - https://github.com/obsidian-toaster/generator-backend > > > [3] - > > > https://github.com/obsidian-toaster/generator-backend/ > > blob/master/src/main/java/org/obsidiantoaster/generator/ > > rest/ObsidianResource.java > > > [4] - https://github.com/obsidian-toaster/obsidian-addon > > > > > > Best Regards, > > > > > > *George Gastaldi* > > > > > > https://onename.com/gastaldi > > > > > > On Tue, Jan 31, 2017 at 9:27 PM, Andrew Lee Rubinger > > > > wrote: > > > > > >> Hi, all: > > >> > > >> We have some requests coming down the pike from the Developer Group > > >> with > > >> regards to running Forge as a Service (which is great!). > > >> > > >> Additionally, I'm suspecting this will soon be very important for us > > >> as > > >> well, as we continue to build out what's currently known as the > > >> "Obsidian > > >> Generator", using Forge as a backend. > > >> > > >> I'm hoping the Forge team will kindly advise as to our preparedness > > >> for > > >> running Forge in a multitenant environment. Concerns may include > > >> resource > > >> consumption (not running an instance-per-user), security, etc. Have > > >> we yet > > >> broached this subject specifically in the prototypes I've seen thus > > >> far > > >> using Forge Online? > > >> > > >> I've requested of the Developer Group a list of requirements they've > > >> devised; will send those along here when I have them. In the > > >> meantime, > > >> mind describing where we currently stand? > > >> > > >> S, > > >> ALR > > >> > > >> -- > > >> Red Hat Developer Group Architecture > > >> @ALRubinger > > >> _______________________________________________ > > >> forge-dev mailing list > > >> forge-dev at lists.jboss.org > > >> https://lists.jboss.org/mailman/listinfo/forge-dev > > >> > > > _______________________________________________ > > > forge-dev mailing list > > > forge-dev at lists.jboss.org > > > https://lists.jboss.org/mailman/listinfo/forge-dev > > > > > > /max > > http://about.me/maxandersen > > _______________________________________________ > > forge-dev mailing list > > forge-dev at lists.jboss.org > > https://lists.jboss.org/mailman/listinfo/forge-dev > > > _______________________________________________ > forge-dev mailing list > forge-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/forge-dev > From ggastald at redhat.com Sat Feb 4 10:30:31 2017 From: ggastald at redhat.com (George Gastaldi) Date: Sat, 4 Feb 2017 13:30:31 -0200 Subject: [forge-dev] Multitenancy for Forge as a Service In-Reply-To: References: Message-ID: Hey Pete, The Obsidia Forge addon uses ProjectFactory.createTempProject (which in turn uses File.newTempFile) to create a temporary project. Files.delete is called once the flow is over (in a finally block) Em 4 de fev de 2017 12:03, "Pete Muir" escreveu: On Wednesday, 1 February 2017, Max Rydahl Andersen wrote: > > > Hey Andrew, > > > > Yes, this is a really important question, and it's something that I > > believe > > that the Forge architecture supports out-of-the-box. Every interaction > > with > > the UI API (the Forge commands, per say) is stored in a request-scoped > > UIContext[1] object that is initialized when a command is invoked and > > destroyed when the command finishes executing. That means that you can > > execute multiple concurrent commands in a single Forge instance and > > they > > will not affect each other. > > Excellent. > My understanding of the flow then is something like this: 1) The user works through one or more screens, inputting fields that describe what they want 2) The web UI creates Forge commands from this, batching them all up to send in a single request to the server 3) On receiving commands the server executes them in Forge, generates a response, and then cleans up any resources generated on the server side In other words, Forge as a Service is not stateful in anyway, so each time you send a request, Forge must rebuild the project context. If so, this sounds good, as IIRC Forge can build project contexts relatively quickly. A couple of things to follow up on in this area: 1) How confident are you with the cleanup step - principally I'm wondering if you ensure a cleanup by using some sort of temporary file system whilst the request is in progress and then removing it once the request completes? Note that our backup thought here was just to cycle the container running Forge regularly so that we don't run risk running out of disk space, so this is not overly critical. 2) Actually this is related - how are you isolating projects on the server side, particularly with disk? > > > When it comes to security, as a good practice, it is important to > > restrict > > the commands that the exposed REST service can execute. > > That's why in the Obsidian backend[2] we only expose a REST > > endpoint[3] > > consuming the Forge API to execute a limited number of commands (more > > specifically, the ones exposed by the obsidian-addon[4]) > Very sensible :-) > > > > Last but not least, as the Forge Online app is just a PoC illustrating > > how > > Forge can work in a web environment, it does not impose restrictions > > to the > > supported commands, therefore it shouldn't be a web app to be used in > > production. > We wouldn't want to use this app directly anyway... > > Since we need something for production I would be curious to know what > you feel are missing ? > > > PS: It would be nice to ping the Fabric8 team for this matter as they > > are > > already using Forge as their project generator solution for a long > > time now. > > In case you missed it - fabric8 team is now part of devtools and > devtools are fabric8 :) > > We'll get James et.al. involved (they are in f2f meetings this week), > but as I understood > Fabric8 have a fork of the PoC adjusted to handle some more complex > workflows they were > exploring. Things they would like to get merged back as I understand it. > > That said - to outline the items we were discussing about forge as a > SaaS: > > Main reason we need it in first place is that Forge is *massive*, takes > up a lot of > resources - even more than Eclipse Che; thus we are very interested in > being able to run > Forge without having one instance per user. > > In that we had some questions and concerns in addition to the ones > above: > > A) can forge run without access to download updates etc. from maven and > similar ? > i.e. can we lock updates down and ensure it and its execution cannot > be altered > just because someone updated a plugin. > (We discussed firewalling it completely to be sure :) > > B) Is there a way to ensure that secrets like GitHub auth keys and > others are > not exposed to users nor in logs ? > > C) Any expectations/ideas on how we would integrate obsidian generators > with > fabric8 ? > Two options we considered: > 1) call out to separate hosted obsidian-toaster and run the > risk of > exposing secrets on wire. > 2) run obsidian-toaster or a copy of it inside fabric8 online so > we don't expose any secrets. > > Pete and others might have more, but above are the ones I wrote down > while being on bluejeans to the conversations. > > > I hope this helps. Looking forward to this list of requirements and > > happy > > to discuss about this anytime. > > > > [1] - > > https://github.com/forge/core/tree/master/ui#uicommand- > execution-lifecycle > > [2] - https://github.com/obsidian-toaster/generator-backend > > [3] - > > https://github.com/obsidian-toaster/generator-backend/ > blob/master/src/main/java/org/obsidiantoaster/generator/ > rest/ObsidianResource.java > > [4] - https://github.com/obsidian-toaster/obsidian-addon > > > > Best Regards, > > > > *George Gastaldi* > > > > https://onename.com/gastaldi > > > > On Tue, Jan 31, 2017 at 9:27 PM, Andrew Lee Rubinger > > > wrote: > > > >> Hi, all: > >> > >> We have some requests coming down the pike from the Developer Group > >> with > >> regards to running Forge as a Service (which is great!). > >> > >> Additionally, I'm suspecting this will soon be very important for us > >> as > >> well, as we continue to build out what's currently known as the > >> "Obsidian > >> Generator", using Forge as a backend. > >> > >> I'm hoping the Forge team will kindly advise as to our preparedness > >> for > >> running Forge in a multitenant environment. Concerns may include > >> resource > >> consumption (not running an instance-per-user), security, etc. Have > >> we yet > >> broached this subject specifically in the prototypes I've seen thus > >> far > >> using Forge Online? > >> > >> I've requested of the Developer Group a list of requirements they've > >> devised; will send those along here when I have them. In the > >> meantime, > >> mind describing where we currently stand? > >> > >> S, > >> ALR > >> > >> -- > >> Red Hat Developer Group Architecture > >> @ALRubinger > >> _______________________________________________ > >> forge-dev mailing list > >> forge-dev at lists.jboss.org > >> https://lists.jboss.org/mailman/listinfo/forge-dev > >> > > _______________________________________________ > > forge-dev mailing list > > forge-dev at lists.jboss.org > > https://lists.jboss.org/mailman/listinfo/forge-dev > > > /max > http://about.me/maxandersen > _______________________________________________ > forge-dev mailing list > forge-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/forge-dev > _______________________________________________ forge-dev mailing list forge-dev at lists.jboss.org https://lists.jboss.org/mailman/listinfo/forge-dev From bartosz at redhat.com Wed Feb 15 07:38:27 2017 From: bartosz at redhat.com (Bartosz Majsak) Date: Wed, 15 Feb 2017 13:38:27 +0100 Subject: [forge-dev] Issues with shell tests in Arquillian Addon Message-ID: Hi, in Forge Arquillian Addon we are struggling with integration tests executed using `org.jboss.forge.addon.shell.test.ShellTest`. We've noticed that particular tests are "randomly" failing every n-th execution. Interestingly enough it?s almost always [`AddArquillianAlgeronPublisherTest`]( https://github.com/forge/addon-arquillian/blob/e6b6e905ec86ceef1c1a72862ded27e680039f71/src/test/java/test/integration/algeron/AddArquillianAlgeronPublisherTest.java) hanging on 2nd or 3rd command. Not sure what makes it so special, but at least it's a starting point for further investigation. Without a serious scientific method I can conclude it's almost 5/10 runs of the full build. We spent some time investigating without that much of luck, but what we have observed is that ```java public boolean isEnabled(UIContext context) { Boolean parent = super.isEnabled(context); if (parent) { return getSelectedProject(context).hasFacet(AlgeronConsumer.class); } return parent; } ``` is sometimes not recognizing that `parent` is a project. This might a be lead. Above snippet comes from [`AbstractAlgeronPublisherCommand`]( https://github.com/forge/addon-arquillian/blob/e6b6e905ec86ceef1c1a72862ded27e680039f71/src/main/java/org/jboss/forge/arquillian/command/algeron/AbstractAlgeronPublisherCommand.java#L64 ). Moreover we are explictly installing all the required facets, as we thought that could be the root cause. But it seems it's not the problem. Or maybe we are doing something wrong here as well? Any hints would be more than welcome :) This is really bothering us at the moment :\ Cheers, Bartosz. From bartosz at redhat.com Wed Feb 15 07:40:34 2017 From: bartosz at redhat.com (Bartosz Majsak) Date: Wed, 15 Feb 2017 13:40:34 +0100 Subject: [forge-dev] Issues with shell tests in Arquillian Addon In-Reply-To: References: Message-ID: *Ignore previous mail, here?s reformatted one.* Hi, in Forge Arquillian Addon we are struggling with integration tests executed using org.jboss.forge.addon.shell.test.ShellTest. We?ve noticed that particular tests are ?randomly? failing every n-th execution. Interestingly enough it?s almost always AddArquillianAlgeronPublisherTest hanging on 2nd or 3rd command. Not sure what makes it so special, but at least it?s a starting point for further investigation. Without a serious scientific method I can conclude it?s almost 5/10 runs of the full build. We spent some time investigating without that much of luck, but what we have observed is that public boolean isEnabled(UIContext context) { Boolean parent = super.isEnabled(context); if (parent) { return getSelectedProject(context).hasFacet(AlgeronConsumer.class); } return parent; } is sometimes not recognizing that parent is a project. This might a be lead. Above snippet comes from AbstractAlgeronPublisherCommand . Moreover we are explictly installing all the required facets, as we thought that could be the root cause. @Beforepublic void setUp() throws Exception { final AddonRegistry addonRegistry = Furnace.instance(getClass().getClassLoader()).getAddonRegistry(); projectFactory = addonRegistry.getServices(ProjectFactory.class).get(); shellTest = addonRegistry.getServices(ShellTest.class).get(); project = projectFactory.createTempProject(asList(JavaSourceFacet.class, ArquillianFacet.class, AlgeronProvider.class, AlgeronConsumer.class, AlgeronPublisherFacet.class)); shellTest.getShell().setCurrentResource(project.getRoot()); } But it seems it?s not the problem. Or maybe we are doing something wrong here as well? Any hints would be more than welcome :) This is really bothering us at the moment :\ Cheers, Bartosz. On Wed, Feb 15, 2017 at 1:38 PM, Bartosz Majsak wrote: Hi, > > in Forge Arquillian Addon we are struggling with integration tests > executed using `org.jboss.forge.addon.shell.test.ShellTest`. We've > noticed that particular tests are "randomly" failing every n-th execution. > Interestingly enough it?s almost always [`AddArquillianAlgeronPublisherT > est`](https://github.com/forge/addon-arquillian/blob/ > e6b6e905ec86ceef1c1a72862ded27e680039f71/src/test/java/test/ > integration/algeron/AddArquillianAlgeronPublisherTest.java) hanging on > 2nd or 3rd command. Not sure what makes it so special, but at least it's a > starting point for further investigation. Without a serious scientific > method I can conclude it's almost 5/10 runs of the full build. > > We spent some time investigating without that much of luck, but what we > have observed is that > > ```java > public boolean isEnabled(UIContext context) { > Boolean parent = super.isEnabled(context); > if (parent) { > return getSelectedProject(context).hasFacet(AlgeronConsumer. > class); > } > return parent; > } > ``` > is sometimes not recognizing that `parent` is a project. This might a be > lead. Above snippet comes from [`AbstractAlgeronPublisherCommand`]( > https://github.com/forge/addon-arquillian/blob/ > e6b6e905ec86ceef1c1a72862ded27e680039f71/src/main/java/org/ > jboss/forge/arquillian/command/algeron/AbstractAlgeronPublisherComman > d.java#L64). > > Moreover we are explictly installing all the required facets, as we > thought that could be the root cause. > > > But it seems it's not the problem. Or maybe we are doing something wrong > here as well? > > Any hints would be more than welcome :) This is really bothering us at the > moment :\ > > Cheers, > Bartosz. > > ? From rmpestano at gmail.com Sun Feb 26 13:01:03 2017 From: rmpestano at gmail.com (Rafael Pestano) Date: Sun, 26 Feb 2017 15:01:03 -0300 Subject: [forge-dev] hasFacet return false after instalation in Forge test Message-ID: Hi guys, I have a simple test which runs the setup command and then verifies if the facet was installed: @Test public void shouldGenerateSwaggerResources() throws Exception { Result result = shellTest.execute("swagger-setup", 25, TimeUnit.SECONDS); Assert.assertThat(result, not(instanceOf(Failed.class))); Assert.assertThat(result.getMessage(), is(equalTo("Swagger setup completed successfully!"))); Assert.assertTrue(project.hasFacet(SwaggerFacet.class)); the problem is that the assertion in returning false and test is failing. Note that the setup command runs with success and also I can confirm the facet is installed when I run the plugin on the IDE. here's the setup command: @Override public Result execute(UIExecutionContext context) throws Exception { Project project = getSelectedProject(context); boolean execute = true; if (project.hasFacet(SwaggerFacet.class) && project.getFacet(SwaggerFacet.class).isInstalled()) { execute = context.getPrompt().promptBoolean("Swagger plugin is already installed, override it?"); } if (execute) { swaggerConfiguration.setResourcesDir(resourcesDir.getValue()); SwaggerFacet facet = facetFactory.create(project, SwaggerFacet.class); facet.setConfiguration(swaggerConfiguration); facetFactory.install(project, facet); copySwaggerUIResources(facet); return Results.success("Swagger setup completed successfully!"); } else { return Results.success(); } } When I debug the test I can see the SwaggerFacet#isInstalled returns true after setup command. The code can be found here: https://github.com/rmpestano/swagger-addon Any help is appreciated. -- Att, Rafael M. Pestano Desenvolvedor Java Cia. de Processamento de Dados do Rio Grande do Sul http://rpestano.wordpress.com/ @realpestano From ggastald at redhat.com Sun Feb 26 13:10:40 2017 From: ggastald at redhat.com (George Gastaldi) Date: Sun, 26 Feb 2017 15:10:40 -0300 Subject: [forge-dev] hasFacet return false after instalation in Forge test In-Reply-To: References: Message-ID: That's because the original Project object remains unchanged and you need to refresh it. Try calling ProjectFactory.findProject again after your command is executed Em 26 de fev de 2017 15:04, "Rafael Pestano" escreveu: > Hi guys, > > I have a simple test which runs the setup command and then verifies if the > facet was installed: > > @Test > public void shouldGenerateSwaggerResources() throws Exception { > Result result = shellTest.execute("swagger-setup", 25, > TimeUnit.SECONDS); > Assert.assertThat(result, not(instanceOf(Failed.class))); > Assert.assertThat(result.getMessage(), is(equalTo("Swagger setup > completed successfully!"))); > Assert.assertTrue(project.hasFacet(SwaggerFacet.class)); > > > the problem is that the assertion in returning false and test is failing. > Note that the setup command runs with success and also I can confirm the > facet is installed when I run the plugin on the IDE. > > here's the setup command: > > @Override > > public Result execute(UIExecutionContext context) throws Exception { > Project project = getSelectedProject(context); > > boolean execute = true; > if (project.hasFacet(SwaggerFacet.class) && > project.getFacet(SwaggerFacet.class).isInstalled()) { > execute = context.getPrompt().promptBoolean("Swagger plugin is > already installed, override it?"); > } > > if (execute) { > swaggerConfiguration.setResourcesDir(resourcesDir.getValue()); > SwaggerFacet facet = facetFactory.create(project, > SwaggerFacet.class); > facet.setConfiguration(swaggerConfiguration); > facetFactory.install(project, facet); > copySwaggerUIResources(facet); > return Results.success("Swagger setup completed successfully!"); > } else { > return Results.success(); > } > > } > > > When I debug the test I can see the SwaggerFacet#isInstalled returns true > after setup command. > > The code can be found here: https://github.com/rmpestano/swagger-addon > > > Any help is appreciated. > > -- > Att, > > Rafael M. Pestano > > Desenvolvedor Java Cia. de Processamento de Dados do Rio Grande do Sul > http://rpestano.wordpress.com/ > @realpestano > _______________________________________________ > forge-dev mailing list > forge-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/forge-dev > From rmpestano at gmail.com Sun Feb 26 20:27:44 2017 From: rmpestano at gmail.com (Rafael Pestano) Date: Sun, 26 Feb 2017 22:27:44 -0300 Subject: [forge-dev] hasFacet return false after instalation in Forge test In-Reply-To: References: Message-ID: Yea! thank you very much George. I think that wasn't necessary in older forge versions, right? anyway, it's all green now ;) 2017-02-26 15:10 GMT-03:00 George Gastaldi : > That's because the original Project object remains unchanged and you need > to refresh it. Try calling ProjectFactory.findProject again after your > command is executed > > Em 26 de fev de 2017 15:04, "Rafael Pestano" > escreveu: > > > Hi guys, > > > > I have a simple test which runs the setup command and then verifies if > the > > facet was installed: > > > > @Test > > public void shouldGenerateSwaggerResources() throws Exception { > > Result result = shellTest.execute("swagger-setup", 25, > > TimeUnit.SECONDS); > > Assert.assertThat(result, not(instanceOf(Failed.class))); > > Assert.assertThat(result.getMessage(), is(equalTo("Swagger setup > > completed successfully!"))); > > Assert.assertTrue(project.hasFacet(SwaggerFacet.class)); > > > > > > the problem is that the assertion in returning false and test is failing. > > Note that the setup command runs with success and also I can confirm the > > facet is installed when I run the plugin on the IDE. > > > > here's the setup command: > > > > @Override > > > > public Result execute(UIExecutionContext context) throws Exception { > > Project project = getSelectedProject(context); > > > > boolean execute = true; > > if (project.hasFacet(SwaggerFacet.class) && > > project.getFacet(SwaggerFacet.class).isInstalled()) { > > execute = context.getPrompt().promptBoolean("Swagger plugin is > > already installed, override it?"); > > } > > > > if (execute) { > > swaggerConfiguration.setResourcesDir(resourcesDir.getValue()); > > SwaggerFacet facet = facetFactory.create(project, > > SwaggerFacet.class); > > facet.setConfiguration(swaggerConfiguration); > > facetFactory.install(project, facet); > > copySwaggerUIResources(facet); > > return Results.success("Swagger setup completed successfully!"); > > } else { > > return Results.success(); > > } > > > > } > > > > > > When I debug the test I can see the SwaggerFacet#isInstalled returns true > > after setup command. > > > > The code can be found here: https://github.com/rmpestano/swagger-addon > > > > > > Any help is appreciated. > > > > -- > > Att, > > > > Rafael M. Pestano > > > > Desenvolvedor Java Cia. de Processamento de Dados do Rio Grande do Sul > > http://rpestano.wordpress.com/ > > @realpestano > > _______________________________________________ > > forge-dev mailing list > > forge-dev at lists.jboss.org > > https://lists.jboss.org/mailman/listinfo/forge-dev > > > _______________________________________________ > forge-dev mailing list > forge-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/forge-dev > -- Att, Rafael M. Pestano Desenvolvedor Java Cia. de Processamento de Dados do Rio Grande do Sul http://rpestano.wordpress.com/ @realpestano From ggastald at redhat.com Sun Feb 26 20:38:17 2017 From: ggastald at redhat.com (George Gastaldi) Date: Sun, 26 Feb 2017 22:38:17 -0300 Subject: [forge-dev] hasFacet return false after instalation in Forge test In-Reply-To: References: Message-ID: Yes, that behavior exists since Forge 2.0.0.Final was released. Glad it works for you! Cheers! Em 26 de fev de 2017 22:36, "Rafael Pestano" escreveu: > Yea! thank you very much George. > > I think that wasn't necessary in older forge versions, right? anyway, it's > all green now ;) > > 2017-02-26 15:10 GMT-03:00 George Gastaldi : > > > That's because the original Project object remains unchanged and you need > > to refresh it. Try calling ProjectFactory.findProject again after your > > command is executed > > > > Em 26 de fev de 2017 15:04, "Rafael Pestano" > > escreveu: > > > > > Hi guys, > > > > > > I have a simple test which runs the setup command and then verifies if > > the > > > facet was installed: > > > > > > @Test > > > public void shouldGenerateSwaggerResources() throws Exception { > > > Result result = shellTest.execute("swagger-setup", 25, > > > TimeUnit.SECONDS); > > > Assert.assertThat(result, not(instanceOf(Failed.class))); > > > Assert.assertThat(result.getMessage(), is(equalTo("Swagger setup > > > completed successfully!"))); > > > Assert.assertTrue(project.hasFacet(SwaggerFacet.class)); > > > > > > > > > the problem is that the assertion in returning false and test is > failing. > > > Note that the setup command runs with success and also I can confirm > the > > > facet is installed when I run the plugin on the IDE. > > > > > > here's the setup command: > > > > > > @Override > > > > > > public Result execute(UIExecutionContext context) throws Exception { > > > Project project = getSelectedProject(context); > > > > > > boolean execute = true; > > > if (project.hasFacet(SwaggerFacet.class) && > > > project.getFacet(SwaggerFacet.class).isInstalled()) { > > > execute = context.getPrompt().promptBoolean("Swagger plugin is > > > already installed, override it?"); > > > } > > > > > > if (execute) { > > > swaggerConfiguration.setResourcesDir(resourcesDir.getValue()); > > > SwaggerFacet facet = facetFactory.create(project, > > > SwaggerFacet.class); > > > facet.setConfiguration(swaggerConfiguration); > > > facetFactory.install(project, facet); > > > copySwaggerUIResources(facet); > > > return Results.success("Swagger setup completed > successfully!"); > > > } else { > > > return Results.success(); > > > } > > > > > > } > > > > > > > > > When I debug the test I can see the SwaggerFacet#isInstalled returns > true > > > after setup command. > > > > > > The code can be found here: https://github.com/rmpestano/swagger-addon > > > > > > > > > Any help is appreciated. > > > > > > -- > > > Att, > > > > > > Rafael M. Pestano > > > > > > Desenvolvedor Java Cia. de Processamento de Dados do Rio Grande do Sul > > > http://rpestano.wordpress.com/ > > > @realpestano > > > _______________________________________________ > > > forge-dev mailing list > > > forge-dev at lists.jboss.org > > > https://lists.jboss.org/mailman/listinfo/forge-dev > > > > > _______________________________________________ > > forge-dev mailing list > > forge-dev at lists.jboss.org > > https://lists.jboss.org/mailman/listinfo/forge-dev > > > > > > -- > Att, > > Rafael M. Pestano > > Desenvolvedor Java Cia. de Processamento de Dados do Rio Grande do Sul > http://rpestano.wordpress.com/ > @realpestano > _______________________________________________ > forge-dev mailing list > forge-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/forge-dev >