[resteasy] Re: RestEasy 4.7.9 - java.lang.IllegalStateException: Connection pool shutdown

Hi James, Yes, you remember right. And that's what we do in every single IVR call before the very first API call when we create the RestClient instance for each call. (this is a part that I haven't touched at all) So the connection pool should be there and working OK for each new, separate call. That's why I don't understand what's happening. So this is what our code does in short at the moment with *v3.0.18* (I try to be short :) ) : Call arrives into the call center and going through routing scripts what forward the calls to our Tomcat application servers --> Call lands on an app server --> IVR app starts --> Wrapper bean created (all app calls are being called through the wrapper) --> Client created (before the very first API call)* --> First API call made --> API response closed after processed --> Wrapper shuts down client --> IVR app goes on until the next API call required --> Second API call made --> API response closed after processed --> Wrapper shuts down client --> IVR app goes on until the next API call required --> etc. etc ... --> when the customer hangs up or is being transferred, then the IVR app terminates. *And yes, I know that the creation part is being done only once in a call only at the beginning, but it is working without any issue! And not working with v4.7.9 anymore...but according to the documentation this is all right. (to be honest, it is logical if you ask me)* *Here comes the interesting part: as I remember I've tested a version with v.4.7.9 where I re-created the client again before every single API call if client.isClosed() gives back "true", and I got the same problem :( . I can do the same test again, just to confirm that re-creating the client does not help.* ------------------------------------------------------------- * the client creation part (this is from the code with, v4.7.9, but it is almost the same..the only difference is the ClientBuilder part what changed in v4.7.9) *PoolingHttpClientConnectionManager cm = new PoolingHttpClientConnectionManager();* *CloseableHttpClient httpClient = HttpClients.custom().setConnectionManager(cm).build();* *cm.setMaxTotal(200); * *cm.setDefaultMaxPerRoute(20); * *engine = new ApacheHttpClient43Engine(httpClient); * *init = true;* *reclient = ((ResteasyClientBuilder) ClientBuilder.newBuilder())* * .connectTimeout(Integer.parseInt(connectionTimeout), TimeUnit.MILLISECONDS)* * .readTimeout(Integer.parseInt(socketTimout), TimeUnit.MILLISECONDS).httpEngine(engine).build();* ------------------------------------------------------------- Regards, Zsolt James Perkins <jperkins(a)redhat.com&gt; ezt írta (időpont: 2023. aug. 16., Sze, 18:39): > Hello Zsolt, > You stated originally that you create a > custom PoolingHttpClientConnectionManager correct? The HTTP Client > documentation seems to indicate that closing the client also closes the > pool > https://hc.apache.org/httpcomponents-client-4.5.x/current/tutorial/html/c.... > This means if you'd have to create the client > and PoolingHttpClientConnectionManager after each close of the client. > > On Wed, Aug 16, 2023 at 12:45 AM Zsolt Melich <stocek86(a)gmail.com&gt; wrote: > >> Hi James, >> >> "*Do you use a CDI container by chance?*" >> No, I don't. >> >> "*However, the client should not be closed if it's going to be used >> again....*" >> Even if we create a new client instance in every single run of the >> deployed war application (a jsp+java based app served by Tomcat)? [that war >> file is using our rest client what is using RestEasy as a dependency] >> >> I believe it should not kill the whole underlying connection pool for >> Tomcat. This is not happening with the current version we are using >> (v3.0.18). >> We can create and close as many client instances as we want as long as >> we always keep in mind if we have closed the previous one or haven't. I >> know it is inefficient and expensive, but it was not written by me and >> that's what I wanted to fix originally. And that was what revealed the >> differences between v3.0.18 and v4.x.x. >> >> Or maybe the mechanism used by v3.0.18 is the faulty one? (so it is not >> OK that v3.0.18 is not killing the pool after a single client.close()) >> >> Best regards, >> Zsolt Melich >> >> >> James Perkins <jperkins(a)redhat.com&gt; ezt írta (időpont: 2023. aug. 9., >> Sze, 17:44): >> >>> >>> >>> On Wed, Aug 9, 2023 at 1:53 AM Zsolt Melich <stocek86(a)gmail.com&gt; wrote: >>> >>>> Hi James, >>>> >>>> Thanks for the info! >>>> >>>> Meanwhile, I've modified one of our test app (that calls all the APIs >>>> we can call in an IVR call) to check if the same happens. I can say, more >>>> or less, the same happens. >>>> >>>> If I close the client somewhere between two API calls, I'm not able to >>>> make any more successful API calls in the same run of the tool. (even if I >>>> create a completely new client instance for the next API call) >>>> >>> >>> If the client is closed, it definitely cannot be used anymore. Do you >>> use a CDI container by chance? There could be a different way to manage the >>> client resource is why I ask. >>> >>> >>>> The only difference I experienced is that I don't need to restart >>>> anything (~Tomcat service) to make it work again in the next run. It works >>>> again until the temporary line closes the client. >>>> It looks like you are right about the pool closure in your last >>>> sentence. >>>> >>>> The official documentation of this RestEasy release says that the >>>> socket closure is done in ApacheHttpClient43.finalize() under the hood, but >>>> I'm not sure if I can really trust in that. >>>> The affected IVR serves 20.000+ calls a day. A memory leak would not >>>> be good. >>>> >>> >>> That should only happen if the client wasn't closed by other means. I >>> wouldn't rely on the finalizer either TBH. Client management can >>> potentially be tricky. However, the client should not be closed if it's >>> going to be used again. In Jakarta REST 3.1, for Jakarta EE 10, the >>> jakarta.ws.rs.client.Client extends AutoCloseable so it can be used in >>> try-with-resources. There is definitely some overhead in creating a client, >>> but it's likely not that significant. However, sharing a client should work >>> fine as well. >>> >>> >>>> >>>> Maybe the same tests should be done with release 5.x.x. >>>> >>>> Regards, >>>> Zsolt Melich >>>> >>>> James Perkins <jperkins(a)redhat.com&gt; ezt írta (időpont: 2023. aug. 7., >>>> H, 16:59): >>>> >>>>> Hello, >>>>> Just an FYI. The 4.7 version is no longer maintained. If you're still >>>>> using Jakarta EE 8, you'd want to use the 5.0 versions. >>>>> >>>>> On Mon, Aug 7, 2023 at 5:01 AM Zsolt Melich <stocek86(a)gmail.com&gt; >>>>> wrote: >>>>> >>>>>> Hi all! >>>>>> >>>>>> I'm facing with the error message in the subject since I had to >>>>>> switch from RestEasy v3.0.18 (I know, quite old) to v4.7.9. >>>>>> We have a great, complex IVR application (written in Java 8), what >>>>>> is using RestEasy for the WS client we have for the API calls throughout a >>>>>> calls. The application is deployed to a Tomcat 9 application server. >>>>>> >>>>>> The current version of the client creates a RestEasy client instance >>>>>> for each API invocation, then close the response and the client (with the >>>>>> close() method). This is quite expensive, so I planned to change this when >>>>>> I switch to v4.7.9. >>>>>> In the new version I create the client at the beginning, closing the >>>>>> response (to let the resource back to the pool) after each invocation, and >>>>>> shut down the client with close() at the very end (only once). >>>>>> We create the PoolingHttpClientConnectionManager instance, the >>>>>> engine instance (ApacheHttpClient43Engine) and the httpClient instance >>>>>> manually, then we pass the engine to the ClientBuilder to create the client >>>>>> instance. >>>>>> >>>>>> The problem is that I can make only one IVR test call without any >>>>>> issue. At the very end of the first call, the connection is being closed. >>>>>> In the second call I got the exception from the subject for the very first >>>>>> API call. >>>>>> I checked the logs, traced my test calls and everything is looking >>>>>> fine (the app does not close the connection before the invocation). After a >>>>>> Tomcat restart I'm able to do a successful test call again, but only for >>>>>> one time! After a successful call, I'm getting the same exception in the >>>>>> next call... >>>>>> >>>>>> Does the close() method of the client instance destroy the whole >>>>>> connection pool for later calls as well when I close the connection at the >>>>>> end of the first call? Is this normal? >>>>>> We didn't have this issue with v.3.0.18. >>>>>> >>>>> >>>>> It shouldn't close the pool, no. If you close the client and attempt >>>>> to reuse the same connection pool, that would likely not work. I believe >>>>> internally when you close the client, the connection pool is also closed. >>>>> >>>>> >>>>>> >>>>>> P.S. : I know I haven't provided any exact code yet, but I try to >>>>>> clarify and understand the mechanics first. >>>>>> >>>>>> Thanks for the helping replies in advance! >>>>>> _______________________________________________ >>>>>> resteasy mailing list -- resteasy(a)lists.jboss.org >>>>>> To unsubscribe send an email to resteasy-leave(a)lists.jboss.org >>>>>> Privacy Statement: https://www.redhat.com/en/about/privacy-policy >>>>>> List Archives: >>>>>> https://lists.jboss.org/archives/list/resteasy@lists.jboss.org/message/WA... >>>>>> >>>>> >>>>> >>>>> -- >>>>> James R. Perkins >>>>> JBoss by Red Hat >>>>> >>>> >>> >>> -- >>> James R. Perkins >>> JBoss by Red Hat >>> >> > > -- > James R. Perkins > JBoss by Red Hat >