[jboss-jira] [JBoss JIRA] Commented: (JBMESSAGING-1428) JBM stops processing message due to switch to PULL ServerInvokerCallbackHandler

Ron Sigal wrote: > > > Tim Fox wrote: >> Ron Sigal wrote: >>> Hi Jiri, >>> >>> Yes, I think you've put your finger on it. Line >>> >>> 2008-09-24 05:25:54,111 DEBUG [Timer-10][org.jboss.remoting.ConnectionValidator] ConnectionValidator[SocketClientInvoker[e56328, bisocket://localhost:4457], pingPeriod=2000 ms]'s connections is invalid >>> >>> indicates that the ConnectionValidator ping has timed out, at which point it will >>> >>> 1. notify all registered listeners, and, by default, >>> 2. shut down the related LeasePinger, >>> >>> Once the LeasePinger shuts down, the following will happen on the server side: >>> >>> 1. the server side Lease will notify its registered listeners of a connection failure, and, by default, >>> 2. the ServerInvokerCallbackHandler will close its client. >>> >>> The latter is what turns push callbacks into apparent pull callbacks. >>> >>> Notice that the ConnectionValidator ping failure occurs about 1 second after the ping is sent. That's the default ping timeout, and it's too small, which is causing false timeout. I would suggest: >>> >>> 1. adding something like >>> >>> <attribute name="validatorPingPeriod" isParam="true">10000</attribute> >>> <attribute name="validatorPingTimeout" isParam="true">5000</attribute> >> Ok, now I confused (as ever). We set clientLeasePeriod, I thought that controlled the ping interval? >> >> What does this other ping config do, and how does it differ. >> >> Apologies, I know I have probably asked you this a million times, but I never have quite got it, and why more than one ping config is necessary ;) > > No problem. Starting over, I would probably just have one pinger, but, historically, we created two. > > 1. LeasePinger pings the server, and the corresponding Lease informs server side listeners of a failure. "clientLeasePeriod" sets the default ping period for LeasePinger. > > 2. ConnectionValidator pings the server and, if it doesn't get a timely answer, informs client side listeners of a failure. "validatorPingPeriod" and "validatorPingTimeout" configure ConnectionValidator. I was under the impression that the following: <attribute name="socket.check_connection" isParam="true">false</attribute> In our config, is supposed to turn the second pinger off. http://anonsvn.jboss.org/repos/messaging/tags/JBossMessaging_1_4_0_SP3_CP...

> > I guess it took a particularly heavy load to flush out the fact that the default ConnectionValidator timeout was too small. > > >>> >>> to remoting-bisocket-service.xml, which will extend the ping timeout and reduce the likelihood of false timeouts, and >>> >>> 2. like I mentioned before, add >>> >>> <attribute name="registerCallbackListener">false</attribute> >>> >>> (if Tim agrees). >>> >>> -Ron >>> >>> Jiri Pechanec wrote: >>>> Hi, >>>> >>>> I have checked the logs yet more (the last one I have sent) and I found this >>>> >>>> 2008-09-24 05:25:54,111 DEBUG [Timer-10][org.jboss.remoting.ConnectionValidator] ConnectionValidator[SocketClientInvoker[e56328, bisocket://localhost:4457], pingPeriod=2000 ms]'s connections is invalid >>>> 2008-09-24 05:25:54,298 TRACE [Timer-10][org.jboss.remoting.InvokerRegistry] destroying client invoker InvokerLocator [bisocket://localhost:4457/?JBM_clientMaxPoolSize=200&clientLeasePeriod=10000&clientSocketClass=org.jboss.jms.client.remoting.ClientSocketWrapper&dataType=jms&marshaller=org.jboss.jms.wireformat.JMSWireFormat&numberOfCallRetries=1&numberOfRetries=10&pingFrequency=214748364&pingWindowFactor=10&socket.check_connection=false&stopLeaseOnFailure=true&timeout=0&unmarshaller=org.jboss.jms.wireformat.JMSWireFormat], config {timeout=1000, NumberOfCallRetries=1, validatorPingPeriod=2000, connection_checker=true} >>>> 2008-09-24 05:25:54,298 DEBUG [Timer-10][org.jboss.remoting.InvokerRegistry] decremented SocketClientInvoker[e56328, bisocket://localhost:4457]'s count, current count 2 >>>> 2008-09-24 05:25:54,298 DEBUG [Timer-10][org.jboss.remoting.ConnectionValidator] ConnectionValidator[SocketClientInvoker[e56328, bisocket://localhost:4457], pingPeriod=2000 ms] stopped, returning true >>>> >>>> And during this operation the ClientInvoker on the server was destoryed >>>> 2008-09-24 05:25:54,304 TRACE [org.jboss.remoting.InvokerRegistry] destroying client invoker InvokerLocator [bisocket://10.18.0.216:655404877/callback?callbackServerHost=10.18.0.216&callbackServerPort=655404877&callbackServerProtocol=bisocket&clientMaxPoolSize=1&clientSocketClass=org.jboss.jms.client.remoting.ClientSocketWrapper&datatype=jms&guid=ai060-cbto0n-flhqxb53-1-flhr24dl-46&isCallbackServer=true&onewayThreadPool=org.jboss.jms.server.remoting.DirectThreadPool&serverSocketClass=org.jboss.jms.server.remoting.ServerSocketWrapper], config {socket.check_connection=false, dataType=jms, serverInvokerCallbackHandler=ServerInvokerCallbackHandler[ai060-cbto0n-flhqxb53-1-flhr24dc-44+ai060-cbto0n-flhqxb53-1-flhr24dw-47], datatype=jms, callbackServerProtocol=bisocket, callbackServerPort=655404877, pingWindowFactor=10, numberOfRetries=10, clientMaxPoolSize=1, JBM_clientMaxPoolSize=200, serverSocketClass=org.jboss.jms.server.remoting.ServerSocketWrapper, isCallbackServer=true, pingFrequency=214748364, timeout=10000, listenerId=ai060-cbto0n-flhqxb53-1-flhr24dw-47, marshaller=org.jboss.jms.wireformat.JMSWireFormat, callbackServerHost=10.18.0.216, serverBindPort=4457, stopLeaseOnFailure=true, serverBindAddress=localhost, callbackTimeout=10000, guid=ai060-cbto0n-flhqxb53-1-flhr24dl-46, onewayThreadPool=org.jboss.jms.server.remoting.DirectThreadPool, unmarshaller=org.jboss.jms.wireformat.JMSWireFormat, clientLeasePeriod=10000, serverInvoker=SocketServerInvoker[127.0.0.1:4457], numberOfCallRetries=1, clientSocketClass=org.jboss.jms.client.remoting.ClientSocketWrapper} >>>> 2008-09-24 05:25:54,304 DEBUG [org.jboss.remoting.InvokerRegistry] removed SocketClientInvoker[de8f62, bisocket://10.18.0.216:655404877] from registry >>>> 2008-09-24 05:25:54,304 TRACE [org.jboss.remoting.InvokerRegistry] disconnecting SocketClientInvoker[de8f62, bisocket://10.18.0.216:655404877] >>>> >>>> And changed from PUSH to PULL >>>> >>>> >>>> >>>> J. >>>> >>>> On Sat, 27 Sep 2008 07:58:32 +0200, Ron Sigal <ron.sigal(a)jboss.com&gt; wrote: >>>> >>>>> Hi Jiri, >>>>> >>>>> Well, I'm trying to figure out the puzzle of client.log, and here's what I know and what I don't know. >>>>> >>>>> 1. The JMS connection starts in several stages. a. SocketClientInvoker[1855562, bisocket://localhost:4457], which is used by the main JBM Client, announces it is connected on line 25. b. SocketClientInvoker[19b46dc, bisocket://localhost:4457], which is used by the JBM oneway Client, announces it is connected on line 38. c. The callback handler, CallbackManager[be49e0], is installed as of line 50. >>>>> >>>>> d. SocketClientInvoker[167d7fb, bisocket://localhost:4457], which is used by the ConnectionValidator associated with SocketClientInvoker[1855562, bisocket://localhost:4457], is started as of line 65. >>>>> >>>>> e. The JMS connection is up as of line 66. >>>>> 2. At that point, JMSMockApplication does a bunch of stuff. >>>>> >>>>> 3. On line 185, JMSMockApplication announces it is exiting. >>>>> >>>>> a. ConnectionValidator[SocketClientInvoker[167d7fb, bisocket://localhost:4457], associated with main JBM connection, is stopped as of line 190. >>>>> b. The callback server is stopped as of line 197. >>>>> c. LeasePinger[SocketClientInvoker[1855562, bisocket://localhost:4457] stops at line 198. >>>>> d. SocketClientInvoker[1855562, bisocket://localhost:4457] is disconnected as of line 200. >>>>> e. SocketClientInvoker[19b46dc, bisocket://localhost:4457] is disconnected as of line 203. >>>>> So far, this makes sense to me. As LeasePinger[SocketClientInvoker[1855562, bisocket://localhost:4457] is stopping, it should send a $DISCONNECT$ message to the server, which is what causes the ServerInvokerCallbackHandler to shut down prematurely. >>>>> >>>>> There's a lot of other stuff going on that I don't understand. For example, there are four other client invokers in the log, as well as another ConnectionValidator. I'm guessing that they are created by your test code and have nothing to do with JBM. For example, the last reference to JMSMockApplication is on line 185. >>>>> >>>>> In other words, as far as I can tell, JBM and JBR are behaving reasonably on the client side. The problem seems to be the premature shut down of ServerInvokerCallbackHandler on the server side, and, if my understanding of JBM is correct (SimpleConnectionManager, in particular), I would suggest turning off the feature that causes ServerInvokerCallbackHandler to register itself as a Lease listener, i.e., set parameter "registerCallbackListener" to false. >>>>> >>>>> Let me know what happens next. >>>>> >>>>> -Ron >>>>> >>>>> Jiri Pechanec wrote: >>>>>> Hi Ron, >>>>>> >>>>>> thanks I guess we are arriving to the core - I have attached full client log - can you see any reason why DISCONNECT message was sent? >>>>>> >>>>>> I can even send the server side log, but it is quite huge >>>>>> >>>>>> J. >>>>>> >>>>>> On Fri, 26 Sep 2008 09:14:51 +0200, Ron Sigal <ron.sigal(a)jboss.com&gt; wrote: >>>>>> >>>>>>> Hi Jiri, >>>>>>> >>>>>>> I have a partial explanation for this log. 1. At 05:25:54,302 the server gets a "$DISCONNECT$" message from the LeasePinger on the client side, which leads to shutting down the Lease on the server side and informing all registered listeners that the connection is broken. I can't tell from the fragments you sent what caused the $DISCONNECT$ message to be sent. >>>>>>> >>>>>>> 2. As of JBREM-873 "Have ServerInvokerCallbackHandler register as connection listener", org.jboss.remoting.callback.ServerInvokerCallbackHandler has, by default, registered itself as a Lease listener. When the Lease shuts down, it informs ServerInvokerCallbackHandler that the connection is broken, and ServerInvokerCallbackHandler disconnects its callback Client. Subsequent calls to ServerInvokerCallbackHandler.handleCallback() see no Client, so it looks like the connection does pull callbacks. The problem is that ServerInvokerCallbackHandler is shutting itself down before JBM shuts down the JMS connection, which seems to occur at 05:28:15,122. >>>>>>> >>>>>>> I recall that Tim asked me, at some point, if the application was required to shut down ServerInvokerCallbackHandler. In the normal course of events, ServerInvoker will get a REMOVELISTENER message and it will call ServerInvokerCallbackHandler.destroy(). But if the connection is really broken, there will be no such message. Instead, the Lease will detect a failure and notify listeners. Ideally, at this point, JBM would shut down the server side of the connection, including shutting down ServerInvokerCallbackHandler. However, the Remoting Guide never documented the need to call ServerInvokerCallbackHandler.destroy(), which was the reason I created JBREM-873. Now, looking at org.jboss.jms.server.connectionmanager.SimpleConnectionManager, which I guess would be informed when a Lease shuts down, it looks like SimpleConnectionManager will call org.jboss.jms.server.endpoint.closeCallbackClient(), which will call ServerInvokerCallbackHandler.callbackHandler.getCallbackClient().disconnect(), which is more or less what ServerInvokerCallbackHandler.destroy() does. Actually, that would throw a NullPointerException if there is no callback Client, which would be true for the http and servlet transports, so it would be better to call ServerInvokerCallbackHandler.destroy(). Anyway, if I understand the SimpleConnectionManager correctly, there's no need for ServerInvokerCallbackHandler to shut itself down. You can turn off this behavior by adding >>>>>>> >>>>>>> <attribute name="registerCallbackListener">false</attribute> to remoting-bisocket-service.xml. >>>>>>> >>>>>>> Tim, if you remember anything in addition to, or at odds with, my recollections, please let me know. >>>>>>> >>>>>>> -Ron >>>>>>> >>>>>>> >>>>>>> Jiri Pechanec wrote: >>>>>>>> Hi Ron, >>>>>>>> >>>>>>>> thanks for your explanation. I think I need you to explain a problem a little bit more >>>>>>>> >>>>>>>> I am putting the snippets from both server and client log correlated together so hopefully we''l find another reason for the wrong behaviour. Please see attached file to prevent any formatting issus with linebreaks >>>>>>>> >>>>>>>> In short there is a callback created, JBM uses the callback (PUSH), then callback is closed which intenrally sets client = null. JBM uses the callback, but now PULL - and then the things are broken because PUSH must be used >>>>>>>> >>>>>>>> I believe there is a BUG, but I am not sure if it is in JBM or Remoting beacuse there are two options now >>>>>>>> 1) remoting closes callback handler although it should not >>>>>>>> 2) JBM uses closed callback handler although it should not >>>>>>>> >>>>>>>> I was disucssing this with Tim and his statement is that there is a problem in JBR. From the log messages I have attached I am now inclined to agree with him. Can you please look at the log snippets that I have attached and try to explain them - I was mainly surprised that the connecteion was closed on server but on client the LeasePinger was still managing the connection and closed it 2 minutes later - which in turn at the same time generated this message >>>>>>>> 2008-09-24 05:28:15,122 DEBUG [org.jboss.jms.server.connectionmanager.SimpleConnectionManager] unregistered connection ConnectionEndpoint[3t-be42rhlf-1-tkiuqhlf-3etw0x-060ia] with remoting session ID ...-flhr24dc-44 >>>>>>>> >>>>>>>> Overall the message flow and timing seems suspicious to me >>>>>>>> >>>>>>>> J. >>>>>>>> >>>>>>>> On Fri, 26 Sep 2008 04:37:04 +0200, Ron Sigal <ron.sigal(a)jboss.com&gt; wrote: >>>>>>>> >>>>>>>>> Hi Jiri, >>>>>>>>> >>>>>>>>> This lease stuff gets a little confusing. I say that as an expert on >>>>>>>>> the subject of being confused by leasing. :) >>>>>>>>> >>>>>>>>> Remoting has a couple of layers on the client side. A user creates an >>>>>>>>> org.jboss.remoting.Client, which it can use to send invocations to the >>>>>>>>> server. But the Client creates a transport specific invoker, e.g., >>>>>>>>> BisocketClientInvoker for the bisocket transport, through which the >>>>>>>>> Client communicates with the server. It's the client invoker (in >>>>>>>>> particular, org.jboss.remoting.MicroRemoteClientInvoker, the parent of >>>>>>>>> all client invokers), which manages leases on the client side. In >>>>>>>>> particular, MicroRemoteClientInvoker creates an >>>>>>>>> org.jboss.remoting.LeasePinger to ping the server. The important thing >>>>>>>>> to note is that a single MicroRemoteClientInvoker could, in principle, >>>>>>>>> be used by multiple Clients. By extension, multiple Clients can be >>>>>>>>> serviced by a single LeasePinger. So, there are two types of IDs >>>>>>>>> involved in leasing: >>>>>>>>> >>>>>>>>> 1. The LeasePinger is identified by the session ID of the >>>>>>>>> MicroRemoteClientInvoker that created it, and >>>>>>>>> >>>>>>>>> 2. with each ping, the LeasePinger sends the set of session IDs of all >>>>>>>>> of the Clients that it is servicing. >>>>>>>>> >>>>>>>>> Now, when a single Client disconnects, it tells the LeasePinger to >>>>>>>>> remove its session ID from the set maintained by the LeasePinger. But >>>>>>>>> when the underlying invoker shuts down, it sends the "$DISCONNECT$" >>>>>>>>> message to the server, which closes the Lease for all Client connections >>>>>>>>> sharing the transport layer connection. Then, for each Client sessionID >>>>>>>>> in the set received from the LeasePinger, the Lease (using >>>>>>>>> org.jboss.remoting.ConnectionNotifier) notifies all registered listeners >>>>>>>>> that the Client has disconnected: >>>>>>>>> >>>>>>>>> 1. line 2: "terminating invoker lease: >>>>>>>>> ai060-cbto0n-flhqxb53-1-flhqz03g-1z:" refers to the >>>>>>>>> *MicroRemoteClientInvoker's* session ID, i.e., the whole Lease is being >>>>>>>>> terminated. >>>>>>>>> >>>>>>>>> 2. line 3: "Terminating lease for session id >>>>>>>>> ai060-cbto0n-flhqxb53-1-flhqz03g-1z" also refers to the >>>>>>>>> MicroRemoteClientInvoker's session ID. >>>>>>>>> >>>>>>>>> 3. line 4: "Client disconnected (session id = >>>>>>>>> ai060-cbto0n-flhqxb53-1-flhr24dc-44" refers to the *Client's* sessionID. >>>>>>>>> >>>>>>>>> So, as far as I can see, leasing is working properly. Let me know if >>>>>>>>> there are any other signs of malfunction. >>>>>>>>> >>>>>>>>> -Ron >>>>>>>>> >>>>>>>>> Jiri Pechanec wrote: >>>>>>>>>> Hi Ron, >>>>>>>>>> >>>>>>>>>> I am hunting problems with JMS under very specific conditions (testing >>>>>>>>>> SOA-P with higher load). It seems that when one connection is closed >>>>>>>>>> it seems that it closes callback handler clinet for another lease >>>>>>>>>> See excerpt of log lines 2-4 >>>>>>>>>> >>>>>>>>>> Session ids are different at lines 3 and 4 >>>>>>>>>> >>>>>>>>>> 2008-09-24 05:25:54,302 TRACE [org.jboss.remoting.ServerInvoker] >>>>>>>>>> SocketServerInvoker[127.0.0.1:4457] received $DISCONNECT$ >>>>>>>>>> 2008-09-24 05:25:54,303 TRACE [org.jboss.remoting.ServerInvoker] >>>>>>>>>> terminating invoker lease: ai060-cbto0n-flhqxb53-1-flhqz03g-1z >>>>>>>>>> 2008-09-24 05:25:54,303 TRACE [org.jboss.remoting.Lease] Terminating >>>>>>>>>> lease for session id ai060-cbto0n-flhqxb53-1-flhqz03g-1z >>>>>>>>>> 2008-09-24 05:25:54,303 TRACE [org.jboss.remoting.ConnectionNotifier] >>>>>>>>>> Client disconnected (session id = ai060-cbto0n-flhqxb53-1-flhr24dc-44 >>>>>>>>>> 2008-09-24 05:25:54,304 TRACE [org.jboss.remoting.InvokerRegistry] >>>>>>>>>> destroying client invoker InvokerLocator >>>>>>>>>> [bisocket://10.18.0.216:655404877/callback?callbackServerHost=10.180.216&callbackServerPort=655404877&callbackServerProtocol=bisocket&clientMaxPoolSize=1&clientSocketClass=org.jboss.jms.client.remoting.ClientSocketWrapper&datatype=jms&guid=ai060-cbto0n-flhqxb53-1-flhr24dl-46&isCallbackServer=true&onewayThreadPool=org.jboss.jms.server.remoting.DirectThreadPool&serverSocketClass=org.jboss.jms.server.remoting.ServerSocketWrapper], config {socket.check_connection=false, dataType=jms, >>>>>>>>>> serverInvokerCallbackHandler=ServerInvokerCallbackHandler[ai060-cbto0n-flhqxb53-1-flhr24dc-44+ai060-cbto0n-flhqxb53-1-flhr24dw-47], datatype=jms, callbackServerProtocol=bisocket, >>>>>>>>>> callbackServerPort=655404877, pingWindowFactor=10, numberOfRetries=10, >>>>>>>>>> clientMaxPoolSize=1, JBM_clientMaxPoolSize=200, >>>>>>>>>> serverSocketClass=org.jboss.jms.server.remoting.ServerSocketWrapper, >>>>>>>>>> isCallbackServer=true, pingFrequency=214748364, timeout=10000, >>>>>>>>>> listenerId=ai060-cbto0n-flhqxb53-1-flhr24dw-47, >>>>>>>>>> marshaller=org.jboss.jms.wireformat.JMSWireFormat, >>>>>>>>>> callbackServerHost=10.18.0.216, serverBindPort=4457, >>>>>>>>>> stopLeaseOnFailure=true, serverBindAddress=localhost, >>>>>>>>>> callbackTimeout=10000, guid=ai060-cbto0n-flhqxb53-1-flhr24dl-46, >>>>>>>>>> onewayThreadPool=org.jboss.jms.server.remoting.DirectThreadPool, >>>>>>>>>> unmarshaller=org.jboss.jms.wireformat.JMSWireFormat, >>>>>>>>>> clientLeasePeriod=10000, >>>>>>>>>> serverInvoker=SocketServerInvoker[127.0.0.1:4457], >>>>>>>>>> numberOfCallRetries=1, >>>>>>>>>> clientSocketClass=org.jboss.jms.client.remoting.ClientSocketWrapper} >>>>>>>>>> 2008-09-24 05:25:54,304 DEBUG [org.jboss.remoting.InvokerRegistry] >>>>>>>>>> removed SocketClientInvoker[de8f62, bisocket://10.18.0.216:655404877] >>>>>>>>>> from registry >>>>>>>>>> 2008-09-24 05:25:54,304 TRACE [org.jboss.remoting.InvokerRegistry] >>>>>>>>>> disconnecting SocketClientInvoker[de8f62, >>>>>>>>>> bisocket://10.18.0.216:655404877] >>>>>>>>>> 2008-09-24 05:25:54,304 DEBUG >>>>>>>>>> [org.jboss.remoting.transport.socket.MicroSocketClientInvoker] >>>>>>>>>> SocketClientInvoker[de8f62, bisocket://10.18.0.216:655404877] >>>>>>>>>> disconnecting ... >>>>>>>>>> 2008-09-24 05:25:54,304 TRACE >>>>>>>>>> [org.jboss.remoting.MicroRemoteClientInvoker] >>>>>>>>>> SocketClientInvoker[de8f62, bisocket://10.18.0.216:655404877] >>>>>>>>>> disconnecting ... >>>>>>>>>> 2008-09-24 05:25:54,304 DEBUG >>>>>>>>>> [org.jboss.remoting.transport.socket.SocketWrapper] >>>>>>>>>> ClientSocketWrapper[Socket[addr=/127.0.0.1,port=56727,localport=47367].122e5be] closing >>>>>>>>>> >>>>>>>>>> When there are no problems I am seeing >>>>>>>>>> Session ids are same at lines 3 and 4 >>>>>>>>>> >>>>>>>>>> 2008-09-24 05:20:44,555 TRACE [org.jboss.remoting.ServerInvoker] >>>>>>>>>> SocketServerInvoker[127.0.0.1:4457] received $DISCONNECT$ >>>>>>>>>> 2008-09-24 05:20:44,555 TRACE [org.jboss.remoting.ServerInvoker] >>>>>>>>>> terminating client lease: ai060-cbto0n-flhqxb53-1-flhqxd29-7 >>>>>>>>>> 2008-09-24 05:20:44,555 TRACE [org.jboss.remoting.Lease] Terminating >>>>>>>>>> lease for session id ai060-cbto0n-flhqxb53-1-flhqxd26-5 >>>>>>>>>> 2008-09-24 05:20:44,556 TRACE [org.jboss.remoting.ConnectionNotifier] >>>>>>>>>> Client disconnected (session id = ai060-cbto0n-flhqxb53-1-flhqxd26-5 >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Can you please take a look at it? >>>>>>>>>> >>>>>>>>>> Thanks >>>>>>>>>> >>>>>>>>>> J. >>>>>>>> >>>>>> >>>>>> >>>>>> >>>> >>>> >>>> >> >>

JBM stops processing message due to switch to PULL ServerInvokerCallbackHandler ------------------------------------------------------------------------------- Key: JBMESSAGING-1428 URL: https://jira.jboss.org/jira/browse/JBMESSAGING-1428 Project: JBoss Messaging Issue Type: Bug Affects Versions: 1.4.0.SP3_CP03 Reporter: Jiri Pechanec Assignee: Tim Fox Fix For: 1.4.0.SP3.CP04 This issue links JBPAPP-1219