[wildfly-dev] Cassandra Subsystem

[David M. Lloyd]

Indeed, when you define a service to control the process, the lifecycle 
of the process should not be tied to the service.  Instead, the service 
represents the desire to have the process running.

A dedicated reaper thread with a small (32k? maybe less on 32-bit?) 
stack should be established to waitFor() and re-create the process (with 
a short/configurable delay to avoid overloading the system).  If you 
want to capture stdout and stderr, one small-stack thread each should be 
created for each to capture the process output and feed it to the 
logging layer.  Alternatively, they can be combined by ProcessBuilder so 
you only need one thread (though the output might be intermixed). 
Pooled threads generally are not a good fit for this application. 
Daemon threads should not be used, as that might provide more 
opportunity for the JVM to exit before the child process has been shut 
down.  In JDK 9, there will likely be options to force child processes 
to terminate when the parent terminates, but we do not yet have this 
capability available unfortunately, so we must live with this limit of 
what we can do to avoid misadventure.

The input stream to the process must be managed so the process never 
hangs waiting for input.  This could be as simple as closing the stream 
immediately on start.

The reaper thread should be cleanly managed to ensure deterministic 
shutdown of the process on service stop().  This can be done through the 
application of a (volatile) stop flag and thread interruption, combined 
with async service stop, which the reaper thread would invoke upon exit. 
  Because you need an async service stop context to stop the thread, you 
could use a field of this type as the stop flag; if null, then no stop 
is requested.

Maybe a ProcessManagementService would be a good addition to MSC...

On 09/19/2014 09:00 AM, Scott Marlow wrote:
> Even if System.exit() isn't called, the process can still terminate for
> other reasons.  How do you want to handle a Cassandra process
> termination?  Perhaps log the error and retry starting it a certain
> number of times (with maybe a time threshold on how many times to
> restart)?  Do we have logic like this elsewhere?
>
> IMO, we should also read the Cassandra process output so that it doesn't
> stay in memory.
>
> On 09/19/2014 08:57 AM, Heiko Braun wrote:
>> It's not my preferred solution, but something to explore.  At the same
>> time I talking to the apache folks if we can get rid of the
>> System.exit() call in C*. Which they seem to consider  a good thing too.
>>
>> But thanks for the pointers, I'll have a look. Bootstrapping modules
>> like we do it for WF didn't come to my mind.
>>
>> Regards, Heiko
>>
>> On 19 Sep 2014, at 14:52, David M. Lloyd <david.lloyd at redhat.com> wrote:
>>
>>> Well, I have a bias of course, but it seems to me like you could just
>>> add Cassandra and its dependencies as modules and then launch it like
>>> this:
>>>
>>>      String javaHome = System.getProperty("java.home");
>>>      ProcessBuilder pb = new ProcssBuilder(
>>>     javaHome + File.pathSeparator + "bin" + File.pathSeparator + "java",
>>>          ...jvm args...,
>>>          "-jar",
>>>          "jboss-modules.jar",
>>>          "-mp",
>>>          "modules",
>>>          "org.whatever.cassandra",
>>>          ...cassandra args...);
>>>      ...set up I/O redirection here...
>>>      Process p = pb.start();
>>>
>>> On 09/19/2014 03:12 AM, Heiko Braun wrote:
>>>>
>>>>
>>>> David,
>>>>
>>>> I've been thinking more about your proposal to treat the cassandra
>>>> daemon as a subprocess. Currently the cassandra dependencies are
>>>> installed along side the module. How would I pass the classpath to
>>>> the subprocess? Is there a way to get to the CP, similar to the way
>>>> an URLClassLoader exposes it? Or how could it be achieved with JBoss
>>>> modules?
>>>>
>>>> /Heiko
>>>>
>>>> On 26 Aug 2014, at 17:36, David M. Lloyd <david.lloyd at redhat.com>
>>>> wrote:
>>>>
>>>>> Yeah, you could have a service that controls the cassandra daemon as a
>>>>> service and uses whatever its default network API is for
>>>>> communications.
>>>>>   This should be safe and reliable as of JDK 7.
>>>>>
>>>>> Just a suggestion in case embedding becomes an untenable proposition.
>>>>>
>>>>> On 8/26/14, 8:23 AM, Heiko Braun wrote:
>>>>>>
>>>>>>
>>>>>> Not sure I can follow. Do you refer to the System.exit() calls?
>>>>>>
>>>>>> On 26 Aug 2014, at 15:10, David M. Lloyd <david.lloyd at redhat.com
>>>>>> <mailto:david.lloyd at redhat.com>> wrote:
>>>>>>
>>>>>>> One possible option might be to run it as a subprocess.
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> - DML
>>>>> _______________________________________________
>>>>> wildfly-dev mailing list
>>>>> wildfly-dev at lists.jboss.org
>>>>> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>>>>
>>>
>>> --
>>> - DML
>>> _______________________________________________
>>> wildfly-dev mailing list
>>> wildfly-dev at lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>>
>>
>> _______________________________________________
>> wildfly-dev mailing list
>> wildfly-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>>
>

-- 
- DML