3:59 p.m.
Kabir Khan [http://community.jboss.org/people/kabir.khan%40jboss.com] created the
discussion
"Re: ServerManager-Server(-ProcessManager) communication"
To view the discussion, visit: http://community.jboss.org/message/560646#560646
--------------------------------------------------------------
The respawn policy should be configurable: https://jira.jboss.org/browse/JBAS-8390
https://jira.jboss.org/browse/JBAS-8390
--------------------------------------------------------------
Reply to this message by going to Community
[http://community.jboss.org/message/560646#560646]
Start a new discussion in JBoss AS7 Development at Community
[http://community.jboss.org/choose-container!input.jspa?contentType=1&...]
12:02 p.m.
Kabir Khan [http://community.jboss.org/people/kabir.khan%40jboss.com] created the
discussion
"Re: ServerManager-Server(-ProcessManager) communication"
To view the discussion, visit: http://community.jboss.org/message/561616#561616
--------------------------------------------------------------
Once this is all properly tested and I get the domain controller to work in my branch I
wonder what to do next. We have two sets of connections at the moment:
1) Server processes connect to SM via a socket
2) Server/SM processes connect to PM via a socket
If 1) goes down I was originally going to suggest routing the commands via 2) instead, but
I think it makes a lot more sense to send the RECONNECT_SERVERS command via 2 instead? The
simplest way would be in SM to stop and start the SM listening socket whenever the direct
connection to ANY server goes down.
If 2) goes down I am not sure what to do. I have thrown away all the stdio stuff that was
originally there. But if 2) falls over I should probably switch to stdio for that process
and route the commands through there and also invent some commands to make the process
reconnect to the process managers socket. Does anybody know if there are any problems with
suddenly starting to consume a processes output and starting to write to its input and
then stopping doing that again?
Once this is all properly tested and I get the domain controller to work in my branch I
wonder what to do next. We have two sets of connections at the moment:
1) Server processes connect to SM via a socket
2) Server/SM processes connect to PM via a socket
If 1) goes down I was originally going to suggest routing the commands via 2) instead, but
I think it makes a lot more sense to send the RECONNECT_SERVERS command via 2 instead? The
simplest way would be in SM to stop and start the SM listening socket whenever the direct
connection to ANY server goes down.
If 2) goes down I am not sure what to do. I have thrown away all the stdio stuff that was
originally there. But if 2) falls over I should switch to stdio for that process and route
the commands through there and also invent some commands to make the process reconnect to
the PM socket. Does anybody know if there are any problems with suddenly starting to
consume a processes standard output and starting to write to its standard input and then
stopping doing that again?
--------------------------------------------------------------
Reply to this message by going to Community
[http://community.jboss.org/message/561616#561616]
Start a new discussion in JBoss AS7 Development at Community
[http://community.jboss.org/choose-container!input.jspa?contentType=1&...]
12:03 p.m.
Brian Stansberry [http://community.jboss.org/people/bstansberry%40jboss.com] created the
discussion
"Re: ServerManager-Server(-ProcessManager) communication"
To view the discussion, visit: http://community.jboss.org/message/561853#561853
--------------------------------------------------------------
Kabir Khan wrote:
> Brian Stansberry wrote:
>
> I agree it makes more sense to handle a failure of 1) by using the PM to tell the
server(s) to reconnect. Stopping and starting the SM's listening socket gives me a bit
of a queasy feeling though; it could interrupt other on-going communication. Is the
advantage mainly simplicity (stop and start the socket triggers the existing 6.3.4 logic
above instead of requiring the SM to send a command to the PM)?
I suggested restarting the listener for two reasons. It sounded simpler, and I am not
sure in what situations the SM listener would be well and truly broken. But I can
definitely send a command to PM instead (and then if we find scenarios where we need to
restart SM's listener I'll deal with those later).
Sounds good.
The PM needs to consume each process' stdout and stderr
anyway.>
Not sure what you mean here? We still consume stderr for logging but I have done away
with the stdout consumption. If you mean for monitoring whether the process is still
alive, that is now done by a simple Process.waitFor().
A key thing the PM needs to do is handle the last sentence in this bit from the
java.lang.Process class javadoc:
"The created subprocess does not have its own terminal or console. All its standard
io (i.e. stdin, stdout, stderr) operations will be redirected to the parent process
through three streams (
http://download.oracle.com/javase/6/docs/api/java/lang/Process.html#getOu...
getOutputStream(),
http://download.oracle.com/javase/6/docs/api/java/lang/Process.html#getIn...
getInputStream(),
http://download.oracle.com/javase/6/docs/api/java/lang/Process.html#getEr...
getErrorStream()). The parent process uses these streams to feed input to and get output
from the subprocess. Because some native platforms only provide limited buffer size for
standard input and output streams, failure to promptly write the input stream or read the
output stream of the subprocess may cause the subprocess to block, and even
deadlock."
So, whatever comes out of stdout or stderr needs to be consumed promptly.
Using stdin all the time for PM -> Process sounds good.
--------------------------------------------------------------
Reply to this message by going to Community
[http://community.jboss.org/message/561853#561853]
Start a new discussion in JBoss AS7 Development at Community
[http://community.jboss.org/choose-container!input.jspa?contentType=1&...]
12:08 p.m.
Kabir Khan [http://community.jboss.org/people/kabir.khan%40jboss.com] created the
discussion
"Re: ServerManager-Server(-ProcessManager) communication"
To view the discussion, visit: http://community.jboss.org/message/561778#561778
--------------------------------------------------------------
Brian Stansberry wrote:
I agree it makes more sense to handle a failure of 1) by using the PM to tell the
server(s) to reconnect. Stopping and starting the SM's listening socket gives me a bit
of a queasy feeling though; it could interrupt other on-going communication. Is the
advantage mainly simplicity (stop and start the socket triggers the existing 6.3.4 logic
above instead of requiring the SM to send a command to the PM)?
I suggested
restarting the listener for two reasons. It sounded simpler, and I am not sure in what
situations the SM listener would be well and truly broken. But I can definitely send a
command to PM instead (and then if we find scenarios where we need to restart SM's
listener I'll deal with those later).
The PM needs to consume each process' stdout and stderr anyway.
Not sure what you mean here? We still consume stderr for logging but I have done
away with the stdout consumption. If you mean for monitoring whether the process is still
alive, that is now done by a simple Process.waitFor().
I don't think it's good though to try and route commands over
stdio besides the ones needed to get the socket communication going away. Otherwise we
face the problem of dealing with junk sent via stdout that led to our using sockets. One
thing though is the PM can safely use each child process' stdin to send commands to
that process.
I agree. To keep things simple:
a) PM will only listen for input from processes via the process sockets
b) PM will only send data to processes via stdin
If the socket goes down PM will send a message to the process via its stdin to reconnect
to its socket. For b) I am currently using the socket to push commands from PM->Process
and was going to switch over to stdin when the socket goes down, but it will be a lot
simpler, with less switching between different communication mechanisms, if we just use
stdin all the time.
--------------------------------------------------------------
Reply to this message by going to Community
[http://community.jboss.org/message/561778#561778]
Start a new discussion in JBoss AS7 Development at Community
[http://community.jboss.org/choose-container!input.jspa?contentType=1&...]
12:15 p.m.
Kabir Khan [http://community.jboss.org/people/kabir.khan%40jboss.com] created the
discussion
"Re: ServerManager-Server(-ProcessManager) communication"
To view the discussion, visit: http://community.jboss.org/message/561862#561862
--------------------------------------------------------------
Brian Stansberry wrote:
A key thing the PM needs to do is handle the last sentence in this bit from the
java.lang.Process class javadoc:
"The created subprocess does not have its own terminal or console. All its standard
io (i.e. stdin, stdout, stderr) operations will be redirected to the parent process
through three streams (
http://download.oracle.com/javase/6/docs/api/java/lang/Process.html#getOu...
getOutputStream(),
http://download.oracle.com/javase/6/docs/api/java/lang/Process.html#getIn...
getInputStream(),
http://download.oracle.com/javase/6/docs/api/java/lang/Process.html#getEr...
getErrorStream()). The parent process uses these streams to feed input to and get output
from the subprocess. Because some native platforms only provide limited buffer size for
standard input and output streams, failure to promptly write the input stream or read the
output stream of the subprocess may cause the subprocess to block, and even
deadlock."
Thanks, that is kind of what I was worried about regarding
stdout/-err. I should have read the javadoc!
--------------------------------------------------------------
Reply to this message by going to Community
[http://community.jboss.org/message/561862#561862]
Start a new discussion in JBoss AS7 Development at Community
[http://community.jboss.org/choose-container!input.jspa?contentType=1&...]
12:30 p.m.
Kabir Khan [http://community.jboss.org/people/kabir.khan%40jboss.com] created the
discussion
"Re: ServerManager-Server(-ProcessManager) communication"
To view the discussion, visit: http://community.jboss.org/message/561863#561863
--------------------------------------------------------------
Kabir Khan wrote:
But I can definitely send a command to PM instead (and then if we find scenarios where
we need to restart SM's listener I'll deal with those later).
When SM
detects that a server's socket connection is down it now sends 'RECONNECT_SERVER
<SERVER_NAME> <SM_ADDRESS> <PSM>' to PM which then tells the server
to reconnect (it results in 6.3.4.1 for the server process)
--------------------------------------------------------------
Reply to this message by going to Community
[http://community.jboss.org/message/561863#561863]
Start a new discussion in JBoss AS7 Development at Community
[http://community.jboss.org/choose-container!input.jspa?contentType=1&...]
12:31 p.m.
Kabir Khan [http://community.jboss.org/people/kabir.khan%40jboss.com] created the
discussion
"Re: ServerManager-Server(-ProcessManager) communication"
To view the discussion, visit: http://community.jboss.org/message/561605#561605
--------------------------------------------------------------
I have implemented the following but still need to test some corner cases
1. PM starts up and listens on a socket on a port (PPM) for connections from the processes
it manages.
2. PM starts SM passing in PPM, PM’s host address and ‘ServerManager’ as the name1. SM
opens a socket on a different port (PSM) which listens for connections from the Server
processes and from the Domain Controller (DC).
2. SM initiates communication with PM, by connecting to port PPM. The first command it
sends is 'CONNECTED ServerManager', which helps PM associate the socket with the
correct ManagedProcess.
3. For each Server configured in SM:1. SM tells PM to add Server
2. SM tells PM to start Server process1. PM launches the Server process, passing in PPM,
PM_ADDRESS, PSM, SM_ADDRESS and the SERVER_NAME
2. Server initiates communication with PM, by connecting to port PPM. The first command it
sends is 'CONNECTED <SERVER_NAME>', which helps PM associate the socket with
the correct ManagedProcess.
3. Server starts listening for commands on the PM socket.
4. Server initiates communication with SM, by connecting to port PSM. The first command it
sends is 'CONNECTED <SERVER_NAME>', which helps SM associate the socket with
the correct Server proxy.
5. Server sends the ‘SERVER_AVAILABLE’ command on the SM socket
6. Server starts listening for commands on the SM socket
3. SM sends the ‘START_SERVER serverConfig’ message to the server via the Server’s
socket1. Server parses the serverConfig, starts up and sends to SM either1.
‘SERVER_STARTED’ if successful.
2. ‘SERVER_START_FAILED’ if failed1. SM tells PM to stop process
2. If Server auto-restart=true and number retries is < respawn_policy_max SM repeats
2.3.2 and 2.3.3 *- The respawn policy should be configurable (*
https://jira.jboss.org/browse/JBAS-8390 *https://jira.jboss.org/browse/JBAS-8390**)*
3. Otherwise tell PM to remove process
3. While a ManagedProcess is registered as started in PM1. Processes are connected to the
PM socket
2. The ManagedProcess monitors whether the process is still alive (with a thread doing
Process.waitFor())1. If a Server process goes down PM stops the process and sends ‘DOWN
<SERVER_NAME>’ to SM on the PM-SM connection.1. SM respawns the server process
according to the rules in 2.3.3.2.
2. If the ServerManager process goes down PM respawns it as in 6.3
4. To shut down a server1. SM sends ‘STOP_SERVER’ to server.1. Server closes down
2. Server sends ‘SERVER_STOPPED’ to SM.
3. SM tells PM to stop the Server process
4. SM tells PM to remove the Server process
5. Closing down everything1. Message to shutdown comes from1. SM gets SHUTDOWN command
from DC or management interface
2. Shutdown hook in PM1. Send ‘SHUTDOWN_SERVERS’ command to SM1. For each server do 4 to
close it down
2. PM sends 'SHUTDOWN' message to SM which closes down SM as in 6
6. Restarting SM1. SM process is stopped by1. Message from DC
2. Process is killed
2. SM is down...
3. SM process is started1. ** PM starts SM passing in PPM, PM’s host address and
‘ServerManager’ as the name along with the -restarted-server-manager flag.
2. See 2.1
3. See 2.2
4. SM sends the ‘RECONNECT_SERVERS <SM_ADDRESS> <PSM>’ command to PM1. For
each Server process PM sends ‘RECONNECT_SERVER_MANAGER <SM_ADDRESS> <PSM>’1.
Server reconnects to SM as in 2.3.2.4
2. Server sends ‘SERVER_RECONNECT_STATUS <Current_State>’ to SM1. if the server is
not in the starting, started, stopping or stopped state (I added some basic state
management) SM does 2.3.3
--------------------------------------------------------------
Reply to this message by going to Community
[http://community.jboss.org/message/561605#561605]
Start a new discussion in JBoss AS7 Development at Community
[http://community.jboss.org/choose-container!input.jspa?contentType=1&...]
12:33 p.m.
Brian Stansberry [http://community.jboss.org/people/bstansberry%40jboss.com] created the
discussion
"Re: ServerManager-Server(-ProcessManager) communication"
To view the discussion, visit: http://community.jboss.org/message/561694#561694
--------------------------------------------------------------
I agree it makes more sense to handle a failure of 1) by using the PM to tell the
server(s) to reconnect. Stopping and starting the SM's listening socket gives me a bit
of a queasy feeling though; it could interrupt other on-going communication. Is the
advantage mainly simplicity (stop and start the socket triggers the existing 6.3.4 logic
above instead of requiring the SM to send a command to the PM)?
If 2) goes down using stdio if needed to coordinate commands to trigger reconnecting
sounds fine. I don't know any reason why using stdio again would be a problem. The PM
needs to consume each process' stdout and stderr anyway.
I don't think it's good though to try and route commands over stdio besides the
ones needed to get the socket communication going away. Otherwise we face the problem of
dealing with junk sent via stdout that led to our using sockets. One thing though is the
PM can safely use each child process' stdin to send commands to that process.
--------------------------------------------------------------
Reply to this message by going to Community
[http://community.jboss.org/message/561694#561694]
Start a new discussion in JBoss AS7 Development at Community
[http://community.jboss.org/choose-container!input.jspa?contentType=1&...]
1:15 p.m.
Kabir Khan [http://community.jboss.org/people/kabir.khan%40jboss.com] created the
discussion
"Re: ServerManager-Server(-ProcessManager) communication"
To view the discussion, visit: http://community.jboss.org/message/563796#563796
--------------------------------------------------------------
Jason,
Back from holidays. In my topic branch I am going to revert the work you did for
http://github.com/jbossas/jboss-as/commit/7f3f5d0f86c4c329d0dd4a041df6ebc...
http://github.com/jbossas/jboss-as/commit/7f3f5d0f86c4c329d0dd4a041df6ebc... since
now all ManagedProcess->actual process communications happen via the process's
stdin
--------------------------------------------------------------
Reply to this message by going to Community
[http://community.jboss.org/message/563796#563796]
Start a new discussion in JBoss AS7 Development at Community
[http://community.jboss.org/choose-container!input.jspa?contentType=1&...]
1:25 p.m.
Jason Greene [http://community.jboss.org/people/jason.greene%40jboss.com] created the
discussion
"Re: ServerManager-Server(-ProcessManager) communication"
To view the discussion, visit: http://community.jboss.org/message/563800#563800
--------------------------------------------------------------
Actually I meant to post about that, but never got around to it. Basically the problem is
that we need to use stdin to ship an early logging context (any any other pre boostrap
data) as soon as the process starts. Brain, David, and I last week talked about the points
above here about switching all communications to STDIN, and really the advantages
aren't that much better over a TCP connection over loopback. If that connection drops
both parties will know right away.
--------------------------------------------------------------
Reply to this message by going to Community
[http://community.jboss.org/message/563800#563800]
Start a new discussion in JBoss AS7 Development at Community
[http://community.jboss.org/choose-container!input.jspa?contentType=1&...]
8:24 p.m.
Kabir Khan [http://community.jboss.org/people/kabir.khan%40jboss.com] created the
discussion
"Re: ServerManager-Server(-ProcessManager) communication"
To view the discussion, visit: http://community.jboss.org/message/564613#564613
--------------------------------------------------------------
I got rid of the ProcessManager send/broadcast methods taking a String list. I think I
can get rid of the methods taking byte[] as well? Nothing is using them as D<->SM
and SM<->Server use direct communication, and PM<->Process should use the PM
protocol. Any objections?
--------------------------------------------------------------
Reply to this message by going to Community
[http://community.jboss.org/message/564613#564613]
Start a new discussion in JBoss AS7 Development at Community
[http://community.jboss.org/choose-container!input.jspa?contentType=1&...]
4964
days inactive
4986
days old
jboss-dev-forums@lists.jboss.org
10 comments
3 participants
participants (3)
-
Brian Stansberry -
Jason Greene
-
Kabir Khan