[undertow-dev] AjpServerRequestConduit hangs
luca boncompagni
lbonco at gmail.com
Thu Dec 21 17:29:38 EST 2017
Hi to all,
We are using undertow 1.4.0 running in wildfly-10.1.0 and we are using
mod_jk as load balencer.
Some times, during some upload, mod_jk and undertow hags-up. Both are
reding from file descriptor.
We are able to capture traffic, If you want, I can upload the full
pcap, but here you can find a decoded export from wireshark.
=================== START DUMP =======================
No. Time Source Destination
Protocol Length Info
1157 17:29:18,964405 10.248.4.81 10.248.4.99
AJP13 1212 0:REQ:Body
Frame 1157: 1212 bytes on wire (9696 bits), 1212 bytes captured (9696 bits)
Ethernet II, Src: Vmware_a6:03:b3 (00:50:56:a6:03:b3), Dst:
Vmware_88:22:48 (00:50:56:88:22:48)
Internet Protocol Version 4, Src: 10.248.4.81, Dst: 10.248.4.99
Transmission Control Protocol, Src Port: 37092, Dst Port: 8007, Seq:
1399449, Ack: 1195323, Len: 1146
Apache JServ Protocol v1.3
Magic: 1234
Length: 1142
Data:
No. Time Source Destination
Protocol Length Info
1158 17:29:18,964540 10.248.4.81 10.248.4.99
TCP 4410 37092 → 8007 [ACK] Seq=1400595 Ack=1195323 Win=342528
Len=4344 TSval=2468302997 TSecr=1035814060 [TCP segment of a
reassembled PDU]
Frame 1158: 4410 bytes on wire (35280 bits), 4410 bytes captured (35280 bits)
Ethernet II, Src: Vmware_a6:03:b3 (00:50:56:a6:03:b3), Dst:
Vmware_88:22:48 (00:50:56:88:22:48)
Internet Protocol Version 4, Src: 10.248.4.81, Dst: 10.248.4.99
Transmission Control Protocol, Src Port: 37092, Dst Port: 8007, Seq:
1400595, Ack: 1195323, Len: 4344
No. Time Source Destination
Protocol Length Info
1159 17:29:18,964563 10.248.4.99 10.248.4.81
TCP 66 8007 → 37092 [ACK] Seq=1195323 Ack=1404939 Win=472448
Len=0 TSval=1035821158 TSecr=2468302997
Frame 1159: 66 bytes on wire (528 bits), 66 bytes captured (528 bits)
Ethernet II, Src: Vmware_88:22:48 (00:50:56:88:22:48), Dst:
Vmware_a6:03:b3 (00:50:56:a6:03:b3)
Internet Protocol Version 4, Src: 10.248.4.99, Dst: 10.248.4.81
Transmission Control Protocol, Src Port: 8007, Dst Port: 37092, Seq:
1195323, Ack: 1404939, Len: 0
No. Time Source Destination
Protocol Length Info
1160 17:29:18,964584 10.248.4.81 10.248.4.99
AJP13 3914 0:REQ:Body
Frame 1160: 3914 bytes on wire (31312 bits), 3914 bytes captured (31312 bits)
Ethernet II, Src: Vmware_a6:03:b3 (00:50:56:a6:03:b3), Dst:
Vmware_88:22:48 (00:50:56:88:22:48)
Internet Protocol Version 4, Src: 10.248.4.81, Dst: 10.248.4.99
Transmission Control Protocol, Src Port: 37092, Dst Port: 8007, Seq:
1404939, Ack: 1195323, Len: 3848
[2 Reassembled TCP Segments (8192 bytes): #1158(4344), #1160(3848)]
Apache JServ Protocol v1.3
Magic: 1234
Length: 8188
Data: -----------------------------123701544520758\r\nContent-Disposition:
form-data; name="img"; filename="blob"\r\nContent-Type:
image/png\r\n\r\n\211PNG\r\n\032\n
No. Time Source Destination
Protocol Length Info
1161 17:29:18,975994 10.248.4.99 10.248.4.81
AJP13 73 0:RSP:GET BODY CHUNK
Frame 1161: 73 bytes on wire (584 bits), 73 bytes captured (584 bits)
Ethernet II, Src: Vmware_88:22:48 (00:50:56:88:22:48), Dst:
Vmware_a6:03:b3 (00:50:56:a6:03:b3)
Internet Protocol Version 4, Src: 10.248.4.99, Dst: 10.248.4.81
Transmission Control Protocol, Src Port: 8007, Dst Port: 37092, Seq:
1195323, Ack: 1408787, Len: 7
Apache JServ Protocol v1.3
Magic: 4142
Length: 3
Code: GET BODY CHUNK (6)
RLEN: 8186
No. Time Source Destination
Protocol Length Info
2123 17:29:26,465777 10.248.4.81 10.248.4.99
AJP13 3914 0:REQ:Body
Frame 2123: 3914 bytes on wire (31312 bits), 3914 bytes captured (31312 bits)
Ethernet II, Src: Vmware_a6:03:b3 (00:50:56:a6:03:b3), Dst:
Vmware_88:22:48 (00:50:56:88:22:48)
Internet Protocol Version 4, Src: 10.248.4.81, Dst: 10.248.4.99
Transmission Control Protocol, Src Port: 37092, Dst Port: 8007, Seq:
3035147, Ack: 1196716, Len: 3848
[2 Reassembled TCP Segments (8192 bytes): #2122(4344), #2123(3848)]
Apache JServ Protocol v1.3
Magic: 1234
Length: 8188
Data: [truncated]
No. Time Source Destination
Protocol Length Info
2124 17:29:26,465799 10.248.4.99 10.248.4.81
TCP 66 8007 → 37092 [ACK] Seq=1196716 Ack=3038995 Win=472448
Len=0 TSval=1035828659 TSecr=2468310498
Frame 2124: 66 bytes on wire (528 bits), 66 bytes captured (528 bits)
Ethernet II, Src: Vmware_88:22:48 (00:50:56:88:22:48), Dst:
Vmware_a6:03:b3 (00:50:56:a6:03:b3)
Internet Protocol Version 4, Src: 10.248.4.99, Dst: 10.248.4.81
Transmission Control Protocol, Src Port: 8007, Dst Port: 37092, Seq:
1196716, Ack: 3038995, Len: 0
No. Time Source Destination
Protocol Length Info
2125 17:29:26,465909 10.248.4.99 10.248.4.81
AJP13 73 0:RSP:GET BODY CHUNK
Frame 2125: 73 bytes on wire (584 bits), 73 bytes captured (584 bits)
Ethernet II, Src: Vmware_88:22:48 (00:50:56:88:22:48), Dst:
Vmware_a6:03:b3 (00:50:56:a6:03:b3)
Internet Protocol Version 4, Src: 10.248.4.99, Dst: 10.248.4.81
Transmission Control Protocol, Src Port: 8007, Dst Port: 37092, Seq:
1196716, Ack: 3038995, Len: 7
Apache JServ Protocol v1.3
Magic: 4142
Length: 3
Code: GET BODY CHUNK (6)
RLEN: 8186
No. Time Source Destination
Protocol Length Info
2126 17:29:26,502147 10.248.4.81 10.248.4.99
TCP 4410 37092 → 8007 [ACK] Seq=3038995 Ack=1196723 Win=342528
Len=4344 TSval=2468310535 TSecr=1035828659 [TCP segment of a
reassembled PDU]
Frame 2126: 4410 bytes on wire (35280 bits), 4410 bytes captured (35280 bits)
Ethernet II, Src: Vmware_a6:03:b3 (00:50:56:a6:03:b3), Dst:
Vmware_88:22:48 (00:50:56:88:22:48)
Internet Protocol Version 4, Src: 10.248.4.81, Dst: 10.248.4.99
Transmission Control Protocol, Src Port: 37092, Dst Port: 8007, Seq:
3038995, Ack: 1196723, Len: 4344
No. Time Source Destination
Protocol Length Info
2127 17:29:26,502218 10.248.4.81 10.248.4.99
AJP13 3914 0:REQ:Body
Frame 2127: 3914 bytes on wire (31312 bits), 3914 bytes captured (31312 bits)
Ethernet II, Src: Vmware_a6:03:b3 (00:50:56:a6:03:b3), Dst:
Vmware_88:22:48 (00:50:56:88:22:48)
Internet Protocol Version 4, Src: 10.248.4.81, Dst: 10.248.4.99
Transmission Control Protocol, Src Port: 37092, Dst Port: 8007, Seq:
3043339, Ack: 1196723, Len: 3848
[2 Reassembled TCP Segments (8192 bytes): #2126(4344), #2127(3848)]
Apache JServ Protocol v1.3
Magic: 1234
Length: 8188
Data [truncated]:
No. Time Source Destination
Protocol Length Info
2128 17:29:26,502232 10.248.4.99 10.248.4.81
TCP 66 8007 → 37092 [ACK] Seq=1196723 Ack=3047187 Win=472448
Len=0 TSval=1035828695 TSecr=2468310535
Frame 2128: 66 bytes on wire (528 bits), 66 bytes captured (528 bits)
Ethernet II, Src: Vmware_88:22:48 (00:50:56:88:22:48), Dst:
Vmware_a6:03:b3 (00:50:56:a6:03:b3)
Internet Protocol Version 4, Src: 10.248.4.99, Dst: 10.248.4.81
Transmission Control Protocol, Src Port: 8007, Dst Port: 37092, Seq:
1196723, Ack: 3047187, Len: 0
No. Time Source Destination
Protocol Length Info
2129 17:29:26,502450 10.248.4.99 10.248.4.81
AJP13 73 0:RSP:GET BODY CHUNK
Frame 2129: 73 bytes on wire (584 bits), 73 bytes captured (584 bits)
Ethernet II, Src: Vmware_88:22:48 (00:50:56:88:22:48), Dst:
Vmware_a6:03:b3 (00:50:56:a6:03:b3)
Internet Protocol Version 4, Src: 10.248.4.99, Dst: 10.248.4.81
Transmission Control Protocol, Src Port: 8007, Dst Port: 37092, Seq:
1196723, Ack: 3047187, Len: 7
Apache JServ Protocol v1.3
Magic: 4142
Length: 3
Code: GET BODY CHUNK (6)
RLEN: 8186
No. Time Source Destination
Protocol Length Info
2130 17:29:26,502691 10.248.4.81 10.248.4.99
TCP 4410 37092 → 8007 [ACK] Seq=3047187 Ack=1196730 Win=342528
Len=4344 TSval=2468310535 TSecr=1035828696 [TCP segment of a
reassembled PDU]
Frame 2130: 4410 bytes on wire (35280 bits), 4410 bytes captured (35280 bits)
Ethernet II, Src: Vmware_a6:03:b3 (00:50:56:a6:03:b3), Dst:
Vmware_88:22:48 (00:50:56:88:22:48)
Internet Protocol Version 4, Src: 10.248.4.81, Dst: 10.248.4.99
Transmission Control Protocol, Src Port: 37092, Dst Port: 8007, Seq:
3047187, Ack: 1196730, Len: 4344
No. Time Source Destination
Protocol Length Info
2131 17:29:26,502729 10.248.4.81 10.248.4.99
AJP13 3914 0:REQ:Body
Frame 2131: 3914 bytes on wire (31312 bits), 3914 bytes captured (31312 bits)
Ethernet II, Src: Vmware_a6:03:b3 (00:50:56:a6:03:b3), Dst:
Vmware_88:22:48 (00:50:56:88:22:48)
Internet Protocol Version 4, Src: 10.248.4.81, Dst: 10.248.4.99
Transmission Control Protocol, Src Port: 37092, Dst Port: 8007, Seq:
3051531, Ack: 1196730, Len: 3848
[2 Reassembled TCP Segments (8192 bytes): #2130(4344), #2131(3848)]
Apache JServ Protocol v1.3
Magic: 1234
Length: 8188
Data: [truncated]
No. Time Source Destination
Protocol Length Info
2132 17:29:26,502739 10.248.4.99 10.248.4.81
TCP 66 8007 → 37092 [ACK] Seq=1196730 Ack=3055379 Win=472448
Len=0 TSval=1035828696 TSecr=2468310535
Frame 2132: 66 bytes on wire (528 bits), 66 bytes captured (528 bits)
Ethernet II, Src: Vmware_88:22:48 (00:50:56:88:22:48), Dst:
Vmware_a6:03:b3 (00:50:56:a6:03:b3)
Internet Protocol Version 4, Src: 10.248.4.99, Dst: 10.248.4.81
Transmission Control Protocol, Src Port: 8007, Dst Port: 37092, Seq:
1196730, Ack: 3055379, Len: 0
No. Time Source Destination
Protocol Length Info
2133 17:29:26,502873 10.248.4.99 10.248.4.81
AJP13 73 0:RSP:GET BODY CHUNK
Frame 2133: 73 bytes on wire (584 bits), 73 bytes captured (584 bits)
Ethernet II, Src: Vmware_88:22:48 (00:50:56:88:22:48), Dst:
Vmware_a6:03:b3 (00:50:56:a6:03:b3)
Internet Protocol Version 4, Src: 10.248.4.99, Dst: 10.248.4.81
Transmission Control Protocol, Src Port: 8007, Dst Port: 37092, Seq:
1196730, Ack: 3055379, Len: 7
Apache JServ Protocol v1.3
Magic: 4142
Length: 3
Code: GET BODY CHUNK (6)
RLEN: 8186
No. Time Source Destination
Protocol Length Info
2134 17:29:26,508393 10.248.4.81 10.248.4.99
AJP13 204 0:REQ:Body
Frame 2134: 204 bytes on wire (1632 bits), 204 bytes captured (1632 bits)
Ethernet II, Src: Vmware_a6:03:b3 (00:50:56:a6:03:b3), Dst:
Vmware_88:22:48 (00:50:56:88:22:48)
Internet Protocol Version 4, Src: 10.248.4.81, Dst: 10.248.4.99
Transmission Control Protocol, Src Port: 37092, Dst Port: 8007, Seq:
3055379, Ack: 1196737, Len: 138
Apache JServ Protocol v1.3
Magic: 1234
Length: 134
Data: [truncated]
No. Time Source Destination
Protocol Length Info
2135 17:29:26,508541 10.248.4.99 10.248.4.81
AJP13 73 0:RSP:GET BODY CHUNK
Frame 2135: 73 bytes on wire (584 bits), 73 bytes captured (584 bits)
Ethernet II, Src: Vmware_88:22:48 (00:50:56:88:22:48), Dst:
Vmware_a6:03:b3 (00:50:56:a6:03:b3)
Internet Protocol Version 4, Src: 10.248.4.99, Dst: 10.248.4.81
Transmission Control Protocol, Src Port: 8007, Dst Port: 37092, Seq:
1196737, Ack: 3055517, Len: 7
Apache JServ Protocol v1.3
Magic: 4142
Length: 3
Code: GET BODY CHUNK (6)
RLEN: 8186
No. Time Source Destination
Protocol Length Info
2136 17:29:26,508751 10.248.4.81 10.248.4.99
AJP13 70 0:REQ:Body
Frame 2136: 70 bytes on wire (560 bits), 70 bytes captured (560 bits)
Ethernet II, Src: Vmware_a6:03:b3 (00:50:56:a6:03:b3), Dst:
Vmware_88:22:48 (00:50:56:88:22:48)
Internet Protocol Version 4, Src: 10.248.4.81, Dst: 10.248.4.99
Transmission Control Protocol, Src Port: 37092, Dst Port: 8007, Seq:
3055517, Ack: 1196744, Len: 4
Apache JServ Protocol v1.3
Magic: 1234
Length: 0
No. Time Source Destination
Protocol Length Info
2137 17:29:26,548349 10.248.4.99 10.248.4.81
TCP 66 8007 → 37092 [ACK] Seq=1196744 Ack=3055521 Win=472448
Len=0 TSval=1035828742 TSecr=2468310541
Frame 2137: 66 bytes on wire (528 bits), 66 bytes captured (528 bits)
Ethernet II, Src: Vmware_88:22:48 (00:50:56:88:22:48), Dst:
Vmware_a6:03:b3 (00:50:56:a6:03:b3)
Internet Protocol Version 4, Src: 10.248.4.99, Dst: 10.248.4.81
Transmission Control Protocol, Src Port: 8007, Dst Port: 37092, Seq:
1196744, Ack: 3055521, Len: 0
=================== END DUMP =======================
If I understand the protocol, undertow send two "GET BODY CHUNK" of
8186 length, bu mod_jk can send only 134 bytes, I think that the
upload finished, so undertow is waiting of filling the first "GET BIDY
CHUNK", but mod_jk is waiting for response.
Do you think that the following commit can solve my issue?
commit 13ed70fea27adba8317260ddb4b1dbdb4602471d
Author: Stuart Douglas <stuart.w.douglas at gmail.com>
Date: Sat Feb 11 12:04:19 2017 +1100
UNDERTOW-994 ajp connection hangs if a post HTTP request header
contains 'Transfer-Encoding: chunked'
Thanks,
Luca
More information about the undertow-dev
mailing list