如何调查通过环回接口连接到服务器的TCP客户端端口上的随机重置

10

我有一个测试套件，该套件在localhost上初始化服务器，然后运行成千上万的压力测试，其中包括启动和停止连接到同一计算机上此本地服务器的客户端。我正在尝试查找客户端发送到服务器的RST的原因。

这是事件的顺序：

  1. 客户端初始化SYN
  2. 服务器发回SYN，ACK
  3. 客户端发送RST（而不是ACK和建立连接）

一个观察结果是，在许多情况下，服务器运行大约99％的CPU，尽管它在几秒钟/分钟后恢复正常。

最初，我认为这可能是因为客户端使用的是先前关闭的连接中的端口，该端口处于TIME_WAIT状态，因此当服务器在该连接上发送SYN，ACK时，客户端会发出RST。尽管我对为什么客户端应该在TIME_WAIT状态下使用端口表示怀疑，但是后来我使用的是旧的fedora核心4，并认为可能没有正确的TCP / IP实现：-)。

现在，我不认为这是原因，因为围绕解决上述问题所做的所有调整都无法缓解问题。仅供参考，以下是我尝试过的更改/etc/sysctl.conf：

net.ipv4.ip_local_port_range = 1024     65535
net.ipv4.tcp_fin_timeout = 15
net.ipv4.tcp_rfc1337 = 1
net.ipv4.tcp_tw_reuse = 1

还有一点，这个问题并不一致。实际上，只有经过15到35个重新启动服务器并运行所有客户端测试的周期后，我才能重现此问题。

我没有所有可以想到的原因，所以请让我知道是否有人对可能发生的事情有任何想法。

封包详细资料

Date & Time                No.     Time        Source                Destination           Protocol Length Source port Dest port Info
2014-06-25 14:49:45.378209 1032719 1858.494749 ::1                   ::1                   TCP      94     netview-aix-5 navisphere netview-aix-5 > navisphere [SYN] Seq=0 Win=32752 Len=0 MSS=16376 SACK_PERM=1 TSval=1587252 TSecr=0 WS=128

Frame 1032719: 94 bytes on wire (752 bits), 94 bytes captured (752 bits)
    Arrival Time: Jun 25, 2014 14:49:45.378209000 EDT
    Epoch Time: 1403722185.378209000 seconds
    [Time delta from previous captured frame: 0.005893000 seconds]
    [Time delta from previous displayed frame: 0.000000000 seconds]
    [Time since reference or first frame: 1858.494749000 seconds]
    Frame Number: 1032719
    Frame Length: 94 bytes (752 bits)
    Capture Length: 94 bytes (752 bits)
    [Frame is marked: False]
    [Frame is ignored: False]
    [Protocols in frame: eth:ipv6:tcp]
    [Coloring Rule Name: TCP SYN/FIN]
    [Coloring Rule String: tcp.flags & 0x02 || tcp.flags.fin == 1]
Ethernet II, Src: 00:00:00_00:00:00 (00:00:00:00:00:00), Dst: 00:00:00_00:00:00 (00:00:00:00:00:00)
    Destination: 00:00:00_00:00:00 (00:00:00:00:00:00)
        Address: 00:00:00_00:00:00 (00:00:00:00:00:00)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
        .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
    Source: 00:00:00_00:00:00 (00:00:00:00:00:00)
        Address: 00:00:00_00:00:00 (00:00:00:00:00:00)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
        .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
    Type: IPv6 (0x86dd)
Internet Protocol Version 6, Src: ::1 (::1), Dst: ::1 (::1)
    0110 .... = Version: 6
        [0110 .... = This field makes the filter "ip.version == 6" possible: 6]
    .... 0000 0000 .... .... .... .... .... = Traffic class: 0x00000000
        .... 0000 00.. .... .... .... .... .... = Differentiated Services Field: Default (0x00000000)
        .... .... ..0. .... .... .... .... .... = ECN-Capable Transport (ECT): Not set
        .... .... ...0 .... .... .... .... .... = ECN-CE: Not set
    .... .... .... 0000 0000 0000 0000 0000 = Flowlabel: 0x00000000
    Payload length: 40
    Next header: TCP (0x06)
    Hop limit: 64
    Source: ::1 (::1)
    Destination: ::1 (::1)
Transmission Control Protocol, Src Port: netview-aix-5 (1665), Dst Port: navisphere (2162), Seq: 0, Len: 0
    Source port: netview-aix-5 (1665)
    Destination port: navisphere (2162)
    [Stream index: 3374]
    Sequence number: 0    (relative sequence number)
    Header length: 40 bytes
    Flags: 0x02 (SYN)
        000. .... .... = Reserved: Not set
        ...0 .... .... = Nonce: Not set
        .... 0... .... = Congestion Window Reduced (CWR): Not set
        .... .0.. .... = ECN-Echo: Not set
        .... ..0. .... = Urgent: Not set
        .... ...0 .... = Acknowledgement: Not set
        .... .... 0... = Push: Not set
        .... .... .0.. = Reset: Not set
        .... .... ..1. = Syn: Set
            [Expert Info (Chat/Sequence): Connection establish request (SYN): server port navisphere]
        .... .... ...0 = Fin: Not set
    Window size value: 32752
    [Calculated window size: 32752]
    Checksum: 0xf489 [validation disabled]
        [Good Checksum: False]
        [Bad Checksum: False]
    Options: (20 bytes)
        Maximum segment size: 16376 bytes
        TCP SACK Permitted Option: True
        Timestamps: TSval 1587252, TSecr 0
        No-Operation (NOP)
        Window scale: 7 (multiply by 128)

Date & Time                No.     Time        Source                Destination           Protocol Length Source port Dest port Info
2014-06-25 14:49:45.378222 1032720 1858.494762 ::1                   ::1                   TCP      94     navisphere  netview-aix-5 navisphere > netview-aix-5 [SYN, ACK] Seq=0 Ack=1 Win=32728 Len=0 MSS=16376 SACK_PERM=1 TSval=1587252 TSecr=1587252 WS=128

Frame 1032720: 94 bytes on wire (752 bits), 94 bytes captured (752 bits)
    Arrival Time: Jun 25, 2014 14:49:45.378222000 EDT
    Epoch Time: 1403722185.378222000 seconds
    [Time delta from previous captured frame: 0.000013000 seconds]
    [Time delta from previous displayed frame: 0.000013000 seconds]
    [Time since reference or first frame: 1858.494762000 seconds]
    Frame Number: 1032720
    Frame Length: 94 bytes (752 bits)
    Capture Length: 94 bytes (752 bits)
    [Frame is marked: False]
    [Frame is ignored: False]
    [Protocols in frame: eth:ipv6:tcp]
    [Coloring Rule Name: TCP SYN/FIN]
    [Coloring Rule String: tcp.flags & 0x02 || tcp.flags.fin == 1]
Ethernet II, Src: 00:00:00_00:00:00 (00:00:00:00:00:00), Dst: 00:00:00_00:00:00 (00:00:00:00:00:00)
    Destination: 00:00:00_00:00:00 (00:00:00:00:00:00)
        Address: 00:00:00_00:00:00 (00:00:00:00:00:00)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
        .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
    Source: 00:00:00_00:00:00 (00:00:00:00:00:00)
        Address: 00:00:00_00:00:00 (00:00:00:00:00:00)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
        .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
    Type: IPv6 (0x86dd)
Internet Protocol Version 6, Src: ::1 (::1), Dst: ::1 (::1)
    0110 .... = Version: 6
        [0110 .... = This field makes the filter "ip.version == 6" possible: 6]
    .... 0000 0000 .... .... .... .... .... = Traffic class: 0x00000000
        .... 0000 00.. .... .... .... .... .... = Differentiated Services Field: Default (0x00000000)
        .... .... ..0. .... .... .... .... .... = ECN-Capable Transport (ECT): Not set
        .... .... ...0 .... .... .... .... .... = ECN-CE: Not set
    .... .... .... 0000 0000 0000 0000 0000 = Flowlabel: 0x00000000
    Payload length: 40
    Next header: TCP (0x06)
    Hop limit: 64
    Source: ::1 (::1)
    Destination: ::1 (::1)
Transmission Control Protocol, Src Port: navisphere (2162), Dst Port: netview-aix-5 (1665), Seq: 0, Ack: 1, Len: 0
    Source port: navisphere (2162)
    Destination port: netview-aix-5 (1665)
    [Stream index: 3374]
    Sequence number: 0    (relative sequence number)
    Acknowledgement number: 1    (relative ack number)
    Header length: 40 bytes
    Flags: 0x12 (SYN, ACK)
        000. .... .... = Reserved: Not set
        ...0 .... .... = Nonce: Not set
        .... 0... .... = Congestion Window Reduced (CWR): Not set
        .... .0.. .... = ECN-Echo: Not set
        .... ..0. .... = Urgent: Not set
        .... ...1 .... = Acknowledgement: Set
        .... .... 0... = Push: Not set
        .... .... .0.. = Reset: Not set
        .... .... ..1. = Syn: Set
            [Expert Info (Chat/Sequence): Connection establish acknowledge (SYN+ACK): server port navisphere]
        .... .... ...0 = Fin: Not set
    Window size value: 32728
    [Calculated window size: 32728]
    Checksum: 0xf7fa [validation disabled]
        [Good Checksum: False]
        [Bad Checksum: False]
    Options: (20 bytes)
        Maximum segment size: 16376 bytes
        TCP SACK Permitted Option: True
        Timestamps: TSval 1587252, TSecr 1587252
        No-Operation (NOP)
        Window scale: 7 (multiply by 128)
    [SEQ/ACK analysis]
        [This is an ACK to the segment in frame: 1032719]
        [The RTT to ACK the segment was: 0.000013000 seconds]

Date & Time                No.     Time        Source                Destination           Protocol Length Source port Dest port Info
2014-06-25 14:49:45.378228 1032721 1858.494768 ::1                   ::1                   TCP      74     netview-aix-5 navisphere netview-aix-5 > navisphere [RST] Seq=1 Win=0 Len=0

Frame 1032721: 74 bytes on wire (592 bits), 74 bytes captured (592 bits)
    Arrival Time: Jun 25, 2014 14:49:45.378228000 EDT
    Epoch Time: 1403722185.378228000 seconds
    [Time delta from previous captured frame: 0.000006000 seconds]
    [Time delta from previous displayed frame: 0.000006000 seconds]
    [Time since reference or first frame: 1858.494768000 seconds]
    Frame Number: 1032721
    Frame Length: 74 bytes (592 bits)
    Capture Length: 74 bytes (592 bits)
    [Frame is marked: False]
    [Frame is ignored: False]
    [Protocols in frame: eth:ipv6:tcp]
    [Coloring Rule Name: TCP RST]
    [Coloring Rule String: tcp.flags.reset eq 1]
Ethernet II, Src: 00:00:00_00:00:00 (00:00:00:00:00:00), Dst: 00:00:00_00:00:00 (00:00:00:00:00:00)
    Destination: 00:00:00_00:00:00 (00:00:00:00:00:00)
        Address: 00:00:00_00:00:00 (00:00:00:00:00:00)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
        .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
    Source: 00:00:00_00:00:00 (00:00:00:00:00:00)
        Address: 00:00:00_00:00:00 (00:00:00:00:00:00)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
        .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
    Type: IPv6 (0x86dd)
Internet Protocol Version 6, Src: ::1 (::1), Dst: ::1 (::1)
    0110 .... = Version: 6
        [0110 .... = This field makes the filter "ip.version == 6" possible: 6]
    .... 0000 0000 .... .... .... .... .... = Traffic class: 0x00000000
        .... 0000 00.. .... .... .... .... .... = Differentiated Services Field: Default (0x00000000)
        .... .... ..0. .... .... .... .... .... = ECN-Capable Transport (ECT): Not set
        .... .... ...0 .... .... .... .... .... = ECN-CE: Not set
    .... .... .... 0000 0000 0000 0000 0000 = Flowlabel: 0x00000000
    Payload length: 20
    Next header: TCP (0x06)
    Hop limit: 64
    Source: ::1 (::1)
    Destination: ::1 (::1)
Transmission Control Protocol, Src Port: netview-aix-5 (1665), Dst Port: navisphere (2162), Seq: 1, Len: 0
    Source port: netview-aix-5 (1665)
    Destination port: navisphere (2162)
    [Stream index: 3374]
    Sequence number: 1    (relative sequence number)
    Header length: 20 bytes
    Flags: 0x04 (RST)
        000. .... .... = Reserved: Not set
        ...0 .... .... = Nonce: Not set
        .... 0... .... = Congestion Window Reduced (CWR): Not set
        .... .0.. .... = ECN-Echo: Not set
        .... ..0. .... = Urgent: Not set
        .... ...0 .... = Acknowledgement: Not set
        .... .... 0... = Push: Not set
        .... .... .1.. = Reset: Set
            [Expert Info (Chat/Sequence): Connection reset (RST)]
        .... .... ..0. = Syn: Not set
        .... .... ...0 = Fin: Not set
    Window size value: 0
    [Calculated window size: 0]
    [Window size scaling factor: 128]
    Checksum: 0x4eea [validation disabled]
        [Good Checksum: False]
        [Bad Checksum: False]

Date & Time                No.     Time        Source                Destination           Protocol Length Source port Dest port Info
2014-06-25 14:49:48.375927 1032722 1861.492467 ::1                   ::1                   TCP      94     netview-aix-5 navisphere netview-aix-5 > navisphere [SYN] Seq=0 Win=32752 Len=0 MSS=16376 SACK_PERM=1 TSval=1588002 TSecr=0 WS=128

Frame 1032722: 94 bytes on wire (752 bits), 94 bytes captured (752 bits)
    Arrival Time: Jun 25, 2014 14:49:48.375927000 EDT
    Epoch Time: 1403722188.375927000 seconds
    [Time delta from previous captured frame: 2.997699000 seconds]
    [Time delta from previous displayed frame: 2.997699000 seconds]
    [Time since reference or first frame: 1861.492467000 seconds]
    Frame Number: 1032722
    Frame Length: 94 bytes (752 bits)
    Capture Length: 94 bytes (752 bits)
    [Frame is marked: False]
    [Frame is ignored: False]
    [Protocols in frame: eth:ipv6:tcp]
    [Coloring Rule Name: TCP SYN/FIN]
    [Coloring Rule String: tcp.flags & 0x02 || tcp.flags.fin == 1]
Ethernet II, Src: 00:00:00_00:00:00 (00:00:00:00:00:00), Dst: 00:00:00_00:00:00 (00:00:00:00:00:00)
    Destination: 00:00:00_00:00:00 (00:00:00:00:00:00)
        Address: 00:00:00_00:00:00 (00:00:00:00:00:00)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
        .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
    Source: 00:00:00_00:00:00 (00:00:00:00:00:00)
        Address: 00:00:00_00:00:00 (00:00:00:00:00:00)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
        .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
    Type: IPv6 (0x86dd)
Internet Protocol Version 6, Src: ::1 (::1), Dst: ::1 (::1)
    0110 .... = Version: 6
        [0110 .... = This field makes the filter "ip.version == 6" possible: 6]
    .... 0000 0000 .... .... .... .... .... = Traffic class: 0x00000000
        .... 0000 00.. .... .... .... .... .... = Differentiated Services Field: Default (0x00000000)
        .... .... ..0. .... .... .... .... .... = ECN-Capable Transport (ECT): Not set
        .... .... ...0 .... .... .... .... .... = ECN-CE: Not set
    .... .... .... 0000 0000 0000 0000 0000 = Flowlabel: 0x00000000
    Payload length: 40
    Next header: TCP (0x06)
    Hop limit: 64
    Source: ::1 (::1)
    Destination: ::1 (::1)
Transmission Control Protocol, Src Port: netview-aix-5 (1665), Dst Port: navisphere (2162), Seq: 0, Len: 0
    Source port: netview-aix-5 (1665)
    Destination port: navisphere (2162)
    [Stream index: 3374]
    Sequence number: 0    (relative sequence number)
    Header length: 40 bytes
    Flags: 0x02 (SYN)
        000. .... .... = Reserved: Not set
        ...0 .... .... = Nonce: Not set
        .... 0... .... = Congestion Window Reduced (CWR): Not set
        .... .0.. .... = ECN-Echo: Not set
        .... ..0. .... = Urgent: Not set
        .... ...0 .... = Acknowledgement: Not set
        .... .... 0... = Push: Not set
        .... .... .0.. = Reset: Not set
        .... .... ..1. = Syn: Set
            [Expert Info (Chat/Sequence): Connection establish request (SYN): server port navisphere]
        .... .... ...0 = Fin: Not set
    Window size value: 32752
    [Calculated window size: 32752]
    Checksum: 0xf19b [validation disabled]
        [Good Checksum: False]
        [Bad Checksum: False]
    Options: (20 bytes)
        Maximum segment size: 16376 bytes
        TCP SACK Permitted Option: True
        Timestamps: TSval 1588002, TSecr 0
        No-Operation (NOP)
        Window scale: 7 (multiply by 128)

Date & Time                No.     Time        Source                Destination           Protocol Length Source port Dest port Info
2014-06-25 14:49:48.375950 1032723 1861.492490 ::1                   ::1                   TCP      94     navisphere  netview-aix-5 [TCP Previous segment lost] navisphere > netview-aix-5 [SYN, ACK] Seq=2997725 Ack=1 Win=32728 Len=0 MSS=16376 SACK_PERM=1 TSval=1588002 TSecr=1588002 WS=128

Frame 1032723: 94 bytes on wire (752 bits), 94 bytes captured (752 bits)
    Arrival Time: Jun 25, 2014 14:49:48.375950000 EDT
    Epoch Time: 1403722188.375950000 seconds
    [Time delta from previous captured frame: 0.000023000 seconds]
    [Time delta from previous displayed frame: 0.000023000 seconds]
    [Time since reference or first frame: 1861.492490000 seconds]
    Frame Number: 1032723
    Frame Length: 94 bytes (752 bits)
    Capture Length: 94 bytes (752 bits)
    [Frame is marked: False]
    [Frame is ignored: False]
    [Protocols in frame: eth:ipv6:tcp]
    [Coloring Rule Name: Bad TCP]
    [Coloring Rule String: tcp.analysis.flags]
Ethernet II, Src: 00:00:00_00:00:00 (00:00:00:00:00:00), Dst: 00:00:00_00:00:00 (00:00:00:00:00:00)
    Destination: 00:00:00_00:00:00 (00:00:00:00:00:00)
        Address: 00:00:00_00:00:00 (00:00:00:00:00:00)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
        .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
    Source: 00:00:00_00:00:00 (00:00:00:00:00:00)
        Address: 00:00:00_00:00:00 (00:00:00:00:00:00)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
        .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
    Type: IPv6 (0x86dd)
Internet Protocol Version 6, Src: ::1 (::1), Dst: ::1 (::1)
    0110 .... = Version: 6
        [0110 .... = This field makes the filter "ip.version == 6" possible: 6]
    .... 0000 0000 .... .... .... .... .... = Traffic class: 0x00000000
        .... 0000 00.. .... .... .... .... .... = Differentiated Services Field: Default (0x00000000)
        .... .... ..0. .... .... .... .... .... = ECN-Capable Transport (ECT): Not set
        .... .... ...0 .... .... .... .... .... = ECN-CE: Not set
    .... .... .... 0000 0000 0000 0000 0000 = Flowlabel: 0x00000000
    Payload length: 40
    Next header: TCP (0x06)
    Hop limit: 64
    Source: ::1 (::1)
    Destination: ::1 (::1)
Transmission Control Protocol, Src Port: navisphere (2162), Dst Port: netview-aix-5 (1665), Seq: 2997725, Ack: 1, Len: 0
    Source port: navisphere (2162)
    Destination port: netview-aix-5 (1665)
    [Stream index: 3374]
    Sequence number: 2997725    (relative sequence number)
    Acknowledgement number: 1    (relative ack number)
    Header length: 40 bytes
    Flags: 0x12 (SYN, ACK)
        000. .... .... = Reserved: Not set
        ...0 .... .... = Nonce: Not set
        .... 0... .... = Congestion Window Reduced (CWR): Not set
        .... .0.. .... = ECN-Echo: Not set
        .... ..0. .... = Urgent: Not set
        .... ...1 .... = Acknowledgement: Set
        .... .... 0... = Push: Not set
        .... .... .0.. = Reset: Not set
        .... .... ..1. = Syn: Set
            [Expert Info (Chat/Sequence): Connection establish acknowledge (SYN+ACK): server port navisphere]
        .... .... ...0 = Fin: Not set
    Window size value: 32728
    [Calculated window size: 32728]
    Checksum: 0x3414 [validation disabled]
        [Good Checksum: False]
        [Bad Checksum: False]
    Options: (20 bytes)
        Maximum segment size: 16376 bytes
        TCP SACK Permitted Option: True
        Timestamps: TSval 1588002, TSecr 1588002
        No-Operation (NOP)
        Window scale: 7 (multiply by 128)
    [SEQ/ACK analysis]
        [This is an ACK to the segment in frame: 1032722]
        [The RTT to ACK the segment was: 0.000023000 seconds]
        [TCP Analysis Flags]
            [A segment before this frame was lost]
                [Expert Info (Warn/Sequence): Previous segment lost (common at capture start)]

Date & Time                No.     Time        Source                Destination           Protocol Length Source port Dest port Info
2014-06-25 14:49:48.375958 1032724 1861.492498 ::1                   ::1                   TCP      74     netview-aix-5 navisphere netview-aix-5 > navisphere [RST] Seq=1 Win=0 Len=0

Frame 1032724: 74 bytes on wire (592 bits), 74 bytes captured (592 bits)
    Arrival Time: Jun 25, 2014 14:49:48.375958000 EDT
    Epoch Time: 1403722188.375958000 seconds
    [Time delta from previous captured frame: 0.000008000 seconds]
    [Time delta from previous displayed frame: 0.000008000 seconds]
    [Time since reference or first frame: 1861.492498000 seconds]
    Frame Number: 1032724
    Frame Length: 74 bytes (592 bits)
    Capture Length: 74 bytes (592 bits)
    [Frame is marked: False]
    [Frame is ignored: False]
    [Protocols in frame: eth:ipv6:tcp]
    [Coloring Rule Name: TCP RST]
    [Coloring Rule String: tcp.flags.reset eq 1]
Ethernet II, Src: 00:00:00_00:00:00 (00:00:00:00:00:00), Dst: 00:00:00_00:00:00 (00:00:00:00:00:00)
    Destination: 00:00:00_00:00:00 (00:00:00:00:00:00)
        Address: 00:00:00_00:00:00 (00:00:00:00:00:00)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
        .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
    Source: 00:00:00_00:00:00 (00:00:00:00:00:00)
        Address: 00:00:00_00:00:00 (00:00:00:00:00:00)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
        .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
    Type: IPv6 (0x86dd)
Internet Protocol Version 6, Src: ::1 (::1), Dst: ::1 (::1)
    0110 .... = Version: 6
        [0110 .... = This field makes the filter "ip.version == 6" possible: 6]
    .... 0000 0000 .... .... .... .... .... = Traffic class: 0x00000000
        .... 0000 00.. .... .... .... .... .... = Differentiated Services Field: Default (0x00000000)
        .... .... ..0. .... .... .... .... .... = ECN-Capable Transport (ECT): Not set
        .... .... ...0 .... .... .... .... .... = ECN-CE: Not set
    .... .... .... 0000 0000 0000 0000 0000 = Flowlabel: 0x00000000
    Payload length: 20
    Next header: TCP (0x06)
    Hop limit: 64
    Source: ::1 (::1)
    Destination: ::1 (::1)
Transmission Control Protocol, Src Port: netview-aix-5 (1665), Dst Port: navisphere (2162), Seq: 1, Len: 0
    Source port: netview-aix-5 (1665)
    Destination port: navisphere (2162)
    [Stream index: 3374]
    Sequence number: 1    (relative sequence number)
    Header length: 20 bytes
    Flags: 0x04 (RST)
        000. .... .... = Reserved: Not set
        ...0 .... .... = Nonce: Not set
        .... 0... .... = Congestion Window Reduced (CWR): Not set
        .... .0.. .... = ECN-Echo: Not set
        .... ..0. .... = Urgent: Not set
        .... ...0 .... = Acknowledgement: Not set
        .... .... 0... = Push: Not set
        .... .... .1.. = Reset: Set
            [Expert Info (Chat/Sequence): Connection reset (RST)]
        .... .... ..0. = Syn: Not set
        .... .... ...0 = Fin: Not set
    Window size value: 0
    [Calculated window size: 0]
    [Window size scaling factor: 128]
    Checksum: 0x4eea [validation disabled]
        [Good Checksum: False]
        [Bad Checksum: False]

Date & Time                No.     Time        Source                Destination           Protocol Length Source port Dest port Info
2014-06-25 14:49:54.376072 1032725 1867.492612 ::1                   ::1                   TCP      94     netview-aix-5 navisphere netview-aix-5 > navisphere [SYN] Seq=0 Win=32752 Len=0 MSS=16376 SACK_PERM=1 TSval=1589502 TSecr=0 WS=128

Frame 1032725: 94 bytes on wire (752 bits), 94 bytes captured (752 bits)
    Arrival Time: Jun 25, 2014 14:49:54.376072000 EDT
    Epoch Time: 1403722194.376072000 seconds
    [Time delta from previous captured frame: 6.000114000 seconds]
    [Time delta from previous displayed frame: 6.000114000 seconds]
    [Time since reference or first frame: 1867.492612000 seconds]
    Frame Number: 1032725
    Frame Length: 94 bytes (752 bits)
    Capture Length: 94 bytes (752 bits)
    [Frame is marked: False]
    [Frame is ignored: False]
    [Protocols in frame: eth:ipv6:tcp]
    [Coloring Rule Name: TCP SYN/FIN]
    [Coloring Rule String: tcp.flags & 0x02 || tcp.flags.fin == 1]
Ethernet II, Src: 00:00:00_00:00:00 (00:00:00:00:00:00), Dst: 00:00:00_00:00:00 (00:00:00:00:00:00)
    Destination: 00:00:00_00:00:00 (00:00:00:00:00:00)
        Address: 00:00:00_00:00:00 (00:00:00:00:00:00)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
        .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
    Source: 00:00:00_00:00:00 (00:00:00:00:00:00)
        Address: 00:00:00_00:00:00 (00:00:00:00:00:00)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
        .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
    Type: IPv6 (0x86dd)
Internet Protocol Version 6, Src: ::1 (::1), Dst: ::1 (::1)
    0110 .... = Version: 6
        [0110 .... = This field makes the filter "ip.version == 6" possible: 6]
    .... 0000 0000 .... .... .... .... .... = Traffic class: 0x00000000
        .... 0000 00.. .... .... .... .... .... = Differentiated Services Field: Default (0x00000000)
        .... .... ..0. .... .... .... .... .... = ECN-Capable Transport (ECT): Not set
        .... .... ...0 .... .... .... .... .... = ECN-CE: Not set
    .... .... .... 0000 0000 0000 0000 0000 = Flowlabel: 0x00000000
    Payload length: 40
    Next header: TCP (0x06)
    Hop limit: 64
    Source: ::1 (::1)
    Destination: ::1 (::1)
Transmission Control Protocol, Src Port: netview-aix-5 (1665), Dst Port: navisphere (2162), Seq: 0, Len: 0
    Source port: netview-aix-5 (1665)
    Destination port: navisphere (2162)
    [Stream index: 3374]
    Sequence number: 0    (relative sequence number)
    Header length: 40 bytes
    Flags: 0x02 (SYN)
        000. .... .... = Reserved: Not set
        ...0 .... .... = Nonce: Not set
        .... 0... .... = Congestion Window Reduced (CWR): Not set
        .... .0.. .... = ECN-Echo: Not set
        .... ..0. .... = Urgent: Not set
        .... ...0 .... = Acknowledgement: Not set
        .... .... 0... = Push: Not set
        .... .... .0.. = Reset: Not set
        .... .... ..1. = Syn: Set
            [Expert Info (Chat/Sequence): Connection establish request (SYN): server port navisphere]
        .... .... ...0 = Fin: Not set
    Window size value: 32752
    [Calculated window size: 32752]
    Checksum: 0xebbf [validation disabled]
        [Good Checksum: False]
        [Bad Checksum: False]
    Options: (20 bytes)
        Maximum segment size: 16376 bytes
        TCP SACK Permitted Option: True
        Timestamps: TSval 1589502, TSecr 0
        No-Operation (NOP)
        Window scale: 7 (multiply by 128)

iptables-save 输出：

# Generated by iptables-save v1.3.0 on Thu Jun 26 10:15:33 2014
*filter
:FORWARD ACCEPT [0:0]
:INPUT ACCEPT [675:236200]
:OUTPUT ACCEPT [241:21540]
COMMIT
# Completed on Thu Jun 26 10:15:33 2014
fedora  tcp  tcpdump  tcp-ip 
— 卡罗莱纳州
source
客户端的套接字是否处于非阻塞模式？服务器接受时是否有可能关闭客户端套接字？您有这些数据包的时间安排吗？一个pcap？
— 斯特凡Chazelas
1
您是否检查的返回值connect()s？您是否有可能connect()s在同一套接字上执行两个非阻塞操作？是否有可能某些客户在某个alarm()中间被杀死或中断（？）connect()？您可以在运行strace -ff或的客户端上重现它autrace吗？
— 斯特凡Chazelas
1
在输出中iptables-save是否有任何花哨的内容（nfqueue，连接限制...）？是否可能达到某些资源限制？有多少个并发客户端？那些bind到特定的源端口吗？
— 斯特凡Chazelas
1
最好将其添加到问题中。问题是测试中是否达到（或可能达到）这些限制，而不是实际达到或达到的极限。
— 斯特凡Chazelas
1
我的猜测是，施加在堆栈上的压力有一个连接在半开放状态的超时客户端下降。该连接被删除（这意味着客户端应用程序应将其视为ETIMEDOUT），并且当堆栈最终处理了syn / ack时，不再有与​​其相关的连接，因此将其重置。您是否尝试增加syn积压（sysctl -w net.ipv4.tcp_max_syn_backlog=???）的大小？
— didierc 2014年

Answers:

1

只是想在没有解决根本原因的情况下以采用的解决方案来结束该线程，我猜想这是在Fedora 4 TCP堆栈实现中。作为解决方案，我仅介绍了一次立即重新连接尝试，以防万一故障是由于ETIMEDOUT引起的，并且使用的协议是IPV6。这已经为我和我的团队永久性地解决了该问题，在任何其他导致ETIMEDOUT的连接问题中报告错误有3分钟的延迟风险。这不是解决问题的真正/理想的解决方案，但是可以使我们继续前进……因为这只会影响我们的自动化测试套件，而不会交付给客户。我希望最终某个对fedora 4 tcp / ip堆栈实现足够了解的人能够永远解决这个难题。

— 卡罗莱纳州
source
By using our site, you acknowledge that you have read and understand our Cookie Policy and Privacy Policy.
Licensed under cc by-sa 3.0 with attribution required.