iperf UDP Packet Loss Analysis

Purpose

Analyze UDP packet loss of iperf test between 2 VMs with networking type of VPC in the same region

Description

User creates 2 VMs belonging to separate VPCs, which was connected by Express Connect in the same region. Then they use following commands to check packet loss when the bandwidth is more than 50M, and the ratio of packet loss increases with higher bandwidth.

ECS A:  iperf -c <ECS_B_IP> -u -b <bandwidth>

ECS B:  iperf -s -u

Analysis

The topo between ECS A and ECS B:

Data stream topo:

ECS A(192.168.104.235)-> NC1(100.105.59.3)-> vgw(10.141.166.253)-> NC2(100.105.59.9)-> ECS B(10.182.83.13)

Troubleshooting

• Firstly, we just think it's a simple networking packet loss issue, so we find a wrong way of packet analysis in which we confirm the communication established between 2 NCs directly without any other networking components like xgw.

• Engaged vpc developers for troubleshooting, they identified that the data stream was indeed going through the switch xgw, while the datagram was encapsulated by the IP address of target NC.

[Time ] 17:32:07.130844   Point: input
[ETHER] 24:4c:07:33:0e:02 -> 00:04:37:28:00:65, eth_type: 0x0800
[IPv4 ] 100.105.59.3 -> 10.141.166.253
proto: 17, ver: 04, ihl: 05, len: 1534, ident: 59824,R: 0, DF: 1, MF: 0, offset: 0, ttl: 60, chksum: 0xfe47
[UDP  ] sport: 46703, dport: 250, size: 1514, chksum: 0x0000
[VxLan] debug_flag: 0, vlan_tag: 0, payload_type: 0, version: 1, tunnel_id: 1878597, tos: 0, tof: 0
[IPv4 ] 192.168.104.235 -> 10.182.83.13
proto: 17, ver: 04, ihl: 05, len: 1498, ident: 55469,R: 0, DF: 1, MF: 0, offset: 0, ttl: 64, chksum: 0xd50e
[UDP  ] sport: 36687, dport: 5001, size: 1478, chksum: 0xa0aa

[Time ] 17:32:07.130854   Point: output
[ETHER] 24:4c:07:33:0e:02 -> 00:04:37:28:00:65, eth_type: 0x0800
[IPv4 ] 100.105.59.3 -> 100.105.59.9
proto: 17, ver: 04, ihl: 05, len: 1534, ident: 59824,R: 0, DF: 1, MF: 0, offset: 0, ttl: 60, chksum: 0x0000
[UDP  ] sport: 46703, dport: 250, size: 1514, chksum: 0x0000
[VxLan] debug_flag: 0, vlan_tag: 0, payload_type: 0, version: 1, tunnel_id: 2125861, tos: 0, tof: 0
[IPv4 ] 192.168.104.235 -> 10.182.83.13
proto: 17, ver: 04, ihl: 05, len: 1498, ident: 55469,R: 0, DF: 1, MF: 0, offset: 0, ttl: 64, chksum: 0xd50e
[UDP  ] sport: 36687, dport: 5001, size: 1478, chksum: 0xa0aa

• Included xgw as a part of packet capture, steps as below

ECS A send out UDP packets：iperf -c 10.182.83.13 -u -b 600m
ECS B receive UDP packets：iperf -u -s

ECS A：

sudo tcpdump -w ~/client.pcap -n -i eth0 src host  192.168.104.25 and src port 1234

ECS B:

sudo tcpdump -w ~/server.pcap -n -i eth0 src host  192.168.104.25 and src port 1234

Packet capture for eth0 of 2 NCs:

NC1: 

sudo houyi-tcpdump -w /apsara/i-6we6pnh19n2q7srkgomd.pcap -nnK -i eth0 udp and src inner_port 1234 and dst inner_host 10.182.83.13

NC2: sudo houyi-tcpdump -B 4096 -w /apsara/i-6we53i9h3ducbju5rmuw.pap -nn -i eth0  udp -K and src inner_host 192.168.104.235 and src inner_port 1234

100.105.59.3:46728 ->10.141.166.253:250

As the datagram was encapsulated by the IP address of target NC, the correct format of datagram is as below:

100.105.59.3:46728 -> 100.105.59.9:250

• Packet capture statistics

ECS A packet loss/packets sent：171/510203

NC1 eth0 packets sent：510204

ASW/LSW packets：510204

NC2 eth0 packets received：510204

ECS B packets received：510204，capture 507442， dropped by kernel 2162

• Per above statistics, we double confirmed that packet loss is inside of VM, known as protocol stack dropout. We modified the UDP buffer size from 212992（208KB）to 2097152（2MB）for target VM.

/proc/sys/net/core/rmem_default
/proc/sys/net/core/rmem_max

• After the modification, no packet loss detected

End

• Above issue is one of the scenarios of protocol stack dropout, the traditional way of troubleshooting is to use tcpdump or wireshark for packet capture, here I wanna introduce Dropwatch to diagnose the issue recommended by genius @褚霸, special thanks to my boss @铁竹 who shared following link to me.

http://blog.yufeng.info/archives/2497

• My handsome colleague @砺辛 developed a new tool for networking issue diagnose, which includes the scenario of UDP packet loss, it helps identify the root cause quickly when reproducing the issue.

https://www.atatech.org/articles/85295