我正在将以太网升级到 10 Gbps,以加快本地网络的连接速度。遗憾的是,我网络上的一台机器的 PCIe 插槽已经用完了,因此无法使用原生 10 GbE PCIe 适配器。USB 适配器是唯一可行的替代方案 - 由于 5 Gbps 以太网交换机和 USB 适配器很难找到且价格昂贵,因此决定在服务器上运行两个基于 Realtek RTL8156 芯片组的 2.5 Gbps 以太网到 USB 3 适配器,该芯片组很容易买到。然后,它们在以太网级别使用 LAG/LACP 中的链路聚合连接在一起,并连接到 2.5/10 Gbps 交换机。
Linux 可以识别这些 USB 适配器,如下所示。我还事先确定,通过为每个 NIC 分配一个 IP 地址并运行 iperf3 测试,两个适配器都可以以 2.5 Gbps 的速度独立运行。
$ dmesg
[ 5.118103] usb 4-4: new SuperSpeed USB device number 2 using xhci_hcd
[ 5.138434] usb 4-4: New USB device found, idVendor=0bda, idProduct=8156, bcdDevice=31.04
[ 5.138436] usb 4-4: New USB device strings: Mfr=1, Product=2, SerialNumber=6
[ 5.138438] usb 4-4: Product: USB 10/100/1G/2.5G LAN
[ 5.138439] usb 4-4: Manufacturer: Realtek
[ 5.138440] usb 4-4: SerialNumber: 401300ÿÿÿÿ
[ 6.970319] cdc_ncm 4-4:2.0: MAC-Address: 00:e0:4c:68:10:e6
[ 6.970325] cdc_ncm 4-4:2.0: setting rx_max = 16384
[ 6.970356] cdc_ncm 4-4:2.0: setting tx_max = 16384
[ 6.984099] cdc_ncm 4-4:2.0 eth0: register 'cdc_ncm' at usb-0000:0a:00.3-4, CDC NCM (NO ZLP), 00:e0:4c:68:10:e6
# lsusb
Bus 004 Device 002: ID 0bda:8156 Realtek Semiconductor Corp. USB 10/100/1G/2.5G LAN
Bus 002 Device 002: ID 0bda:8156 Realtek Semiconductor Corp. USB 10/100/1G/2.5G LAN
有问题的机器正在运行 Linux 虚拟机管理程序 (Proxmox),并且两个网络接口被识别为 enx00e04c680152 和 enx00e04c6810e6(我相信 udev/systemd 根据它们的 MAC 地址生成了名称)。为了将两个接口绑定在一起,我在 Proxmox 上使用了以下设置:
Linux Bond
* Name: bond0
* Autostart: Yes
* Slaves: enx00e04c680152 enx00e04c6810e6
* Mode: LACP (802.3ad)
* Hash policy: layer3+4
在交换机上,我创建了一个端口通道组,并在主动模式下启用了 LACP,并选择了两个连接到服务器的 2.5 Gbps 端口。Linux 还报告说,从它自己的角度来看,两个卡都已被检测到并绑定:
# ip link | grep enx
3: enx00e04c6810e6: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master bond0 state UP mode DEFAULT group default qlen 1000
4: enx00e04c680152: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master bond0 state UP mode DEFAULT group default qlen 1000
不幸的是,由于某种原因,无论如何,Linux 都无法与交换机建立 LACP 链路,Linux 内核一直告诉我“对于绑定中的任何适配器,链路伙伴均无 802.3ad 响应”。
# dmesg
[ 1004.491253] bond0: Warning: No 802.3ad response from the link partner for any adapters in the bond
[ 1034.527234] bond0: Warning: No 802.3ad response from the link partner for any adapters in the bond
[ 1064.547217] bond0: Warning: No 802.3ad response from the link partner for any adapters in the bond
状态文件/proc/net/bonding/bond0
显示 IEEE 802.3ad 处于活动状态,但未成功建立链接。这可以从两个 NIC 的“聚合器 ID”不同以及它们都进入“搅动”状态(这实际上意味着链接失败)这一事实中看出。
# cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v6.1.15-1-pve
Bonding Mode: IEEE 802.3ad Dynamic link aggregation
Transmit Hash Policy: layer3+4 (1)
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0
Peer Notification Delay (ms): 0
802.3ad info
LACP active: on
LACP rate: slow
Min links: 0
Aggregator selection policy (ad_select): stable
System priority: 65535
System MAC address: 8a:f5:1c:f4:8b:70
Active Aggregator Info:
Aggregator ID: 1
Number of ports: 1
Actor Key: 10
Partner Key: 1
Partner Mac Address: 00:00:00:00:00:00
Slave Interface: enx00e04c680152
MII Status: up
Speed: 2500 Mbps
Duplex: half
Link Failure Count: 0
Permanent HW addr: 00:e0:4c:68:01:52
Slave queue ID: 0
Aggregator ID: 1
Actor Churn State: none
Partner Churn State: churned
Actor Churned Count: 0
Partner Churned Count: 1
details actor lacp pdu:
system priority: 65535
system mac address: 8a:f5:1c:f4:8b:70
port key: 10
port priority: 255
port number: 1
port state: 77
details partner lacp pdu:
system priority: 65535
system mac address: 00:00:00:00:00:00
oper key: 1
port priority: 255
port number: 1
port state: 1
Slave Interface: enx00e04c6810e6
MII Status: up
Speed: 2500 Mbps
Duplex: half
Link Failure Count: 0
Permanent HW addr: 00:e0:4c:68:10:e6
Slave queue ID: 0
Aggregator ID: 2
Actor Churn State: churned
Partner Churn State: churned
Actor Churned Count: 1
Partner Churned Count: 1
details actor lacp pdu:
system priority: 65535
system mac address: 8a:f5:1c:f4:8b:70
port key: 10
port priority: 255
port number: 2
port state: 69
details partner lacp pdu:
system priority: 65535
system mac address: 00:00:00:00:00:00
oper key: 1
port priority: 255
port number: 1
port state: 1