主页 / computer / 问题 / 1857994
Accepted
比尔盖子
Asked: 2024-10-07 18:04:55 +0800 CST

无法在 Linux 上通过 LACP 绑定两个 2.5 Gbps Realtek RTL8156 以太网适配器

  • 772

我正在将以太网升级到 10 Gbps，以加快本地网络的连接速度。遗憾的是，我网络上的一台机器的 PCIe 插槽已经用完了，因此无法使用原生 10 GbE PCIe 适配器。USB 适配器是唯一可行的替代方案 - 由于 5 Gbps 以太网交换机和 USB 适配器很难找到且价格昂贵，因此决定在服务器上运行两个基于 Realtek RTL8156 芯片组的 2.5 Gbps 以太网到 USB 3 适配器，该芯片组很容易买到。然后，它们在以太网级别使用 LAG/LACP 中的链路聚合连接在一起，并连接到 2.5/10 Gbps 交换机。

Linux 可以识别这些 USB 适配器，如下所示。我还事先确定，通过为每个 NIC 分配一个 IP 地址并运行 iperf3 测试，两个适配器都可以以 2.5 Gbps 的速度独立运行。

$ dmesg
[    5.118103] usb 4-4: new SuperSpeed USB device number 2 using xhci_hcd
[    5.138434] usb 4-4: New USB device found, idVendor=0bda, idProduct=8156, bcdDevice=31.04
[    5.138436] usb 4-4: New USB device strings: Mfr=1, Product=2, SerialNumber=6
[    5.138438] usb 4-4: Product: USB 10/100/1G/2.5G LAN
[    5.138439] usb 4-4: Manufacturer: Realtek
[    5.138440] usb 4-4: SerialNumber: 401300ÿÿÿÿ
[    6.970319] cdc_ncm 4-4:2.0: MAC-Address: 00:e0:4c:68:10:e6
[    6.970325] cdc_ncm 4-4:2.0: setting rx_max = 16384
[    6.970356] cdc_ncm 4-4:2.0: setting tx_max = 16384
[    6.984099] cdc_ncm 4-4:2.0 eth0: register 'cdc_ncm' at usb-0000:0a:00.3-4, CDC NCM (NO ZLP), 00:e0:4c:68:10:e6

# lsusb
Bus 004 Device 002: ID 0bda:8156 Realtek Semiconductor Corp. USB 10/100/1G/2.5G LAN
Bus 002 Device 002: ID 0bda:8156 Realtek Semiconductor Corp. USB 10/100/1G/2.5G LAN

有问题的机器正在运行 Linux 虚拟机管理程序 (Proxmox)，并且两个网络接口被识别为 enx00e04c680152 和 enx00e04c6810e6（我相信 udev/systemd 根据它们的 MAC 地址生成了名称）。为了将两个接口绑定在一起，我在 Proxmox 上使用了以下设置：

Linux Bond
* Name: bond0
* Autostart: Yes
* Slaves: enx00e04c680152 enx00e04c6810e6
* Mode: LACP (802.3ad)
* Hash policy: layer3+4

在交换机上，我创建了一个端口通道组，并在主动模式下启用了 LACP，并选择了两个连接到服务器的 2.5 Gbps 端口。Linux 还报告说，从它自己的角度来看，两个卡都已被检测到并绑定：

# ip link | grep enx
3: enx00e04c6810e6: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master bond0 state UP mode DEFAULT group default qlen 1000
4: enx00e04c680152: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master bond0 state UP mode DEFAULT group default qlen 1000

不幸的是，由于某种原因，无论如何，Linux 都无法与交换机建立 LACP 链路，Linux 内核一直告诉我“对于绑定中的任何适配器，链路伙伴均无 802.3ad 响应”。

# dmesg
[ 1004.491253] bond0: Warning: No 802.3ad response from the link partner for any adapters in the bond
[ 1034.527234] bond0: Warning: No 802.3ad response from the link partner for any adapters in the bond
[ 1064.547217] bond0: Warning: No 802.3ad response from the link partner for any adapters in the bond

状态文件/proc/net/bonding/bond0显示 IEEE 802.3ad 处于活动状态，但未成功建立链接。这可以从两个 NIC 的“聚合器 ID”不同以及它们都进入“搅动”状态（这实际上意味着链接失败）这一事实中看出。

# cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v6.1.15-1-pve

Bonding Mode: IEEE 802.3ad Dynamic link aggregation
Transmit Hash Policy: layer3+4 (1)
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0
Peer Notification Delay (ms): 0

802.3ad info
LACP active: on
LACP rate: slow
Min links: 0
Aggregator selection policy (ad_select): stable
System priority: 65535
System MAC address: 8a:f5:1c:f4:8b:70
Active Aggregator Info:
    Aggregator ID: 1
    Number of ports: 1
    Actor Key: 10
    Partner Key: 1
    Partner Mac Address: 00:00:00:00:00:00

Slave Interface: enx00e04c680152
MII Status: up
Speed: 2500 Mbps
Duplex: half
Link Failure Count: 0
Permanent HW addr: 00:e0:4c:68:01:52
Slave queue ID: 0
Aggregator ID: 1
Actor Churn State: none
Partner Churn State: churned
Actor Churned Count: 0
Partner Churned Count: 1
details actor lacp pdu:
    system priority: 65535
    system mac address: 8a:f5:1c:f4:8b:70
    port key: 10
    port priority: 255
    port number: 1
    port state: 77
details partner lacp pdu:
    system priority: 65535
    system mac address: 00:00:00:00:00:00
    oper key: 1
    port priority: 255
    port number: 1
    port state: 1

Slave Interface: enx00e04c6810e6
MII Status: up
Speed: 2500 Mbps
Duplex: half
Link Failure Count: 0
Permanent HW addr: 00:e0:4c:68:10:e6
Slave queue ID: 0
Aggregator ID: 2
Actor Churn State: churned
Partner Churn State: churned
Actor Churned Count: 1
Partner Churned Count: 1
details actor lacp pdu:
    system priority: 65535
    system mac address: 8a:f5:1c:f4:8b:70
    port key: 10
    port priority: 255
    port number: 2
    port state: 69
details partner lacp pdu:
    system priority: 65535
    system mac address: 00:00:00:00:00:00
    oper key: 1
    port priority: 255
    port number: 1
    port state: 1
networking

1 个回答

  1. Best Answer

    总结

    这些网络适配器应切换到供应商特定模式 (bConfigurationValue: 1) 并由 Linux 驱动程序控制r8152。通用驱动程序的默认模式 (bConfigurationValue: 2)cdc_ncm 功能有限，因此并非所有功能都能正常工作。

    分析

    经过几天的故障排除，我终于找到了根本原因。线索是报告的硬件信息ethtool：

    # ethtool enx00e04c6810e6
Settings for enx00e04c6810e6:
    Supported ports: [  ]
    Supported link modes:   Not reported
    Supported pause frame use: No
    Supports auto-negotiation: No
    Supported FEC modes: Not reported
    Advertised link modes:  Not reported
    Advertised pause frame use: No
    Advertised auto-negotiation: No
    Advertised FEC modes: Not reported
    Speed: 2500Mb/s
    Duplex: Half
    Auto-negotiation: off
    Port: Twisted Pair
    PHYAD: 0
    Transceiver: internal
    MDI-X: Unknown
        Current message level: 0x00000007 (7)
                               drv probe link
    Link detected: yes

    请注意，设备驱动程序无法提供重要的硬件信息，例如自动协商或支持的链接模式。还请注意，驱动程序声称适配器以 2.5 Gbps 半双工运行，这是不寻常的。我怀疑 Linux 的bonding模块无法工作是因为硬件信息或功能支持不完整。

    经过进一步调查，我发现默认情况下，这些 Realtek 2.5 Gbps USB 适配器在通用 USB CDC-NCM 模式下运行。这是 USB 定义的标准模式，但仅提供有限的功能。要启用这些适配器的完整硬件功能，必须将它们从标准模式 (bConfigurationValue: 2) 切换到供应商特定模式 (bConfigurationValue: 1)。可以通过检查bConfigurationValue中的文件来检查当前模式/sys/bus/usb/devices/。

    lsusb首先，通过和检查 USB 硬件lsusb -t：

    # lsusb
Bus 004 Device 002: ID 0bda:8156 Realtek Semiconductor Corp. USB 10/100/1G/2.5G LAN
Bus 002 Device 002: ID 0bda:8156 Realtek Semiconductor Corp. USB 10/100/1G/2.5G LAN

# lsusb -t
/:  Bus 04.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/4p, 10000M
    |__ Port 4: Dev 2, If 0, Class=Communications, Driver=cdc_ncm, 5000M
    |__ Port 4: Dev 2, If 1, Class=CDC Data, Driver=cdc_ncm, 5000M
/:  Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/3p, 10000M
    |__ Port 2: Dev 2, If 0, Class=Communications, Driver=cdc_ncm, 5000M
    |__ Port 2: Dev 2, If 1, Class=CDC Data, Driver=cdc_ncm, 5000M

    可以看出该驱动程序cdc_ncm已被使用。

    接下来，通过检查内容 确保您在 中查看正确的设备/sys/bus/usb/devices（您需要根据 的输出调整/2-2/和） ，首先它应该是。然后，检查同一目录下的 。4-4lsusb -tmanufacturerRealtekbConfigurationValue

    # cat /sys/bus/usb/devices/2-2/manufacturer 
Realtek
# cat /sys/bus/usb/devices/2-2/bConfigurationValue 
2

# cat /sys/bus/usb/devices/4-4/manufacturer 
Realtek
# cat /sys/bus/usb/devices/4-4/bConfigurationValue 
2

    是bConfigurationValue，2表示处于 CDC_NCM 模式。要手动切换调试模式，请运行：

    # echo 1 > /sys/bus/usb/devices/2-2/bConfigurationValue 
# echo 1 > /sys/bus/usb/devices/4-4/bConfigurationValue

    切换后，Linux 将开始使用r8152驱动程序（而不是通用cdc_ncm 驱动程序）来控制它们。在 中dmesg，可以看到：

    [ 8439.270745] r8152 2-2:1.0: load rtl8156b-2 v1 04/15/21 successfully
[ 8439.397031] r8152 2-2:1.0 (unnamed net_device) (uninitialized): netif_napi_add_weight() called with weight 256
[ 8439.397202] r8152 2-2:1.0 eth0: v1.12.13
[ 8439.397220] usbcore: registered new interface driver r8152
[ 8439.399436] r8152 2-2:1.0 enx00e04c680152: renamed from eth0
[ 8505.003557] r8152 4-4:1.0: load rtl8156b-2 v1 04/15/21 successfully
[ 8505.044832] r8152 4-4:1.0 eth0: v1.12.13
[ 8505.047901] r8152 4-4:1.0 enx00e04c6810e6: renamed from eth0

    lsusb -t还将报告r8152现在负责网络适配器。

    # lsusb -t
/:  Bus 04.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/4p, 10000M
    |__ Port 4: Dev 2, If 0, Class=Vendor Specific Class, Driver=r8152, 5000M
/:  Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/3p, 10000M
    |__ Port 2: Dev 2, If 0, Class=Vendor Specific Class, Driver=r8152, 5000M

    如果适配器可以被 r8152 识别，则创建一个 udev 规则以自动应用它（请注意，此时，所有适配器都已从绑定模式分离，因此除非您手动重新创建它，否则绑定将不起作用，因此只需添加 udev 规则并重新启动就更容易了）。

    解决方案

    首先，找到网络适配器的 VID 和 PID，这可能因品牌而异。

    # lsusb
Bus 004 Device 002: ID 0bda:8156 Realtek Semiconductor Corp. USB 10/100/1G/2.5G LAN
Bus 002 Device 002: ID 0bda:8156 Realtek Semiconductor Corp. USB 10/100/1G/2.5G LAN

    然后创建以下 udev 规则：

    ACTION=="add", SUBSYSTEMS=="usb", ATTR{idVendor}=="0bda", ATTR{idProduct}=="8156", ATTR{bConfigurationValue}="1"

    将规则另存为文件，例如/etc/udev/rules.d/90-usb-r8152-ethernet.rules，然后重新启动系统。请注意，应调整此处的idVendor和idProduct以匹配 列出的 VID 和 PID lsusb。

    切换后，ethtool能够获取适配器的硬件信息：

    # ethtool enx00e04c680152 
Settings for enx00e04c680152:
    Supported ports: [ TP    MII ]
    Supported link modes:   10baseT/Half 10baseT/Full
                            100baseT/Half 100baseT/Full
                            1000baseT/Half 1000baseT/Full
                            2500baseT/Full
    Supported pause frame use: No
    Supports auto-negotiation: Yes
    Supported FEC modes: Not reported
    Advertised link modes:  10baseT/Half 10baseT/Full
                            100baseT/Half 100baseT/Full
                            1000baseT/Full
                            2500baseT/Full
    Advertised pause frame use: No
    Advertised auto-negotiation: Yes
    Advertised FEC modes: Not reported
    Link partner advertised link modes:  10baseT/Half 10baseT/Full
                                         100baseT/Half 100baseT/Full
                                         1000baseT/Full
                                         2500baseT/Full
    Link partner advertised pause frame use: No
    Link partner advertised auto-negotiation: Yes
    Link partner advertised FEC modes: Not reported
    Speed: 2500Mb/s
    Duplex: Full
    Auto-negotiation: on
    Port: MII
    PHYAD: 32
    Transceiver: internal
    Supports Wake-on: pumbg
    Wake-on: g
        Current message level: 0x00007fff (32767)
                               drv probe link timer ifdown ifup rx_err tx_err tx_queued intr tx_done rx_status pktdata hw wol
    Link detected: yes

    还可以建立 LACP：

    # cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v6.1.15-1-pve

Bonding Mode: IEEE 802.3ad Dynamic link aggregation
Transmit Hash Policy: layer3+4 (1)
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0
Peer Notification Delay (ms): 0

802.3ad info
LACP active: on
LACP rate: slow
Min links: 0
Aggregator selection policy (ad_select): stable
System priority: 65535
System MAC address: 8a:f5:1c:f4:8b:70
Active Aggregator Info:
    Aggregator ID: 1
    Number of ports: 2
    Actor Key: 11
    Partner Key: 1
    Partner Mac Address: 8c:1f:64:9a:c2:5f

Slave Interface: enx00e04c680152
MII Status: up
Speed: 2500 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 00:e0:4c:68:01:52
Slave queue ID: 0
Aggregator ID: 1
Actor Churn State: monitoring
Partner Churn State: monitoring
Actor Churned Count: 0
Partner Churned Count: 1
details actor lacp pdu:
    system priority: 65535
    system mac address: 8a:f5:1c:f4:8b:70
    port key: 11
    port priority: 255
    port number: 1
    port state: 61
details partner lacp pdu:
    system priority: 65535
    system mac address: 8c:1f:64:9a:c2:5f
    oper key: 1
    port priority: 32768
    port number: 2
    port state: 61

Slave Interface: enx00e04c6810e6
MII Status: up
Speed: 2500 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 00:e0:4c:68:10:e6
Slave queue ID: 0
Aggregator ID: 1
Actor Churn State: monitoring
Partner Churn State: monitoring
Actor Churned Count: 1
Partner Churned Count: 1
details actor lacp pdu:
    system priority: 65535
    system mac address: 8a:f5:1c:f4:8b:70
    port key: 11
    port priority: 255
    port number: 2
    port state: 61
details partner lacp pdu:
    system priority: 65535
    system mac address: 8c:1f:64:9a:c2:5f
    oper key: 1
    port priority: 32768
    port number: 3
    port state: 61
    • 35

相关问题