我们目前有一个 kubernetes 集群（使用 calico），其中所有工作节点都通过 lacp 绑定连接在一起（2x10GB 链接，LACP 模式 4 - 802.3ad）。
但是，我们设法在 2 个 pod 之间获得的最大吞吐量是 10 GB/s。根据一些文档，我们应该能够达到 20 GB/s。根据wikipedia，这似乎是一种正常行为：
这将分别为每个目标 MAC 地址、IP 地址或 IP 地址和端口组合选择相同的 NIC 从站
这导致理解只有一个链接用于给定的目标 IP + 端口组合。
我的问题如下：我们是否应该能够在 pod 之间达到 20 GB/s 的吞吐量？或者我们卡在 10 GB/s 是正常的吗？
我有全新安装的 Ubuntu、全新安装的 k3s 和全新下载的 calicoctl。我已经按照以下方式安装了它。
curl -sfL https://get.k3s.io | K3S_KUBECONFIG_MODE="644"\
INSTALL_K3S_EXEC="--flannel-backend=none --cluster-cidr=192.168.0.0/16\
--disable-network-policy --disable=traefik" sh -
kubectl create -f https://docs.projectcalico.org/manifests/tigera-operator.yaml
kubectl create -f https://docs.projectcalico.org/manifests/custom-resources.yaml
curl -o calicoctl -O -L "https://github.com/projectcalico/calicoctl/releases/download/v3.20.2/calicoctl"
当我运行 kubectl 时，一切正常。当我运行 calicoctl 时，出现证书错误。
# calicoctl apply -f V000_000-host-policy.yaml
Unable to get Cluster Information to verify version mismatch: Get "https://127.0.0.1:6443/apis/crd.projectcalico.org/v1/clusterinformations/default": x509: certificate signed by unknown authority
Use --allow-version-mismatch to override.
我已经从to复制request-header-ca.crt和证书client-ca.crt并将它们应用于。我可以确认证书列在.server-ca.crt/var/lib/rancher/k3s/server/tls/usr/local/share/ca-certificatesupdate-ca-certificates/etc/ssl/certs/ca-certificates.crt
此外，我的~/.kube/config文件具有以下内容（我会定期重新安装，我希望这些都不是机密的 - 如果我错了，请纠正我）
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: LS0t...LS0K
server: https://127.0.0.1:6443
name: default
contexts:
- context:
cluster: default
user: default
name: default
current-context: default
kind: Config
preferences: {}
users:
- name: default
user:
client-certificate-data: LS0t...LS0K
client-key-data: LS0t...LQo=
我有以下配置/etc/cni/net.d/calico-kubeconfig
# Kubeconfig file for Calico CNI plugin. Installed by calico/node.
apiVersion: v1
kind: Config
clusters:
- name: local
cluster:
server: https://10.43.0.1:443
certificate-authority-data: "LS0t...tLS0K"
users:
- name: calico
user:
token: eyJhb...tk4Q
contexts:
- name: calico-context
context:
cluster: local
user: calico
current-context: calico-context
我已将 calico-kubeconfig 中的地址从 更改为10.43.0.1:443，127.0.0.1:6443但这没有任何区别。
有谁知道如何解决这个问题？我看到的证书错误是 CA 或令牌的结果吗？卷曲到同一个地址也抱怨 CA，所以这让我认为这与令牌无关。
我的网络设置：
使用此设置，只有同一子网上的节点可以建立 bgp 连接。其他节点（执行完整的 3 路 tcp 握手），使用 [FIN, ACK] 然后 [RST] 响应 hte OPEN 消息，因此我的calicoctl node status<- 中的对等消息重置连接在控制器 3 上（10.0.3.100）
IPv4 BGP status
+--------------+-------------------+-------+----------+--------------------------------+
| PEER ADDRESS | PEER TYPE | STATE | SINCE | INFO |
+--------------+-------------------+-------+----------+--------------------------------+
| 10.0.1.100 | node-to-node mesh | start | 07:12:01 | Connect Socket: Connection |
| | | | | closed |
| 10.0.2.100 | node-to-node mesh | start | 07:12:01 | Connect |
| 10.0.1.101 | node-to-node mesh | start | 07:12:01 | Connect Socket: Connection |
| | | | | reset by peer |
| 10.0.1.102 | node-to-node mesh | start | 07:12:01 | Connect Socket: Connection |
| | | | | reset by peer |
| 10.0.2.102 | node-to-node mesh | start | 07:12:01 | Connect Socket: Connection |
| | | | | reset by peer |
| 10.0.3.101 | node-to-node mesh | up | 07:14:13 | Established |
| 10.0.3.102 | node-to-node mesh | up | 07:12:02 | Established |
+--------------+-------------------+-------+----------+--------------------------------+
我从控制器 3 (10.0.3.100) 到 node4 (10.0.2.102) 的握手 + OPEN 消息的wireshark 转储
10.0.3.100 和 10.0.2.102 之间的
Wireshark bgp 跟踪 10.0.0.4(10.0.3.100) 和 10.0.2.102 之间的 Wireshark bgp 跟踪
也许问题是节点 4 看到的数据来自 10.0.0.4 而不是 10.0.3.100？
我已经安装了 kubernetes master 和一个 node v 1.20。我部署了 nginx
kubectl run nginxpod --image=nginx
$ kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginxpod 1/1 Running 0 19s 192.168.2.195 xps15-9560 <none> <none>
当我在 master 上卷曲时，在 master 上超时：
$ curl 192.168.2.195
curl: (7) Failed to connect to 192.168.2.195 port 80: Connection timed out
在它工作的节点上。我已经尝试过我网络上的其他主机，但它们也超时了。为什么我只能从 Pod 实际运行的节点进行连接？
- - 编辑 - -
calico 节点正在运行，但尚未准备好。我不知道这是什么意思：
$ kubectl get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
default nginxpod 1/1 Running 0 64m
kube-system calico-kube-controllers-5f6cfd688c-wk5jp 1/1 Running 0 69m
kube-system calico-node-t47kf 0/1 Running 0 45m
kube-system calico-node-vqj6m 0/1 Running 0 68m
kube-system calico-node-wzwzb 0/1 Running 0 69m
kube-system coredns-74ff55c5b-mb2vj 1/1 Running 0 69m
kube-system coredns-74ff55c5b-pvsgz 1/1 Running 0 69m
kube-system etcd-ubuntu-18-extssd 1/1 Running 0 69m
kube-system kube-apiserver-ubuntu-18-extssd 1/1 Running 0 69m
kube-system kube-controller-manager-ubuntu-18-extssd 1/1 Running 0 69m
kube-system kube-proxy-5fq9b 1/1 Running 0 68m
kube-system kube-proxy-bxhfm 1/1 Running 0 69m
kube-system kube-proxy-pp9sb 1/1 Running 0 45m
kube-system kube-scheduler-ubuntu-18-extssd 1/1 Running 0 69m
--------编辑 2-----------------
印花布节点未找到：
$ kubectl describe pod calico-node-t47kf
Error from server (NotFound): pods "calico-node-t47kf" not found
$ kubectl describe pod calico-node-vqj6m
Error from server (NotFound): pods "calico-node-vqj6m" not found
$ kubectl describe pod calico-node-*****
Error from server (NotFound): pods "calico-node-*****" not found
----编辑 3---- 描述 pod calico-node-t47kf 的输出：
$ kubectl -n kube-system describe pod calico-node-t47kf
Name: calico-node-t47kf
Namespace: kube-system
Priority: 2000001000
Priority Class Name: system-node-critical
Node: xps15-9560/192.168.0.71
Start Time: Sun, 05 Sep 2021 16:51:37 -0600
Labels: controller-revision-hash=b8998dcb
k8s-app=calico-node
pod-template-generation=1
Annotations: <none>
Status: Running
IP: 192.168.0.71
IPs:
IP: 192.168.0.71
Controlled By: DaemonSet/calico-node
Init Containers:
upgrade-ipam:
Container ID: docker://3d393316548badf75bb2c2ad881ffd7a4d2c37a1762d84ec973715c8a398072e
Image: docker.io/calico/cni:v3.20.0
Image ID: docker-pullable://calico/cni@sha256:9906e2cca8006e1fe9fc3f358a3a06da6253afdd6fad05d594e884e8298ffe1d
Port: <none>
Host Port: <none>
Command:
/opt/cni/bin/calico-ipam
-upgrade
State: Terminated
Reason: Completed
Exit Code: 0
Started: Sun, 05 Sep 2021 16:51:38 -0600
Finished: Sun, 05 Sep 2021 16:51:38 -0600
Ready: True
Restart Count: 0
Environment Variables from:
kubernetes-services-endpoint ConfigMap Optional: true
Environment:
KUBERNETES_NODE_NAME: (v1:spec.nodeName)
CALICO_NETWORKING_BACKEND: <set to the key 'calico_backend' of config map 'calico-config'> Optional: false
Mounts:
/host/opt/cni/bin from cni-bin-dir (rw)
/var/lib/cni/networks from host-local-net-dir (rw)
/var/run/secrets/kubernetes.io/serviceaccount from calico-node-token-qfrcw (ro)
install-cni:
Container ID: docker://9e3b23f12657fe343117ec0cf54e104a8eb69c6133fe2dd10c1aabbc9260189f
Image: docker.io/calico/cni:v3.20.0
Image ID: docker-pullable://calico/cni@sha256:9906e2cca8006e1fe9fc3f358a3a06da6253afdd6fad05d594e884e8298ffe1d
Port: <none>
Host Port: <none>
Command:
/opt/cni/bin/install
State: Terminated
Reason: Completed
Exit Code: 0
Started: Sun, 05 Sep 2021 16:51:39 -0600
Finished: Sun, 05 Sep 2021 16:51:40 -0600
Ready: True
Restart Count: 0
Environment Variables from:
kubernetes-services-endpoint ConfigMap Optional: true
Environment:
CNI_CONF_NAME: 10-calico.conflist
CNI_NETWORK_CONFIG: <set to the key 'cni_network_config' of config map 'calico-config'> Optional: false
KUBERNETES_NODE_NAME: (v1:spec.nodeName)
CNI_MTU: <set to the key 'veth_mtu' of config map 'calico-config'> Optional: false
SLEEP: false
Mounts:
/host/etc/cni/net.d from cni-net-dir (rw)
/host/opt/cni/bin from cni-bin-dir (rw)
/var/run/secrets/kubernetes.io/serviceaccount from calico-node-token-qfrcw (ro)
flexvol-driver:
Container ID: docker://090008276cf33c2fd64aa141405e90feff00fc70f55769372655e45bf2e2dc92
Image: docker.io/calico/pod2daemon-flexvol:v3.20.0
Image ID: docker-pullable://calico/pod2daemon-flexvol@sha256:c17e3e9871682bed00bfd33f8d6f00db1d1a126034a25bf5380355978e0c548d
Port: <none>
Host Port: <none>
State: Terminated
Reason: Completed
Exit Code: 0
Started: Sun, 05 Sep 2021 16:51:40 -0600
Finished: Sun, 05 Sep 2021 16:51:40 -0600
Ready: True
Restart Count: 0
Environment: <none>
Mounts:
/host/driver from flexvol-driver-host (rw)
/var/run/secrets/kubernetes.io/serviceaccount from calico-node-token-qfrcw (ro)
Containers:
calico-node:
Container ID: docker://2c1456bb2f346564f88b3e8deba3baef1669f0aa61aa0e654d52c915eaf462cf
Image: docker.io/calico/node:v3.20.0
Image ID: docker-pullable://calico/node@sha256:7f9aa7e31fbcea7be64b153f8bcfd494de023679ec10d851a05667f0adb42650
Port: <none>
Host Port: <none>
State: Running
Started: Sun, 05 Sep 2021 16:51:41 -0600
Ready: False
Restart Count: 0
Requests:
cpu: 250m
Liveness: exec [/bin/calico-node -felix-live -bird-live] delay=10s timeout=10s period=10s #success=1 #failure=6
Readiness: exec [/bin/calico-node -felix-ready -bird-ready] delay=0s timeout=10s period=10s #success=1 #failure=3
Environment Variables from:
kubernetes-services-endpoint ConfigMap Optional: true
Environment:
DATASTORE_TYPE: kubernetes
WAIT_FOR_DATASTORE: true
NODENAME: (v1:spec.nodeName)
CALICO_NETWORKING_BACKEND: <set to the key 'calico_backend' of config map 'calico-config'> Optional: false
CLUSTER_TYPE: k8s,bgp
IP: autodetect
CALICO_IPV4POOL_IPIP: Always
CALICO_IPV4POOL_VXLAN: Never
FELIX_IPINIPMTU: <set to the key 'veth_mtu' of config map 'calico-config'> Optional: false
FELIX_VXLANMTU: <set to the key 'veth_mtu' of config map 'calico-config'> Optional: false
FELIX_WIREGUARDMTU: <set to the key 'veth_mtu' of config map 'calico-config'> Optional: false
CALICO_DISABLE_FILE_LOGGING: true
FELIX_DEFAULTENDPOINTTOHOSTACTION: ACCEPT
FELIX_IPV6SUPPORT: false
FELIX_HEALTHENABLED: true
Mounts:
/host/etc/cni/net.d from cni-net-dir (rw)
/lib/modules from lib-modules (ro)
/run/xtables.lock from xtables-lock (rw)
/sys/fs/ from sysfs (rw)
/var/lib/calico from var-lib-calico (rw)
/var/log/calico/cni from cni-log-dir (ro)
/var/run/calico from var-run-calico (rw)
/var/run/nodeagent from policysync (rw)
/var/run/secrets/kubernetes.io/serviceaccount from calico-node-token-qfrcw (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
lib-modules:
Type: HostPath (bare host directory volume)
Path: /lib/modules
HostPathType:
var-run-calico:
Type: HostPath (bare host directory volume)
Path: /var/run/calico
HostPathType:
var-lib-calico:
Type: HostPath (bare host directory volume)
Path: /var/lib/calico
HostPathType:
xtables-lock:
Type: HostPath (bare host directory volume)
Path: /run/xtables.lock
HostPathType: FileOrCreate
sysfs:
Type: HostPath (bare host directory volume)
Path: /sys/fs/
HostPathType: DirectoryOrCreate
cni-bin-dir:
Type: HostPath (bare host directory volume)
Path: /opt/cni/bin
HostPathType:
cni-net-dir:
Type: HostPath (bare host directory volume)
Path: /etc/cni/net.d
HostPathType:
cni-log-dir:
Type: HostPath (bare host directory volume)
Path: /var/log/calico/cni
HostPathType:
host-local-net-dir:
Type: HostPath (bare host directory volume)
Path: /var/lib/cni/networks
HostPathType:
policysync:
Type: HostPath (bare host directory volume)
Path: /var/run/nodeagent
HostPathType: DirectoryOrCreate
flexvol-driver-host:
Type: HostPath (bare host directory volume)
Path: /usr/libexec/kubernetes/kubelet-plugins/volume/exec/nodeagent~uds
HostPathType: DirectoryOrCreate
calico-node-token-qfrcw:
Type: Secret (a volume populated by a Secret)
SecretName: calico-node-token-qfrcw
Optional: false
QoS Class: Burstable
Node-Selectors: kubernetes.io/os=linux
Tolerations: :NoSchedule op=Exists
:NoExecute op=Exists
CriticalAddonsOnly op=Exists
node.kubernetes.io/disk-pressure:NoSchedule op=Exists
node.kubernetes.io/memory-pressure:NoSchedule op=Exists
node.kubernetes.io/network-unavailable:NoSchedule op=Exists
node.kubernetes.io/not-ready:NoExecute op=Exists
node.kubernetes.io/pid-pressure:NoSchedule op=Exists
node.kubernetes.io/unreachable:NoExecute op=Exists
node.kubernetes.io/unschedulable:NoSchedule op=Exists
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning Unhealthy 93s (x41181 over 4d18h) kubelet (combined from similar events): Readiness probe failed: 2021-09-10 17:16:42.497 [INFO][1620263] confd/health.go 180: Number of node(s) with BGP peering established = 0
calico/node is not ready: BIRD is not ready: BGP not established with 192.168.55.1,192.168.0.12
------编辑 4--------
尝试删除并重新安装印花布，这使事情变得更糟。
我们即将开始在我们自己的数据中心的裸机上建立一个新的 kubernetes 集群。k8s 模块和服务的文档很棒，但是我找不到任何关于满足我们要求所需组件的全面顶视图文档：
Pod 需要可通过 IPv4 和 IPv6 访问 Pod 需要能够在主机之间移动，并且仍然可以通过两种协议访问 Pod 需要通过 IPv4 和 IPv6 访问外部资源 我知道可以使用 MetalLb 进行入口流量管理。但是，当 pod 尝试访问外部 v4/v6 资源时，这是否也有效？
总体而言，要满足这些要求需要什么？
我有一个运行印花布豆荚 [ calico-node-& calico-kube-controllers-] 的 K3s 设置。在卸载 K3s 时，calico pod 被删除，但我看到它calicoctl并且iptables -S命令仍在运行并显示数据。
我想彻底删除calico（包括calico创建的calicoctl和Iptables）。哪些命令可以帮助我这样做？
K3s 卸载命令： /usr/local/bin/k3s-uninstall.sh删除所有 k3s pod，包括 calico，但calicoctl仍然iptables -S有效。
PS：我已经尝试了几件事-
kubectl delete -f https://raw.githubusercontent.com/aws/amazon-vpc-cni-k8s/release-1.5/config/v1.5/calico.yaml删除calico-node-but calico-kube-controller，calicoctl并且iptables -S仍然存在Kubectl delete这个队列中的命令也对我不起作用，在执行这两个命令之后calicoctl仍然iptables -S存在
我的集群上安装了 Calico，但几天前 pod 路由停止工作。我注意到印花布豆荚脱机了，从日志中我得到了这个：
2021-07-12 08:36:43.524 [INFO][1] main.go 92: Loaded configuration from environment config=&config.Config{LogLevel:"info", WorkloadEndpointWorkers:1, ProfileWorkers:1, PolicyWorkers:1, NodeWorkers:1, Kubeconfig:"", DatastoreType:"kubernetes"}
2021-07-12 08:36:43.525 [FATAL][1] main.go 105: Failed to start error=failed to build Calico client: open /var/run/secrets/kubernetes.io/serviceaccount/token: permission denied
我该如何解决这个问题？先感谢您
Kubernetes 版本：v1.21.1
印花布版本：最新
我有一个使用 kubernetes 编排器运行的应用程序。我想实现基于域名或通配符的 calico 网络策略，以便可以使用域名 ( FQDN/ DNS) 来允许从一个 pod 或一组 pod 访问（通过标签选择器）。
我遇到了calico doc，它说了同样的话，但不确定这是免费的还是付费的？有人可以证实这一点吗？我也可以在哪里得到这个例子？
我有一个使用 kubernetes 编排器运行的应用程序。我想实现基于域名正则表达式匹配的印花布网络策略。我在calico doc中看到他们使用 ip range(CIDR) 来控制来电和去电。
可以使用域名正则表达式匹配而不是 CIDR 来完成同样的事情吗？一些例子会有所帮助。
例如，我希望我的 calico 网络策略允许域的入口调用：
"^.+\\.app\\.ubuntu\\.net$ ^.+\\.aws-tools.org$ ^local\\.aws-tools\\.org$"
我无法让 kubernetes IPv6SingleStack LoadBalancer服务通过正确的源 IP 地址传递到 pod。SingleStack LoadBalancer它在将流量传递到相同 pod的姊妹 IPv4 上运行良好。
该集群是一个裸机 v1.21.1 双栈集群kubeadm，使用 Calico v3.18 作为 cni 和 MetalLB 来为配置了type: LoadBalancer. 然后将 Calico 配置为通过 BGP 向本地路由器宣布负载均衡器 IP。以具有两个服务（一个用于 IPv4，一个用于 IPv6）的单个nginx部署为例，如果我通过 IPv4 地址卷曲 IP，则 nginx 访问日志会在以下位置打印正确的客户端 IP 192.168.2.0/24：
192.168.2.128 - - [01/Jun/2021:19:32:37 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.64.1" "-"
但是将来自同一客户端的 IPv6 地址卷曲在 中2001:8b0:c8f:e8b0::/64，nginx 显示的客户端 IP 地址为fd5a:1111:1111::f31f
fd5a:1111:1111::f31f - - [01/Jun/2021:19:34:23 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.64.1" "-"
该地址来自集群的serviceSubnet，fd5a:1111:1111::/112并且恰好是clusterIPIPv6 服务的地址。似乎有些东西实际上在这里做了一些 TCP 代理（ipvs？），但目前尚不清楚它为什么会这样。如果是的话，我希望这externalTrafficPolicy是Cluster- 事实上，如果我将服务从 更改Local为Cluster，我将获得在 IPv4 上转发请求的集群节点的本地 IP 地址（如预期的那样），以及在 IPv6 上相同的集群 IP 地址。externalTrafficPolicy在 IPv6 的情况下似乎没有效果。
我是否遗漏了一些明显的东西，或者这些服务的行为方式是否应该相同？
测试清单：
---
apiVersion: v1
kind: Service
metadata:
name: test-service-source-ip-v4
namespace: default
labels:
k8s-app: test-service-source-ip
spec:
selector:
k8s-app: test-service-source-ip
type: LoadBalancer
ipFamilies:
- IPv4
ipFamilyPolicy: SingleStack
loadBalancerIP: 192.168.254.11
externalTrafficPolicy: "Local"
ports:
- name: http-tcp
protocol: TCP
port: 80
---
apiVersion: v1
kind: Service
metadata:
name: test-service-source-ip-v6
namespace: default
labels:
k8s-app: test-service-source-ip
spec:
selector:
k8s-app: test-service-source-ip
type: LoadBalancer
ipFamilies:
- IPv6
ipFamilyPolicy: SingleStack
loadBalancerIP: 2001:8b0:c8f:e8b1:beef:f00d::11
externalTrafficPolicy: "Local"
ports:
- name: http-tcp
protocol: TCP
port: 80
---
apiVersion: apps/v1
kind: Deployment
metadata:
namespace: default
name: test-service-source-ip
labels:
k8s-app: test-service-source-ip
spec:
replicas: 1
selector:
matchLabels:
k8s-app: test-service-source-ip
template:
metadata:
labels:
k8s-app: test-service-source-ip
spec:
containers:
- name: test-service-source-ip
image: nginx:1
ports:
- containerPort: 80
protocol: TCP