主页 / server / 问题 / 805053
Accepted
ufk
Asked: 2016-09-24 15:37:39 +0800 CST

带有 etcd2-tls 的 kubelet 服务无法连接到 127.0.0.1:8080 - getsockopt：连接被拒绝

  • 772

我安装了 CoreOS stable v1122.2.0。

我已经用 tls 配置了 etcd2 并且工作正常。我基于https://github.com/coreos/etcd/tree/master/hack/tls-setup使用我为我的服务器创建的子域而不是特定的 IP 地址创建了证书，以使 calico tls 工作。

etcd2 和 calcio-node 已配置并正常工作。现在我想配置 Kubernetes。我使用了https://coreos.com/kubernetes/docs/latest/deploy-master.html上的说明，现在我只配置了一个 coreos 服务器。

当我启动 kubelet 并执行时，journalctl -f -u kubelet我收到以下消息：

 Sep 23 23:30:11 coreos-2.tux-in.com kubelet-wrapper[1473]: E0923 23:30:11.495381    1473 reflector.go:205] pkg/kubelet/kubelet.go:286: Failed to list *api.Node: Get http://127.0.0.1:8080/api/v1/nodes?fieldSelector=metadata.name%3Dcoreos-2.tux-in.com&resourceVersion=0: dial tcp 127.0.0.1:8080: getsockopt: connection refused
 Sep 23 23:30:11 coreos-2.tux-in.com kubelet-wrapper[1473]: E0923 23:30:11.889187    1473 reflector.go:205] pkg/kubelet/kubelet.go:267: Failed to list *api.Service: Get http://127.0.0.1:8080/api/v1/services?resourceVersion=0: dial tcp 127.0.0.1:8080: getsockopt: connection refused
 Sep 23 23:30:12 coreos-2.tux-in.com kubelet-wrapper[1473]: E0923 23:30:12.292061    1473 reflector.go:205] pkg/kubelet/config/apiserver.go:43: Failed to list *api.Pod: Get http://127.0.0.1:8080/api/v1/pods?fieldSelector=spec.nodeName%3Dcoreos-2.tux-in.com&resourceVersion=0: dial tcp 127.0.0.1:8080: getsockopt: connection refused
 Sep 23 23:30:12 coreos-2.tux-in.com kubelet-wrapper[1473]: E0923 23:30:12.307222    1473 event.go:207] Unable to write event: 'Post http://127.0.0.1:8080/api/v1/namespaces/default/events: dial tcp 127.0.0.1:8080: getsockopt: connection refused' (may retry after sleeping)
 Sep 23 23:30:12 coreos-2.tux-in.com kubelet-wrapper[1473]: E0923 23:30:12.495982    1473 reflector.go:205] pkg/kubelet/kubelet.go:286: Failed to list *api.Node: Get http://127.0.0.1:8080/api/v1/nodes?fieldSelector=metadata.name%3Dcoreos-2.tux-in.com&resourceVersion=0: dial tcp 127.0.0.1:8080: getsockopt: connection refused
 Sep 23 23:30:12 coreos-2.tux-in.com kubelet-wrapper[1473]: E0923 23:30:12.889756    1473 reflector.go:205] pkg/kubelet/kubelet.go:267: Failed to list *api.Service: Get http://127.0.0.1:8080/api/v1/services?resourceVersion=0: dial tcp 127.0.0.1:8080: getsockopt: connection refused
 Sep 23 23:30:13 coreos-2.tux-in.com kubelet-wrapper[1473]: E0923 23:30:13.292671    1473 reflector.go:205] pkg/kubelet/config/apiserver.go:43: Failed to list *api.Pod: Get http://127.0.0.1:8080/api/v1/pods?fieldSelector=spec.nodeName%3Dcoreos-2.tux-in.com&resourceVersion=0: dial tcp 127.0.0.1:8080: getsockopt: connection refused
 Sep 23 23:30:13 coreos-2.tux-in.com kubelet-wrapper[1473]: E0923 23:30:13.496732    1473 reflector.go:205] pkg/kubelet/kubelet.go:286: Failed to list *api.Node: Get http://127.0.0.1:8080/api/v1/nodes?fieldSelector=metadata.name%3Dcoreos-2.tux-in.com&resourceVersion=0: dial tcp 127.0.0.1:8080: getsockopt: connection refused
 Sep 23 23:30:13 coreos-2.tux-in.com kubelet-wrapper[1473]: E0923 23:30:13.589335    1473 kubelet.go:1938] Failed creating a mirror pod for "kube-apiserver-coreos-2.tux-in.com_kube-system(9b41319800532574b4c4ac760c920bee)": Post http://127.0.0.1:8080/api/v1/namespaces/kube-system/pods: dial tcp 127.0.0.1:8080: getsockopt: connection refused
 Sep 23 23:30:13 coreos-2.tux-in.com kubelet-wrapper[1473]: E0923 23:30:13.890294    1473 reflector.go:205] pkg/kubelet/kubelet.go:267: Failed to list *api.Service: Get http://127.0.0.1:8080/api/v1/services?resourceVersion=0: dial tcp 127.0.0.1:8080: getsockopt: connection refused
 Sep 23 23:30:13 coreos-2.tux-in.com kubelet-wrapper[1473]: I0923 23:30:13.979257    1473 docker_manager.go:2289] checking backoff for container "kube-apiserver" in pod "kube-apiserver-coreos-2.tux-in.com"
 Sep 23 23:30:13 coreos-2.tux-in.com kubelet-wrapper[1473]: I0923 23:30:13.980071    1473 docker_manager.go:2303] Back-off 2m40s restarting failed container=kube-apiserver pod=kube-apiserver-coreos-2.tux-in.com_kube-system(9b41319800532574b4c4ac760c920bee)
 Sep 23 23:30:13 coreos-2.tux-in.com kubelet-wrapper[1473]: E0923 23:30:13.980144    1473 pod_workers.go:183] Error syncing pod 9b41319800532574b4c4ac760c920bee, skipping: failed to "StartContainer" for "kube-apiserver" with CrashLoopBackOff: "Back-off 2m40s restarting failed container=kube-apiserver pod=kube-apiserver-coreos-2.tux-in.com_kube-system(9b41319800532574b4c4ac760c920bee)"

这是我的/var/lib/coreos-install/user_data文件：

 #cloud-config

 hostname: "coreos-2.tux-in.com"
 write_files:
  - path: "/etc/ssl/etcd/ca.pem"
    permissions: "0666"
    owner: "etcd:etcd"
    content: |
     ...
  - path: "/etc/ssl/etcd/etcd1.pem"
    permissions: "0666"
    owner: "etcd:etcd"
    content: |
     ...
  - path: "/etc/ssl/etcd/etcd1-key.pem"
    permissions: "0666"
    owner: "etcd:etcd"
    content: |
     ...
  - path: "/etc/kubernetes/ssl/ca.pem"
    permissions: "0600"
    owner: "root:root"
    content: |
     ...
  - path: "/etc/kubernetes/ssl/apiserver.pem"
    permissions: "0600"
    owner: "root:root"
    content: |
     ...
  - path: "/etc/kubernetes/ssl/apiserver-key.pem"
    permissions: "0600"
    owner: "root:root"
    content: |
     ...
  - path: "/etc/kubernetes/cni/net.d/10-calico.conf"
    content: |
      {
          "name": "calico",
          "type": "flannel",
          "delegate": {
              "type": "calico",
              "etcd_endpoints": "https://coreos-2.tux-in.com:2379",
              "log_level": "none",
              "log_level_stderr": "info",
              "hostname": "coreos-2.tux-in.com",
              "policy": {
                  "type": "k8s",
                  "k8s_api_root": "http://127.0.0.1:8080/api/v1/"
              }
          }
      }
  - path: "/etc/kubernetes/manifests/policy-controller.yaml"
    content: |
     apiVersion: v1
      kind: Pod
      metadata:
        name: calico-policy-controller
        namespace: calico-system
      spec:
        hostNetwork: true
        containers:
          # The Calico policy controller.
          - name: k8s-policy-controller
            image: calico/kube-policy-controller:v0.2.0
            env:
              - name: ETCD_ENDPOINTS
                value: "https://coreos-2.tux-in.com:2379"
              - name: K8S_API
                value: "http://127.0.0.1:8080"
              - name: LEADER_ELECTION
                value: "true"
          # Leader election container used by the policy controller.
          - name: leader-elector
            image: quay.io/calico/leader-elector:v0.1.0
            imagePullPolicy: IfNotPresent
            args:
              - "--election=calico-policy-election"
              - "--election-namespace=calico-system"
              - "--http=127.0.0.1:4040"
  - path: "/etc/kubernetes/manifests/kube-scheduler.yaml"
    content: |
      apiVersion: v1
      kind: Pod
      metadata:
        name: kube-scheduler
        namespace: kube-system
      spec:
        hostNetwork: true
        containers:
        - name: kube-scheduler
          image: quay.io/coreos/hyperkube:v1.3.6_coreos.0
          command:
          - /hyperkube
          - scheduler
          - --master=http://127.0.0.1:8080
          - --leader-elect=true
          livenessProbe:
            httpGet:
              host: 127.0.0.1
              path: /healthz
              port: 10251
            initialDelaySeconds: 15
            timeoutSeconds: 1
  - path: "/etc/kubernetes/manifests/kube-controller-manager.yaml"
    content: |
      apiVersion: v1
      kind: Pod
      metadata:
        name: kube-controller-manager
        namespace: kube-system
      spec:
        hostNetwork: true
        containers:
        - name: kube-controller-manager
          image: quay.io/coreos/hyperkube:v1.3.6_coreos.0
          command:
          - /hyperkube
          - controller-manager
          - --master=http://127.0.0.1:8080
          - --leader-elect=true
          - --service-account-private-key-file=/etc/kubernetes/ssl/apiserver-key.pem
          - --root-ca-file=/etc/kubernetes/ssl/ca.pem
          livenessProbe:
            httpGet:
              host: 127.0.0.1
              path: /healthz
              port: 10252
            initialDelaySeconds: 15
            timeoutSeconds: 1
          volumeMounts:
          - mountPath: /etc/kubernetes/ssl
            name: ssl-certs-kubernetes
            readOnly: true
          - mountPath: /etc/ssl/certs
            name: ssl-certs-host
            readOnly: true
        volumes:
        - hostPath:
            path: /etc/kubernetes/ssl
          name: ssl-certs-kubernetes
        - hostPath:
            path: /usr/share/ca-certificates
          name: ssl-certs-host
  - path: "/etc/kubernetes/manifests/kube-proxy.yaml"
    content: |
      apiVersion: v1
      kind: Pod
      metadata:
        name: kube-proxy
        namespace: kube-system
      spec:
        hostNetwork: true
        containers:
        - name: kube-proxy
          image: quay.io/coreos/hyperkube:v1.3.6_coreos.0
          command:
          - /hyperkube
          - proxy
          - --master=http://127.0.0.1:8080
          - --proxy-mode=iptables
          securityContext:
            privileged: true
          volumeMounts:
          - mountPath: /etc/ssl/certs
            name: ssl-certs-host
            readOnly: true
        volumes:
        - hostPath:
            path: /usr/share/ca-certificates
          name: ssl-certs-host
  - path: "/etc/kubernetes/manifests/kube-apiserver.yaml"
    content: |
      apiVersion: v1
      kind: Pod
      metadata:
        name: kube-apiserver
        namespace: kube-system
      spec:
        hostNetwork: true
        containers:
        - name: kube-apiserver
          image: quay.io/coreos/hyperkube:v1.3.6_coreos.0
          command:
          - /hyperkube
          - apiserver
          - --bind-address=0.0.0.0
          - --etcd-servers=https://coreos-2.tux-in.com:2379
          - --allow-privileged=true
          - --service-cluster-ip-range=10.0.0.0/24
          - --secure-port=443
          - --advertise-address=coreos-2.tux-in.com
          - --admission-control=NamespaceLifecycle,LimitRanger,ServiceAccount,ResourceQuota
          - --tls-cert-file=/etc/kubernetes/ssl/apiserver.pem
          - --tls-private-key-file=/etc/kubernetes/ssl/apiserver-key.pem
          - --client-ca-file=/etc/kubernetes/ssl/ca.pem
          - --service-account-key-file=/etc/kubernetes/ssl/apiserver-key.pem
          - --runtime-config=extensions/v1beta1=true,extensions/v1beta1/networkpolicies=true
          ports:
          - containerPort: 443
            hostPort: 443
            name: https
          - containerPort: 8080
            hostPort: 8080
            name: local
          volumeMounts:
          - mountPath: /etc/kubernetes/ssl
            name: ssl-certs-kubernetes
            readOnly: true
          - mountPath: /etc/ssl/certs
            name: ssl-certs-host
            readOnly: true
        volumes:
        - hostPath:
            path: /etc/kubernetes/ssl
          name: ssl-certs-kubernetes
        - hostPath:
            path: /usr/share/ca-certificates
          name: ssl-certs-host
 ssh_authorized_keys:
          - ...
 coreos:
   etcd2:
     # generate a new token for each unique cluster from https://discovery.etcd.io/new?size=3
     # specify the initial size of your cluster with ?size=X
     discovery: ...
     advertise-client-urls: https://coreos-2.tux-in.com:2379,https://coreos-2.tux-in.com:4001
     initial-advertise-peer-urls: https://coreos-2.tux-in.com:2380
     # listen on both the official ports and the legacy ports
     # legacy ports can be omitted if your application doesn't depend on them
     listen-client-urls: https://0.0.0.0:2379,https://0.0.0.0:4001
     listen-peer-urls: https://coreos-2.tux-in.com:2380
   flannel:
     etcd_endpoints: "https://coreos-2.tux-in.com:2379"
     etcd_cafile: /etc/ssl/etcd/ca.pem
     etcd_certfile: /etc/ssl/etcd/etcd1.pem
     etcd_keyfile: /etc/ssl/etcd/etcd1-key.pem
   update:
     reboot-strategy: etcd-lock
   units:
     - name: 00-enp4s0.network
       runtime: true
       content: |
        [Match]
        Name=enp4s0

        [Network]
        Address=10.79.218.2/24
        Gateway=10.79.218.232
        DNS=8.8.8.8
     - name: var-lib-rkt.mount
       enable: true
       command: start
       content: |
         [Mount]
         What=/dev/disk/by-uuid/daca9515-5040-4f1d-ac0b-b69de3b91343
         Where=/var/lib/rkt
         Type=btrfs
         Options=loop,discard
     - name: etcd2.service
       command: start
       drop-ins:
        - name: 30-certs.conf
          content: |
           [Service]
           Environment="ETCD_CERT_FILE=/etc/ssl/etcd/etcd1.pem"
           Environment="ETCD_KEY_FILE=/etc/ssl/etcd/etcd1-key.pem"
           Environment="ETCD_TRUSTED_CA_FILE=/etc/ssl/etcd/ca.pem"
           Environment="ETCD_CLIENT_CERT_AUTH=true"
           Environment="ETCD_PEER_CERT_FILE=/etc/ssl/etcd/etcd1.pem"
           Environment="ETCD_PEER_KEY_FILE=/etc/ssl/etcd/etcd1-key.pem"
           Environment="ETCD_PEER_TRUSTED_CA_FILE=/etc/ssl/etcd/ca.pem"
           Environment="ETCD_PEER_CLIENT_CERT_AUTH=true"
     - name: flanneld.service
       command: start
       drop-ins:
        - name: 50-network-config.conf
          content: |
           [Service]
           ExecStartPre=/usr/bin/etcdctl --ca-file=/etc/ssl/etcd/ca.pem --cert-file=/etc/ssl/etcd/etcd1.pem --key-file=/etc/ssl/etcd/etcd1-key.pem --endpoint=https://coreos-2.tux-in.com:2379 set /coreos.com/network/config '{"Network":"10.1.0.0/16", "Backend": {"Type": "vxlan"}}'
     - name: calico-node.service
       command: start
       content: |
        [Unit]
        Description=Calico per-host agent
        Requires=network-online.target
        After=network-online.target

        [Service]
        Slice=machine.slice
        Environment=CALICO_DISABLE_FILE_LOGGING=true
        Environment=HOSTNAME=coreos-2.tux-in.com
        Environment=IP=10.79.218.2
        Environment=FELIX_FELIXHOSTNAME=coreos-2.tux-in.com
        Environment=CALICO_NETWORKING=false
        Environment=NO_DEFAULT_POOLS=true
        Environment=ETCD_ENDPOINTS=https://coreos-2.tux-in.com:2379
        Environment=ETCD_AUTHORITY=coreos-2.tux-in.com:2379
        Environment=ETCD_SCHEME=https
        Environment=ETCD_CA_CERT_FILE=/etc/ssl/etcd/ca.pem
        Environment=ETCD_CERT_FILE=/etc/ssl/etcd/etcd1.pem
        Environment=ETCD_KEY_FILE=/etc/ssl/etcd/etcd1-key.pem
        ExecStart=/usr/bin/rkt run --volume=resolv-conf,kind=host,source=/etc/resolv.conf,readOnly=true \
        --volume=etcd-tls-certs,kind=host,source=/etc/ssl/etcd,readOnly=true --inherit-env --stage1-from-dir=stage1-fly.aci \
        --volume=modules,kind=host,source=/lib/modules,readOnly=false \
        --mount=volume=modules,target=/lib/modules \
        --trust-keys-from-https quay.io/calico/node:v0.19.0 \
        --mount=volume=etcd-tls-certs,target=/etc/ssl/etcd \
        --mount=volume=resolv-conf,target=/etc/resolv.conf

        KillMode=mixed
        Restart=always
        TimeoutStartSec=0

        [Install]
        WantedBy=multi-user.target
     - name: kubelet.service
       command: start
       content: |
        [Service]
        ExecStartPre=/usr/bin/mkdir -p /etc/kubernetes/manifests
        ExecStartPre=/usr/bin/mkdir -p /var/log/containers

        Environment=KUBELET_VERSION=v1.3.7_coreos.0
        Environment="RKT_OPTS=--volume var-log,kind=host,source=/var/log \
          --mount volume=var-log,target=/var/log \
          --volume dns,kind=host,source=/etc/resolv.conf \
          --mount volume=dns,target=/etc/resolv.conf"

        ExecStart=/usr/lib/coreos/kubelet-wrapper \
          --api-servers=http://127.0.0.1:8080 \
          --network-plugin-dir=/etc/kubernetes/cni/net.d \
          --network-plugin=cni \
          --register-schedulable=false \
          --allow-privileged=true \
          --config=/etc/kubernetes/manifests \
          --hostname-override=coreos-2.tux-in.com \
          --cluster-dns=8.8.8.8 \
          --cluster-domain=tux-in.com
        Restart=always
        RestartSec=10
        [Install]
        WantedBy=multi-user.target

127.0.0.1:8080应该由 kubelet-apiserver 打开吗？我在这里想念什么？

谢谢！

kubernetes coreos

1 个回答

  1. Best Answer

    在许多情况下，API 服务器是由 Kubelet 启动的，导致在 API 端点可用之前出现初始连接错误。如果此错误在一段时间后仍然存在，您可能希望查看您的 API 服务器是否正在启动以及是否启动。

    Kubelet 将自动启动/etc/kubernetes/manifests您的kube-apiserver.yaml所在位置的服务。

    如果您的 API Server 未启动，您将需要：

    1：检查 Kubelet 命令行选项以确保使用这些--config=/etc/kubernetes/manifests选项启用清单。这可以检查 ps aux | grep kubelet

    2：检查 Kubelet 和 API 容器日志以查看 API 启动期间发生的问题。这通常是证书不匹配、依赖失败、etcd 服务未侦听等。

    Kubelet 服务日志：

       $ journalctl -fu kubelet.service

    这个例子我通过 'docker logs' 从 API 服务器收集日志并显示我的 Kubelet 启动 API 服务器。在服务器侦听和最终启动之前注意类似的连接问题。

       $ docker ps -l
    543022a70bc6        gcr.io/google_containers/hyperkube:v1.3.7   "/hyperkube apiserver"   3 seconds ago        Exited (1) 3 seconds ago

   $ docker logs 543022a70bc6
    I0920 00:26:33.903861       1 genericapiserver.go:606] Will report 10.0.104.100 as public IP address.
    E0920 00:26:33.937478       1 reflector.go:205] k8s.io/kubernetes/plugin/pkg/admission/namespace/exists/admission.go:86: Failed to list *api.Namespace: Get http://0.0.0.0:8080/api/v1/namespaces?resourceVersion=0: dial tcp 0.0.0.0:8080: getsockopt: connection refused
    E0920 00:26:33.937651       1 reflector.go:205] k8s.io/kubernetes/plugin/pkg/admission/namespace/lifecycle/admission.go:116: Failed to list *api.Namespace: Get http://0.0.0.0:8080/api/v1/namespaces?resourceVersion=0: dial tcp 0.0.0.0:8080: getsockopt: connection refused
    E0920 00:26:33.937821       1 reflector.go:205] k8s.io/kubernetes/plugin/pkg/admission/limitranger/admission.go:154: Failed to list *api.LimitRange: Get http://0.0.0.0:8080/api/v1/limitranges?resourceVersion=0: dial tcp 0.0.0.0:8080: getsockopt: connection refused
    E0920 00:26:33.939508       1 reflector.go:216] k8s.io/kubernetes/plugin/pkg/admission/serviceaccount/admission.go:119: Failed to list *api.Secret: Get http://0.0.0.0:8080/api/v1/secrets?fieldSelector=type%3Dkubernetes.io%2Fservice-account-token&resourceVersion=0: dial tcp 0.0.0.0:8080: getsockopt: connection refused
    E0920 00:26:33.939741       1 reflector.go:216] k8s.io/kubernetes/plugin/pkg/admission/serviceaccount/admission.go:103: Failed to list *api.ServiceAccount: Get http://0.0.0.0:8080/api/v1/serviceaccounts?resourceVersion=0: dial tcp 0.0.0.0:8080: getsockopt: connection refused
    E0920 00:26:33.947780       1 reflector.go:205] k8s.io/kubernetes/plugin/pkg/admission/resourcequota/controller.go:121: Failed to list *api.ResourceQuota: Get http://0.0.0.0:8080/api/v1/resourcequotas?resourceVersion=0: dial tcp 0.0.0.0:8080: getsockopt: connection refused
    [restful] 2016/09/20 00:26:34 log.go:30: [restful/swagger] listing is available at https://10.0.104.100:6443/swaggerapi/
    [restful] 2016/09/20 00:26:34 log.go:30: [restful/swagger] https://10.0.104.100:6443/swaggerui/ is mapped to folder /swagger-ui/
    I0920 00:26:34.235914       1 genericapiserver.go:690] Serving securely on 0.0.0.0:6443
    I0920 00:26:34.235941       1 genericapiserver.go:734] Serving insecurely on 0.0.0.0:8080
    • 2

相关问题