主页 / user-1015906

Liquid's questions

Martin Hope
Liquid
Asked: 2024-10-08 00:11:44 +0800 CST
  • 5

我正在运行通过 apt 安装的 slurm 21.08.5。

我的集群有 4 台 gpu 机器 (nd-gpu[001-005])，每台有 8 个 gpu。我可以按如下方式运行作业

srun --gres=gpu:8 nvidia-smi -L

我看到了 GPU。我还可以使用 0 到 8 个 GPU 中的任何一个来安排实际作业。但是，资源调度工作不正常。如果我运行：

 srun --gres=gpu:1 sleep 1000

整个节点将被分配，并且我将无法使用该节点上剩余的 7 个 gpu。

以下是gres.conf：

NodeName=nd-gpu001 Name=gpu File=/dev/nvidia0
NodeName=nd-gpu001 Name=gpu File=/dev/nvidia1
NodeName=nd-gpu001 Name=gpu File=/dev/nvidia2
NodeName=nd-gpu001 Name=gpu File=/dev/nvidia3
NodeName=nd-gpu001 Name=gpu File=/dev/nvidia4
NodeName=nd-gpu001 Name=gpu File=/dev/nvidia5
NodeName=nd-gpu001 Name=gpu File=/dev/nvidia6
NodeName=nd-gpu001 Name=gpu File=/dev/nvidia7
NodeName=nd-gpu002 Name=gpu File=/dev/nvidia0
NodeName=nd-gpu002 Name=gpu File=/dev/nvidia1
NodeName=nd-gpu002 Name=gpu File=/dev/nvidia2
NodeName=nd-gpu002 Name=gpu File=/dev/nvidia3
NodeName=nd-gpu002 Name=gpu File=/dev/nvidia4
NodeName=nd-gpu002 Name=gpu File=/dev/nvidia5
NodeName=nd-gpu002 Name=gpu File=/dev/nvidia6
NodeName=nd-gpu002 Name=gpu File=/dev/nvidia7
NodeName=nd-gpu003 Name=gpu File=/dev/nvidia0
NodeName=nd-gpu003 Name=gpu File=/dev/nvidia1
NodeName=nd-gpu003 Name=gpu File=/dev/nvidia2
NodeName=nd-gpu003 Name=gpu File=/dev/nvidia3
NodeName=nd-gpu003 Name=gpu File=/dev/nvidia4
NodeName=nd-gpu003 Name=gpu File=/dev/nvidia5
NodeName=nd-gpu003 Name=gpu File=/dev/nvidia6
NodeName=nd-gpu003 Name=gpu File=/dev/nvidia7
NodeName=nd-gpu004 Name=gpu File=/dev/nvidia0
NodeName=nd-gpu004 Name=gpu File=/dev/nvidia1
NodeName=nd-gpu004 Name=gpu File=/dev/nvidia2
NodeName=nd-gpu004 Name=gpu File=/dev/nvidia3
NodeName=nd-gpu004 Name=gpu File=/dev/nvidia4
NodeName=nd-gpu004 Name=gpu File=/dev/nvidia5
NodeName=nd-gpu004 Name=gpu File=/dev/nvidia6
NodeName=nd-gpu004 Name=gpu File=/dev/nvidia7

slurm.conf：

# See the slurm.conf man page for more information.  
#  
ClusterName=cluster  
SlurmctldHost=nd-cpu01  
SlurmctldHost=nd-cpu02  
#  

#GresTypes=  
#GroupUpdateForce=0  

ProctrackType=proctrack/cgroup  
ReturnToService=1  
SlurmctldPidFile=/var/run/slurmctld.pid  
SlurmctldPort=6817  
SlurmdPidFile=/var/run/slurmd.pid  
SlurmdPort=6818  
SlurmdSpoolDir=/var/spool/slurmd  
SlurmUser=slurm  
StateSaveLocation=/home/slurm/slurmctd  
TaskPlugin=task/affinity,task/cgroup  

# TIMERS  
InactiveLimit=0  
KillWait=30  

MinJobAge=300  

SlurmctldTimeout=120  
SlurmdTimeout=300  
Waittime=0  

# SCHEDULING  
SchedulerType=sched/backfill  
SelectType=select/cons_tres  

# LOGGING AND ACCOUNTING  
JobCompType=jobcomp/none  
JobAcctGatherFrequency=30  
SlurmctldDebug=info  
SlurmctldLogFile=/var/log/slurmctld.log  
SlurmdDebug=info  
SlurmdLogFile=/var/log/slurmd.log  

# COMPUTE NODES  
GresTypes=gpu
NodeName=nd-gpu[001-004] Sockets=2 CoresPerSocket=56 ThreadsPerCore=1 State=UNKNOWN Gres=gpu:8
PartitionName=debug Nodes=ALL Default=YES MaxTime=INFINITE State=UP

以及 cgroups.conf：

ConstrainDevices=yes
slurm
Martin Hope
Liquid
Asked: 2024-09-06 00:07:40 +0800 CST
  • 5

我试图将 ceph 从 17 升级到 18.2.4，如下所述

ceph orch upgrade start --ceph-version 18.2.4
Initiating upgrade to quay.io/ceph/ceph:v18.2.4

但此后，协调器不再响应

ceph orch upgrade status
Error ENOENT: Module not found

将后端重新设置为 orchestrator 或 cephadm 失败，因为服务显示为“已禁用”。Ceph mgr 却表示服务已开启。

据我目前所知，我被困在一个运行 reef 的 mgr 守护进程上，而集群的其余部分在 quincy 上运行。

ceph versions
{
    "mon": {
        "ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable)": 5
    },
    "mgr": {
        "ceph version 18.2.4 (e7ad5345525c7aa95470c26863873b581076945d) reef (stable)": 1
    },
    "osd": {
        "ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable)": 31
    },
    "mds": {
        "ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable)": 4
    },
    "overall": {
        "ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable)": 40,
        "ceph version 18.2.4 (e7ad5345525c7aa95470c26863873b581076945d) reef (stable)": 1
    }
}

如何将集群恢复到健康状态？

编辑1 Ceph健康：

  cluster:
    id:     16249ca6-4060-11ef-a8a1-7509512e051b
    health: HEALTH_WARN
            insufficient standby MDS daemons available
            mon gpu001 is low on available space
            1/5 mons down, quorum ***
            Degraded data redundancy: 92072087/608856489 objects degraded (15.122%), 97 pgs degraded, 97 pgs undersized
            7 pgs not deep-scrubbed in time

  services:
    mon: 5 daemons, quorum ***
    mgr: cpu01.fcxjpi(active, since 5m)
    mds: 4/4 daemons up
    osd: 34 osds: 31 up (since 45h), 31 in (since 46h); 31 remapped pgs
 
  data:
    volumes: 1/1 healthy
    pools:   4 pools, 193 pgs
    objects: 121.96M objects, 32 TiB
    usage:   36 TiB used, 73 TiB / 108 TiB avail
    pgs:     92072087/608856489 objects degraded (15.122%)
             29422795/608856489 objects misplaced (4.832%)
             97 active+undersized+degraded
             65 active+clean
             31 active+clean+remapped
  io:
    client:   253 KiB/s rd, 51 KiB/s wr, 3 op/s rd, 2 op/s wr

注意：该问题最初是在 SO [https://stackoverflow.com/posts/78949269] 上提出的，我被建议将其移到这里。我目前正在搜索 MGR 日志以调查状态，并最终强制降级。

ubuntu