Liquid's questions

Estou executando o slurm 21.08.5 instalado via apt.

Meu cluster tem 4 máquinas gpu (nd-gpu[001-005]) com 8 gpus cada. Posso executar trabalhos como

srun --gres=gpu:8 nvidia-smi -L

E eu vejo minhas gpus. Eu também posso agendar trabalhos reais com qualquer um de 0 a 8 gpus. No entanto, o agendamento de recursos não está funcionando corretamente. Se eu executar:

 srun --gres=gpu:1 sleep 1000

O nó inteiro será alocado e não poderei usar as 7 GPUs restantes naquele nó.

O que se segue é o gres.conf:

NodeName=nd-gpu001 Name=gpu File=/dev/nvidia0
NodeName=nd-gpu001 Name=gpu File=/dev/nvidia1
NodeName=nd-gpu001 Name=gpu File=/dev/nvidia2
NodeName=nd-gpu001 Name=gpu File=/dev/nvidia3
NodeName=nd-gpu001 Name=gpu File=/dev/nvidia4
NodeName=nd-gpu001 Name=gpu File=/dev/nvidia5
NodeName=nd-gpu001 Name=gpu File=/dev/nvidia6
NodeName=nd-gpu001 Name=gpu File=/dev/nvidia7
NodeName=nd-gpu002 Name=gpu File=/dev/nvidia0
NodeName=nd-gpu002 Name=gpu File=/dev/nvidia1
NodeName=nd-gpu002 Name=gpu File=/dev/nvidia2
NodeName=nd-gpu002 Name=gpu File=/dev/nvidia3
NodeName=nd-gpu002 Name=gpu File=/dev/nvidia4
NodeName=nd-gpu002 Name=gpu File=/dev/nvidia5
NodeName=nd-gpu002 Name=gpu File=/dev/nvidia6
NodeName=nd-gpu002 Name=gpu File=/dev/nvidia7
NodeName=nd-gpu003 Name=gpu File=/dev/nvidia0
NodeName=nd-gpu003 Name=gpu File=/dev/nvidia1
NodeName=nd-gpu003 Name=gpu File=/dev/nvidia2
NodeName=nd-gpu003 Name=gpu File=/dev/nvidia3
NodeName=nd-gpu003 Name=gpu File=/dev/nvidia4
NodeName=nd-gpu003 Name=gpu File=/dev/nvidia5
NodeName=nd-gpu003 Name=gpu File=/dev/nvidia6
NodeName=nd-gpu003 Name=gpu File=/dev/nvidia7
NodeName=nd-gpu004 Name=gpu File=/dev/nvidia0
NodeName=nd-gpu004 Name=gpu File=/dev/nvidia1
NodeName=nd-gpu004 Name=gpu File=/dev/nvidia2
NodeName=nd-gpu004 Name=gpu File=/dev/nvidia3
NodeName=nd-gpu004 Name=gpu File=/dev/nvidia4
NodeName=nd-gpu004 Name=gpu File=/dev/nvidia5
NodeName=nd-gpu004 Name=gpu File=/dev/nvidia6
NodeName=nd-gpu004 Name=gpu File=/dev/nvidia7

o slurm.conf:

# See the slurm.conf man page for more information.  
#  
ClusterName=cluster  
SlurmctldHost=nd-cpu01  
SlurmctldHost=nd-cpu02  
#  

#GresTypes=  
#GroupUpdateForce=0  

ProctrackType=proctrack/cgroup  
ReturnToService=1  
SlurmctldPidFile=/var/run/slurmctld.pid  
SlurmctldPort=6817  
SlurmdPidFile=/var/run/slurmd.pid  
SlurmdPort=6818  
SlurmdSpoolDir=/var/spool/slurmd  
SlurmUser=slurm  
StateSaveLocation=/home/slurm/slurmctd  
TaskPlugin=task/affinity,task/cgroup  

# TIMERS  
InactiveLimit=0  
KillWait=30  

MinJobAge=300  

SlurmctldTimeout=120  
SlurmdTimeout=300  
Waittime=0  

# SCHEDULING  
SchedulerType=sched/backfill  
SelectType=select/cons_tres  

# LOGGING AND ACCOUNTING  
JobCompType=jobcomp/none  
JobAcctGatherFrequency=30  
SlurmctldDebug=info  
SlurmctldLogFile=/var/log/slurmctld.log  
SlurmdDebug=info  
SlurmdLogFile=/var/log/slurmd.log  

# COMPUTE NODES  
GresTypes=gpu
NodeName=nd-gpu[001-004] Sockets=2 CoresPerSocket=56 ThreadsPerCore=1 State=UNKNOWN Gres=gpu:8
PartitionName=debug Nodes=ALL Default=YES MaxTime=INFINITE State=UP

e o cgroups.conf:

ConstrainDevices=yes

Eu estava tentando emitir uma atualização do ceph de 17 para 18.2.4, conforme descrito aqui

ceph orch upgrade start --ceph-version 18.2.4
Initiating upgrade to quay.io/ceph/ceph:v18.2.4

Depois disso, porém, o orquestrador não responde mais

ceph orch upgrade status
Error ENOENT: Module not found

Definir o backend de volta para o orchestrator ou cephadm falha, porque o serviço aparece como 'desabilitado'. O gerente do Ceph jura que o serviço está ligado.

Pelo que pude perceber até agora, estou preso com um daemon mgr executando o reef, enquanto o resto do cluster executa o quincy.

ceph versions
{
    "mon": {
        "ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable)": 5
    },
    "mgr": {
        "ceph version 18.2.4 (e7ad5345525c7aa95470c26863873b581076945d) reef (stable)": 1
    },
    "osd": {
        "ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable)": 31
    },
    "mds": {
        "ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable)": 4
    },
    "overall": {
        "ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable)": 40,
        "ceph version 18.2.4 (e7ad5345525c7aa95470c26863873b581076945d) reef (stable)": 1
    }
}

Como faço para restabelecer o cluster em um estado saudável?

EDIT 1 Saúde Ceph:

  cluster:
    id:     16249ca6-4060-11ef-a8a1-7509512e051b
    health: HEALTH_WARN
            insufficient standby MDS daemons available
            mon gpu001 is low on available space
            1/5 mons down, quorum ***
            Degraded data redundancy: 92072087/608856489 objects degraded (15.122%), 97 pgs degraded, 97 pgs undersized
            7 pgs not deep-scrubbed in time

  services:
    mon: 5 daemons, quorum ***
    mgr: cpu01.fcxjpi(active, since 5m)
    mds: 4/4 daemons up
    osd: 34 osds: 31 up (since 45h), 31 in (since 46h); 31 remapped pgs
 
  data:
    volumes: 1/1 healthy
    pools:   4 pools, 193 pgs
    objects: 121.96M objects, 32 TiB
    usage:   36 TiB used, 73 TiB / 108 TiB avail
    pgs:     92072087/608856489 objects degraded (15.122%)
             29422795/608856489 objects misplaced (4.832%)
             97 active+undersized+degraded
             65 active+clean
             31 active+clean+remapped
  io:
    client:   253 KiB/s rd, 51 KiB/s wr, 3 op/s rd, 2 op/s wr

Nota : a pergunta foi feita originalmente no SO [https://stackoverflow.com/posts/78949269], fui aconselhado a movê-la para cá. Atualmente, estou no processo de pesquisa dos logs do MGR para investigar o status e, eventualmente, forçar um downgrade.