主页 / coding / 问题 / 77164954
Accepted
PesKchan
Asked: 2023-09-24 05:18:10 +0800 CST

在 shell 脚本中添加重试

  • 772

我的目标是增加此 fastq 验证的重试次数，该验证有时会因网络问题而失败，尽管有给定 NCBI SRA id 的数据，但它失败了。所以我需要在中止之前增加至少5次重试。

我该如何做同样的事情？

set -x
PS4='[\\d \\t] '

# Check parameter for error
check=0
# Print fastq-dump executable path
echo \$(which fastq-dump)

# Loop through all parameters to check validity
for file in \$@;
do
    cp \${file} .
    # Extract filename for sampleID
    file_basename=\$(basename \${file})
    id=\${file_basename%".id"}
    # Start validation
    echo "Checking \${id}..."
    # Download start of fastq
    fastq-dump $(get_ngc()) -X 1 -Z --split-spot \${id} > \${id}.test.fastq 2> \${id}.test.log
    # Get number of lines downloaded to valildate for error
    numLines=\$(cat \${id}.test.fastq | wc -l)
    if [ \$numLines -gt 0 ]; then
        echo "\${id} has data... OK"
    else
        echo "\${id} does not have data... ERROR"
        check=1
    fi
done
# Exit with error if some fastqs not accessible
if [ \$check -gt 0 ]; then
    echo "ERROR: One or more samples have inaccessible fastqs.. exiting"
    exit 1
fi

＃＃＃＃＃＃＃＃＃＃＃＃ 更新 ＃＃＃＃＃＃＃＃＃＃＃＃＃＃＃

#!/usr/bin/env cwl-runner

cwlVersion: cwl:v1.0
class: CommandLineTool
requirements:
- class: DockerRequirement
  dockerPull: kcm1400/validate_fastq_ncbi_sra:v1
- class: InlineJavascriptRequirement
  expressionLib:
  - var get_ngc= function(){ if(inputs["ngc_file"]==null){ return " "; }else{ return
    "--ngc "+inputs["ngc_file"].path+" "; } }
- class: InitialWorkDirRequirement
  listing:
  - entry: |-
      set -x
      PS4='[\\d \\t] '

      # Check parameter for error
      check=0
      # Print fastq-dump executable path
      echo \$(which fastq-dump)

      # Loop through all parameters to check validity
      for file in \$@;
      do
          cp \${file} .
          # Extract filename for sampleID
          file_basename=\$(basename \${file})
          id=\${file_basename%".id"}
          # Start validation
          echo "Checking \${id}..."
          # Download start of fastq
          fastq-dump $(get_ngc()) -X 1 -Z --split-spot \${id} > \${id}.test.fastq 2> \${id}.test.log
          # Get number of lines downloaded to valildate for error
          numLines=\$(cat \${id}.test.fastq | wc -l)
          if [ \$numLines -gt 0 ]; then
              echo "\${id} has data... OK"
          else
              echo "\${id} does not have data... ERROR"
              check=1
          fi
      done
      # Exit with error if some fastqs not accessible
      if [ \$check -gt 0 ]; then
          echo "ERROR: One or more samples have inaccessible fastqs.. exiting"
          exit 1
      fi
    entryname: validate_fastq.sh
    writable: false
label: validate_fastq_ncbi_sra
stdout: validate_fastq.log.txt
inputs:
  sra_ID:
    type:
    - type: array
      items: string
    - 'null'
    inputBinding:
      position: 1
  id_file:
    type:
    - type: array
      items: File
    - 'null'
    inputBinding:
      position: 2
  ngc_file:
    type:
    - File
    - 'null'
    inputBinding:
      position: 3
outputs:
  output_log:
    type: stdout
    label: output log file
  ncbi_ids:
    type:
      type: array
      items: File
    outputBinding:
      glob:
      - '*.id'
baseCommand:
- sh
- validate_fastq.sh

相同的 CWL 文件

shell

1 个回答

  1. Best Answer

    这是一个尝试性的答案。希望至少它可以帮助您提出一个更完善的问题。

    您在评论中提到反斜杠在“cwl 文件”中对您有用，但您没有解释什么是“cwl 文件”。如果您指的是通用工作流语言，那么我们需要了解有关脚本周围的 YAML 结构的更多信息；当然，YAML 标量格式选项中不需要反斜杠。

    没有更多细节，我们只能说，对于“shell 脚本”，这些反斜杠正在破坏功能，因此我已将其删除。

    也不清楚您到底想要捕获哪种错误情况。大概fastq-dump是这些错误的根源。如果它写得很完整，你应该能够简单地说

    if fastq-dump --options arguments; then ...

    但为了安全起见，我坚持使用你计算输出线数量的笨拙方法。

    #!/bin/sh
# ^ explicitly name which shell you are using
# See also https://en.wikipedia.org/wiki/Shebang_(Unix)

set -x
PS4='[\\d \\t] '

check=0
# Avoid useless use of echo
# https://www.iki.fi/era/unix/award.html#echo
# Prefer POSIX "command -v" over nonstandard "which"
command -v fastq-dump

# Quote all file names
# https://stackoverflow.com/questions/10067266/when-to-wrap-quotes-around-a-shell-variable
for file in "$@"; do
    cp "$file" .
    # Basename knows how to trim extension
    id=$(basename "$file" .id)
    # Write diagnostics to stderr
    echo "$0: Checking $id..." >&2
    # Truncate log so we can append in a loop; see below
    : > "$id".test.log
    # Loop until success, or retries exhausted
    for retry in 1 2 3 4 5; do
        # Append rather than overwrite error log, in case we retry
        fastq-dump $(get_ngc()) -X 1 -Z --split-spot "$id" > "$id".test.fastq 2>> "$id".test.log
        # Avoid useless cat
        # https://stackoverflow.com/questions/11710552/useless-use-of-cat 
        numLines=$(wc -l < "$id".test.fastq)
        if [ $numLines -gt 0 ]; then
            echo "$0: $id has data... OK" >&2
            break
        else
            echo "$0: $id does not have data... ERROR" >&2
            case $retry in
             5) echo "$0: $id: aborting, after 5 attempts" >&2
                check=1;;
             *) # Sleep before retry
                sleep 5;;
            esac
        fi
    done
done
# Exit with error if some fastqs not accessible
if [ $check -gt 0 ]; then
    echo "$0: ERROR: One or more samples have inaccessible fastqs.. exiting" >&2
    exit 1
fi

    这里的许多初学者错误都会被https://shellcheck.net/ 检测到甚至修复；在此处询问之前，可能会通过此工具运行您的脚本，以避免分散回答者的注意力。显然，有时修复也可以解决您想问的问题。

    $(get_ngc())看起来仍然像一个语法错误，但我猜它是 CWL 的一部分......？

    • 2

相关问题