dylanjm's questions

我目前正在研究一个课程，让我们将代码提交给自动评分器，然后返回我们的结果。它返回的格式很难在视觉上解析，所以我想编写一个可以在管道中使用的脚本，以使其更易于阅读。

这是自动评分器的输出：

Problem,Correct?,Correct Answer,Agent's Answer
"Challenge Problem B-04",0,4,-1
"Basic Problem B-12",0,1,-1
"Challenge Problem B-05",0,6,-1
"Challenge Problem B-07",0,6,-1
"Challenge Problem B-06",0,3,-1
"Basic Problem B-11",0,1,-1
"Basic Problem B-10",0,3,-1
"Challenge Problem B-03",0,3,-1
"Challenge Problem B-02",0,1,-1
"Challenge Problem B-01",0,6,-1
"Challenge Problem B-09",0,4,-1
"Challenge Problem B-08",0,4,-1
"Basic Problem B-08",0,6,-1
"Basic Problem B-09",0,5,-1
"Basic Problem B-04",0,3,-1
"Basic Problem B-05",0,4,-1
"Basic Problem B-06",0,5,-1
"Basic Problem B-07",0,6,-1
"Basic Problem B-01",0,2,-1
"Basic Problem B-02",0,5,-1
"Basic Problem B-03",0,1,-1
"Challenge Problem B-10",0,4,-1
"Challenge Problem B-11",0,5,-1
"Challenge Problem B-12",0,1,-1
{
    "Basic Problems B": {
        "Incorrect": "0",
        "Skipped": "12",
        "Correct": "0",
        "Set": "Basic Problems B"
    },
    "Challenge Problems B": {
        "Incorrect": "0",
        "Skipped": "12",
        "Correct": "0",
        "Set": "Challenge Problems B"
    }
}

它是逗号分隔值和 JSON 的混合。把这一切放在一张我可以阅读的漂亮桌子上会很好。

目前，我有类似的东西

python submit.py --provider gt --assignment error-check | column -t -s, | less -S

哪个输出：

{
    "Basic Problems B": {
        "Incorrect": "0",
        "Skipped": "12",
        "Correct": "0",
        "Set": "Basic Problems B"
    },
    "Challenge Problems B": {
        "Incorrect": "0",
        "Skipped": "12",
        "Correct": "0",
        "Set": "Challenge Problems B"
    }
}
Problem                   Correct?  Correct Answer  Agent's Answer
"Challenge Problem B-04"  0         4               -1
"Basic Problem B-12"      0         1               -1
"Challenge Problem B-05"  0         6               -1
"Challenge Problem B-07"  0         6               -1
"Challenge Problem B-06"  0         3               -1
"Basic Problem B-11"      0         1               -1
"Basic Problem B-10"      0         3               -1
"Challenge Problem B-03"  0         3               -1
"Challenge Problem B-02"  0         1               -1
"Challenge Problem B-01"  0         6               -1
"Challenge Problem B-09"  0         4               -1
"Challenge Problem B-08"  0         4               -1
"Basic Problem B-08"      0         6               -1
"Basic Problem B-09"      0         5               -1
"Basic Problem B-04"      0         3               -1
"Basic Problem B-05"      0         4               -1
"Basic Problem B-06"      0         5               -1
"Basic Problem B-07"      0         6               -1
"Basic Problem B-01"      0         2               -1
"Basic Problem B-02"      0         5               -1
"Basic Problem B-03"      0         1               -1
"Challenge Problem B-10"  0         4               -1
"Challenge Problem B-11"  0         5               -1
"Challenge Problem B-12"  0         1               -1

这让我大部分时间都在那里。现在我想知道是否有一种方法可以处理 JSON？

我不能依赖在某个行号处拆分输出，但我想我可以在输出第一次找到{.

我想尽可能少地这样做，以便与同学分享。所以依赖越少越好。

我看过其他建议使用外部代码的 JSON 解析帖子。

理想的输出如下所示：

Problem                   Correct?  Correct Answer  Agent's Answer
"Challenge Problem B-04"  0         4               -1
"Basic Problem B-12"      0         1               -1
"Challenge Problem B-05"  0         6               -1
"Challenge Problem B-07"  0         6               -1
"Challenge Problem B-06"  0         3               -1
"Basic Problem B-11"      0         1               -1
"Basic Problem B-10"      0         3               -1
"Challenge Problem B-03"  0         3               -1
"Challenge Problem B-02"  0         1               -1
"Challenge Problem B-01"  0         6               -1
"Challenge Problem B-09"  0         4               -1
"Challenge Problem B-08"  0         4               -1
"Basic Problem B-08"      0         6               -1
"Basic Problem B-09"      0         5               -1
"Basic Problem B-04"      0         3               -1
"Basic Problem B-05"      0         4               -1
"Basic Problem B-06"      0         5               -1
"Basic Problem B-07"      0         6               -1
"Basic Problem B-01"      0         2               -1
"Basic Problem B-02"      0         5               -1
"Basic Problem B-03"      0         1               -1
"Challenge Problem B-10"  0         4               -1
"Challenge Problem B-11"  0         5               -1
"Challenge Problem B-12"  0         1               -1

Set                   Incorrect Skipped Correct
Basic Problems B      0         12      0
Challenge Problems B  0         12      0

我正在编写一个脚本，用于搜索远程服务器上的文件并将它们传输回我的本地计算机。我希望能够先进行试运行，所以我知道要带回哪些文件。

我目前正在使用我在这里getopts找到的一些代码的混合和输出重定向。

在我看来，通过我的研究，从 ZSH 或 Bash 函数返回数组是不切实际的。对我来说，这让我很难理解如何编写这个脚本而不必重复自己。

这是我当前的脚本：

编辑：请原谅我将一些 bashisms 与 zsh 的东西混合在一起，我开始使用#!/bin/bash但切换到 zsh 编写这个脚本。

#!/usr/local/bin/zsh
RED='\033[0;31m'
NC='\033[0m'
GREEN='\033[0;32m'
YELLOW='\033[0;33m'

dry_run=0
yesterday=1

# Establish -n flag means to do a dry run.
while getopts "ny:" flag; do
    case "$flag" in
        n) dry_run=1 ;;
        y) yesterday=${OPTARG} ;;
        *) echo 'error in command line parsing' >&2
           exit 1
    esac
done
shift $(($OPTIND-1))

# This is the folder I'm interested in getting files from
folder=${1:?"You must define a folder of interest"}

# Check to see if dry-run, if not proceed with copying the files over. 
if [ "$dry_run" -eq 1 ]; then
    print -Pn "\n%S%11F%{Initiating Dry-Run%}%s%f"

    # SSH onto server and find the most recently updated folder.
    # Then place that newest folder and folder of interest into the absolute file path.
    # Then SSH again and use find with that file-path
    # Return array of file paths
    # TODO: **THIS IS THE SECTION I NEED TO REFACTOR INTO A FUNCTION**
    bison_remote_files=($(
        {
            {
                bison_latest_run=$(ssh -qn falcon1 'find /projects/bison/git/* -mindepth 0 -maxdepth 0 -type d -printf "%T@\t%f\n"' |
                    sort -t$'\t' -r -nk1,5 |
                    sed -n "$yesterday"p |
                    cut -f2-)

                bison_remote_path=$(
                    echo $bison_latest_run |
                        awk -v folder="$folder" '{print "/projects/bison/git/"$1"/assessment/LWR/validation/"folder}')

                ssh -qn falcon1 \
                    "find $bison_remote_path -type f -name '*_out.csv' -not -path '*/doc/*' 2>/dev/null" >&3 3>&-; echo "$?"

                print -Pn "\n\n%U%B%13F%{Fetching data from:%}%u %B%12F%{ /projects/bison/git/${bison_latest_run}%}%b%f\n" >&2

            } | {
                until read -t1 ret; do
                    print -Pn "%S%11F%{.%}%s%f" >&2
                done
                exit "$ret"
            }
        } 3>&1))


    # Maninpulate remote file paths to match the local machine directory
    local_file_path=($(for i in "${bison_remote_files[@]}"; do
                           echo $i |
                               gsed -E "s|/projects/bison/git/bison_[0-9]{8}|$HOME/Documents/projects/bison|g"
                       done
                     ))

    # Loop through remote and local and show where they will be placed
    for ((i=1; i<=${#bison_remote_files[@]}; i++)); do
        print -P "\u251C\U2500%B%1F%{Remote File ->%}%b%f ${bison_remote_files[i]}"
        print -P "\u251C\U2500%B%10F%{Local File  ->%}%b%f ${local_file_path[i]}"

        if [[ $i -lt ${#bison_remote_files[@]} ]]; then
            print -Pn "\U2502\n"
        else
            print -Pn "\U2514\U2500\U2504\U27E2\n"
        fi
    done

# If it's not a dry run, grab all the files using scp
# This is the part I am stuck...
# All my defined variables are un-run in the scope above
# How do I craft a function (or something else) so I don't have to do all the above all over again?    
else
    printf "${YELLOW}Fetching Data from ${NC}(${GREEN}${bison_latest_run}${NC})${YELLOW}...${NC}\n"

    for ((i=0; i<${#NEW_RFILEP[@]}; i++)); do

        scp -qp mcdodyla@falcon1:"${NEW_RFILEP[i]}" "${LOCAL_FILEP[i]}"

        # Check if scp was successful, if it was show green.
        if [ ${PIPESTATUS[0]} -eq 0 ]; then      
            printf "${GREEN}File Created/Updated at:${NC} ${LOCAL_FILEP[i]}\n"
        else
            printf "${RED}Error Fetching File:${NC} ${NEW_RFILEP[i]}\n"
        fi
    done
    printf "${YELLOW}Bison Remote Fetch Complete!${NC}\n"
fi

如您所见，我的所有数据都卡在第一个 if 语句案例中，因此如果我不想进行空运行，那么我必须再次运行所有代码。由于 bash/zsh 并没有真正返回数组，我该如何重构这段代码？

编辑：这是一个示例用例：

> bfetch -n "HBEP"

Initiating Dry-Run...

Fetching data from:  /projects/bison/git/bison_20190827
├─Remote File -> /projects/bison/git/bison_20190827/assessment/LWR/validation/HBEP/analysis/BK370/HBEP_BK370_out.csv
├─Local File  -> /Users/mcdodj/Documents/projects/bison/assessment/LWR/validation/HBEP/analysis/BK370/HBEP_BK370_out.csv
│
├─Remote File -> /projects/bison/git/bison_20190827/assessment/LWR/validation/HBEP/analysis/BK363/HBEP_BK363_out.csv
├─Local File  -> /Users/mcdodj/Documents/projects/bison/assessment/LWR/validation/HBEP/analysis/BK363/HBEP_BK363_out.csv
│
├─Remote File -> /projects/bison/git/bison_20190827/assessment/LWR/validation/HBEP/analysis/BK365/HBEP_BK365_out.csv
├─Local File  -> /Users/mcdodj/Documents/projects/bison/assessment/LWR/validation/HBEP/analysis/BK365/HBEP_BK365_out.csv

我有一个大型项目，我正在尝试查找不包含*_out.csv文件的目录。我看过其他类似的答案，我想我快到了。

我遇到的问题是我只想查看正在进行的目录，analysis/但我也不想查看一些也进行分析的特定目录。

我设置了一个小示例问题：

$ tree
.
├── case1
│   ├── analysis
│   │   ├── test1
│   │   │   ├── gold
│   │   │   └── test1_out.csv
│   │   └── test2
│   └── doc
└── case2
    ├── analysis
    │   ├── test3
    │   │   └── gold
    │   └── test4
    │       └── test4_out.csv
    └── doc

12 directories, 2 files

我不想查看标题为*/doc/*或的目录*/gold/*。我当前的命令是：

find . -type d -not -name "doc" -not -name "gold" '!' -exec test -e "{}/*_out.csv" ';' -print

结果是：

.
./case1
./case1/analysis
./case1/analysis/test1
./case1/analysis/test2
./case2
./case2/analysis
./case2/analysis/test3
./case2/analysis/test4

我理想的输出看起来像

./case1/analysis/test2
./case2/analysis/test3

如您所见，我当前find的命令排除了docandgold目录，但不排除有*_out.csv文件的目录，也不排除不继续的目录analysis/。

我正在尝试使用 bash 操作文件路径。我的目标的第一部分已经完成，我将文件路径的前半部分替换为我本地计算机上的路径。

$> fp=$"/projects/bison/git/bison_20190405/assessment/LWR/validation/HBEP/analysis/BK363/HBEP_BK363_out.csv"
$> echo $fp | 
    sed -E "s#/projects/bison/git/bison_[0-9]{8}#/Users/djm/Documents/projects/bison#"

$> /Users/djm/Documents/projects/bison/assessment/LWR/validation/HBEP/analysis/BK363/HBEP_BK363_out.csv

下一部分我遇到了麻烦。我想用 和 替换/analysis/文件名之间的所有内容/doc/figures/FILENAME，包括分析。我已经能够使用 Rscript 创建此功能，但无法弄清楚如何将文件名保留在管道语句中。

预期输入：

/Users/djm/Documents/projects/bison/assessment/LWR/validation/HBEP/analysis/BK363/HBEP_BK363_out.csv

预期输出：

/Users/djm/Documents/projects/bison/assessment/LWR/validation/HBEP/doc/figures/HBEP_BK363_out.csv

这是我当前的 Rscript 和必要的正则表达式：

library(dplyr)
library(stringr)

test <- "/projects/bison/git/bison_20190405/assessment/LWR/validation/HBEP/analysis/BK363/HBEP_BK363_out.csv"

str_replace_all(test, "/projects/bison/git/bison_[0-9]{8}",
                      "Users/djm/Documents/projects/bison") %>% 
  str_replace_all("(?:analysis).*$", 
                  paste0("doc/figures/", basename(.)))

#> [1] "Users/djm/Documents/projects/bison/assessment/LWR/validation/HBEP/doc/figures/HBEP_BK363_out.csv"

^{由reprex 包（v0.2.1）于 2019 年 4 月 9 日创建}

我正在编写一个 bash 脚本，该脚本需要*_out.csv从远程服务器上的目录中获取所有内容。所有这些文件都是另一个目录深处的几个目录。例如，假设目录名为ox_20190404/. 我可以通过以下方式找到我的所有文件：

find ox_20190404/assessment/LWR/validation -type f -name "*_out.csv"

这个问题回答了我的部分问题，但是由于我不想完整地复制目录，所以我需要弄清楚如何实现上述代码。假设我从这个开始：

$ dir="/projects/ox/git"
$ server="myusername@server"
$ scp $server:$dir/$(ssh $server 'ls -t $dir | head -1') .

我将如何从那里获取我需要的文件？

我的问题的最后一部分想知道是否有一种方法可以获取所有复制的文件并将它们放在远程服务器上的相同文件路径和目录中。