主页 / user-16704

Franck Dernoncourt's questions

Martin Hope
Franck Dernoncourt
Asked: 2023-12-22 06:22:57 +0800 CST
  • 5

我想下载很多文件（>数千万）。我有每个文件的 URL。我的文件中有 URL 列表URLs.txt：

http://mydomain.com/0wd.pdf
http://mydomain.com/asz.pdf
http://mydomain.com/axz.pdf
http://mydomain.com/b00.pdf
http://mydomain.com/bb0.pdf
etc.

我可以通过下载它们wget -i URLs.txt，但是它会超过一个文件夹中可以放置的最大文件数。

如何下载这么大的 URL 列表，以便将下载的文件分成包含文件名首字母的子文件夹？例如，：

0/0wd.pdf
a/asz.pdf
a/axz.pdf
b/b00.pdf
b/bb0.pdf
etc.

如果这很重要的话，我使用 Ubuntu。

wget
Martin Hope
Franck Dernoncourt
Asked: 2023-08-12 09:20:17 +0800 CST
  • 6

我  在运行 Ubuntu 22.04 的 AWS EC2 实例上使用Doccano 。

为了运行 Doccano，我使用doccano webserver --port 8000.

但现在我想让 Doccano 在端口 80 上提供其网页。因此，我必须以 root 身份运行 Doccano。

在不知道 EC2 上的 root 密码的情况下，如何以 root 用户身份运行 doccano？

我试过

(doccanopy310) ubuntu@ip-172-30-33-321:~$ sudo doccano webserver --port 80
sudo: doccano: command not found

和

(doccanopy310) ubuntu@ip-172-30-33-321:~$ su
Password:

我不知道此 EC2 实例中的 root 密码。我没有任何 AWS 管理员权限：我只是获得了 IP + .pem 文件以通过 SSH 连接到它。

http.server重现此问题的另一种方法是在 Python 3（或SimpleHTTPServerPython 2）中使用：

python -m http.server 80

将产生：

(doccanopy310) ubuntu@ip-172-30-33-321:~$ sudo python -m http.server 80
sudo: python: command not found
root
Martin Hope
Franck Dernoncourt
Asked: 2023-08-12 08:31:30 +0800 CST
  • 5

我在 Ubuntu 22.04 上使用Doccano 。

当要求 Doccano 在端口 80 上提供服务时，我收到错误 "Can't connect to ('0.0.0.0', 80)"：问题是什么？

要求 Doccano 在端口 8000 上提供服务效果很好。

完整错误消息：

(doccanopy310) ubuntu@ip-172-30-33-321:~$ doccano webserver --port 80
[2023-08-12 00:16:09 +0000] [36905] [INFO] [django_drf_filepond.apps::ready::61] App init: no django-storages backend configured, using default (local) storage backend if set, otherwise you need to manage file storage independently of this app.
Starting server with port 80.
[2023-08-12 00:16:09 +0000] [36905] [INFO] Starting gunicorn 20.1.0
[2023-08-12 00:16:09 +0000] [36905] [ERROR] Retrying in 1 second.
[2023-08-12 00:16:10 +0000] [36905] [ERROR] Retrying in 1 second.
[2023-08-12 00:16:11 +0000] [36905] [ERROR] Retrying in 1 second.
[2023-08-12 00:16:12 +0000] [36905] [ERROR] Retrying in 1 second.
[2023-08-12 00:16:13 +0000] [36905] [ERROR] Retrying in 1 second.
[2023-08-12 00:16:14 +0000] [36905] [ERROR] Can't connect to ('0.0.0.0', 80)
(doccanopy310) ubuntu@ip-172-30-33-321:~$

重现：启动 AWS EC2 实例，然后运行以下命令。

我安装了 Doccano：

# Install conda
wget https://repo.anaconda.com/archive/Anaconda3-2023.03-1-Linux-x86_64.sh 
bash Anaconda3-2023.03-1-Linux-x86_64.sh -b
if ! [[ $PATH =~ "$HOME/anaconda3/bin" ]]; then
  PATH="$HOME/anaconda3/bin:$PATH"
fi
conda init bash
source ~/.bashrc 

# create conda env and install doccano
conda create -n doccanopy310 python=3.10 anaconda
conda activate doccanopy310
pip install doccano

为了启动 Doccano，我遵循了官方文档）。我跑：

# Initialize database. First time only.
doccano init
# Create a super user. First time only.
doccano createuser --username admin --password pass
# Start a web server.
doccano webserver --port 80

并从运行另一个终端：

# Start the task queue to handle file upload/download.
doccano task

要求 Doccano 在端口 8000 上提供服务效果很好：

(doccanopy310) ubuntu@ip-172-30-33-321:~$ doccano webserver --port 8000
[2023-08-12 00:21:53 +0000] [37206] [INFO] [django_drf_filepond.apps::ready::61] App init: no django-storages backend configured, using default (local) storage backend if set, otherwise you need to manage file storage independently of this app.
Starting server with port 8000.
[2023-08-12 00:21:53 +0000] [37206] [INFO] Starting gunicorn 20.1.0
[2023-08-12 00:21:53 +0000] [37206] [INFO] Listening at: http://0.0.0.0:8000 (37206)
[2023-08-12 00:21:53 +0000] [37206] [INFO] Using worker: sync
[2023-08-12 00:21:53 +0000] [37207] [INFO] Booting worker with pid: 37207
[2023-08-12 00:21:53 +0000] [37208] [INFO] Booting worker with pid: 37208
[2023-08-12 00:21:53 +0000] [37209] [INFO] Booting worker with pid: 37209
[2023-08-12 00:21:54 +0000] [37210] [INFO] Booting worker with pid: 37210
[2023-08-12 00:21:54 +0000] [37211] [INFO] Booting worker with pid: 37211

我没有看到任何程序使用端口 80：

debug to see which ports are being used: From within the machine netstat -ntlp / netstat -nulp will show all open TCP / UDP ports (and associated programs) respectively.
(doccanopy310) ubuntu@ip-172-30-33-321:~$ netstat -nulp
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
udp        0      0 127.0.0.53:53           0.0.0.0:*                           -
udp        0      0 172.30.33.321:68        0.0.0.0:*                           -
udp        0      0 127.0.0.1:323           0.0.0.0:*                           -
udp6       0      0 ::1:323                 :::*                                -

(doccanopy310) ubuntu@ip-172-30-33-321:~$ netstat -ntlp
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      -
tcp        0      0 0.0.0.0:8000            0.0.0.0:*               LISTEN      37206/python
tcp        0      0 127.0.0.53:53           0.0.0.0:*               LISTEN      -
tcp6       0      0 :::22                   :::*                    LISTEN      -
(doccanopy310) ubuntu@ip-172-30-33-321:~$

环境：

  • Ubuntu 22.04.2 LTS (GNU/Linux 5.15.0-1039-aws x86_64))。
ubuntu