主页 / user-955882

Hooman Bahreini's questions

Martin Hope
Hooman Bahreini
Asked: 2020-10-30 12:24:31 +0800 CST
  • 8

我写了一个小的 Python 爬虫（使用 Scrapy 框架）。刮板需要无头浏览...我正在使用 ChromeDriver。

当我在没有任何 GUI 的 Ubuntu 服务器上运行此代码时，我必须安装 Xvfb 才能在我的 Ubuntu 服务器上运行 ChromeDriver（我遵循了本指南）

这是我的代码：

class MySpider(scrapy.Spider):
    name = 'my_spider'

    def __init__(self):
        # self.driver = webdriver.Chrome(ChromeDriverManager().install())
        chrome_options = Options()
        chrome_options.add_argument('--headless')
        chrome_options.add_argument('--no-sandbox')
        chrome_options.add_argument('--disable-dev-shm-usage')
        self.driver = webdriver.Chrome('/usr/bin/chromedriver', chrome_options=chrome_options)

我可以从 Ubuntu shell 运行上面的代码，它执行时没有任何错误：

ubuntu@ip-1-2-3-4:~/scrapers/my_scraper$ scrapy crawl my_spider

现在我想设置一个 cron 作业来每天运行上述命令：

# m h  dom mon dow   command
PATH=/usr/local/bin:/home/ubuntu/.local/bin/
05 12 * * * cd /home/ubuntu/scrapers/my_scraper && scrapy crawl my_spider >> /tmp/scraper.log 2>&1

但是 crontab 作业给了我以下错误：

Traceback (most recent call last):
  File "/home/ubuntu/.local/lib/python3.6/site-packages/scrapy/crawler.py", line 192, in crawl
    return self._crawl(crawler, *args, **kwargs)
  File "/home/ubuntu/.local/lib/python3.6/site-packages/scrapy/crawler.py", line 196, in _crawl
    d = crawler.crawl(*args, **kwargs)
  File "/home/ubuntu/.local/lib/python3.6/site-packages/twisted/internet/defer.py", line 1613, in unwindGenerator
    return _cancellableInlineCallbacks(gen)
  File "/home/ubuntu/.local/lib/python3.6/site-packages/twisted/internet/defer.py", line 1529, in _cancellableInlineCallbacks
    _inlineCallbacks(None, g, status)
--- <exception caught here> ---
  File "/home/ubuntu/.local/lib/python3.6/site-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
    result = g.send(result)
  File "/home/ubuntu/.local/lib/python3.6/site-packages/scrapy/crawler.py", line 86, in crawl
    self.spider = self._create_spider(*args, **kwargs)
  File "/home/ubuntu/.local/lib/python3.6/site-packages/scrapy/crawler.py", line 98, in _create_spider
    return self.spidercls.from_crawler(self, *args, **kwargs)
  File "/home/ubuntu/.local/lib/python3.6/site-packages/scrapy/spiders/__init__.py", line 19, in from_crawler
    spider = cls(*args, **kwargs)
  File "/home/ubuntu/scrapers/my_scraper/my_scraper/spiders/spider.py", line 27, in __init__
    self.driver = webdriver.Chrome('/usr/bin/chromedriver', chrome_options=chrome_options)
  File "/home/ubuntu/.local/lib/python3.6/site-packages/selenium/webdriver/chrome/webdriver.py", line 81, in __init__
    desired_capabilities=desired_capabilities)
  File "/home/ubuntu/.local/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 157, in __init__
    self.start_session(capabilities, browser_profile)
  File "/home/ubuntu/.local/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 252, in start_session
    response = self.execute(Command.NEW_SESSION, parameters)
  File "/home/ubuntu/.local/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute
    self.error_handler.check_response(response)
  File "/home/ubuntu/.local/lib/python3.6/site-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: unknown error: Chrome failed to start: exited abnormally
  (unknown error: DevToolsActivePort file doesn't exist)
  (The process started from chrome location /usr/bin/google-chrome is no longer running, so ChromeDriver is assuming that Chrome has crashed.)
  (Driver info: chromedriver=2.41.578700 (2f1ed5f9343c13f73144538f15c00b370eda6706),platform=Linux 5.4.0-1029-aws x86_64)

更新

这个答案帮助我解决了这个问题（但我不太明白为什么）

我echo $PATH在我的 Ubuntu shell 上运行并将值复制到 crontab 中：

PATH=/home/ubuntu/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
05 12 * * * cd /home/ubuntu/scrapers/my_scraper && scrapy crawl my_spider >> /tmp/scraper.log 2>&1

注意：由于我为这个问题创建了一个赏金，我很高兴将它奖励给任何解释为什么更改 PATH 解决了问题的答案。

command-line python google-chrome cron selenium
Martin Hope
Hooman Bahreini
Asked: 2019-10-14 16:55:23 +0800 CST
  • 2

我正在使用 s3fs 在我的 Ubuntu 服务器上安装一个 s3 存储桶：

sudo apt-get install s3fs

sudo vim /etc/passwd-s3fs // <--put user access key and secret key in this file

sudo chmod 640 /etc/passwd-s3fs // <-- change permission to password file

sudo vim /etc/fuse.conf // uncomment user_allow_other

现在挂载 s3 存储桶：

sudo s3fs -o allow_other s3-bucket-name /home/myuser/s3

我想确保重新启动后自动挂载存储桶，我该怎么做？

fstab mount startup automount
Martin Hope
Hooman Bahreini
Asked: 2019-08-22 21:10:46 +0800 CST
  • 1

我正在学习 bash，但无法理解以下示例中的输出重定向出了什么问题：

我有一个名为 myfile.txt 的文件，其内容如下。

熟能生巧

我将使用tr命令将P替换为p：

cat myfile.txt | tr P p

这就是我想要的，现在我要将结果放回原始文件中：

cat myfile.txt | tr P p > myfile.txt

但是执行完上面的命令后myfile.txt是空的……为什么会这样呢？

更新：

如果我将输出发送到另一个文件，那么它会按预期工作：

cat myfile.txt | tr P p > anotherfile.txt
command-line bash redirect
Martin Hope
Hooman Bahreini
Asked: 2019-08-19 19:41:43 +0800 CST
  • 1

我想下载并安装fuse，我正在关注本教程

下载并编译最新版本的fuse源代码。对于本文，我们使用 fuse 版本 3.5。按照这组命令将编译 fuse 并在内核中添加 fuse 模块。

cd /usr/src/
wget https://github.com/libfuse/libfuse/releases/download/fuse-3.5.0/fuse-3.5.0.tar.xz

我已经执行了上述 2 个命令，但似乎没有奏效 - 输出：

HTTP request sent, awaiting response... 302 Found
Location: https://github-production-release-asset-2e65be.s3.amazonaws.com/48296177/f5f40080-6080-11e9-831e-924025a26137?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20190819%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20190819T032718Z&X-Amz-Expires=300&X-Amz-Signature=9daad2fc48ac63c665e7bb3c4af49d43b4d9f04170b13e31cd134f8f4bf37c72&X-Amz-SignedHeaders=host&actor_id=0&response-content-disposition=attachment%3B%20filename%3Dfuse-3.5.0.tar.xz&response-content-type=application%2Foctet-stream [following]
--2019-08-19 03:27:18--  https://github-production-release-asset-2e65be.s3.amazonaws.com/48296177/f5f40080-6080-11e9-831e-924025a26137?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20190819%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20190819T032718Z&X-Amz-Expires=300&X-Amz-Signature=9daad2fc48ac63c665e7bb3c4af49d43b4d9f04170b13e31cd134f8f4bf37c72&X-Amz-SignedHeaders=host&actor_id=0&response-content-disposition=attachment%3B%20filename%3Dfuse-3.5.0.tar.xz&response-content-type=application%2Foctet-stream
Resolving github-production-release-asset-2e65be.s3.amazonaws.com (github-production-release-asset-2e65be.s3.amazonaws.com)... 52.217.36.188
Connecting to github-production-release-asset-2e65be.s3.amazonaws.com (github-production-release-asset-2e65be.s3.amazonaws.com)|52.217.36.188|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1021676 (998K) [application/octet-stream]
fuse-3.5.0.tar.xz: Permission denied

Cannot write to ‘fuse-3.5.0.tar.xz’ (Success).

我看不到任何正在下载的代码：/usr/src

software-installation wget
Martin Hope
Hooman Bahreini
Asked: 2019-05-22 02:17:17 +0800 CST
  • 4

我添加了以下 crontab 作业：

sudo crontab -e

在此处输入图像描述

58 * * * * ubuntu /home/ubuntu/backup/mysqlbackup.sh
>/home/ubuntu/backup/log/backup.log

我已经尝试过 root 和 ubuntu 用户。

当我运行命令时：

sudo crontab -l

在此处输入图像描述

当我运行时：

systemctl status cron

在此处输入图像描述

但是mysqlbackup.sh没有被执行。知道问题出在哪里，或者我如何诊断出了什么问题？

注意我只添加了几分钟来测试脚本

command-line bash cron
Martin Hope
Hooman Bahreini
Asked: 2019-05-21 20:09:03 +0800 CST
  • 1

我创建了以下 shell 脚本来备份 MySQL DB，对其进行压缩，然后将其复制到 s3 存储桶中：

#vim /home/ubuntu/backup/mysqlbackup2.sh

#!/bin/bash
## Backup mysql DB, zip it and then copy it to s3 bucket

mysqldump -hhostname -uusername dbName -p'p@ssW0rd' > db.sql

if [ $? -le 1 ] 
then
    # zip the file and copy it s3 bucket
    sudo gzip -9 db.sql
    s3cmd put db.sql.gz s://mys3bucket/
else 
    echo "Fail to backup MySQL DB" >> "backup.log"
fi

它做的一切都很好，备份被复制到 s3 存储桶。但我无法理解 shell 脚本的输出：

在此处输入图像描述

我了解密码警告，但为什么显示：'[' 0 -le 1 ']'？我的 if 条件有什么问题吗？

scripts bash
Martin Hope
Hooman Bahreini
Asked: 2019-05-21 17:24:06 +0800 CST
  • 2

我想创建一个 shell 脚本来自动备份 MySQL 数据库。稍后我会将其复制到 s3 存储桶中。

我创建了以下 shell 脚本：

#vim /home/ubuntu/backup/mysqlbackup.sh

#!/bin/bash
## Specify the name of the database that you want to backupbackup

# Database credentials
USER="user1"
PASSWORD="password"
HOST="hostname.compute.amazonaws.com"
DB_NAME="db1"

#Backup_Directory_Locations
BACKUPROOT="/home/ubuntu/backup"
TSTAMP=$(date +"%d-%b-%Y-%H-%M-%S")
S3BUCKET="s3://s3bucket"
#LOG_FILE="/home/ubuntu/backup/log/dump.log"

mysqldump  -h <HOST>  -u <USER>  --database <DB_NAME>  -p"$PASSWORD" > $BACKUPROOT/$DB_NAME-$TSTAMP.sql

然后从命令行，我运行脚本：

sudo bash -x ./mysqlbackup.sh

它失败了，告诉我：

主机：没有这样的文件或目录

在此处输入图像描述

command-line bash mysql
Martin Hope
Hooman Bahreini
Asked: 2019-05-20 04:55:22 +0800 CST
  • 12

我已按照此处的说明设置 Galera 集群。指令说我需要禁用 appArmor：

禁用 AppArmor

默认情况下，某些服务器（例如 Ubuntu）包括 AppArmor，这可能会阻止 mysqld 打开其他端口或运行脚本。您必须禁用 AppArmor 或将其配置为允许 mysqld...

所以我按照说明执行了以下命令：

sudo ln -s /etc/apparmor.d/usr /etc/apparmor.d/disable/.sbin.mysqld
sudo service apparmor restart

我已经完成了集群配置。但我不确定我是否正确禁用了 AppArmor，因为当我运行时：

sudo aa-status

我得到：

... some more output here
2 processes have profiles defined.
1 processes are in enforce mode.
   /usr/sbin/mysqld (1938)
1 processes are in complain mode.
snap.amazon-ssm-agent.amazon-ssm-agent (1295)
0 processes are unconfined but have a profile defined.

我看到 mysqld 处于强制模式...这是什么意思？这是否意味着对 MySQL 禁用 AppArmor？是否可以一起禁用 AppArmor？

mysql apparmor