AskOverflow.Dev

AskOverflow.Dev Logo AskOverflow.Dev Logo

AskOverflow.Dev Navigation

    • 主页
    • 系统&网络
    • Ubuntu
    • Unix
    • DBA
    • Computer
    • Coding
    • LangChain

Mobile menu

Close
  • 主页
  • 系统&网络
    • 最新
    • 热门
    • 标签
  • Ubuntu
    • 最新
    • 热门
    • 标签
  • Unix
    • 最新
    • 标签
  • DBA
    • 最新
    • 标签
  • Computer
    • 最新
    • 标签
  • Coding
    • 最新
    • 标签
主页 / server / 问题

问题[celery](server)

Martin Hope
artifex_knowledge
Asked: 2020-07-07 05:50:35 +0800 CST

使用 Nginx/Celery/Django 的 EC2 上的高 CPU 负载导致服务器失败

  • 0

我正在使用 Nginx 和 uWSGI 在 EC2 服务器上运行 Django Web 应用程序。我也让 Celery 运行一些后台任务(没有 CRON 作业,只是偶尔的用户操作)。

该应用程序处于早期封闭测试阶段,目前没有活跃用户。

在过去的三天里,服务器在经历超高 CPU 负载后会倒塌,似乎是随机的(见屏幕截图)。

在此之前,该应用程序运行数周都没有问题。我对网站进行了一些编程更改,但没有对服务器配置进行更改(主要是合并模型)。

我试图从日志中提取一些东西(Nginx access.log、error.log 和 Django debug.log),但我没有看到任何错误或奇怪的东西(现在无法访问日志)。

此外,如果我事先没有重新启动服务器,我在迁移模型更改(在 venv 中)时遇到了类似的效果。有时,即使重新启动服务器,它也会变得如此缓慢,我不得不等待几分钟才能重新启动 Celery。

我需要帮助才能找到调查问题的起点。有任何想法吗?

服务器负载

django nginx celery
  • 1 个回答
  • 213 Views
Martin Hope
vowel-house-might
Asked: 2016-09-10 07:33:23 +0800 CST

Oracle 和 Django 之间的 DatabaseError 与 Celery

  • 1

我在部署 Django-Celery-RabbitMQ 项目的最后一步时遇到了麻烦。

通过 systemd 运行 celery、httpd 和 rabbitmq-server 服务后,当我在 Django 中触发任务时,我看到了消息Received task: smartalec.tasks.match_loadfile_task

smartalec 是我的应用程序,match_loadfile_task 是我的任务,该项目称为 tcida_api。

但是大约 30 秒后,有一个巨大的堆栈跟踪,如下所示(完整的调试日志,下面有异常)。

[2016-09-09 15:34:20,040: ERROR/MainProcess] Task smartalec.tasks.match_loadfile_task[740dc792-7410-4418-aa44-dd9c56b63b04] raised unexpected: DatabaseError(' <cx_Oracle._Error object at 0x3a3ed50>',)

谷歌搜索“Django Oracle Celery DatabaseError”会得到一些关于在单个连接上共享线程的查询,但我不确定这是否适用于我,因为我在 project.settings 的数据库配置中有“threaded=TRUE”。

当我从命令行运行 celery 时,我得到了同样的结果。

sudo /opt/tcida/virtualenv/smartalec/bin/celery -A my_django_project worker -l info

我没有安装 django celery,django 可以让数据库查询没问题,通过 shell 和网页,配置在我的本地机器上使用 django 的开发服务器。

什么会导致或解决此异常?

完整的 Celery 调试日志,不包括完整的堆栈跟踪。

[amartin@ip-10-50-7-148 tcida_api]$ sudo /opt/tcida/virtualenv/smartalec/bin/celery -A tcida_api worker --concurrency 1 -l debug
/opt/tcida/virtualenv/smartalec/lib/python2.7/site-packages/celery/platforms.py:812: RuntimeWarning: You are running the worker with superuser privileges, which is absolutely not recommended!

Please specify a different user using the -u option.

User information: uid=0 euid=0 gid=0 egid=0

  uid=uid, euid=euid, gid=gid, egid=egid,
[2016-09-09 16:14:22,995: DEBUG/MainProcess] | Worker: Preparing bootsteps.
[2016-09-09 16:14:22,997: DEBUG/MainProcess] | Worker: Building graph...
[2016-09-09 16:14:22,997: DEBUG/MainProcess] | Worker: New boot order: {Timer, Hub, Queues (intra), Pool, Autoscaler, StateDB, Autoreloader, Beat, Consumer}
[2016-09-09 16:14:22,999: DEBUG/MainProcess] | Consumer: Preparing bootsteps.
[2016-09-09 16:14:23,000: DEBUG/MainProcess] | Consumer: Building graph...
[2016-09-09 16:14:23,003: DEBUG/MainProcess] | Consumer: New boot order: {Connection, Events, Mingle, Tasks, Control, Agent, Gossip, Heart, event loop}

 -------------- celery@ip-10-50-7-148.eu-west-1.compute.internal v3.1.23 (Cipater)
---- **** ----- 
--- * ***  * -- Linux-3.10.0-229.11.1.el7.x86_64-x86_64-with-redhat-7.1-Maipo
-- * - **** --- 
- ** ---------- [config]
- ** ---------- .> app:         tcida_api:0x1a91850
- ** ---------- .> transport:   amqp://guest:**@localhost:5672//
- ** ---------- .> results:     disabled://
- *** --- * --- .> concurrency: 1 (prefork)
-- ******* ---- 
--- ***** ----- [queues]
 -------------- .> celery           exchange=celery(direct) key=celery


[tasks]
  . celery.backend_cleanup
  . celery.chain
  . celery.chord
  . celery.chord_unlock
  . celery.chunks
  . celery.group
  . celery.map
  . celery.starmap
  . smartalec.tasks.match_loadfile_task

[2016-09-09 16:14:23,005: DEBUG/MainProcess] | Worker: Starting Hub
[2016-09-09 16:14:23,005: DEBUG/MainProcess] ^-- substep ok
[2016-09-09 16:14:23,005: DEBUG/MainProcess] | Worker: Starting Pool
[2016-09-09 16:14:23,029: DEBUG/MainProcess] ^-- substep ok
[2016-09-09 16:14:23,030: DEBUG/MainProcess] | Worker: Starting Consumer
[2016-09-09 16:14:23,030: DEBUG/MainProcess] | Consumer: Starting Connection
[2016-09-09 16:14:23,038: DEBUG/MainProcess] Start from server, version: 0.9, properties: {u'information': u'Licensed under the MPL.  See http://www.rabbitmq.com/', u'product': u'RabbitMQ', u'copyright': u'Copyright (C) 2007-2013 GoPivotal, Inc.', u'capabilities': {u'exchange_exchange_bindings': True, u'consumer_cancel_notify': True, u'publisher_confirms': True, u'basic.nack': True}, u'platform': u'Erlang/OTP', u'version': u'3.1.5'}, mechanisms: [u'PLAIN', u'AMQPLAIN'], locales: [u'en_US']
[2016-09-09 16:14:23,039: DEBUG/MainProcess] Open OK!
[2016-09-09 16:14:23,039: INFO/MainProcess] Connected to amqp://guest:**@127.0.0.1:5672//
[2016-09-09 16:14:23,040: DEBUG/MainProcess] ^-- substep ok
[2016-09-09 16:14:23,040: DEBUG/MainProcess] | Consumer: Starting Events
[2016-09-09 16:14:23,047: DEBUG/MainProcess] Start from server, version: 0.9, properties: {u'information': u'Licensed under the MPL.  See http://www.rabbitmq.com/', u'product': u'RabbitMQ', u'copyright': u'Copyright (C) 2007-2013 GoPivotal, Inc.', u'capabilities': {u'exchange_exchange_bindings': True, u'consumer_cancel_notify': True, u'publisher_confirms': True, u'basic.nack': True}, u'platform': u'Erlang/OTP', u'version': u'3.1.5'}, mechanisms: [u'PLAIN', u'AMQPLAIN'], locales: [u'en_US']
[2016-09-09 16:14:23,047: DEBUG/MainProcess] Open OK!
[2016-09-09 16:14:23,048: DEBUG/MainProcess] using channel_id: 1
[2016-09-09 16:14:23,048: DEBUG/MainProcess] Channel open
[2016-09-09 16:14:23,049: DEBUG/MainProcess] ^-- substep ok
[2016-09-09 16:14:23,049: DEBUG/MainProcess] | Consumer: Starting Mingle
[2016-09-09 16:14:23,049: INFO/MainProcess] mingle: searching for neighbors
[2016-09-09 16:14:23,050: DEBUG/MainProcess] using channel_id: 1
[2016-09-09 16:14:23,050: DEBUG/MainProcess] Channel open
[2016-09-09 16:14:24,056: INFO/MainProcess] mingle: all alone
[2016-09-09 16:14:24,057: DEBUG/MainProcess] ^-- substep ok
[2016-09-09 16:14:24,057: DEBUG/MainProcess] | Consumer: Starting Tasks
[2016-09-09 16:14:24,059: DEBUG/MainProcess] ^-- substep ok
[2016-09-09 16:14:24,059: DEBUG/MainProcess] | Consumer: Starting Control
[2016-09-09 16:14:24,059: DEBUG/MainProcess] using channel_id: 2
[2016-09-09 16:14:24,060: DEBUG/MainProcess] Channel open
[2016-09-09 16:14:24,062: DEBUG/MainProcess] ^-- substep ok
[2016-09-09 16:14:24,062: DEBUG/MainProcess] | Consumer: Starting Gossip
[2016-09-09 16:14:24,062: DEBUG/MainProcess] using channel_id: 3
[2016-09-09 16:14:24,063: DEBUG/MainProcess] Channel open
[2016-09-09 16:14:24,065: DEBUG/MainProcess] ^-- substep ok
[2016-09-09 16:14:24,065: DEBUG/MainProcess] | Consumer: Starting Heart
[2016-09-09 16:14:24,066: DEBUG/MainProcess] ^-- substep ok
[2016-09-09 16:14:24,066: DEBUG/MainProcess] | Consumer: Starting event loop
[2016-09-09 16:14:24,067: WARNING/MainProcess] celery@ip-10-50-7-148.eu-west-1.compute.internal ready.
[2016-09-09 16:14:24,067: DEBUG/MainProcess] | Worker: Hub.register Pool...
[2016-09-09 16:14:24,067: DEBUG/MainProcess] basic.qos: prefetch_count->4
[2016-09-09 16:14:31,300: INFO/MainProcess] Received task: smartalec.tasks.match_loadfile_task[51fed74e-f0ce-4cad-9a3d-283b830939d6]
[2016-09-09 16:14:31,300: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x1c50500> (args:(u'smartalec.tasks.match_loadfile_task', u'51fed74e-f0ce-4cad-9a3d-283b830939d6', [1], {}, {u'utc': True, u'is_eager': False, u'chord': None, u'group': None, u'args': [1], u'retries': 0, u'delivery_info': {u'priority': 0, u'redelivered': False, u'routing_key': u'celery', u'exchange': u'celery'}, u'expires': None, u'hostname': 'celery@ip-10-50-7-148.eu-west-1.compute.internal', u'task': u'smartalec.tasks.match_loadfile_task', u'callbacks': None, u'correlation_id': u'51fed74e-f0ce-4cad-9a3d-283b830939d6', u'errbacks': None, u'timelimit': [None, None], u'taskset': None, u'kwargs': {}, u'eta': None, u'reply_to': u'950dab7a-8608-37e0-991a-5e7c6dadbf4a', u'id': u'51fed74e-f0ce-4cad-9a3d-283b830939d6', u'headers': {}}) kwargs:{})
[2016-09-09 16:14:31,310: DEBUG/MainProcess] Task accepted: smartalec.tasks.match_loadfile_task[51fed74e-f0ce-4cad-9a3d-283b830939d6] pid:25239
[2016-09-09 16:15:26,475: ERROR/MainProcess] Task smartalec.tasks.match_loadfile_task[51fed74e-f0ce-4cad-9a3d-283b830939d6] raised unexpected: DatabaseError('<cx_Oracle._Error object at 0x36c9d50>',)
Traceback (most recent call last):
  File "/opt/tcida/virtualenv/smartalec/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/opt/tcida/virtualenv/smartalec/lib/python2.7/site-packages/celery/app/trace.py", line 438, in __protected_call__
    return self.run(*args, **kwargs)
  File "/opt/tcida/virtualenv/smartalec/lib/python2.7/site-packages/smartalec/tasks.py", line 14, in match_loadfile_task
    load_file = LoadFile.objects.get(pk=loadfile_pk)
  File "/opt/tcida/virtualenv/smartalec/lib/python2.7/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/opt/tcida/virtualenv/smartalec/lib/python2.7/site-packages/django/db/models/query.py", line 379, in get
    num = len(clone)
  File "/opt/tcida/virtualenv/smartalec/lib/python2.7/site-packages/django/db/models/query.py", line 238, in __len__
    self._fetch_all()
  File "/opt/tcida/virtualenv/smartalec/lib/python2.7/site-packages/django/db/models/query.py", line 1087, in _fetch_all
    self._result_cache = list(self.iterator())
  File "/opt/tcida/virtualenv/smartalec/lib/python2.7/site-packages/django/db/models/query.py", line 54, in __iter__
    results = compiler.execute_sql()
  File "/opt/tcida/virtualenv/smartalec/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 824, in execute_sql
    sql, params = self.as_sql()
  File "/opt/tcida/virtualenv/smartalec/lib/python2.7/site-packages/django/db/backends/oracle/compiler.py", line 22, in as_sql
    subquery=subquery,
  File "/opt/tcida/virtualenv/smartalec/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 376, in as_sql
    where, w_params = self.compile(self.where) if self.where is not None else ("", [])
  File "/opt/tcida/virtualenv/smartalec/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 353, in compile
    sql, params = node.as_sql(self, self.connection)
  File "/opt/tcida/virtualenv/smartalec/lib/python2.7/site-packages/django/db/models/sql/where.py", line 79, in as_sql
    sql, params = compiler.compile(child)
  File "/opt/tcida/virtualenv/smartalec/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 353, in compile
    sql, params = node.as_sql(self, self.connection)
  File "/opt/tcida/virtualenv/smartalec/lib/python2.7/site-packages/django/db/models/lookups.py", line 158, in as_sql
    rhs_sql = self.get_rhs_op(connection, rhs_sql)
  File "/opt/tcida/virtualenv/smartalec/lib/python2.7/site-packages/django/db/models/lookups.py", line 162, in get_rhs_op
    return connection.operators[self.lookup_name] % rhs
  File "/opt/tcida/virtualenv/smartalec/lib/python2.7/site-packages/django/db/backends/oracle/base.py", line 80, in __get__
    instance.cursor().close()
  File "/opt/tcida/virtualenv/smartalec/lib/python2.7/site-packages/django/db/backends/base/base.py", line 233, in cursor
    cursor = self.make_cursor(self._cursor())
  File "/opt/tcida/virtualenv/smartalec/lib/python2.7/site-packages/django/db/backends/base/base.py", line 204, in _cursor
    self.ensure_connection()
  File "/opt/tcida/virtualenv/smartalec/lib/python2.7/site-packages/django/db/backends/base/base.py", line 199, in ensure_connection
    self.connect()
  File "/opt/tcida/virtualenv/smartalec/lib/python2.7/site-packages/django/db/utils.py", line 94, in __exit__
    six.reraise(dj_exc_type, dj_exc_value, traceback)
  File "/opt/tcida/virtualenv/smartalec/lib/python2.7/site-packages/django/db/backends/base/base.py", line 199, in ensure_connection
    self.connect()
  File "/opt/tcida/virtualenv/smartalec/lib/python2.7/site-packages/django/db/backends/base/base.py", line 171, in connect
    self.connection = self.get_new_connection(conn_params)
  File "/opt/tcida/virtualenv/smartalec/lib/python2.7/site-packages/django/db/backends/oracle/base.py", line 216, in get_new_connection
    return Database.connect(conn_string, **conn_params)
DatabaseError: <cx_Oracle._Error object at 0x36c9d50>
django oracle rabbitmq rhel6 celery
  • 1 个回答
  • 372 Views
Martin Hope
user35348
Asked: 2010-02-19 13:22:14 +0800 CST

是否可以在启动时自动运行芹菜?

  • 6

我有基于 django 和 celery 的 python 服务器。每次计算机重新启动时,apache2 都会启动,因此我的服务器正在工作,但我必须手动重新启动 celery(转到我的项目目录并执行“python manage.py celeryd”)。什么是生产的正确解决方案?我的意思是,是否有可能将芹菜作为守护进程启动?

在这里http://github.com/ask/celery/tree/master/contrib/debian/init.d/我发现了两个脚本,它们看起来应该在 /etc/init.d/ 中,就像启动 apache2 的 apache2 脚本一样。但我把它们放到 /etc/init.d/ 中,我发现没有任何变化。

linux django celery
  • 3 个回答
  • 4605 Views

Sidebar

Stats

  • 问题 205573
  • 回答 270741
  • 最佳答案 135370
  • 用户 68524
  • 热门
  • 回答
  • Marko Smith

    新安装后 postgres 的默认超级用户用户名/密码是什么?

    • 5 个回答
  • Marko Smith

    SFTP 使用什么端口?

    • 6 个回答
  • Marko Smith

    命令行列出 Windows Active Directory 组中的用户?

    • 9 个回答
  • Marko Smith

    什么是 Pem 文件,它与其他 OpenSSL 生成的密钥文件格式有何不同?

    • 3 个回答
  • Marko Smith

    如何确定bash变量是否为空?

    • 15 个回答
  • Martin Hope
    Tom Feiner 如何按大小对 du -h 输出进行排序 2009-02-26 05:42:42 +0800 CST
  • Martin Hope
    Noah Goodrich 什么是 Pem 文件,它与其他 OpenSSL 生成的密钥文件格式有何不同? 2009-05-19 18:24:42 +0800 CST
  • Martin Hope
    Brent 如何确定bash变量是否为空? 2009-05-13 09:54:48 +0800 CST
  • Martin Hope
    cletus 您如何找到在 Windows 中打开文件的进程? 2009-05-01 16:47:16 +0800 CST

热门标签

linux nginx windows networking ubuntu domain-name-system amazon-web-services active-directory apache-2.4 ssh

Explore

  • 主页
  • 问题
    • 最新
    • 热门
  • 标签
  • 帮助
subwaysurfers
my femboy roommate

Footer

AskOverflow.Dev

关于我们

  • 关于我们
  • 联系我们

Legal Stuff

  • Privacy Policy

Language

  • Pt
  • Server
  • Unix

© 2023 AskOverflow.DEV All Rights Reserve