一尘不染

Heroku和Redistogo Nano达到了最大的客户数量

redis

我在带有RedisToGo Nano插件的Heroku上使用celerybeat

有一个Web dyno和一个worker dyno

celerybeat工作者设置为每分钟执行一次任务。

问题是:每当我部署新的提交时,dynos重新启动,并且出现此错误

2014-02-27T13:19:31.552352+00:00 app[worker.1]: Traceback (most recent call last):
2014-02-27T13:19:31.552352+00:00 app[worker.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/celery/worker/consumer.py", line 389, in start
2014-02-27T13:19:31.552352+00:00 app[worker.1]:     self.reset_connection()
2014-02-27T13:19:31.552352+00:00 app[worker.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/celery/worker/consumer.py", line 727, in reset_connection
2014-02-27T13:19:31.552352+00:00 app[worker.1]:     self.connection = self._open_connection()
2014-02-27T13:19:31.552352+00:00 app[worker.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/celery/worker/consumer.py", line 792, in _open_connection
2014-02-27T13:19:31.552352+00:00 app[worker.1]:     callback=self.maybe_shutdown)
2014-02-27T13:18:23.864287+00:00 app[worker.1]:     self.on_connect()
2014-02-27T13:18:23.864287+00:00 app[worker.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/redis/connection.py", line 263, in on_connect
2014-02-27T13:18:23.864287+00:00 app[worker.1]:     if nativestr(self.read_response()) != 'OK':
2014-02-27T13:18:23.864287+00:00 app[worker.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/redis/connection.py", line 314, in read_response
2014-02-27T13:18:23.864287+00:00 app[worker.1]:     raise response
2014-02-27T13:18:23.864287+00:00 app[worker.1]: ResponseError: max number of clients reached
2014-02-27T13:19:31.552352+00:00 app[worker.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/kombu/connection.py", line 272, in ensure_connection
2014-02-27T13:19:31.552352+00:00 app[worker.1]:     interval_start, interval_step, interval_max, callback)
2014-02-27T13:19:31.552591+00:00 app[worker.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/kombu/utils/__init__.py", line 218, in retry_over_time
2014-02-27T13:19:31.552591+00:00 app[worker.1]:     return fun(*args, **kwargs)
2014-02-27T13:19:31.552591+00:00 app[worker.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/kombu/connection.py", line 162, in connect
2014-02-27T13:19:31.552591+00:00 app[worker.1]:     return self.connection
2014-02-27T13:19:31.552591+00:00 app[worker.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/kombu/connection.py", line 617, in connection
2014-02-27T13:18:23.870811+00:00 app[worker.1]: [2014-02-27 13:18:23,870: ERROR/MainProcess] consumer: Connection to broker lost. Trying to re-establish the connection...

那些日志不断地继续下去。直到我停止两个测功机并重新启动它们。

这已经成为一个问题,因为它几乎在每次部署新提交时都会发生。

有什么想法为什么会发生以及如何解决呢?


阅读 318

收藏
2020-06-20

共1个答案

一尘不染

纳米redistogo计划将同时进行的redis连接限制为 10

使用的redis连接数将根据您的前端和celery worker设置而有所不同。听起来您正在为生产堆栈使用 > = 5 redis连接

部署新代码时,Heroku会启动一个全新的堆栈。这意味着您在部署时正在使用 > = 10 redis连接

有两种方法可以解决此问题:

  • 通过升级到更大的计划($$$),增加允许的redistogo连接的最大数量
  • 减少堆栈使用的连接数(减少网络工作者使用的celery并发或redis连接)

这是资源枯竭的简单问题。我只需要支付更大的RedisToGo计划。

2020-06-20