新节点页面加载缓慢 日志报错 Start ws client failed: unexpected EOF
- JumpServer 版本：3.10.3 社区版
- 运行方式: 通过离线包 ./jmsctl.sh install 安装 使用外部mysql和redis
- 浏览器版本: Chrom 122.0.6261.69
事故描述： 原来在系统为centos8的机器 cbh-node1 和 cbh-node2 上部署了JumpServer 2.22.4 版本，近日按照官方文档，通过下载离线安装包的方式升级堡垒机, 升级版本依次为： 2.22.4 》 2.28.21 》 3.10.2
升级完成后 在系统为rocky linux 9.3的机器cbh-node3上 通过下载离线安装包jumpserver-offline-installer-v3.10.3-amd64.tar.gz 来安装 JumpServer 3.10.3 。安装完成后 通过http直接访问 cbh-node3的80端口 发现堡垒机UI页面加载及其缓慢(登陆后 1min 才能主页面才加载完成)。cbh-node1 cbh-node2 加载速度正常(登陆后5秒内主页面加载完成)
其中cbh-node3的硬件配置与cbh-node1 cbh-node2的硬件配置 网络条件完全一致。
以下分别是在cbh-node1 和 cbh-node3上调用core健康检查的表现，cbh-node3的接口耗时明显更久
haproxy 健康检查如下
根据官方文档，找到了 cbh-node3上有的 cbh-node1上没有的错误日志
koko.log 和 lion.log
2024-02-26 23:31:15 [ERRO] Start ws client failed: unexpected EOF
celery_default.log 报错信息
/opt/py3/lib/python3.11/site-packages/celery/app/utils.py:203: CDeprecationWarning:
The 'CELERY_QUEUES' setting is deprecated and scheduled for removal in
version 6.0.0. Use the task_queues instead
deprecated.warn(description=f'The {setting!r} setting',
gunicorn.log
[2024-02-26 01:39:37 +0800] [69] [ERROR] Exception in ASGI application
Traceback (most recent call last):
File "/opt/py3/lib/python3.11/site-packages/uvicorn/protocols/websockets/websockets_impl.py", line 254, in run_asgi
result = await self.app(self.scope, self.asgi_receive, self.asgi_send)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/py3/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 78, in __call__
return await self.app(scope, receive, send)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/py3/lib/python3.11/site-packages/channels/routing.py", line 62, in __call__
return await application(scope, receive, send)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/jumpserver/apps/jumpserver/routing.py", line 53, in __call__
return await self.app(scope, receive, send)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/py3/lib/python3.11/site-packages/channels/sessions.py", line 47, in __call__
return await self.inner(dict(scope, cookies=cookies), receive, send)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/py3/lib/python3.11/site-packages/channels/sessions.py", line 263, in __call__
return await self.inner(wrapper.scope, receive, wrapper.send)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/py3/lib/python3.11/site-packages/channels/auth.py", line 185, in __call__
return await super().__call__(scope, receive, send)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/py3/lib/python3.11/site-packages/channels/middleware.py", line 24, in __call__
return await self.inner(scope, receive, send)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/py3/lib/python3.11/site-packages/channels/routing.py", line 116, in __call__
return await application(
^^^^^^^^^^^^^^^^^^
File "/opt/py3/lib/python3.11/site-packages/channels/consumer.py", line 94, in app
return await consumer(scope, receive, send)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/py3/lib/python3.11/site-packages/channels/consumer.py", line 58, in __call__
await await_many_dispatch(
File "/opt/py3/lib/python3.11/site-packages/channels/utils.py", line 50, in await_many_dispatch
await dispatch(result)
File "/opt/py3/lib/python3.11/site-packages/asgiref/sync.py", line 479, in __call__
ret: _R = await loop.run_in_executor(
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/py3/lib/python3.11/site-packages/channels/db.py", line 13, in thread_handler
return super().thread_handler(loop, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/py3/lib/python3.11/site-packages/asgiref/sync.py", line 538, in thread_handler
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/opt/py3/lib/python3.11/site-packages/channels/consumer.py", line 125, in dispatch
handler(message)
File "/opt/py3/lib/python3.11/site-packages/channels/generic/websocket.py", line 38, in websocket_connect
self.connect()
File "/opt/jumpserver/apps/terminal/ws.py", line 27, in connect
self.sub = self.watch_component_event()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/jumpserver/apps/terminal/ws.py", line 63, in watch_component_event
self.send_tasks_msg()
File "/opt/jumpserver/apps/terminal/ws.py", line 50, in send_tasks_msg
self.send(bytes_data=content)
File "/opt/py3/lib/python3.11/site-packages/channels/generic/websocket.py", line 76, in send
super().send({"type": "websocket.send", "bytes": bytes_data})
File "/opt/py3/lib/python3.11/site-packages/channels/consumer.py", line 133, in send
self.base_send(message)
File "/opt/py3/lib/python3.11/site-packages/asgiref/sync.py", line 277, in __call__
return call_result.result()
^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/concurrent/futures/_base.py", line 449, in result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
raise self._exception
File "/opt/py3/lib/python3.11/site-packages/asgiref/sync.py", line 353, in main_wrap
result = await self.awaitable(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/py3/lib/python3.11/site-packages/channels/sessions.py", line 226, in send
return await self.real_send(message)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/py3/lib/python3.11/site-packages/uvicorn/protocols/websockets/websockets_impl.py", line 327, in asgi_send
await self.send(data) # type: ignore[arg-type]
^^^^^^^^^^^^^^^^^^^^^
File "/opt/py3/lib/python3.11/site-packages/websockets/legacy/protocol.py", line 635, in send
await self.ensure_open()
File "/opt/py3/lib/python3.11/site-packages/websockets/legacy/protocol.py", line 935, in ensure_open
raise self.connection_closed_exc()
websockets.exceptions.ConnectionClosedError: no close frame received or sent
完整日已上传
celery_default.log (34.5 KB)
gunicorn.log (1.7 MB)
koko.log (253.6 KB)
lion.log (322.8 KB)
希望万能的社区能帮忙定位问题原因，或者有类似经历的小伙伴可以提供一些解决问题的思路！！
致谢 JumpServer!!