k8s使用多节点部署jms_all连接到同一数据库时sftp功能经常断连报错400

jms_all版本:v3.8.1

架构模式:

当节点为1时:

部署架构:

sftp功能正常:

当节点为1个以上时:
部署架构:

sftp功能报错:

查看日志:

补充说明:

  • sftp功能不能正常使用,但是ssh功能是正常的
  • 在两台机器使用jmsctl.sh安装jumpserver并连接到同一数据库时不会出现该问题

补充部署yaml文件:

  • jumpserver-svc.yaml:
---
apiVersion: v1
kind: Service
metadata:
  name: jumpserver-hs
  namespace: jumpserver
  labels:
    app: jumpserver
spec:
  selector:
    app: jumpserver
  clusterIP: None
  ports:
  - port: 80
    name: jumpserver-web
  - port: 2222
    name: jumpserver-connect

---
apiVersion: v1
kind: Service
metadata:
  name: jumpserver
  namespace: jumpserver
  labels:
    app: jumpserver
spec:
  selector:
    app: jumpserver
  clusterIP: 10.97.2.1
  ports:
  - port: 80
    targetPort: 80
    name: jumpserver-web
  - port: 2222
    targetPort: 2222
    name: jumpserver-connect

---
apiVersion: v1
kind: Service
metadata:
  name: jumpserver-np
  namespace: jumpserver
  labels:
    app: jumpserver
spec:
  selector:
    app: jumpserver
  type: NodePort
  ports:
  - port: 80
    targetPort: 80
    nodePort: 30200
    name: jumpserver-web
  - port: 2222
    targetPort: 2222
    nodePort: 30201
    name: jumpserver-connect
  • jumpserver.yaml
---
apiVersion: apps/v1
kind: StatefulSet 
metadata:
  name: jumpserver
  namespace: jumpserver
spec:
  serviceName: jumpserver-hs
  replicas: 2
  selector:
    matchLabels:
      app: jumpserver
  template:
    metadata:
      name: jumpserver
      namespace: jumpserver
      labels:
        app: jumpserver
    spec:
      # 解决pvc卷覆盖问题
      initContainers:
      - name: init-jumpserver-1
        image: jumpserver/jms_all:v3.8.1
        command:
        - /bin/sh
        - -c
        - if [ -z "$(ls -A /temp-volume-core-data)" ]; then cp -R /opt/jumpserver/data/* /temp-volume-core-data/; fi;
        volumeMounts:
        - name: jumpserver-data
          subPath: core-data
          mountPath: /temp-volume-core-data
      # 解决pvc卷覆盖的问题
      - name: init-jumpserver-2
        image: jumpserver/jms_all:v3.8.1
        command:
        - /bin/sh
        - -c
        - if [ -z "$(ls -A /temp-volume-nginx-log)" ]; then cp -R /var/log/nginx/* /temp-volume-nginx-log/; fi;
        volumeMounts:
        - name: jumpserver-data
          subPath: nginx-log
          mountPath: /temp-volume-nginx-log
      containers:
      - name: jumpserver
        image: jumpserver/jms_all:v3.8.1
        ports:
        - containerPort: 80
        - containerPort: 2222
        - containerPort: 30000
        securityContext: 
          privileged: true
        # 就绪探针
        readinessProbe:
          httpGet:
            path: /api/health
            port: 8080
          initialDelaySeconds: 10
          periodSeconds: 10
          successThreshold: 1
          failureThreshold: 2
        env:
        # 自行生成的随机字符串,不能包含特殊字符,长度推荐大于等于50
        - name: SECRET_KEY
          value: "66666666666666666666666666666666666666666666666666"
        # 自行生成的随机字符串,不能包含特殊字符,长度推荐大于等于24
        - name: BOOTSTRAP_TOKEN
          value: "666666666666666666666666"
        # 日志等级,测试环境推荐设置为DEBUG
        - name: LOG_LEVEL
          value: DEBUG
        # mysql地址
        - name: DB_HOST
          value: "192.xxx.xxx.xxx"
        # mysql端口
        - name: DB_PORT
          value: "30250"
        # mysql用户名
        - name: DB_USER
          value: jumpserver
        # mysql密码
        - name: DB_PASSWORD
          value: jumpserver
        # mysql数据库名称
        - name: DB_NAME
          value: jumpserver
        # redis
        - name: REDIS_HOST
          value: "192.xxx.xxx.xxx"
        # 自行生成的随机字符串,不能包含特殊字符,长度推荐大于等于24
        - name: REDIS_PORT
          value: "30251"
        # 自行生成的随机字符串,不能包含特殊字符,长度推荐大于等于50
        - name: REDIS_PASSWORD
          value: jumpserver
        # 防止跨域设置访问地址
        - name: DOMAINS
          value: 192.xxx.xxx.xxx:30200
        # koko lion使用redis共享
        # - name: SHARE_ROOM_TYPE
        #   value: redis
        volumeMounts:
        # Core持久化目录,存储录像日志
        - name: jumpserver-data
          subPath: core-data
          mountPath: /opt/jumpserver/data
        # Koko持久化目录
        - name: jumpserver-data
          subPath: koko-data
          mountPath: /opt/koko/data
        # Lion持久化目录
        - name: jumpserver-data
          subPath: lion-data
          mountPath: /opt/lion/data
        # Magnus持久化目录
        - name: jumpserver-data
          subPath: magnus-data
          mountPath: /opt/magnus/data
        # Kael持久化目录
        - name: jumpserver-data
          subPath: kael-data
          mountPath: /opt/kael/data
        # Chen持久化目录
        - name: jumpserver-data
          subPath: chen-data
          mountPath: /opt/chen/data
        # Nginx日志持久化目录
        - name: jumpserver-data
          subPath: nginx-log
          mountPath: /var/log/nginx
        # ssh密钥持久化目录
        - name: jumpserver-data
          subPath: ssh-data
          mountPath: /root/.ssh
  volumeClaimTemplates: 
  - metadata:
      name: jumpserver-data
    spec:
      accessModes: ["ReadWriteMany"]
      storageClassName: managed-nfs-storage-retain
      resources:
        requests:
          storage: 100Gi

单个节点状态正常,多个节点异常,还是k8s的。。。请问,你koko组件的会话保持做了么?

请问koko会话保持需要做什么设置吗,因为jms_all的部署文档只是使用docker部署也没有提到更改配置文件的操作,所以我也没做什么额外的设置。我之前倒是试过配置环境变量SHARE_ROOM_TYPE=redis,不过没什么效果

节选参考这里的配置哈反向代理 - JumpServer 文档

谢谢您的提示,我之前没有从负载均衡的角度去思考问题,查阅了一下资料后发现k8s的service虽然能做负载均衡,但是对于长连接这种,如果客户端和其中一个pod建立连接之后这个长连接没有断开,客户端也不会再和其他pod建立连接。我试验了一下为每个pod单独设置service,再各自单独访问之后发现sftp功能均正常了,说明问题就是出现在了service上,即负载均衡器的问题。我打算去试验一下在外部配置nginx对这些service再做负载均衡看看能不能成功。

有效果的话,麻烦脱敏分享一下配置哈,我们也学习学习

更改后的架构:

nginx配置:

    # 自定义变量$connection upgrade
    # 配置既支持ws请求,又支持http请求
    map $http_upgrade $connection_upgrade {
        default keep-alive;  # 默认keep alive
        'websocket' upgrade; # 如果为websocket则为upgrade可升级的
    }

    upstream jms_list {
        server 192.198.30.112:30202;
        server 192.198.30.112:30204;
    }

    server {
       listen       80;
       server_name  bj-jms.sfdomain.com;

       location / {
           proxy_pass http://jms_list;
            # proxy_pass http://192.198.30.112:30202;
            proxy_http_version 1.1;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection $connection_upgrade;
            proxy_redirect off;
            proxy_set_header Host $host; # 请确保 proxy_set_header Host $host; 是个标准的域名或者ip,只支持 -,不支持 _ 等其余符号
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
       }
    }

jumpserver.yaml(k8s-pod配置):

---
apiVersion: apps/v1
kind: StatefulSet 
metadata:
  name: jumpserver
  namespace: jumpserver
spec:
  serviceName: jumpserver-hs
  replicas: 2
  selector:
    matchLabels:
      app: jumpserver
  template:
    metadata:
      name: jumpserver
      namespace: jumpserver
      labels:
        app: jumpserver
    spec:
      # 解决pvc卷覆盖问题
      initContainers:
      - name: init-jumpserver-1
        image: jumpserver/jms_all:v3.8.1
        command:
        - /bin/sh
        - -c
        - if [ -z "$(ls -A /temp-volume-core-data)" ]; then cp -R /opt/jumpserver/data/* /temp-volume-core-data/; fi;
        volumeMounts:
        - name: jumpserver-data
          subPath: core-data
          mountPath: /temp-volume-core-data
      # 解决pvc卷覆盖的问题
      - name: init-jumpserver-2
        image: jumpserver/jms_all:v3.8.1
        command:
        - /bin/sh
        - -c
        - if [ -z "$(ls -A /temp-volume-nginx-log)" ]; then cp -R /var/log/nginx/* /temp-volume-nginx-log/; fi;
        volumeMounts:
        - name: jumpserver-data
          subPath: nginx-log
          mountPath: /temp-volume-nginx-log
      containers:
      - name: jumpserver
        image: jumpserver/jms_all:v3.8.1
        ports:
        - containerPort: 80
        - containerPort: 2222
        - containerPort: 30000
        securityContext: 
          privileged: true
        # 就绪探针
        readinessProbe:
          httpGet:
            path: /api/health
            port: 8080
          initialDelaySeconds: 10
          periodSeconds: 10
          successThreshold: 1
          failureThreshold: 2
        env:
        # 自行生成的随机字符串,不能包含特殊字符,长度推荐大于等于50
        - name: SECRET_KEY
          value: "******"
        # 自行生成的随机字符串,不能包含特殊字符,长度推荐大于等于24
        - name: BOOTSTRAP_TOKEN
          value: "******"
        # 日志等级,测试环境推荐设置为DEBUG
        - name: LOG_LEVEL
          value: DEBUG
        # mysql地址
        - name: DB_HOST
          value: "192.xxx.xxx.xxx"
        # mysql端口
        - name: DB_PORT
          value: "30250"
        # mysql用户名
        - name: DB_USER
          value: jumpserver
        # mysql密码
        - name: DB_PASSWORD
          value: jumpserver
        # mysql数据库名称
        - name: DB_NAME
          value: jumpserver
        # redis
        - name: REDIS_HOST
          value: "192.xxx.xxx.xxx"
        # 自行生成的随机字符串,不能包含特殊字符,长度推荐大于等于24
        - name: REDIS_PORT
          value: "30251"
        # 自行生成的随机字符串,不能包含特殊字符,长度推荐大于等于50
        - name: REDIS_PASSWORD
          value: jumpserver
        # 防止跨域设置访问地址
        - name: DOMAINS
          value: jms.xxx.com,192.xxx.xxx.xxx:30202,192.xxx.xxx.xxx:30204
        # koko lion使用redis共享
        - name: SHARE_ROOM_TYPE
          value: redis
        volumeMounts:
        # Core持久化目录,存储录像日志
        - name: jumpserver-data
          subPath: core-data
          mountPath: /opt/jumpserver/data
        # Koko持久化目录
        - name: jumpserver-data
          subPath: koko-data
          mountPath: /opt/koko/data
        # Lion持久化目录
        - name: jumpserver-data
          subPath: lion-data
          mountPath: /opt/lion/data
        # Magnus持久化目录
        - name: jumpserver-data
          subPath: magnus-data
          mountPath: /opt/magnus/data
        # Kael持久化目录
        - name: jumpserver-data
          subPath: kael-data
          mountPath: /opt/kael/data
        # Chen持久化目录
        - name: jumpserver-data
          subPath: chen-data
          mountPath: /opt/chen/data
        # Nginx日志持久化目录
        - name: jumpserver-data
          subPath: nginx-log
          mountPath: /var/log/nginx
        # ssh密钥持久化目录
        - name: jumpserver-data
          subPath: ssh-data
          mountPath: /root/.ssh
  volumeClaimTemplates: 
  - metadata:
      name: jumpserver-data
    spec:
      accessModes: ["ReadWriteMany"]
      storageClassName: managed-nfs-storage-retain
      resources:
        requests:
          storage: 100Gi

jumpserver-svc.yaml(k8s-service配置):

---
apiVersion: v1
kind: Service
metadata:
  name: jumpserver-hs
  namespace: jumpserver
  labels:
    app: jumpserver
spec:
  selector:
    app: jumpserver
  clusterIP: None
  ports:
  - port: 80
    name: jumpserver-web
  - port: 2222
    name: jumpserver-connect

---
apiVersion: v1
kind: Service
metadata:
  name: jumpserver
  namespace: jumpserver
  labels:
    app: jumpserver
spec:
  selector:
    app: jumpserver
  clusterIP: 10.97.2.1
  ports:
  - port: 80
    targetPort: 80
    name: jumpserver-web
  - port: 2222
    targetPort: 2222
    name: jumpserver-connect

---
apiVersion: v1
kind: Service
metadata:
  name: jumpserver-np
  namespace: jumpserver
  labels:
    app: jumpserver
spec:
  selector:
    app: jumpserver
  type: NodePort
  ports:
  - port: 80
    targetPort: 80
    nodePort: 30200
    name: jumpserver-web
  - port: 2222
    targetPort: 2222
    nodePort: 30201
    name: jumpserver-connect

---
apiVersion: v1
kind: Service
metadata:
  name: jumpserver-0-np
  namespace: jumpserver
  labels:
    app: jumpserver
spec:
  selector:
    statefulset.kubernetes.io/pod-name: jumpserver-0
  type: NodePort
  ports:
  - port: 80
    targetPort: 80
    nodePort: 30202
    name: jumpserver-web
  - port: 2222
    targetPort: 2222
    nodePort: 30203
    name: jumpserver-connect

---
apiVersion: v1
kind: Service
metadata:
  name: jumpserver-1-np
  namespace: jumpserver
  labels:
    app: jumpserver
spec:
  selector:
    statefulset.kubernetes.io/pod-name: jumpserver-1
  type: NodePort
  ports:
  - port: 80
    targetPort: 80
    nodePort: 30204
    name: jumpserver-web
  - port: 2222
    targetPort: 2222
    nodePort: 30205
    name: jumpserver-connect

已分享,我这个不是咱们提供的标准部署方案,所以可以会有问题,之后会再测试观察,您有建议的话也请多多指教。目前测试的结果是通过域名转发之后访问jms正常,使用ssh和sftp功能也正常,在尝试配置域名转发jms的ssh端口,不知道您有没有思路