Merge pull request #13236 from jumpserver/dev

v3.10.10-lts

.github/ISSUE_TEMPLATE/----.md

@@ -1,11 +1,35 @@
 ---
 name: 需求建议
 about: 提出针对本项目的想法和建议
-title: "[Feature] "
+title: "[Feature] 需求标题"
 labels: 类型:需求
 assignees: 
   - ibuler
   - baijiangjie
 ---
 
-**请描述您的需求或者改进建议.**
+## 注意
+_针对过于简单的需求描述不予考虑。请确保提供足够的细节和信息以支持功能的开发和实现。_
+
+## 功能名称
+[在这里输入功能的名称或标题]
+
+## 功能描述
+[在这里描述该功能的详细内容，包括其作用、目的和所需的功能]
+
+## 用户故事（可选）
+[如果适用，可以提供用户故事来更好地理解该功能的使用场景和用户期望]
+
+## 功能要求
+- [要求1：描述该功能的具体要求，如界面设计、交互逻辑等]
+- [要求2：描述该功能的另一个具体要求]
+- [以此类推，列出所有相关的功能要求]
+
+## 示例或原型（可选）
+[如果有的话，提供该功能的示例或原型图以更好地说明功能的实现方式]
+
+## 优先级
+[描述该功能的优先级，如高、中、低，或使用数字等其他标识]
+
+## 备注（可选）
+[在这里添加任何其他相关信息或备注]

.github/ISSUE_TEMPLATE/bug---.md

@@ -1,22 +1,51 @@
 ---
 name: Bug 提交
 about: 提交产品缺陷帮助我们更好的改进
-title: "[Bug] "
+title: "[Bug] Bug 标题"
 labels: 类型:Bug
 assignees: 
    - baijiangjie
 ---
 
-**JumpServer 版本( v2.28 之前的版本不再支持 )**
+## 注意
+**JumpServer 版本( v2.28 之前的版本不再支持 )** <br>
+_针对过于简单的 Bug 描述不予考虑。请确保提供足够的细节和信息以支持 Bug 的复现和修复。_
+
+## 当前使用的 JumpServer 版本 (必填)
+[在这里输入当前使用的 JumpServer 的版本号]
+
+## 使用的版本类型 (必填)
+- [ ] 社区版
+- [ ] 企业版
+- [ ] 企业试用版
 
 
-**浏览器版本**
+## 版本安装方式 (必填)
+- [ ] 在线安装 (一键命令)
+- [ ] 离线安装 (下载离线包)
+- [ ] All-in-One
+- [ ] 1Panel 安装
+- [ ] Kubernetes 安装
+- [ ] 源码安装
 
+## Bug 描述 (详细)
+[在这里描述 Bug 的详细情况，包括其影响和出现的具体情况]
 
-**Bug 描述**
+## 复现步骤
+1. [描述如何复现 Bug 的第一步]
+2. [描述如何复现 Bug 的第二步]
+3. [以此类推，列出所有复现 Bug 所需的步骤]
 
+## 期望行为
+[描述 Bug 出现时期望的系统行为或结果]
 
-**Bug 重现步骤(有截图更好)**
-1.
-2.
-3.
+## 实际行为
+[描述实际上发生了什么，以及 Bug 出现的具体情况]
+
+## 系统环境
+- 操作系统：[例如：Windows 10, macOS Big Sur]
+- 浏览器/应用版本：[如果适用，请提供相关版本信息]
+- 其他相关环境信息：[如果有其他相关环境信息，请在此处提供]
+
+## 附加信息（可选）
+[在这里添加任何其他相关信息，如截图、错误信息等]

.github/ISSUE_TEMPLATE/question.md

@@ -1,10 +1,50 @@
 ---
 name: 问题咨询
 about: 提出针对本项目安装部署、使用及其他方面的相关问题
-title: "[Question] "
+title: "[Question] 问题标题"
 labels: 类型:提问
 assignees: 
   - baijiangjie
 ---
+## 注意
+**请描述您的问题.** <br>
+**JumpServer 版本( v2.28 之前的版本不再支持 )** <br>
+_针对过于简单的 Bug 描述不予考虑。请确保提供足够的细节和信息以支持 Bug 的复现和修复。_
+
+## 当前使用的 JumpServer 版本 (必填)
+[在这里输入当前使用的 JumpServer 的版本号]
+
+## 使用的版本类型 (必填)
+- [ ] 社区版
+- [ ] 企业版
+- [ ] 企业试用版
+
+
+## 版本安装方式 (必填)
+- [ ] 在线安装 (一键命令)
+- [ ] 离线安装 (下载离线包)
+- [ ] All-in-One
+- [ ] 1Panel 安装
+- [ ] Kubernetes 安装
+- [ ] 源码安装
+
+## 问题描述 (详细)
+[在这里描述你遇到的问题]
+
+## 背景信息
+- 操作系统：[例如：Windows 10, macOS Big Sur]
+- 浏览器/应用版本：[如果适用，请提供相关版本信息]
+- 其他相关环境信息：[如果有其他相关环境信息，请在此处提供]
+
+## 具体问题
+[在这里详细描述你的问题，包括任何相关细节或错误信息]
+
+## 尝试过的解决方法
+[如果你已经尝试过解决问题，请在这里列出你已经尝试过的解决方法]
+
+## 预期结果
+[描述你期望的解决方案或结果]
+
+## 我们的期望
+[描述你希望我们提供的帮助或支持]
 
-**请描述您的问题.**

.github/workflows/jms-build-test.yml

@@ -1,26 +1,32 @@
 name: "Run Build Test"
 on:
   push:
-    branches:
-      - pr@*
-      - repr@*
+    paths:
+      - 'Dockerfile'
+      - 'Dockerfile-*'
+      - 'pyproject.toml'
+      - 'poetry.lock'
 
 jobs:
   build:
     runs-on: ubuntu-latest
     steps:
     - uses: actions/checkout@v3
+    - uses: docker/setup-qemu-action@v3
+    - uses: docker/setup-buildx-action@v3
 
-    - uses: docker/setup-qemu-action@v2
+    - name: Check Dockerfile
+      run: |
+        test -f Dockerfile-ce || cp -f Dockerfile Dockerfile-ce
 
-    - uses: docker/setup-buildx-action@v2
-
-    - uses: docker/build-push-action@v3
+    - name: Build CE Image
+      uses: docker/build-push-action@v5
       with:
         context: .
         push: false
-        tags: jumpserver/core-ce:test
         file: Dockerfile-ce
+        tags: jumpserver/core-ce:test
+        platforms: linux/amd64
         build-args: |
           APT_MIRROR=http://deb.debian.org
           PIP_MIRROR=https://pypi.org/simple
@@ -28,9 +34,22 @@ jobs:
         cache-from: type=gha
         cache-to: type=gha,mode=max
 
-    - uses: LouisBrunner/checks-action@v1.5.0
-      if: always()
+    - name: Prepare EE Image
+      run: |
+        sed -i 's@^FROM registry.fit2cloud.com@# FROM registry.fit2cloud.com@g' Dockerfile-ee
+        sed -i 's@^COPY --from=build-xpack@# COPY --from=build-xpack@g' Dockerfile-ee
+
+    - name: Build EE Image
+      uses: docker/build-push-action@v5
       with:
-        token: ${{ secrets.GITHUB_TOKEN }}
-        name: Check Build
-        conclusion: ${{ job.status }}
+        context: .
+        push: false
+        file: Dockerfile-ee
+        tags: jumpserver/core-ee:test
+        platforms: linux/amd64
+        build-args: |
+          APT_MIRROR=http://deb.debian.org
+          PIP_MIRROR=https://pypi.org/simple
+          PIP_JMS_MIRROR=https://pypi.org/simple
+        cache-from: type=gha
+        cache-to: type=gha,mode=max

.gitignore

@@ -43,3 +43,4 @@ releashe
 data/*
 test.py
 .history/
+.test/

Dockerfile-ce

@@ -19,11 +19,11 @@ ARG BUILD_DEPENDENCIES="              \
 
 ARG DEPENDENCIES="                    \
         freetds-dev                   \
-        libpq-dev                     \
         libffi-dev                    \
         libjpeg-dev                   \
         libkrb5-dev                   \
         libldap2-dev                  \
+        libpq-dev                     \
         libsasl2-dev                  \
         libssl-dev                    \
         libxml2-dev                   \
@@ -75,6 +75,7 @@ ENV LANG=zh_CN.UTF-8 \
 
 ARG DEPENDENCIES="                    \
         libjpeg-dev                   \
+        libpq-dev                     \
         libx11-dev                    \
         freerdp2-dev                  \
         libxmlsec1-openssl"
@@ -86,6 +87,7 @@ ARG TOOLS="                           \
         default-mysql-client          \
         iputils-ping                  \
         locales                       \
+        netcat-openbsd                \
         nmap                          \
         openssh-client                \
         patch                         \
@@ -110,8 +112,17 @@ RUN --mount=type=cache,target=/var/cache/apt,sharing=locked,id=core-apt \
     && sed -i "s@# export @export @g" ~/.bashrc \
     && sed -i "s@# alias @alias @g" ~/.bashrc
 
+ARG RECEPTOR_VERSION=v1.4.5
+RUN set -ex \
+    && wget -O /opt/receptor.tar.gz https://github.com/ansible/receptor/releases/download/${RECEPTOR_VERSION}/receptor_${RECEPTOR_VERSION/v/}_linux_${TARGETARCH}.tar.gz \
+    && tar -xf /opt/receptor.tar.gz -C /usr/local/bin/ \
+    && chown root:root /usr/local/bin/receptor \
+    && chmod 755 /usr/local/bin/receptor \
+    && rm -f /opt/receptor.tar.gz
+
 COPY --from=stage-2 /opt/py3 /opt/py3
 COPY --from=stage-1 /opt/jumpserver/release/jumpserver /opt/jumpserver
+COPY --from=stage-1 /opt/jumpserver/release/jumpserver/apps/libs/ansible/ansible.cfg /etc/ansible/
 
 WORKDIR /opt/jumpserver
 

README_EN.md

@@ -85,7 +85,7 @@ If you find a security problem, please contact us directly：
 - 400-052-0755
 
 ### License & Copyright
-Copyright (c) 2014-2022 FIT2CLOUD Tech, Inc., All rights reserved.
+Copyright (c) 2014-2024 FIT2CLOUD Tech, Inc., All rights reserved.
 
 Licensed under The GNU General Public License version 3 (GPLv3)  (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
 

apps/accounts/api/account/task.py

@@ -1,11 +1,12 @@
+from django.db.models import Q
 from rest_framework.generics import CreateAPIView
 
 from accounts import serializers
+from accounts.models import Account
 from accounts.permissions import AccountTaskActionPermission
 from accounts.tasks import (
     remove_accounts_task, verify_accounts_connectivity_task, push_accounts_to_assets_task
 )
-from assets.exceptions import NotSupportedTemporarilyError
 from authentication.permissions import UserConfirmation, ConfirmType
 
 __all__ = [
@@ -26,25 +27,35 @@ class AccountsTaskCreateAPI(CreateAPIView):
             ]
         return super().get_permissions()
 
-    def perform_create(self, serializer):
-        data = serializer.validated_data
-        accounts = data.get('accounts', [])
-        params = data.get('params')
+    @staticmethod
+    def get_account_ids(data, action):
+        account_type = 'gather_accounts' if action == 'remove' else 'accounts'
+        accounts = data.get(account_type, [])
         account_ids = [str(a.id) for a in accounts]
 
-        if data['action'] == 'push':
-            task = push_accounts_to_assets_task.delay(account_ids, params)
-        elif data['action'] == 'remove':
-            gather_accounts = data.get('gather_accounts', [])
-            gather_account_ids = [str(a.id) for a in gather_accounts]
-            task = remove_accounts_task.delay(gather_account_ids)
+        if action == 'remove':
+            return account_ids
+
+        assets = data.get('assets', [])
+        asset_ids = [str(a.id) for a in assets]
+        ids = Account.objects.filter(
+            Q(id__in=account_ids) | Q(asset_id__in=asset_ids)
+        ).distinct().values_list('id', flat=True)
+        return [str(_id) for _id in ids]
+
+    def perform_create(self, serializer):
+        data = serializer.validated_data
+        action = data['action']
+        ids = self.get_account_ids(data, action)
+
+        if action == 'push':
+            task = push_accounts_to_assets_task.delay(ids, data.get('params'))
+        elif action == 'remove':
+            task = remove_accounts_task.delay(ids)
+        elif action == 'verify':
+            task = verify_accounts_connectivity_task.delay(ids)
         else:
-            account = accounts[0]
-            asset = account.asset
-            if not asset.auto_config['ansible_enabled'] or \
-                    not asset.auto_config['ping_enabled']:
-                raise NotSupportedTemporarilyError()
-            task = verify_accounts_connectivity_task.delay(account_ids)
+            raise ValueError(f"Invalid action: {action}")
 
         data = getattr(serializer, '_data', {})
         data["task"] = task.id

apps/accounts/api/automations/backup.py

@@ -18,9 +18,8 @@ __all__ = [
 
 class AccountBackupPlanViewSet(OrgBulkModelViewSet):
     model = AccountBackupAutomation
-    filter_fields = ('name',)
-    search_fields = filter_fields
-    ordering = ('name',)
+    filterset_fields = ('name',)
+    search_fields = filterset_fields
     serializer_class = serializers.AccountBackupSerializer
 
 

apps/accounts/api/automations/base.py

@@ -20,8 +20,8 @@ __all__ = [
 class AutomationAssetsListApi(generics.ListAPIView):
     model = BaseAutomation
     serializer_class = serializers.AutomationAssetsSerializer
-    filter_fields = ("name", "address")
-    search_fields = filter_fields
+    filterset_fields = ("name", "address")
+    search_fields = filterset_fields
 
     def get_object(self):
         pk = self.kwargs.get('pk')

apps/accounts/api/automations/change_secret.py

@@ -6,9 +6,12 @@ from rest_framework.response import Response
 
 from accounts import serializers
 from accounts.const import AutomationTypes
+from accounts.filters import ChangeSecretRecordFilterSet
 from accounts.models import ChangeSecretAutomation, ChangeSecretRecord
 from accounts.tasks import execute_automation_record_task
+from authentication.permissions import UserConfirmation, ConfirmType
 from orgs.mixins.api import OrgBulkModelViewSet, OrgGenericViewSet
+from rbac.permissions import RBACPermission
 from .base import (
     AutomationAssetsListApi, AutomationRemoveAssetApi, AutomationAddAssetApi,
     AutomationNodeAddRemoveApi, AutomationExecutionViewSet
@@ -24,35 +27,54 @@ __all__ = [
 
 class ChangeSecretAutomationViewSet(OrgBulkModelViewSet):
     model = ChangeSecretAutomation
-    filter_fields = ('name', 'secret_type', 'secret_strategy')
-    search_fields = filter_fields
+    filterset_fields = ('name', 'secret_type', 'secret_strategy')
+    search_fields = filterset_fields
     serializer_class = serializers.ChangeSecretAutomationSerializer
 
 
 class ChangeSecretRecordViewSet(mixins.ListModelMixin, OrgGenericViewSet):
-    serializer_class = serializers.ChangeSecretRecordSerializer
-    filterset_fields = ('asset_id', 'execution_id')
+    filterset_class = ChangeSecretRecordFilterSet
     search_fields = ('asset__address',)
     tp = AutomationTypes.change_secret
+    serializer_classes = {
+        'default': serializers.ChangeSecretRecordSerializer,
+        'secret': serializers.ChangeSecretRecordViewSecretSerializer,
+    }
     rbac_perms = {
         'execute': 'accounts.add_changesecretexecution',
+        'secret': 'accounts.view_changesecretrecord',
     }
 
+    def get_permissions(self):
+        if self.action == 'secret':
+            self.permission_classes = [
+                RBACPermission,
+                UserConfirmation.require(ConfirmType.MFA)
+            ]
+        return super().get_permissions()
+
     def get_queryset(self):
         return ChangeSecretRecord.objects.all()
 
     @action(methods=['post'], detail=False, url_path='execute')
     def execute(self, request, *args, **kwargs):
-        record_id = request.data.get('record_id')
-        record = self.get_queryset().filter(pk=record_id)
-        if not record:
+        record_ids = request.data.get('record_ids')
+        records = self.get_queryset().filter(id__in=record_ids)
+        execution_count = records.values_list('execution_id', flat=True).distinct().count()
+        if execution_count != 1:
             return Response(
-                {'detail': 'record not found'},
-                status=status.HTTP_404_NOT_FOUND
+                {'detail': 'Only one execution is allowed to execute'},
+                status=status.HTTP_400_BAD_REQUEST
             )
-        task = execute_automation_record_task.delay(record_id, self.tp)
+        task = execute_automation_record_task.delay(record_ids, self.tp)
         return Response({'task': task.id}, status=status.HTTP_200_OK)
 
+    @action(methods=['get'], detail=True, url_path='secret')
+    def secret(self, request, *args, **kwargs):
+        instance = self.get_object()
+        serializer = self.get_serializer(instance)
+        return Response(serializer.data)
+
 
 class ChangSecretExecutionViewSet(AutomationExecutionViewSet):
     rbac_perms = (

apps/accounts/api/automations/gather_accounts.py

@@ -20,8 +20,8 @@ __all__ = [
 
 class GatherAccountsAutomationViewSet(OrgBulkModelViewSet):
     model = GatherAccountsAutomation
-    filter_fields = ('name',)
-    search_fields = filter_fields
+    filterset_fields = ('name',)
+    search_fields = filterset_fields
     serializer_class = serializers.GatherAccountAutomationSerializer
 
 

apps/accounts/api/automations/push_account.py

@@ -20,8 +20,8 @@ __all__ = [
 
 class PushAccountAutomationViewSet(OrgBulkModelViewSet):
     model = PushAccountAutomation
-    filter_fields = ('name', 'secret_type', 'secret_strategy')
-    search_fields = filter_fields
+    filterset_fields = ('name', 'secret_type', 'secret_strategy')
+    search_fields = filterset_fields
     serializer_class = serializers.PushAccountAutomationSerializer
 
 

apps/accounts/automations/backup_account/handlers.py

@@ -6,7 +6,7 @@ from django.conf import settings
 from rest_framework import serializers
 from xlsxwriter import Workbook
 
-from accounts.const.automation import AccountBackupType
+from accounts.const import AccountBackupType
 from accounts.models.automations.backup_account import AccountBackupAutomation
 from accounts.notifications import AccountBackupExecutionTaskMsg, AccountBackupByObjStorageExecutionTaskMsg
 from accounts.serializers import AccountSecretSerializer
@@ -168,9 +168,8 @@ class AccountBackupHandler:
             if not user.secret_key:
                 attachment_list = []
             else:
-                password = user.secret_key.encode('utf8')
                 attachment = os.path.join(PATH, f'{plan_name}-{local_now_filename()}-{time.time()}.zip')
-                encrypt_and_compress_zip_file(attachment, password, files)
+                encrypt_and_compress_zip_file(attachment, user.secret_key, files)
                 attachment_list = [attachment, ]
             AccountBackupExecutionTaskMsg(plan_name, user).publish(attachment_list)
             print('邮件已发送至{}({})'.format(user, user.email))
@@ -191,7 +190,6 @@ class AccountBackupHandler:
             attachment = os.path.join(PATH, f'{plan_name}-{local_now_filename()}-{time.time()}.zip')
             if password:
                 print('\033[32m>>> 使用加密密码对文件进行加密中\033[0m')
-                password = password.encode('utf8')
                 encrypt_and_compress_zip_file(attachment, password, files)
             else:
                 zip_files(attachment, files)

apps/accounts/automations/change_secret/custom/ssh/main.yml

@@ -18,6 +18,8 @@
         become_user: "{{ custom_become_user | default('') }}"
         become_password: "{{ custom_become_password | default('') }}"
         become_private_key_path: "{{ custom_become_private_key_path | default(None) }}"
+        old_ssh_version: "{{ jms_asset.old_ssh_version | default(False) }}"
+        gateway_args: "{{ jms_asset.ansible_ssh_common_args | default(None) }}"
       register: ping_info
       delegate_to: localhost
 
@@ -54,4 +56,6 @@
         become_user: "{{ account.become.ansible_user | default('') }}"
         become_password: "{{ account.become.ansible_password | default('') }}"
         become_private_key_path: "{{ account.become.ansible_ssh_private_key_file | default(None) }}"
+        old_ssh_version: "{{ jms_asset.old_ssh_version | default(False) }}"
+        gateway_args: "{{ jms_asset.ansible_ssh_common_args | default(None) }}"
       delegate_to: localhost

apps/accounts/automations/change_secret/custom/ssh/manifest.yml

@@ -7,6 +7,7 @@ type:
   - all
 method: change_secret
 protocol: ssh
+priority: 50
 params:
   - name: commands
     type: list

apps/accounts/automations/change_secret/database/oracle/main.yml

@@ -39,3 +39,4 @@
         login_host: "{{ jms_asset.address }}"
         login_port: "{{ jms_asset.port }}"
         login_database: "{{ jms_asset.spec_info.db_name }}"
+        mode: "{{ account.mode }}"

apps/accounts/automations/change_secret/host/aix/main.yml

@@ -85,6 +85,7 @@
         become_user: "{{ account.become.ansible_user | default('') }}"
         become_password: "{{ account.become.ansible_password | default('') }}"
         become_private_key_path: "{{ account.become.ansible_ssh_private_key_file | default(None) }}"
+        old_ssh_version: "{{ jms_asset.old_ssh_version | default(False) }}"
       when: account.secret_type == "password"
       delegate_to: localhost
 
@@ -95,5 +96,6 @@
         login_user: "{{ account.username }}"
         login_private_key_path: "{{ account.private_key_path  }}"
         gateway_args: "{{ jms_asset.ansible_ssh_common_args | default('') }}"
+        old_ssh_version: "{{ jms_asset.old_ssh_version | default(False) }}"
       when: account.secret_type == "ssh_key"
       delegate_to: localhost

apps/accounts/automations/change_secret/host/posix/main.yml

@@ -85,6 +85,7 @@
         become_user: "{{ account.become.ansible_user | default('') }}"
         become_password: "{{ account.become.ansible_password | default('') }}"
         become_private_key_path: "{{ account.become.ansible_ssh_private_key_file | default(None) }}"
+        old_ssh_version: "{{ jms_asset.old_ssh_version | default(False) }}"
       when: account.secret_type == "password"
       delegate_to: localhost
 
@@ -95,5 +96,6 @@
         login_user: "{{ account.username }}"
         login_private_key_path: "{{ account.private_key_path  }}"
         gateway_args: "{{ jms_asset.ansible_ssh_common_args | default('') }}"
+        old_ssh_version: "{{ jms_asset.old_ssh_version | default(False) }}"
       when: account.secret_type == "ssh_key"
       delegate_to: localhost

apps/accounts/automations/change_secret/host/windows_rdp_verify/manifest.yml

@@ -5,6 +5,7 @@ method: change_secret
 category: host
 type:
   - windows
+priority: 49
 params:
   - name: groups
     type: str

apps/accounts/automations/change_secret/manager.py

@@ -4,11 +4,12 @@ from copy import deepcopy
 
 from django.conf import settings
 from django.utils import timezone
+from django.utils.translation import gettext_lazy as _
 from xlsxwriter import Workbook
 
-from accounts.const import AutomationTypes, SecretType, SSHKeyStrategy, SecretStrategy
+from accounts.const import AutomationTypes, SecretType, SSHKeyStrategy, SecretStrategy, ChangeSecretRecordStatusChoice
 from accounts.models import ChangeSecretRecord
-from accounts.notifications import ChangeSecretExecutionTaskMsg
+from accounts.notifications import ChangeSecretExecutionTaskMsg, ChangeSecretFailedMsg
 from accounts.serializers import ChangeSecretRecordBackUpSerializer
 from assets.const import HostTypes
 from common.utils import get_logger
@@ -26,7 +27,7 @@ class ChangeSecretManager(AccountBasePlaybookManager):
 
     def __init__(self, *args, **kwargs):
         super().__init__(*args, **kwargs)
-        self.record_id = self.execution.snapshot.get('record_id')
+        self.record_map = self.execution.snapshot.get('record_map', {})
         self.secret_type = self.execution.snapshot.get('secret_type')
         self.secret_strategy = self.execution.snapshot.get(
             'secret_strategy', SecretStrategy.custom
@@ -118,14 +119,24 @@ class ChangeSecretManager(AccountBasePlaybookManager):
             else:
                 new_secret = self.get_secret(secret_type)
 
-            if self.record_id is None:
+            if new_secret is None:
+                print(f'new_secret is None, account: {account}')
+                continue
+
+            asset_account_id = f'{asset.id}-{account.id}'
+            if asset_account_id not in self.record_map:
                 recorder = ChangeSecretRecord(
                     asset=asset, account=account, execution=self.execution,
                     old_secret=account.secret, new_secret=new_secret,
                 )
                 records.append(recorder)
             else:
-                recorder = ChangeSecretRecord.objects.get(id=self.record_id)
+                record_id = self.record_map[asset_account_id]
+                try:
+                    recorder = ChangeSecretRecord.objects.get(id=record_id)
+                except ChangeSecretRecord.DoesNotExist:
+                    print(f"Record {record_id} not found")
+                    continue
 
             self.name_recorder_mapper[h['name']] = recorder
 
@@ -153,25 +164,43 @@ class ChangeSecretManager(AccountBasePlaybookManager):
         recorder = self.name_recorder_mapper.get(host)
         if not recorder:
             return
-        recorder.status = 'success'
+        recorder.status = ChangeSecretRecordStatusChoice.success.value
         recorder.date_finished = timezone.now()
-        recorder.save()
+
         account = recorder.account
         if not account:
             print("Account not found, deleted ?")
             return
         account.secret = recorder.new_secret
         account.date_updated = timezone.now()
-        account.save(update_fields=['secret', 'date_updated'])
+
+        max_retries = 3
+        retry_count = 0
+
+        while retry_count < max_retries:
+            try:
+                recorder.save()
+                account.save(update_fields=['secret', 'version', 'date_updated'])
+                break
+            except Exception as e:
+                retry_count += 1
+                if retry_count == max_retries:
+                    self.on_host_error(host, str(e), result)
+                else:
+                    print(f'retry {retry_count} times for {host} recorder save error: {e}')
+                    time.sleep(1)
 
     def on_host_error(self, host, error, result):
         recorder = self.name_recorder_mapper.get(host)
         if not recorder:
             return
-        recorder.status = 'failed'
+        recorder.status = ChangeSecretRecordStatusChoice.failed.value
         recorder.date_finished = timezone.now()
         recorder.error = error
-        recorder.save()
+        try:
+            recorder.save()
+        except Exception as e:
+            print(f"\033[31m Save {host} recorder error: {e} \033[0m\n")
 
     def on_runner_failed(self, runner, e):
         logger.error("Account error: ", e)
@@ -183,23 +212,56 @@ class ChangeSecretManager(AccountBasePlaybookManager):
             return False
         return True
 
+    @staticmethod
+    def get_summary(recorders):
+        total, succeed, failed = 0, 0, 0
+        for recorder in recorders:
+            if recorder.status == ChangeSecretRecordStatusChoice.success.value:
+                succeed += 1
+            else:
+                failed += 1
+            total += 1
+
+        summary = _('Success: %s, Failed: %s, Total: %s') % (succeed, failed, total)
+        return summary
+
     def run(self, *args, **kwargs):
         if self.secret_type and not self.check_secret():
             return
         super().run(*args, **kwargs)
-        if self.record_id:
-            return
-        recorders = self.name_recorder_mapper.values()
-        recorders = list(recorders)
-        self.send_recorder_mail(recorders)
+        recorders = list(self.name_recorder_mapper.values())
+        summary = self.get_summary(recorders)
+        print(summary, end='')
+
+        if self.record_map:
+            return
+
+        failed_recorders = [
+            r for r in recorders
+            if r.status == ChangeSecretRecordStatusChoice.failed.value
+        ]
 
-    def send_recorder_mail(self, recorders):
         recipients = self.execution.recipients
-        if not recorders or not recipients:
+        recipients = User.objects.filter(id__in=list(recipients.keys()))
+        if not recipients:
             return
 
-        recipients = User.objects.filter(id__in=list(recipients.keys()))
+        if failed_recorders:
+            name = self.execution.snapshot.get('name')
+            execution_id = str(self.execution.id)
+            _ids = [r.id for r in failed_recorders]
+            asset_account_errors = ChangeSecretRecord.objects.filter(
+                id__in=_ids).values_list('asset__name', 'account__username', 'error')
 
+            for user in recipients:
+                ChangeSecretFailedMsg(name, execution_id, user, asset_account_errors).publish()
+
+        if not recorders:
+            return
+
+        self.send_recorder_mail(recipients, recorders, summary)
+
+    def send_recorder_mail(self, recipients, recorders, summary):
         name = self.execution.snapshot['name']
         path = os.path.join(os.path.dirname(settings.BASE_DIR), 'tmp')
         filename = os.path.join(path, f'{name}-{local_now_filename()}-{time.time()}.xlsx')
@@ -209,11 +271,10 @@ class ChangeSecretManager(AccountBasePlaybookManager):
         for user in recipients:
             attachments = []
             if user.secret_key:
-                password = user.secret_key.encode('utf8')
                 attachment = os.path.join(path, f'{name}-{local_now_filename()}-{time.time()}.zip')
-                encrypt_and_compress_zip_file(attachment, password, [filename])
+                encrypt_and_compress_zip_file(attachment, user.secret_key, [filename])
                 attachments = [attachment]
-            ChangeSecretExecutionTaskMsg(name, user).publish(attachments)
+            ChangeSecretExecutionTaskMsg(name, user, summary).publish(attachments)
         os.remove(filename)
 
     @staticmethod

apps/accounts/automations/gather_accounts/host/windows/main.yml

@@ -1,9 +1,10 @@
 - hosts: demo
   gather_facts: no
   tasks:
-    - name: Gather posix account
+    - name: Gather windows account
       ansible.builtin.win_shell: net user
       register: result
+      ignore_errors: true
 
     - name: Define info by set_fact
       ansible.builtin.set_fact:

apps/accounts/automations/gather_accounts/manager.py

@@ -51,14 +51,22 @@ class GatherAccountsManager(AccountBasePlaybookManager):
         data = self.generate_data(asset, result)
         self.asset_account_info[asset] = data
 
+    @staticmethod
+    def get_nested_info(data, *keys):
+        for key in keys:
+            data = data.get(key, {})
+            if not data:
+                break
+        return data
+
     def on_host_success(self, host, result):
-        info = result.get('debug', {}).get('res', {}).get('info', {})
+        info = self.get_nested_info(result, 'debug', 'res', 'info')
         asset = self.host_asset_mapper.get(host)
         if asset and info:
             result = self.filter_success_result(asset.type, info)
             self.collect_asset_account_info(asset, result)
         else:
-            logger.error(f'Not found {host} info')
+            print(f'\033[31m Not found {host} info \033[0m\n')
 
     def update_or_create_accounts(self):
         for asset, data in self.asset_account_info.items():

apps/accounts/automations/push_account/database/oracle/main.yml

@@ -39,3 +39,4 @@
         login_host: "{{ jms_asset.address }}"
         login_port: "{{ jms_asset.port }}"
         login_database: "{{ jms_asset.spec_info.db_name }}"
+        mode: "{{ account.mode }}"

apps/accounts/automations/push_account/host/aix/main.yml

@@ -85,6 +85,7 @@
         become_user: "{{ account.become.ansible_user | default('') }}"
         become_password: "{{ account.become.ansible_password | default('') }}"
         become_private_key_path: "{{ account.become.ansible_ssh_private_key_file | default(None) }}"
+        old_ssh_version: "{{ jms_asset.old_ssh_version | default(False) }}"
       when: account.secret_type == "password"
       delegate_to: localhost
 
@@ -95,6 +96,7 @@
         login_user: "{{ account.username }}"
         login_private_key_path: "{{ account.private_key_path  }}"
         gateway_args: "{{ jms_asset.ansible_ssh_common_args | default('') }}"
+        old_ssh_version: "{{ jms_asset.old_ssh_version | default(False) }}"
       when: account.secret_type == "ssh_key"
       delegate_to: localhost
 

apps/accounts/automations/push_account/host/posix/main.yml

@@ -85,6 +85,7 @@
         become_user: "{{ account.become.ansible_user | default('') }}"
         become_password: "{{ account.become.ansible_password | default('') }}"
         become_private_key_path: "{{ account.become.ansible_ssh_private_key_file | default(None) }}"
+        old_ssh_version: "{{ jms_asset.old_ssh_version | default(False) }}"
       when: account.secret_type == "password"
       delegate_to: localhost
 
@@ -95,6 +96,7 @@
         login_user: "{{ account.username }}"
         login_private_key_path: "{{ account.private_key_path  }}"
         gateway_args: "{{ jms_asset.ansible_ssh_common_args | default('') }}"
+        old_ssh_version: "{{ jms_asset.old_ssh_version | default(False) }}"
       when: account.secret_type == "ssh_key"
       delegate_to: localhost
 

apps/accounts/automations/push_account/host/windows_rdp_verify/manifest.yml

@@ -5,6 +5,7 @@ method: push_account
 category: host
 type:
   - windows
+priority: 49
 params:
   - name: groups
     type: str

apps/accounts/automations/remove_account/manager.py

@@ -60,8 +60,11 @@ class RemoveAccountManager(AccountBasePlaybookManager):
         if not tuple_asset_gather_account:
             return
         asset, gather_account = tuple_asset_gather_account
-        Account.objects.filter(
-            asset_id=asset.id,
-            username=gather_account.username
-        ).delete()
-        gather_account.delete()
+        try:
+            Account.objects.filter(
+                asset_id=asset.id,
+                username=gather_account.username
+            ).delete()
+            gather_account.delete()
+        except Exception as e:
+            print(f'\033[31m Delete account {gather_account.username} failed: {e} \033[0m\n')

apps/accounts/automations/verify_account/custom/rdp/main.yml

@@ -3,6 +3,7 @@
   vars:
     ansible_shell_type: sh
     ansible_connection: local
+    ansible_python_interpreter: /opt/py3/bin/python
 
   tasks:
     - name: Verify account (pyfreerdp)

apps/accounts/automations/verify_account/custom/rdp/manifest.yml

@@ -6,6 +6,7 @@ type:
   - windows
 method: verify_account
 protocol: rdp
+priority: 1
 
 i18n:
   Windows rdp account verify:

apps/accounts/automations/verify_account/custom/ssh/main.yml

@@ -19,3 +19,5 @@
         become_user: "{{ account.become.ansible_user | default('') }}"
         become_password: "{{ account.become.ansible_password | default('') }}"
         become_private_key_path: "{{ account.become.ansible_ssh_private_key_file | default(None) }}"
+        old_ssh_version: "{{ jms_asset.old_ssh_version | default(False) }}"
+        gateway_args: "{{ jms_asset.ansible_ssh_common_args | default(None) }}"

apps/accounts/automations/verify_account/custom/ssh/manifest.yml

@@ -7,6 +7,7 @@ type:
   - all
 method: verify_account
 protocol: ssh
+priority: 50
 
 i18n:
   SSH account verify:

apps/accounts/automations/verify_account/manager.py

@@ -51,6 +51,9 @@ class VerifyAccountManager(AccountBasePlaybookManager):
             h['name'] += '(' + account.username + ')'
             self.host_account_mapper[h['name']] = account
             secret = account.secret
+            if secret is None:
+                print(f'account {account.name} secret is None')
+                continue
 
             private_key_path = None
             if account.secret_type == SecretType.SSH_KEY:
@@ -62,7 +65,7 @@ class VerifyAccountManager(AccountBasePlaybookManager):
                 'name': account.name,
                 'username': account.username,
                 'secret_type': account.secret_type,
-                'secret':  account.escape_jinja2_syntax(secret),
+                'secret': account.escape_jinja2_syntax(secret),
                 'private_key_path': private_key_path,
                 'become': account.get_ansible_become_auth(),
             }
@@ -73,8 +76,14 @@ class VerifyAccountManager(AccountBasePlaybookManager):
 
     def on_host_success(self, host, result):
         account = self.host_account_mapper.get(host)
-        account.set_connectivity(Connectivity.OK)
+        try:
+            account.set_connectivity(Connectivity.OK)
+        except Exception as e:
+            print(f'\033[31m Update account {account.name} connectivity failed: {e} \033[0m\n')
 
     def on_host_error(self, host, error, result):
         account = self.host_account_mapper.get(host)
-        account.set_connectivity(Connectivity.ERR)
+        try:
+            account.set_connectivity(Connectivity.ERR)
+        except Exception as e:
+            print(f'\033[31m Update account {account.name} connectivity failed: {e} \033[0m\n')

apps/accounts/const/account.py

@@ -15,6 +15,7 @@ class AliasAccount(TextChoices):
     INPUT = '@INPUT', _('Manual input')
     USER = '@USER', _('Dynamic user')
     ANON = '@ANON', _('Anonymous account')
+    SPEC = '@SPEC', _('Specified account')
 
     @classmethod
     def virtual_choices(cls):

apps/accounts/const/automation.py

@@ -16,7 +16,7 @@ DEFAULT_PASSWORD_RULES = {
 __all__ = [
     'AutomationTypes', 'SecretStrategy', 'SSHKeyStrategy', 'Connectivity',
     'DEFAULT_PASSWORD_LENGTH', 'DEFAULT_PASSWORD_RULES', 'TriggerChoice',
-    'PushAccountActionChoice', 'AccountBackupType'
+    'PushAccountActionChoice', 'AccountBackupType', 'ChangeSecretRecordStatusChoice',
 ]
 
 
@@ -103,3 +103,9 @@ class AccountBackupType(models.TextChoices):
     email = 'email', _('Email')
     # 目前只支持sftp方式
     object_storage = 'object_storage', _('SFTP')
+
+
+class ChangeSecretRecordStatusChoice(models.TextChoices):
+    failed = 'failed', _('Failed')
+    success = 'success', _('Success')
+    pending = 'pending', _('Pending')

apps/accounts/filters.py

@@ -5,7 +5,7 @@ from django_filters import rest_framework as drf_filters
 
 from assets.models import Node
 from common.drf.filters import BaseFilterSet
-from .models import Account, GatheredAccount
+from .models import Account, GatheredAccount, ChangeSecretRecord
 
 
 class AccountFilterSet(BaseFilterSet):
@@ -52,6 +52,7 @@ class AccountFilterSet(BaseFilterSet):
 class GatheredAccountFilterSet(BaseFilterSet):
     node_id = drf_filters.CharFilter(method='filter_nodes')
     asset_id = drf_filters.CharFilter(field_name='asset_id', lookup_expr='exact')
+    asset_name = drf_filters.CharFilter(field_name='asset__name', lookup_expr='icontains')
 
     @staticmethod
     def filter_nodes(queryset, name, value):
@@ -60,3 +61,13 @@ class GatheredAccountFilterSet(BaseFilterSet):
     class Meta:
         model = GatheredAccount
         fields = ['id', 'username']
+
+
+class ChangeSecretRecordFilterSet(BaseFilterSet):
+    asset_name = drf_filters.CharFilter(field_name='asset__name', lookup_expr='icontains')
+    account_username = drf_filters.CharFilter(field_name='account__username', lookup_expr='icontains')
+    execution_id = drf_filters.CharFilter(field_name='execution_id', lookup_expr='exact')
+
+    class Meta:
+        model = ChangeSecretRecord
+        fields = ['id', 'status', 'asset_id', 'execution']

apps/accounts/migrations/0014_virtualaccount.py

@@ -1,8 +1,9 @@
 # Generated by Django 4.1.10 on 2023-08-01 09:12
 
-from django.db import migrations, models
 import uuid
 
+from django.db import migrations, models
+
 
 class Migration(migrations.Migration):
 
@@ -20,7 +21,7 @@ class Migration(migrations.Migration):
                 ('date_updated', models.DateTimeField(auto_now=True, verbose_name='Date updated')),
                 ('id', models.UUIDField(default=uuid.uuid4, primary_key=True, serialize=False)),
                 ('org_id', models.CharField(blank=True, db_index=True, default='', max_length=36, verbose_name='Organization')),
-                ('alias', models.CharField(choices=[('@INPUT', 'Manual input'), ('@USER', 'Dynamic user'), ('@ANON', 'Anonymous account')], max_length=128, verbose_name='Alias')),
+                ('alias', models.CharField(choices=[('@INPUT', 'Manual input'), ('@USER', 'Dynamic user'), ('@ANON', 'Anonymous account'), ('@SPEC', 'Specified account')], max_length=128, verbose_name='Alias')),
                 ('secret_from_login', models.BooleanField(default=None, null=True, verbose_name='Secret from login')),
             ],
             options={

apps/accounts/models/automations/backup_account.py

@@ -8,7 +8,7 @@ from django.db import models
 from django.db.models import F
 from django.utils.translation import gettext_lazy as _
 
-from accounts.const.automation import AccountBackupType
+from accounts.const import AccountBackupType
 from common.const.choices import Trigger
 from common.db import fields
 from common.db.encoder import ModelJSONFieldEncoder

apps/accounts/models/automations/change_secret.py

@@ -2,7 +2,7 @@ from django.db import models
 from django.utils.translation import gettext_lazy as _
 
 from accounts.const import (
-    AutomationTypes
+    AutomationTypes, ChangeSecretRecordStatusChoice
 )
 from common.db import fields
 from common.db.models import JMSBaseModel
@@ -40,7 +40,10 @@ class ChangeSecretRecord(JMSBaseModel):
     new_secret = fields.EncryptTextField(blank=True, null=True, verbose_name=_('New secret'))
     date_started = models.DateTimeField(blank=True, null=True, verbose_name=_('Date started'))
     date_finished = models.DateTimeField(blank=True, null=True, verbose_name=_('Date finished'))
-    status = models.CharField(max_length=16, default='pending', verbose_name=_('Status'))
+    status = models.CharField(
+        max_length=16, verbose_name=_('Status'),
+        default=ChangeSecretRecordStatusChoice.pending.value
+    )
     error = models.TextField(blank=True, null=True, verbose_name=_('Error'))
 
     class Meta:

apps/accounts/models/base.py

@@ -137,16 +137,13 @@ class BaseAccount(VaultModelMixin, JMSOrgBaseModel):
         else:
             return None
 
-    @property
-    def private_key_path(self):
+    def get_private_key_path(self, path):
         if self.secret_type != SecretType.SSH_KEY \
                 or not self.secret \
                 or not self.private_key:
             return None
-        project_dir = settings.PROJECT_DIR
-        tmp_dir = os.path.join(project_dir, 'tmp')
         key_name = '.' + md5(self.private_key.encode('utf-8')).hexdigest()
-        key_path = os.path.join(tmp_dir, key_name)
+        key_path = os.path.join(path, key_name)
         if not os.path.exists(key_path):
             # https://github.com/ansible/ansible-runner/issues/544
             # ssh requires OpenSSH format keys to have a full ending newline.
@@ -158,6 +155,12 @@ class BaseAccount(VaultModelMixin, JMSOrgBaseModel):
             os.chmod(key_path, 0o400)
         return key_path
 
+    @property
+    def private_key_path(self):
+        project_dir = settings.PROJECT_DIR
+        tmp_dir = os.path.join(project_dir, 'tmp')
+        return self.get_private_key_path(tmp_dir)
+
     def get_private_key(self):
         if not self.private_key:
             return None

apps/accounts/notifications.py

@@ -1,6 +1,7 @@
 from django.template.loader import render_to_string
 from django.utils.translation import gettext_lazy as _
 
+from accounts.models import ChangeSecretRecord
 from common.tasks import send_mail_attachment_async, upload_backup_to_obj_storage
 from notifications.notifications import UserMessage
 from terminal.models.component.storage import ReplayStorage
@@ -54,20 +55,23 @@ class AccountBackupByObjStorageExecutionTaskMsg(object):
 class ChangeSecretExecutionTaskMsg(object):
     subject = _('Notification of implementation result of encryption change plan')
 
-    def __init__(self, name: str, user: User):
+    def __init__(self, name: str, user: User, summary):
         self.name = name
         self.user = user
+        self.summary = summary
 
     @property
     def message(self):
         name = self.name
         if self.user.secret_key:
-            return _('{} - The encryption change task has been completed. '
-                     'See the attachment for details').format(name)
+            default_message = _('{} - The encryption change task has been completed. '
+                                'See the attachment for details').format(name)
+
         else:
-            return _("{} - The encryption change task has been completed: the encryption "
-                     "password has not been set - please go to personal information -> "
-                     "file encryption password to set the encryption password").format(name)
+            default_message = _("{} - The encryption change task has been completed: the encryption "
+                                "password has not been set - please go to personal information -> "
+                                "set encryption password in preferences").format(name)
+        return self.summary + '\n' + default_message
 
     def publish(self, attachments=None):
         send_mail_attachment_async(
@@ -95,3 +99,35 @@ class GatherAccountChangeMsg(UserMessage):
     def gen_test_msg(cls):
         user = User.objects.first()
         return cls(user, {})
+
+
+class ChangeSecretFailedMsg(UserMessage):
+    subject = _('Change secret or push account failed information')
+
+    def __init__(self, name, execution_id, user, asset_account_errors: list):
+        self.name = name
+        self.execution_id = execution_id
+        self.asset_account_errors = asset_account_errors
+        super().__init__(user)
+
+    def get_html_msg(self) -> dict:
+        context = {
+            'name': self.name,
+            'recipient': self.user,
+            'execution_id': self.execution_id,
+            'asset_account_errors': self.asset_account_errors
+        }
+        message = render_to_string('accounts/change_secret_failed_info.html', context)
+
+        return {
+            'subject': str(self.subject),
+            'message': message
+        }
+
+    @classmethod
+    def gen_test_msg(cls):
+        name = 'test'
+        user = User.objects.first()
+        record = ChangeSecretRecord.objects.first()
+        execution_id = str(record.execution_id)
+        return cls(name, execution_id, user, [])

apps/accounts/serializers/account/account.py

@@ -58,7 +58,7 @@ class AccountCreateUpdateSerializerMixin(serializers.Serializer):
         for data in initial_data:
             if not data.get('asset') and not self.instance:
                 raise serializers.ValidationError({'asset': UniqueTogetherValidator.missing_message})
-            asset = data.get('asset') or self.instance.asset
+            asset = data.get('asset') or getattr(self.instance, 'asset', None)
             self.from_template_if_need(data)
             self.set_uniq_name_if_need(data, asset)
 
@@ -431,8 +431,11 @@ class AssetAccountBulkSerializer(
 
 class AccountSecretSerializer(SecretReadableMixin, AccountSerializer):
     class Meta(AccountSerializer.Meta):
+        fields = AccountSerializer.Meta.fields + ['spec_info']
         extra_kwargs = {
+            **AccountSerializer.Meta.extra_kwargs,
             'secret': {'write_only': False},
+            'spec_info': {'label': _('Spec info')},
         }
 
 
@@ -455,12 +458,14 @@ class AccountHistorySerializer(serializers.ModelSerializer):
 
 class AccountTaskSerializer(serializers.Serializer):
     ACTION_CHOICES = (
-        ('test', 'test'),
         ('verify', 'verify'),
         ('push', 'push'),
         ('remove', 'remove'),
     )
     action = serializers.ChoiceField(choices=ACTION_CHOICES, write_only=True)
+    assets = serializers.PrimaryKeyRelatedField(
+        queryset=Asset.objects, required=False, allow_empty=True, many=True
+    )
     accounts = serializers.PrimaryKeyRelatedField(
         queryset=Account.objects, required=False, allow_empty=True, many=True
     )

apps/accounts/serializers/account/base.py

@@ -67,15 +67,14 @@ class BaseAccountSerializer(AuthValidateMixin, ResourceLabelsMixin, BulkOrgResou
         fields_mini = ['id', 'name', 'username']
         fields_small = fields_mini + [
             'secret_type', 'secret', 'passphrase',
-            'privileged', 'is_active', 'spec_info',
+            'privileged', 'is_active',
         ]
         fields_other = ['created_by', 'date_created', 'date_updated', 'comment']
         fields = fields_small + fields_other + ['labels']
         read_only_fields = [
-            'spec_info', 'date_verified', 'created_by', 'date_created',
+            'date_verified', 'created_by', 'date_created',
         ]
         extra_kwargs = {
-            'spec_info': {'label': _('Spec info')},
             'username': {'help_text': _(
                 "Tip: If no username is required for authentication, fill in `null`, "
                 "If AD account, like `username@domain`"

apps/accounts/serializers/account/template.py

@@ -35,6 +35,7 @@ class AccountTemplateSerializer(BaseAccountSerializer):
             'su_from'
         ]
         extra_kwargs = {
+            **BaseAccountSerializer.Meta.extra_kwargs,
             'secret_strategy': {'help_text': _('Secret generation strategy for account creation')},
             'auto_push': {'help_text': _('Whether to automatically push the account to the asset')},
             'platforms': {
@@ -64,6 +65,9 @@ class AccountTemplateSerializer(BaseAccountSerializer):
 
 class AccountTemplateSecretSerializer(SecretReadableMixin, AccountTemplateSerializer):
     class Meta(AccountTemplateSerializer.Meta):
+        fields = AccountTemplateSerializer.Meta.fields + ['spec_info']
         extra_kwargs = {
+            **AccountTemplateSerializer.Meta.extra_kwargs,
             'secret': {'write_only': False},
+            'spec_info': {'label': _('Spec info')},
         }

apps/accounts/serializers/automations/base.py

@@ -21,6 +21,7 @@ __all__ = [
 class BaseAutomationSerializer(PeriodTaskSerializerMixin, BulkOrgResourceModelSerializer):
     assets = ObjectRelatedField(many=True, required=False, queryset=Asset.objects, label=_('Assets'))
     nodes = ObjectRelatedField(many=True, required=False, queryset=Node.objects, label=_('Nodes'))
+    is_periodic = serializers.BooleanField(default=False, required=False, label=_("Periodic perform"))
 
     class Meta:
         read_only_fields = [

apps/accounts/serializers/automations/change_secret.py

@@ -4,7 +4,8 @@ from django.utils.translation import gettext_lazy as _
 from rest_framework import serializers
 
 from accounts.const import (
-    AutomationTypes, SecretType, SecretStrategy, SSHKeyStrategy
+    AutomationTypes, SecretType, SecretStrategy,
+    SSHKeyStrategy, ChangeSecretRecordStatusChoice
 )
 from accounts.models import (
     Account, ChangeSecretAutomation,
@@ -21,6 +22,7 @@ logger = get_logger(__file__)
 __all__ = [
     'ChangeSecretAutomationSerializer',
     'ChangeSecretRecordSerializer',
+    'ChangeSecretRecordViewSecretSerializer',
     'ChangeSecretRecordBackUpSerializer',
     'ChangeSecretUpdateAssetSerializer',
     'ChangeSecretUpdateNodeSerializer',
@@ -104,7 +106,10 @@ class ChangeSecretAutomationSerializer(AuthValidateMixin, BaseAutomationSerializ
 class ChangeSecretRecordSerializer(serializers.ModelSerializer):
     is_success = serializers.SerializerMethodField(label=_('Is success'))
     asset = ObjectRelatedField(queryset=Asset.objects, label=_('Asset'))
-    account = ObjectRelatedField(queryset=Account.objects, label=_('Account'))
+    account = ObjectRelatedField(
+        queryset=Account.objects, label=_('Account'),
+        attrs=("id", "name", "username")
+    )
     execution = ObjectRelatedField(
         queryset=AutomationExecution.objects, label=_('Automation task execution')
     )
@@ -119,7 +124,16 @@ class ChangeSecretRecordSerializer(serializers.ModelSerializer):
 
     @staticmethod
     def get_is_success(obj):
-        return obj.status == 'success'
+        return obj.status == ChangeSecretRecordStatusChoice.success.value
+
+
+class ChangeSecretRecordViewSecretSerializer(serializers.ModelSerializer):
+    class Meta:
+        model = ChangeSecretRecord
+        fields = [
+            'id', 'old_secret', 'new_secret',
+        ]
+        read_only_fields = fields
 
 
 class ChangeSecretRecordBackUpSerializer(serializers.ModelSerializer):
@@ -145,7 +159,7 @@ class ChangeSecretRecordBackUpSerializer(serializers.ModelSerializer):
 
     @staticmethod
     def get_is_success(obj):
-        if obj.status == 'success':
+        if obj.status == ChangeSecretRecordStatusChoice.success.value:
             return _("Success")
         return _("Failed")
 

apps/accounts/signal_handlers.py

@@ -63,7 +63,7 @@ def create_accounts_activities(account, action='create'):
 def on_account_create_by_template(sender, instance, created=False, **kwargs):
     if not created or instance.source != 'template':
         return
-    push_accounts_if_need(accounts=(instance,))
+    push_accounts_if_need.delay(accounts=(instance,))
     create_accounts_activities(instance, action='create')
 
 

apps/accounts/tasks/automation.py

@@ -36,14 +36,14 @@ def execute_account_automation_task(pid, trigger, tp):
         instance.execute(trigger)
 
 
-def record_task_activity_callback(self, record_id, *args, **kwargs):
+def record_task_activity_callback(self, record_ids, *args, **kwargs):
     from accounts.models import ChangeSecretRecord
     with tmp_to_root_org():
-        record = get_object_or_none(ChangeSecretRecord, id=record_id)
-    if not record:
+        records = ChangeSecretRecord.objects.filter(id__in=record_ids)
+    if not records:
         return
-    resource_ids = [record.id]
-    org_id = record.execution.org_id
+    resource_ids = [str(i.id) for i in records]
+    org_id = records[0].execution.org_id
     return resource_ids, org_id
 
 
@@ -51,22 +51,26 @@ def record_task_activity_callback(self, record_id, *args, **kwargs):
     queue='ansible', verbose_name=_('Execute automation record'),
     activity_callback=record_task_activity_callback
 )
-def execute_automation_record_task(record_id, tp):
+def execute_automation_record_task(record_ids, tp):
     from accounts.models import ChangeSecretRecord
+    task_name = gettext_noop('Execute automation record')
+
     with tmp_to_root_org():
-        instance = get_object_or_none(ChangeSecretRecord, pk=record_id)
-    if not instance:
-        logger.error("No automation record found: {}".format(record_id))
+        records = ChangeSecretRecord.objects.filter(id__in=record_ids)
+
+    if not records:
+        logger.error('No automation record found: {}'.format(record_ids))
         return
 
-    task_name = gettext_noop('Execute automation record')
+    record = records[0]
+    record_map = {f'{record.asset_id}-{record.account_id}': str(record.id) for record in records}
     task_snapshot = {
-        'secret': instance.new_secret,
-        'secret_type': instance.execution.snapshot.get('secret_type'),
-        'accounts': [str(instance.account_id)],
-        'assets': [str(instance.asset_id)],
         'params': {},
-        'record_id': record_id,
+        'record_map': record_map,
+        'secret': record.new_secret,
+        'secret_type': record.execution.snapshot.get('secret_type'),
+        'assets': [str(instance.asset_id) for instance in records],
+        'accounts': [str(instance.account_id) for instance in records],
     }
-    with tmp_to_org(instance.execution.org_id):
+    with tmp_to_org(record.execution.org_id):
         quickstart_automation_by_snapshot(task_name, tp, task_snapshot)

apps/accounts/tasks/template.py

@@ -29,7 +29,8 @@ def template_sync_related_accounts(template_id, user_id=None):
     name = template.name
     username = template.username
     secret_type = template.secret_type
-    print(f'\033[32m>>> 开始同步模版名称、用户名、密钥类型到相关联的账号 ({datetime.now().strftime("%Y-%m-%d %H:%M:%S")})')
+    print(
+        f'\033[32m>>> 开始同步模板名称、用户名、密钥类型到相关联的账号 ({datetime.now().strftime("%Y-%m-%d %H:%M:%S")})')
     with tmp_to_org(org_id):
         for account in accounts:
             account.name = name

apps/accounts/templates/accounts/asset_account_change_info.html

@@ -1,10 +1,10 @@
 {% load i18n %}
 
-
+<h3>{% trans 'Gather account change information' %}</h3>
 <table style="width: 100%; border-collapse: collapse; max-width: 100%; text-align: left; margin-top: 20px;">
     <caption></caption>
     <tr style="background-color: #f2f2f2;">
-        <th style="border: 1px solid #ddd; padding: 10px; font-weight: bold;">{% trans 'Asset' %}</th>
+        <th style="border: 1px solid #ddd; padding: 10px;">{% trans 'Asset' %}</th>
         <th style="border: 1px solid #ddd; padding: 10px;">{% trans 'Added account' %}</th>
         <th style="border: 1px solid #ddd; padding: 10px;">{% trans 'Deleted account' %}</th>
     </tr>

apps/accounts/templates/accounts/change_secret_failed_info.html

@@ -0,0 +1,36 @@
+{% load i18n %}
+
+<h3>{% trans 'Task name' %}: {{ name }}</h3>
+<h3>{% trans 'Task execution id' %}: {{ execution_id }}</h3>
+<p>{% trans 'Respectful' %} {{ recipient }}</p>
+<p>{% trans 'Hello! The following is the failure of changing the password of your assets or pushing the account. Please check and handle it in time.' %}</p>
+<table style="width: 100%; border-collapse: collapse; max-width: 100%; text-align: left; margin-top: 20px;">
+    <caption></caption>
+    <thead>
+    <tr style="background-color: #f2f2f2;">
+        <th style="border: 1px solid #ddd; padding: 10px;">{% trans 'Asset' %}</th>
+        <th style="border: 1px solid #ddd; padding: 10px;">{% trans 'Account' %}</th>
+        <th style="border: 1px solid #ddd; padding: 10px;">{% trans 'Error' %}</th>
+    </tr>
+    </thead>
+    <tbody>
+    {% for asset_name, account_username, error in asset_account_errors %}
+        <tr>
+            <td style="border: 1px solid #ddd; padding: 10px;">{{ asset_name }}</td>
+            <td style="border: 1px solid #ddd; padding: 10px;">{{ account_username }}</td>
+            <td style="border: 1px solid #ddd; padding: 10px;">
+                <div style="
+                max-width: 90%;
+                white-space: nowrap;
+                overflow: hidden;
+                text-overflow: ellipsis;
+                display: block;"
+                     title="{{ error }}"
+                >
+                    {{ error }}
+                </div>
+            </td>
+        </tr>
+    {% endfor %}
+    </tbody>
+</table>

apps/acls/notifications.py

@@ -41,21 +41,21 @@ class UserLoginReminderMsg(UserMessage):
 class AssetLoginReminderMsg(UserMessage):
     subject = _('Asset login reminder')
 
-    def __init__(self, user, asset: Asset, login_user: User, account_username):
+    def __init__(self, user, asset: Asset, login_user: User, account: Account, input_username):
         self.asset = asset
         self.login_user = login_user
-        self.account_username = account_username
+        self.account = account
+        self.input_username = input_username
         super().__init__(user)
 
     def get_html_msg(self) -> dict:
-        account = Account.objects.get(asset=self.asset, username=self.account_username)
         context = {
             'recipient': self.user,
             'username': self.login_user.username,
             'name': self.login_user.name,
             'asset': str(self.asset),
-            'account': self.account_username,
-            'account_name': account.name,
+            'account': self.input_username,
+            'account_name': self.account.name,
         }
         message = render_to_string('acls/asset_login_reminder.html', context)
 

apps/assets/api/asset/asset.py

@@ -32,6 +32,7 @@ __all__ = [
 
 class AssetFilterSet(BaseFilterSet):
     platform = django_filters.CharFilter(method='filter_platform')
+    exclude_platform = django_filters.CharFilter(field_name="platform__name", lookup_expr='exact', exclude=True)
     domain = django_filters.CharFilter(method='filter_domain')
     type = django_filters.CharFilter(field_name="platform__type", lookup_expr="exact")
     category = django_filters.CharFilter(field_name="platform__category", lookup_expr="exact")

apps/assets/api/asset/permission.py

@@ -48,7 +48,7 @@ class AssetPermUserListApi(BaseAssetPermUserOrUserGroupListApi):
 
     def get_queryset(self):
         perms = self.get_asset_related_perms()
-        users = User.objects.filter(
+        users = User.get_queryset().filter(
             Q(assetpermissions__in=perms) | Q(groups__assetpermissions__in=perms)
         ).distinct()
         return users

apps/assets/api/domain.py

@@ -19,7 +19,6 @@ class DomainViewSet(OrgBulkModelViewSet):
     model = Domain
     filterset_fields = ("name",)
     search_fields = filterset_fields
-    ordering = ('name',)
     serializer_classes = {
         'default': serializers.DomainSerializer,
         'list': serializers.DomainListSerializer,
@@ -30,6 +29,10 @@ class DomainViewSet(OrgBulkModelViewSet):
             return serializers.DomainWithGatewaySerializer
         return super().get_serializer_class()
 
+    def partial_update(self, request, *args, **kwargs):
+        kwargs['partial'] = True
+        return self.update(request, *args, **kwargs)
+
 
 class GatewayViewSet(HostViewSet):
     perm_model = Gateway

apps/assets/api/node.py

@@ -22,6 +22,7 @@ from orgs.utils import current_org
 from rbac.permissions import RBACPermission
 from .. import serializers
 from ..models import Node
+from ..signal_handlers import update_nodes_assets_amount
 from ..tasks import (
     update_node_assets_hardware_info_manual,
     test_node_assets_connectivity_manual,
@@ -94,6 +95,7 @@ class NodeAddChildrenApi(generics.UpdateAPIView):
         children = Node.objects.filter(id__in=node_ids)
         for node in children:
             node.parent = instance
+        update_nodes_assets_amount.delay(ttl=5, node_ids=(instance.id,))
         return Response("OK")
 
 

apps/assets/api/platform.py

@@ -21,6 +21,7 @@ class AssetPlatformViewSet(JMSModelViewSet):
     }
     filterset_fields = ['name', 'category', 'type']
     search_fields = ['name']
+    ordering = ['-internal', 'name']
     rbac_perms = {
         'categories': 'assets.view_platform',
         'type_constraints': 'assets.view_platform',

apps/assets/automations/__init__.py

@@ -1,2 +1,2 @@
 from .endpoint import ExecutionManager
-from .methods import platform_automation_methods, filter_platform_methods
+from .methods import platform_automation_methods, filter_platform_methods, sorted_methods

apps/assets/automations/base/manager.py

@@ -12,7 +12,8 @@ from sshtunnel import SSHTunnelForwarder
 
 from assets.automations.methods import platform_automation_methods
 from common.utils import get_logger, lazyproperty, is_openssh_format_key, ssh_pubkey_gen
-from ops.ansible import JMSInventory, PlaybookRunner, DefaultCallback
+from ops.ansible import JMSInventory, DefaultCallback, SuperPlaybookRunner
+from ops.ansible.interface import interface
 
 logger = get_logger(__name__)
 
@@ -54,7 +55,9 @@ class SSHTunnelManager:
                 not_valid.append(k)
             else:
                 local_bind_port = server.local_bind_port
-                host['ansible_host'] = jms_asset['address'] = host['login_host'] = '127.0.0.1'
+
+                host['ansible_host'] = jms_asset['address'] = host[
+                    'login_host'] = interface.get_gateway_proxy_host()
                 host['ansible_port'] = jms_asset['port'] = host['login_port'] = local_bind_port
                 servers.append(server)
 
@@ -269,7 +272,7 @@ class BasePlaybookManager:
                 if not playbook_path:
                     continue
 
-                runer = PlaybookRunner(
+                runer = SuperPlaybookRunner(
                     inventory_path,
                     playbook_path,
                     self.runtime_dir,
@@ -297,12 +300,16 @@ class BasePlaybookManager:
             for host in hosts:
                 result = cb.host_results.get(host)
                 if state == 'ok':
-                    self.on_host_success(host, result)
+                    self.on_host_success(host, result.get('ok', ''))
                 elif state == 'skipped':
                     pass
                 else:
                     error = hosts.get(host)
-                    self.on_host_error(host, error, result)
+                    self.on_host_error(
+                        host, error,
+                        result.get('failures', '')
+                        or result.get('dark', '')
+                    )
 
     def on_runner_failed(self, runner, e):
         print("Runner failed: {} {}".format(e, self))
@@ -314,7 +321,7 @@ class BasePlaybookManager:
     def delete_runtime_dir(self):
         if settings.DEBUG_DEV:
             return
-        shutil.rmtree(self.runtime_dir)
+        shutil.rmtree(self.runtime_dir, ignore_errors=True)
 
     def run(self, *args, **kwargs):
         print(">>> 任务准备阶段\n")
@@ -333,6 +340,7 @@ class BasePlaybookManager:
             ssh_tunnel = SSHTunnelManager()
             ssh_tunnel.local_gateway_prepare(runner)
             try:
+                kwargs.update({"clean_workspace": False})
                 cb = runner.run(**kwargs)
                 self.on_runner_success(runner, cb)
             except Exception as e:

apps/assets/automations/methods.py

@@ -68,6 +68,10 @@ def filter_platform_methods(category, tp_name, method=None, methods=None):
     return methods
 
 
+def sorted_methods(methods):
+    return sorted(methods, key=lambda x: x.get('priority', 10))
+
+
 BASE_DIR = os.path.dirname(os.path.abspath(__file__))
 platform_automation_methods = get_platform_automation_methods(BASE_DIR)
 

apps/assets/automations/ping/custom/rdp/main.yml

@@ -3,6 +3,7 @@
   vars:
     ansible_shell_type: sh
     ansible_connection: local
+    ansible_python_interpreter: /opt/py3/bin/python
 
   tasks:
     - name: Test asset connection (pyfreerdp)

apps/assets/automations/ping/custom/rdp/manifest.yml

@@ -7,6 +7,7 @@ type:
   - windows
 method: ping
 protocol: rdp
+priority: 1
 
 i18n:
   Ping by pyfreerdp:

apps/assets/automations/ping/custom/ssh/main.yml

@@ -19,3 +19,6 @@
         become_user: "{{ custom_become_user | default('') }}"
         become_password: "{{ custom_become_password | default('') }}"
         become_private_key_path: "{{ custom_become_private_key_path | default(None) }}"
+        old_ssh_version: "{{ jms_asset.old_ssh_version | default(False) }}"
+        gateway_args: "{{ jms_asset.ansible_ssh_common_args | default(None) }}"
+

apps/assets/automations/ping/custom/ssh/manifest.yml

@@ -7,6 +7,7 @@ type:
   - all
 method: ping
 protocol: ssh
+priority: 50
 
 i18n:
   Ping by paramiko:

apps/assets/automations/ping/custom/telnet/main.yml

@@ -0,0 +1,11 @@
+- hosts: custom
+  gather_facts: no
+  vars:
+    ansible_connection: local
+    ansible_shell_type: sh
+
+  tasks:
+    - name: Test asset connection (telnet)
+      telnet_ping:
+        login_host: "{{ jms_asset.address }}"
+        login_port: "{{ jms_asset.port }}"

apps/assets/automations/ping/custom/telnet/manifest.yml

@@ -0,0 +1,16 @@
+id: ping_by_telnet
+name: "{{ 'Ping by telnet' | trans }}"
+category:
+  - device
+  - host
+type:
+  - all
+method: ping
+protocol: telnet
+priority: 50
+
+i18n:
+  Ping by telnet:
+    zh: '使用 Python 模块 telnet 测试主机可连接性'
+    en: 'Ping by telnet module'
+    ja: 'Pythonモジュールtelnetを使用したホスト接続性のテスト'

apps/assets/automations/ping/manager.py

@@ -25,14 +25,22 @@ class PingManager(BasePlaybookManager):
 
     def on_host_success(self, host, result):
         asset, account = self.host_asset_and_account_mapper.get(host)
-        asset.set_connectivity(Connectivity.OK)
-        if not account:
-            return
-        account.set_connectivity(Connectivity.OK)
+        try:
+            asset.set_connectivity(Connectivity.OK)
+            if not account:
+                return
+            account.set_connectivity(Connectivity.OK)
+        except Exception as e:
+            print(f'\033[31m Update account {account.name} or '
+                  f'update asset {asset.name} connectivity failed: {e} \033[0m\n')
 
     def on_host_error(self, host, error, result):
         asset, account = self.host_asset_and_account_mapper.get(host)
-        asset.set_connectivity(Connectivity.ERR)
-        if not account:
-            return
-        account.set_connectivity(Connectivity.ERR)
+        try:
+            asset.set_connectivity(Connectivity.ERR)
+            if not account:
+                return
+            account.set_connectivity(Connectivity.ERR)
+        except Exception as e:
+            print(f'\033[31m Update account {account.name} or '
+                  f'update asset {asset.name} connectivity failed: {e} \033[0m\n')

apps/assets/automations/ping_gateway/manager.py

@@ -92,18 +92,26 @@ class PingGatewayManager:
     @staticmethod
     def on_host_success(gateway, account):
         print('\033[32m {} -> {}\033[0m\n'.format(gateway, account))
-        gateway.set_connectivity(Connectivity.OK)
-        if not account:
-            return
-        account.set_connectivity(Connectivity.OK)
+        try:
+            gateway.set_connectivity(Connectivity.OK)
+            if not account:
+                return
+            account.set_connectivity(Connectivity.OK)
+        except Exception as e:
+            print(f'\033[31m Update account {account.name} or '
+                  f'update asset {gateway.name} connectivity failed: {e} \033[0m\n')
 
     @staticmethod
     def on_host_error(gateway, account, error):
         print('\033[31m {} -> {} 原因: {} \033[0m\n'.format(gateway, account, error))
-        gateway.set_connectivity(Connectivity.ERR)
-        if not account:
-            return
-        account.set_connectivity(Connectivity.ERR)
+        try:
+            gateway.set_connectivity(Connectivity.ERR)
+            if not account:
+                return
+            account.set_connectivity(Connectivity.ERR)
+        except Exception as e:
+            print(f'\033[31m Update account {account.name} or '
+                  f'update asset {gateway.name} connectivity failed: {e} \033[0m\n')
 
     @staticmethod
     def before_runner_start():

apps/assets/const/protocol.py

@@ -38,6 +38,14 @@ class Protocol(ChoicesMixin, models.TextChoices):
             cls.ssh: {
                 'port': 22,
                 'secret_types': ['password', 'ssh_key'],
+                'setting': {
+                    'old_ssh_version': {
+                        'type': 'bool',
+                        'default': False,
+                        'label': _('Old SSH version'),
+                        'help_text': _('Old SSH version like openssh 5.x or 6.x')
+                    }
+                }
             },
             cls.sftp: {
                 'port': 22,
@@ -187,6 +195,14 @@ class Protocol(ChoicesMixin, models.TextChoices):
                 'port': 27017,
                 'required': True,
                 'secret_types': ['password'],
+                'setting': {
+                    'auth_source': {
+                        'type': 'str',
+                        'default': 'admin',
+                        'label': _('Auth source'),
+                        'help_text': _('The database to authenticate against')
+                    }
+                }
             },
             cls.redis: {
                 'port': 6379,
@@ -270,7 +286,7 @@ class Protocol(ChoicesMixin, models.TextChoices):
                         'label': _('API mode'),
                         'choices': [
                             ('gpt-3.5-turbo', 'GPT-3.5 Turbo'),
-                            ('gpt-3.5-turbo-16k', 'GPT-3.5 Turbo 16K'),
+                            ('gpt-3.5-turbo-1106', 'GPT-3.5 Turbo 1106'),
                         ]
                     }
                 }
@@ -280,7 +296,8 @@ class Protocol(ChoicesMixin, models.TextChoices):
             choices = protocols[cls.chatgpt]['setting']['api_mode']['choices']
             choices.extend([
                 ('gpt-4', 'GPT-4'),
-                ('gpt-4-32k', 'GPT-4 32K'),
+                ('gpt-4-turbo', 'GPT-4 Turbo'),
+                ('gpt-4o', 'GPT-4o'),
             ])
         return protocols
 

apps/assets/const/types.py

@@ -90,7 +90,7 @@ class AllTypes(ChoicesMixin):
 
     @classmethod
     def set_automation_methods(cls, category, tp_name, constraints):
-        from assets.automations import filter_platform_methods
+        from assets.automations import filter_platform_methods, sorted_methods
         automation = constraints.get('automation', {})
         automation_methods = {}
         platform_automation_methods = cls.get_automation_methods()
@@ -101,6 +101,7 @@ class AllTypes(ChoicesMixin):
             methods = filter_platform_methods(
                 category, tp_name, item_name, methods=platform_automation_methods
             )
+            methods = sorted_methods(methods)
             methods = [{'name': m['name'], 'id': m['id']} for m in methods]
             automation_methods[item_name + '_methods'] = methods
         automation.update(automation_methods)

apps/assets/migrations/0100_auto_20220711_1413.py

@@ -1,6 +1,7 @@
 # Generated by Django 3.2.12 on 2022-07-11 06:13
 
 import time
+import math
 from django.utils import timezone
 from itertools import groupby
 from django.db import migrations
@@ -40,9 +41,13 @@ def migrate_asset_accounts(apps, schema_editor):
             if system_user:
                 # 更新一次系统用户的认证属性
                 account_values.update({attr: getattr(system_user, attr, '') for attr in all_attrs})
-                account_values['created_by'] = str(system_user.id)
                 account_values['privileged'] = system_user.type == 'admin' \
                                                 or system_user.username in ['root', 'Administrator']
+                if system_user.su_enabled and system_user.su_from:
+                    created_by = f'{str(system_user.id)}::{str(system_user.su_from.username)}'
+                else:
+                    created_by = str(system_user.id)
+                account_values['created_by'] = created_by
 
             auth_book_auth = {attr: getattr(auth_book, attr, '') for attr in all_attrs if getattr(auth_book, attr, '')}
             # 最终优先使用 auth_book 的认证属性
@@ -117,6 +122,70 @@ def migrate_asset_accounts(apps, schema_editor):
         print("\t      - histories: {}".format(len(accounts_to_history)))
 
 
+def update_asset_accounts_su_from(apps, schema_editor):
+    # Update accounts su_from
+    print("\n\tStart update asset accounts su_from field")
+    account_model = apps.get_model('accounts', 'Account')
+    platform_model = apps.get_model('assets', 'Platform')
+    asset_model = apps.get_model('assets', 'Asset')
+    platform_ids = list(platform_model.objects.filter(su_enabled=True).values_list('id', flat=True))
+
+    count = 0
+    step_size = 1000
+    count_account = 0
+    while True:
+        start = time.time()
+        asset_ids = asset_model.objects \
+                    .filter(platform_id__in=platform_ids) \
+                    .values_list('id', flat=True)[count:count + step_size]
+        asset_ids = list(asset_ids)
+        if not asset_ids:
+            break
+        count += len(asset_ids)
+
+        accounts = list(account_model.objects.filter(asset_id__in=asset_ids))
+
+        # {asset_id_account_username: account.id}}
+        asset_accounts_mapper = {}
+        for a in accounts:
+            try:
+                k = f'{a.asset_id}_{a.username}'
+                asset_accounts_mapper[k] = str(a.id)
+            except Exception as e:
+                pass
+
+        update_accounts = []
+        for a in accounts:
+            try:
+                if not a.created_by:
+                    continue
+                created_by_list = a.created_by.split('::')
+                if len(created_by_list) != 2:
+                    continue
+                su_from_username = created_by_list[1]
+                if not su_from_username:
+                    continue
+                k = f'{a.asset_id}_{su_from_username}'
+                su_from_id = asset_accounts_mapper.get(k)
+                if not su_from_id:
+                    continue
+                a.su_from_id = su_from_id
+                update_accounts.append(a)
+            except Exception as e:
+                pass
+
+        count_account += len(update_accounts)
+
+        log_msg = "\t      - [{}]: Update accounts su_from: {}-{} {:.2f}s"
+        try:
+            account_model.objects.bulk_update(update_accounts, ['su_from_id'])
+        except Exception as e:
+            status = 'Failed'
+        else:
+            status = 'Success'
+        print(log_msg.format(status, count_account - len(update_accounts), count_account, time.time() - start))
+
+
 def migrate_db_accounts(apps, schema_editor):
     app_perm_model = apps.get_model('perms', 'ApplicationPermission')
     account_model = apps.get_model('accounts', 'Account')
@@ -196,5 +265,6 @@ class Migration(migrations.Migration):
 
     operations = [
         migrations.RunPython(migrate_asset_accounts),
+        migrations.RunPython(update_asset_accounts_su_from),
         migrations.RunPython(migrate_db_accounts),
     ]

apps/assets/migrations/0109_alter_asset_options.py

@@ -12,6 +12,6 @@ class Migration(migrations.Migration):
     operations = [
         migrations.AlterModelOptions(
             name='asset',
-            options={'ordering': ['name'], 'permissions': [('refresh_assethardwareinfo', 'Can refresh asset hardware info'), ('test_assetconnectivity', 'Can test asset connectivity'), ('match_asset', 'Can match asset'), ('change_assetnodes', 'Can change asset nodes')], 'verbose_name': 'Asset'},
+            options={'ordering': [], 'permissions': [('refresh_assethardwareinfo', 'Can refresh asset hardware info'), ('test_assetconnectivity', 'Can test asset connectivity'), ('match_asset', 'Can match asset'), ('change_assetnodes', 'Can change asset nodes')], 'verbose_name': 'Asset'},
         ),
     ]

apps/assets/models/asset/common.py

@@ -348,7 +348,7 @@ class Asset(NodesRelationMixin, LabeledMixin, AbsConnectivity, JSONFilterMixin,
     class Meta:
         unique_together = [('org_id', 'name')]
         verbose_name = _("Asset")
-        ordering = ["name", ]
+        ordering = []
         permissions = [
             ('refresh_assethardwareinfo', _('Can refresh asset hardware info')),
             ('test_assetconnectivity', _('Can test asset connectivity')),

apps/assets/models/gateway.py

@@ -73,3 +73,7 @@ class Gateway(Host):
     def private_key_path(self):
         account = self.select_account
         return account.private_key_path if account else None
+
+    def get_private_key_path(self, path):
+        account = self.select_account
+        return account.get_private_key_path(path) if account else None

apps/assets/models/node.py

@@ -73,6 +73,10 @@ class FamilyMixin:
     @classmethod
     def get_nodes_all_children(cls, nodes, with_self=True):
         pattern = cls.get_nodes_children_key_pattern(nodes, with_self=with_self)
+        if not pattern:
+            # 如果 pattern = ''
+            # key__iregex 报错 (1139, "Got error 'empty (sub)expression' from regexp")
+            return cls.objects.none()
         return Node.objects.filter(key__iregex=pattern)
 
     @classmethod
@@ -429,7 +433,7 @@ class NodeAssetsMixin(NodeAllAssetsMappingMixin):
 
     @classmethod
     @timeit
-    def get_nodes_all_assets(cls, *nodes):
+    def get_nodes_all_assets(cls, *nodes, distinct=True):
         from .asset import Asset
         node_ids = set()
         descendant_node_query = Q()
@@ -439,7 +443,10 @@ class NodeAssetsMixin(NodeAllAssetsMappingMixin):
         if descendant_node_query:
             _ids = Node.objects.order_by().filter(descendant_node_query).values_list('id', flat=True)
             node_ids.update(_ids)
-        return Asset.objects.order_by().filter(nodes__id__in=node_ids).distinct()
+        assets = Asset.objects.order_by().filter(nodes__id__in=node_ids)
+        if distinct:
+            assets = assets.distinct()
+        return assets
 
     def get_all_asset_ids(self):
         asset_ids = self.get_all_asset_ids_by_node_key(org_id=self.org_id, node_key=self.key)

apps/assets/models/platform.py

@@ -1,8 +1,7 @@
 from django.db import models
 from django.utils.translation import gettext_lazy as _
 
-from assets.const import AllTypes
-from assets.const import Protocol
+from assets.const import AllTypes, Category, Protocol
 from common.db.fields import JsonDictTextField
 from common.db.models import JMSBaseModel
 
@@ -119,6 +118,15 @@ class Platform(LabeledMixin, JMSBaseModel):
         )
         return linux.id
 
+    def is_huawei(self):
+        if self.category != Category.DEVICE:
+            return False
+        if 'huawei' in self.name.lower():
+            return True
+        if '华为' in self.name:
+            return True
+        return False
+
     def __str__(self):
         return self.name
 

apps/assets/serializers/asset/info/spec.py

@@ -22,6 +22,36 @@ class WebSpecSerializer(serializers.ModelSerializer):
             'submit_selector', 'script'
         ]
 
+    def get_fields(self):
+        fields = super().get_fields()
+        if self.is_retrieve():
+            # 查看 Web 资产详情时
+            self.pop_fields_if_need(fields)
+        return fields
+    
+    def is_retrieve(self):
+        try:
+            self.context.get('request').method and self.parent.instance.web
+            return True
+        except Exception:
+            return False
+
+    def pop_fields_if_need(self, fields):
+        fields_script = ['script']
+        fields_basic = ['username_selector', 'password_selector', 'submit_selector']
+        autofill = self.parent.instance.web.autofill
+        pop_fields_mapper = {
+            FillType.no: fields_script + fields_basic,
+            FillType.basic: fields_script,
+            FillType.script: fields_basic,
+        }
+        fields_pop = pop_fields_mapper.get(autofill, [])
+        for f in fields_pop:
+            fields.pop(f, None)
+        return fields
+
+
+
 
 category_spec_serializer_map = {
     'database': DatabaseSpecSerializer,

apps/assets/serializers/domain.py

@@ -1,12 +1,13 @@
 # -*- coding: utf-8 -*-
 #
-from django.db.models import Count
+from django.db.models import Count, Q
 from django.utils.translation import gettext_lazy as _
 from rest_framework import serializers
 
 from common.serializers import ResourceLabelsMixin
 from common.serializers.fields import ObjectRelatedField
 from orgs.mixins.serializers import BulkOrgResourceModelSerializer
+from assets.models.gateway import Gateway
 from .gateway import GatewayWithAccountSecretSerializer
 from ..models import Domain
 
@@ -15,7 +16,7 @@ __all__ = ['DomainSerializer', 'DomainWithGatewaySerializer', 'DomainListSeriali
 
 class DomainSerializer(ResourceLabelsMixin, BulkOrgResourceModelSerializer):
     gateways = ObjectRelatedField(
-        many=True, required=False, label=_('Gateway'), read_only=True,
+        many=True, required=False, label=_('Gateway'), queryset=Gateway.objects
     )
 
     class Meta:
@@ -25,6 +26,9 @@ class DomainSerializer(ResourceLabelsMixin, BulkOrgResourceModelSerializer):
         fields_m2m = ['assets', 'gateways']
         read_only_fields = ['date_created']
         fields = fields_small + fields_m2m + read_only_fields
+        extra_kwargs = {
+            'assets': {'required': False},
+        }
 
     def to_representation(self, instance):
         data = super().to_representation(instance)
@@ -35,12 +39,17 @@ class DomainSerializer(ResourceLabelsMixin, BulkOrgResourceModelSerializer):
         data['assets'] = [i for i in assets if str(i['id']) not in gateway_ids]
         return data
 
-    def update(self, instance, validated_data):
+    def create(self, validated_data):
         assets = validated_data.pop('assets', [])
-        assets = assets + list(instance.gateways)
-        validated_data['assets'] = assets
-        instance = super().update(instance, validated_data)
-        return instance
+        gateways = validated_data.pop('gateways', [])
+        validated_data['assets'] = assets + gateways
+        return super().create(validated_data)
+
+    def update(self, instance, validated_data):
+        assets = validated_data.pop('assets', list(instance.assets.all()))
+        gateways = validated_data.pop('gateways', list(instance.gateways.all()))
+        validated_data['assets'] = assets + gateways
+        return super().update(instance, validated_data)
 
     @classmethod
     def setup_eager_loading(cls, queryset):
@@ -58,7 +67,7 @@ class DomainListSerializer(DomainSerializer):
     @classmethod
     def setup_eager_loading(cls, queryset):
         queryset = queryset.annotate(
-            assets_amount=Count('assets'),
+            assets_amount=Count('assets', filter=~Q(assets__platform__name='Gateway'), distinct=True),
         )
         return queryset
 

apps/assets/signal_handlers/asset.py

@@ -63,13 +63,13 @@ def on_asset_create(sender, instance=None, created=False, **kwargs):
         return
     logger.info("Asset create signal recv: {}".format(instance))
 
-    ensure_asset_has_node(assets=(instance,))
+    ensure_asset_has_node.delay(assets=(instance,))
 
     # 获取资产硬件信息
     auto_config = instance.auto_config
     if auto_config.get('ping_enabled'):
         logger.debug('Asset {} ping enabled, test connectivity'.format(instance.name))
-        test_assets_connectivity_handler(assets=(instance,))
+        test_assets_connectivity_handler.delay(assets=(instance,))
     if auto_config.get('gather_facts_enabled'):
         logger.debug('Asset {} gather facts enabled, gather facts'.format(instance.name))
         gather_assets_facts_handler(assets=(instance,))

apps/assets/signal_handlers/node_assets_amount.py

@@ -2,14 +2,16 @@
 #
 from operator import add, sub
 
+from django.conf import settings
 from django.db.models.signals import m2m_changed
 from django.dispatch import receiver
 
 from assets.models import Asset, Node
 from common.const.signals import PRE_CLEAR, POST_ADD, PRE_REMOVE
 from common.decorators import on_transaction_commit, merge_delay_run
+from common.signals import django_ready
 from common.utils import get_logger
-from orgs.utils import tmp_to_org
+from orgs.utils import tmp_to_org, tmp_to_root_org
 from ..tasks import check_node_assets_amount_task
 
 logger = get_logger(__file__)
@@ -34,7 +36,7 @@ def on_node_asset_change(sender, action, instance, reverse, pk_set, **kwargs):
             node_ids = [instance.id]
         else:
             node_ids = list(pk_set)
-        update_nodes_assets_amount(node_ids=node_ids)
+        update_nodes_assets_amount.delay(node_ids=node_ids)
 
 
 @merge_delay_run(ttl=30)
@@ -52,3 +54,18 @@ def update_nodes_assets_amount(node_ids=()):
         node.assets_amount = node.get_assets_amount()
 
     Node.objects.bulk_update(nodes, ['assets_amount'])
+
+
+@receiver(django_ready)
+def set_assets_size_to_setting(sender, **kwargs):
+    from assets.models import Asset
+    try:
+        with tmp_to_root_org():
+            amount = Asset.objects.order_by().count()
+    except:
+        amount = 0
+
+    if amount > 20000:
+        settings.ASSET_SIZE = 'large'
+    elif amount > 5000:
+        settings.ASSET_SIZE = 'medium'

apps/assets/signal_handlers/node_assets_mapping.py

@@ -44,18 +44,18 @@ def on_node_post_create(sender, instance, created, update_fields, **kwargs):
         need_expire = False
 
     if need_expire:
-        expire_node_assets_mapping(org_ids=(instance.org_id,))
+        expire_node_assets_mapping.delay(org_ids=(instance.org_id,))
 
 
 @receiver(post_delete, sender=Node)
 def on_node_post_delete(sender, instance, **kwargs):
-    expire_node_assets_mapping(org_ids=(instance.org_id,))
+    expire_node_assets_mapping.delay(org_ids=(instance.org_id,))
 
 
 @receiver(m2m_changed, sender=Asset.nodes.through)
 def on_node_asset_change(sender, instance, action='pre_remove', **kwargs):
     if action.startswith('post'):
-        expire_node_assets_mapping(org_ids=(instance.org_id,))
+        expire_node_assets_mapping.delay(org_ids=(instance.org_id,))
 
 
 @receiver(django_ready)

apps/assets/utils/k8s.py

@@ -88,8 +88,7 @@ class KubernetesClient:
             try:
                 data = getattr(self, func_name)(*args)
             except Exception as e:
-                logger.error(e)
-                raise e
+                logger.error(f'K8S tree get {tp} error: {e}')
 
         if self.server:
             self.server.stop()

apps/audits/api.py

@@ -1,6 +1,5 @@
 # -*- coding: utf-8 -*-
 #
-
 from importlib import import_module
 
 from django.conf import settings
@@ -20,6 +19,7 @@ from common.const.http import GET, POST
 from common.drf.filters import DatetimeRangeFilterBackend
 from common.permissions import IsServiceAccount
 from common.plugins.es import QuerySet as ESQuerySet
+from common.sessions.cache import user_session_manager
 from common.storage.ftp_file import FTPFileStorageHandler
 from common.utils import is_uuid, get_logger, lazyproperty
 from orgs.mixins.api import OrgReadonlyModelViewSet, OrgModelViewSet
@@ -30,7 +30,7 @@ from terminal.models import default_storage
 from users.models import User
 from .backends import TYPE_ENGINE_MAPPING
 from .const import ActivityChoices
-from .filters import UserSessionFilterSet
+from .filters import UserSessionFilterSet, OperateLogFilterSet
 from .models import (
     FTPLog, UserLoginLog, OperateLog, PasswordChangeLog,
     ActivityLog, JobLog, UserSession
@@ -65,7 +65,7 @@ class FTPLogViewSet(OrgModelViewSet):
     date_range_filter_fields = [
         ('date_start', ('date_from', 'date_to'))
     ]
-    filterset_fields = ['user', 'asset', 'account', 'filename']
+    filterset_fields = ['user', 'asset', 'account', 'filename', 'session']
     search_fields = filterset_fields
     ordering = ['-date_start']
     http_method_names = ['post', 'get', 'head', 'options', 'patch']
@@ -204,10 +204,7 @@ class OperateLogViewSet(OrgReadonlyModelViewSet):
     date_range_filter_fields = [
         ('datetime', ('date_from', 'date_to'))
     ]
-    filterset_fields = [
-        'user', 'action', 'resource_type', 'resource',
-        'remote_addr'
-    ]
+    filterset_class = OperateLogFilterSet
     search_fields = ['resource', 'user']
     ordering = ['-datetime']
 
@@ -271,7 +268,7 @@ class UserSessionViewSet(CommonApiMixin, viewsets.ModelViewSet):
         return user_ids
 
     def get_queryset(self):
-        keys = UserSession.get_keys()
+        keys = user_session_manager.get_keys()
         queryset = UserSession.objects.filter(key__in=keys)
         if current_org.is_root():
             return queryset
@@ -289,8 +286,7 @@ class UserSessionViewSet(CommonApiMixin, viewsets.ModelViewSet):
             return Response(status=status.HTTP_200_OK)
 
         keys = queryset.values_list('key', flat=True)
-        session_store_cls = import_module(settings.SESSION_ENGINE).SessionStore
         for key in keys:
-            session_store_cls(key).delete()
+            user_session_manager.remove(key)
         queryset.delete()
         return Response(status=status.HTTP_200_OK)

apps/audits/filters.py

@@ -1,12 +1,13 @@
-from django.core.cache import cache
+from django.apps import apps
+from django.utils import translation
+
 from django_filters import rest_framework as drf_filters
 from rest_framework import filters
 from rest_framework.compat import coreapi, coreschema
-
 from common.drf.filters import BaseFilterSet
-from notifications.ws import WS_SESSION_KEY
+from common.sessions.cache import user_session_manager
 from orgs.utils import current_org
-from .models import UserSession
+from .models import UserSession, OperateLog
 
 __all__ = ['CurrentOrgMembersFilter']
 
@@ -41,15 +42,32 @@ class UserSessionFilterSet(BaseFilterSet):
 
     @staticmethod
     def filter_is_active(queryset, name, is_active):
-        redis_client = cache.client.get_client()
-        members = redis_client.smembers(WS_SESSION_KEY)
-        members = [member.decode('utf-8') for member in members]
+        keys = user_session_manager.get_active_keys()
         if is_active:
-            queryset = queryset.filter(key__in=members)
+            queryset = queryset.filter(key__in=keys)
         else:
-            queryset = queryset.exclude(key__in=members)
+            queryset = queryset.exclude(key__in=keys)
         return queryset
 
     class Meta:
         model = UserSession
         fields = ['id', 'ip', 'city', 'type']
+
+
+class OperateLogFilterSet(BaseFilterSet):
+    resource_type = drf_filters.CharFilter(method='filter_resource_type')
+
+    @staticmethod
+    def filter_resource_type(queryset, name, resource_type):
+        current_lang = translation.get_language()
+        with translation.override(current_lang):
+            mapper = {str(m._meta.verbose_name): m._meta.verbose_name_raw for m in apps.get_models()}
+        tp = mapper.get(resource_type)
+        queryset = queryset.filter(resource_type=tp)
+        return queryset
+
+    class Meta:
+        model = OperateLog
+        fields = [
+            'user', 'action', 'resource', 'remote_addr'
+        ]

apps/audits/handler.py

@@ -12,7 +12,10 @@ from common.utils.timezone import as_current_tz
 from jumpserver.utils import current_request
 from orgs.models import Organization
 from orgs.utils import get_current_org_id
+from settings.models import Setting
 from settings.serializers import SettingsSerializer
+from users.models import Preference
+from users.serializers import PreferenceSerializer
 from .backends import get_operate_log_storage
 
 logger = get_logger(__name__)
@@ -87,19 +90,15 @@ class OperatorLogHandler(metaclass=Singleton):
         return log_id, before, after
 
     @staticmethod
-    def get_resource_display_from_setting(resource):
-        resource_display = None
-        setting_serializer = SettingsSerializer()
-        label = setting_serializer.get_field_label(resource)
-        if label is not None:
-            resource_display = label
-        return resource_display
-
-    def get_resource_display(self, resource):
-        resource_display = str(resource)
-        return_value = self.get_resource_display_from_setting(resource_display)
-        if return_value is not None:
-            resource_display = return_value
+    def get_resource_display(resource):
+        if isinstance(resource, Setting):
+            serializer = SettingsSerializer()
+            resource_display = serializer.get_field_label(resource.name)
+        elif isinstance(resource, Preference):
+            serializer = PreferenceSerializer()
+            resource_display = serializer.get_field_label(resource.name)
+        else:
+            resource_display = str(resource)
         return resource_display
 
     @staticmethod

apps/audits/models.py

@@ -4,15 +4,15 @@ from datetime import timedelta
 from importlib import import_module
 
 from django.conf import settings
-from django.core.cache import caches, cache
+from django.core.cache import caches
 from django.db import models
 from django.db.models import Q
 from django.utils import timezone
 from django.utils.translation import gettext, gettext_lazy as _
 
 from common.db.encoder import ModelJSONFieldEncoder
+from common.sessions.cache import user_session_manager
 from common.utils import lazyproperty, i18n_trans
-from notifications.ws import WS_SESSION_KEY
 from ops.models import JobExecution
 from orgs.mixins.models import OrgModelMixin, Organization
 from orgs.utils import current_org
@@ -278,8 +278,7 @@ class UserSession(models.Model):
 
     @property
     def is_active(self):
-        redis_client = cache.client.get_client()
-        return redis_client.sismember(WS_SESSION_KEY, self.key)
+        return user_session_manager.check_active(self.key)
 
     @property
     def date_expired(self):
@@ -289,16 +288,9 @@ class UserSession(models.Model):
         ttl = caches[settings.SESSION_CACHE_ALIAS].ttl(cache_key)
         return timezone.now() + timedelta(seconds=ttl)
 
-    @staticmethod
-    def get_keys():
-        session_store_cls = import_module(settings.SESSION_ENGINE).SessionStore
-        cache_key_prefix = session_store_cls.cache_key_prefix
-        keys = caches[settings.SESSION_CACHE_ALIAS].iter_keys('*')
-        return [k.replace(cache_key_prefix, '') for k in keys]
-
     @classmethod
     def clear_expired_sessions(cls):
-        keys = cls.get_keys()
+        keys = user_session_manager.get_keys()
         cls.objects.exclude(key__in=keys).delete()
 
     class Meta:

apps/audits/serializers.py

@@ -23,7 +23,7 @@ class JobLogSerializer(JobExecutionSerializer):
     class Meta:
         model = models.JobLog
         read_only_fields = [
-            "id", "material", "time_cost", 'date_start',
+            "id", "material", 'job_type', "time_cost", 'date_start',
             'date_finished', 'date_created',
             'is_finished', 'is_success',
             'task_id', 'creator_name'
@@ -43,7 +43,7 @@ class FTPLogSerializer(serializers.ModelSerializer):
         fields_small = fields_mini + [
             "user", "remote_addr", "asset", "account",
             "org_id", "operate", "filename", "date_start",
-            "is_success", "has_file",
+            "is_success", "has_file", "session"
         ]
         fields = fields_small
 

apps/audits/signal_handlers/login_log.py

@@ -36,6 +36,7 @@ class AuthBackendLabelMapping(LazyObject):
         backend_label_mapping[settings.AUTH_BACKEND_AUTH_TOKEN] = _("Auth Token")
         backend_label_mapping[settings.AUTH_BACKEND_WECOM] = _("WeCom")
         backend_label_mapping[settings.AUTH_BACKEND_FEISHU] = _("FeiShu")
+        backend_label_mapping[settings.AUTH_BACKEND_LARK] = 'Lark'
         backend_label_mapping[settings.AUTH_BACKEND_SLACK] = _("Slack")
         backend_label_mapping[settings.AUTH_BACKEND_DINGTALK] = _("DingTalk")
         backend_label_mapping[settings.AUTH_BACKEND_TEMP_TOKEN] = _("Temporary token")

apps/audits/signal_handlers/operate_log.py

@@ -178,7 +178,7 @@ def on_django_start_set_operate_log_monitor_models(sender, **kwargs):
         'PermedAsset', 'PermedAccount', 'MenuPermission',
         'Permission', 'TicketSession', 'ApplyLoginTicket',
         'ApplyCommandTicket', 'ApplyLoginAssetTicket',
-        'FavoriteAsset', 'Asset'
+        'FavoriteAsset',
     }
     for i, app in enumerate(apps.get_models(), 1):
         app_name = app._meta.app_label

apps/audits/tasks.py

@@ -7,19 +7,18 @@ import subprocess
 from celery import shared_task
 from django.conf import settings
 from django.core.files.storage import default_storage
+from django.db import transaction
 from django.utils import timezone
 from django.utils.translation import gettext_lazy as _
 
 from common.const.crontab import CRONTAB_AT_AM_TWO
-from common.utils import get_log_keep_day, get_logger
 from common.storage.ftp_file import FTPFileStorageHandler
-from ops.celery.decorator import (
-    register_as_period_task, after_app_shutdown_clean_periodic
-)
+from common.utils import get_log_keep_day, get_logger
+from ops.celery.decorator import register_as_period_task
 from ops.models import CeleryTaskExecution
-from terminal.models import Session, Command
 from terminal.backends import server_replay_storage
-from .models import UserLoginLog, OperateLog, FTPLog, ActivityLog
+from terminal.models import Session, Command
+from .models import UserLoginLog, OperateLog, FTPLog, ActivityLog, PasswordChangeLog
 
 logger = get_logger(__name__)
 
@@ -38,6 +37,14 @@ def clean_operation_log_period():
     OperateLog.objects.filter(datetime__lt=expired_day).delete()
 
 
+def clean_password_change_log_period():
+    now = timezone.now()
+    days = get_log_keep_day('PASSWORD_CHANGE_LOG_KEEP_DAYS')
+    expired_day = now - datetime.timedelta(days=days)
+    PasswordChangeLog.objects.filter(datetime__lt=expired_day).delete()
+    logger.info("Clean password change log done")
+
+
 def clean_activity_log_period():
     now = timezone.now()
     days = get_log_keep_day('ACTIVITY_LOG_KEEP_DAYS')
@@ -49,9 +56,9 @@ def clean_ftp_log_period():
     now = timezone.now()
     days = get_log_keep_day('FTP_LOG_KEEP_DAYS')
     expired_day = now - datetime.timedelta(days=days)
-    file_store_dir = os.path.join(default_storage.base_location, 'ftp_file')
+    file_store_dir = os.path.join(default_storage.base_location, FTPLog.upload_to)
     FTPLog.objects.filter(date_start__lt=expired_day).delete()
-    command = "find %s -mtime +%s -exec rm -f {} \\;" % (
+    command = "find %s -mtime +%s -type f -exec rm -f {} \\;" % (
         file_store_dir, days
     )
     subprocess.call(command, shell=True)
@@ -76,6 +83,15 @@ def clean_celery_tasks_period():
     subprocess.call(command, shell=True)
 
 
+def batch_delete(queryset, batch_size=3000):
+    model = queryset.model
+    count = queryset.count()
+    with transaction.atomic():
+        for i in range(0, count, batch_size):
+            pks = queryset[i:i + batch_size].values_list('id', flat=True)
+            model.objects.filter(id__in=list(pks)).delete()
+
+
 def clean_expired_session_period():
     logger.info("Start clean expired session record, commands and replay")
     days = get_log_keep_day('TERMINAL_SESSION_KEEP_DURATION')
@@ -85,9 +101,9 @@ def clean_expired_session_period():
     expired_commands = Command.objects.filter(timestamp__lt=timestamp)
     replay_dir = os.path.join(default_storage.base_location, 'replay')
 
-    expired_sessions.delete()
+    batch_delete(expired_sessions)
     logger.info("Clean session item done")
-    expired_commands.delete()
+    batch_delete(expired_commands)
     logger.info("Clean session command done")
     command = "find %s -mtime +%s \\( -name '*.json' -o -name '*.tar' -o -name '*.gz' \\) -exec rm -f {} \\;" % (
         replay_dir, days
@@ -100,7 +116,6 @@ def clean_expired_session_period():
 
 @shared_task(verbose_name=_('Clean audits session task log'))
 @register_as_period_task(crontab=CRONTAB_AT_AM_TWO)
-@after_app_shutdown_clean_periodic
 def clean_audits_log_period():
     print("Start clean audit session task log")
     clean_login_log_period()
@@ -109,6 +124,7 @@ def clean_audits_log_period():
     clean_activity_log_period()
     clean_celery_tasks_period()
     clean_expired_session_period()
+    clean_password_change_log_period()
 
 
 @shared_task(verbose_name=_('Upload FTP file to external storage'))

apps/authentication/api/__init__.py

@@ -2,13 +2,15 @@
 #
 
 from .access_key import *
+from .common import *
 from .confirm import *
 from .connection_token import *
 from .feishu import *
+from .lark import *
 from .login_confirm import *
 from .mfa import *
 from .password import *
+from .session import *
 from .sso import *
 from .temp_token import *
 from .token import *
-from .common import *

apps/authentication/api/common.py

@@ -12,7 +12,6 @@ from common.permissions import IsValidUser, OnlySuperUser
 from common.utils import get_logger
 from users.models import User
 
-
 logger = get_logger(__file__)
 
 
@@ -24,6 +23,7 @@ class QRUnBindBase(APIView):
             'wecom': {'user_field': 'wecom_id', 'not_bind_err': errors.WeComNotBound},
             'dingtalk': {'user_field': 'dingtalk_id', 'not_bind_err': errors.DingTalkNotBound},
             'feishu': {'user_field': 'feishu_id', 'not_bind_err': errors.FeiShuNotBound},
+            'lark': {'user_field': 'lark_id', 'not_bind_err': errors.LarkNotBound},
             'slack': {'user_field': 'slack_id', 'not_bind_err': errors.SlackNotBound},
         }
         user = self.user

apps/authentication/api/connection_token.py

@@ -205,7 +205,7 @@ class RDPFileClientProtocolURLMixin:
         return data
 
     def get_smart_endpoint(self, protocol, asset=None):
-        endpoint = Endpoint.match_by_instance_label(asset, protocol)
+        endpoint = Endpoint.match_by_instance_label(asset, protocol, self.request)
         if not endpoint:
             target_ip = asset.get_target_ip() if asset else ''
             endpoint = EndpointRule.match_endpoint(
@@ -223,12 +223,17 @@ class ExtraActionApiMixin(RDPFileClientProtocolURLMixin):
     validate_exchange_token: callable
 
     @action(methods=['POST', 'GET'], detail=True, url_path='rdp-file')
-    def get_rdp_file(self, *args, **kwargs):
+    def get_rdp_file(self, request, *args, **kwargs):
         token = self.get_object()
         token.is_valid()
         filename, content = self.get_rdp_file_info(token)
-        filename = '{}.rdp'.format(filename)
         response = HttpResponse(content, content_type='application/octet-stream')
+
+        if is_true(request.query_params.get('reusable')):
+            token.set_reusable(True)
+            filename = '{}-{}'.format(filename, token.date_expired.strftime('%Y%m%d_%H%M%S'))
+
+        filename += '.rdp'
         response['Content-Disposition'] = 'attachment; filename*=UTF-8\'\'%s' % filename
         return response
 
@@ -379,6 +384,7 @@ class ConnectionTokenViewSet(ExtraActionApiMixin, RootOrgViewMixin, JMSModelView
 
         if account.username != AliasAccount.INPUT:
             data['input_username'] = ''
+
         ticket = self._validate_acl(user, asset, account)
         if ticket:
             data['from_ticket'] = ticket
@@ -413,7 +419,10 @@ class ConnectionTokenViewSet(ExtraActionApiMixin, RootOrgViewMixin, JMSModelView
 
     def _validate_acl(self, user, asset, account):
         from acls.models import LoginAssetACL
-        acls = LoginAssetACL.filter_queryset(user=user, asset=asset, account=account)
+        kwargs = {'user': user, 'asset': asset, 'account': account}
+        if account.username == AliasAccount.INPUT:
+            kwargs['account_username'] = self.input_username
+        acls = LoginAssetACL.filter_queryset(**kwargs)
         ip = get_request_ip_or_data(self.request)
         acl = LoginAssetACL.get_match_rule_acls(user, ip, acls)
         if not acl:
@@ -443,7 +452,7 @@ class ConnectionTokenViewSet(ExtraActionApiMixin, RootOrgViewMixin, JMSModelView
             self._record_operate_log(acl, asset)
             for reviewer in reviewers:
                 AssetLoginReminderMsg(
-                    reviewer, asset, user, self.input_username
+                    reviewer, asset, user, account, self.input_username
                 ).publish_async()
 
     def create(self, request, *args, **kwargs):
@@ -503,20 +512,16 @@ class SuperConnectionTokenViewSet(ConnectionTokenViewSet):
         token.is_valid()
         serializer = self.get_serializer(instance=token)
 
-        expire_now = request.data.get('expire_now', None)
+        expire_now = request.data.get('expire_now', True)
         asset_type = token.asset.type
         # 设置默认值
-        if expire_now is None:
-            # TODO 暂时特殊处理 k8s 不过期
-            if asset_type in ['k8s', 'kubernetes']:
-                expire_now = False
-            else:
-                expire_now = not settings.CONNECTION_TOKEN_REUSABLE
+        if asset_type in ['k8s', 'kubernetes']:
+            expire_now = False
 
-        if is_false(expire_now):
-            logger.debug('Api specified, now expire now')
-        elif token.is_reusable and settings.CONNECTION_TOKEN_REUSABLE:
+        if token.is_reusable and settings.CONNECTION_TOKEN_REUSABLE:
             logger.debug('Token is reusable, not expire now')
+        elif is_false(expire_now):
+            logger.debug('Api specified, now expire now')
         else:
             token.expire()
 

apps/authentication/api/lark.py

@@ -0,0 +1,8 @@
+from common.utils import get_logger
+from .feishu import FeiShuEventSubscriptionCallback
+
+logger = get_logger(__name__)
+
+
+class LarkEventSubscriptionCallback(FeiShuEventSubscriptionCallback):
+    pass

apps/authentication/api/login_confirm.py

@@ -9,6 +9,7 @@ from common.utils import get_logger
 from .. import errors, mixins
 
 __all__ = ['TicketStatusApi']
+
 logger = get_logger(__name__)
 
 

apps/authentication/api/mfa.py

@@ -9,7 +9,7 @@ from rest_framework.permissions import AllowAny
 from rest_framework.response import Response
 from rest_framework.serializers import ValidationError
 
-from common.exceptions import UnexpectError
+from common.exceptions import JMSException, UnexpectError
 from common.utils import get_logger
 from users.models.user import User
 from .. import errors
@@ -50,7 +50,10 @@ class MFASendCodeApi(AuthMixin, CreateAPIView):
         mfa_type = serializer.validated_data['type']
 
         if not username:
-            user = self.get_user_from_session()
+            try:
+                user = self.get_user_from_session()
+            except errors.SessionEmptyError as e:
+                raise ValidationError({'error': e})
         else:
             user = self.get_user_from_db(username)
 
@@ -61,6 +64,8 @@ class MFASendCodeApi(AuthMixin, CreateAPIView):
 
         try:
             mfa_backend.send_challenge()
+        except JMSException:
+            raise
         except Exception as e:
             raise UnexpectError(str(e))
 

apps/authentication/api/session.py

@@ -0,0 +1,68 @@
+import time
+from threading import Thread
+
+from django.conf import settings
+from django.contrib.auth import logout
+from django.contrib.auth.models import AnonymousUser
+from rest_framework import generics
+from rest_framework import status
+from rest_framework.response import Response
+
+from common.sessions.cache import user_session_manager
+from common.utils import get_logger
+
+__all__ = ['UserSessionApi']
+
+logger = get_logger(__name__)
+
+
+class UserSessionManager:
+
+    def __init__(self, request):
+        self.request = request
+        self.session = request.session
+
+    def connect(self):
+        user_session_manager.add_or_increment(self.session.session_key)
+
+    def disconnect(self):
+        user_session_manager.decrement(self.session.session_key)
+        if self.should_delete_session():
+            thread = Thread(target=self.delay_delete_session)
+            thread.start()
+
+    def should_delete_session(self):
+        return (self.session.modified or settings.SESSION_SAVE_EVERY_REQUEST) and \
+            not self.session.is_empty() and \
+            self.session.get_expire_at_browser_close() and \
+            not user_session_manager.check_active(self.session.session_key)
+
+    def delay_delete_session(self):
+        timeout = 6
+        check_interval = 0.5
+
+        start_time = time.time()
+        while time.time() - start_time < timeout:
+            time.sleep(check_interval)
+            if user_session_manager.check_active(self.session.session_key):
+                return
+
+        logout(self.request)
+
+
+class UserSessionApi(generics.RetrieveDestroyAPIView):
+    permission_classes = ()
+
+    def retrieve(self, request, *args, **kwargs):
+        if isinstance(request.user, AnonymousUser):
+            return Response(status=status.HTTP_200_OK)
+
+        UserSessionManager(request).connect()
+        return Response(status=status.HTTP_200_OK)
+
+    def destroy(self, request, *args, **kwargs):
+        if isinstance(request.user, AnonymousUser):
+            return Response(status=status.HTTP_200_OK)
+
+        UserSessionManager(request).disconnect()
+        return Response(status=status.HTTP_204_NO_CONTENT)

apps/authentication/api/sso.py

@@ -5,11 +5,13 @@ from django.conf import settings
 from django.contrib.auth import login
 from django.http.response import HttpResponseRedirect
 from rest_framework import serializers
+from rest_framework import status
 from rest_framework.decorators import action
 from rest_framework.permissions import AllowAny
 from rest_framework.request import Request
 from rest_framework.response import Response
 
+from authentication.errors import ACLError
 from common.api import JMSGenericViewSet
 from common.const.http import POST, GET
 from common.permissions import OnlySuperUser
@@ -17,7 +19,10 @@ from common.serializers import EmptySerializer
 from common.utils import reverse, safe_next_url
 from common.utils.timezone import utc_now
 from users.models import User
-from ..errors import SSOAuthClosed
+from users.utils import LoginBlockUtil, LoginIpBlockUtil
+from ..errors import (
+    SSOAuthClosed, AuthFailedError, LoginConfirmBaseError, SSOAuthKeyTTLError
+)
 from ..filters import AuthKeyQueryDeclaration
 from ..mixins import AuthMixin
 from ..models import SSOToken
@@ -63,31 +68,58 @@ class SSOViewSet(AuthMixin, JMSGenericViewSet):
         此接口违反了 `Restful` 的规范
         `GET` 应该是安全的方法，但此接口是不安全的
         """
+        status_code = status.HTTP_400_BAD_REQUEST
         request.META['HTTP_X_JMS_LOGIN_TYPE'] = 'W'
         authkey = request.query_params.get(AUTH_KEY)
         next_url = request.query_params.get(NEXT_URL)
         if not next_url or not next_url.startswith('/'):
             next_url = reverse('index')
 
-        if not authkey:
-            raise serializers.ValidationError("authkey is required")
-
         try:
+            if not authkey:
+                raise serializers.ValidationError("authkey is required")
+
             authkey = UUID(authkey)
             token = SSOToken.objects.get(authkey=authkey, expired=False)
-            # 先过期，只能访问这一次
+        except (ValueError, SSOToken.DoesNotExist, serializers.ValidationError) as e:
+            error_msg = str(e)
+            self.send_auth_signal(success=False, reason=error_msg)
+            return Response({'error': error_msg}, status=status_code)
+
+        error_msg = None
+        user = token.user
+        username = user.username
+        ip = self.get_request_ip()
+
+        try:
+            if (utc_now().timestamp() - token.date_created.timestamp()) > settings.AUTH_SSO_AUTHKEY_TTL:
+                raise SSOAuthKeyTTLError()
+
+            self._check_is_block(username, True)
+            self._check_only_allow_exists_user_auth(username)
+            self._check_login_acl(user, ip)
+            self.check_user_login_confirm_if_need(user)
+
+            self.request.session['auth_backend'] = settings.AUTH_BACKEND_SSO
+            login(self.request, user, settings.AUTH_BACKEND_SSO)
+            self.send_auth_signal(success=True, user=user)
+            self.mark_mfa_ok('otp', user)
+
+            LoginIpBlockUtil(ip).clean_block_if_need()
+            LoginBlockUtil(username, ip).clean_failed_count()
+            self.clear_auth_mark()
+        except (ACLError, LoginConfirmBaseError):  # 无需记录日志
+            pass
+        except (AuthFailedError, SSOAuthKeyTTLError) as e:
+            error_msg = e.msg
+        except Exception as e:
+            error_msg = str(e)
+        finally:
             token.expired = True
             token.save()
-        except (ValueError, SSOToken.DoesNotExist):
-            self.send_auth_signal(success=False, reason='authkey_invalid')
-            return HttpResponseRedirect(next_url)
 
-        # 判断是否过期
-        if (utc_now().timestamp() - token.date_created.timestamp()) > settings.AUTH_SSO_AUTHKEY_TTL:
-            self.send_auth_signal(success=False, reason='authkey_timeout')
+        if error_msg:
+            self.send_auth_signal(success=False, username=username, reason=error_msg)
+            return Response({'error': error_msg}, status=status_code)
+        else:
             return HttpResponseRedirect(next_url)
-
-        user = token.user
-        login(self.request, user, settings.AUTH_BACKEND_SSO)
-        self.send_auth_signal(success=True, user=user)
-        return HttpResponseRedirect(next_url)

apps/authentication/backends/drf.py

@@ -10,6 +10,7 @@ from rest_framework import authentication, exceptions
 from common.auth import signature
 from common.decorators import merge_delay_run
 from common.utils import get_object_or_none, get_request_ip_or_data, contains_ip
+from users.models import User
 from ..models import AccessKey, PrivateToken
 
 
@@ -19,22 +20,23 @@ def date_more_than(d, seconds):
 
 @merge_delay_run(ttl=60)
 def update_token_last_used(tokens=()):
-    for token in tokens:
-        token.date_last_used = timezone.now()
-        token.save(update_fields=['date_last_used'])
+    access_keys_ids = [token.id for token in tokens if isinstance(token, AccessKey)]
+    private_token_keys = [token.key for token in tokens if isinstance(token, PrivateToken)]
+    if len(access_keys_ids) > 0:
+        AccessKey.objects.filter(id__in=access_keys_ids).update(date_last_used=timezone.now())
+    if len(private_token_keys) > 0:
+        PrivateToken.objects.filter(key__in=private_token_keys).update(date_last_used=timezone.now())
 
 
 @merge_delay_run(ttl=60)
 def update_user_last_used(users=()):
-    for user in users:
-        user.date_api_key_last_used = timezone.now()
-        user.save(update_fields=['date_api_key_last_used'])
+    User.objects.filter(id__in=users).update(date_api_key_last_used=timezone.now())
 
 
 def after_authenticate_update_date(user, token=None):
-    update_user_last_used(users=(user,))
+    update_user_last_used.delay(users=(user.id,))
     if token:
-        update_token_last_used(tokens=(token,))
+        update_token_last_used.delay(tokens=(token,))
 
 
 class AccessTokenAuthentication(authentication.BaseAuthentication):

apps/authentication/backends/oauth2/backends.py

@@ -98,16 +98,19 @@ class OAuth2Backend(JMSModelBackend):
         access_token_url = '{url}{separator}{query}'.format(
             url=settings.AUTH_OAUTH2_ACCESS_TOKEN_ENDPOINT, separator=separator, query=urlencode(query_dict)
         )
+        # token_method -> get, post(post_data), post_json
         token_method = settings.AUTH_OAUTH2_ACCESS_TOKEN_METHOD.lower()
-        requests_func = getattr(requests, token_method, requests.get)
         logger.debug(log_prompt.format('Call the access token endpoint[method: %s]' % token_method))
         headers = {
             'Accept': 'application/json'
         }
-        if token_method == 'post':
-            access_token_response = requests_func(access_token_url, headers=headers, data=query_dict)
+        if token_method.startswith('post'):
+            body_key = 'json' if token_method.endswith('json') else 'data'
+            access_token_response = requests.post(
+                access_token_url, headers=headers, **{body_key: query_dict}
+            )
         else:
-            access_token_response = requests_func(access_token_url, headers=headers)
+            access_token_response = requests.get(access_token_url, headers=headers)
         try:
             access_token_response.raise_for_status()
             access_token_response_data = access_token_response.json()