Merge branch 'master' into v2.0

[Update] assets/gathered_user 添加过滤字段

.github/ISSUE_TEMPLATE.md

@@ -0,0 +1,17 @@
+[简述你的问题]
+
+
+##### 使用版本
+[请提供你使用的JumpServer版本 如 2.0.1 注: 1.4及以下版本不再提供支持]
+
+##### 问题复现步骤
+1. [步骤1]
+2. [步骤2]
+
+##### 具体表现[截图可能会更好些,最好能截全]
+
+
+##### 其他
+
+
+[注:] 完成后请关闭 issue

.github/ISSUE_TEMPLATE/----.md

@@ -1,10 +0,0 @@
----
-name: 需求建议
-about: 提出针对本项目的想法和建议
-title: "[Feature] "
-labels: 类型:需求
-assignees: ibuler
-
----
-
-**请描述您的需求或者改进建议.**

.github/ISSUE_TEMPLATE/bug---.md

@@ -1,22 +0,0 @@
----
-name: Bug 提交
-about: 提交产品缺陷帮助我们更好的改进
-title: "[Bug] "
-labels: 类型:bug
-assignees: wojiushixiaobai
-
----
-
-**JumpServer 版本(v1.5.9以下不再支持)**
-
-
-**浏览器版本**
-
-
-**Bug 描述**
-
-
-**Bug 重现步骤(有截图更好)**
-1.  
-2. 
-3. 

.github/ISSUE_TEMPLATE/question.md

@@ -1,10 +0,0 @@
----
-name: 问题咨询
-about: 提出针对本项目安装部署、使用及其他方面的相关问题
-title: "[Question] "
-labels: 类型:提问
-assignees: wojiushixiaobai
-
----
-
-**请描述您的问题.**

.github/workflows/jms-generic-action-handler.yml

@@ -1,12 +0,0 @@
-on: [push, pull_request, release]
-
-name: JumpServer repos generic handler
-
-jobs:
-  generic_handler:
-    name: Run generic handler
-    runs-on: ubuntu-latest
-    steps:
-      - uses: jumpserver/action-generic-handler@master
-        env:
-          GITHUB_TOKEN: ${{ secrets.PRIVATE_TOKEN }}

Dockerfile

@@ -1,46 +1,25 @@
-# 编译代码
-FROM python:3.8.6-slim as stage-build
-MAINTAINER JumpServer Team <ibuler@qq.com>
-ARG VERSION
-ENV VERSION=$VERSION
+FROM registry.fit2cloud.com/public/python:v3
+MAINTAINER Jumpserver Team <ibuler@qq.com>
 
 WORKDIR /opt/jumpserver
-ADD . .
-RUN cd utils && bash -ixeu build.sh
+RUN useradd jumpserver
 
+COPY ./requirements /tmp/requirements
 
-# 构建运行时环境
-FROM python:3.8.6-slim
-ARG PIP_MIRROR=https://pypi.douban.com/simple
-ENV PIP_MIRROR=$PIP_MIRROR
-ARG PIP_JMS_MIRROR=https://pypi.douban.com/simple
-ENV PIP_JMS_MIRROR=$PIP_JMS_MIRROR
-
-WORKDIR /opt/jumpserver
-
-COPY ./requirements/deb_buster_requirements.txt ./requirements/deb_buster_requirements.txt
-RUN sed -i 's/deb.debian.org/mirrors.aliyun.com/g' /etc/apt/sources.list \
-    && sed -i 's/security.debian.org/mirrors.aliyun.com/g' /etc/apt/sources.list \
-    && apt update \
-    && grep -v '^#' ./requirements/deb_buster_requirements.txt | xargs apt -y install \
-    && localedef -c -f UTF-8 -i zh_CN zh_CN.UTF-8 \
-    && cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime
-
-COPY ./requirements/requirements.txt ./requirements/requirements.txt
-RUN pip install --upgrade pip==20.2.4 setuptools==49.6.0 wheel==0.34.2 -i ${PIP_MIRROR} \
-    && pip config set global.index-url ${PIP_MIRROR} \
-    && pip install --no-cache-dir $(grep 'jms' requirements/requirements.txt) -i ${PIP_JMS_MIRROR} \
-    && pip install --no-cache-dir -r requirements/requirements.txt
-
-COPY --from=stage-build /opt/jumpserver/release/jumpserver /opt/jumpserver
-RUN mkdir -p /root/.ssh/ \
-    && echo "Host *\n\tStrictHostKeyChecking no\n\tUserKnownHostsFile /dev/null" > /root/.ssh/config
+RUN yum -y install epel-release && \
+      echo -e "[mysql]\nname=mysql\nbaseurl=https://mirrors.tuna.tsinghua.edu.cn/mysql/yum/mysql57-community-el6/\ngpgcheck=0\nenabled=1" > /etc/yum.repos.d/mysql.repo
+RUN cd /tmp/requirements && yum -y install $(cat rpm_requirements.txt)
+RUN cd /tmp/requirements && pip install --upgrade pip setuptools && pip install wheel && \
+    pip install -i https://pypi.tuna.tsinghua.edu.cn/simple -r requirements.txt || pip install -r requirements.txt
+RUN mkdir -p /root/.ssh/ && echo -e "Host *\n\tStrictHostKeyChecking no\n\tUserKnownHostsFile /dev/null" > /root/.ssh/config
 
+COPY . /opt/jumpserver
 RUN echo > config.yml
 VOLUME /opt/jumpserver/data
 VOLUME /opt/jumpserver/logs
 
 ENV LANG=zh_CN.UTF-8
+ENV LC_ALL=zh_CN.UTF-8
 
 EXPOSE 8070
 EXPOSE 8080

README.md

@@ -4,118 +4,6 @@
 [![Django](https://img.shields.io/badge/django-2.2-brightgreen.svg?style=plastic)](https://www.djangoproject.com/)
 [![Docker Pulls](https://img.shields.io/docker/pulls/jumpserver/jms_all.svg)](https://hub.docker.com/u/jumpserver)
 
-- [ENGLISH](https://github.com/jumpserver/jumpserver/blob/master/README_EN.md)
-
-## 紧急BUG修复通知
-JumpServer发现远程执行漏洞，请速度修复
-
-非常感谢  **reactivity of Alibaba Hackerone bug bounty program**(瑞典) 向我们报告了此 BUG
-
-**影响版本:**
-```
-< v2.6.2
-< v2.5.4
-< v2.4.5 
-= v1.5.9
->= v1.5.3
-```
-**安全版本:**
-```
->= v2.6.2
->= v2.5.4
->= v2.4.5 
-= v1.5.9 （版本号没变）
-< v1.5.3
-```
-
-**修复方案:**
-
-将JumpServer升级至安全版本；
-
-**临时修复方案:**
-
-修改 Nginx 配置文件屏蔽漏洞接口 
-
-```
-/api/v1/authentication/connection-token/
-/api/v1/users/connection-token/
-```
-
-Nginx 配置文件位置
-```
-# 社区老版本
-/etc/nginx/conf.d/jumpserver.conf
-
-# 企业老版本
-jumpserver-release/nginx/http_server.conf
- 
-# 新版本在 
-jumpserver-release/compose/config_static/http_server.conf
-```
-
-修改 Nginx 配置文件实例
-```
-### 保证在 /api 之前 和 / 之前
-location /api/v1/authentication/connection-token/ {
-   return 403;
-}
- 
-location /api/v1/users/connection-token/ {
-   return 403;
-}
-### 新增以上这些
- 
-location /api/ {
-    proxy_set_header X-Real-IP $remote_addr;
-    proxy_set_header Host $host;
-    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
-    proxy_pass http://core:8080;
-  }
- 
-...
-```
-
-修改完成后重启 nginx
-
-```
-docker方式: 
-docker restart jms_nginx
-
-nginx方式:
-systemctl restart nginx
-
-```
-
-**修复验证**
-
-```
-$ wget https://github.com/jumpserver/jumpserver/releases/download/v2.6.2/jms_bug_check.sh 
-
-# 使用方法 bash jms_bug_check.sh HOST 
-$ bash jms_bug_check.sh demo.jumpserver.org
-漏洞已修复
-```
-
-**入侵检测**
-
-下载脚本到 jumpserver 日志目录，这个目录中存在 gunicorn.log，然后执行
-
-```
-$ pwd
-/opt/jumpserver/core/logs
-
-$ ls gunicorn.log 
-gunicorn.log
-
-$ wget 'https://github.com/jumpserver/jumpserver/releases/download/v2.6.2/jms_check_attack.sh'
-$ bash jms_check_attack.sh
-系统未被入侵
-```
-
---------------------------
-
-JumpServer 正在寻找开发者，一起为改变世界做些贡献吧，哪怕一点点，联系我 <ibuler@fit2cloud.com> 
-
 JumpServer 是全球首款开源的堡垒机，使用 GNU GPL v2.0 开源协议，是符合 4A 规范的运维安全审计系统。
 
 JumpServer 使用 Python / Django 为主进行开发，遵循 Web 2.0 规范，配备了业界领先的 Web Terminal 方案，交互界面美观、用户体验好。
@@ -128,13 +16,12 @@ JumpServer 采纳分布式架构，支持多机房跨区域部署，支持横向
 
 ## 特色优势
 
-- 开源: 零门槛，线上快速获取和安装；
-- 分布式: 轻松支持大规模并发访问；
+- 开源: 零门槛，线上快速获取和安装, 修复版本视情况而定；
+, 修复版本视情况而定- 分布式: 轻松支持大规模并发访问；
 - 无插件: 仅需浏览器，极致的 Web Terminal 使用体验；
 - 多云支持: 一套系统，同时管理不同云上面的资产；
 - 云端存储: 审计录像云端存储，永不丢失；
-- 多租户: 一套系统，多个子公司和部门同时使用；
-- 多应用支持: 数据库，Windows远程应用，Kubernetes。
+- 多租户: 一套系统，多个子公司和部门同时使用。
 
 ## 版本说明
 
@@ -307,76 +194,13 @@ v2.1.0 是 v2.0.0 之后的功能版本。
     <td>文件传输</td>
     <td>可对文件的上传、下载记录进行审计</td>
   </tr>
-  <tr>
-    <td rowspan="20">数据库审计<br>Database</td>
-    <td rowspan="2">连接方式</td>
-    <td>命令方式</td>
-  </tr>
-  <tr>
-    <td>Web UI方式 (X-PACK)</td>
-  </tr>
-
-  <tr>
-    <td rowspan="4">支持的数据库</td>
-    <td>MySQL</td>
-  </tr>
-  <tr>
-    <td>Oracle (X-PACK)</td>
-  </tr>
-  <tr>
-    <td>MariaDB (X-PACK)</td>
-  </tr>
-  <tr>
-    <td>PostgreSQL (X-PACK)</td>
-  </tr>
-  <tr>
-    <td rowspan="6">功能亮点</td>
-    <td>语法高亮</td>
-  </tr>
-  <tr>
-    <td>SQL格式化</td>
-  </tr>
-  <tr>
-    <td>支持快捷键</td>
-  </tr>
-  <tr>
-    <td>支持选中执行</td>
-  </tr>
-  <tr>
-    <td>SQL历史查询</td>
-  </tr>
-  <tr>
-    <td>支持页面创建 DB, TABLE</td>
-  </tr>
-  <tr>
-    <td rowspan="2">会话审计</td>
-    <td>命令记录</td>
-  </tr>
-  <tr>
-    <td>录像回放</td>
-  </tr>
 </table>
 
 ## 快速开始
 
 - [极速安装](https://docs.jumpserver.org/zh/master/install/setup_by_fast/)
 - [完整文档](https://docs.jumpserver.org)
-- [演示视频](https://www.bilibili.com/video/BV1ZV41127GB)
-
-## 组件项目
-- [Lina](https://github.com/jumpserver/lina) JumpServer Web UI 项目
-- [Luna](https://github.com/jumpserver/luna) JumpServer Web Terminal 项目
-- [Koko](https://github.com/jumpserver/koko) JumpServer 字符协议 Connector 项目，替代原来 Python 版本的 [Coco](https://github.com/jumpserver/coco)
-- [Guacamole](https://github.com/jumpserver/docker-guacamole) JumpServer 图形协议 Connector 项目，依赖 [Apache Guacamole](https://guacamole.apache.org/)
-
-## 致谢
-- [Apache Guacamole](https://guacamole.apache.org/) Web页面连接 RDP, SSH, VNC协议设备，JumpServer 图形化连接依赖
-- [OmniDB](https://omnidb.org/) Web页面连接使用数据库，JumpServer Web数据库依赖
-
-
-## JumpServer 企业版 
-- [申请企业版试用](https://jinshuju.net/f/kyOYpi)
-> 注：企业版支持离线安装，申请通过后会提供高速下载链接。
+- [演示视频](https://jumpserver.oss-cn-hangzhou.aliyuncs.com/jms-media/%E3%80%90%E6%BC%94%E7%A4%BA%E8%A7%86%E9%A2%91%E3%80%91Jumpserver%20%E5%A0%A1%E5%9E%92%E6%9C%BA%20V1.5.0%20%E6%BC%94%E7%A4%BA%E8%A7%86%E9%A2%91%20-%20final.mp4)
 
 ## 案例研究
 

README_EN.md

@@ -1,135 +1,22 @@
 ## Jumpserver 
 
+![Total visitor](https://visitor-count-badge.herokuapp.com/total.svg?repo_id=jumpserver)
+![Visitors in today](https://visitor-count-badge.herokuapp.com/today.svg?repo_id=jumpserver)
 [![Python3](https://img.shields.io/badge/python-3.6-green.svg?style=plastic)](https://www.python.org/)
-[![Django](https://img.shields.io/badge/django-2.2-brightgreen.svg?style=plastic)](https://www.djangoproject.com/)
-[![Docker Pulls](https://img.shields.io/docker/pulls/jumpserver/jms_all.svg)](https://hub.docker.com/u/jumpserver)
+[![Django](https://img.shields.io/badge/django-2.1-brightgreen.svg?style=plastic)](https://www.djangoproject.com/)
+[![Ansible](https://img.shields.io/badge/ansible-2.4.2.0-blue.svg?style=plastic)](https://www.ansible.com/)
+[![Paramiko](https://img.shields.io/badge/paramiko-2.4.1-green.svg?style=plastic)](http://www.paramiko.org/)
 
----- 
-## CRITICAL BUG WARNING
-
-Recently we have found a critical bug for remote execution vulnerability which leads to pre-auth and info leak, please fix it as soon as possible.
-
-Thanks for **reactivity from Alibaba Hackerone bug bounty program** report us this bug
-
-**Vulnerable version:**
-```
-< v2.6.2
-< v2.5.4
-< v2.4.5 
-= v1.5.9
->= v1.5.3
-```
-
-**Safe and Stable version:**
-```
->= v2.6.2
->= v2.5.4
->= v2.4.5 
-= v1.5.9 （version tag didn't change）
-< v1.5.3
-```
-
-**Bug Fix Solution:**
-Upgrade to the latest version or the version mentioned above
-
-
-**Temporary Solution (upgrade asap):**
-
-Modify the Nginx config file and disable the vulnerable api listed below
-
-```
-/api/v1/authentication/connection-token/
-/api/v1/users/connection-token/
-```
-
-Path to Nginx config file
-
-```
-# Previous Community version
-/etc/nginx/conf.d/jumpserver.conf
-
-# Previous Enterprise version
-jumpserver-release/nginx/http_server.conf
- 
-# Latest version
-jumpserver-release/compose/config_static/http_server.conf
-```
-
-Changes in Nginx config file
-
-```
-### Put the following code on top of location server, or before /api and /
-location /api/v1/authentication/connection-token/ {
-   return 403;
-}
- 
-location /api/v1/users/connection-token/ {
-   return 403;
-}
-### End right here
- 
-location /api/ {
-    proxy_set_header X-Real-IP $remote_addr;
-    proxy_set_header Host $host;
-    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
-    proxy_pass http://core:8080;
-  }
- 
-...
-```
-
-Save the file and restart Nginx
-
-```
-docker deployment: 
-$ docker restart jms_nginx
-
-rpm or other deployment:
-$ systemctl restart nginx
-
-```
-
-**Bug Fix Verification**
-
-```
-# Download the following script to check if it is fixed
-$ wget https://github.com/jumpserver/jumpserver/releases/download/v2.6.2/jms_bug_check.sh 
-
-# Run the code to verify it
-$ bash jms_bug_check.sh demo.jumpserver.org
-漏洞已修复 (It means the bug is fixed)
-漏洞未修复 (It means the bug is not fixed and the system is still vulnerable)
-```
-
-
-**Attack Simulation**
-
-Go to the logs directory which should contain gunicorn.log file. Then download the "attack" script and execute it
-
-```
-$ pwd
-/opt/jumpserver/core/logs
-
-$ ls gunicorn.log
-gunicorn.log
-
-$ wget 'https://github.com/jumpserver/jumpserver/releases/download/v2.6.2/jms_check_attack.sh'
-$ bash jms_check_attack.sh
-系统未被入侵 (It means the system is safe)
-系统已被入侵 (It means the system is being attacked)
-```
-
---------------------------
 
 ----
 
-- [中文版](https://github.com/jumpserver/jumpserver/blob/master/README.md)
+- [中文版](https://github.com/jumpserver/jumpserver/blob/master/README_EN.md)
 
-Jumpserver is the world's first open-source PAM (Privileged Access Management System) and is licensed under the GNU GPL v2.0. It is a 4A-compliant professional operation and maintenance security audit system.
+Jumpserver is the first fully open source bastion in the world, based on the GNU GPL v2.0 open source protocol. Jumpserver is a  professional operation and maintenance audit system conforms to 4A specifications.
 
-Jumpserver uses Python / Django for development, follows Web 2.0 specifications, and is equipped with an industry-leading Web Terminal solution that provides a beautiful user interface and great user experience
+Jumpserver is developed using Python / Django, conforms to the Web 2.0 specification, and is equipped with the industry-leading Web Terminal solution which have beautiful interface and great user experience.
 
-Jumpserver adopts a distributed architecture to support multi-branch deployment across multiple cross-regional areas. The central node provides APIs, and login nodes are deployed in each branch. It can be scaled horizontally without concurrency restrictions.
+Jumpserver adopts a distributed architecture to support multi-branch deployment across multiple areas. The central node provides APIs, and login nodes are deployed in each branch. It can be scaled horizontally without concurrency restrictions.
 
 Change the world, starting from little things.
 
@@ -160,7 +47,7 @@ We provide online demo, demo video and screenshots to get you started quickly.
 We provide the SDK for your other systems to quickly interact with the Jumpserver API.
 
 - [Python](https://github.com/jumpserver/jumpserver-python-sdk) Jumpserver other components use this SDK to complete the interaction.
-- [Java](https://github.com/KaiJunYan/jumpserver-java-sdk.git) Thanks to 恺珺 for providing his Java SDK vesrion.
+- [Java](https://github.com/KaiJunYan/jumpserver-java-sdk.git) 恺珺同学提供的Java版本的SDK thanks to 恺珺 for provide Java SDK
 
 
 ### License & Copyright

apps/.gitattributes

@@ -1,2 +0,0 @@
-*.js linguist-language=python
-*.html linguist-language=python

apps/applications/api/__init__.py

@@ -1,3 +1,2 @@
-from .application import *
-from .mixin import *
 from .remote_app import *
+from .database_app import *

apps/applications/api/application.py

@@ -1,19 +0,0 @@
-# coding: utf-8
-#
-
-from orgs.mixins.api import OrgBulkModelViewSet
-
-from ..hands import IsOrgAdminOrAppUser
-from .. import models, serializers
-
-
-__all__ = ['ApplicationViewSet']
-
-
-class ApplicationViewSet(OrgBulkModelViewSet):
-    model = models.Application
-    filterset_fields = ('name', 'type', 'category')
-    search_fields = filterset_fields
-    permission_classes = (IsOrgAdminOrAppUser,)
-    serializer_class = serializers.ApplicationSerializer
-

apps/applications/api/database_app.py

@@ -0,0 +1,20 @@
+# coding: utf-8
+# 
+
+from orgs.mixins.api import OrgBulkModelViewSet
+
+from .. import models
+from .. import serializers
+from ..hands import IsOrgAdminOrAppUser
+
+__all__ = [
+    'DatabaseAppViewSet',
+]
+
+
+class DatabaseAppViewSet(OrgBulkModelViewSet):
+    model = models.DatabaseApp
+    filter_fields = ('name',)
+    search_fields = filter_fields
+    permission_classes = (IsOrgAdminOrAppUser,)
+    serializer_class = serializers.DatabaseAppSerializer

apps/applications/api/mixin.py

@@ -1,89 +0,0 @@
-from orgs.models import Organization
-
-
-__all__ = ['SerializeApplicationToTreeNodeMixin']
-
-
-class SerializeApplicationToTreeNodeMixin:
-
-    @staticmethod
-    def _serialize_db(db):
-        return {
-            'id': db.id,
-            'name': db.name,
-            'title': db.name,
-            'pId': '',
-            'open': False,
-            'iconSkin': 'database',
-            'meta': {'type': 'database_app'}
-        }
-
-    @staticmethod
-    def _serialize_remote_app(remote_app):
-        return {
-            'id': remote_app.id,
-            'name': remote_app.name,
-            'title': remote_app.name,
-            'pId': '',
-            'open': False,
-            'isParent': False,
-            'iconSkin': 'chrome',
-            'meta': {'type': 'remote_app'}
-        }
-
-    @staticmethod
-    def _serialize_cloud(cloud):
-        return {
-            'id': cloud.id,
-            'name': cloud.name,
-            'title': cloud.name,
-            'pId': '',
-            'open': False,
-            'isParent': False,
-            'iconSkin': 'k8s',
-            'meta': {'type': 'k8s_app'}
-        }
-
-    def _serialize_application(self, application):
-        method_name = f'_serialize_{application.category}'
-        data = getattr(self, method_name)(application)
-        data.update({
-            'pId': application.org.id,
-            'org_name': application.org_name
-        })
-        return data
-
-    def serialize_applications(self, applications):
-        data = [self._serialize_application(application) for application in applications]
-        return data
-
-    @staticmethod
-    def _serialize_organization(org):
-        return {
-            'id': org.id,
-            'name': org.name,
-            'title': org.name,
-            'pId': '',
-            'open': True,
-            'isParent': True,
-            'meta': {
-                'type': 'node'
-            }
-        }
-
-    def serialize_organizations(self, organizations):
-        data = [self._serialize_organization(org) for org in organizations]
-        return data
-
-    @staticmethod
-    def filter_organizations(applications):
-        organizations_id = set(applications.values_list('org_id', flat=True))
-        organizations = [Organization.get_instance(org_id) for org_id in organizations_id]
-        return organizations
-
-    def serialize_applications_with_org(self, applications):
-        organizations = self.filter_organizations(applications)
-        data_organizations = self.serialize_organizations(organizations)
-        data_applications = self.serialize_applications(applications)
-        data = data_organizations + data_applications
-        return data

apps/applications/api/remote_app.py

@@ -1,19 +1,27 @@
 # coding: utf-8
 #
 
+from orgs.mixins.api import OrgBulkModelViewSet
 from orgs.mixins import generics
-from ..hands import IsAppUser
-from .. import models
-from ..serializers import RemoteAppConnectionInfoSerializer
-from ..permissions import IsRemoteApp
+from ..hands import IsOrgAdmin, IsAppUser
+from ..models import RemoteApp
+from ..serializers import RemoteAppSerializer, RemoteAppConnectionInfoSerializer
 
 
 __all__ = [
-    'RemoteAppConnectionInfoApi',
+    'RemoteAppViewSet', 'RemoteAppConnectionInfoApi',
 ]
 
 
+class RemoteAppViewSet(OrgBulkModelViewSet):
+    model = RemoteApp
+    filter_fields = ('name', 'type', 'comment')
+    search_fields = filter_fields
+    permission_classes = (IsOrgAdmin,)
+    serializer_class = RemoteAppSerializer
+
+
 class RemoteAppConnectionInfoApi(generics.RetrieveAPIView):
-    model = models.Application
-    permission_classes = (IsAppUser, IsRemoteApp)
+    model = RemoteApp
+    permission_classes = (IsAppUser, )
     serializer_class = RemoteAppConnectionInfoSerializer

apps/applications/const.py

@@ -1,49 +1,63 @@
 #  coding: utf-8
 #
 
-from django.db.models import TextChoices
 from django.utils.translation import ugettext_lazy as _
 
 
-class ApplicationCategoryChoices(TextChoices):
-    db = 'db', _('Database')
-    remote_app = 'remote_app', _('Remote app')
-    cloud = 'cloud', 'Cloud'
+# RemoteApp
 
-    @classmethod
-    def get_label(cls, category):
-        return dict(cls.choices).get(category, '')
+REMOTE_APP_BOOT_PROGRAM_NAME = '||jmservisor'
+
+REMOTE_APP_TYPE_CHROME = 'chrome'
+REMOTE_APP_TYPE_MYSQL_WORKBENCH = 'mysql_workbench'
+REMOTE_APP_TYPE_VMWARE_CLIENT = 'vmware_client'
+REMOTE_APP_TYPE_CUSTOM = 'custom'
+
+# Fields attribute write_only default => False
+
+REMOTE_APP_TYPE_CHROME_FIELDS = [
+    {'name': 'chrome_target'},
+    {'name': 'chrome_username'},
+    {'name': 'chrome_password', 'write_only': True}
+]
+REMOTE_APP_TYPE_MYSQL_WORKBENCH_FIELDS = [
+    {'name': 'mysql_workbench_ip'},
+    {'name': 'mysql_workbench_name'},
+    {'name': 'mysql_workbench_username'},
+    {'name': 'mysql_workbench_password', 'write_only': True}
+]
+REMOTE_APP_TYPE_VMWARE_CLIENT_FIELDS = [
+    {'name': 'vmware_target'},
+    {'name': 'vmware_username'},
+    {'name': 'vmware_password', 'write_only': True}
+]
+REMOTE_APP_TYPE_CUSTOM_FIELDS = [
+    {'name': 'custom_cmdline'},
+    {'name': 'custom_target'},
+    {'name': 'custom_username'},
+    {'name': 'custom_password', 'write_only': True}
+]
+
+REMOTE_APP_TYPE_FIELDS_MAP = {
+    REMOTE_APP_TYPE_CHROME: REMOTE_APP_TYPE_CHROME_FIELDS,
+    REMOTE_APP_TYPE_MYSQL_WORKBENCH: REMOTE_APP_TYPE_MYSQL_WORKBENCH_FIELDS,
+    REMOTE_APP_TYPE_VMWARE_CLIENT: REMOTE_APP_TYPE_VMWARE_CLIENT_FIELDS,
+    REMOTE_APP_TYPE_CUSTOM: REMOTE_APP_TYPE_CUSTOM_FIELDS
+}
+
+REMOTE_APP_TYPE_CHOICES = (
+    (REMOTE_APP_TYPE_CHROME, 'Chrome'),
+    (REMOTE_APP_TYPE_MYSQL_WORKBENCH, 'MySQL Workbench'),
+    (REMOTE_APP_TYPE_VMWARE_CLIENT, 'vSphere Client'),
+    (REMOTE_APP_TYPE_CUSTOM, _('Custom')),
+)
 
 
-class ApplicationTypeChoices(TextChoices):
-    # db category
-    mysql = 'mysql', 'MySQL'
-    oracle = 'oracle', 'Oracle'
-    pgsql = 'postgresql', 'PostgreSQL'
-    mariadb = 'mariadb', 'MariaDB'
+# DatabaseApp
 
-    # remote-app category
-    chrome = 'chrome', 'Chrome'
-    mysql_workbench = 'mysql_workbench', 'MySQL Workbench'
-    vmware_client = 'vmware_client', 'vSphere Client'
-    custom = 'custom', _('Custom')
 
-    # cloud category
-    k8s = 'k8s', 'Kubernetes'
-
-    @classmethod
-    def get_label(cls, tp):
-        return dict(cls.choices).get(tp, '')
-
-    @classmethod
-    def db_types(cls):
-        return [cls.mysql.value, cls.oracle.value, cls.pgsql.value, cls.mariadb.value]
-
-    @classmethod
-    def remote_app_types(cls):
-        return [cls.chrome.value, cls.mysql_workbench.value, cls.vmware_client.value, cls.custom.value]
-
-    @classmethod
-    def cloud_types(cls):
-        return [cls.k8s.value]
+DATABASE_APP_TYPE_MYSQL = 'mysql'
 
+DATABASE_APP_TYPE_CHOICES = (
+    (DATABASE_APP_TYPE_MYSQL, 'MySQL'),
+)

apps/applications/migrations/0005_k8sapp.py

@@ -1,34 +0,0 @@
-# Generated by Django 2.2.13 on 2020-08-07 07:13
-
-from django.db import migrations, models
-import uuid
-
-
-class Migration(migrations.Migration):
-
-    dependencies = [
-        ('applications', '0004_auto_20191218_1705'),
-    ]
-
-    operations = [
-        migrations.CreateModel(
-            name='K8sApp',
-            fields=[
-                ('created_by', models.CharField(blank=True, max_length=32, null=True, verbose_name='Created by')),
-                ('updated_by', models.CharField(blank=True, max_length=32, null=True, verbose_name='Updated by')),
-                ('date_created', models.DateTimeField(auto_now_add=True, null=True, verbose_name='Date created')),
-                ('date_updated', models.DateTimeField(auto_now=True, verbose_name='Date updated')),
-                ('id', models.UUIDField(default=uuid.uuid4, primary_key=True, serialize=False)),
-                ('org_id', models.CharField(blank=True, db_index=True, default='', max_length=36, verbose_name='Organization')),
-                ('name', models.CharField(max_length=128, verbose_name='Name')),
-                ('type', models.CharField(choices=[('k8s', 'Kubernetes')], default='k8s', max_length=128, verbose_name='Type')),
-                ('cluster', models.CharField(max_length=1024, verbose_name='Cluster')),
-                ('comment', models.TextField(blank=True, default='', max_length=128, verbose_name='Comment')),
-            ],
-            options={
-                'verbose_name': 'KubernetesApp',
-                'ordering': ('name',),
-                'unique_together': {('org_id', 'name')},
-            },
-        ),
-    ]

apps/applications/migrations/0006_application.py

@@ -1,140 +0,0 @@
-# Generated by Django 2.2.13 on 2020-10-19 12:01
-
-from django.db import migrations, models
-import django.db.models.deletion
-import django_mysql.models
-import uuid
-
-
-CATEGORY_DB_LIST = ['mysql', 'oracle', 'postgresql', 'mariadb']
-CATEGORY_REMOTE_LIST = ['chrome', 'mysql_workbench', 'vmware_client', 'custom']
-CATEGORY_CLOUD_LIST = ['k8s']
-
-CATEGORY_DB = 'db'
-CATEGORY_REMOTE = 'remote_app'
-CATEGORY_CLOUD = 'cloud'
-CATEGORY_LIST = [CATEGORY_DB, CATEGORY_REMOTE, CATEGORY_CLOUD]
-
-
-def get_application_category(old_app):
-    _type = old_app.type
-    if _type in CATEGORY_DB_LIST:
-        category = CATEGORY_DB
-    elif _type in CATEGORY_REMOTE_LIST:
-        category = CATEGORY_REMOTE
-    elif _type in CATEGORY_CLOUD_LIST:
-        category = CATEGORY_CLOUD
-    else:
-        category = None
-    return category
-
-
-def common_to_application_json(old_app):
-    category = get_application_category(old_app)
-    date_updated = old_app.date_updated if hasattr(old_app, 'date_updated') else old_app.date_created
-    return {
-        'id': old_app.id,
-        'name': old_app.name,
-        'type': old_app.type,
-        'category': category,
-        'comment': old_app.comment,
-        'created_by': old_app.created_by,
-        'date_created': old_app.date_created,
-        'date_updated': date_updated,
-        'org_id': old_app.org_id
-    }
-
-
-def db_to_application_json(database):
-    app_json = common_to_application_json(database)
-    app_json.update({
-        'attrs': {
-            'host': database.host,
-            'port': database.port,
-            'database': database.database
-        }
-    })
-    return app_json
-
-
-def remote_to_application_json(remote):
-    app_json = common_to_application_json(remote)
-    attrs = {
-        'asset': str(remote.asset.id),
-        'path': remote.path,
-    }
-    attrs.update(remote.params)
-    app_json.update({
-        'attrs': attrs
-    })
-    return app_json
-
-
-def k8s_to_application_json(k8s):
-    app_json = common_to_application_json(k8s)
-    app_json.update({
-        'attrs': {
-            'cluster': k8s.cluster
-        }
-    })
-    return app_json
-
-
-def migrate_and_integrate_applications(apps, schema_editor):
-    db_alias = schema_editor.connection.alias
-
-    database_app_model = apps.get_model("applications", "DatabaseApp")
-    remote_app_model = apps.get_model("applications", "RemoteApp")
-    k8s_app_model = apps.get_model("applications", "K8sApp")
-
-    database_apps = database_app_model.objects.using(db_alias).all()
-    remote_apps = remote_app_model.objects.using(db_alias).all()
-    k8s_apps = k8s_app_model.objects.using(db_alias).all()
-
-    database_applications = [db_to_application_json(db_app) for db_app in database_apps]
-    remote_applications = [remote_to_application_json(remote_app) for remote_app in remote_apps]
-    k8s_applications = [k8s_to_application_json(k8s_app) for k8s_app in k8s_apps]
-
-    applications_json = database_applications + remote_applications + k8s_applications
-    application_model = apps.get_model("applications", "Application")
-    applications = [
-        application_model(**application_json)
-        for application_json in applications_json
-        if application_json['category'] in CATEGORY_LIST
-    ]
-    for application in applications:
-        if application_model.objects.using(db_alias).filter(name=application.name).exists():
-            application.name = '{}-{}'.format(application.name, application.type)
-        application.save()
-
-
-class Migration(migrations.Migration):
-
-    dependencies = [
-        ('assets', '0057_fill_node_value_assets_amount_and_parent_key'),
-        ('applications', '0005_k8sapp'),
-    ]
-
-    operations = [
-        migrations.CreateModel(
-            name='Application',
-            fields=[
-                ('org_id', models.CharField(blank=True, db_index=True, default='', max_length=36, verbose_name='Organization')),
-                ('id', models.UUIDField(default=uuid.uuid4, primary_key=True, serialize=False)),
-                ('created_by', models.CharField(blank=True, max_length=32, null=True, verbose_name='Created by')),
-                ('date_created', models.DateTimeField(auto_now_add=True, null=True, verbose_name='Date created')),
-                ('date_updated', models.DateTimeField(auto_now=True, verbose_name='Date updated')),
-                ('name', models.CharField(max_length=128, verbose_name='Name')),
-                ('category', models.CharField(choices=[('db', 'Database'), ('remote_app', 'Remote app'), ('cloud', 'Cloud')], max_length=16, verbose_name='Category')),
-                ('type', models.CharField(choices=[('mysql', 'MySQL'), ('oracle', 'Oracle'), ('postgresql', 'PostgreSQL'), ('mariadb', 'MariaDB'), ('chrome', 'Chrome'), ('mysql_workbench', 'MySQL Workbench'), ('vmware_client', 'vSphere Client'), ('custom', 'Custom'), ('k8s', 'Kubernetes')], max_length=16, verbose_name='Type')),
-                ('attrs', django_mysql.models.JSONField(default=dict)),
-                ('comment', models.TextField(blank=True, default='', max_length=128, verbose_name='Comment')),
-                ('domain', models.ForeignKey(blank=True, null=True, on_delete=django.db.models.deletion.SET_NULL, related_name='applications', to='assets.Domain', verbose_name='Domain')),
-            ],
-            options={
-                'ordering': ('name',),
-                'unique_together': {('org_id', 'name')},
-            },
-        ),
-        migrations.RunPython(migrate_and_integrate_applications),
-    ]

apps/applications/migrations/0007_auto_20201119_1110.py

@@ -1,18 +0,0 @@
-# Generated by Django 3.1 on 2020-11-19 03:10
-
-from django.db import migrations, models
-
-
-class Migration(migrations.Migration):
-
-    dependencies = [
-        ('applications', '0006_application'),
-    ]
-
-    operations = [
-        migrations.AlterField(
-            model_name='application',
-            name='attrs',
-            field=models.JSONField(),
-        ),
-    ]

apps/applications/migrations/0008_auto_20210104_0435.py

@@ -1,28 +0,0 @@
-# Generated by Django 3.1 on 2021-01-03 20:35
-
-from django.db import migrations, models
-
-
-class Migration(migrations.Migration):
-
-    dependencies = [
-        ('perms', '0017_auto_20210104_0435'),
-        ('applications', '0007_auto_20201119_1110'),
-    ]
-
-    operations = [
-        migrations.DeleteModel(
-            name='DatabaseApp',
-        ),
-        migrations.DeleteModel(
-            name='K8sApp',
-        ),
-        migrations.AlterField(
-            model_name='application',
-            name='attrs',
-            field=models.JSONField(default=dict, verbose_name='Attrs'),
-        ),
-        migrations.DeleteModel(
-            name='RemoteApp',
-        ),
-    ]

apps/applications/models/__init__.py

@@ -1 +1,2 @@
-from .application import *
+from .remote_app import *
+from .database_app import *

apps/applications/models/application.py

@@ -1,37 +0,0 @@
-from django.db import models
-from django.utils.translation import ugettext_lazy as _
-
-from orgs.mixins.models import OrgModelMixin
-from common.mixins import CommonModelMixin
-from .. import const
-
-
-class Application(CommonModelMixin, OrgModelMixin):
-    name = models.CharField(max_length=128, verbose_name=_('Name'))
-    category = models.CharField(
-        max_length=16, choices=const.ApplicationCategoryChoices.choices, verbose_name=_('Category')
-    )
-    type = models.CharField(
-        max_length=16, choices=const.ApplicationTypeChoices.choices, verbose_name=_('Type')
-    )
-    domain = models.ForeignKey(
-        'assets.Domain', null=True, blank=True, related_name='applications',
-        on_delete=models.SET_NULL, verbose_name=_("Domain"),
-    )
-    attrs = models.JSONField(default=dict, verbose_name=_('Attrs'))
-    comment = models.TextField(
-        max_length=128, default='', blank=True, verbose_name=_('Comment')
-    )
-
-    class Meta:
-        unique_together = [('org_id', 'name')]
-        ordering = ('name',)
-
-    def __str__(self):
-        category_display = self.get_category_display()
-        type_display = self.get_type_display()
-        return f'{self.name}({type_display})[{category_display}]'
-
-    @property
-    def category_remote_app(self):
-        return self.category == const.ApplicationCategoryChoices.remote_app.value

apps/applications/models/database_app.py

@@ -0,0 +1,42 @@
+# coding: utf-8
+#
+
+import uuid
+from django.db import models
+from django.utils.translation import ugettext_lazy as _
+
+from orgs.mixins.models import OrgModelMixin
+from common.mixins import CommonModelMixin
+from .. import const
+
+
+__all__ = ['DatabaseApp']
+
+
+class DatabaseApp(CommonModelMixin, OrgModelMixin):
+    id = models.UUIDField(default=uuid.uuid4, primary_key=True)
+    name = models.CharField(max_length=128, verbose_name=_('Name'))
+    type = models.CharField(
+        default=const.DATABASE_APP_TYPE_MYSQL,
+        choices=const.DATABASE_APP_TYPE_CHOICES,
+        max_length=128, verbose_name=_('Type')
+    )
+    host = models.CharField(
+        max_length=128, verbose_name=_('Host'), db_index=True
+    )
+    port = models.IntegerField(default=3306, verbose_name=_('Port'))
+    database = models.CharField(
+        max_length=128, blank=True, null=True, verbose_name=_('Database'),
+        db_index=True
+    )
+    comment = models.TextField(
+        max_length=128, default='', blank=True, verbose_name=_('Comment')
+    )
+
+    def __str__(self):
+        return self.name
+
+    class Meta:
+        unique_together = [('org_id', 'name'), ]
+        verbose_name = _("DatabaseApp")
+        ordering = ('name', )

apps/applications/models/remote_app.py

@@ -0,0 +1,78 @@
+# coding: utf-8
+#
+
+import uuid
+from django.db import models
+from django.utils.translation import ugettext_lazy as _
+
+from orgs.mixins.models import OrgModelMixin
+from common.fields.model import EncryptJsonDictTextField
+
+from .. import const
+
+
+__all__ = [
+    'RemoteApp',
+]
+
+
+class RemoteApp(OrgModelMixin):
+    id = models.UUIDField(default=uuid.uuid4, primary_key=True)
+    name = models.CharField(max_length=128, verbose_name=_('Name'))
+    asset = models.ForeignKey(
+        'assets.Asset', on_delete=models.CASCADE, verbose_name=_('Asset')
+    )
+    type = models.CharField(
+        default=const.REMOTE_APP_TYPE_CHROME,
+        choices=const.REMOTE_APP_TYPE_CHOICES,
+        max_length=128, verbose_name=_('App type')
+    )
+    path = models.CharField(
+        max_length=128, blank=False, null=False,
+        verbose_name=_('App path')
+    )
+    params = EncryptJsonDictTextField(
+        max_length=4096, default={}, blank=True, null=True,
+        verbose_name=_('Parameters')
+    )
+    created_by = models.CharField(
+        max_length=32, null=True, blank=True, verbose_name=_('Created by')
+    )
+    date_created = models.DateTimeField(
+        auto_now_add=True, null=True, blank=True, verbose_name=_('Date created')
+    )
+    comment = models.TextField(
+        max_length=128, default='', blank=True, verbose_name=_('Comment')
+    )
+
+    class Meta:
+        verbose_name = _("RemoteApp")
+        unique_together = [('org_id', 'name')]
+        ordering = ('name', )
+
+    def __str__(self):
+        return self.name
+
+    @property
+    def parameters(self):
+        """
+        返回Guacamole需要的RemoteApp配置参数信息中的parameters参数
+        """
+        _parameters = list()
+        _parameters.append(self.type)
+        path = '\"%s\"' % self.path
+        _parameters.append(path)
+        for field in const.REMOTE_APP_TYPE_FIELDS_MAP[self.type]:
+            value = self.params.get(field['name'])
+            if value is None:
+                continue
+            _parameters.append(value)
+        _parameters = ' '.join(_parameters)
+        return _parameters
+
+    @property
+    def asset_info(self):
+        return {
+            'id': self.asset.id,
+            'hostname': self.asset.hostname
+        }

apps/applications/permissions.py

@@ -1,9 +0,0 @@
-from rest_framework import permissions
-
-
-__all__ = ['IsRemoteApp']
-
-
-class IsRemoteApp(permissions.BasePermission):
-    def has_object_permission(self, request, view, obj):
-        return obj.category_remote_app

apps/applications/serializers/__init__.py

@@ -1,2 +1,2 @@
-from .application import *
 from .remote_app import *
+from .database_app import *

apps/applications/serializers/application.py

@@ -1,64 +0,0 @@
-# coding: utf-8
-#
-
-from rest_framework import serializers
-from django.utils.translation import ugettext_lazy as _
-from orgs.mixins.serializers import BulkOrgResourceModelSerializer
-from common.drf.serializers import MethodSerializer
-from .attrs import category_serializer_classes_mapping, type_serializer_classes_mapping
-
-from .. import models
-
-__all__ = [
-    'ApplicationSerializer', 'ApplicationSerializerMixin',
-]
-
-
-class ApplicationSerializerMixin(serializers.Serializer):
-    attrs = MethodSerializer()
-
-    def get_attrs_serializer(self):
-        default_serializer = serializers.Serializer(read_only=True)
-        if isinstance(self.instance, models.Application):
-            _type = self.instance.type
-            _category = self.instance.category
-        else:
-            _type = self.context['request'].query_params.get('type')
-            _category = self.context['request'].query_params.get('category')
-
-        if _type:
-            serializer_class = type_serializer_classes_mapping.get(_type)
-        elif _category:
-            serializer_class = category_serializer_classes_mapping.get(_category)
-        else:
-            serializer_class = default_serializer
-
-        if not serializer_class:
-            serializer_class = default_serializer
-
-        if isinstance(serializer_class, type):
-            serializer = serializer_class()
-        else:
-            serializer = serializer_class
-        return serializer
-
-
-class ApplicationSerializer(ApplicationSerializerMixin, BulkOrgResourceModelSerializer):
-    category_display = serializers.ReadOnlyField(source='get_category_display', label=_('Category'))
-    type_display = serializers.ReadOnlyField(source='get_type_display', label=_('Type'))
-
-    class Meta:
-        model = models.Application
-        fields = [
-            'id', 'name', 'category', 'category_display', 'type', 'type_display', 'attrs',
-            'domain', 'created_by', 'date_created', 'date_updated', 'comment'
-        ]
-        read_only_fields = [
-            'created_by', 'date_created', 'date_updated', 'get_type_display',
-        ]
-
-    def validate_attrs(self, attrs):
-        _attrs = self.instance.attrs if self.instance else {}
-        _attrs.update(attrs)
-        return _attrs
-

apps/applications/serializers/attrs/__init__.py

@@ -1 +0,0 @@
-from .attrs import *

apps/applications/serializers/attrs/application_category/__init__.py

@@ -1,3 +0,0 @@
-from .remote_app import *
-from .db import *
-from .cloud import *

apps/applications/serializers/attrs/application_category/cloud.py

@@ -1,9 +0,0 @@
-from rest_framework import serializers
-from django.utils.translation import ugettext_lazy as _
-
-
-__all__ = ['CloudSerializer']
-
-
-class CloudSerializer(serializers.Serializer):
-    cluster = serializers.CharField(max_length=1024, label=_('Cluster'), allow_null=True)

apps/applications/serializers/attrs/application_category/db.py

@@ -1,15 +0,0 @@
-# coding: utf-8
-#
-from rest_framework import serializers
-from django.utils.translation import ugettext_lazy as _
-
-
-__all__ = ['DBSerializer']
-
-
-class DBSerializer(serializers.Serializer):
-    host = serializers.CharField(max_length=128, label=_('Host'), allow_null=True)
-    port = serializers.IntegerField(label=_('Port'), allow_null=True)
-    database = serializers.CharField(
-        max_length=128, required=True, allow_null=True, label=_('Database')
-    )

apps/applications/serializers/attrs/application_category/remote_app.py

@@ -1,52 +0,0 @@
-# coding: utf-8
-#
-
-from rest_framework import serializers
-from django.utils.translation import ugettext_lazy as _
-from django.core.exceptions import ObjectDoesNotExist
-
-from common.utils import get_logger, is_uuid
-from assets.models import Asset
-
-logger = get_logger(__file__)
-
-
-__all__ = ['RemoteAppSerializer']
-
-
-class CharPrimaryKeyRelatedField(serializers.PrimaryKeyRelatedField):
-
-    def to_internal_value(self, data):
-        instance = super().to_internal_value(data)
-        return str(instance.id)
-
-    def to_representation(self, value):
-        # value is instance.id
-        if self.pk_field is not None:
-            return self.pk_field.to_representation(value)
-        return value
-
-
-class RemoteAppSerializer(serializers.Serializer):
-    asset_info = serializers.SerializerMethodField()
-    asset = CharPrimaryKeyRelatedField(
-        queryset=Asset.objects, required=False, label=_("Asset"), allow_null=True
-    )
-    path = serializers.CharField(
-        max_length=128, label=_('Application path'), allow_null=True
-    )
-
-    @staticmethod
-    def get_asset_info(obj):
-        asset_id = obj.get('asset')
-        if not asset_id or is_uuid(asset_id):
-            return {}
-        try:
-            asset = Asset.objects.filter(id=str(asset_id)).values_list('id', 'hostname')
-        except ObjectDoesNotExist as e:
-            logger.error(e)
-            return {}
-        if not asset:
-            return {}
-        asset_info = {'id': str(asset[0]), 'hostname': asset[1]}
-        return asset_info

apps/applications/serializers/attrs/application_type/__init__.py

@@ -1,12 +0,0 @@
-
-from .mysql import *
-from .mariadb import *
-from .oracle import *
-from .pgsql import *
-
-from .chrome import *
-from .mysql_workbench import *
-from .vmware_client import *
-from .custom import *
-
-from .k8s import *

apps/applications/serializers/attrs/application_type/chrome.py

@@ -1,26 +0,0 @@
-from django.utils.translation import ugettext_lazy as _
-from rest_framework import serializers
-
-from ..application_category import RemoteAppSerializer
-
-
-__all__ = ['ChromeSerializer']
-
-
-class ChromeSerializer(RemoteAppSerializer):
-    CHROME_PATH = 'C:\Program Files (x86)\Google\Chrome\Application\chrome.exe'
-
-    path = serializers.CharField(
-        max_length=128, label=_('Application path'), default=CHROME_PATH, allow_null=True,
-    )
-    chrome_target = serializers.CharField(
-        max_length=128, allow_blank=True, required=False, label=_('Target URL'), allow_null=True,
-    )
-    chrome_username = serializers.CharField(
-        max_length=128, allow_blank=True, required=False, label=_('Username'), allow_null=True,
-    )
-    chrome_password = serializers.CharField(
-        max_length=128, allow_blank=True, required=False, write_only=True, label=_('Password'),
-        allow_null=True
-    )
-

apps/applications/serializers/attrs/application_type/custom.py

@@ -1,27 +0,0 @@
-
-from django.utils.translation import ugettext_lazy as _
-from rest_framework import serializers
-
-from ..application_category import RemoteAppSerializer
-
-
-__all__ = ['CustomSerializer']
-
-
-class CustomSerializer(RemoteAppSerializer):
-    custom_cmdline = serializers.CharField(
-        max_length=128, allow_blank=True, required=False, label=_('Operating parameter'),
-        allow_null=True,
-    )
-    custom_target = serializers.CharField(
-        max_length=128, allow_blank=True, required=False, label=_('Target url'),
-        allow_null=True,
-    )
-    custom_username = serializers.CharField(
-        max_length=128, allow_blank=True, required=False, label=_('Username'),
-        allow_null=True,
-    )
-    custom_password = serializers.CharField(
-        max_length=128, allow_blank=True, required=False, write_only=True, label=_('Password'),
-        allow_null=True,
-    )

apps/applications/serializers/attrs/application_type/k8s.py

@@ -1,8 +0,0 @@
-from ..application_category import CloudSerializer
-
-
-__all__ = ['K8SSerializer']
-
-
-class K8SSerializer(CloudSerializer):
-    pass

apps/applications/serializers/attrs/application_type/mariadb.py

@@ -1,8 +0,0 @@
-from .mysql import MySQLSerializer
-
-
-__all__ = ['MariaDBSerializer']
-
-
-class MariaDBSerializer(MySQLSerializer):
-    pass

apps/applications/serializers/attrs/application_type/mysql.py

@@ -1,15 +0,0 @@
-from rest_framework import serializers
-from django.utils.translation import ugettext_lazy as _
-
-from ..application_category import DBSerializer
-
-
-__all__ = ['MySQLSerializer']
-
-
-class MySQLSerializer(DBSerializer):
-    port = serializers.IntegerField(default=3306, label=_('Port'), allow_null=True)
-
-
-
-

apps/applications/serializers/attrs/application_type/mysql_workbench.py

@@ -1,36 +0,0 @@
-from django.utils.translation import ugettext_lazy as _
-from rest_framework import serializers
-
-from ..application_category import RemoteAppSerializer
-
-
-__all__ = ['MySQLWorkbenchSerializer']
-
-
-class MySQLWorkbenchSerializer(RemoteAppSerializer):
-    MYSQL_WORKBENCH_PATH = 'C:\Program Files\MySQL\MySQL Workbench 8.0 CE\MySQLWorkbench.exe'
-
-    path = serializers.CharField(
-        max_length=128, label=_('Application path'), default=MYSQL_WORKBENCH_PATH,
-        allow_null=True,
-    )
-    mysql_workbench_ip = serializers.CharField(
-        max_length=128, allow_blank=True, required=False, label=_('IP'),
-        allow_null=True,
-    )
-    mysql_workbench_port = serializers.IntegerField(
-        required=False, label=_('Port'),
-        allow_null=True,
-    )
-    mysql_workbench_name = serializers.CharField(
-        max_length=128, allow_blank=True, required=False, label=_('Database'),
-        allow_null=True,
-    )
-    mysql_workbench_username = serializers.CharField(
-        max_length=128, allow_blank=True, required=False, label=_('Username'),
-        allow_null=True,
-    )
-    mysql_workbench_password = serializers.CharField(
-        max_length=128, allow_blank=True, required=False, write_only=True, label=_('Password'),
-        allow_null=True,
-    )

apps/applications/serializers/attrs/application_type/oracle.py

@@ -1,12 +0,0 @@
-from rest_framework import serializers
-from django.utils.translation import ugettext_lazy as _
-
-from ..application_category import DBSerializer
-
-
-__all__ = ['OracleSerializer']
-
-
-class OracleSerializer(DBSerializer):
-    port = serializers.IntegerField(default=1521, label=_('Port'), allow_null=True)
-

apps/applications/serializers/attrs/application_type/pgsql.py

@@ -1,12 +0,0 @@
-from rest_framework import serializers
-from django.utils.translation import ugettext_lazy as _
-
-from ..application_category import DBSerializer
-
-
-__all__ = ['PostgreSerializer']
-
-
-class PostgreSerializer(DBSerializer):
-    port = serializers.IntegerField(default=5432, label=_('Port'), allow_null=True)
-

apps/applications/serializers/attrs/application_type/vmware_client.py

@@ -1,32 +0,0 @@
-from django.utils.translation import ugettext_lazy as _
-from rest_framework import serializers
-
-from ..application_category import RemoteAppSerializer
-
-
-__all__ = ['VMwareClientSerializer']
-
-
-class VMwareClientSerializer(RemoteAppSerializer):
-    PATH = r'''
-    C:\Program Files (x86)\VMware\Infrastructure\Virtual Infrastructure Client\Launcher\VpxClient
-    .exe
-    '''
-    VMWARE_CLIENT_PATH = ''.join(PATH.split())
-
-    path = serializers.CharField(
-        max_length=128, label=_('Application path'), default=VMWARE_CLIENT_PATH,
-        allow_null=True
-    )
-    vmware_target = serializers.CharField(
-        max_length=128, allow_blank=True, required=False, label=_('Target URL'),
-        allow_null=True
-    )
-    vmware_username = serializers.CharField(
-        max_length=128, allow_blank=True, required=False, label=_('Username'),
-        allow_null=True
-    )
-    vmware_password = serializers.CharField(
-        max_length=128, allow_blank=True, required=False, write_only=True, label=_('Password'),
-        allow_null=True
-    )

apps/applications/serializers/attrs/attrs.py

@@ -1,42 +0,0 @@
-from rest_framework import serializers
-from applications import const
-from . import application_category, application_type
-
-
-__all__ = [
-    'category_serializer_classes_mapping',
-    'type_serializer_classes_mapping',
-    'get_serializer_class_by_application_type',
-]
-
-
-# define `attrs` field `category serializers mapping`
-# ---------------------------------------------------
-
-category_serializer_classes_mapping = {
-    const.ApplicationCategoryChoices.db.value: application_category.DBSerializer,
-    const.ApplicationCategoryChoices.remote_app.value: application_category.RemoteAppSerializer,
-    const.ApplicationCategoryChoices.cloud.value: application_category.CloudSerializer,
-}
-
-# define `attrs` field `type serializers mapping`
-# -----------------------------------------------
-
-type_serializer_classes_mapping = {
-    # db
-    const.ApplicationTypeChoices.mysql.value: application_type.MySQLSerializer,
-    const.ApplicationTypeChoices.mariadb.value: application_type.MariaDBSerializer,
-    const.ApplicationTypeChoices.oracle.value: application_type.OracleSerializer,
-    const.ApplicationTypeChoices.pgsql.value: application_type.PostgreSerializer,
-    # remote-app
-    const.ApplicationTypeChoices.chrome.value: application_type.ChromeSerializer,
-    const.ApplicationTypeChoices.mysql_workbench.value: application_type.MySQLWorkbenchSerializer,
-    const.ApplicationTypeChoices.vmware_client.value: application_type.VMwareClientSerializer,
-    const.ApplicationTypeChoices.custom.value: application_type.CustomSerializer,
-    # cloud
-    const.ApplicationTypeChoices.k8s.value: application_type.K8SSerializer
-}
-
-
-def get_serializer_class_by_application_type(_application_type):
-    return type_serializer_classes_mapping.get(_application_type)

apps/applications/serializers/database_app.py

@@ -0,0 +1,26 @@
+# coding: utf-8
+#
+
+from orgs.mixins.serializers import BulkOrgResourceModelSerializer
+from common.serializers import AdaptedBulkListSerializer
+
+from .. import models
+
+__all__ = [
+    'DatabaseAppSerializer',
+]
+
+
+class DatabaseAppSerializer(BulkOrgResourceModelSerializer):
+
+    class Meta:
+        model = models.DatabaseApp
+        list_serializer_class = AdaptedBulkListSerializer
+        fields = [
+            'id', 'name', 'type', 'get_type_display', 'host', 'port',
+            'database', 'comment', 'created_by', 'date_created', 'date_updated',
+        ]
+        read_only_fields = [
+            'created_by', 'date_created', 'date_updated'
+            'get_type_display',
+        ]

apps/applications/serializers/remote_app.py

@@ -1,57 +1,86 @@
 # coding: utf-8
 #
+
+import copy
 from rest_framework import serializers
-from common.utils import get_logger
-from ..models import Application
+
+from common.serializers import AdaptedBulkListSerializer
+from common.fields.serializer import CustomMetaDictField
+from orgs.mixins.serializers import BulkOrgResourceModelSerializer
+
+from .. import const
+from ..models import RemoteApp
 
 
-logger = get_logger(__file__)
+__all__ = [
+    'RemoteAppSerializer', 'RemoteAppConnectionInfoSerializer',
+]
 
 
-__all__ = ['RemoteAppConnectionInfoSerializer']
+class RemoteAppParamsDictField(CustomMetaDictField):
+    type_fields_map = const.REMOTE_APP_TYPE_FIELDS_MAP
+    default_type = const.REMOTE_APP_TYPE_CHROME
+    convert_key_remove_type_prefix = False
+    convert_key_to_upper = False
+
+
+class RemoteAppSerializer(BulkOrgResourceModelSerializer):
+    params = RemoteAppParamsDictField()
+    type_fields_map = const.REMOTE_APP_TYPE_FIELDS_MAP
+
+    class Meta:
+        model = RemoteApp
+        list_serializer_class = AdaptedBulkListSerializer
+        fields = [
+            'id', 'name', 'asset', 'asset_info', 'type', 'get_type_display',
+            'path', 'params', 'date_created', 'created_by', 'comment',
+        ]
+        read_only_fields = [
+            'created_by', 'date_created', 'asset_info',
+            'get_type_display'
+        ]
+
+    def process_params(self, instance, validated_data):
+        new_params = copy.deepcopy(validated_data.get('params', {}))
+        tp = validated_data.get('type', '')
+
+        if tp != instance.type:
+            return new_params
+
+        old_params = instance.params
+        fields = self.type_fields_map.get(instance.type, [])
+        for field in fields:
+            if not field.get('write_only', False):
+                continue
+            field_name = field['name']
+            new_value = new_params.get(field_name, '')
+            old_value = old_params.get(field_name, '')
+            field_value = new_value if new_value else old_value
+            new_params[field_name] = field_value
+
+        return new_params
+
+    def update(self, instance, validated_data):
+        params = self.process_params(instance, validated_data)
+        validated_data['params'] = params
+        return super().update(instance, validated_data)
 
 
 class RemoteAppConnectionInfoSerializer(serializers.ModelSerializer):
     parameter_remote_app = serializers.SerializerMethodField()
-    asset = serializers.SerializerMethodField()
 
     class Meta:
-        model = Application
+        model = RemoteApp
         fields = [
             'id', 'name', 'asset', 'parameter_remote_app',
         ]
         read_only_fields = ['parameter_remote_app']
 
     @staticmethod
-    def get_asset(obj):
-        return obj.attrs.get('asset')
-
-    @staticmethod
-    def get_parameters(obj):
-        """
-        返回Guacamole需要的RemoteApp配置参数信息中的parameters参数
-        """
-        from .attrs import get_serializer_class_by_application_type
-        serializer_class = get_serializer_class_by_application_type(obj.type)
-        fields = serializer_class().get_fields()
-
-        parameters = [obj.type]
-        for field_name in list(fields.keys()):
-            if field_name in ['asset']:
-                continue
-            value = obj.attrs.get(field_name)
-            if not value:
-                continue
-            if field_name == 'path':
-                value = '\"%s\"' % value
-            parameters.append(str(value))
-
-        parameters = ' '.join(parameters)
-        return parameters
-
-    def get_parameter_remote_app(self, obj):
-        return {
-            'program': '||jmservisor',
+    def get_parameter_remote_app(obj):
+        parameter = {
+            'program': const.REMOTE_APP_BOOT_PROGRAM_NAME,
             'working_directory': '',
-            'parameters': self.get_parameters(obj)
+            'parameters': obj.parameters,
         }
+        return parameter

apps/applications/urls/api_urls.py

@@ -1,20 +1,24 @@
 # coding:utf-8
 #
-from django.urls import path
-from rest_framework_bulk.routes import BulkRouter
-from .. import api
 
+from django.urls import path, re_path
+from rest_framework_bulk.routes import BulkRouter
+
+from common import api as capi
+from .. import api
 
 app_name = 'applications'
 
-
 router = BulkRouter()
-router.register(r'applications', api.ApplicationViewSet, 'application')
-
+router.register(r'remote-apps', api.RemoteAppViewSet, 'remote-app')
+router.register(r'database-apps', api.DatabaseAppViewSet, 'database-app')
 
 urlpatterns = [
     path('remote-apps/<uuid:pk>/connection-info/', api.RemoteAppConnectionInfoApi.as_view(), name='remote-app-connection-info'),
 ]
 
+old_version_urlpatterns = [
+    re_path('(?P<resource>remote-app)/.*', capi.redirect_plural_name_api)
+]
 
-urlpatterns += router.urls
+urlpatterns += router.urls + old_version_urlpatterns

apps/applications/urls/views_urls.py

@@ -0,0 +1,7 @@
+# coding:utf-8
+from django.urls import path
+
+app_name = 'applications'
+
+urlpatterns = [
+]

apps/assets/api/__init__.py

@@ -1,4 +1,3 @@
-from .mixin import *
 from .admin_user import *
 from .asset import *
 from .label import *

apps/assets/api/admin_user.py

@@ -29,8 +29,8 @@ class AdminUserViewSet(OrgBulkModelViewSet):
     Admin user api set, for add,delete,update,list,retrieve resource
     """
     model = AdminUser
-    filterset_fields = ("name", "username")
-    search_fields = filterset_fields
+    filter_fields = ("name", "username")
+    search_fields = filter_fields
     serializer_class = serializers.AdminUserSerializer
     permission_classes = (IsOrgAdmin,)
 
@@ -93,8 +93,9 @@ class AdminUserTestConnectiveApi(generics.RetrieveAPIView):
 class AdminUserAssetsListView(generics.ListAPIView):
     permission_classes = (IsOrgAdmin,)
     serializer_class = serializers.AssetSimpleSerializer
-    filterset_fields = ("hostname", "ip")
-    search_fields = filterset_fields
+    filter_fields = ("hostname", "ip")
+    http_method_names = ['get']
+    search_fields = filter_fields
 
     def get_object(self):
         pk = self.kwargs.get('pk')

apps/assets/api/asset.py

@@ -1,10 +1,8 @@
 # -*- coding: utf-8 -*-
 #
-from assets.api import FilterAssetByNodeMixin
+
 from rest_framework.viewsets import ModelViewSet
 from rest_framework.generics import RetrieveAPIView
-from rest_framework.response import Response
-from rest_framework import status
 from django.shortcuts import get_object_or_404
 
 from common.utils import get_logger, get_object_or_none
@@ -14,33 +12,28 @@ from orgs.mixins import generics
 from ..models import Asset, Node, Platform
 from .. import serializers
 from ..tasks import (
-    update_assets_hardware_info_manual, test_assets_connectivity_manual
+    update_asset_hardware_info_manual, test_asset_connectivity_manual
 )
-from ..filters import FilterAssetByNodeFilterBackend, LabelFilterBackend, IpInFilterBackend
+from ..filters import AssetByNodeFilterBackend, LabelFilterBackend
 
 
 logger = get_logger(__file__)
 __all__ = [
     'AssetViewSet', 'AssetPlatformRetrieveApi',
     'AssetGatewayListApi', 'AssetPlatformViewSet',
-    'AssetTaskCreateApi', 'AssetsTaskCreateApi',
+    'AssetTaskCreateApi',
 ]
 
 
-class AssetViewSet(FilterAssetByNodeMixin, OrgBulkModelViewSet):
+class AssetViewSet(OrgBulkModelViewSet):
     """
     API endpoint that allows Asset to be viewed or edited.
     """
     model = Asset
-    filterset_fields = {
-        'hostname': ['exact'],
-        'ip': ['exact'],
-        'systemuser__id': ['exact'],
-        'admin_user__id': ['exact'],
-        'platform__base': ['exact'],
-        'is_active': ['exact'],
-        'protocols': ['exact', 'icontains']
-    }
+    filter_fields = (
+        "hostname", "ip", "systemuser__id", "admin_user__id", "platform__base",
+        "is_active"
+    )
     search_fields = ("hostname", "ip")
     ordering_fields = ("hostname", "ip", "port", "cpu_cores")
     serializer_classes = {
@@ -48,7 +41,7 @@ class AssetViewSet(FilterAssetByNodeMixin, OrgBulkModelViewSet):
         'display': serializers.AssetDisplaySerializer,
     }
     permission_classes = (IsOrgAdminOrAppUser,)
-    extra_filter_backends = [FilterAssetByNodeFilterBackend, LabelFilterBackend, IpInFilterBackend]
+    extra_filter_backends = [AssetByNodeFilterBackend, LabelFilterBackend]
 
     def set_assets_node(self, assets):
         if not isinstance(assets, list):
@@ -81,7 +74,7 @@ class AssetPlatformViewSet(ModelViewSet):
     queryset = Platform.objects.all()
     permission_classes = (IsSuperUser,)
     serializer_class = serializers.PlatformSerializer
-    filterset_fields = ['name', 'base']
+    filter_fields = ['name', 'base']
     search_fields = ['name']
 
     def get_permissions(self):
@@ -97,43 +90,32 @@ class AssetPlatformViewSet(ModelViewSet):
         return super().check_object_permissions(request, obj)
 
 
-class AssetsTaskMixin:
-    def perform_assets_task(self, serializer):
-        data = serializer.validated_data
-        assets = data['assets']
-        action = data['action']
+class AssetTaskCreateApi(generics.CreateAPIView):
+    model = Asset
+    serializer_class = serializers.AssetTaskSerializer
+    permission_classes = (IsOrgAdmin,)
+
+    def get_object(self):
+        pk = self.kwargs.get("pk")
+        instance = get_object_or_404(Asset, pk=pk)
+        return instance
+
+    def perform_create(self, serializer):
+        asset = self.get_object()
+        action = serializer.validated_data["action"]
         if action == "refresh":
-            task = update_assets_hardware_info_manual.delay(assets)
+            task = update_asset_hardware_info_manual.delay(asset)
         else:
-            task = test_assets_connectivity_manual.delay(assets)
+            task = test_asset_connectivity_manual.delay(asset)
         data = getattr(serializer, '_data', {})
         data["task"] = task.id
         setattr(serializer, '_data', data)
 
-    def perform_create(self, serializer):
-        self.perform_assets_task(serializer)
-
-
-class AssetTaskCreateApi(AssetsTaskMixin, generics.CreateAPIView):
-    model = Asset
-    serializer_class = serializers.AssetTaskSerializer
-    permission_classes = (IsOrgAdmin,)
-
-    def create(self, request, *args, **kwargs):
-        pk = self.kwargs.get('pk')
-        request.data['assets'] = [pk]
-        return super().create(request, *args, **kwargs)
-
-
-class AssetsTaskCreateApi(AssetsTaskMixin, generics.CreateAPIView):
-    model = Asset
-    serializer_class = serializers.AssetTaskSerializer
-    permission_classes = (IsOrgAdmin,)
-
 
 class AssetGatewayListApi(generics.ListAPIView):
     permission_classes = (IsOrgAdminOrAppUser,)
     serializer_class = serializers.GatewayWithAuthSerializer
+    model = Asset
 
     def get_queryset(self):
         asset_id = self.kwargs.get('pk')

apps/assets/api/asset_user.py

@@ -28,7 +28,7 @@ logger = get_logger(__name__)
 class AssetUserFilterBackend(filters.BaseFilterBackend):
     def filter_queryset(self, request, queryset, view):
         kwargs = {}
-        for field in view.filterset_fields:
+        for field in view.filter_fields:
             value = request.GET.get(field)
             if not value:
                 continue
@@ -78,7 +78,7 @@ class AssetUserViewSet(CommonApiMixin, BulkModelViewSet):
         'retrieve': serializers.AssetUserReadSerializer,
     }
     permission_classes = [IsOrgAdminOrAppUser]
-    filterset_fields = [
+    filter_fields = [
         "id", "ip", "hostname", "username",
         "asset_id", "node_id",
         "prefer", "prefer_id",
@@ -131,7 +131,7 @@ class AssetUserTaskCreateAPI(generics.CreateAPIView):
     permission_classes = (IsOrgAdminOrAppUser,)
     serializer_class = serializers.AssetUserTaskSerializer
     filter_backends = AssetUserViewSet.filter_backends
-    filterset_fields = AssetUserViewSet.filterset_fields
+    filter_fields = AssetUserViewSet.filter_fields
 
     def get_asset_users(self):
         manager = AssetUserManager()

apps/assets/api/cmd_filter.py

@@ -14,16 +14,16 @@ __all__ = ['CommandFilterViewSet', 'CommandFilterRuleViewSet']
 
 class CommandFilterViewSet(OrgBulkModelViewSet):
     model = CommandFilter
-    filterset_fields = ("name",)
-    search_fields = filterset_fields
+    filter_fields = ("name",)
+    search_fields = filter_fields
     permission_classes = (IsOrgAdmin,)
     serializer_class = serializers.CommandFilterSerializer
 
 
 class CommandFilterRuleViewSet(OrgBulkModelViewSet):
     model = CommandFilterRule
-    filterset_fields = ("content",)
-    search_fields = filterset_fields
+    filter_fields = ("content",)
+    search_fields = filter_fields
     permission_classes = (IsOrgAdmin,)
     serializer_class = serializers.CommandFilterRuleSerializer
 

apps/assets/api/domain.py

@@ -1,9 +1,7 @@
 # ~*~ coding: utf-8 ~*~
 
-from django.views.generic.detail import SingleObjectMixin
-from django.utils.translation import ugettext as _
 from rest_framework.views import APIView, Response
-from rest_framework.serializers import ValidationError
+from django.views.generic.detail import SingleObjectMixin
 
 from common.utils import get_logger
 from common.permissions import IsOrgAdmin, IsOrgAdminOrAppUser
@@ -18,8 +16,8 @@ __all__ = ['DomainViewSet', 'GatewayViewSet', "GatewayTestConnectionApi"]
 
 class DomainViewSet(OrgBulkModelViewSet):
     model = Domain
-    filterset_fields = ("name", )
-    search_fields = filterset_fields
+    filter_fields = ("name", )
+    search_fields = filter_fields
     permission_classes = (IsOrgAdminOrAppUser,)
     serializer_class = serializers.DomainSerializer
 
@@ -31,7 +29,7 @@ class DomainViewSet(OrgBulkModelViewSet):
 
 class GatewayViewSet(OrgBulkModelViewSet):
     model = Gateway
-    filterset_fields = ("domain__name", "name", "username", "ip", "domain")
+    filter_fields = ("domain__name", "name", "username", "ip", "domain")
     search_fields = ("domain__name", "name", "username", "ip")
     permission_classes = (IsOrgAdmin,)
     serializer_class = serializers.GatewaySerializer
@@ -44,10 +42,6 @@ class GatewayTestConnectionApi(SingleObjectMixin, APIView):
     def post(self, request, *args, **kwargs):
         self.object = self.get_object(Gateway.objects.all())
         local_port = self.request.data.get('port') or self.object.port
-        try:
-            local_port = int(local_port)
-        except ValueError:
-            raise ValidationError({'port': _('Number required')})
         ok, e = self.object.test_connective(local_port=local_port)
         if ok:
             return Response("ok")

apps/assets/api/favorite_asset.py

@@ -13,7 +13,7 @@ __all__ = ['FavoriteAssetViewSet']
 class FavoriteAssetViewSet(BulkModelViewSet):
     serializer_class = FavoriteAssetSerializer
     permission_classes = (IsValidUser,)
-    filterset_fields = ['asset']
+    filter_fields = ['asset']
 
     def dispatch(self, request, *args, **kwargs):
         with tmp_to_root_org():

apps/assets/api/gathered_user.py

@@ -18,5 +18,5 @@ class GatheredUserViewSet(OrgModelViewSet):
     permission_classes = [IsOrgAdmin]
     extra_filter_backends = [AssetRelatedByNodeFilterBackend]
 
-    filterset_fields = ['asset', 'username', 'present', 'asset__ip', 'asset__hostname', 'asset_id']
+    filter_fields = ['asset', 'username', 'present', 'asset__ip', 'asset__hostname', 'asset_id']
     search_fields = ['username', 'asset__ip', 'asset__hostname']

apps/assets/api/label.py

@@ -28,8 +28,8 @@ __all__ = ['LabelViewSet']
 
 class LabelViewSet(OrgBulkModelViewSet):
     model = Label
-    filterset_fields = ("name", "value")
-    search_fields = filterset_fields
+    filter_fields = ("name", "value")
+    search_fields = filter_fields
     permission_classes = (IsOrgAdmin,)
     serializer_class = serializers.LabelSerializer
 

apps/assets/api/mixin.py

@@ -1,90 +0,0 @@
-from typing import List
-
-from assets.models import Node, Asset
-from assets.pagination import AssetLimitOffsetPagination
-from common.utils import lazyproperty, dict_get_any, is_uuid, get_object_or_none
-from assets.utils import get_node, is_query_node_all_assets
-
-
-class SerializeToTreeNodeMixin:
-    permission_classes = ()
-
-    def serialize_nodes(self, nodes: List[Node], with_asset_amount=False):
-        if with_asset_amount:
-            def _name(node: Node):
-                return '{} ({})'.format(node.value, node.assets_amount)
-        else:
-            def _name(node: Node):
-                return node.value
-        data = [
-            {
-                'id': node.key,
-                'name': _name(node),
-                'title': _name(node),
-                'pId': node.parent_key,
-                'isParent': True,
-                'open': node.is_org_root(),
-                'meta': {
-                    'node': {
-                        "id": node.id,
-                        "key": node.key,
-                        "value": node.value,
-                    },
-                    'type': 'node'
-                }
-            }
-            for node in nodes
-        ]
-        return data
-
-    def get_platform(self, asset: Asset):
-        default = 'file'
-        icon = {'windows', 'linux'}
-        platform = asset.platform_base.lower()
-        if platform in icon:
-            return platform
-        return default
-
-    def serialize_assets(self, assets, node_key=None):
-        if node_key is None:
-            get_pid = lambda asset: getattr(asset, 'parent_key', '')
-        else:
-            get_pid = lambda asset: node_key
-
-        data = [
-            {
-                'id': str(asset.id),
-                'name': asset.hostname,
-                'title': asset.ip,
-                'pId': get_pid(asset),
-                'isParent': False,
-                'open': False,
-                'iconSkin': self.get_platform(asset),
-                'chkDisabled': not asset.is_active,
-                'meta': {
-                    'type': 'asset',
-                    'asset': {
-                        'id': asset.id,
-                        'hostname': asset.hostname,
-                        'ip': asset.ip,
-                        'protocols': asset.protocols_as_list,
-                        'platform': asset.platform_base,
-                        'org_name': asset.org_name
-                    },
-                }
-            }
-            for asset in assets
-        ]
-        return data
-
-
-class FilterAssetByNodeMixin:
-    pagination_class = AssetLimitOffsetPagination
-
-    @lazyproperty
-    def is_query_node_all_assets(self):
-        return is_query_node_all_assets(self.request)
-
-    @lazyproperty
-    def node(self):
-        return get_node(self.request)

apps/assets/api/node.py

@@ -1,28 +1,16 @@
 # ~*~ coding: utf-8 ~*~
-from functools import partial
-from collections import namedtuple, defaultdict
 
+from collections import namedtuple
 from rest_framework import status
 from rest_framework.serializers import ValidationError
 from rest_framework.response import Response
-from rest_framework.decorators import action
 from django.utils.translation import ugettext_lazy as _
 from django.shortcuts import get_object_or_404, Http404
-from django.utils.decorators import method_decorator
-from django.db.models.signals import m2m_changed
 
-from common.const.http import POST
-from common.exceptions import SomeoneIsDoingThis
-from common.const.signals import PRE_REMOVE, POST_REMOVE
-from assets.models import Asset
 from common.utils import get_logger, get_object_or_none
 from common.tree import TreeNodeSerializer
-from common.const.distributed_lock_key import UPDATE_NODE_TREE_LOCK_KEY
 from orgs.mixins.api import OrgModelViewSet
 from orgs.mixins import generics
-from orgs.lock import org_level_transaction_lock
-from orgs.utils import current_org
-from assets.tasks import check_node_assets_amount_task
 from ..hands import IsOrgAdmin
 from ..models import Node
 from ..tasks import (
@@ -30,13 +18,12 @@ from ..tasks import (
     test_node_assets_connectivity_manual,
 )
 from .. import serializers
-from .mixin import SerializeToTreeNodeMixin
 
 
 logger = get_logger(__file__)
 __all__ = [
     'NodeViewSet', 'NodeChildrenApi', 'NodeAssetsApi',
-    'NodeAddAssetsApi', 'NodeRemoveAssetsApi', 'MoveAssetsToNodeApi',
+    'NodeAddAssetsApi', 'NodeRemoveAssetsApi', 'NodeReplaceAssetsApi',
     'NodeAddChildrenApi', 'NodeListAsTreeApi',
     'NodeChildrenAsTreeApi',
     'NodeTaskCreateApi',
@@ -45,16 +32,11 @@ __all__ = [
 
 class NodeViewSet(OrgModelViewSet):
     model = Node
-    filterset_fields = ('value', 'key', 'id')
+    filter_fields = ('value', 'key', 'id')
     search_fields = ('value', )
     permission_classes = (IsOrgAdmin,)
     serializer_class = serializers.NodeSerializer
 
-    @action(methods=[POST], detail=False, url_name='launch-check-assets-amount-task')
-    def launch_check_assets_amount_task(self, request):
-        task = check_node_assets_amount_task.delay(current_org.id)
-        return Response(data={'task': task.id})
-
     # 仅支持根节点指直接创建，子节点下的节点需要通过children接口创建
     def perform_create(self, serializer):
         child_key = Node.org_root().get_next_child_key()
@@ -70,9 +52,6 @@ class NodeViewSet(OrgModelViewSet):
 
     def destroy(self, request, *args, **kwargs):
         node = self.get_object()
-        if node.is_org_root():
-            error = _("You can't delete the root node ({})".format(node.value))
-            return Response(data={'error': error}, status=status.HTTP_403_FORBIDDEN)
         if node.has_children_or_has_assets():
             error = _("Deletion failed and the node contains children or assets")
             return Response(data={'error': error}, status=status.HTTP_403_FORBIDDEN)
@@ -157,7 +136,7 @@ class NodeChildrenApi(generics.ListCreateAPIView):
         return queryset
 
 
-class NodeChildrenAsTreeApi(SerializeToTreeNodeMixin, NodeChildrenApi):
+class NodeChildrenAsTreeApi(NodeChildrenApi):
     """
     节点子节点作为树返回，
     [
@@ -171,23 +150,31 @@ class NodeChildrenAsTreeApi(SerializeToTreeNodeMixin, NodeChildrenApi):
 
     """
     model = Node
+    serializer_class = TreeNodeSerializer
+    http_method_names = ['get']
 
-    def list(self, request, *args, **kwargs):
-        nodes = self.get_queryset().order_by('value')
-        nodes = self.serialize_nodes(nodes, with_asset_amount=True)
-        assets = self.get_assets()
-        data = [*nodes, *assets]
-        return Response(data=data)
+    def get_queryset(self):
+        queryset = super().get_queryset()
+        queryset = [node.as_tree_node() for node in queryset]
+        queryset = self.add_assets_if_need(queryset)
+        queryset = sorted(queryset)
+        return queryset
 
-    def get_assets(self):
+    def add_assets_if_need(self, queryset):
         include_assets = self.request.query_params.get('assets', '0') == '1'
         if not include_assets:
-            return []
+            return queryset
         assets = self.instance.get_assets().only(
             "id", "hostname", "ip", "os",
-            "org_id", "protocols", "is_active"
+            "org_id", "protocols",
         )
-        return self.serialize_assets(assets, self.instance.key)
+        for asset in assets:
+            queryset.append(asset.as_tree_node(self.instance))
+        return queryset
+
+    def check_need_refresh_nodes(self):
+        if self.request.query_params.get('refresh', '0') == '1':
+            Node.refresh_nodes()
 
 
 class NodeAssetsApi(generics.ListAPIView):
@@ -213,14 +200,14 @@ class NodeAddChildrenApi(generics.UpdateAPIView):
     def put(self, request, *args, **kwargs):
         instance = self.get_object()
         nodes_id = request.data.get("nodes")
-        children = Node.objects.filter(id__in=nodes_id)
+        children = [get_object_or_none(Node, id=pk) for pk in nodes_id]
         for node in children:
+            if not node:
+                continue
             node.parent = instance
         return Response("OK")
 
 
-@method_decorator(org_level_transaction_lock(UPDATE_NODE_TREE_LOCK_KEY), name='patch')
-@method_decorator(org_level_transaction_lock(UPDATE_NODE_TREE_LOCK_KEY), name='put')
 class NodeAddAssetsApi(generics.UpdateAPIView):
     model = Node
     serializer_class = serializers.NodeAssetsSerializer
@@ -233,8 +220,6 @@ class NodeAddAssetsApi(generics.UpdateAPIView):
         instance.assets.add(*tuple(assets))
 
 
-@method_decorator(org_level_transaction_lock(UPDATE_NODE_TREE_LOCK_KEY), name='patch')
-@method_decorator(org_level_transaction_lock(UPDATE_NODE_TREE_LOCK_KEY), name='put')
 class NodeRemoveAssetsApi(generics.UpdateAPIView):
     model = Node
     serializer_class = serializers.NodeAssetsSerializer
@@ -243,17 +228,15 @@ class NodeRemoveAssetsApi(generics.UpdateAPIView):
 
     def perform_update(self, serializer):
         assets = serializer.validated_data.get('assets')
-        node = self.get_object()
-        node.assets.remove(*assets)
-
-        # 把孤儿资产添加到 root 节点
-        orphan_assets = Asset.objects.filter(id__in=[a.id for a in assets], nodes__isnull=True).distinct()
-        Node.org_root().assets.add(*orphan_assets)
+        instance = self.get_object()
+        if instance != Node.org_root():
+            instance.assets.remove(*tuple(assets))
+        else:
+            assets = [asset for asset in assets if asset.nodes.count() > 1]
+            instance.assets.remove(*tuple(assets))
 
 
-@method_decorator(org_level_transaction_lock(UPDATE_NODE_TREE_LOCK_KEY), name='patch')
-@method_decorator(org_level_transaction_lock(UPDATE_NODE_TREE_LOCK_KEY), name='put')
-class MoveAssetsToNodeApi(generics.UpdateAPIView):
+class NodeReplaceAssetsApi(generics.UpdateAPIView):
     model = Node
     serializer_class = serializers.NodeAssetsSerializer
     permission_classes = (IsOrgAdmin,)
@@ -261,39 +244,9 @@ class MoveAssetsToNodeApi(generics.UpdateAPIView):
 
     def perform_update(self, serializer):
         assets = serializer.validated_data.get('assets')
-        node = self.get_object()
-        self.remove_old_nodes(assets)
-        node.assets.add(*assets)
-
-    def remove_old_nodes(self, assets):
-        m2m_model = Asset.nodes.through
-
-        # 查询资产与节点关系表，查出要移动资产与节点的所有关系
-        relates = m2m_model.objects.filter(asset__in=assets).values_list('asset_id', 'node_id')
-        if relates:
-            # 对关系以资产进行分组，用来发 `reverse=False` 信号
-            asset_nodes_mapper = defaultdict(set)
-            for asset_id, node_id in relates:
-                asset_nodes_mapper[asset_id].add(node_id)
-
-            # 组建一个资产 id -> Asset 的 mapper
-            asset_mapper = {asset.id: asset for asset in assets}
-
-            # 创建删除关系信号发送函数
-            senders = []
-            for asset_id, node_id_set in asset_nodes_mapper.items():
-                senders.append(partial(m2m_changed.send, sender=m2m_model, instance=asset_mapper[asset_id],
-                                       reverse=False, model=Node, pk_set=node_id_set))
-            # 发送 pre 信号
-            [sender(action=PRE_REMOVE) for sender in senders]
-            num = len(relates)
-            asset_ids, node_ids = zip(*relates)
-            # 删除之前的关系
-            rows, _i = m2m_model.objects.filter(asset_id__in=asset_ids, node_id__in=node_ids).delete()
-            if rows != num:
-                raise SomeoneIsDoingThis
-            # 发送 post 信号
-            [sender(action=POST_REMOVE) for sender in senders]
+        instance = self.get_object()
+        for asset in assets:
+            asset.nodes.set([instance])
 
 
 class NodeTaskCreateApi(generics.CreateAPIView):
@@ -314,6 +267,7 @@ class NodeTaskCreateApi(generics.CreateAPIView):
 
     @staticmethod
     def refresh_nodes_cache():
+        Node.refresh_nodes()
         Task = namedtuple('Task', ['id'])
         task = Task(id="0")
         return task

apps/assets/api/system_user.py

@@ -3,8 +3,7 @@ from django.shortcuts import get_object_or_404
 from rest_framework.response import Response
 
 from common.utils import get_logger
-from common.permissions import IsOrgAdmin, IsOrgAdminOrAppUser
-from common.drf.filters import CustomFilter
+from common.permissions import IsOrgAdmin, IsOrgAdminOrAppUser, IsAppUser
 from orgs.mixins.api import OrgBulkModelViewSet
 from orgs.mixins import generics
 from orgs.utils import tmp_to_org
@@ -13,14 +12,14 @@ from .. import serializers
 from ..serializers import SystemUserWithAuthInfoSerializer
 from ..tasks import (
     push_system_user_to_assets_manual, test_system_user_connectivity_manual,
-    push_system_user_to_assets
+    push_system_user_a_asset_manual,
 )
 
 
 logger = get_logger(__file__)
 __all__ = [
     'SystemUserViewSet', 'SystemUserAuthInfoApi', 'SystemUserAssetAuthInfoApi',
-    'SystemUserCommandFilterRuleListApi', 'SystemUserTaskApi', 'SystemUserAssetsListView',
+    'SystemUserCommandFilterRuleListApi', 'SystemUserTaskApi',
 ]
 
 
@@ -29,12 +28,8 @@ class SystemUserViewSet(OrgBulkModelViewSet):
     System user api set, for add,delete,update,list,retrieve resource
     """
     model = SystemUser
-    filterset_fields = {
-        'name': ['exact'],
-        'username': ['exact'],
-        'protocol': ['exact', 'in']
-    }
-    search_fields = filterset_fields
+    filter_fields = ("name", "username", "protocol")
+    search_fields = filter_fields
     serializer_class = serializers.SystemUserSerializer
     serializer_classes = {
         'default': serializers.SystemUserSerializer,
@@ -87,18 +82,18 @@ class SystemUserTaskApi(generics.CreateAPIView):
     permission_classes = (IsOrgAdmin,)
     serializer_class = serializers.SystemUserTaskSerializer
 
-    def do_push(self, system_user, assets_id=None):
-        if assets_id is None:
+    def do_push(self, system_user, asset=None):
+        if asset is None:
             task = push_system_user_to_assets_manual.delay(system_user)
         else:
             username = self.request.query_params.get('username')
-            task = push_system_user_to_assets.delay(
-                system_user.id, assets_id, username=username
+            task = push_system_user_a_asset_manual.delay(
+                system_user, asset, username=username
             )
         return task
 
     @staticmethod
-    def do_test(system_user):
+    def do_test(system_user, asset=None):
         task = test_system_user_connectivity_manual.delay(system_user)
         return task
 
@@ -109,16 +104,11 @@ class SystemUserTaskApi(generics.CreateAPIView):
     def perform_create(self, serializer):
         action = serializer.validated_data["action"]
         asset = serializer.validated_data.get('asset')
-        assets = serializer.validated_data.get('assets') or []
-
         system_user = self.get_object()
         if action == 'push':
-            assets = [asset] if asset else assets
-            assets_id = [asset.id for asset in assets]
-            assets_id = assets_id if assets_id else None
-            task = self.do_push(system_user, assets_id)
+            task = self.do_push(system_user, asset)
         else:
-            task = self.do_test(system_user)
+            task = self.do_test(system_user, asset)
         data = getattr(serializer, '_data', {})
         data["task"] = task.id
         setattr(serializer, '_data', data)
@@ -135,18 +125,3 @@ class SystemUserCommandFilterRuleListApi(generics.ListAPIView):
         pk = self.kwargs.get('pk', None)
         system_user = get_object_or_404(SystemUser, pk=pk)
         return system_user.cmd_filter_rules
-
-
-class SystemUserAssetsListView(generics.ListAPIView):
-    permission_classes = (IsOrgAdmin,)
-    serializer_class = serializers.AssetSimpleSerializer
-    filterset_fields = ("hostname", "ip")
-    search_fields = filterset_fields
-
-    def get_object(self):
-        pk = self.kwargs.get('pk')
-        return get_object_or_404(SystemUser, pk=pk)
-
-    def get_queryset(self):
-        system_user = self.get_object()
-        return system_user.get_all_assets()

apps/assets/api/system_user_relation.py

@@ -65,7 +65,7 @@ class SystemUserAssetRelationViewSet(BaseRelationViewSet):
     serializer_class = serializers.SystemUserAssetRelationSerializer
     model = models.SystemUser.assets.through
     permission_classes = (IsOrgAdmin,)
-    filterset_fields = [
+    filter_fields = [
         'id', 'asset', 'systemuser',
     ]
     search_fields = [
@@ -91,11 +91,11 @@ class SystemUserNodeRelationViewSet(BaseRelationViewSet):
     serializer_class = serializers.SystemUserNodeRelationSerializer
     model = models.SystemUser.nodes.through
     permission_classes = (IsOrgAdmin,)
-    filterset_fields = [
+    filter_fields = [
         'id', 'node', 'systemuser',
     ]
     search_fields = [
-        "node__value", "systemuser__name", "systemuser__username"
+        "node__value", "systemuser__name", "systemuser_username"
     ]
 
     def get_objects_attr(self):
@@ -112,7 +112,7 @@ class SystemUserUserRelationViewSet(BaseRelationViewSet):
     serializer_class = serializers.SystemUserUserRelationSerializer
     model = models.SystemUser.users.through
     permission_classes = (IsOrgAdmin,)
-    filterset_fields = [
+    filter_fields = [
         'id', 'user', 'systemuser',
     ]
     search_fields = [

apps/assets/exceptions.py

@@ -1,6 +0,0 @@
-from rest_framework import status
-from common.exceptions import JMSException
-
-
-class NodeIsBeingUpdatedByOthers(JMSException):
-    status_code = status.HTTP_409_CONFLICT

apps/assets/filters.py

@@ -1,12 +1,12 @@
 # -*- coding: utf-8 -*-
 #
 
-from rest_framework.compat import coreapi, coreschema
+import coreapi
 from rest_framework import filters
 from django.db.models import Q
 
-from .models import Label
-from assets.utils import is_query_node_all_assets, get_node
+from common.utils import dict_get_any, is_uuid, get_object_or_none
+from .models import Node, Label
 
 
 class AssetByNodeFilterBackend(filters.BaseFilterBackend):
@@ -21,54 +21,47 @@ class AssetByNodeFilterBackend(filters.BaseFilterBackend):
             for field in self.fields
         ]
 
-    def filter_node_related_all(self, queryset, node):
-        return queryset.filter(
-            Q(nodes__key__istartswith=f'{node.key}:') |
-            Q(nodes__key=node.key)
-        ).distinct()
+    @staticmethod
+    def is_query_all(request):
+        query_all_arg = request.query_params.get('all')
+        show_current_asset_arg = request.query_params.get('show_current_asset')
 
-    def filter_node_related_direct(self, queryset, node):
-        return queryset.filter(nodes__key=node.key).distinct()
+        query_all = query_all_arg == '1'
+        if show_current_asset_arg is not None:
+            query_all = show_current_asset_arg != '1'
+        return query_all
+
+    @staticmethod
+    def get_query_node(request):
+        node_id = dict_get_any(request.query_params, ['node', 'node_id'])
+        if not node_id:
+            return None, False
+
+        if is_uuid(node_id):
+            node = get_object_or_none(Node, id=node_id)
+        else:
+            node = get_object_or_none(Node, key=node_id)
+        return node, True
+
+    @staticmethod
+    def perform_query(pattern, queryset):
+        return queryset.filter(nodes__key__regex=pattern).distinct()
 
     def filter_queryset(self, request, queryset, view):
-        node = get_node(request)
-        if node is None:
+        node, has_query_arg = self.get_query_node(request)
+        if not has_query_arg:
             return queryset
 
-        query_all = is_query_node_all_assets(request)
-        if query_all:
-            return self.filter_node_related_all(queryset, node)
-        else:
-            return self.filter_node_related_direct(queryset, node)
-
-
-class FilterAssetByNodeFilterBackend(filters.BaseFilterBackend):
-    """
-    需要与 `assets.api.mixin.FilterAssetByNodeMixin` 配合使用
-    """
-    fields = ['node', 'all']
-
-    def get_schema_fields(self, view):
-        return [
-            coreapi.Field(
-                name=field, location='query', required=False,
-                type='string', example='', description='', schema=None,
-            )
-            for field in self.fields
-        ]
-
-    def filter_queryset(self, request, queryset, view):
-        node = view.node
         if node is None:
             return queryset
-        query_all = view.is_query_node_all_assets
+        query_all = self.is_query_all(request)
         if query_all:
-            return queryset.filter(
-                Q(nodes__key__istartswith=f'{node.key}:') |
-                Q(nodes__key=node.key)
-            ).distinct()
+            pattern = node.get_all_children_pattern(with_self=True)
         else:
-            return queryset.filter(nodes__key=node.key).distinct()
+            # pattern = node.get_children_key_pattern(with_self=True)
+            # 只显示当前节点下资产
+            pattern = r"^{}$".format(node.key)
+        return self.perform_query(pattern, queryset)
 
 
 class LabelFilterBackend(filters.BaseFilterBackend):
@@ -120,32 +113,7 @@ class LabelFilterBackend(filters.BaseFilterBackend):
 
 
 class AssetRelatedByNodeFilterBackend(AssetByNodeFilterBackend):
-    def filter_node_related_all(self, queryset, node):
-        return queryset.filter(
-            Q(asset__nodes__key__istartswith=f'{node.key}:') |
-            Q(asset__nodes__key=node.key)
-        ).distinct()
+    @staticmethod
+    def perform_query(pattern, queryset):
+        return queryset.filter(asset__nodes__key__regex=pattern).distinct()
 
-    def filter_node_related_direct(self, queryset, node):
-        return queryset.filter(asset__nodes__key=node.key).distinct()
-
-
-class IpInFilterBackend(filters.BaseFilterBackend):
-    def filter_queryset(self, request, queryset, view):
-        ips = request.query_params.get('ips')
-        if not ips:
-            return queryset
-        ip_list = [i.strip() for i in ips.split(',')]
-        queryset = queryset.filter(ip__in=ip_list)
-        return queryset
-
-    def get_schema_fields(self, view):
-        return [
-            coreapi.Field(
-                name='ips', location='query', required=False, type='string',
-                schema=coreschema.String(
-                    title='ips',
-                    description='ip in filter'
-                )
-            )
-        ]

apps/assets/migrations/0050_auto_20200711_1740.py

@@ -1,18 +0,0 @@
-# Generated by Django 2.2.10 on 2020-07-11 09:40
-
-from django.db import migrations, models
-
-
-class Migration(migrations.Migration):
-
-    dependencies = [
-        ('assets', '0049_systemuser_sftp_root'),
-    ]
-
-    operations = [
-        migrations.AlterField(
-            model_name='asset',
-            name='created_by',
-            field=models.CharField(blank=True, max_length=128, null=True, verbose_name='Created by'),
-        ),
-    ]

apps/assets/migrations/0051_auto_20200713_1143.py

@@ -1,22 +0,0 @@
-# Generated by Django 2.2.10 on 2020-07-13 03:43
-
-from django.db import migrations, models
-
-
-class Migration(migrations.Migration):
-
-    dependencies = [
-        ('assets', '0050_auto_20200711_1740'),
-    ]
-
-    operations = [
-        migrations.AlterField(
-            model_name='domain',
-            name='name',
-            field=models.CharField(max_length=128, verbose_name='Name'),
-        ),
-        migrations.AlterUniqueTogether(
-            name='domain',
-            unique_together={('org_id', 'name')},
-        ),
-    ]

apps/assets/migrations/0052_auto_20200715_1535.py

@@ -1,22 +0,0 @@
-# Generated by Django 2.2.10 on 2020-07-15 07:35
-
-from django.db import migrations, models
-
-
-class Migration(migrations.Migration):
-
-    dependencies = [
-        ('assets', '0051_auto_20200713_1143'),
-    ]
-
-    operations = [
-        migrations.AlterField(
-            model_name='commandfilter',
-            name='name',
-            field=models.CharField(max_length=64, verbose_name='Name'),
-        ),
-        migrations.AlterUniqueTogether(
-            name='commandfilter',
-            unique_together={('org_id', 'name')},
-        ),
-    ]

apps/assets/migrations/0053_auto_20200723_1232.py

@@ -1,34 +0,0 @@
-# Generated by Django 2.2.10 on 2020-07-23 04:32
-
-import django.core.validators
-from django.db import migrations, models
-
-
-class Migration(migrations.Migration):
-
-    dependencies = [
-        ('assets', '0052_auto_20200715_1535'),
-    ]
-
-    operations = [
-        migrations.AlterField(
-            model_name='adminuser',
-            name='username',
-            field=models.CharField(blank=True, db_index=True, max_length=128, validators=[django.core.validators.RegexValidator('^[0-9a-zA-Z_@\\-\\.]*$', 'Special char not allowed')], verbose_name='Username'),
-        ),
-        migrations.AlterField(
-            model_name='authbook',
-            name='username',
-            field=models.CharField(blank=True, db_index=True, max_length=128, validators=[django.core.validators.RegexValidator('^[0-9a-zA-Z_@\\-\\.]*$', 'Special char not allowed')], verbose_name='Username'),
-        ),
-        migrations.AlterField(
-            model_name='gateway',
-            name='username',
-            field=models.CharField(blank=True, db_index=True, max_length=128, validators=[django.core.validators.RegexValidator('^[0-9a-zA-Z_@\\-\\.]*$', 'Special char not allowed')], verbose_name='Username'),
-        ),
-        migrations.AlterField(
-            model_name='systemuser',
-            name='username',
-            field=models.CharField(blank=True, db_index=True, max_length=128, validators=[django.core.validators.RegexValidator('^[0-9a-zA-Z_@\\-\\.]*$', 'Special char not allowed')], verbose_name='Username'),
-        ),
-    ]

apps/assets/migrations/0054_auto_20200807_1032.py

@@ -1,23 +0,0 @@
-# Generated by Django 2.2.13 on 2020-08-07 02:32
-
-from django.db import migrations, models
-
-
-class Migration(migrations.Migration):
-
-    dependencies = [
-        ('assets', '0053_auto_20200723_1232'),
-    ]
-
-    operations = [
-        migrations.AddField(
-            model_name='systemuser',
-            name='token',
-            field=models.TextField(default='', verbose_name='Token'),
-        ),
-        migrations.AlterField(
-            model_name='systemuser',
-            name='protocol',
-            field=models.CharField(choices=[('ssh', 'ssh'), ('rdp', 'rdp'), ('telnet', 'telnet'), ('vnc', 'vnc'), ('mysql', 'mysql'), ('k8s', 'k8s')], default='ssh', max_length=16, verbose_name='Protocol'),
-        ),
-    ]

apps/assets/migrations/0055_auto_20200811_1845.py

@@ -1,23 +0,0 @@
-# Generated by Django 2.2.13 on 2020-08-11 10:45
-
-from django.db import migrations, models
-
-
-class Migration(migrations.Migration):
-
-    dependencies = [
-        ('assets', '0054_auto_20200807_1032'),
-    ]
-
-    operations = [
-        migrations.AddField(
-            model_name='systemuser',
-            name='home',
-            field=models.CharField(blank=True, default='', max_length=4096, verbose_name='Home'),
-        ),
-        migrations.AddField(
-            model_name='systemuser',
-            name='system_groups',
-            field=models.CharField(blank=True, default='', max_length=4096, verbose_name='System groups'),
-        ),
-    ]

apps/assets/migrations/0056_auto_20200904_1751.py

@@ -1,23 +0,0 @@
-# Generated by Django 2.2.13 on 2020-09-04 09:51
-
-from django.db import migrations, models
-
-
-class Migration(migrations.Migration):
-
-    dependencies = [
-        ('assets', '0055_auto_20200811_1845'),
-    ]
-
-    operations = [
-        migrations.AddField(
-            model_name='node',
-            name='assets_amount',
-            field=models.IntegerField(default=0),
-        ),
-        migrations.AddField(
-            model_name='node',
-            name='parent_key',
-            field=models.CharField(db_index=True, default='', max_length=64, verbose_name='Parent key'),
-        ),
-    ]

apps/assets/migrations/0057_fill_node_value_assets_amount_and_parent_key.py

@@ -1,38 +0,0 @@
-# Generated by Django 2.2.13 on 2020-08-21 08:20
-
-from django.db import migrations
-from django.db.models import Q
-
-
-def fill_node_value(apps, schema_editor):
-    Node = apps.get_model('assets', 'Node')
-    Asset = apps.get_model('assets', 'Asset')
-    node_queryset = Node.objects.all()
-    node_amount = node_queryset.count()
-    width = len(str(node_amount))
-    print('\n')
-    for i, node in enumerate(node_queryset):
-        print(f'\t{i+1:0>{width}}/{node_amount} compute node[{node.key}]`s assets_amount ...')
-        assets_amount = Asset.objects.filter(
-            Q(nodes__key__istartswith=f'{node.key}:') | Q(nodes=node)
-        ).distinct().count()
-        key = node.key
-        try:
-            parent_key = key[:key.rindex(':')]
-        except ValueError:
-            parent_key = ''
-        node.assets_amount = assets_amount
-        node.parent_key = parent_key
-        node.save()
-    print('  ' + '.'*65, end='')
-
-
-class Migration(migrations.Migration):
-
-    dependencies = [
-        ('assets', '0056_auto_20200904_1751'),
-    ]
-
-    operations = [
-        migrations.RunPython(fill_node_value)
-    ]

apps/assets/migrations/0058_auto_20201023_1115.py

@@ -1,27 +0,0 @@
-# Generated by Django 2.2.13 on 2020-10-23 03:15
-
-from django.db import migrations, models
-
-
-class Migration(migrations.Migration):
-
-    dependencies = [
-        ('assets', '0057_fill_node_value_assets_amount_and_parent_key'),
-    ]
-
-    operations = [
-        migrations.AlterModelOptions(
-            name='asset',
-            options={'ordering': ['-date_created'], 'verbose_name': 'Asset'},
-        ),
-        migrations.AlterField(
-            model_name='asset',
-            name='comment',
-            field=models.TextField(blank=True, default='', verbose_name='Comment'),
-        ),
-        migrations.AlterField(
-            model_name='commandfilterrule',
-            name='content',
-            field=models.TextField(help_text='One line one command', verbose_name='Content'),
-        ),
-    ]

apps/assets/migrations/0059_auto_20201027_1905.py

@@ -1,28 +0,0 @@
-# Generated by Django 2.2.13 on 2020-10-27 11:05
-
-from django.db import migrations, models
-
-
-class Migration(migrations.Migration):
-
-    dependencies = [
-        ('assets', '0058_auto_20201023_1115'),
-    ]
-
-    operations = [
-        migrations.AlterField(
-            model_name='systemuser',
-            name='protocol',
-            field=models.CharField(choices=[('ssh', 'ssh'), ('rdp', 'rdp'), ('telnet', 'telnet'), ('vnc', 'vnc'), ('mysql', 'mysql'), ('oracle', 'oracle'), ('mariadb', 'mariadb'), ('postgresql', 'postgresql'), ('k8s', 'k8s')], default='ssh', max_length=16, verbose_name='Protocol'),
-        ),
-        migrations.AddField(
-            model_name='systemuser',
-            name='ad_domain',
-            field=models.CharField(default='', max_length=256),
-        ),
-        migrations.AlterField(
-            model_name='gateway',
-            name='ip',
-            field=models.CharField(db_index=True, max_length=128, verbose_name='IP'),
-        ),
-    ]

apps/assets/migrations/0060_node_full_value.py

@@ -1,58 +0,0 @@
-# Generated by Django 2.2.13 on 2020-10-26 11:31
-
-from django.db import migrations, models
-
-
-def get_node_ancestor_keys(key, with_self=False):
-    parent_keys = []
-    key_list = key.split(":")
-    if not with_self:
-        key_list.pop()
-    for i in range(len(key_list)):
-        parent_keys.append(":".join(key_list))
-        key_list.pop()
-    return parent_keys
-
-
-def migrate_nodes_value_with_slash(apps, schema_editor):
-    model = apps.get_model("assets", "Node")
-    db_alias = schema_editor.connection.alias
-    nodes = model.objects.using(db_alias).filter(value__contains='/')
-    print('')
-    print("- Start migrate node value if has /")
-    for i, node in enumerate(list(nodes)):
-        new_value = node.value.replace('/', '|')
-        print("{} start migrate node value: {} => {}".format(i, node.value, new_value))
-        node.value = new_value
-        node.save()
-
-
-def migrate_nodes_full_value(apps, schema_editor):
-    model = apps.get_model("assets", "Node")
-    db_alias = schema_editor.connection.alias
-    nodes = model.objects.using(db_alias).all()
-    print("- Start migrate node full value")
-    for i, node in enumerate(list(nodes)):
-        print("{} start migrate {} node full value".format(i, node.value))
-        ancestor_keys = get_node_ancestor_keys(node.key, True)
-        values = model.objects.filter(key__in=ancestor_keys).values_list('key', 'value')
-        values = [v for k, v in sorted(values, key=lambda x: len(x[0]))]
-        node.full_value = '/' + '/'.join(values)
-        node.save()
-
-
-class Migration(migrations.Migration):
-
-    dependencies = [
-        ('assets', '0059_auto_20201027_1905'),
-    ]
-
-    operations = [
-        migrations.AddField(
-            model_name='node',
-            name='full_value',
-            field=models.CharField(default='', max_length=4096, verbose_name='Full value'),
-        ),
-        migrations.RunPython(migrate_nodes_value_with_slash),
-        migrations.RunPython(migrate_nodes_full_value)
-    ]

apps/assets/migrations/0061_auto_20201116_1757.py

@@ -1,17 +0,0 @@
-# Generated by Django 2.2.13 on 2020-11-16 09:57
-
-from django.db import migrations
-
-
-class Migration(migrations.Migration):
-
-    dependencies = [
-        ('assets', '0060_node_full_value'),
-    ]
-
-    operations = [
-        migrations.AlterModelOptions(
-            name='node',
-            options={'ordering': ['value'], 'verbose_name': 'Node'},
-        ),
-    ]

apps/assets/migrations/0062_auto_20201117_1938.py

@@ -1,17 +0,0 @@
-# Generated by Django 2.2.13 on 2020-11-17 11:38
-
-from django.db import migrations
-
-
-class Migration(migrations.Migration):
-
-    dependencies = [
-        ('assets', '0061_auto_20201116_1757'),
-    ]
-
-    operations = [
-        migrations.AlterModelOptions(
-            name='asset',
-            options={'ordering': ['hostname', 'ip'], 'verbose_name': 'Asset'},
-        ),
-    ]

apps/assets/migrations/0063_migrate_default_node_key.py

@@ -1,72 +0,0 @@
-# Generated by Jiangjie.Bai on 2020-12-01 10:47
-
-from django.db import migrations
-from django.db.models import Q
-
-default_node_value = 'Default'  # Always
-old_default_node_key = '0'  # Version <= 1.4.3
-new_default_node_key = '1'  # Version >= 1.4.4
-
-
-def compute_parent_key(key):
-    try:
-        return key[:key.rindex(':')]
-    except ValueError:
-        return ''
-
-
-def migrate_default_node_key(apps, schema_editor):
-    """ 将已经存在的Default节点的key从0修改为1 """
-    # 1.4.3版本中Default节点的key为0
-    print('')
-    Node = apps.get_model('assets', 'Node')
-    Asset = apps.get_model('assets', 'Asset')
-
-    # key为0的节点
-    old_default_node = Node.objects.filter(key=old_default_node_key, value=default_node_value).first()
-    if not old_default_node:
-        print(f'Check old default node `key={old_default_node_key} value={default_node_value}` not exists')
-        return
-    print(f'Check old default node `key={old_default_node_key} value={default_node_value}` exists')
-    # key为1的节点
-    new_default_node = Node.objects.filter(key=new_default_node_key, value=default_node_value).first()
-    if new_default_node:
-        print(f'Check new default node `key={new_default_node_key} value={default_node_value}` exists')
-        all_assets = Asset.objects.filter(
-            Q(nodes__key__startswith=f'{new_default_node_key}:') | Q(nodes__key=new_default_node_key)
-        ).distinct()
-        if all_assets:
-            print(f'Check new default node has assets (count: {len(all_assets)})')
-            return
-        all_children = Node.objects.filter(key__startswith=f'{new_default_node_key}:')
-        if all_children:
-            print(f'Check new default node has children nodes (count: {len(all_children)})')
-            return
-        print(f'Check new default node not has assets and children nodes, delete it.')
-        new_default_node.delete()
-    # 执行修改
-    print(f'Modify old default node `key` from `{old_default_node_key}` to `{new_default_node_key}`')
-    nodes = Node.objects.filter(
-        Q(key__istartswith=f'{old_default_node_key}:') | Q(key=old_default_node_key)
-    )
-    for node in nodes:
-        old_key = node.key
-        key_list = old_key.split(':', maxsplit=1)
-        key_list[0] = new_default_node_key
-        new_key = ':'.join(key_list)
-        node.key = new_key
-        node.parent_key = compute_parent_key(node.key)
-    # 批量更新
-    print(f'Bulk update nodes `key` and `parent_key`, (count: {len(nodes)})')
-    Node.objects.bulk_update(nodes, ['key', 'parent_key'])
-
-
-class Migration(migrations.Migration):
-
-    dependencies = [
-        ('assets', '0062_auto_20201117_1938'),
-    ]
-
-    operations = [
-        migrations.RunPython(migrate_default_node_key)
-    ]

apps/assets/migrations/0064_auto_20201203_1100.py

@@ -1,17 +0,0 @@
-# Generated by Django 3.1 on 2020-12-03 03:00
-
-from django.db import migrations
-
-
-class Migration(migrations.Migration):
-
-    dependencies = [
-        ('assets', '0063_migrate_default_node_key'),
-    ]
-
-    operations = [
-        migrations.AlterModelOptions(
-            name='node',
-            options={'ordering': ['parent_key', 'value'], 'verbose_name': 'Node'},
-        ),
-    ]

apps/assets/migrations/0065_auto_20210121_1549.py

@@ -1,17 +0,0 @@
-# Generated by Django 3.1 on 2021-01-21 07:49
-
-from django.db import migrations
-
-
-class Migration(migrations.Migration):
-
-    dependencies = [
-        ('assets', '0064_auto_20201203_1100'),
-    ]
-
-    operations = [
-        migrations.AlterModelOptions(
-            name='domain',
-            options={'ordering': ('name',), 'verbose_name': 'Domain'},
-        ),
-    ]

apps/assets/models/asset.py

@@ -47,10 +47,6 @@ class AssetManager(OrgManager):
         )
 
 
-class AssetOrgManager(OrgManager):
-    pass
-
-
 class AssetQuerySet(models.QuerySet):
     def active(self):
         return self.filter(is_active=True)
@@ -225,12 +221,11 @@ class Asset(ProtocolsMixin, NodesRelationMixin, OrgModelMixin):
     hostname_raw = models.CharField(max_length=128, blank=True, null=True, verbose_name=_('Hostname raw'))
 
     labels = models.ManyToManyField('assets.Label', blank=True, related_name='assets', verbose_name=_("Labels"))
-    created_by = models.CharField(max_length=128, null=True, blank=True, verbose_name=_('Created by'))
+    created_by = models.CharField(max_length=32, null=True, blank=True, verbose_name=_('Created by'))
     date_created = models.DateTimeField(auto_now_add=True, null=True, blank=True, verbose_name=_('Date created'))
-    comment = models.TextField(default='', blank=True, verbose_name=_('Comment'))
+    comment = models.TextField(max_length=128, default='', blank=True, verbose_name=_('Comment'))
 
     objects = AssetManager.from_queryset(AssetQuerySet)()
-    org_objects = AssetOrgManager.from_queryset(AssetQuerySet)()
     _connectivity = None
 
     def __str__(self):
@@ -249,6 +244,10 @@ class Asset(ProtocolsMixin, NodesRelationMixin, OrgModelMixin):
     def platform_base(self):
         return self.platform.base
 
+    @lazyproperty
+    def admin_user_display(self):
+        return self.admin_user.name
+
     @lazyproperty
     def admin_user_username(self):
         """求可连接性时，直接用用户名去取，避免再查一次admin user
@@ -313,12 +312,6 @@ class Asset(ProtocolsMixin, NodesRelationMixin, OrgModelMixin):
         }
         return info
 
-    def nodes_display(self):
-        names = []
-        for n in self.nodes.all():
-            names.append(n.full_value)
-        return names
-
     def as_node(self):
         from .node import Node
         fake_node = Node()
@@ -361,4 +354,36 @@ class Asset(ProtocolsMixin, NodesRelationMixin, OrgModelMixin):
     class Meta:
         unique_together = [('org_id', 'hostname')]
         verbose_name = _("Asset")
-        ordering = ["hostname", "ip"]
+
+    @classmethod
+    def generate_fake(cls, count=100):
+        from .user import AdminUser, SystemUser
+        from random import seed, choice
+        from django.db import IntegrityError
+        from .node import Node
+        from orgs.utils import get_current_org
+        from orgs.models import Organization
+        org = get_current_org()
+        if not org or not org.is_real():
+            Organization.default().change_to()
+
+        nodes = list(Node.objects.all())
+        seed()
+        for i in range(count):
+            ip = [str(i) for i in random.sample(range(255), 4)]
+            asset = cls(ip='.'.join(ip),
+                        hostname='.'.join(ip),
+                        admin_user=choice(AdminUser.objects.all()),
+                        created_by='Fake')
+            try:
+                asset.save()
+                asset.protocols = 'ssh/22'
+                if nodes and len(nodes) > 3:
+                    _nodes = random.sample(nodes, 3)
+                else:
+                    _nodes = [Node.default_node()]
+                asset.nodes.set(_nodes)
+                logger.debug('Generate fake asset : %s' % asset.ip)
+            except IntegrityError:
+                print('Error continue')
+                continue

apps/assets/models/authbook.py

@@ -3,6 +3,7 @@
 
 from django.db import models, transaction
 from django.db.models import Max
+from django.core.cache import cache
 from django.utils.translation import ugettext_lazy as _
 
 from orgs.mixins.models import OrgManager
@@ -57,18 +58,20 @@ class AuthBook(BaseUser):
         同时设置自己的 is_latest=True, version=max_version + 1
         """
         username = kwargs['username']
-        asset = kwargs.get('asset') or kwargs.get('asset_id')
-        with transaction.atomic():
-            # 使用select_for_update限制并发创建相同的username、asset条目
-            instances = cls.objects.select_for_update().filter(username=username, asset=asset)
-            instances.filter(is_latest=True).update(is_latest=False)
-            max_version = cls.get_max_version(username, asset)
-            kwargs.update({
-                'version': max_version + 1,
-                'is_latest': True
-            })
-            obj = cls.objects.create(**kwargs)
-            return obj
+        asset = kwargs['asset']
+        key_lock = 'KEY_LOCK_CREATE_AUTH_BOOK_{}_{}'.format(username, asset.id)
+        with cache.lock(key_lock):
+            with transaction.atomic():
+                cls.objects.filter(
+                    username=username, asset=asset, is_latest=True
+                ).update(is_latest=False)
+                max_version = cls.get_max_version(username, asset)
+                kwargs.update({
+                    'version': max_version + 1,
+                    'is_latest': True
+                })
+                obj = cls.objects.create(**kwargs)
+                return obj
 
     @property
     def connectivity(self):

apps/assets/models/base.py

@@ -158,11 +158,9 @@ class AuthMixin:
         if update_fields:
             self.save(update_fields=update_fields)
 
-    def has_special_auth(self, asset=None, username=None):
+    def has_special_auth(self, asset=None):
         from .authbook import AuthBook
-        if username is None:
-            username = self.username
-        queryset = AuthBook.objects.filter(username=username)
+        queryset = AuthBook.objects.filter(username=self.username)
         if asset:
             queryset = queryset.filter(asset=asset)
         return queryset.exists()
@@ -232,7 +230,7 @@ class AuthMixin:
 class BaseUser(OrgModelMixin, AuthMixin, ConnectivityMixin):
     id = models.UUIDField(default=uuid.uuid4, primary_key=True)
     name = models.CharField(max_length=128, verbose_name=_('Name'))
-    username = models.CharField(max_length=128, blank=True, verbose_name=_('Username'), validators=[alphanumeric], db_index=True)
+    username = models.CharField(max_length=32, blank=True, verbose_name=_('Username'), validators=[alphanumeric], db_index=True)
     password = fields.EncryptCharField(max_length=256, blank=True, null=True, verbose_name=_('Password'))
     private_key = fields.EncryptTextField(blank=True, null=True, verbose_name=_('SSH private key'))
     public_key = fields.EncryptTextField(blank=True, null=True, verbose_name=_('SSH public key'))

apps/assets/models/cluster.py

@@ -38,3 +38,27 @@ class Cluster(models.Model):
     class Meta:
         ordering = ['name']
         verbose_name = _("Cluster")
+
+    @classmethod
+    def generate_fake(cls, count=5):
+        from random import seed, choice
+        import forgery_py
+        from django.db import IntegrityError
+
+        seed()
+        for i in range(count):
+            cluster = cls(name=forgery_py.name.full_name(),
+                          bandwidth='200M',
+                          contact=forgery_py.name.full_name(),
+                          phone=forgery_py.address.phone(),
+                          address=forgery_py.address.city() + forgery_py.address.street_address(),
+                          # operator=choice(['北京联通', '北京电信', 'BGP全网通']),
+                          operator=choice([_('Beijing unicom'), _('Beijing telecom'), _('BGP full netcom')]),
+                          comment=forgery_py.lorem_ipsum.sentence(),
+                          created_by='Fake')
+            try:
+                cluster.save()
+                logger.debug('Generate fake asset group: %s' % cluster.name)
+            except IntegrityError:
+                print('Error continue')
+                continue

apps/assets/models/cmd_filter.py

@@ -18,7 +18,7 @@ __all__ = [
 
 class CommandFilter(OrgModelMixin):
     id = models.UUIDField(default=uuid.uuid4, primary_key=True)
-    name = models.CharField(max_length=64, verbose_name=_("Name"))
+    name = models.CharField(max_length=64, unique=True, verbose_name=_("Name"))
     is_active = models.BooleanField(default=True, verbose_name=_('Is active'))
     comment = models.TextField(blank=True, default='', verbose_name=_("Comment"))
     date_created = models.DateTimeField(auto_now_add=True)
@@ -29,7 +29,6 @@ class CommandFilter(OrgModelMixin):
         return self.name
 
     class Meta:
-        unique_together = [('org_id', 'name')]
         verbose_name = _("Command filter")
 
 
@@ -52,7 +51,7 @@ class CommandFilterRule(OrgModelMixin):
     type = models.CharField(max_length=16, default=TYPE_COMMAND, choices=TYPE_CHOICES, verbose_name=_("Type"))
     priority = models.IntegerField(default=50, verbose_name=_("Priority"), help_text=_("1-100, the higher will be match first"),
                                    validators=[MinValueValidator(1), MaxValueValidator(100)])
-    content = models.TextField(verbose_name=_("Content"), help_text=_("One line one command"))
+    content = models.TextField(max_length=1024, verbose_name=_("Content"), help_text=_("One line one command"))
     action = models.IntegerField(default=ACTION_DENY, choices=ACTION_CHOICES, verbose_name=_("Action"))
     comment = models.CharField(max_length=64, blank=True, default='', verbose_name=_("Comment"))
     date_created = models.DateTimeField(auto_now_add=True)

apps/assets/models/domain.py

@@ -9,7 +9,6 @@ import paramiko
 from django.db import models
 from django.utils.translation import ugettext_lazy as _
 
-from common.utils.strings import no_special_chars
 from orgs.mixins.models import OrgModelMixin
 from .base import BaseUser
 
@@ -18,15 +17,13 @@ __all__ = ['Domain', 'Gateway']
 
 class Domain(OrgModelMixin):
     id = models.UUIDField(default=uuid.uuid4, primary_key=True)
-    name = models.CharField(max_length=128, verbose_name=_('Name'))
+    name = models.CharField(max_length=128, unique=True, verbose_name=_('Name'))
     comment = models.TextField(blank=True, verbose_name=_('Comment'))
     date_created = models.DateTimeField(auto_now_add=True, null=True,
                                         verbose_name=_('Date created'))
 
     class Meta:
         verbose_name = _("Domain")
-        unique_together = [('org_id', 'name')]
-        ordering = ('name',)
 
     def __str__(self):
         return self.name
@@ -49,7 +46,7 @@ class Gateway(BaseUser):
         (PROTOCOL_SSH, 'ssh'),
         (PROTOCOL_RDP, 'rdp'),
     )
-    ip = models.CharField(max_length=128, verbose_name=_('IP'), db_index=True)
+    ip = models.GenericIPAddressField(max_length=32, verbose_name=_('IP'), db_index=True)
     port = models.IntegerField(default=22, verbose_name=_('Port'))
     protocol = models.CharField(choices=PROTOCOL_CHOICES, max_length=16, default=PROTOCOL_SSH, verbose_name=_("Protocol"))
     domain = models.ForeignKey(Domain, on_delete=models.CASCADE, verbose_name=_("Domain"))
@@ -66,8 +63,8 @@ class Gateway(BaseUser):
     def test_connective(self, local_port=None):
         if local_port is None:
             local_port = self.port
-        if self.password and not no_special_chars(self.password):
-            return False, _("Password should not contains special characters")
+        if self.password and not re.match(r'\w+$', self.password):
+            return False, _("Password should not contain special characters")
 
         client = paramiko.SSHClient()
         client.set_missing_host_key_policy(paramiko.AutoAddPolicy())

apps/assets/models/favorite_asset.py

@@ -18,15 +18,3 @@ class FavoriteAsset(CommonModelMixin):
     @classmethod
     def get_user_favorite_assets_id(cls, user):
         return cls.objects.filter(user=user).values_list('asset', flat=True)
-
-    @classmethod
-    def get_user_favorite_assets(cls, user, asset_perms_id=None):
-        from assets.models import Asset
-        from perms.utils.asset.user_permission import get_user_granted_all_assets
-        asset_ids = get_user_granted_all_assets(
-            user,
-            via_mapping_node=False,
-            asset_perms_id=asset_perms_id
-        ).values_list('id', flat=True)
-        query_name = cls.asset.field.related_query_name()
-        return Asset.org_objects.filter(**{f'{query_name}__user_id': user.id}, id__in=asset_ids).distinct()

apps/assets/models/group.py

@@ -33,3 +33,21 @@ class AssetGroup(models.Model):
     def initial(cls):
         asset_group = cls(name=_('Default'), comment=_('Default asset group'))
         asset_group.save()
+
+    @classmethod
+    def generate_fake(cls, count=100):
+        from random import seed
+        import forgery_py
+        from django.db import IntegrityError
+
+        seed()
+        for i in range(count):
+            group = cls(name=forgery_py.name.full_name(),
+                        comment=forgery_py.lorem_ipsum.sentence(),
+                        created_by='Fake')
+            try:
+                group.save()
+                logger.debug('Generate fake asset group: %s' % group.name)
+            except IntegrityError:
+                print('Error continue')
+                continue

apps/assets/models/node.py

@@ -2,35 +2,132 @@
 #
 import uuid
 import re
+import time
 
 from django.db import models, transaction
 from django.db.models import Q
 from django.db.utils import IntegrityError
 from django.utils.translation import ugettext_lazy as _
 from django.utils.translation import ugettext
-from django.db.transaction import atomic
+from django.core.cache import cache
 
-from common.utils import get_logger
-from common.utils.common import lazyproperty
+from common.utils import get_logger, lazyproperty
 from orgs.mixins.models import OrgModelMixin, OrgManager
-from orgs.utils import get_current_org, tmp_to_org
+from orgs.utils import get_current_org, tmp_to_org, current_org
 from orgs.models import Organization
 
 
-__all__ = ['Node', 'FamilyMixin', 'compute_parent_key']
+__all__ = ['Node']
 logger = get_logger(__name__)
 
 
-def compute_parent_key(key):
-    try:
-        return key[:key.rindex(':')]
-    except ValueError:
-        return ''
-
-
 class NodeQuerySet(models.QuerySet):
     def delete(self):
-        raise NotImplementedError
+        raise PermissionError("Bulk delete node deny")
+
+
+class TreeCache:
+    updated_time_cache_key = 'NODE_TREE_UPDATED_AT_{}'
+    cache_time = 3600
+    assets_updated_time_cache_key = 'NODE_TREE_ASSETS_UPDATED_AT_{}'
+
+    def __init__(self, tree, org_id):
+        now = time.time()
+        self.created_time = now
+        self.assets_created_time = now
+        self.tree = tree
+        self.org_id = org_id
+
+    def _has_changed(self, tp="tree"):
+        if tp == "assets":
+            key = self.assets_updated_time_cache_key.format(self.org_id)
+        else:
+            key = self.updated_time_cache_key.format(self.org_id)
+        updated_time = cache.get(key, 0)
+        if updated_time > self.created_time:
+            return True
+        else:
+            return False
+
+    @classmethod
+    def set_changed(cls, tp="tree", t=None, org_id=None):
+        if org_id is None:
+            org_id = current_org.id
+        if tp == "assets":
+            key = cls.assets_updated_time_cache_key.format(org_id)
+        else:
+            key = cls.updated_time_cache_key.format(org_id)
+        ttl = cls.cache_time
+        if not t:
+            t = time.time()
+        cache.set(key, t, ttl)
+
+    def tree_has_changed(self):
+        return self._has_changed("tree")
+
+    def set_tree_changed(self, t=None):
+        logger.debug("Set tree tree changed")
+        self.__class__.set_changed(t=t, tp="tree")
+
+    def assets_has_changed(self):
+        return self._has_changed("assets")
+
+    def set_tree_assets_changed(self, t=None):
+        logger.debug("Set tree assets changed")
+        self.__class__.set_changed(t=t, tp="assets")
+
+    def get(self):
+        if self.tree_has_changed():
+            self.renew()
+            return self.tree
+        if self.assets_has_changed():
+            self.tree.init_assets()
+        return self.tree
+
+    def renew(self):
+        new_obj = self.__class__.new(self.org_id)
+        self.tree = new_obj.tree
+        self.created_time = new_obj.created_time
+        self.assets_created_time = new_obj.assets_created_time
+
+    @classmethod
+    def new(cls, org_id=None):
+        from ..utils import TreeService
+        logger.debug("Create node tree")
+        if not org_id:
+            org_id = current_org.id
+        with tmp_to_org(org_id):
+            tree = TreeService.new()
+            obj = cls(tree, org_id)
+            obj.tree = tree
+            return obj
+
+
+class TreeMixin:
+    _org_tree_map = {}
+
+    @classmethod
+    def tree(cls):
+        org_id = current_org.org_id()
+        t = cls.get_local_tree_cache(org_id)
+
+        if t is None:
+            t = TreeCache.new()
+            cls._org_tree_map[org_id] = t
+        return t.get()
+
+    @classmethod
+    def get_local_tree_cache(cls, org_id=None):
+        t = cls._org_tree_map.get(org_id)
+        return t
+
+    @classmethod
+    def refresh_tree(cls, t=None):
+        TreeCache.set_changed(tp="tree", t=t, org_id=current_org.id)
+
+    @classmethod
+    def refresh_node_assets(cls, t=None):
+        TreeCache.set_changed(tp="assets", t=t, org_id=current_org.id)
 
 
 class FamilyMixin:
@@ -38,20 +135,19 @@ class FamilyMixin:
     __children = None
     __all_children = None
     is_node = True
-    child_mark: int
 
     @staticmethod
     def clean_children_keys(nodes_keys):
-        sort_key = lambda k: [int(i) for i in k.split(':')]
-        nodes_keys = sorted(list(nodes_keys), key=sort_key)
-
+        nodes_keys = sorted(list(nodes_keys), key=lambda x: (len(x), x))
         nodes_keys_clean = []
-        base_key = ''
-        for key in nodes_keys:
-            if key.startswith(base_key + ':'):
-                continue
-            nodes_keys_clean.append(key)
-            base_key = key
+        for key in nodes_keys[::-1]:
+            found = False
+            for k in nodes_keys:
+                if key.startswith(k + ':'):
+                    found = True
+                    break
+            if not found:
+                nodes_keys_clean.append(key)
         return nodes_keys_clean
 
     @classmethod
@@ -79,16 +175,13 @@ class FamilyMixin:
         return re.match(children_pattern, self.key)
 
     def get_children(self, with_self=False):
-        q = Q(parent_key=self.key)
-        if with_self:
-            q |= Q(key=self.key)
-        return Node.objects.filter(q)
+        pattern = self.get_children_key_pattern(with_self=with_self)
+        return Node.objects.filter(key__regex=pattern)
 
     def get_all_children(self, with_self=False):
-        q = Q(key__istartswith=f'{self.key}:')
-        if with_self:
-            q |= Q(key=self.key)
-        return Node.objects.filter(q)
+        pattern = self.get_all_children_pattern(with_self=with_self)
+        children = Node.objects.filter(key__regex=pattern)
+        return children
 
     @property
     def children(self):
@@ -98,46 +191,19 @@ class FamilyMixin:
     def all_children(self):
         return self.get_all_children(with_self=False)
 
-    def create_child(self, value=None, _id=None):
-        with atomic(savepoint=False):
+    def create_child(self, value, _id=None):
+        with transaction.atomic():
             child_key = self.get_next_child_key()
-            if value is None:
-                value = child_key
             child = self.__class__.objects.create(
                 id=_id, key=child_key, value=value
             )
             return child
 
-    def get_or_create_child(self, value, _id=None):
-        """
-        :return: Node, bool (created)
-        """
-        children = self.get_children()
-        exist = children.filter(value=value).exists()
-        if exist:
-            child = children.filter(value=value).first()
-            created = False
-        else:
-            child = self.create_child(value, _id)
-            created = True
-        return child, created
-
-    def get_valid_child_mark(self):
-        key = "{}:{}".format(self.key, self.child_mark)
-        if not self.__class__.objects.filter(key=key).exists():
-            return self.child_mark
-        children_keys = self.get_children().values_list('key', flat=True)
-        children_keys_last = [key.split(':')[-1] for key in children_keys]
-        children_keys_last = [int(k) for k in children_keys_last if k.strip().isdigit()]
-        max_key_last = max(children_keys_last) if children_keys_last else 1
-        return max_key_last + 1
-
     def get_next_child_key(self):
-        child_mark = self.get_valid_child_mark()
-        key = "{}:{}".format(self.key, child_mark)
-        self.child_mark = child_mark + 1
+        mark = self.child_mark
+        self.child_mark += 1
         self.save()
-        return key
+        return "{}:{}".format(self.key, mark)
 
     def get_next_child_preset_name(self):
         name = ugettext("New node")
@@ -175,13 +241,10 @@ class FamilyMixin:
         ancestor_keys = self.get_ancestor_keys(with_self=with_self)
         return self.__class__.objects.filter(key__in=ancestor_keys)
 
-    # @property
-    # def parent_key(self):
-    #     parent_key = ":".join(self.key.split(":")[:-1])
-    #     return parent_key
-
-    def compute_parent_key(self):
-        return compute_parent_key(self.key)
+    @property
+    def parent_key(self):
+        parent_key = ":".join(self.key.split(":")[:-1])
+        return parent_key
 
     def is_parent(self, other):
         return other.is_children(self)
@@ -217,63 +280,45 @@ class FamilyMixin:
             sibling = sibling.exclude(key=self.key)
         return sibling
 
-    @classmethod
-    def create_node_by_full_value(cls, full_value):
-        if not full_value:
-            return []
-        nodes_family = full_value.split('/')
-        nodes_family = [v for v in nodes_family if v]
-        org_root = cls.org_root()
-        if nodes_family[0] == org_root.value:
-            nodes_family = nodes_family[1:]
-        return cls.create_nodes_recurse(nodes_family, org_root)
-
-    @classmethod
-    def create_nodes_recurse(cls, values, parent=None):
-        values = [v for v in values if v]
-        if not values:
-            return None
-        if parent is None:
-            parent = cls.org_root()
-        value = values[0]
-        child, created = parent.get_or_create_child(value=value)
-        if len(values) == 1:
-            return child
-        return cls.create_nodes_recurse(values[1:], child)
-
     def get_family(self):
         ancestors = self.get_ancestors()
         children = self.get_all_children()
         return [*tuple(ancestors), self, *tuple(children)]
 
 
+class FullValueMixin:
+    key = ''
+
+    @lazyproperty
+    def full_value(self):
+        if self.is_org_root():
+            return self.value
+        value = self.tree().get_node_full_tag(self.key)
+        return value
+
+
 class NodeAssetsMixin:
     key = ''
     id = None
 
+    @lazyproperty
+    def assets_amount(self):
+        amount = self.tree().assets_amount(self.key)
+        return amount
+
     def get_all_assets(self):
         from .asset import Asset
-        q = Q(nodes__key__startswith=f'{self.key}:') | Q(nodes__key=self.key)
-        return Asset.objects.filter(q).distinct()
-
-    @classmethod
-    def get_node_all_assets_by_key_v2(cls, key):
-        # 最初的写法是：
-        #   Asset.objects.filter(Q(nodes__key__startswith=f'{node.key}:') | Q(nodes__id=node.id))
-        #   可是 startswith 会导致表关联时 Asset 索引失效
-        from .asset import Asset
-        node_ids = cls.objects.filter(
-            Q(key__startswith=f'{key}:') |
-            Q(key=key)
-        ).values_list('id', flat=True).distinct()
-        assets = Asset.objects.filter(
-            nodes__id__in=list(node_ids)
-        ).distinct()
-        return assets
+        if self.is_org_root():
+            return Asset.objects.filter(org_id=self.org_id)
+        pattern = '^{0}$|^{0}:'.format(self.key)
+        return Asset.objects.filter(nodes__key__regex=pattern).distinct()
 
     def get_assets(self):
         from .asset import Asset
-        assets = Asset.objects.filter(nodes=self)
+        if self.is_org_root():
+            assets = Asset.objects.filter(Q(nodes=self) | Q(nodes__isnull=True))
+        else:
+            assets = Asset.objects.filter(nodes=self)
         return assets.distinct()
 
     def get_valid_assets(self):
@@ -282,54 +327,51 @@ class NodeAssetsMixin:
     def get_all_valid_assets(self):
         return self.get_all_assets().valid()
 
+    @classmethod
+    def _get_nodes_all_assets(cls, nodes_keys):
+        """
+        当节点比较多的时候，这种正则方式性能差极了
+        :param nodes_keys:
+        :return:
+        """
+        from .asset import Asset
+        nodes_keys = cls.clean_children_keys(nodes_keys)
+        nodes_children_pattern = set()
+        for key in nodes_keys:
+            children_pattern = cls.get_node_all_children_key_pattern(key)
+            nodes_children_pattern.add(children_pattern)
+        pattern = '|'.join(nodes_children_pattern)
+        return Asset.objects.filter(nodes__key__regex=pattern).distinct()
+
     @classmethod
     def get_nodes_all_assets_ids(cls, nodes_keys):
-        assets_ids = cls.get_nodes_all_assets(nodes_keys).values_list('id', flat=True)
+        nodes_keys = cls.clean_children_keys(nodes_keys)
+        assets_ids = set()
+        for key in nodes_keys:
+            node_assets_ids = cls.tree().all_assets(key)
+            assets_ids.update(set(node_assets_ids))
         return assets_ids
 
     @classmethod
     def get_nodes_all_assets(cls, nodes_keys, extra_assets_ids=None):
         from .asset import Asset
         nodes_keys = cls.clean_children_keys(nodes_keys)
-        q = Q()
-        node_ids = ()
-        for key in nodes_keys:
-            q |= Q(key__startswith=f'{key}:')
-            q |= Q(key=key)
-        if q:
-            node_ids = Node.objects.filter(q).distinct().values_list('id', flat=True)
-
-        q = Q(nodes__id__in=list(node_ids))
+        assets_ids = cls.get_nodes_all_assets_ids(nodes_keys)
         if extra_assets_ids:
-            q |= Q(id__in=extra_assets_ids)
-        if q:
-            return Asset.org_objects.filter(q).distinct()
-        else:
-            return Asset.objects.none()
+            assets_ids.update(set(extra_assets_ids))
+        return Asset.objects.filter(id__in=assets_ids)
 
 
 class SomeNodesMixin:
     key = ''
     default_key = '1'
     default_value = 'Default'
+    ungrouped_key = '-10'
+    ungrouped_value = _('ungrouped')
     empty_key = '-11'
     empty_value = _("empty")
-
-    @classmethod
-    def default_node(cls):
-        with tmp_to_org(Organization.default()):
-            defaults = {'value': cls.default_value}
-            try:
-                obj, created = cls.objects.get_or_create(
-                    defaults=defaults, key=cls.default_key,
-                )
-            except IntegrityError as e:
-                logger.error("Create default node failed: {}".format(e))
-                cls.modify_other_org_root_node_key()
-                obj, created = cls.objects.get_or_create(
-                    defaults=defaults, key=cls.default_key,
-                )
-            return obj
+    favorite_key = '-12'
+    favorite_value = _("favorite")
 
     def is_default_node(self):
         return self.key == self.default_key
@@ -364,18 +406,51 @@ class SomeNodesMixin:
 
     @classmethod
     def org_root(cls):
-        root = cls.objects.filter(parent_key='')\
-            .filter(key__regex=r'^[0-9]+$')\
-            .exclude(key__startswith='-')\
-            .order_by('key')
+        root = cls.objects.filter(key__regex=r'^[0-9]+$')
         if root:
             return root[0]
         else:
             return cls.create_org_root_node()
 
+    @classmethod
+    def ungrouped_node(cls):
+        with tmp_to_org(Organization.system()):
+            defaults = {'value': cls.ungrouped_value}
+            obj, created = cls.objects.get_or_create(
+                defaults=defaults, key=cls.ungrouped_key
+            )
+            return obj
+
+    @classmethod
+    def default_node(cls):
+        with tmp_to_org(Organization.default()):
+            defaults = {'value': cls.default_value}
+            try:
+                obj, created = cls.objects.get_or_create(
+                    defaults=defaults, key=cls.default_key,
+                )
+            except IntegrityError as e:
+                logger.error("Create default node failed: {}".format(e))
+                cls.modify_other_org_root_node_key()
+                obj, created = cls.objects.get_or_create(
+                    defaults=defaults, key=cls.default_key,
+                )
+            return obj
+
+    @classmethod
+    def favorite_node(cls):
+        with tmp_to_org(Organization.system()):
+            defaults = {'value': cls.favorite_value}
+            obj, created = cls.objects.get_or_create(
+                defaults=defaults, key=cls.favorite_key
+            )
+            return obj
+
     @classmethod
     def initial_some_nodes(cls):
         cls.default_node()
+        cls.ungrouped_node()
+        cls.favorite_node()
 
     @classmethod
     def modify_other_org_root_node_key(cls):
@@ -407,16 +482,12 @@ class SomeNodesMixin:
                     logger.info('Modify key ( {} > {} )'.format(old_key, new_key))
 
 
-class Node(OrgModelMixin, SomeNodesMixin, FamilyMixin, NodeAssetsMixin):
+class Node(OrgModelMixin, SomeNodesMixin, TreeMixin, FamilyMixin, FullValueMixin, NodeAssetsMixin):
     id = models.UUIDField(default=uuid.uuid4, primary_key=True)
     key = models.CharField(unique=True, max_length=64, verbose_name=_("Key"))  # '1:1:1:1'
     value = models.CharField(max_length=128, verbose_name=_("Value"))
-    full_value = models.CharField(max_length=4096, verbose_name=_('Full value'), default='')
     child_mark = models.IntegerField(default=0)
     date_create = models.DateTimeField(auto_now_add=True)
-    parent_key = models.CharField(max_length=64, verbose_name=_("Parent key"),
-                                  db_index=True, default='')
-    assets_amount = models.IntegerField(default=0)
 
     objects = OrgManager.from_queryset(NodeQuerySet)()
     is_node = True
@@ -424,10 +495,10 @@ class Node(OrgModelMixin, SomeNodesMixin, FamilyMixin, NodeAssetsMixin):
 
     class Meta:
         verbose_name = _("Node")
-        ordering = ['parent_key', 'value']
+        ordering = ['key']
 
     def __str__(self):
-        return self.full_value
+        return self.value
 
     # def __eq__(self, other):
     #     if not other:
@@ -451,19 +522,18 @@ class Node(OrgModelMixin, SomeNodesMixin, FamilyMixin, NodeAssetsMixin):
     def name(self):
         return self.value
 
-    def computed_full_value(self):
-        # 不要在列表中调用该属性
-        values = self.__class__.objects.filter(
-            key__in=self.get_ancestor_keys()
-        ).values_list('key', 'value')
-        values = [v for k, v in sorted(values, key=lambda x: len(x[0]))]
-        values.append(str(self.value))
-        return '/' + '/'.join(values)
-
     @property
     def level(self):
         return len(self.key.split(':'))
 
+    @classmethod
+    def refresh_nodes(cls):
+        cls.refresh_tree()
+
+    @classmethod
+    def refresh_assets(cls):
+        cls.refresh_node_assets()
+
     def as_tree_node(self):
         from common.tree import TreeNode
         name = '{} ({})'.format(self.value, self.assets_amount)
@@ -498,26 +568,19 @@ class Node(OrgModelMixin, SomeNodesMixin, FamilyMixin, NodeAssetsMixin):
             return
         return super().delete(using=using, keep_parents=keep_parents)
 
-    def update_child_full_value(self):
-        nodes = self.get_all_children(with_self=True)
-        sort_key_func = lambda n: [int(i) for i in n.key.split(':')]
-        nodes_sorted = sorted(list(nodes), key=sort_key_func)
-        nodes_mapper = {n.key: n for n in nodes_sorted}
-        if not self.is_org_root():
-            # 如果是org_root，那么parent_key为'', parent为自己，所以这种情况不处理
-            # 更新自己时，自己的parent_key获取不到
-            nodes_mapper.update({self.parent_key: self.parent})
-        for node in nodes_sorted:
-            parent = nodes_mapper.get(node.parent_key)
-            if not parent:
-                if node.parent_key:
-                    logger.error(f'Node parent node in mapper: {node.parent_key} {node.value}')
-                continue
-            node.full_value = parent.full_value + '/' + node.value
-        self.__class__.objects.bulk_update(nodes, ['full_value'])
+    @classmethod
+    def generate_fake(cls, count=100):
+        import random
+        org = get_current_org()
+        if not org or not org.is_real():
+            Organization.default().change_to()
+        nodes = list(cls.objects.all())
+        if count > 100:
+            length = 100
+        else:
+            length = count
 
-    def save(self, *args, **kwargs):
-        self.full_value = self.computed_full_value()
-        instance = super().save(*args, **kwargs)
-        self.update_child_full_value()
-        return instance
+        for i in range(length):
+            node = random.choice(nodes)
+            child = node.create_child('Node {}'.format(i))
+            print("{}. {}".format(i, child))

apps/assets/models/user.py

@@ -9,7 +9,6 @@ from django.utils.translation import ugettext_lazy as _
 from django.core.validators import MinValueValidator, MaxValueValidator
 
 from common.utils import signer
-from common.fields.model import JsonListCharField
 from .base import BaseUser
 from .asset import Asset
 
@@ -65,6 +64,26 @@ class AdminUser(BaseUser):
         unique_together = [('name', 'org_id')]
         verbose_name = _("Admin user")
 
+    @classmethod
+    def generate_fake(cls, count=10):
+        from random import seed
+        import forgery_py
+        from django.db import IntegrityError
+
+        seed()
+        for i in range(count):
+            obj = cls(name=forgery_py.name.full_name(),
+                      username=forgery_py.internet.user_name(),
+                      password=forgery_py.lorem_ipsum.word(),
+                      comment=forgery_py.lorem_ipsum.sentence(),
+                      created_by='Fake')
+            try:
+                obj.save()
+                logger.debug('Generate fake asset group: %s' % obj.name)
+            except IntegrityError:
+                print('Error continue')
+                continue
+
 
 class SystemUser(BaseUser):
     PROTOCOL_SSH = 'ssh'
@@ -72,38 +91,13 @@ class SystemUser(BaseUser):
     PROTOCOL_TELNET = 'telnet'
     PROTOCOL_VNC = 'vnc'
     PROTOCOL_MYSQL = 'mysql'
-    PROTOCOL_ORACLE = 'oracle'
-    PROTOCOL_MARIADB = 'mariadb'
-    PROTOCOL_POSTGRESQL = 'postgresql'
-    PROTOCOL_K8S = 'k8s'
     PROTOCOL_CHOICES = (
         (PROTOCOL_SSH, 'ssh'),
         (PROTOCOL_RDP, 'rdp'),
         (PROTOCOL_TELNET, 'telnet'),
         (PROTOCOL_VNC, 'vnc'),
         (PROTOCOL_MYSQL, 'mysql'),
-        (PROTOCOL_ORACLE, 'oracle'),
-        (PROTOCOL_MARIADB, 'mariadb'),
-        (PROTOCOL_POSTGRESQL, 'postgresql'),
-        (PROTOCOL_K8S, 'k8s'),
     )
-    ASSET_CATEGORY_PROTOCOLS = [
-        PROTOCOL_SSH, PROTOCOL_RDP, PROTOCOL_TELNET, PROTOCOL_VNC
-    ]
-    APPLICATION_CATEGORY_REMOTE_APP_PROTOCOLS = [
-        PROTOCOL_RDP
-    ]
-    APPLICATION_CATEGORY_DB_PROTOCOLS = [
-        PROTOCOL_MYSQL, PROTOCOL_ORACLE, PROTOCOL_MARIADB, PROTOCOL_POSTGRESQL
-    ]
-    APPLICATION_CATEGORY_CLOUD_PROTOCOLS = [
-        PROTOCOL_K8S
-    ]
-    APPLICATION_CATEGORY_PROTOCOLS = [
-        *APPLICATION_CATEGORY_REMOTE_APP_PROTOCOLS,
-        *APPLICATION_CATEGORY_DB_PROTOCOLS,
-        *APPLICATION_CATEGORY_CLOUD_PROTOCOLS
-    ]
 
     LOGIN_AUTO = 'auto'
     LOGIN_MANUAL = 'manual'
@@ -124,10 +118,6 @@ class SystemUser(BaseUser):
     login_mode = models.CharField(choices=LOGIN_MODE_CHOICES, default=LOGIN_AUTO, max_length=10, verbose_name=_('Login mode'))
     cmd_filters = models.ManyToManyField('CommandFilter', related_name='system_users', verbose_name=_("Command filter"), blank=True)
     sftp_root = models.CharField(default='tmp', max_length=128, verbose_name=_("SFTP Root"))
-    token = models.TextField(default='', verbose_name=_('Token'))
-    home = models.CharField(max_length=4096, default='', verbose_name=_('Home'), blank=True)
-    system_groups = models.CharField(default='', max_length=4096, verbose_name=_('System groups'), blank=True)
-    ad_domain = models.CharField(default='', max_length=256)
     _prefer = 'system_user'
 
     def __str__(self):
@@ -162,16 +152,11 @@ class SystemUser(BaseUser):
 
     @property
     def is_need_test_asset_connective(self):
-        return self.protocol in self.ASSET_CATEGORY_PROTOCOLS
-
-    def has_special_auth(self, asset=None, username=None):
-        if username is None and self.username_same_with_user:
-            raise TypeError('System user is dynamic, username should be pass')
-        return super().has_special_auth(asset=asset, username=username)
+        return self.protocol not in [self.PROTOCOL_MYSQL]
 
     @property
     def can_perm_to_asset(self):
-        return self.protocol in self.ASSET_CATEGORY_PROTOCOLS
+        return self.protocol not in [self.PROTOCOL_MYSQL]
 
     def _merge_auth(self, other):
         super()._merge_auth(other)
@@ -204,18 +189,27 @@ class SystemUser(BaseUser):
         assets = Asset.objects.filter(id__in=assets_ids)
         return assets
 
-    @classmethod
-    def get_protocol_by_application_type(cls, app_type):
-        from applications.const import ApplicationTypeChoices
-        if app_type in cls.APPLICATION_CATEGORY_PROTOCOLS:
-            protocol = app_type
-        elif app_type in ApplicationTypeChoices.remote_app_types():
-            protocol = cls.PROTOCOL_RDP
-        else:
-            protocol = None
-        return protocol
-
     class Meta:
         ordering = ['name']
         unique_together = [('name', 'org_id')]
         verbose_name = _("System user")
+
+    @classmethod
+    def generate_fake(cls, count=10):
+        from random import seed
+        import forgery_py
+        from django.db import IntegrityError
+
+        seed()
+        for i in range(count):
+            obj = cls(name=forgery_py.name.full_name(),
+                      username=forgery_py.internet.user_name(),
+                      password=forgery_py.lorem_ipsum.word(),
+                      comment=forgery_py.lorem_ipsum.sentence(),
+                      created_by='Fake')
+            try:
+                obj.save()
+                logger.debug('Generate fake asset group: %s' % obj.name)
+            except IntegrityError:
+                print('Error continue')
+                continue

apps/assets/pagination.py

@@ -1,39 +0,0 @@
-from rest_framework.pagination import LimitOffsetPagination
-from rest_framework.request import Request
-
-from assets.models import Node
-
-
-class AssetLimitOffsetPagination(LimitOffsetPagination):
-    """
-    需要与 `assets.api.mixin.FilterAssetByNodeMixin` 配合使用
-    """
-    def get_count(self, queryset):
-        """
-        1. 如果查询节点下的所有资产，那 count 使用 Node.assets_amount
-        2. 如果有其他过滤条件使用 super
-        3. 如果只查询该节点下的资产使用 super
-        """
-        exclude_query_params = {
-            self.limit_query_param,
-            self.offset_query_param,
-            'node', 'all', 'show_current_asset',
-            'node_id', 'display', 'draw', 'fields_size',
-        }
-
-        for k, v in self._request.query_params.items():
-            if k not in exclude_query_params and v is not None:
-                return super().get_count(queryset)
-
-        is_query_all = self._view.is_query_node_all_assets
-        if is_query_all:
-            node = self._view.node
-            if not node:
-                node = Node.org_root()
-            return node.assets_amount
-        return super().get_count(queryset)
-
-    def paginate_queryset(self, queryset, request: Request, view=None):
-        self._request = request
-        self._view = view
-        return super().paginate_queryset(queryset, request, view=None)

apps/assets/serializers/admin_user.py

@@ -3,7 +3,7 @@
 from django.utils.translation import ugettext_lazy as _
 from rest_framework import serializers
 
-from common.drf.serializers import AdaptedBulkListSerializer
+from common.serializers import AdaptedBulkListSerializer
 
 from ..models import Node, AdminUser
 from orgs.mixins.serializers import BulkOrgResourceModelSerializer

apps/assets/serializers/asset.py

@@ -1,12 +1,13 @@
 # -*- coding: utf-8 -*-
 #
 from rest_framework import serializers
-from django.db.models import F
-from django.core.validators import RegexValidator
+from django.db.models import Prefetch, F, Count
+
 from django.utils.translation import ugettext_lazy as _
 
 from orgs.mixins.serializers import BulkOrgResourceModelSerializer
-from ..models import Asset, Node, Platform
+from common.serializers import AdaptedBulkListSerializer
+from ..models import Asset, Node, Label, Platform
 from .base import ConnectivitySerializer
 
 __all__ = [
@@ -66,15 +67,12 @@ class AssetSerializer(BulkOrgResourceModelSerializer):
         slug_field='name', queryset=Platform.objects.all(), label=_("Platform")
     )
     protocols = ProtocolsField(label=_('Protocols'), required=False)
-    domain_display = serializers.ReadOnlyField(source='domain.name', label=_('Domain name'))
-    admin_user_display = serializers.ReadOnlyField(source='admin_user.name', label=_('Admin user name'))
-    nodes_display = serializers.ListField(child=serializers.CharField(), label=_('Nodes name'), required=False)
-
     """
     资产的数据结构
     """
     class Meta:
         model = Asset
+        list_serializer_class = AdaptedBulkListSerializer
         fields_mini = ['id', 'hostname', 'ip']
         fields_small = fields_mini + [
             'protocol', 'port', 'protocols', 'is_active', 'public_ip',
@@ -84,13 +82,13 @@ class AssetSerializer(BulkOrgResourceModelSerializer):
             'created_by', 'date_created', 'hardware_info',
         ]
         fields_fk = [
-            'admin_user', 'admin_user_display', 'domain', 'domain_display', 'platform'
+            'admin_user', 'admin_user_display', 'domain', 'platform'
         ]
         fk_only_fields = {
             'platform': ['name']
         }
         fields_m2m = [
-            'nodes', 'nodes_display', 'labels',
+            'nodes', 'labels',
         ]
         annotates_fields = {
             # 'admin_user_display': 'admin_user__name'
@@ -98,6 +96,9 @@ class AssetSerializer(BulkOrgResourceModelSerializer):
         fields_as = list(annotates_fields.keys())
         fields = fields_small + fields_fk + fields_m2m + fields_as
         read_only_fields = [
+            'vendor', 'model', 'sn', 'cpu_model', 'cpu_count',
+            'cpu_cores', 'cpu_vcpus', 'memory', 'disk_total', 'disk_info',
+            'os', 'os_version', 'os_arch', 'hostname_raw',
             'created_by', 'date_created',
         ] + fields_as
 
@@ -112,7 +113,6 @@ class AssetSerializer(BulkOrgResourceModelSerializer):
     def setup_eager_loading(cls, queryset):
         """ Perform necessary eager loading of data. """
         queryset = queryset.select_related('admin_user', 'domain', 'platform')
-        queryset = queryset.prefetch_related('nodes', 'labels')
         return queryset
 
     def compatible_with_old_protocol(self, validated_data):
@@ -130,32 +130,14 @@ class AssetSerializer(BulkOrgResourceModelSerializer):
         if protocols_data:
             validated_data["protocols"] = ' '.join(protocols_data)
 
-    def perform_nodes_display_create(self, instance, nodes_display):
-        if not nodes_display:
-            return
-        nodes_to_set = []
-        for full_value in nodes_display:
-            node = Node.objects.filter(full_value=full_value).first()
-            if node:
-                nodes_to_set.append(node)
-            else:
-                node = Node.create_node_by_full_value(full_value)
-            nodes_to_set.append(node)
-        instance.nodes.set(nodes_to_set)
-
     def create(self, validated_data):
         self.compatible_with_old_protocol(validated_data)
-        nodes_display = validated_data.pop('nodes_display', '')
         instance = super().create(validated_data)
-        self.perform_nodes_display_create(instance, nodes_display)
         return instance
 
     def update(self, instance, validated_data):
-        nodes_display = validated_data.pop('nodes_display', '')
         self.compatible_with_old_protocol(validated_data)
-        instance = super().update(instance, validated_data)
-        self.perform_nodes_display_create(instance, nodes_display)
-        return instance
+        return super().update(instance, validated_data)
 
 
 class AssetDisplaySerializer(AssetSerializer):
@@ -168,7 +150,7 @@ class AssetDisplaySerializer(AssetSerializer):
 
     @classmethod
     def setup_eager_loading(cls, queryset):
-        queryset = super().setup_eager_loading(queryset)
+        """ Perform necessary eager loading of data. """
         queryset = queryset\
             .annotate(admin_user_username=F('admin_user__username'))
         return queryset
@@ -177,14 +159,6 @@ class AssetDisplaySerializer(AssetSerializer):
 class PlatformSerializer(serializers.ModelSerializer):
     meta = serializers.DictField(required=False, allow_null=True)
 
-    def __init__(self, *args, **kwargs):
-        super().__init__(*args, **kwargs)
-
-        # TODO 修复 drf SlugField RegexValidator bug，之后记得删除
-        validators = self.fields['name'].validators
-        if isinstance(validators[-1], RegexValidator):
-            validators.pop()
-
     class Meta:
         model = Platform
         fields = [
@@ -212,6 +186,3 @@ class AssetTaskSerializer(serializers.Serializer):
     )
     task = serializers.CharField(read_only=True)
     action = serializers.ChoiceField(choices=ACTION_CHOICES, write_only=True)
-    assets = serializers.PrimaryKeyRelatedField(
-        queryset=Asset.objects, required=False, allow_empty=True, many=True
-    )

apps/assets/serializers/asset_user.py

@@ -4,7 +4,7 @@
 from django.utils.translation import ugettext as _
 from rest_framework import serializers
 
-from common.drf.serializers import AdaptedBulkListSerializer
+from common.serializers import AdaptedBulkListSerializer
 from orgs.mixins.serializers import BulkOrgResourceModelSerializer
 from ..models import AuthBook, Asset
 from ..backends import AssetUserManager

apps/assets/serializers/cmd_filter.py

@@ -3,7 +3,8 @@
 import re
 from rest_framework import serializers
 
-from common.drf.serializers import AdaptedBulkListSerializer
+from common.fields import ChoiceDisplayField
+from common.serializers import AdaptedBulkListSerializer
 from ..models import CommandFilter, CommandFilterRule, SystemUser
 from orgs.mixins.serializers import BulkOrgResourceModelSerializer
 
@@ -25,6 +26,7 @@ class CommandFilterSerializer(BulkOrgResourceModelSerializer):
 
 
 class CommandFilterRuleSerializer(BulkOrgResourceModelSerializer):
+    # serializer_choice_field = ChoiceDisplayField
     invalid_pattern = re.compile(r'[\.\*\+\[\\\?\{\}\^\$\|\(\)\#\<\>]')
     type_display = serializers.ReadOnlyField(source='get_type_display')
     action_display = serializers.ReadOnlyField(source='get_action_display')

apps/assets/serializers/domain.py

@@ -1,19 +1,17 @@
 # -*- coding: utf-8 -*-
 #
 from rest_framework import serializers
-from django.utils.translation import ugettext_lazy as _
 
-from common.drf.serializers import AdaptedBulkListSerializer
+from common.serializers import AdaptedBulkListSerializer
 from orgs.mixins.serializers import BulkOrgResourceModelSerializer
-from common.validators import NoSpecialChars
+
 from ..models import Domain, Gateway
 from .base import AuthSerializerMixin
 
 
 class DomainSerializer(BulkOrgResourceModelSerializer):
-    asset_count = serializers.SerializerMethodField(label=_('Assets count'))
-    application_count = serializers.SerializerMethodField(label=_('Applications count'))
-    gateway_count = serializers.SerializerMethodField(label=_('Gateways count'))
+    asset_count = serializers.SerializerMethodField()
+    gateway_count = serializers.SerializerMethodField()
 
     class Meta:
         model = Domain
@@ -22,12 +20,12 @@ class DomainSerializer(BulkOrgResourceModelSerializer):
             'comment', 'date_created'
         ]
         fields_m2m = [
-            'asset_count', 'assets', 'application_count', 'gateway_count',
+            'asset_count', 'assets', 'gateway_count',
         ]
         fields = fields_small + fields_m2m
         read_only_fields = ('asset_count', 'gateway_count', 'date_created')
         extra_kwargs = {
-            'assets': {'required': False, 'label': _('Assets')},
+            'assets': {'required': False}
         }
         list_serializer_class = AdaptedBulkListSerializer
 
@@ -35,10 +33,6 @@ class DomainSerializer(BulkOrgResourceModelSerializer):
     def get_asset_count(obj):
         return obj.assets.count()
 
-    @staticmethod
-    def get_application_count(obj):
-        return obj.applications.count()
-
     @staticmethod
     def get_gateway_count(obj):
         return obj.gateway_set.all().count()
@@ -53,9 +47,6 @@ class GatewaySerializer(AuthSerializerMixin, BulkOrgResourceModelSerializer):
             'private_key', 'public_key', 'domain', 'is_active', 'date_created',
             'date_updated', 'created_by', 'comment',
         ]
-        extra_kwargs = {
-            'password': {'validators': [NoSpecialChars()]}
-        }
 
     def __init__(self, *args, **kwargs):
         super().__init__(*args, **kwargs)

apps/assets/serializers/favorite_asset.py

@@ -4,7 +4,7 @@
 from rest_framework import serializers
 
 from orgs.utils import tmp_to_root_org
-from common.drf.serializers import AdaptedBulkListSerializer
+from common.serializers import AdaptedBulkListSerializer
 from common.mixins import BulkSerializerMixin
 from ..models import FavoriteAsset
 

apps/assets/serializers/label.py

@@ -2,7 +2,7 @@
 #
 from rest_framework import serializers
 
-from common.drf.serializers import AdaptedBulkListSerializer
+from common.serializers import AdaptedBulkListSerializer
 from orgs.mixins.serializers import BulkOrgResourceModelSerializer
 
 from ..models import Label

apps/assets/serializers/node.py

@@ -25,9 +25,6 @@ class NodeSerializer(BulkOrgResourceModelSerializer):
         read_only_fields = ['key', 'org_id']
 
     def validate_value(self, data):
-        if '/' in data:
-            error = _("Can't contains: " + "/")
-            raise serializers.ValidationError(error)
         if self.instance:
             instance = self.instance
             siblings = instance.get_siblings()

apps/assets/serializers/system_user.py

@@ -2,10 +2,11 @@ from rest_framework import serializers
 from django.utils.translation import ugettext_lazy as _
 from django.db.models import Count
 
-from common.drf.serializers import AdaptedBulkListSerializer
+from common.serializers import AdaptedBulkListSerializer
 from common.mixins.serializers import BulkSerializerMixin
 from common.utils import ssh_pubkey_gen
 from orgs.mixins.serializers import BulkOrgResourceModelSerializer
+from assets.models import Node
 from ..models import SystemUser, Asset
 from .base import AuthSerializerMixin
 
@@ -32,20 +33,17 @@ class SystemUserSerializer(AuthSerializerMixin, BulkOrgResourceModelSerializer):
             'login_mode', 'login_mode_display',
             'priority', 'username_same_with_user',
             'auto_push', 'cmd_filters', 'sudo', 'shell', 'comment',
-            'auto_generate_key', 'sftp_root', 'token',
-            'assets_amount', 'date_created', 'created_by',
-            'home', 'system_groups', 'ad_domain'
+            'auto_generate_key', 'sftp_root',
+            'assets_amount', 'date_created', 'created_by'
         ]
         extra_kwargs = {
             'password': {"write_only": True},
             'public_key': {"write_only": True},
             'private_key': {"write_only": True},
-            'token': {"write_only": True},
-            'nodes_amount': {'label': _('Nodes amount')},
-            'assets_amount': {'label': _('Assets amount')},
+            'nodes_amount': {'label': _('Node')},
+            'assets_amount': {'label': _('Asset')},
             'login_mode_display': {'label': _('Login mode display')},
             'created_by': {'read_only': True},
-            'ad_domain': {'required': False, 'allow_blank': True, 'label': _('Ad domain')},
         }
 
     def validate_auto_push(self, value):
@@ -145,28 +143,16 @@ class SystemUserSerializer(AuthSerializerMixin, BulkOrgResourceModelSerializer):
 
 
 class SystemUserListSerializer(SystemUserSerializer):
-
     class Meta(SystemUserSerializer.Meta):
         fields = [
             'id', 'name', 'username', 'protocol',
-            'password', 'public_key', 'private_key',
             'login_mode', 'login_mode_display',
             'priority', "username_same_with_user",
             'auto_push', 'sudo', 'shell', 'comment',
-            "assets_amount", 'home', 'system_groups',
-            'auto_generate_key', 'ad_domain',
+            "assets_amount",
+            'auto_generate_key',
             'sftp_root',
         ]
-        extra_kwargs = {
-            'password': {"write_only": True},
-            'public_key': {"write_only": True},
-            'private_key': {"write_only": True},
-            'nodes_amount': {'label': _('Nodes amount')},
-            'assets_amount': {'label': _('Assets amount')},
-            'login_mode_display': {'label': _('Login mode display')},
-            'created_by': {'read_only': True},
-            'ad_domain': {'label': _('Ad domain')},
-        }
 
     @classmethod
     def setup_eager_loading(cls, queryset):
@@ -183,8 +169,7 @@ class SystemUserWithAuthInfoSerializer(SystemUserSerializer):
             'login_mode', 'login_mode_display',
             'priority', 'username_same_with_user',
             'auto_push', 'sudo', 'shell', 'comment',
-            'auto_generate_key', 'sftp_root', 'token',
-            'ad_domain',
+            'auto_generate_key', 'sftp_root',
         ]
         extra_kwargs = {
             'nodes_amount': {'label': _('Node')},
@@ -234,8 +219,15 @@ class SystemUserNodeRelationSerializer(RelationMixin, serializers.ModelSerialize
             'id', 'node', "node_display",
         ]
 
+    def __init__(self, *args, **kwargs):
+        super().__init__(*args, **kwargs)
+        self.tree = Node.tree()
+
     def get_node_display(self, obj):
-        return obj.node.full_value
+        if hasattr(obj, 'node_key'):
+            return self.tree.get_node_full_tag(obj.node_key)
+        else:
+            return obj.node.full_value
 
 
 class SystemUserUserRelationSerializer(RelationMixin, serializers.ModelSerializer):
@@ -257,8 +249,4 @@ class SystemUserTaskSerializer(serializers.Serializer):
     asset = serializers.PrimaryKeyRelatedField(
         queryset=Asset.objects, allow_null=True, required=False, write_only=True
     )
-    assets = serializers.PrimaryKeyRelatedField(
-        queryset=Asset.objects, allow_null=True, required=False, write_only=True,
-        many=True
-    )
     task = serializers.CharField(read_only=True)

apps/assets/signals_handler.py

@@ -1,20 +1,17 @@
 # -*- coding: utf-8 -*-
 #
-from operator import add, sub
-
-from assets.utils import is_asset_exists_in_node
+from collections import defaultdict
 from django.db.models.signals import (
-    post_save, m2m_changed, pre_delete, post_delete, pre_save
+    post_save, m2m_changed, post_delete
 )
-from django.db.models import Q, F
+from django.db.models.aggregates import Count
 from django.dispatch import receiver
 
-from common.exceptions import M2MReverseNotAllowed
-from common.const.signals import PRE_ADD, POST_ADD, POST_REMOVE, PRE_CLEAR, PRE_REMOVE
 from common.utils import get_logger
 from common.decorator import on_transaction_commit
-from .models import Asset, SystemUser, Node, compute_parent_key
-from users.models import User
+from orgs.utils import tmp_to_root_org
+from .models import Asset, SystemUser, Node, AuthBook
+from .utils import TreeService
 from .tasks import (
     update_assets_hardware_info_util,
     test_asset_connectivity_util,
@@ -37,11 +34,6 @@ def test_asset_conn_on_created(asset):
     test_asset_connectivity_util.delay([asset])
 
 
-@receiver(pre_save, sender=Node)
-def on_node_pre_save(sender, instance: Node, **kwargs):
-    instance.parent_key = instance.compute_parent_key()
-
-
 @receiver(post_save, sender=Asset)
 @on_transaction_commit
 def on_asset_created_or_update(sender, instance=None, created=False, **kwargs):
@@ -62,9 +54,17 @@ def on_asset_created_or_update(sender, instance=None, created=False, **kwargs):
             instance.nodes.add(Node.org_root())
 
 
+@receiver(post_delete, sender=Asset)
+def on_asset_delete(sender, instance=None, **kwargs):
+    """
+    当资产删除时，刷新节点，节点中存在节点和资产的关系
+    """
+    logger.debug("Asset delete signal recv: {}".format(instance))
+    Node.refresh_assets()
+
+
 @receiver(post_save, sender=SystemUser, dispatch_uid="jms")
-@on_transaction_commit
-def on_system_user_update(instance: SystemUser, created, **kwargs):
+def on_system_user_update(sender, instance=None, created=True, **kwargs):
     """
     当系统用户更新时，可能更新了秘钥，用户名等，这时要自动推送系统用户到资产上,
     其实应该当 用户名，密码，秘钥 sudo等更新时再推送，这里偷个懒,
@@ -74,57 +74,53 @@ def on_system_user_update(instance: SystemUser, created, **kwargs):
     if instance and not created:
         logger.info("System user update signal recv: {}".format(instance))
         assets = instance.assets.all().valid()
-        push_system_user_to_assets.delay(instance.id, [_asset.id for _asset in assets])
+        push_system_user_to_assets.delay(instance, assets)
 
 
 @receiver(m2m_changed, sender=SystemUser.assets.through)
-@on_transaction_commit
-def on_system_user_assets_change(instance, action, model, pk_set, **kwargs):
+def on_system_user_assets_change(sender, instance=None, action='', model=None, pk_set=None, **kwargs):
     """
     当系统用户和资产关系发生变化时，应该重新推送系统用户到新添加的资产中
     """
-    if action != POST_ADD:
+    if action != "post_add":
         return
     logger.debug("System user assets change signal recv: {}".format(instance))
+    queryset = model.objects.filter(pk__in=pk_set)
     if model == Asset:
-        system_users_id = [instance.id]
-        assets_id = pk_set
+        system_users = [instance]
+        assets = queryset
     else:
-        system_users_id = pk_set
-        assets_id = [instance.id]
-    for system_user_id in system_users_id:
-        push_system_user_to_assets.delay(system_user_id, assets_id)
+        system_users = queryset
+        assets = [instance]
+    for system_user in system_users:
+        push_system_user_to_assets.delay(system_user, assets)
 
 
 @receiver(m2m_changed, sender=SystemUser.users.through)
-@on_transaction_commit
-def on_system_user_users_change(sender, instance: SystemUser, action, model, pk_set, reverse, **kwargs):
+def on_system_user_users_change(sender, instance=None, action='', model=None, pk_set=None, **kwargs):
     """
     当系统用户和用户关系发生变化时，应该重新推送系统用户资产中
     """
-    if action != POST_ADD:
+    if action != "post_add":
         return
-
-    if reverse:
-        raise M2MReverseNotAllowed
-
     if not instance.username_same_with_user:
         return
-
     logger.debug("System user users change signal recv: {}".format(instance))
-    usernames = model.objects.filter(pk__in=pk_set).values_list('username', flat=True)
-
-    for username in usernames:
-        push_system_user_to_assets_manual.delay(instance, username)
+    queryset = model.objects.filter(pk__in=pk_set)
+    if model == SystemUser:
+        system_users = queryset
+    else:
+        system_users = [instance]
+    for s in system_users:
+        push_system_user_to_assets_manual.delay(s)
 
 
 @receiver(m2m_changed, sender=SystemUser.nodes.through)
-@on_transaction_commit
 def on_system_user_nodes_change(sender, instance=None, action=None, model=None, pk_set=None, **kwargs):
     """
     当系统用户和节点关系发生变化时，应该将节点下资产关联到新的系统用户上
     """
-    if action != POST_ADD:
+    if action != "post_add":
         return
     logger.info("System user nodes update signal recv: {}".format(instance))
 
@@ -139,217 +135,102 @@ def on_system_user_nodes_change(sender, instance=None, action=None, model=None,
 
 
 @receiver(m2m_changed, sender=SystemUser.groups.through)
-def on_system_user_groups_change(instance, action, pk_set, reverse, **kwargs):
+def on_system_user_groups_change(sender, instance=None, action=None, model=None,
+                                 pk_set=None, reverse=False, **kwargs):
     """
     当系统用户和用户组关系发生变化时，应该将组下用户关联到新的系统用户上
     """
-    if action != POST_ADD:
+    if action != "post_add" or reverse:
         return
-    if reverse:
-        raise M2MReverseNotAllowed
     logger.info("System user groups update signal recv: {}".format(instance))
-
-    users = User.objects.filter(groups__id__in=pk_set).distinct()
-    instance.users.add(*users)
+    groups = model.objects.filter(pk__in=pk_set).annotate(users_count=Count("users"))
+    users = groups.filter(users_count__gt=0).values_list('users', flat=True)
+    instance.users.add(*tuple(users))
 
 
 @receiver(m2m_changed, sender=Asset.nodes.through)
-def on_asset_nodes_add(instance, action, reverse, pk_set, **kwargs):
+def on_asset_nodes_change(sender, instance=None, action='', **kwargs):
     """
-    本操作共访问 4 次数据库
+    资产节点发生变化时，刷新节点
+    """
+    if action.startswith('post'):
+        logger.debug("Asset nodes change signal recv: {}".format(instance))
+        Node.refresh_assets()
+        with tmp_to_root_org():
+            Node.refresh_assets()
 
+
+@receiver(m2m_changed, sender=Asset.nodes.through)
+def on_asset_nodes_add(sender, instance=None, action='', model=None, pk_set=None, **kwargs):
+    """
     当资产的节点发生变化时，或者 当节点的资产关系发生变化时，
     节点下新增的资产，添加到节点关联的系统用户中
     """
-    if action != POST_ADD:
+    if action != "post_add":
         return
     logger.debug("Assets node add signal recv: {}".format(action))
-    if reverse:
-        nodes = [instance.key]
-        asset_ids = pk_set
+    if model == Node:
+        nodes = model.objects.filter(pk__in=pk_set).values_list('key', flat=True)
+        assets = [instance.id]
     else:
-        nodes = Node.objects.filter(pk__in=pk_set).values_list('key', flat=True)
-        asset_ids = [instance.id]
-
+        nodes = [instance.key]
+        assets = model.objects.filter(pk__in=pk_set).values_list('id', flat=True)
     # 节点资产发生变化时，将资产关联到节点及祖先节点关联的系统用户, 只关注新增的
     nodes_ancestors_keys = set()
+    node_tree = TreeService.new()
     for node in nodes:
-        nodes_ancestors_keys.update(Node.get_node_ancestor_keys(node, with_self=True))
+        ancestors_keys = node_tree.ancestors_ids(nid=node)
+        nodes_ancestors_keys.update(ancestors_keys)
+    system_users = SystemUser.objects.filter(nodes__key__in=nodes_ancestors_keys)
 
-    # 查询所有祖先节点关联的系统用户，都是要跟资产建立关系的
-    system_user_ids = SystemUser.objects.filter(
-        nodes__key__in=nodes_ancestors_keys
-    ).distinct().values_list('id', flat=True)
-
-    # 查询所有已存在的关系
-    m2m_model = SystemUser.assets.through
-    exist = set(m2m_model.objects.filter(
-        systemuser_id__in=system_user_ids, asset_id__in=asset_ids
-    ).values_list('systemuser_id', 'asset_id'))
-    # TODO 优化
-    to_create = []
-    for system_user_id in system_user_ids:
-        asset_ids_to_push = []
-        for asset_id in asset_ids:
-            if (system_user_id, asset_id) in exist:
-                continue
-            asset_ids_to_push.append(asset_id)
-            to_create.append(m2m_model(
-                systemuser_id=system_user_id,
-                asset_id=asset_id
-            ))
-        push_system_user_to_assets.delay(system_user_id, asset_ids_to_push)
-    m2m_model.objects.bulk_create(to_create)
-
-
-def _update_node_assets_amount(node: Node, asset_pk_set: set, operator=add):
-    """
-    一个节点与多个资产关系变化时，更新计数
-
-    :param node: 节点实例
-    :param asset_pk_set: 资产的`id`集合, 内部不会修改该值
-    :param operator: 操作
-    * -> Node
-    # -> Asset
-
-           * [3]
-          / \
-         *   * [2]
-        /     \
-       *       * [1]
-      /       / \
-     *   [a] #  # [b]
-
-    """
-    # 获取节点[1]祖先节点的 `key` 含自己，也就是[1, 2, 3]节点的`key`
-    ancestor_keys = node.get_ancestor_keys(with_self=True)
-    ancestors = Node.objects.filter(key__in=ancestor_keys).order_by('-key')
-    to_update = []
-    for ancestor in ancestors:
-        # 迭代祖先节点的`key`，顺序是 [1] -> [2] -> [3]
-        # 查询该节点及其后代节点是否包含要操作的资产，将包含的从要操作的
-        # 资产集合中去掉，他们是重复节点，无论增加或删除都不会影响节点的资产数量
-
-        asset_pk_set -= set(Asset.objects.filter(
-            id__in=asset_pk_set
-        ).filter(
-            Q(nodes__key__istartswith=f'{ancestor.key}:') |
-            Q(nodes__key=ancestor.key)
-        ).distinct().values_list('id', flat=True))
-        if not asset_pk_set:
-            # 要操作的资产集合为空，说明都是重复资产，不用改变节点资产数量
-            # 而且既然它包含了，它的祖先节点肯定也包含了，所以祖先节点都不用
-            # 处理了
-            break
-        ancestor.assets_amount = operator(F('assets_amount'), len(asset_pk_set))
-        to_update.append(ancestor)
-    Node.objects.bulk_update(to_update, fields=('assets_amount', 'parent_key'))
-
-
-def _remove_ancestor_keys(ancestor_key, tree_set):
-    # 这里判断 `ancestor_key` 不能是空，防止数据错误导致的死循环
-    # 判断是否在集合里，来区分是否已被处理过
-    while ancestor_key and ancestor_key in tree_set:
-        tree_set.remove(ancestor_key)
-        ancestor_key = compute_parent_key(ancestor_key)
-
-
-def _update_nodes_asset_amount(node_keys, asset_pk, operator):
-    """
-    一个资产与多个节点关系变化时，更新计数
-
-    :param node_keys: 节点 id 的集合
-    :param asset_pk: 资产 id
-    :param operator: 操作
-    """
-
-    # 所有相关节点的祖先节点，组成一棵局部树
-    ancestor_keys = set()
-    for key in node_keys:
-        ancestor_keys.update(Node.get_node_ancestor_keys(key))
-
-    # 相关节点可能是其他相关节点的祖先节点，如果是从相关节点里干掉
-    node_keys -= ancestor_keys
-
-    to_update_keys = []
-    for key in node_keys:
-        # 遍历相关节点，处理它及其祖先节点
-        # 查询该节点是否包含待处理资产
-        exists = is_asset_exists_in_node(asset_pk, key)
-        parent_key = compute_parent_key(key)
-
-        if exists:
-            # 如果资产在该节点，那么他及其祖先节点都不用处理
-            _remove_ancestor_keys(parent_key, ancestor_keys)
-            continue
-        else:
-            # 不存在，要更新本节点
-            to_update_keys.append(key)
-            # 这里判断 `parent_key` 不能是空，防止数据错误导致的死循环
-            # 判断是否在集合里，来区分是否已被处理过
-            while parent_key and parent_key in ancestor_keys:
-                exists = is_asset_exists_in_node(asset_pk, parent_key)
-                if exists:
-                    _remove_ancestor_keys(parent_key, ancestor_keys)
-                    break
-                else:
-                    to_update_keys.append(parent_key)
-                    ancestor_keys.remove(parent_key)
-                    parent_key = compute_parent_key(parent_key)
-
-    Node.objects.filter(key__in=to_update_keys).update(
-        assets_amount=operator(F('assets_amount'), 1)
-    )
+    system_users_assets = defaultdict(set)
+    for system_user in system_users:
+        system_users_assets[system_user].update(set(assets))
+    for system_user, _assets in system_users_assets.items():
+        system_user.assets.add(*tuple(_assets))
 
 
 @receiver(m2m_changed, sender=Asset.nodes.through)
-def update_nodes_assets_amount(action, instance, reverse, pk_set, **kwargs):
-    # 不允许 `pre_clear` ，因为该信号没有 `pk_set`
-    # [官网](https://docs.djangoproject.com/en/3.1/ref/signals/#m2m-changed)
-    refused = (PRE_CLEAR,)
-    if action in refused:
-        raise ValueError
+def on_asset_nodes_remove(sender, instance=None, action='', model=None,
+                          pk_set=None, **kwargs):
 
-    mapper = {
-        PRE_ADD: add,
-        POST_REMOVE: sub
-    }
-    if action not in mapper:
+    """
+    监控资产删除节点关系, 或节点删除资产，避免产生游离资产
+    """
+    if action not in ["post_remove", "pre_clear", "post_clear"]:
         return
-
-    operator = mapper[action]
-
-    if reverse:
-        node: Node = instance
-        asset_pk_set = set(pk_set)
-        _update_node_assets_amount(node, asset_pk_set, operator)
+    if action == "pre_clear":
+        if model == Node:
+            instance._nodes = list(instance.nodes.all())
+        else:
+            instance._assets = list(instance.assets.all())
+        return
+    logger.debug("Assets node remove signal recv: {}".format(action))
+    if action == "post_remove":
+        queryset = model.objects.filter(pk__in=pk_set)
     else:
-        asset_pk = instance.id
-        # 与资产直接关联的节点
-        node_keys = set(Node.objects.filter(id__in=pk_set).values_list('key', flat=True))
-        _update_nodes_asset_amount(node_keys, asset_pk, operator)
+        if model == Node:
+            queryset = instance._nodes
+        else:
+            queryset = instance._assets
+    if model == Node:
+        assets = [instance]
+    else:
+        assets = queryset
+    if isinstance(assets, list):
+        assets_not_has_node = []
+        for asset in assets:
+            if asset.nodes.all().count() == 0:
+                assets_not_has_node.append(asset.id)
+    else:
+        assets_not_has_node = assets.annotate(nodes_count=Count('nodes'))\
+            .filter(nodes_count=0).values_list('id', flat=True)
+    Node.org_root().assets.add(*tuple(assets_not_has_node))
 
 
-RELATED_NODE_IDS = '_related_node_ids'
-
-
-@receiver(pre_delete, sender=Asset)
-def on_asset_delete(instance: Asset, using, **kwargs):
-    node_ids = set(Node.objects.filter(
-        assets=instance
-    ).distinct().values_list('id', flat=True))
-    setattr(instance, RELATED_NODE_IDS, node_ids)
-    m2m_changed.send(
-        sender=Asset.nodes.through, instance=instance, reverse=False,
-        model=Node, pk_set=node_ids, using=using, action=PRE_REMOVE
-    )
-
-
-@receiver(post_delete, sender=Asset)
-def on_asset_post_delete(instance: Asset, using, **kwargs):
-    node_ids = getattr(instance, RELATED_NODE_IDS, None)
-    if node_ids:
-        m2m_changed.send(
-            sender=Asset.nodes.through, instance=instance, reverse=False,
-            model=Node, pk_set=node_ids, using=using, action=POST_REMOVE
-        )
+@receiver([post_save, post_delete], sender=Node)
+def on_node_update_or_created(sender, **kwargs):
+    # 刷新节点
+    Node.refresh_nodes()
+    with tmp_to_root_org():
+        Node.refresh_nodes()

apps/assets/tasks/__init__.py

@@ -9,4 +9,3 @@ from .gather_asset_users import *
 from .gather_asset_hardware_info import *
 from .push_system_user import *
 from .system_user_connectivity import *
-from .nodes_amount import *

apps/assets/tasks/asset_connectivity.py

@@ -14,7 +14,7 @@ from .utils import clean_ansible_task_hosts, group_asset_by_platform
 logger = get_logger(__file__)
 __all__ = [
     'test_asset_connectivity_util', 'test_asset_connectivity_manual',
-    'test_node_assets_connectivity_manual', 'test_assets_connectivity_manual',
+    'test_node_assets_connectivity_manual',
 ]
 
 
@@ -82,17 +82,6 @@ def test_asset_connectivity_manual(asset):
         return True, ""
 
 
-@shared_task(queue="ansible")
-def test_assets_connectivity_manual(assets):
-    task_name = _("Test assets connectivity: {}").format([asset.hostname for asset in assets])
-    summary = test_asset_connectivity_util(assets, task_name=task_name)
-
-    if summary.get('dark'):
-        return False, summary['dark']
-    else:
-        return True, ""
-
-
 @shared_task(queue="ansible")
 def test_node_assets_connectivity_manual(node):
     task_name = _("Test if the assets under the node are connectable: {}".format(node.name))