fix(assets): 修复网关信息没有密码的bug

Merge dev to master

Dockerfile

@@ -23,6 +23,7 @@ RUN sed -i 's/deb.debian.org/mirrors.aliyun.com/g' /etc/apt/sources.list \
     && sed -i 's/security.debian.org/mirrors.aliyun.com/g' /etc/apt/sources.list \
     && apt update \
     && grep -v '^#' ./requirements/deb_buster_requirements.txt | xargs apt -y install \
+    && rm -rf /var/lib/apt/lists/* \
     && localedef -c -f UTF-8 -i zh_CN zh_CN.UTF-8 \
     && cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime
 

README.md

@@ -1,121 +1,18 @@
 # JumpServer 多云环境下更好用的堡垒机
 
-[![Python3](https://img.shields.io/badge/python-3.6-green.svg?style=plastic)](https://www.python.org/)
-[![Django](https://img.shields.io/badge/django-2.2-brightgreen.svg?style=plastic)](https://www.djangoproject.com/)
+[![License](https://shields.io/github/license/jumpserver/jumpserver)](https://www.gnu.org/licenses/old-licenses/gpl-2.0.html)
+[![Release Downloads](https://shields.io/github/downloads/jumpserver/jumpserver/total)](https://github.com/jumpserver/jumpserver/releases)
 [![Docker Pulls](https://img.shields.io/docker/pulls/jumpserver/jms_all.svg)](https://hub.docker.com/u/jumpserver)
 
 - [ENGLISH](https://github.com/jumpserver/jumpserver/blob/master/README_EN.md)
 
-## 紧急BUG修复通知
-JumpServer发现远程执行漏洞，请速度修复
 
-非常感谢  **reactivity of Alibaba Hackerone bug bounty program**(瑞典) 向我们报告了此 BUG
-
-**影响版本:**
-```
-< v2.6.2
-< v2.5.4
-< v2.4.5 
-= v1.5.9
->= v1.5.3
-```
-**安全版本:**
-```
->= v2.6.2
->= v2.5.4
->= v2.4.5 
-= v1.5.9 （版本号没变）
-< v1.5.3
-```
-
-**修复方案:**
-
-将JumpServer升级至安全版本；
-
-**临时修复方案:**
-
-修改 Nginx 配置文件屏蔽漏洞接口 
-
-```
-/api/v1/authentication/connection-token/
-/api/v1/users/connection-token/
-```
-
-Nginx 配置文件位置
-```
-# 社区老版本
-/etc/nginx/conf.d/jumpserver.conf
-
-# 企业老版本
-jumpserver-release/nginx/http_server.conf
- 
-# 新版本在 
-jumpserver-release/compose/config_static/http_server.conf
-```
-
-修改 Nginx 配置文件实例
-```
-### 保证在 /api 之前 和 / 之前
-location /api/v1/authentication/connection-token/ {
-   return 403;
-}
- 
-location /api/v1/users/connection-token/ {
-   return 403;
-}
-### 新增以上这些
- 
-location /api/ {
-    proxy_set_header X-Real-IP $remote_addr;
-    proxy_set_header Host $host;
-    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
-    proxy_pass http://core:8080;
-  }
- 
-...
-```
-
-修改完成后重启 nginx
-
-```
-docker方式: 
-docker restart jms_nginx
-
-nginx方式:
-systemctl restart nginx
-
-```
-
-**修复验证**
-
-```
-$ wget https://github.com/jumpserver/jumpserver/releases/download/v2.6.2/jms_bug_check.sh 
-
-# 使用方法 bash jms_bug_check.sh HOST 
-$ bash jms_bug_check.sh demo.jumpserver.org
-漏洞已修复
-```
-
-**入侵检测**
-
-下载脚本到 jumpserver 日志目录，这个目录中存在 gunicorn.log，然后执行
-
-```
-$ pwd
-/opt/jumpserver/core/logs
-
-$ ls gunicorn.log 
-gunicorn.log
-
-$ wget 'https://github.com/jumpserver/jumpserver/releases/download/v2.6.2/jms_check_attack.sh'
-$ bash jms_check_attack.sh
-系统未被入侵
-```
+|《新一代堡垒机建设指南》开放下载|
+|------------------|
+|本白皮书由JumpServer开源项目组编著而成。编写团队从企业实践和技术演进的双重视角出发，结合自身在身份与访问安全领域长期研发及落地经验组织撰写，同时积极听取行业内专家的意见和建议，在此基础上完成了本白皮书的编写任务。下载链接：https://jinshuju.net/f/E0qAl8|
 
 --------------------------
 
-JumpServer 正在寻找开发者，一起为改变世界做些贡献吧，哪怕一点点，联系我 <ibuler@fit2cloud.com> 
-
 JumpServer 是全球首款开源的堡垒机，使用 GNU GPL v2.0 开源协议，是符合 4A 规范的运维安全审计系统。
 
 JumpServer 使用 Python / Django 为主进行开发，遵循 Web 2.0 规范，配备了业界领先的 Web Terminal 方案，交互界面美观、用户体验好。
@@ -124,7 +21,6 @@ JumpServer 采纳分布式架构，支持多机房跨区域部署，支持横向
 
 改变世界，从一点点开始。
 
-> 注: [KubeOperator](https://github.com/KubeOperator/KubeOperator) 是 JumpServer 团队在 Kubernetes 领域的的又一全新力作，欢迎关注和使用。
 
 ## 特色优势
 
@@ -136,21 +32,6 @@ JumpServer 采纳分布式架构，支持多机房跨区域部署，支持横向
 - 多租户: 一套系统，多个子公司和部门同时使用；
 - 多应用支持: 数据库，Windows远程应用，Kubernetes。
 
-## 版本说明
-
-自 v2.0.0 发布后， JumpServer 版本号命名将变更为：v大版本.功能版本.Bug修复版本。比如：
-
-```
-v2.0.1 是 v2.0.0 之后的Bug修复版本；
-v2.1.0 是 v2.0.0 之后的功能版本。
-```
-
-像其它优秀开源项目一样，JumpServer 每个月会发布一个功能版本，并同时维护 3 个功能版本。比如：
-
-```
-在 v2.4 发布前，我们会同时维护 v2.1、v2.2、v2.3；
-在 v2.4 发布后，我们会同时维护 v2.2、v2.3、v2.4；v2.1 会停止维护。
-```
 
 ## 功能列表
 
@@ -180,8 +61,8 @@ v2.1.0 是 v2.0.0 之后的功能版本。
     <td>RADIUS 二次认证</td>
   </tr>
   <tr>
-    <td>登录复核（X-PACK）</td>
-    <td>用户登录行为受管理员的监管与控制</td>
+    <td>登录复核</td>
+    <td>用户登录行为受管理员的监管与控制:small_orange_diamond:</td>
   </tr>
   <tr>
     <td rowspan="11">账号管理<br>Account</td>
@@ -205,23 +86,23 @@ v2.1.0 是 v2.0.0 之后的功能版本。
     <td>密码过期设置</td>
   </tr>
   <tr>
-    <td rowspan="2">批量改密（X-PACK）</td>
-    <td>定期批量改密</td>
+    <td rowspan="2">批量改密</td>
+    <td>定期批量改密:small_orange_diamond:</td>
   </tr>
   <tr>
-    <td>多种密码策略</td>
+    <td>多种密码策略:small_orange_diamond:</td>
   </tr>
   <tr>
-    <td>多云纳管（X-PACK）</td>
-    <td>对私有云、公有云资产自动统一纳管</td>
+    <td>多云纳管 </td>
+    <td>对私有云、公有云资产自动统一纳管:small_orange_diamond:</td>
   </tr>
   <tr>
-    <td>收集用户（X-PACK）</td>
-    <td>自定义任务定期收集主机用户</td>
+    <td>收集用户 </td>
+    <td>自定义任务定期收集主机用户:small_orange_diamond:</td>
   </tr>
   <tr>
-    <td>密码匣子（X-PACK）</td>
-    <td>统一对资产主机的用户密码进行查看、更新、测试操作</td>
+    <td>密码匣子 </td>
+    <td>统一对资产主机的用户密码进行查看、更新、测试操作:small_orange_diamond:</td>
   </tr>
   <tr>
     <td rowspan="15">授权控制<br>Authorization</td>
@@ -246,7 +127,7 @@ v2.1.0 是 v2.0.0 之后的功能版本。
     <td>实现更细粒度的应用级授权</td>
   </tr>
   <tr>
-    <td>MySQL 数据库应用、RemoteApp 远程应用（X-PACK）</td>
+    <td>MySQL 数据库应用、RemoteApp 远程应用:small_orange_diamond: </td>
   </tr>
   <tr>
     <td>动作授权</td>
@@ -273,12 +154,12 @@ v2.1.0 是 v2.0.0 之后的功能版本。
     <td>实现 Web SFTP 文件管理</td>
   </tr>
   <tr>
-    <td>工单管理（X-PACK）</td>
-    <td>支持对用户登录请求行为进行控制</td>
+    <td>工单管理</td>
+    <td>支持对用户登录请求行为进行控制:small_orange_diamond:</td>
   </tr>
   <tr>
-    <td>组织管理（X-PACK）</td>
-    <td>实现多租户管理与权限隔离</td>
+    <td>组织管理</td>
+    <td>实现多租户管理与权限隔离:small_orange_diamond:</td>
   </tr>
   <tr>
     <td rowspan="7">安全审计<br>Audit</td>
@@ -297,7 +178,7 @@ v2.1.0 是 v2.0.0 之后的功能版本。
     <td>支持对 Linux、Windows 等资产操作的录像进行回放审计</td>
   </tr>
   <tr>
-    <td>支持对 RemoteApp（X-PACK）、MySQL 等应用操作的录像进行回放审计</td>
+    <td>支持对 RemoteApp:small_orange_diamond:、MySQL 等应用操作的录像进行回放审计</td>
   </tr>
   <tr>
     <td>指令审计</td>
@@ -313,7 +194,7 @@ v2.1.0 是 v2.0.0 之后的功能版本。
     <td>命令方式</td>
   </tr>
   <tr>
-    <td>Web UI方式 (X-PACK)</td>
+    <td>Web UI方式 :small_orange_diamond:</td>
   </tr>
 
   <tr>
@@ -321,13 +202,13 @@ v2.1.0 是 v2.0.0 之后的功能版本。
     <td>MySQL</td>
   </tr>
   <tr>
-    <td>Oracle (X-PACK)</td>
+    <td>Oracle :small_orange_diamond:</td>
   </tr>
   <tr>
-    <td>MariaDB (X-PACK)</td>
+    <td>MariaDB :small_orange_diamond:</td>
   </tr>
   <tr>
-    <td>PostgreSQL (X-PACK)</td>
+    <td>PostgreSQL :small_orange_diamond:</td>
   </tr>
   <tr>
     <td rowspan="6">功能亮点</td>
@@ -357,6 +238,8 @@ v2.1.0 是 v2.0.0 之后的功能版本。
   </tr>
 </table>
 
+**说明**: 带 :small_orange_diamond: 后缀的是 X-PACK 插件有的功能
+
 ## 快速开始
 
 - [极速安装](https://docs.jumpserver.org/zh/master/install/setup_by_fast/)
@@ -366,17 +249,26 @@ v2.1.0 是 v2.0.0 之后的功能版本。
 ## 组件项目
 - [Lina](https://github.com/jumpserver/lina) JumpServer Web UI 项目
 - [Luna](https://github.com/jumpserver/luna) JumpServer Web Terminal 项目
-- [Koko](https://github.com/jumpserver/koko) JumpServer 字符协议 Connector 项目，替代原来 Python 版本的 [Coco](https://github.com/jumpserver/coco)
-- [Guacamole](https://github.com/jumpserver/docker-guacamole) JumpServer 图形协议 Connector 项目，依赖 [Apache Guacamole](https://guacamole.apache.org/)
+- [KoKo](https://github.com/jumpserver/koko) JumpServer 字符协议 Connector 项目，替代原来 Python 版本的 [Coco](https://github.com/jumpserver/coco)
+- [Lion](https://github.com/jumpserver/lion-release) JumpServer 图形协议 Connector 项目，依赖 [Apache Guacamole](https://guacamole.apache.org/)
+
+## 贡献
+如果有你好的想法创意，或者帮助我们修复了 Bug, 欢迎提交 Pull Request
+
+感谢以下贡献者，让 JumpServer 更加完善
+
+<a href="https://github.com/jumpserver/jumpserver/graphs/contributors">
+  <img src="https://contrib.rocks/image?repo=jumpserver/jumpserver" />
+</a>
+
 
 ## 致谢
-- [Apache Guacamole](https://guacamole.apache.org/) Web页面连接 RDP, SSH, VNC协议设备，JumpServer 图形化连接依赖
+- [Apache Guacamole](https://guacamole.apache.org/) Web页面连接 RDP, SSH, VNC协议设备，JumpServer 图形化组件 Lion 依赖 
 - [OmniDB](https://omnidb.org/) Web页面连接使用数据库，JumpServer Web数据库依赖
 
 
 ## JumpServer 企业版 
 - [申请企业版试用](https://jinshuju.net/f/kyOYpi)
-> 注：企业版支持离线安装，申请通过后会提供高速下载链接。
 
 ## 案例研究
 

README_EN.md

@@ -1,143 +1,245 @@
-## Jumpserver 
+# Jumpserver - The Bastion Host for Multi-Cloud Environment
 
 [![Python3](https://img.shields.io/badge/python-3.6-green.svg?style=plastic)](https://www.python.org/)
 [![Django](https://img.shields.io/badge/django-2.2-brightgreen.svg?style=plastic)](https://www.djangoproject.com/)
 [![Docker Pulls](https://img.shields.io/docker/pulls/jumpserver/jms_all.svg)](https://hub.docker.com/u/jumpserver)
 
----- 
-## CRITICAL BUG WARNING
+- [中文版](https://github.com/jumpserver/jumpserver/blob/master/README.md)
 
-Recently we have found a critical bug for remote execution vulnerability which leads to pre-auth and info leak, please fix it as soon as possible.
-
-Thanks for **reactivity from Alibaba Hackerone bug bounty program** report us this bug
-
-**Vulnerable version:**
-```
-< v2.6.2
-< v2.5.4
-< v2.4.5 
-= v1.5.9
->= v1.5.3
-```
-
-**Safe and Stable version:**
-```
->= v2.6.2
->= v2.5.4
->= v2.4.5 
-= v1.5.9 （version tag didn't change）
-< v1.5.3
-```
-
-**Bug Fix Solution:**
-Upgrade to the latest version or the version mentioned above
-
-
-**Temporary Solution (upgrade asap):**
-
-Modify the Nginx config file and disable the vulnerable api listed below
-
-```
-/api/v1/authentication/connection-token/
-/api/v1/users/connection-token/
-```
-
-Path to Nginx config file
-
-```
-# Previous Community version
-/etc/nginx/conf.d/jumpserver.conf
-
-# Previous Enterprise version
-jumpserver-release/nginx/http_server.conf
- 
-# Latest version
-jumpserver-release/compose/config_static/http_server.conf
-```
-
-Changes in Nginx config file
-
-```
-### Put the following code on top of location server, or before /api and /
-location /api/v1/authentication/connection-token/ {
-   return 403;
-}
- 
-location /api/v1/users/connection-token/ {
-   return 403;
-}
-### End right here
- 
-location /api/ {
-    proxy_set_header X-Real-IP $remote_addr;
-    proxy_set_header Host $host;
-    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
-    proxy_pass http://core:8080;
-  }
- 
-...
-```
-
-Save the file and restart Nginx
-
-```
-docker deployment: 
-$ docker restart jms_nginx
-
-rpm or other deployment:
-$ systemctl restart nginx
-
-```
-
-**Bug Fix Verification**
-
-```
-# Download the following script to check if it is fixed
-$ wget https://github.com/jumpserver/jumpserver/releases/download/v2.6.2/jms_bug_check.sh 
-
-# Run the code to verify it
-$ bash jms_bug_check.sh demo.jumpserver.org
-漏洞已修复 (It means the bug is fixed)
-漏洞未修复 (It means the bug is not fixed and the system is still vulnerable)
-```
-
-
-**Attack Simulation**
-
-Go to the logs directory which should contain gunicorn.log file. Then download the "attack" script and execute it
-
-```
-$ pwd
-/opt/jumpserver/core/logs
-
-$ ls gunicorn.log
-gunicorn.log
-
-$ wget 'https://github.com/jumpserver/jumpserver/releases/download/v2.6.2/jms_check_attack.sh'
-$ bash jms_check_attack.sh
-系统未被入侵 (It means the system is safe)
-系统已被入侵 (It means the system is being attacked)
-```
+|![notification](https://raw.githubusercontent.com/goharbor/website/master/docs/img/readme/bell-outline-badged.svg)Security Notice|
+|------------------|
+|On 15th January 2021, JumpServer found a critical bug for remote execution vulnerability. Please fix it asap! [For more detail](https://github.com/jumpserver/jumpserver/issues/5533) Thanks for **reactivity of Alibaba Hackerone bug bounty program** report use the bug|
 
 --------------------------
 
-----
-
-- [中文版](https://github.com/jumpserver/jumpserver/blob/master/README.md)
-
-Jumpserver is the world's first open-source PAM (Privileged Access Management System) and is licensed under the GNU GPL v2.0. It is a 4A-compliant professional operation and maintenance security audit system.
+Jumpserver is the world's first open-source Bastion Host and is licensed under the GNU GPL v2.0. It is a 4A-compliant professional operation and maintenance security audit system.
 
 Jumpserver uses Python / Django for development, follows Web 2.0 specifications, and is equipped with an industry-leading Web Terminal solution that provides a beautiful user interface and great user experience
 
 Jumpserver adopts a distributed architecture to support multi-branch deployment across multiple cross-regional areas. The central node provides APIs, and login nodes are deployed in each branch. It can be scaled horizontally without concurrency restrictions.
 
-Change the world, starting from little things.
+Change the world by taking every little step
 
 ----
+### Advantages
 
-### Features
+- Open Source: huge transparency and free to access with quick installation process.
+- Distributed: support large-scale concurrent access with ease.
+- No Plugin required: all you need is a browser, the ultimate Web Terminal experience.
+- Multi-Cloud supported: a unified system to manage assets on different clouds at the same time
+- Cloud storage: audit records are stored in the cloud. Data lost no more!
+- Multi-Tenant system: multiple subsidiary companies or departments access the same system simultaneously.
+- Many applications supported: link to databases, windows remote applications, and Kubernetes cluster, etc.
 
- ![Jumpserver 功能](https://jumpserver-release.oss-cn-hangzhou.aliyuncs.com/Jumpserver148.jpeg "Jumpserver 功能")
+## Features List
+
+<table>
+  <tr>
+    <td rowspan="8">Authentication</td>
+    <td rowspan="5">Login</td>
+    <td>Unified way to access and authenticate resources</td>
+  </tr>
+  <tr>
+    <td>LDAP/AD Authentication</td>
+  </tr>
+  <tr>
+    <td>RADIUS Authentication</td>
+  </tr>
+  <tr>
+    <td>OpenID Authentication（Single Sign-On）</td>
+  </tr>
+  <tr>
+    <td>CAS Authentication （Single Sign-On）</td>
+  </tr>
+  <tr>
+    <td rowspan="2">MFA (Multi-Factor Authentication)</td>
+    <td>Use Google Authenticator for MFA</td>
+  </tr>
+  <tr>
+    <td>RADIUS (Remote Authentication Dial In User Service)</td>
+  </tr>
+  <tr>
+    <td>Login Supervision</td>
+    <td>Any user’s login behavior is supervised and controlled by the administrator:small_orange_diamond:</td>
+  </tr>
+  <tr>
+    <td rowspan="11">Accounting</td>
+    <td rowspan="2">Centralized Accounts Management</td>
+    <td>Admin Users management</td>
+  </tr>
+  <tr>
+    <td>System Users management</td>
+  </tr>
+  <tr>
+    <td rowspan="4">Unified Password Management</td>
+    <td>Asset password custody (a matrix storing all asset password with dense security)</td>
+  </tr>
+  <tr>
+    <td>Auto-generated passwords</td>
+  </tr>
+  <tr>
+    <td>Automatic password handling (auto login assets)</td>
+  </tr>
+  <tr>
+    <td>Password expiration settings</td>
+  </tr>
+  <tr>
+    <td rowspan="2">Password change Schedular</td>
+    <td>Support regular batch Linux/Windows assets password changing:small_orange_diamond:</td>
+  </tr>
+  <tr>
+    <td>Implement multiple password strategies:small_orange_diamond:</td>
+  </tr>
+  <tr>
+    <td>Multi-Cloud Management</td>
+    <td>Automatically manage private cloud and public cloud assets in a unified platform :small_orange_diamond:</td>
+  </tr>
+  <tr>
+    <td>Users Acquisition </td>
+    <td>Create regular custom tasks to collect system users in selected assets to identify and track the privileges ownership:small_orange_diamond:</td>
+  </tr>
+  <tr>
+    <td>Password Vault </td>
+    <td>Unified operations to check, update, and test system user password to prevent stealing or unauthorised sharing of passwords:small_orange_diamond:</td>
+  </tr>
+  <tr>
+    <td rowspan="15">Authorization</td>
+    <td>Multi-Dimensional</td>
+    <td>Granting users or user groups to access assets, asset nodes, or applications through system users. Providing precise access control to different roles of users</td>
+  </tr>
+  <tr>
+    <td rowspan="4">Assets</td>
+    <td>Assets are arranged and displayed in a tree structure </td>
+  </tr>
+  <tr>
+    <td>Assets and Nodes have immense flexibility for authorizing</td>
+  </tr>
+  <tr>
+    <td>Assets in nodes inherit authorization automatically</td>
+  </tr>
+  <tr>
+    <td>child nodes automatically inherit authorization from parent nodes</td>
+  </tr>
+  <tr>
+    <td rowspan="2">Application</td>
+    <td>Provides granular access control for privileged users on application level to protect from unauthorized access and unintentional errors</td>
+  </tr>
+  <tr>
+    <td>Database applications (MySQL, Oracle, PostgreSQL, MariaDB, etc.) and Remote App:small_orange_diamond: </td>
+  </tr>
+  <tr>
+    <td>Actions</td>
+    <td>Deeper restriction on the control of file upload, download and connection actions of authorized assets. Control the permission of clipboard copy/paste (from outer terminal to current asset)</td>
+  </tr>
+  <tr>
+    <td>Time Bound</td>
+    <td>Sharply limited the available (accessible) time for account access to the authorized resources to reduce the risk and attack surface drastically</td>
+  </tr>
+  <tr>
+    <td>Privileged Assignment</td>
+    <td>Assign the denied/allowed command lists to different system users as privilege elevation, with the latter taking the form of allowing particular commands to be run with a higher level of privileges. (Minimize insider threat)</td>
+  </tr>
+  <tr>
+    <td>Command Filtering</td>
+    <td>Creating list of restriction commands that you would like to assign to different authorized system users for filtering purpose</td>
+  </tr>
+  <tr>
+    <td>File Transfer and Management</td>
+    <td>Support SFTP file upload/download</td>
+  </tr>
+  <tr>
+    <td>File Management</td>
+    <td>Provide a Web UI for SFTP file management</td>
+  </tr>
+  <tr>
+    <td>Workflow Management</td>
+    <td>Manage user login confirmation requests and assets or applications authorization requests for Just-In-Time Privileges functionality:small_orange_diamond:</td>
+  </tr>
+  <tr>
+    <td>Group Management </td>
+    <td>Establishing a multi-tenant ecosystem that able authority isolation to keep malicious actors away from sensitive administrative backends:small_orange_diamond:</td>
+  </tr>
+  <tr>
+    <td rowspan="8">Auditing</td>
+    <td>Operations</td>
+    <td>Auditing user operation behaviors for any access or usage of given privileged accounts</td>
+  </tr>
+  <tr>
+    <td rowspan="2">Session</td>
+    <td>Support real-time session audit</td>
+  </tr>
+  <tr>
+    <td>Full history of all previous session audits</td>
+  </tr>
+  <tr>
+    <td rowspan="3">Video</td>
+    <td>Complete session audit and playback recordings on assets operation (Linux, Windows)</td>
+  </tr>
+  <tr>
+    <td>Full recordings of RemoteApp, MySQL, and Kubernetes:small_orange_diamond:</td>
+  </tr>
+  <tr>
+    <td>Supports uploading recordings to public clouds</td>
+  </tr>
+  <tr>
+    <td>Command</td>
+    <td>Command auditing on assets and applications operation. Send warning alerts when executing illegal commands</td>
+  </tr>
+  <tr>
+    <td>File Transfer</td>
+    <td>Full recordings of file upload and download</td>
+  </tr>
+  <tr>
+    <td rowspan="20">Database</td>
+    <td rowspan="2">How to connect</td>
+    <td>Command line</td>
+  </tr>
+  <tr>
+    <td>Built-in Web UI:small_orange_diamond:</td>
+  </tr>
+
+  <tr>
+    <td rowspan="4">Supported Database</td>
+    <td>MySQL</td>
+  </tr>
+  <tr>
+    <td>Oracle :small_orange_diamond:</td>
+  </tr>
+  <tr>
+    <td>MariaDB :small_orange_diamond:</td>
+  </tr>
+  <tr>
+    <td>PostgreSQL :small_orange_diamond:</td>
+  </tr>
+  <tr>
+    <td rowspan="6">Feature Highlights</td>
+    <td>Syntax highlights</td>
+  </tr>
+  <tr>
+    <td>Prettier SQL formmating</td>
+  </tr>
+  <tr>
+    <td>Support Shortcuts</td>
+  </tr>
+  <tr>
+    <td>Support selected SQL statements</td>
+  </tr>
+  <tr>
+    <td>SQL commands history query</td>
+  </tr>
+  <tr>
+    <td>Support page creation: DB, TABLE</td>
+  </tr>
+  <tr>
+    <td rowspan="2">Session Auditing</td>
+    <td>Full records of command</td>
+  </tr>
+  <tr>
+    <td>Playback videos</td>
+  </tr>
+</table>
+
+**Note**: Rows with :small_orange_diamond: at the end of the sentence means that it is X-PACK features exclusive ([Apply for X-PACK Trial](https://jinshuju.net/f/kyOYpi))
 
 ### Start 
 
@@ -162,6 +264,50 @@ We provide the SDK for your other systems to quickly interact with the Jumpserve
 - [Python](https://github.com/jumpserver/jumpserver-python-sdk) Jumpserver other components use this SDK to complete the interaction.
 - [Java](https://github.com/KaiJunYan/jumpserver-java-sdk.git) Thanks to 恺珺 for providing his Java SDK vesrion.
 
+## JumpServer Component Projects
+- [Lina](https://github.com/jumpserver/lina) JumpServer Web UI
+- [Luna](https://github.com/jumpserver/luna) JumpServer Web Terminal
+- [KoKo](https://github.com/jumpserver/koko) JumpServer Character protocaol Connector, replace original Python Version [Coco](https://github.com/jumpserver/coco)
+- [Guacamole](https://github.com/jumpserver/docker-guacamole) JumpServer Graphics protocol Connector，rely on [Apache Guacamole](https://guacamole.apache.org/)
+
+## Contribution
+If you have any good ideas or helping us to fix bugs, please submit a Pull Request and accept our thanks :)
+
+Thanks to the following contributors for making JumpServer better everyday!
+
+<a href="https://github.com/jumpserver/jumpserver/graphs/contributors">
+  <img src="https://contrib.rocks/image?repo=jumpserver/jumpserver" />
+</a>
+
+
+## Thanks to
+- [Apache Guacamole](https://guacamole.apache.org/) Web page connection RDP, SSH, VNC protocol equipment. JumpServer graphical connection dependent.
+- [OmniDB](https://omnidb.org/) Web page connection to databases. JumpServer Web database dependent.
+
+
+## JumpServer Enterprise Version 
+- [Apply for it](https://jinshuju.net/f/kyOYpi)
+
+## Case Study
+
+- [JumpServer 堡垒机护航顺丰科技超大规模资产安全运维](https://blog.fit2cloud.com/?p=1147)；
+- [JumpServer 堡垒机让“大智慧”的混合 IT 运维更智慧](https://blog.fit2cloud.com/?p=882)；
+- [携程 JumpServer 堡垒机部署与运营实战](https://blog.fit2cloud.com/?p=851)；
+- [小红书的JumpServer堡垒机大规模资产跨版本迁移之路](https://blog.fit2cloud.com/?p=516)；
+- [JumpServer堡垒机助力中手游提升多云环境下安全运维能力](https://blog.fit2cloud.com/?p=732)；
+- [中通快递：JumpServer主机安全运维实践](https://blog.fit2cloud.com/?p=708)；
+- [东方明珠：JumpServer高效管控异构化、分布式云端资产](https://blog.fit2cloud.com/?p=687)；
+- [江苏农信：JumpServer堡垒机助力行业云安全运维](https://blog.fit2cloud.com/?p=666)。
+
+## For safety instructions
+
+JumpServer is a security product. Please refer to [Basic Security Recommendations](https://docs.jumpserver.org/zh/master/install/install_security/) for deployment and installation.
+
+If you find a security problem, please contact us directly：
+
+- ibuler@fit2cloud.com 
+- support@fit2cloud.com 
+- 400-052-0755
 
 ### License & Copyright
 Copyright (c) 2014-2019 Beijing Duizhan Tech, Inc., All rights reserved.

apps/acls/admin.py

@@ -0,0 +1,3 @@
+from django.contrib import admin
+
+# Register your models here.

apps/acls/api/__init__.py

@@ -0,0 +1,3 @@
+from .login_acl import *
+from .login_asset_acl import *
+from .login_asset_check import *

apps/acls/api/login_acl.py

@@ -0,0 +1,19 @@
+from common.permissions import IsOrgAdmin, HasQueryParamsUserAndIsCurrentOrgMember
+from common.drf.api import JMSBulkModelViewSet
+from ..models import LoginACL
+from .. import serializers
+
+__all__ = ['LoginACLViewSet', ]
+
+
+class LoginACLViewSet(JMSBulkModelViewSet):
+    queryset = LoginACL.objects.all()
+    filterset_fields = ('name', 'user', )
+    search_fields = filterset_fields
+    permission_classes = (IsOrgAdmin, )
+    serializer_class = serializers.LoginACLSerializer
+
+    def get_permissions(self):
+        if self.action in ["retrieve", "list"]:
+            self.permission_classes = (IsOrgAdmin, HasQueryParamsUserAndIsCurrentOrgMember)
+        return super().get_permissions()

apps/acls/api/login_asset_acl.py

@@ -0,0 +1,15 @@
+
+from orgs.mixins.api import OrgBulkModelViewSet
+from common.permissions import IsOrgAdmin
+from .. import models, serializers
+
+
+__all__ = ['LoginAssetACLViewSet']
+
+
+class LoginAssetACLViewSet(OrgBulkModelViewSet):
+    model = models.LoginAssetACL
+    filterset_fields = ('name', )
+    search_fields = filterset_fields
+    permission_classes = (IsOrgAdmin, )
+    serializer_class = serializers.LoginAssetACLSerializer

apps/acls/api/login_asset_check.py

@@ -0,0 +1,77 @@
+from django.shortcuts import get_object_or_404
+from rest_framework.response import Response
+from rest_framework.generics import CreateAPIView, RetrieveDestroyAPIView
+
+from common.permissions import IsAppUser
+from common.utils import reverse, lazyproperty
+from orgs.utils import tmp_to_org, tmp_to_root_org
+from tickets.api import GenericTicketStatusRetrieveCloseAPI
+from ..models import LoginAssetACL
+from .. import serializers
+
+
+__all__ = ['LoginAssetCheckAPI', 'LoginAssetConfirmStatusAPI']
+
+
+class LoginAssetCheckAPI(CreateAPIView):
+    permission_classes = (IsAppUser, )
+    serializer_class = serializers.LoginAssetCheckSerializer
+
+    def create(self, request, *args, **kwargs):
+        is_need_confirm, response_data = self.check_if_need_confirm()
+        return Response(data=response_data, status=200)
+
+    def check_if_need_confirm(self):
+        queries = {
+            'user': self.serializer.user, 'asset': self.serializer.asset,
+            'system_user': self.serializer.system_user,
+            'action': LoginAssetACL.ActionChoices.login_confirm
+        }
+        with tmp_to_org(self.serializer.org):
+            acl = LoginAssetACL.filter(**queries).valid().first()
+
+        if not acl:
+            is_need_confirm = False
+            response_data = {}
+        else:
+            is_need_confirm = True
+            response_data = self._get_response_data_of_need_confirm(acl)
+        response_data['need_confirm'] = is_need_confirm
+        return is_need_confirm, response_data
+
+    def _get_response_data_of_need_confirm(self, acl):
+        ticket = LoginAssetACL.create_login_asset_confirm_ticket(
+            user=self.serializer.user,
+            asset=self.serializer.asset,
+            system_user=self.serializer.system_user,
+            assignees=acl.reviewers.all(),
+            org_id=self.serializer.org.id
+        )
+        confirm_status_url = reverse(
+            view_name='api-acls:login-asset-confirm-status',
+            kwargs={'pk': str(ticket.id)}
+        )
+        ticket_detail_url = reverse(
+            view_name='api-tickets:ticket-detail',
+            kwargs={'pk': str(ticket.id)},
+            external=True, api_to_ui=True
+        )
+        ticket_detail_url = '{url}?type={type}'.format(url=ticket_detail_url, type=ticket.type)
+        data = {
+            'check_confirm_status': {'method': 'GET', 'url': confirm_status_url},
+            'close_confirm': {'method': 'DELETE', 'url': confirm_status_url},
+            'ticket_detail_url': ticket_detail_url,
+            'reviewers': [str(user) for user in ticket.assignees.all()],
+        }
+        return data
+
+    @lazyproperty
+    def serializer(self):
+        serializer = self.get_serializer(data=self.request.data)
+        serializer.is_valid(raise_exception=True)
+        return serializer
+
+
+class LoginAssetConfirmStatusAPI(GenericTicketStatusRetrieveCloseAPI):
+    pass
+

apps/acls/apps.py

@@ -0,0 +1,5 @@
+from django.apps import AppConfig
+
+
+class AclsConfig(AppConfig):
+    name = 'acls'

apps/acls/migrations/0001_initial.py

@@ -0,0 +1,61 @@
+# Generated by Django 3.1 on 2021-03-11 09:53
+
+from django.conf import settings
+import django.core.validators
+from django.db import migrations, models
+import django.db.models.deletion
+import uuid
+
+
+class Migration(migrations.Migration):
+
+    initial = True
+
+    dependencies = [
+        migrations.swappable_dependency(settings.AUTH_USER_MODEL),
+    ]
+
+    operations = [
+        migrations.CreateModel(
+            name='LoginACL',
+            fields=[
+                ('id', models.UUIDField(default=uuid.uuid4, primary_key=True, serialize=False)),
+                ('created_by', models.CharField(blank=True, max_length=32, null=True, verbose_name='Created by')),
+                ('date_created', models.DateTimeField(auto_now_add=True, null=True, verbose_name='Date created')),
+                ('date_updated', models.DateTimeField(auto_now=True, verbose_name='Date updated')),
+                ('name', models.CharField(max_length=128, verbose_name='Name')),
+                ('priority', models.IntegerField(default=50, help_text='1-100, the lower the value will be match first', validators=[django.core.validators.MinValueValidator(1), django.core.validators.MaxValueValidator(100)], verbose_name='Priority')),
+                ('is_active', models.BooleanField(default=True, verbose_name='Active')),
+                ('comment', models.TextField(blank=True, default='', verbose_name='Comment')),
+                ('ip_group', models.JSONField(default=list, verbose_name='Login IP')),
+                ('action', models.CharField(choices=[('reject', 'Reject'), ('allow', 'Allow')], default='reject', max_length=64, verbose_name='Action')),
+                ('user', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, related_name='login_acls', to=settings.AUTH_USER_MODEL, verbose_name='User')),
+            ],
+            options={
+                'ordering': ('priority', '-date_updated', 'name'),
+            },
+        ),
+        migrations.CreateModel(
+            name='LoginAssetACL',
+            fields=[
+                ('org_id', models.CharField(blank=True, db_index=True, default='', max_length=36, verbose_name='Organization')),
+                ('id', models.UUIDField(default=uuid.uuid4, primary_key=True, serialize=False)),
+                ('created_by', models.CharField(blank=True, max_length=32, null=True, verbose_name='Created by')),
+                ('date_created', models.DateTimeField(auto_now_add=True, null=True, verbose_name='Date created')),
+                ('date_updated', models.DateTimeField(auto_now=True, verbose_name='Date updated')),
+                ('name', models.CharField(max_length=128, verbose_name='Name')),
+                ('priority', models.IntegerField(default=50, help_text='1-100, the lower the value will be match first', validators=[django.core.validators.MinValueValidator(1), django.core.validators.MaxValueValidator(100)], verbose_name='Priority')),
+                ('is_active', models.BooleanField(default=True, verbose_name='Active')),
+                ('comment', models.TextField(blank=True, default='', verbose_name='Comment')),
+                ('users', models.JSONField(verbose_name='User')),
+                ('system_users', models.JSONField(verbose_name='System User')),
+                ('assets', models.JSONField(verbose_name='Asset')),
+                ('action', models.CharField(choices=[('login_confirm', 'Login confirm')], default='login_confirm', max_length=64, verbose_name='Action')),
+                ('reviewers', models.ManyToManyField(blank=True, related_name='review_login_asset_acls', to=settings.AUTH_USER_MODEL, verbose_name='Reviewers')),
+            ],
+            options={
+                'ordering': ('priority', '-date_updated', 'name'),
+                'unique_together': {('name', 'org_id')},
+            },
+        ),
+    ]

apps/acls/models/__init__.py

@@ -0,0 +1,2 @@
+from .login_acl import *
+from .login_asset_acl import *

apps/acls/models/base.py

@@ -0,0 +1,35 @@
+from django.db import models
+from django.utils.translation import ugettext_lazy as _
+from django.core.validators import MinValueValidator, MaxValueValidator
+from common.mixins import CommonModelMixin
+
+
+__all__ = ['BaseACL', 'BaseACLQuerySet']
+
+
+class BaseACLQuerySet(models.QuerySet):
+    def active(self):
+        return self.filter(is_active=True)
+
+    def inactive(self):
+        return self.filter(is_active=False)
+
+    def valid(self):
+        return self.active()
+
+    def invalid(self):
+        return self.inactive()
+
+
+class BaseACL(CommonModelMixin):
+    name = models.CharField(max_length=128, verbose_name=_('Name'))
+    priority = models.IntegerField(
+        default=50, verbose_name=_("Priority"),
+        help_text=_("1-100, the lower the value will be match first"),
+        validators=[MinValueValidator(1), MaxValueValidator(100)]
+    )
+    is_active = models.BooleanField(default=True, verbose_name=_("Active"))
+    comment = models.TextField(default='', blank=True, verbose_name=_('Comment'))
+
+    class Meta:
+        abstract = True

apps/acls/models/login_acl.py

@@ -0,0 +1,57 @@
+
+from django.db import models
+from django.utils.translation import ugettext_lazy as _
+from .base import BaseACL, BaseACLQuerySet
+from ..utils import contains_ip
+
+
+class ACLManager(models.Manager):
+
+    def valid(self):
+        return self.get_queryset().valid()
+
+
+class LoginACL(BaseACL):
+    class ActionChoices(models.TextChoices):
+        reject = 'reject', _('Reject')
+        allow = 'allow', _('Allow')
+
+    # 条件
+    ip_group = models.JSONField(default=list, verbose_name=_('Login IP'))
+    # 动作
+    action = models.CharField(
+        max_length=64, choices=ActionChoices.choices, default=ActionChoices.reject,
+        verbose_name=_('Action')
+    )
+    # 关联
+    user = models.ForeignKey(
+        'users.User', on_delete=models.CASCADE, related_name='login_acls', verbose_name=_('User')
+    )
+
+    objects = ACLManager.from_queryset(BaseACLQuerySet)()
+
+    class Meta:
+        ordering = ('priority', '-date_updated', 'name')
+
+    def __str__(self):
+        return self.name
+
+    @property
+    def action_reject(self):
+        return self.action == self.ActionChoices.reject
+
+    @property
+    def action_allow(self):
+        return self.action == self.ActionChoices.allow
+
+    @staticmethod
+    def allow_user_to_login(user, ip):
+        acl = user.login_acls.valid().first()
+        if not acl:
+            return True
+        is_contained = contains_ip(ip, acl.ip_group)
+        if acl.action_allow and is_contained:
+            return True
+        if acl.action_reject and not is_contained:
+            return True
+        return False

apps/acls/models/login_asset_acl.py

@@ -0,0 +1,102 @@
+from django.db import models
+from django.db.models import Q
+from django.utils.translation import ugettext_lazy as _
+from orgs.mixins.models import OrgModelMixin, OrgManager
+from .base import BaseACL, BaseACLQuerySet
+from ..utils import contains_ip
+
+
+class ACLManager(OrgManager):
+
+    def valid(self):
+        return self.get_queryset().valid()
+
+
+class LoginAssetACL(BaseACL, OrgModelMixin):
+    class ActionChoices(models.TextChoices):
+        login_confirm = 'login_confirm', _('Login confirm')
+
+    # 条件
+    users = models.JSONField(verbose_name=_('User'))
+    system_users = models.JSONField(verbose_name=_('System User'))
+    assets = models.JSONField(verbose_name=_('Asset'))
+    # 动作
+    action = models.CharField(
+        max_length=64, choices=ActionChoices.choices, default=ActionChoices.login_confirm,
+        verbose_name=_('Action')
+    )
+    # 动作: 附加字段
+    # - login_confirm
+    reviewers = models.ManyToManyField(
+        'users.User', related_name='review_login_asset_acls', blank=True,
+        verbose_name=_("Reviewers")
+    )
+
+    objects = ACLManager.from_queryset(BaseACLQuerySet)()
+
+    class Meta:
+        unique_together = ('name', 'org_id')
+        ordering = ('priority', '-date_updated', 'name')
+
+    def __str__(self):
+        return self.name
+
+    @classmethod
+    def filter(cls, user, asset, system_user, action):
+        queryset = cls.objects.filter(action=action)
+        queryset = cls.filter_user(user, queryset)
+        queryset = cls.filter_asset(asset, queryset)
+        queryset = cls.filter_system_user(system_user, queryset)
+        return queryset
+
+    @classmethod
+    def filter_user(cls, user, queryset):
+        queryset = queryset.filter(
+            Q(users__username_group__contains=user.username) |
+            Q(users__username_group__contains='*')
+        )
+        return queryset
+
+    @classmethod
+    def filter_asset(cls, asset, queryset):
+        queryset = queryset.filter(
+            Q(assets__hostname_group__contains=asset.hostname) |
+            Q(assets__hostname_group__contains='*')
+        )
+        ids = [q.id for q in queryset if contains_ip(asset.ip, q.assets.get('ip_group', []))]
+        queryset = cls.objects.filter(id__in=ids)
+        return queryset
+
+    @classmethod
+    def filter_system_user(cls, system_user, queryset):
+        queryset = queryset.filter(
+            Q(system_users__name_group__contains=system_user.name) |
+            Q(system_users__name_group__contains='*')
+        ).filter(
+            Q(system_users__username_group__contains=system_user.username) |
+            Q(system_users__username_group__contains='*')
+        ).filter(
+            Q(system_users__protocol_group__contains=system_user.protocol) |
+            Q(system_users__protocol_group__contains='*')
+        )
+        return queryset
+
+    @classmethod
+    def create_login_asset_confirm_ticket(cls, user, asset, system_user, assignees, org_id):
+        from tickets.const import TicketTypeChoices
+        from tickets.models import Ticket
+        data = {
+            'title': _('Login asset confirm') + ' ({})'.format(user),
+            'type': TicketTypeChoices.login_asset_confirm,
+            'meta': {
+                'apply_login_user': str(user),
+                'apply_login_asset': str(asset),
+                'apply_login_system_user': str(system_user),
+            },
+            'org_id': org_id,
+        }
+        ticket = Ticket.objects.create(**data)
+        ticket.assignees.set(assignees)
+        ticket.open(applicant=user)
+        return ticket
+

apps/acls/serializers/__init__.py

@@ -0,0 +1,3 @@
+from .login_acl import *
+from .login_asset_acl import *
+from .login_asset_check import *

apps/acls/serializers/login_acl.py

@@ -0,0 +1,59 @@
+from django.utils.translation import ugettext as _
+from rest_framework import serializers
+from common.drf.serializers import BulkModelSerializer
+from orgs.utils import current_org
+from ..models import LoginACL
+from ..utils import is_ip_address, is_ip_network,  is_ip_segment
+
+
+__all__ = ['LoginACLSerializer', ]
+
+
+def ip_group_child_validator(ip_group_child):
+    is_valid = ip_group_child == '*' \
+               or is_ip_address(ip_group_child) \
+               or is_ip_network(ip_group_child) \
+               or is_ip_segment(ip_group_child)
+    if not is_valid:
+        error = _('IP address invalid: `{}`').format(ip_group_child)
+        raise serializers.ValidationError(error)
+
+
+class LoginACLSerializer(BulkModelSerializer):
+    ip_group_help_text = _(
+        'Format for comma-delimited string, with * indicating a match all. '
+        'Such as: '
+        '192.168.10.1, 192.168.1.0/24, 10.1.1.1-10.1.1.20, 2001:db8:2de::e13, 2001:db8:1a:1110::/64 '
+    )
+
+    ip_group = serializers.ListField(
+        default=['*'], label=_('IP'), help_text=ip_group_help_text,
+        child=serializers.CharField(max_length=1024, validators=[ip_group_child_validator])
+    )
+    user_display = serializers.ReadOnlyField(source='user.name', label=_('User'))
+    action_display = serializers.ReadOnlyField(source='get_action_display', label=_('Action'))
+
+    class Meta:
+        model = LoginACL
+        fields_mini = ['id', 'name']
+        fields_small = fields_mini + [
+            'priority', 'ip_group', 'action', 'action_display',
+            'is_active',
+            'date_created', 'date_updated',
+            'comment', 'created_by',
+        ]
+        fields_fk = ['user', 'user_display',]
+        fields = fields_small + fields_fk
+        extra_kwargs = {
+            'priority': {'default': 50},
+            'is_active': {'default': True},
+        }
+
+    @staticmethod
+    def validate_user(user):
+        if user not in current_org.get_members():
+            error = _('The user `{}` is not in the current organization: `{}`').format(
+                user, current_org
+            )
+            raise serializers.ValidationError(error)
+        return user

apps/acls/serializers/login_asset_acl.py

@@ -0,0 +1,105 @@
+from rest_framework import serializers
+from django.utils.translation import ugettext_lazy as _
+from orgs.mixins.serializers import BulkOrgResourceModelSerializer
+from assets.models import SystemUser
+from acls import models
+from orgs.models import Organization
+
+
+__all__ = ['LoginAssetACLSerializer']
+
+
+common_help_text = _('Format for comma-delimited string, with * indicating a match all. ')
+
+
+class LoginAssetACLUsersSerializer(serializers.Serializer):
+    username_group = serializers.ListField(
+        default=['*'], child=serializers.CharField(max_length=128), label=_('Username'),
+        help_text=common_help_text
+    )
+
+
+class LoginAssetACLAssestsSerializer(serializers.Serializer):
+    ip_group_help_text = _(
+        'Format for comma-delimited string, with * indicating a match all. '
+        'Such as: '
+        '192.168.10.1, 192.168.1.0/24, 10.1.1.1-10.1.1.20, 2001:db8:2de::e13, 2001:db8:1a:1110::/64 '
+        '(Domain name support)'
+    )
+
+    ip_group = serializers.ListField(
+        default=['*'], child=serializers.CharField(max_length=1024), label=_('IP'),
+        help_text=ip_group_help_text
+    )
+    hostname_group = serializers.ListField(
+        default=['*'], child=serializers.CharField(max_length=128), label=_('Hostname'),
+        help_text=common_help_text
+    )
+
+
+class LoginAssetACLSystemUsersSerializer(serializers.Serializer):
+    protocol_group_help_text = _(
+        'Format for comma-delimited string, with * indicating a match all. '
+        'Protocol options: {}'
+    )
+
+    name_group = serializers.ListField(
+        default=['*'], child=serializers.CharField(max_length=128), label=_('Name'),
+        help_text=common_help_text
+    )
+    username_group = serializers.ListField(
+        default=['*'], child=serializers.CharField(max_length=128), label=_('Username'),
+        help_text=common_help_text
+    )
+    protocol_group = serializers.ListField(
+        default=['*'], child=serializers.CharField(max_length=16), label=_('Protocol'),
+        help_text=protocol_group_help_text.format(
+            ', '.join([SystemUser.PROTOCOL_SSH, SystemUser.PROTOCOL_TELNET])
+        )
+    )
+
+    @staticmethod
+    def validate_protocol_group(protocol_group):
+        unsupported_protocols = set(protocol_group) - set(SystemUser.ASSET_CATEGORY_PROTOCOLS + ['*'])
+        if unsupported_protocols:
+            error = _('Unsupported protocols: {}').format(unsupported_protocols)
+            raise serializers.ValidationError(error)
+        return protocol_group
+
+
+class LoginAssetACLSerializer(BulkOrgResourceModelSerializer):
+    users = LoginAssetACLUsersSerializer()
+    assets = LoginAssetACLAssestsSerializer()
+    system_users = LoginAssetACLSystemUsersSerializer()
+    reviewers_amount = serializers.IntegerField(read_only=True, source='reviewers.count')
+    action_display = serializers.ReadOnlyField(source='get_action_display', label=_('Action'))
+
+    class Meta:
+        model = models.LoginAssetACL
+        fields_mini = ['id', 'name']
+        fields_small = fields_mini + [
+            'users', 'system_users', 'assets',
+            'is_active',
+            'date_created', 'date_updated',
+            'priority', 'action', 'action_display', 'comment', 'created_by', 'org_id'
+        ]
+        fields_m2m = ['reviewers', 'reviewers_amount']
+        fields = fields_small + fields_m2m
+        extra_kwargs = {
+            "reviewers": {'allow_null': False, 'required': True},
+            'priority': {'default': 50},
+            'is_active': {'default': True},
+        }
+
+    def validate_reviewers(self, reviewers):
+        org_id = self.fields['org_id'].default()
+        org = Organization.get_instance(org_id)
+        if not org:
+            error = _('The organization `{}` does not exist'.format(org_id))
+            raise serializers.ValidationError(error)
+        users = org.get_members()
+        valid_reviewers = list(set(reviewers) & set(users))
+        if not valid_reviewers:
+            error = _('None of the reviewers belong to Organization `{}`'.format(org.name))
+            raise serializers.ValidationError(error)
+        return valid_reviewers

apps/acls/serializers/login_asset_check.py

@@ -0,0 +1,71 @@
+from django.utils.translation import ugettext_lazy as _
+from rest_framework import serializers
+from orgs.utils import tmp_to_root_org
+from common.utils import get_object_or_none, lazyproperty
+from users.models import User
+from assets.models import Asset, SystemUser
+
+
+__all__ = ['LoginAssetCheckSerializer']
+
+
+class LoginAssetCheckSerializer(serializers.Serializer):
+    user_id = serializers.UUIDField(required=True, allow_null=False)
+    asset_id = serializers.UUIDField(required=True, allow_null=False)
+    system_user_id = serializers.UUIDField(required=True, allow_null=False)
+    system_user_username = serializers.CharField(max_length=128, default='')
+
+    def __init__(self, *args, **kwargs):
+        super().__init__(*args, **kwargs)
+        self.user = None
+        self.asset = None
+        self._system_user = None
+        self._system_user_username = None
+
+    def validate_user_id(self, user_id):
+        self.user = self.validate_object_exist(User, user_id)
+        return user_id
+
+    def validate_asset_id(self, asset_id):
+        self.asset = self.validate_object_exist(Asset, asset_id)
+        return asset_id
+
+    def validate_system_user_id(self, system_user_id):
+        self._system_user = self.validate_object_exist(SystemUser, system_user_id)
+        return system_user_id
+
+    def validate_system_user_username(self, system_user_username):
+        system_user_id = self.initial_data.get('system_user_id')
+        system_user = self.validate_object_exist(SystemUser, system_user_id)
+        if self._system_user.login_mode == SystemUser.LOGIN_MANUAL \
+                and not system_user.username \
+                and not system_user.username_same_with_user \
+                and not system_user_username:
+            error = 'Missing parameter: system_user_username'
+            raise serializers.ValidationError(error)
+        self._system_user_username = system_user_username
+        return system_user_username
+
+    @staticmethod
+    def validate_object_exist(model, field_id):
+        with tmp_to_root_org():
+            obj = get_object_or_none(model, pk=field_id)
+        if not obj:
+            error = '{} Model object does not exist'.format(model.__name__)
+            raise serializers.ValidationError(error)
+        return obj
+
+    @lazyproperty
+    def system_user(self):
+        if self._system_user.username_same_with_user:
+            username = self.user.username
+        elif self._system_user.login_mode == SystemUser.LOGIN_MANUAL:
+            username = self._system_user_username
+        else:
+            username = self._system_user.username
+        self._system_user.username = username
+        return self._system_user
+
+    @lazyproperty
+    def org(self):
+        return self.asset.org

apps/acls/tests.py

@@ -0,0 +1,3 @@
+from django.test import TestCase
+
+# Create your tests here.

apps/acls/urls/__init__.py

@@ -0,0 +1 @@
+from .api_urls import *

apps/acls/urls/api_urls.py

@@ -0,0 +1,18 @@
+from django.urls import path
+from rest_framework_bulk.routes import BulkRouter
+from .. import api
+
+
+app_name = 'acls'
+
+
+router = BulkRouter()
+router.register(r'login-acls', api.LoginACLViewSet, 'login-acl')
+router.register(r'login-asset-acls', api.LoginAssetACLViewSet, 'login-asset-acl')
+
+urlpatterns = [
+    path('login-asset/check/', api.LoginAssetCheckAPI.as_view(), name='login-asset-check'),
+    path('login-asset-confirm/<uuid:pk>/status/', api.LoginAssetConfirmStatusAPI.as_view(), name='login-asset-confirm-status')
+]
+
+urlpatterns += router.urls

apps/acls/utils.py

@@ -0,0 +1,68 @@
+from ipaddress import ip_network, ip_address
+
+
+def is_ip_address(address):
+    """ 192.168.10.1 """
+    try:
+        ip_address(address)
+    except ValueError:
+        return False
+    else:
+        return True
+
+
+def is_ip_network(ip):
+    """ 192.168.1.0/24 """
+    try:
+        ip_network(ip)
+    except ValueError:
+        return False
+    else:
+        return True
+
+
+def is_ip_segment(ip):
+    """ 10.1.1.1-10.1.1.20 """
+    if '-' not in ip:
+        return False
+    ip_address1, ip_address2 = ip.split('-')
+    return is_ip_address(ip_address1) and is_ip_address(ip_address2)
+
+
+def in_ip_segment(ip, ip_segment):
+    ip1, ip2 = ip_segment.split('-')
+    ip1 = int(ip_address(ip1))
+    ip2 = int(ip_address(ip2))
+    ip = int(ip_address(ip))
+    return min(ip1, ip2) <= ip <= max(ip1, ip2)
+
+
+def contains_ip(ip, ip_group):
+    """
+    ip_group:
+    [192.168.10.1, 192.168.1.0/24, 10.1.1.1-10.1.1.20, 2001:db8:2de::e13, 2001:db8:1a:1110::/64.]
+
+    """
+
+    if '*' in ip_group:
+        return True
+
+    for _ip in ip_group:
+        if is_ip_address(_ip):
+            # 192.168.10.1
+            if ip == _ip:
+                return True
+        elif is_ip_network(_ip) and is_ip_address(ip):
+            # 192.168.1.0/24
+            if ip_address(ip) in ip_network(_ip):
+                return True
+        elif is_ip_segment(_ip) and is_ip_address(ip):
+            # 10.1.1.1-10.1.1.20
+            if in_ip_segment(ip, _ip):
+                return True
+        else:
+            # is domain name
+            if ip == _ip:
+                return True
+
+    return False

apps/applications/api/mixin.py

@@ -77,8 +77,8 @@ class SerializeApplicationToTreeNodeMixin:
 
     @staticmethod
     def filter_organizations(applications):
-        organizations_id = set(applications.values_list('org_id', flat=True))
-        organizations = [Organization.get_instance(org_id) for org_id in organizations_id]
+        organization_ids = set(applications.values_list('org_id', flat=True))
+        organizations = [Organization.get_instance(org_id) for org_id in organization_ids]
         return organizations
 
     def serialize_applications_with_org(self, applications):

apps/applications/models/application.py

@@ -3,6 +3,7 @@ from django.utils.translation import ugettext_lazy as _
 
 from orgs.mixins.models import OrgModelMixin
 from common.mixins import CommonModelMixin
+from assets.models import Asset
 from .. import const
 
 
@@ -35,3 +36,35 @@ class Application(CommonModelMixin, OrgModelMixin):
     @property
     def category_remote_app(self):
         return self.category == const.ApplicationCategoryChoices.remote_app.value
+
+    def get_rdp_remote_app_setting(self):
+        from applications.serializers.attrs import get_serializer_class_by_application_type
+        if not self.category_remote_app:
+            raise ValueError(f"Not a remote app application: {self.name}")
+        serializer_class = get_serializer_class_by_application_type(self.type)
+        fields = serializer_class().get_fields()
+
+        parameters = [self.type]
+        for field_name in list(fields.keys()):
+            if field_name in ['asset']:
+                continue
+            value = self.attrs.get(field_name)
+            if not value:
+                continue
+            if field_name == 'path':
+                value = '\"%s\"' % value
+            parameters.append(str(value))
+
+        parameters = ' '.join(parameters)
+        return {
+            'program': '||jmservisor',
+            'working_directory': '',
+            'parameters': parameters
+        }
+
+    def get_remote_app_asset(self):
+        asset_id = self.attrs.get('asset')
+        if not asset_id:
+            raise ValueError("Remote App not has asset attr")
+        asset = Asset.objects.filter(id=asset_id).first()
+        return asset

apps/applications/serializers/application.py

@@ -44,15 +44,19 @@ class ApplicationSerializerMixin(serializers.Serializer):
 
 
 class ApplicationSerializer(ApplicationSerializerMixin, BulkOrgResourceModelSerializer):
-    category_display = serializers.ReadOnlyField(source='get_category_display', label=_('Category'))
-    type_display = serializers.ReadOnlyField(source='get_type_display', label=_('Type'))
+    category_display = serializers.ReadOnlyField(source='get_category_display', label=_('Category(Display)'))
+    type_display = serializers.ReadOnlyField(source='get_type_display', label=_('Type(Dispaly)'))
 
     class Meta:
         model = models.Application
-        fields = [
-            'id', 'name', 'category', 'category_display', 'type', 'type_display', 'attrs',
-            'domain', 'created_by', 'date_created', 'date_updated', 'comment'
+        fields_mini = ['id', 'name']
+        fields_small = fields_mini + [
+            'category', 'category_display', 'type', 'type_display', 'attrs',
+            'date_created', 'date_updated',
+            'created_by', 'comment'
         ]
+        fields_fk = ['domain']
+        fields = fields_small + fields_fk
         read_only_fields = [
             'created_by', 'date_created', 'date_updated', 'get_type_display',
         ]

apps/applications/serializers/attrs/application_category/remote_app.py

@@ -39,14 +39,14 @@ class RemoteAppSerializer(serializers.Serializer):
     @staticmethod
     def get_asset_info(obj):
         asset_id = obj.get('asset')
-        if not asset_id or is_uuid(asset_id):
+        if not asset_id or not is_uuid(asset_id):
             return {}
         try:
-            asset = Asset.objects.filter(id=str(asset_id)).values_list('id', 'hostname')
+            asset = Asset.objects.get(id=str(asset_id))
         except ObjectDoesNotExist as e:
             logger.error(e)
             return {}
         if not asset:
             return {}
-        asset_info = {'id': str(asset[0]), 'hostname': asset[1]}
+        asset_info = {'id': str(asset.id), 'hostname': asset.hostname}
         return asset_info

apps/applications/serializers/remote_app.py

@@ -27,31 +27,5 @@ class RemoteAppConnectionInfoSerializer(serializers.ModelSerializer):
         return obj.attrs.get('asset')
 
     @staticmethod
-    def get_parameters(obj):
-        """
-        返回Guacamole需要的RemoteApp配置参数信息中的parameters参数
-        """
-        from .attrs import get_serializer_class_by_application_type
-        serializer_class = get_serializer_class_by_application_type(obj.type)
-        fields = serializer_class().get_fields()
-
-        parameters = [obj.type]
-        for field_name in list(fields.keys()):
-            if field_name in ['asset']:
-                continue
-            value = obj.attrs.get(field_name)
-            if not value:
-                continue
-            if field_name == 'path':
-                value = '\"%s\"' % value
-            parameters.append(str(value))
-
-        parameters = ' '.join(parameters)
-        return parameters
-
-    def get_parameter_remote_app(self, obj):
-        return {
-            'program': '||jmservisor',
-            'working_directory': '',
-            'parameters': self.get_parameters(obj)
-        }
+    def get_parameter_remote_app(obj):
+        return obj.get_rdp_remote_app_setting()

apps/assets/api/admin_user.py

@@ -33,6 +33,10 @@ class AdminUserViewSet(OrgBulkModelViewSet):
     search_fields = filterset_fields
     serializer_class = serializers.AdminUserSerializer
     permission_classes = (IsOrgAdmin,)
+    serializer_classes = {
+        'default': serializers.AdminUserSerializer,
+        'retrieve': serializers.AdminUserDetailSerializer,
+    }
 
     def get_queryset(self):
         queryset = super().get_queryset()

apps/assets/api/asset.py

@@ -3,8 +3,6 @@
 from assets.api import FilterAssetByNodeMixin
 from rest_framework.viewsets import ModelViewSet
 from rest_framework.generics import RetrieveAPIView
-from rest_framework.response import Response
-from rest_framework import status
 from django.shortcuts import get_object_or_404
 
 from common.utils import get_logger, get_object_or_none

apps/assets/api/asset_user.py

@@ -10,10 +10,10 @@ from common.permissions import IsOrgAdminOrAppUser, NeedMFAVerify
 from common.utils import get_object_or_none, get_logger
 from common.mixins import CommonApiMixin
 from ..backends import AssetUserManager
-from ..models import Asset, Node, SystemUser
+from ..models import Node
 from .. import serializers
 from ..tasks import (
-    test_asset_users_connectivity_manual, push_system_user_a_asset_manual
+    test_asset_users_connectivity_manual
 )
 
 
@@ -100,12 +100,6 @@ class AssetUserViewSet(CommonApiMixin, BulkModelViewSet):
         obj = queryset.get(id=pk)
         return obj
 
-    def get_exception_handler(self):
-        def handler(e, context):
-            logger.error(e, exc_info=True)
-            return Response({"error": str(e)}, status=400)
-        return handler
-
     def perform_destroy(self, instance):
         manager = AssetUserManager()
         manager.delete(instance)

apps/assets/api/cmd_filter.py

@@ -1,15 +1,25 @@
 # -*- coding: utf-8 -*-
 #
 
+from rest_framework.response import Response
+from rest_framework.generics import CreateAPIView, RetrieveDestroyAPIView
 from django.shortcuts import get_object_or_404
 
+from common.utils import reverse
+from common.utils import lazyproperty
 from orgs.mixins.api import OrgBulkModelViewSet
-from ..hands import IsOrgAdmin
+from orgs.utils import tmp_to_root_org
+from tickets.models import Ticket
+from tickets.api import GenericTicketStatusRetrieveCloseAPI
+from ..hands import IsOrgAdmin, IsAppUser
 from ..models import CommandFilter, CommandFilterRule
 from .. import serializers
 
 
-__all__ = ['CommandFilterViewSet', 'CommandFilterRuleViewSet']
+__all__ = [
+    'CommandFilterViewSet', 'CommandFilterRuleViewSet', 'CommandConfirmAPI',
+    'CommandConfirmStatusAPI'
+]
 
 
 class CommandFilterViewSet(OrgBulkModelViewSet):
@@ -35,3 +45,50 @@ class CommandFilterRuleViewSet(OrgBulkModelViewSet):
         return cmd_filter.rules.all()
 
 
+class CommandConfirmAPI(CreateAPIView):
+    permission_classes = (IsAppUser, )
+    serializer_class = serializers.CommandConfirmSerializer
+
+    def create(self, request, *args, **kwargs):
+        ticket = self.create_command_confirm_ticket()
+        response_data = self.get_response_data(ticket)
+        return Response(data=response_data, status=200)
+
+    def create_command_confirm_ticket(self):
+        ticket = self.serializer.cmd_filter_rule.create_command_confirm_ticket(
+            run_command=self.serializer.data.get('run_command'),
+            session=self.serializer.session,
+            cmd_filter_rule=self.serializer.cmd_filter_rule,
+            org_id=self.serializer.org.id
+        )
+        return ticket
+
+    @staticmethod
+    def get_response_data(ticket):
+        confirm_status_url = reverse(
+            view_name='api-assets:command-confirm-status',
+            kwargs={'pk': str(ticket.id)}
+        )
+        ticket_detail_url = reverse(
+            view_name='api-tickets:ticket-detail',
+            kwargs={'pk': str(ticket.id)},
+            external=True, api_to_ui=True
+        )
+        ticket_detail_url = '{url}?type={type}'.format(url=ticket_detail_url, type=ticket.type)
+        return {
+            'check_confirm_status': {'method': 'GET', 'url': confirm_status_url},
+            'close_confirm': {'method': 'DELETE', 'url': confirm_status_url},
+            'ticket_detail_url': ticket_detail_url,
+            'reviewers': [str(user) for user in ticket.assignees.all()]
+        }
+
+    @lazyproperty
+    def serializer(self):
+        serializer = self.get_serializer(data=self.request.data)
+        serializer.is_valid(raise_exception=True)
+        return serializer
+
+
+class CommandConfirmStatusAPI(GenericTicketStatusRetrieveCloseAPI):
+    pass
+

apps/assets/api/mixin.py

@@ -1,14 +1,15 @@
 from typing import List
 
+from common.utils.common import timeit
 from assets.models import Node, Asset
-from assets.pagination import AssetLimitOffsetPagination
-from common.utils import lazyproperty, dict_get_any, is_uuid, get_object_or_none
+from assets.pagination import NodeAssetTreePagination
+from common.utils import lazyproperty
 from assets.utils import get_node, is_query_node_all_assets
 
 
 class SerializeToTreeNodeMixin:
-    permission_classes = ()
 
+    @timeit
     def serialize_nodes(self, nodes: List[Node], with_asset_amount=False):
         if with_asset_amount:
             def _name(node: Node):
@@ -45,6 +46,7 @@ class SerializeToTreeNodeMixin:
             return platform
         return default
 
+    @timeit
     def serialize_assets(self, assets, node_key=None):
         if node_key is None:
             get_pid = lambda asset: getattr(asset, 'parent_key', '')
@@ -79,7 +81,7 @@ class SerializeToTreeNodeMixin:
 
 
 class FilterAssetByNodeMixin:
-    pagination_class = AssetLimitOffsetPagination
+    pagination_class = NodeAssetTreePagination
 
     @lazyproperty
     def is_query_node_all_assets(self):

apps/assets/api/node.py

@@ -8,7 +8,6 @@ from rest_framework.response import Response
 from rest_framework.decorators import action
 from django.utils.translation import ugettext_lazy as _
 from django.shortcuts import get_object_or_404, Http404
-from django.utils.decorators import method_decorator
 from django.db.models.signals import m2m_changed
 
 from common.const.http import POST
@@ -17,20 +16,19 @@ from common.const.signals import PRE_REMOVE, POST_REMOVE
 from assets.models import Asset
 from common.utils import get_logger, get_object_or_none
 from common.tree import TreeNodeSerializer
-from common.const.distributed_lock_key import UPDATE_NODE_TREE_LOCK_KEY
 from orgs.mixins.api import OrgModelViewSet
 from orgs.mixins import generics
-from orgs.lock import org_level_transaction_lock
 from orgs.utils import current_org
-from assets.tasks import check_node_assets_amount_task
 from ..hands import IsOrgAdmin
 from ..models import Node
 from ..tasks import (
     update_node_assets_hardware_info_manual,
     test_node_assets_connectivity_manual,
+    check_node_assets_amount_task
 )
 from .. import serializers
 from .mixin import SerializeToTreeNodeMixin
+from assets.locks import NodeAddChildrenLock
 
 
 logger = get_logger(__file__)
@@ -50,17 +48,17 @@ class NodeViewSet(OrgModelViewSet):
     permission_classes = (IsOrgAdmin,)
     serializer_class = serializers.NodeSerializer
 
-    @action(methods=[POST], detail=False, url_name='launch-check-assets-amount-task')
-    def launch_check_assets_amount_task(self, request):
-        task = check_node_assets_amount_task.delay(current_org.id)
-        return Response(data={'task': task.id})
-
     # 仅支持根节点指直接创建，子节点下的节点需要通过children接口创建
     def perform_create(self, serializer):
         child_key = Node.org_root().get_next_child_key()
         serializer.validated_data["key"] = child_key
         serializer.save()
 
+    @action(methods=[POST], detail=False, url_path='check_assets_amount_task')
+    def check_assets_amount_task(self, request):
+        task = check_node_assets_amount_task.delay(current_org.id)
+        return Response(data={'task': task.id})
+
     def perform_update(self, serializer):
         node = self.get_object()
         if node.is_org_root() and node.value != serializer.validated_data['value']:
@@ -73,8 +71,8 @@ class NodeViewSet(OrgModelViewSet):
         if node.is_org_root():
             error = _("You can't delete the root node ({})".format(node.value))
             return Response(data={'error': error}, status=status.HTTP_403_FORBIDDEN)
-        if node.has_children_or_has_assets():
-            error = _("Deletion failed and the node contains children or assets")
+        if node.has_offspring_assets():
+            error = _("Deletion failed and the node contains assets")
             return Response(data={'error': error}, status=status.HTTP_403_FORBIDDEN)
         return super().destroy(request, *args, **kwargs)
 
@@ -117,22 +115,27 @@ class NodeChildrenApi(generics.ListCreateAPIView):
         return super().initial(request, *args, **kwargs)
 
     def perform_create(self, serializer):
-        data = serializer.validated_data
-        _id = data.get("id")
-        value = data.get("value")
-        if not value:
-            value = self.instance.get_next_child_preset_name()
-        node = self.instance.create_child(value=value, _id=_id)
-        # 避免查询 full value
-        node._full_value = node.value
-        serializer.instance = node
+        with NodeAddChildrenLock(self.instance):
+            data = serializer.validated_data
+            _id = data.get("id")
+            value = data.get("value")
+            if not value:
+                value = self.instance.get_next_child_preset_name()
+            node = self.instance.create_child(value=value, _id=_id)
+            # 避免查询 full value
+            node._full_value = node.value
+            serializer.instance = node
 
     def get_object(self):
         pk = self.kwargs.get('pk') or self.request.query_params.get('id')
         key = self.request.query_params.get("key")
+
         if not pk and not key:
-            node = Node.org_root()
             self.is_initial = True
+            if current_org.is_root():
+                node = None
+            else:
+                node = Node.org_root()
             return node
         if pk:
             node = get_object_or_404(Node, pk=pk)
@@ -140,16 +143,26 @@ class NodeChildrenApi(generics.ListCreateAPIView):
             node = get_object_or_404(Node, key=key)
         return node
 
+    def get_org_root_queryset(self, query_all):
+        if query_all:
+            return Node.objects.all()
+        else:
+            return Node.org_root_nodes()
+
     def get_queryset(self):
         query_all = self.request.query_params.get("all", "0") == "all"
-        if not self.instance:
-            return Node.objects.none()
+
+        if self.is_initial and current_org.is_root():
+            return self.get_org_root_queryset(query_all)
 
         if self.is_initial:
             with_self = True
         else:
             with_self = False
 
+        if not self.instance:
+            return Node.objects.none()
+
         if query_all:
             queryset = self.instance.get_all_children(with_self=with_self)
         else:
@@ -181,12 +194,12 @@ class NodeChildrenAsTreeApi(SerializeToTreeNodeMixin, NodeChildrenApi):
 
     def get_assets(self):
         include_assets = self.request.query_params.get('assets', '0') == '1'
-        if not include_assets:
+        if not self.instance or not include_assets:
             return []
         assets = self.instance.get_assets().only(
-            "id", "hostname", "ip", "os",
-            "org_id", "protocols", "is_active"
-        )
+            "id", "hostname", "ip", "os", "platform_id",
+            "org_id", "protocols", "is_active",
+        ).prefetch_related('platform')
         return self.serialize_assets(assets, self.instance.key)
 
 
@@ -210,17 +223,16 @@ class NodeAddChildrenApi(generics.UpdateAPIView):
     serializer_class = serializers.NodeAddChildrenSerializer
     instance = None
 
-    def put(self, request, *args, **kwargs):
+    def update(self, request, *args, **kwargs):
+        """ 同时支持 put 和 patch 方法"""
         instance = self.get_object()
-        nodes_id = request.data.get("nodes")
-        children = Node.objects.filter(id__in=nodes_id)
+        node_ids = request.data.get("nodes")
+        children = Node.objects.filter(id__in=node_ids)
         for node in children:
             node.parent = instance
         return Response("OK")
 
 
-@method_decorator(org_level_transaction_lock(UPDATE_NODE_TREE_LOCK_KEY), name='patch')
-@method_decorator(org_level_transaction_lock(UPDATE_NODE_TREE_LOCK_KEY), name='put')
 class NodeAddAssetsApi(generics.UpdateAPIView):
     model = Node
     serializer_class = serializers.NodeAssetsSerializer
@@ -233,8 +245,6 @@ class NodeAddAssetsApi(generics.UpdateAPIView):
         instance.assets.add(*tuple(assets))
 
 
-@method_decorator(org_level_transaction_lock(UPDATE_NODE_TREE_LOCK_KEY), name='patch')
-@method_decorator(org_level_transaction_lock(UPDATE_NODE_TREE_LOCK_KEY), name='put')
 class NodeRemoveAssetsApi(generics.UpdateAPIView):
     model = Node
     serializer_class = serializers.NodeAssetsSerializer
@@ -247,12 +257,13 @@ class NodeRemoveAssetsApi(generics.UpdateAPIView):
         node.assets.remove(*assets)
 
         # 把孤儿资产添加到 root 节点
-        orphan_assets = Asset.objects.filter(id__in=[a.id for a in assets], nodes__isnull=True).distinct()
+        orphan_assets = Asset.objects.filter(
+            id__in=[a.id for a in assets],
+            nodes__isnull=True
+        ).distinct()
         Node.org_root().assets.add(*orphan_assets)
 
 
-@method_decorator(org_level_transaction_lock(UPDATE_NODE_TREE_LOCK_KEY), name='patch')
-@method_decorator(org_level_transaction_lock(UPDATE_NODE_TREE_LOCK_KEY), name='put')
 class MoveAssetsToNodeApi(generics.UpdateAPIView):
     model = Node
     serializer_class = serializers.NodeAssetsSerializer

apps/assets/api/system_user.py

@@ -87,19 +87,19 @@ class SystemUserTaskApi(generics.CreateAPIView):
     permission_classes = (IsOrgAdmin,)
     serializer_class = serializers.SystemUserTaskSerializer
 
-    def do_push(self, system_user, assets_id=None):
-        if assets_id is None:
+    def do_push(self, system_user, asset_ids=None):
+        if asset_ids is None:
             task = push_system_user_to_assets_manual.delay(system_user)
         else:
             username = self.request.query_params.get('username')
             task = push_system_user_to_assets.delay(
-                system_user.id, assets_id, username=username
+                system_user.id, asset_ids, username=username
             )
         return task
 
     @staticmethod
-    def do_test(system_user):
-        task = test_system_user_connectivity_manual.delay(system_user)
+    def do_test(system_user, asset_ids):
+        task = test_system_user_connectivity_manual.delay(system_user, asset_ids)
         return task
 
     def get_object(self):
@@ -109,16 +109,20 @@ class SystemUserTaskApi(generics.CreateAPIView):
     def perform_create(self, serializer):
         action = serializer.validated_data["action"]
         asset = serializer.validated_data.get('asset')
-        assets = serializer.validated_data.get('assets') or []
+
+        if asset:
+            assets = [asset]
+        else:
+            assets = serializer.validated_data.get('assets') or []
+
+        asset_ids = [asset.id for asset in assets]
+        asset_ids = asset_ids if asset_ids else None
 
         system_user = self.get_object()
         if action == 'push':
-            assets = [asset] if asset else assets
-            assets_id = [asset.id for asset in assets]
-            assets_id = assets_id if assets_id else None
-            task = self.do_push(system_user, assets_id)
+            task = self.do_push(system_user, asset_ids)
         else:
-            task = self.do_test(system_user)
+            task = self.do_test(system_user, asset_ids)
         data = getattr(serializer, '_data', {})
         data["task"] = task.id
         setattr(serializer, '_data', data)

apps/assets/api/system_user_relation.py

@@ -19,9 +19,10 @@ __all__ = [
 class RelationMixin:
     def get_queryset(self):
         queryset = self.model.objects.all()
-        org_id = current_org.org_id()
-        if org_id is not None:
+        if not current_org.is_root():
+            org_id = current_org.org_id()
             queryset = queryset.filter(systemuser__org_id=org_id)
+
         queryset = queryset.annotate(systemuser_display=Concat(
             F('systemuser__name'), Value('('), F('systemuser__username'),
             Value(')')

apps/assets/apps.py

@@ -1,16 +1,6 @@
 from __future__ import unicode_literals
 
 from django.apps import AppConfig
-from django.db.models.signals import post_migrate
-
-
-def initial_some_nodes():
-    from .models import Node
-    Node.initial_some_nodes()
-
-
-def initial_some_nodes_callback(sender, **kwargs):
-    initial_some_nodes()
 
 
 class AssetsConfig(AppConfig):
@@ -19,7 +9,3 @@ class AssetsConfig(AppConfig):
     def ready(self):
         super().ready()
         from . import signals_handler
-        try:
-            initial_some_nodes()
-        except Exception:
-            post_migrate.connect(initial_some_nodes_callback, sender=self)

apps/assets/backends/base.py

@@ -40,7 +40,7 @@ class BaseBackend:
         return values
 
     @staticmethod
-    def make_assets_as_id(assets):
+    def make_assets_as_ids(assets):
         if not assets:
             return []
         if isinstance(assets[0], Asset):

apps/assets/backends/db.py

@@ -69,9 +69,9 @@ class DBBackend(BaseBackend):
         self.queryset = self.queryset.filter(union_id=union_id)
 
     def _filter_assets(self, assets):
-        assets_id = self.make_assets_as_id(assets)
-        if assets_id:
-            self.queryset = self.queryset.filter(asset_id__in=assets_id)
+        asset_ids = self.make_assets_as_ids(assets)
+        if asset_ids:
+            self.queryset = self.queryset.filter(asset_id__in=asset_ids)
 
     def _filter_node(self, node):
         pass
@@ -165,7 +165,7 @@ class SystemUserBackend(DBBackend):
         kwargs = self.get_annotate()
         filters = self.get_filter()
         qs = self.model.objects.all().annotate(**kwargs)
-        if current_org.org_id() is not None:
+        if not current_org.is_root():
             filters['org_id'] = current_org.org_id()
         qs = qs.filter(**filters)
         qs = self.qs_to_values(qs)

apps/assets/locks.py

@@ -0,0 +1,29 @@
+from orgs.utils import current_org
+from common.utils.lock import DistributedLock
+from assets.models import Node
+
+
+class NodeTreeUpdateLock(DistributedLock):
+    name_template = 'assets.node.tree.update.<org_id:{org_id}>'
+
+    def get_name(self):
+        if current_org:
+            org_id = current_org.id
+        else:
+            org_id = 'current_org_is_null'
+        name = self.name_template.format(
+            org_id=org_id
+        )
+        return name
+
+    def __init__(self):
+        name = self.get_name()
+        super().__init__(name=name, release_on_transaction_commit=True, reentrant=True)
+
+
+class NodeAddChildrenLock(DistributedLock):
+    name_template = 'assets.node.add_children.<org_id:{org_id}>'
+
+    def __init__(self, node: Node):
+        name = self.name_template.format(org_id=node.org_id)
+        super().__init__(name=name, release_on_transaction_commit=True)

apps/assets/migrations/0066_auto_20210208_1802.py

@@ -0,0 +1,17 @@
+# Generated by Django 3.1 on 2021-02-08 10:02
+
+from django.db import migrations
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('assets', '0065_auto_20210121_1549'),
+    ]
+
+    operations = [
+        migrations.AlterModelOptions(
+            name='asset',
+            options={'ordering': ['hostname'], 'verbose_name': 'Asset'},
+        ),
+    ]

apps/assets/migrations/0067_auto_20210311_1113.py

@@ -0,0 +1,48 @@
+# Generated by Django 3.1 on 2021-03-11 03:13
+
+import django.core.validators
+from django.db import migrations, models
+
+
+def migrate_cmd_filter_priority(apps, schema_editor):
+    cmd_filter_rule_model = apps.get_model('assets', 'CommandFilterRule')
+    cmd_filter_rules = cmd_filter_rule_model.objects.all()
+    for cmd_filter_rule in cmd_filter_rules:
+        cmd_filter_rule.priority = 100 - cmd_filter_rule.priority + 1
+
+    cmd_filter_rule_model.objects.bulk_update(cmd_filter_rules, fields=['priority'])
+
+
+def migrate_system_user_priority(apps, schema_editor):
+    system_user_model = apps.get_model('assets', 'SystemUser')
+    system_users = system_user_model.objects.all()
+    for system_user in system_users:
+        system_user.priority = 100 - system_user.priority + 1
+
+    system_user_model.objects.bulk_update(system_users, fields=['priority'])
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('assets', '0066_auto_20210208_1802'),
+    ]
+
+    operations = [
+        migrations.RunPython(migrate_cmd_filter_priority),
+        migrations.RunPython(migrate_system_user_priority),
+        migrations.AlterModelOptions(
+            name='commandfilterrule',
+            options={'ordering': ('priority', 'action'), 'verbose_name': 'Command filter rule'},
+        ),
+        migrations.AlterField(
+            model_name='commandfilterrule',
+            name='priority',
+            field=models.IntegerField(default=50, help_text='1-100, the lower the value will be match first', validators=[django.core.validators.MinValueValidator(1), django.core.validators.MaxValueValidator(100)], verbose_name='Priority'),
+        ),
+        migrations.AlterField(
+            model_name='systemuser',
+            name='priority',
+            field=models.IntegerField(default=20, help_text='1-100, the lower the value will be match first', validators=[django.core.validators.MinValueValidator(1), django.core.validators.MaxValueValidator(100)], verbose_name='Priority'),
+        ),
+    ]

apps/assets/migrations/0068_auto_20210312_1455.py

@@ -0,0 +1,19 @@
+# Generated by Django 3.1 on 2021-03-12 06:55
+
+import django.core.validators
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('assets', '0067_auto_20210311_1113'),
+    ]
+
+    operations = [
+        migrations.AlterField(
+            model_name='systemuser',
+            name='priority',
+            field=models.IntegerField(default=81, help_text='1-100, the lower the value will be match first', validators=[django.core.validators.MinValueValidator(1), django.core.validators.MaxValueValidator(100)], verbose_name='Priority'),
+        ),
+    ]

apps/assets/migrations/0069_change_node_key0_to_key1.py

@@ -0,0 +1,61 @@
+from django.db import migrations
+from django.db.transaction import atomic
+
+default_id = '00000000-0000-0000-0000-000000000002'
+
+
+def change_key0_to_key1(apps, schema_editor):
+    from orgs.utils import set_current_org
+
+    # https://stackoverflow.com/questions/28777338/django-migrations-runpython-not-able-to-call-model-methods
+    Organization = apps.get_model('orgs', 'Organization')
+    Node = apps.get_model('assets', 'Node')
+
+    print()
+    org = Organization.objects.get(id=default_id)
+    set_current_org(org)
+
+    exists_0 = Node.objects.filter(key__startswith='0').exists()
+    if not exists_0:
+        print(f'--> Not exist key=0 nodes, do nothing.')
+        return
+
+    key_1_count = Node.objects.filter(key__startswith='1').count()
+    if key_1_count > 1:
+        print(f'--> Node key=1 have children, can`t just delete it. Please contact JumpServer team')
+        return
+
+    root_node = Node.objects.filter(key='1').first()
+    if root_node and root_node.assets.exists():
+        print(f'--> Node key=1 has assets, do nothing.')
+        return
+
+    with atomic():
+        if root_node:
+            print(f'--> Delete node key=1')
+            root_node.delete()
+
+        nodes_0 = Node.objects.filter(key__startswith='0')
+
+        for n in nodes_0:
+            old_key = n.key
+            key_list = n.key.split(':')
+            key_list[0] = '1'
+            new_key = ':'.join(key_list)
+            new_parent_key = ':'.join(key_list[:-1])
+            n.key = new_key
+            n.parent_key = new_parent_key
+            n.save()
+            print('--> Modify key ( {} > {} )'.format(old_key, new_key))
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('orgs', '0010_auto_20210219_1241'),
+        ('assets', '0068_auto_20210312_1455'),
+    ]
+
+    operations = [
+        migrations.RunPython(change_key0_to_key1)
+    ]

apps/assets/migrations/0070_auto_20210426_1515.py

@@ -0,0 +1,25 @@
+# Generated by Django 3.1 on 2021-04-26 07:15
+
+from django.conf import settings
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        migrations.swappable_dependency(settings.AUTH_USER_MODEL),
+        ('assets', '0069_change_node_key0_to_key1'),
+    ]
+
+    operations = [
+        migrations.AddField(
+            model_name='commandfilterrule',
+            name='reviewers',
+            field=models.ManyToManyField(blank=True, related_name='review_cmd_filter_rules', to=settings.AUTH_USER_MODEL, verbose_name='Reviewers'),
+        ),
+        migrations.AlterField(
+            model_name='commandfilterrule',
+            name='action',
+            field=models.IntegerField(choices=[(0, 'Deny'), (1, 'Allow'), (2, 'Reconfirm')], default=0, verbose_name='Action'),
+        ),
+    ]

apps/assets/models/asset.py

@@ -17,7 +17,7 @@ from orgs.mixins.models import OrgModelMixin, OrgManager
 from .base import ConnectivityMixin
 from .utils import Connectivity
 
-__all__ = ['Asset', 'ProtocolsMixin', 'Platform']
+__all__ = ['Asset', 'ProtocolsMixin', 'Platform', 'AssetQuerySet']
 logger = logging.getLogger(__name__)
 
 
@@ -35,19 +35,12 @@ def default_node():
     try:
         from .node import Node
         root = Node.org_root()
-        return root
+        return Node.objects.filter(id=root.id)
     except:
         return None
 
 
 class AssetManager(OrgManager):
-    def get_queryset(self):
-        return super().get_queryset().annotate(
-            platform_base=models.F('platform__base')
-        )
-
-
-class AssetOrgManager(OrgManager):
     pass
 
 
@@ -230,7 +223,6 @@ class Asset(ProtocolsMixin, NodesRelationMixin, OrgModelMixin):
     comment = models.TextField(default='', blank=True, verbose_name=_('Comment'))
 
     objects = AssetManager.from_queryset(AssetQuerySet)()
-    org_objects = AssetOrgManager.from_queryset(AssetQuerySet)()
     _connectivity = None
 
     def __str__(self):
@@ -361,4 +353,4 @@ class Asset(ProtocolsMixin, NodesRelationMixin, OrgModelMixin):
     class Meta:
         unique_together = [('org_id', 'hostname')]
         verbose_name = _("Asset")
-        ordering = ["hostname", "ip"]
+        ordering = ["hostname", ]

apps/assets/models/base.py

@@ -11,9 +11,12 @@ from django.db import models
 from django.utils.translation import ugettext_lazy as _
 from django.conf import settings
 
+from common.db.models import ChoiceSet
+from common.utils import random_string
 from common.utils import (
     ssh_key_string_to_obj, ssh_key_gen, get_logger, lazyproperty
 )
+from common.utils.encode import ssh_pubkey_gen
 from common.validators import alphanumeric
 from common import fields
 from orgs.mixins.models import OrgModelMixin
@@ -105,6 +108,19 @@ class AuthMixin:
     username = ''
     _prefer = 'system_user'
 
+    @property
+    def ssh_key_fingerprint(self):
+        if self.public_key:
+            public_key = self.public_key
+        elif self.private_key:
+            public_key = ssh_pubkey_gen(private_key=self.private_key, password=self.password)
+        else:
+            return ''
+
+        public_key_obj = sshpubkeys.SSHKey(public_key)
+        fingerprint = public_key_obj.hash_md5()
+        return fingerprint
+
     @property
     def private_key_obj(self):
         if self.private_key:
@@ -204,8 +220,8 @@ class AuthMixin:
         self.save()
 
     @staticmethod
-    def gen_password():
-        return str(uuid.uuid4())
+    def gen_password(length=36):
+        return random_string(length, special_char=True)
 
     @staticmethod
     def gen_key(username):

apps/assets/models/cmd_filter.py

@@ -41,26 +41,33 @@ class CommandFilterRule(OrgModelMixin):
         (TYPE_COMMAND, _('Command')),
     )
 
-    ACTION_DENY, ACTION_ALLOW, ACTION_UNKNOWN = range(3)
-    ACTION_CHOICES = (
-        (ACTION_DENY, _('Deny')),
-        (ACTION_ALLOW, _('Allow')),
-    )
+    ACTION_UNKNOWN = 10
+
+    class ActionChoices(models.IntegerChoices):
+        deny = 0, _('Deny')
+        allow = 1, _('Allow')
+        confirm = 2, _('Reconfirm')
 
     id = models.UUIDField(default=uuid.uuid4, primary_key=True)
     filter = models.ForeignKey('CommandFilter', on_delete=models.CASCADE, verbose_name=_("Filter"), related_name='rules')
     type = models.CharField(max_length=16, default=TYPE_COMMAND, choices=TYPE_CHOICES, verbose_name=_("Type"))
-    priority = models.IntegerField(default=50, verbose_name=_("Priority"), help_text=_("1-100, the higher will be match first"),
+    priority = models.IntegerField(default=50, verbose_name=_("Priority"), help_text=_("1-100, the lower the value will be match first"),
                                    validators=[MinValueValidator(1), MaxValueValidator(100)])
     content = models.TextField(verbose_name=_("Content"), help_text=_("One line one command"))
-    action = models.IntegerField(default=ACTION_DENY, choices=ACTION_CHOICES, verbose_name=_("Action"))
+    action = models.IntegerField(default=ActionChoices.deny, choices=ActionChoices.choices, verbose_name=_("Action"))
+    # 动作: 附加字段
+    # - confirm: 命令复核人
+    reviewers = models.ManyToManyField(
+        'users.User', related_name='review_cmd_filter_rules', blank=True,
+        verbose_name=_("Reviewers")
+    )
     comment = models.CharField(max_length=64, blank=True, default='', verbose_name=_("Comment"))
     date_created = models.DateTimeField(auto_now_add=True)
     date_updated = models.DateTimeField(auto_now=True)
     created_by = models.CharField(max_length=128, blank=True, default='', verbose_name=_('Created by'))
 
     class Meta:
-        ordering = ('-priority', 'action')
+        ordering = ('priority', 'action')
         verbose_name = _("Command filter rule")
 
     @lazyproperty
@@ -89,10 +96,32 @@ class CommandFilterRule(OrgModelMixin):
         if not found:
             return self.ACTION_UNKNOWN, ''
 
-        if self.action == self.ACTION_ALLOW:
-            return self.ACTION_ALLOW, found.group()
+        if self.action == self.ActionChoices.allow:
+            return self.ActionChoices.allow, found.group()
         else:
-            return self.ACTION_DENY, found.group()
+            return self.ActionChoices.deny, found.group()
 
     def __str__(self):
         return '{} % {}'.format(self.type, self.content)
+
+    def create_command_confirm_ticket(self, run_command, session, cmd_filter_rule, org_id):
+        from tickets.const import TicketTypeChoices
+        from tickets.models import Ticket
+        data = {
+            'title': _('Command confirm') + ' ({})'.format(session.user),
+            'type': TicketTypeChoices.command_confirm,
+            'meta': {
+                'apply_run_user': session.user,
+                'apply_run_asset': session.asset,
+                'apply_run_system_user': session.system_user,
+                'apply_run_command': run_command,
+                'apply_from_session_id': str(session.id),
+                'apply_from_cmd_filter_rule_id': str(cmd_filter_rule.id),
+                'apply_from_cmd_filter_id': str(cmd_filter_rule.filter.id)
+            },
+            'org_id': org_id,
+        }
+        ticket = Ticket.objects.create(**data)
+        ticket.assignees.set(self.reviewers.all())
+        ticket.open(applicant=session.user_obj)
+        return ticket

apps/assets/models/favorite_asset.py

@@ -16,17 +16,5 @@ class FavoriteAsset(CommonModelMixin):
         unique_together = ('user', 'asset')
 
     @classmethod
-    def get_user_favorite_assets_id(cls, user):
+    def get_user_favorite_asset_ids(cls, user):
         return cls.objects.filter(user=user).values_list('asset', flat=True)
-
-    @classmethod
-    def get_user_favorite_assets(cls, user, asset_perms_id=None):
-        from assets.models import Asset
-        from perms.utils.asset.user_permission import get_user_granted_all_assets
-        asset_ids = get_user_granted_all_assets(
-            user,
-            via_mapping_node=False,
-            asset_perms_id=asset_perms_id
-        ).values_list('id', flat=True)
-        query_name = cls.asset.field.related_query_name()
-        return Asset.org_objects.filter(**{f'{query_name}__user_id': user.id}, id__in=asset_ids).distinct()

apps/assets/models/node.py

@@ -1,23 +1,32 @@
 # -*- coding: utf-8 -*-
 #
-import uuid
 import re
+import time
+import uuid
+import threading
+import os
+import time
+import uuid
 
+from collections import defaultdict
 from django.db import models, transaction
-from django.db.models import Q
+from django.db.models import Q, Manager
 from django.db.utils import IntegrityError
 from django.utils.translation import ugettext_lazy as _
 from django.utils.translation import ugettext
 from django.db.transaction import atomic
+from django.core.cache import cache
 
+from common.utils.lock import DistributedLock
+from common.utils.common import timeit
+from common.db.models import output_as_string
 from common.utils import get_logger
-from common.utils.common import lazyproperty
 from orgs.mixins.models import OrgModelMixin, OrgManager
-from orgs.utils import get_current_org, tmp_to_org
+from orgs.utils import get_current_org, tmp_to_org, tmp_to_root_org
 from orgs.models import Organization
 
 
-__all__ = ['Node', 'FamilyMixin', 'compute_parent_key']
+__all__ = ['Node', 'FamilyMixin', 'compute_parent_key', 'NodeQuerySet']
 logger = get_logger(__name__)
 
 
@@ -29,8 +38,7 @@ def compute_parent_key(key):
 
 
 class NodeQuerySet(models.QuerySet):
-    def delete(self):
-        raise NotImplementedError
+    pass
 
 
 class FamilyMixin:
@@ -247,9 +255,156 @@ class FamilyMixin:
         return [*tuple(ancestors), self, *tuple(children)]
 
 
-class NodeAssetsMixin:
+class NodeAllAssetsMappingMixin:
+    # Use a new plan
+
+    # { org_id: { node_key: [ asset1_id, asset2_id ] } }
+    orgid_nodekey_assetsid_mapping = defaultdict(dict)
+    locks_for_get_mapping_from_cache = defaultdict(threading.Lock)
+
+    @classmethod
+    def get_lock(cls, org_id):
+        lock = cls.locks_for_get_mapping_from_cache[str(org_id)]
+        return lock
+
+    @classmethod
+    def get_node_all_asset_ids_mapping(cls, org_id):
+        _mapping = cls.get_node_all_asset_ids_mapping_from_memory(org_id)
+        if _mapping:
+            return _mapping
+
+        logger.debug(f'Get node asset mapping from memory failed, acquire thread lock: '
+                     f'thread={threading.get_ident()} '
+                     f'org_id={org_id}')
+        with cls.get_lock(org_id):
+            logger.debug(f'Acquired thread lock ok. check if mapping is in memory now: '
+                         f'thread={threading.get_ident()} '
+                         f'org_id={org_id}')
+            _mapping = cls.get_node_all_asset_ids_mapping_from_memory(org_id)
+            if _mapping:
+                logger.debug(f'Mapping is already in memory now: '
+                             f'thread={threading.get_ident()} '
+                             f'org_id={org_id}')
+                return _mapping
+
+            _mapping = cls.get_node_all_asset_ids_mapping_from_cache_or_generate_to_cache(org_id)
+            cls.set_node_all_asset_ids_mapping_to_memory(org_id, mapping=_mapping)
+        return _mapping
+
+    # from memory
+    @classmethod
+    def get_node_all_asset_ids_mapping_from_memory(cls, org_id):
+        mapping = cls.orgid_nodekey_assetsid_mapping.get(org_id, {})
+        return mapping
+
+    @classmethod
+    def set_node_all_asset_ids_mapping_to_memory(cls, org_id, mapping):
+        cls.orgid_nodekey_assetsid_mapping[org_id] = mapping
+
+    @classmethod
+    def expire_node_all_asset_ids_mapping_from_memory(cls, org_id):
+        org_id = str(org_id)
+        cls.orgid_nodekey_assetsid_mapping.pop(org_id, None)
+
+    @classmethod
+    def expire_all_orgs_node_all_asset_ids_mapping_from_memory(cls):
+        orgs = Organization.objects.all()
+        org_ids = [str(org.id) for org in orgs]
+        org_ids.append(Organization.ROOT_ID)
+
+        for id in org_ids:
+            cls.expire_node_all_asset_ids_mapping_from_memory(id)
+
+    # get order: from memory -> (from cache -> to generate)
+    @classmethod
+    def get_node_all_asset_ids_mapping_from_cache_or_generate_to_cache(cls, org_id):
+        mapping = cls.get_node_all_asset_ids_mapping_from_cache(org_id)
+        if mapping:
+            return mapping
+
+        lock_key = f'KEY_LOCK_GENERATE_ORG_{org_id}_NODE_ALL_ASSET_ids_MAPPING'
+        with DistributedLock(lock_key):
+            # 这里使用无限期锁，原因是如果这里卡住了，就卡在数据库了，说明
+            # 数据库繁忙，所以不应该再有线程执行这个操作，使数据库忙上加忙
+
+            _mapping = cls.get_node_all_asset_ids_mapping_from_cache(org_id)
+            if _mapping:
+                return _mapping
+
+            _mapping = cls.generate_node_all_asset_ids_mapping(org_id)
+            cls.set_node_all_asset_ids_mapping_to_cache(org_id=org_id, mapping=_mapping)
+            return _mapping
+
+    @classmethod
+    def get_node_all_asset_ids_mapping_from_cache(cls, org_id):
+        cache_key = cls._get_cache_key_for_node_all_asset_ids_mapping(org_id)
+        mapping = cache.get(cache_key)
+        logger.info(f'Get node asset mapping from cache {bool(mapping)}: '
+                    f'thread={threading.get_ident()} '
+                    f'org_id={org_id}')
+        return mapping
+
+    @classmethod
+    def set_node_all_asset_ids_mapping_to_cache(cls, org_id, mapping):
+        cache_key = cls._get_cache_key_for_node_all_asset_ids_mapping(org_id)
+        cache.set(cache_key, mapping, timeout=None)
+
+    @classmethod
+    def expire_node_all_asset_ids_mapping_from_cache(cls, org_id):
+        cache_key = cls._get_cache_key_for_node_all_asset_ids_mapping(org_id)
+        cache.delete(cache_key)
+
+    @staticmethod
+    def _get_cache_key_for_node_all_asset_ids_mapping(org_id):
+        return 'ASSETS_ORG_NODE_ALL_ASSET_ids_MAPPING_{}'.format(org_id)
+
+    @classmethod
+    def generate_node_all_asset_ids_mapping(cls, org_id):
+        from .asset import Asset
+
+        logger.info(f'Generate node asset mapping: '
+                    f'thread={threading.get_ident()} '
+                    f'org_id={org_id}')
+        t1 = time.time()
+        with tmp_to_org(org_id):
+            node_ids_key = Node.objects.annotate(
+                char_id=output_as_string('id')
+            ).values_list('char_id', 'key')
+
+            # * 直接取出全部. filter(node__org_id=org_id)(大规模下会更慢)
+            nodes_asset_ids = Asset.nodes.through.objects.all() \
+                .annotate(char_node_id=output_as_string('node_id')) \
+                .annotate(char_asset_id=output_as_string('asset_id')) \
+                .values_list('char_node_id', 'char_asset_id')
+
+            node_id_ancestor_keys_mapping = {
+                node_id: cls.get_node_ancestor_keys(node_key, with_self=True)
+                for node_id, node_key in node_ids_key
+            }
+
+            nodeid_assetsid_mapping = defaultdict(set)
+            for node_id, asset_id in nodes_asset_ids:
+                nodeid_assetsid_mapping[node_id].add(asset_id)
+
+        t2 = time.time()
+
+        mapping = defaultdict(set)
+        for node_id, node_key in node_ids_key:
+            asset_ids = nodeid_assetsid_mapping[node_id]
+            node_ancestor_keys = node_id_ancestor_keys_mapping[node_id]
+            for ancestor_key in node_ancestor_keys:
+                mapping[ancestor_key].update(asset_ids)
+
+        t3 = time.time()
+        logger.info('t1-t2(DB Query): {} s, t3-t2(Generate mapping): {} s'.format(t2-t1, t3-t2))
+        return mapping
+
+
+class NodeAssetsMixin(NodeAllAssetsMappingMixin):
+    org_id: str
     key = ''
     id = None
+    objects: Manager
 
     def get_all_assets(self):
         from .asset import Asset
@@ -263,8 +418,7 @@ class NodeAssetsMixin:
         #   可是 startswith 会导致表关联时 Asset 索引失效
         from .asset import Asset
         node_ids = cls.objects.filter(
-            Q(key__startswith=f'{key}:') |
-            Q(key=key)
+            Q(key__startswith=f'{key}:') | Q(key=key)
         ).values_list('id', flat=True).distinct()
         assets = Asset.objects.filter(
             nodes__id__in=list(node_ids)
@@ -283,54 +437,42 @@ class NodeAssetsMixin:
         return self.get_all_assets().valid()
 
     @classmethod
-    def get_nodes_all_assets_ids(cls, nodes_keys):
-        assets_ids = cls.get_nodes_all_assets(nodes_keys).values_list('id', flat=True)
-        return assets_ids
+    def get_nodes_all_asset_ids_by_keys(cls, nodes_keys):
+        nodes = Node.objects.filter(key__in=nodes_keys)
+        asset_ids = cls.get_nodes_all_assets(*nodes).values_list('id', flat=True)
+        return asset_ids
 
     @classmethod
-    def get_nodes_all_assets(cls, nodes_keys, extra_assets_ids=None):
+    def get_nodes_all_assets(cls, *nodes):
         from .asset import Asset
-        nodes_keys = cls.clean_children_keys(nodes_keys)
-        q = Q()
-        node_ids = ()
-        for key in nodes_keys:
-            q |= Q(key__startswith=f'{key}:')
-            q |= Q(key=key)
-        if q:
-            node_ids = Node.objects.filter(q).distinct().values_list('id', flat=True)
+        node_ids = set()
+        descendant_node_query = Q()
+        for n in nodes:
+            node_ids.add(n.id)
+            descendant_node_query |= Q(key__istartswith=f'{n.key}:')
+        if descendant_node_query:
+            _ids = Node.objects.order_by().filter(descendant_node_query).values_list('id', flat=True)
+            node_ids.update(_ids)
+        return Asset.objects.order_by().filter(nodes__id__in=node_ids).distinct()
 
-        q = Q(nodes__id__in=list(node_ids))
-        if extra_assets_ids:
-            q |= Q(id__in=extra_assets_ids)
-        if q:
-            return Asset.org_objects.filter(q).distinct()
-        else:
-            return Asset.objects.none()
+    def get_all_asset_ids(self):
+        asset_ids = self.get_all_asset_ids_by_node_key(org_id=self.org_id, node_key=self.key)
+        return set(asset_ids)
+
+    @classmethod
+    def get_all_asset_ids_by_node_key(cls, org_id, node_key):
+        org_id = str(org_id)
+        nodekey_assetsid_mapping = cls.get_node_all_asset_ids_mapping(org_id)
+        asset_ids = nodekey_assetsid_mapping.get(node_key, [])
+        return set(asset_ids)
 
 
 class SomeNodesMixin:
     key = ''
     default_key = '1'
-    default_value = 'Default'
     empty_key = '-11'
     empty_value = _("empty")
 
-    @classmethod
-    def default_node(cls):
-        with tmp_to_org(Organization.default()):
-            defaults = {'value': cls.default_value}
-            try:
-                obj, created = cls.objects.get_or_create(
-                    defaults=defaults, key=cls.default_key,
-                )
-            except IntegrityError as e:
-                logger.error("Create default node failed: {}".format(e))
-                cls.modify_other_org_root_node_key()
-                obj, created = cls.objects.get_or_create(
-                    defaults=defaults, key=cls.default_key,
-                )
-            return obj
-
     def is_default_node(self):
         return self.key == self.default_key
 
@@ -341,70 +483,61 @@ class SomeNodesMixin:
             return False
 
     @classmethod
-    def get_next_org_root_node_key(cls):
-        with tmp_to_org(Organization.root()):
-            org_nodes_roots = cls.objects.filter(key__regex=r'^[0-9]+$')
-            org_nodes_roots_keys = org_nodes_roots.values_list('key', flat=True)
-            if not org_nodes_roots_keys:
-                org_nodes_roots_keys = ['1']
-            max_key = max([int(k) for k in org_nodes_roots_keys])
-            key = str(max_key + 1) if max_key != 0 else '2'
-            return key
+    def org_root(cls):
+        # 如果使用current_org 在set_current_org时会死循环
+        ori_org = get_current_org()
+
+        if ori_org and ori_org.is_default():
+            return cls.default_node()
+
+        if ori_org and ori_org.is_root():
+            return None
+
+        org_roots = cls.org_root_nodes()
+        org_roots_length = len(org_roots)
+
+        if org_roots_length == 1:
+            root = org_roots[0]
+            return root
+        elif org_roots_length == 0:
+            root = cls.create_org_root_node()
+            return root
+        else:
+            error = 'Current org {} root node not 1, get {}'.format(ori_org, org_roots_length)
+            raise ValueError(error)
+
+    @classmethod
+    def default_node(cls):
+        default_org = Organization.default()
+        with tmp_to_org(default_org):
+            defaults = {'value': default_org.name}
+            obj, created = cls.objects.get_or_create(defaults=defaults, key=cls.default_key)
+            return obj
 
     @classmethod
     def create_org_root_node(cls):
-        # 如果使用current_org 在set_current_org时会死循环
         ori_org = get_current_org()
         with transaction.atomic():
-            if not ori_org.is_real():
-                return cls.default_node()
             key = cls.get_next_org_root_node_key()
             root = cls.objects.create(key=key, value=ori_org.name)
             return root
 
     @classmethod
-    def org_root(cls):
-        root = cls.objects.filter(parent_key='')\
-            .filter(key__regex=r'^[0-9]+$')\
-            .exclude(key__startswith='-')\
-            .order_by('key')
-        if root:
-            return root[0]
-        else:
-            return cls.create_org_root_node()
+    def get_next_org_root_node_key(cls):
+        with tmp_to_root_org():
+            org_nodes_roots = cls.org_root_nodes()
+            org_nodes_roots_keys = org_nodes_roots.values_list('key', flat=True)
+            if not org_nodes_roots_keys:
+                org_nodes_roots_keys = ['1']
+            max_key = max([int(k) for k in org_nodes_roots_keys])
+            key = str(max_key + 1) if max_key > 0 else '2'
+            return key
 
     @classmethod
-    def initial_some_nodes(cls):
-        cls.default_node()
-
-    @classmethod
-    def modify_other_org_root_node_key(cls):
-        """
-        解决创建 default 节点失败的问题，
-        因为在其他组织下存在 default 节点，故在 DEFAULT 组织下 get 不到 create 失败
-        """
-        logger.info("Modify other org root node key")
-
-        with tmp_to_org(Organization.root()):
-            node_key1 = cls.objects.filter(key='1').first()
-            if not node_key1:
-                logger.info("Not found node that `key` = 1")
-                return
-            if not node_key1.org.is_real():
-                logger.info("Org is not real for node that `key` = 1")
-                return
-
-        with transaction.atomic():
-            with tmp_to_org(node_key1.org):
-                org_root_node_new_key = cls.get_next_org_root_node_key()
-                for n in cls.objects.all():
-                    old_key = n.key
-                    key_list = n.key.split(':')
-                    key_list[0] = org_root_node_new_key
-                    new_key = ':'.join(key_list)
-                    n.key = new_key
-                    n.save()
-                    logger.info('Modify key ( {} > {} )'.format(old_key, new_key))
+    def org_root_nodes(cls):
+        root_nodes = cls.objects.filter(parent_key='', key__regex=r'^[0-9]+$') \
+            .exclude(key__startswith='-').order_by('key')
+        return root_nodes
 
 
 class Node(OrgModelMixin, SomeNodesMixin, FamilyMixin, NodeAssetsMixin):
@@ -488,14 +621,14 @@ class Node(OrgModelMixin, SomeNodesMixin, FamilyMixin, NodeAssetsMixin):
         tree_node = TreeNode(**data)
         return tree_node
 
-    def has_children_or_has_assets(self):
-        if self.children or self.get_assets().exists():
-            return True
-        return False
+    def has_offspring_assets(self):
+        # 拥有后代资产
+        return self.get_all_assets().exists()
 
     def delete(self, using=None, keep_parents=False):
-        if self.has_children_or_has_assets():
+        if self.has_offspring_assets():
             return
+        self.all_children.delete()
         return super().delete(using=using, keep_parents=keep_parents)
 
     def update_child_full_value(self):

apps/assets/models/user.py

@@ -87,6 +87,9 @@ class SystemUser(BaseUser):
         (PROTOCOL_POSTGRESQL, 'postgresql'),
         (PROTOCOL_K8S, 'k8s'),
     )
+
+    SUPPORT_PUSH_PROTOCOLS = [PROTOCOL_SSH, PROTOCOL_RDP]
+
     ASSET_CATEGORY_PROTOCOLS = [
         PROTOCOL_SSH, PROTOCOL_RDP, PROTOCOL_TELNET, PROTOCOL_VNC
     ]
@@ -116,7 +119,7 @@ class SystemUser(BaseUser):
     assets = models.ManyToManyField('assets.Asset', blank=True, verbose_name=_("Assets"))
     users = models.ManyToManyField('users.User', blank=True, verbose_name=_("Users"))
     groups = models.ManyToManyField('users.UserGroup', blank=True, verbose_name=_("User groups"))
-    priority = models.IntegerField(default=20, verbose_name=_("Priority"), validators=[MinValueValidator(1), MaxValueValidator(100)])
+    priority = models.IntegerField(default=81, verbose_name=_("Priority"), help_text=_("1-100, the lower the value will be match first"), validators=[MinValueValidator(1), MaxValueValidator(100)])
     protocol = models.CharField(max_length=16, choices=PROTOCOL_CHOICES, default='ssh', verbose_name=_('Protocol'))
     auto_push = models.BooleanField(default=True, verbose_name=_('Auto push'))
     sudo = models.TextField(default='/bin/whoami', verbose_name=_('Sudo'))
@@ -151,11 +154,15 @@ class SystemUser(BaseUser):
         return self.get_login_mode_display()
 
     def is_need_push(self):
-        if self.auto_push and self.protocol in [self.PROTOCOL_SSH, self.PROTOCOL_RDP]:
+        if self.auto_push and self.is_protocol_support_push:
             return True
         else:
             return False
 
+    @property
+    def is_protocol_support_push(self):
+        return self.protocol in self.SUPPORT_PUSH_PROTOCOLS
+
     @property
     def is_need_cmd_filter(self):
         return self.protocol not in [self.PROTOCOL_RDP, self.PROTOCOL_VNC]
@@ -189,19 +196,19 @@ class SystemUser(BaseUser):
     def is_command_can_run(self, command):
         for rule in self.cmd_filter_rules:
             action, matched_cmd = rule.match(command)
-            if action == rule.ACTION_ALLOW:
+            if action == rule.ActionChoices.allow:
                 return True, None
-            elif action == rule.ACTION_DENY:
+            elif action == rule.ActionChoices.deny:
                 return False, matched_cmd
         return True, None
 
     def get_all_assets(self):
         from assets.models import Node
         nodes_keys = self.nodes.all().values_list('key', flat=True)
-        assets_ids = set(self.assets.all().values_list('id', flat=True))
-        nodes_assets_ids = Node.get_nodes_all_assets_ids(nodes_keys)
-        assets_ids.update(nodes_assets_ids)
-        assets = Asset.objects.filter(id__in=assets_ids)
+        asset_ids = set(self.assets.all().values_list('id', flat=True))
+        nodes_asset_ids = Node.get_nodes_all_asset_ids_by_keys(nodes_keys)
+        asset_ids.update(nodes_asset_ids)
+        assets = Asset.objects.filter(id__in=asset_ids)
         return assets
 
     @classmethod

apps/assets/pagination.py

@@ -1,39 +1,52 @@
 from rest_framework.pagination import LimitOffsetPagination
 from rest_framework.request import Request
 
+from common.utils import get_logger
 from assets.models import Node
 
+logger = get_logger(__name__)
+
+
+class AssetPaginationBase(LimitOffsetPagination):
+
+    def init_attrs(self, queryset, request: Request, view=None):
+        self._request = request
+        self._view = view
+        self._user = request.user
+
+    def paginate_queryset(self, queryset, request: Request, view=None):
+        self.init_attrs(queryset, request, view)
+        return super().paginate_queryset(queryset, request, view=None)
 
-class AssetLimitOffsetPagination(LimitOffsetPagination):
-    """
-    需要与 `assets.api.mixin.FilterAssetByNodeMixin` 配合使用
-    """
     def get_count(self, queryset):
-        """
-        1. 如果查询节点下的所有资产，那 count 使用 Node.assets_amount
-        2. 如果有其他过滤条件使用 super
-        3. 如果只查询该节点下的资产使用 super
-        """
         exclude_query_params = {
             self.limit_query_param,
             self.offset_query_param,
-            'node', 'all', 'show_current_asset',
-            'node_id', 'display', 'draw', 'fields_size',
+            'key', 'all', 'show_current_asset',
+            'cache_policy', 'display', 'draw',
+            'order', 'node', 'node_id', 'fields_size',
         }
-
         for k, v in self._request.query_params.items():
             if k not in exclude_query_params and v is not None:
+                logger.warn(f'Not hit node.assets_amount because find a unknow query_param `{k}` -> {self._request.get_full_path()}')
                 return super().get_count(queryset)
+        node_assets_count = self.get_count_from_nodes(queryset)
+        if node_assets_count is None:
+            return super().get_count(queryset)
+        return node_assets_count
 
+    def get_count_from_nodes(self, queryset):
+        raise NotImplementedError
+
+
+class NodeAssetTreePagination(AssetPaginationBase):
+    def get_count_from_nodes(self, queryset):
         is_query_all = self._view.is_query_node_all_assets
         if is_query_all:
             node = self._view.node
             if not node:
                 node = Node.org_root()
-            return node.assets_amount
-        return super().get_count(queryset)
-
-    def paginate_queryset(self, queryset, request: Request, view=None):
-        self._request = request
-        self._view = view
-        return super().paginate_queryset(queryset, request, view=None)
+            if node:
+                logger.debug(f'Hit node.assets_amount[{node.assets_amount}] -> {self._request.get_full_path()}')
+                return node.assets_amount
+        return None

apps/assets/serializers/admin_user.py

@@ -3,8 +3,6 @@
 from django.utils.translation import ugettext_lazy as _
 from rest_framework import serializers
 
-from common.drf.serializers import AdaptedBulkListSerializer
-
 from ..models import Node, AdminUser
 from orgs.mixins.serializers import BulkOrgResourceModelSerializer
 
@@ -17,15 +15,19 @@ class AdminUserSerializer(AuthSerializerMixin, BulkOrgResourceModelSerializer):
     """
 
     class Meta:
-        list_serializer_class = AdaptedBulkListSerializer
         model = AdminUser
-        fields = [
-            'id', 'name', 'username', 'password', 'private_key', 'public_key',
-            'comment', 'assets_amount', 'date_created', 'date_updated', 'created_by',
+        fields_mini  = ['id', 'name', 'username']
+        fields_write_only = ['password', 'private_key', 'public_key']
+        fields_small = fields_mini + fields_write_only + [
+            'date_created', 'date_updated',
+            'comment', 'created_by'
         ]
+        fields_fk = ['assets_amount']
+        fields = fields_small + fields_fk
         read_only_fields = ['date_created', 'date_updated', 'created_by', 'assets_amount']
 
         extra_kwargs = {
+            'username': {"required": True},
             'password': {"write_only": True},
             'private_key': {"write_only": True},
             'public_key': {"write_only": True},
@@ -33,6 +35,11 @@ class AdminUserSerializer(AuthSerializerMixin, BulkOrgResourceModelSerializer):
         }
 
 
+class AdminUserDetailSerializer(AdminUserSerializer):
+    class Meta(AdminUserSerializer.Meta):
+        fields = AdminUserSerializer.Meta.fields + ['ssh_key_fingerprint']
+
+
 class AdminUserAuthSerializer(AuthSerializer):
 
     class Meta:

apps/assets/serializers/asset.py

@@ -65,7 +65,7 @@ class AssetSerializer(BulkOrgResourceModelSerializer):
     platform = serializers.SlugRelatedField(
         slug_field='name', queryset=Platform.objects.all(), label=_("Platform")
     )
-    protocols = ProtocolsField(label=_('Protocols'), required=False)
+    protocols = ProtocolsField(label=_('Protocols'), required=False, default=['ssh/22'])
     domain_display = serializers.ReadOnlyField(source='domain.name', label=_('Domain name'))
     admin_user_display = serializers.ReadOnlyField(source='admin_user.name', label=_('Admin user name'))
     nodes_display = serializers.ListField(child=serializers.CharField(), label=_('Nodes name'), required=False)
@@ -111,7 +111,7 @@ class AssetSerializer(BulkOrgResourceModelSerializer):
     @classmethod
     def setup_eager_loading(cls, queryset):
         """ Perform necessary eager loading of data. """
-        queryset = queryset.select_related('admin_user', 'domain', 'platform')
+        queryset = queryset.prefetch_related('admin_user', 'domain', 'platform')
         queryset = queryset.prefetch_related('nodes', 'labels')
         return queryset
 
@@ -166,16 +166,9 @@ class AssetDisplaySerializer(AssetSerializer):
             'connectivity',
         ]
 
-    @classmethod
-    def setup_eager_loading(cls, queryset):
-        queryset = super().setup_eager_loading(queryset)
-        queryset = queryset\
-            .annotate(admin_user_username=F('admin_user__username'))
-        return queryset
-
 
 class PlatformSerializer(serializers.ModelSerializer):
-    meta = serializers.DictField(required=False, allow_null=True)
+    meta = serializers.DictField(required=False, allow_null=True, label=_('Meta'))
 
     def __init__(self, *args, **kwargs):
         super().__init__(*args, **kwargs)

apps/assets/serializers/asset_user.py

@@ -22,10 +22,11 @@ class AssetUserWriteSerializer(AuthSerializerMixin, BulkOrgResourceModelSerializ
     class Meta:
         model = AuthBook
         list_serializer_class = AdaptedBulkListSerializer
-        fields = [
-            'id', 'username', 'password', 'private_key', "public_key",
-            'asset', 'comment',
-        ]
+        fields_mini = ['id', 'username']
+        fields_write_only = ['password', 'private_key', "public_key"]
+        fields_small = fields_mini + fields_write_only + ['comment']
+        fields_fk = ['asset']
+        fields = fields_small + fields_fk
         extra_kwargs = {
             'username': {'required': True},
             'password': {'write_only': True},
@@ -52,11 +53,15 @@ class AssetUserReadSerializer(AssetUserWriteSerializer):
             'date_created', 'date_updated',
             'created_by', 'version',
         )
-        fields = [
-            'id', 'username', 'password', 'private_key', "public_key",
-            'asset', 'hostname', 'ip', 'backend', 'version',
-            'date_created', "date_updated", 'comment',
+        fields_mini = ['id', 'username']
+        fields_write_only = ['password', 'private_key', "public_key"]
+        fields_small = fields_mini + fields_write_only + [
+            'backend', 'version',
+            'date_created', "date_updated",
+            'comment'
         ]
+        fields_fk = ['asset', 'hostname', 'ip']
+        fields = fields_small + fields_fk
         extra_kwargs = {
             'username': {'required': True},
             'password': {'write_only': True},

apps/assets/serializers/base.py

@@ -41,10 +41,6 @@ class AuthSerializerMixin:
     def validate_private_key(self, private_key):
         if not private_key:
             return
-        if 'OPENSSH' in private_key:
-            msg = _("Not support openssh format key, using "
-                    "ssh-keygen -t rsa -m pem to generate")
-            raise serializers.ValidationError(msg)
         password = self.initial_data.get("password")
         valid = validate_ssh_private_key(private_key, password)
         if not valid:

apps/assets/serializers/cmd_filter.py

@@ -4,8 +4,11 @@ import re
 from rest_framework import serializers
 
 from common.drf.serializers import AdaptedBulkListSerializer
-from ..models import CommandFilter, CommandFilterRule, SystemUser
+from ..models import CommandFilter, CommandFilterRule
 from orgs.mixins.serializers import BulkOrgResourceModelSerializer
+from orgs.utils import tmp_to_root_org
+from common.utils import get_object_or_none, lazyproperty
+from terminal.models import Session
 
 
 class CommandFilterSerializer(BulkOrgResourceModelSerializer):
@@ -13,11 +16,16 @@ class CommandFilterSerializer(BulkOrgResourceModelSerializer):
     class Meta:
         model = CommandFilter
         list_serializer_class = AdaptedBulkListSerializer
-        fields = [
-            'id', 'name', 'org_id', 'org_name', 'is_active', 'comment',
-            'created_by', 'date_created', 'date_updated', 'rules', 'system_users'
+        fields_mini = ['id', 'name']
+        fields_small = fields_mini + [
+            'org_id', 'org_name',
+            'is_active',
+            'date_created', 'date_updated',
+            'comment', 'created_by',
         ]
-
+        fields_fk = ['rules']
+        fields_m2m = ['system_users']
+        fields = fields_small + fields_fk + fields_m2m
         extra_kwargs = {
             'rules': {'read_only': True},
             'system_users': {'required': False},
@@ -34,13 +42,28 @@ class CommandFilterRuleSerializer(BulkOrgResourceModelSerializer):
         fields_mini = ['id']
         fields_small = fields_mini + [
            'type', 'type_display', 'content', 'priority',
-           'action', 'action_display',
-           'comment', 'created_by', 'date_created', 'date_updated'
+           'action', 'action_display', 'reviewers',
+           'date_created', 'date_updated',
+           'comment', 'created_by',
         ]
         fields_fk = ['filter']
         fields = '__all__'
         list_serializer_class = AdaptedBulkListSerializer
 
+    def __init__(self, *args, **kwargs):
+        super().__init__(*args, **kwargs)
+        self.set_action_choices()
+
+    def set_action_choices(self):
+        from django.conf import settings
+        action = self.fields.get('action')
+        if not action:
+            return
+        choices = action._choices
+        if not settings.XPACK_ENABLED:
+            choices.pop(CommandFilterRule.ActionChoices.confirm, None)
+        action._choices = choices
+
     # def validate_content(self, content):
     #     tp = self.initial_data.get("type")
     #     if tp == CommandFilterRule.TYPE_REGEX:
@@ -50,3 +73,35 @@ class CommandFilterRuleSerializer(BulkOrgResourceModelSerializer):
     #         msg = _("Content should not be contain: {}").format(invalid_char)
     #         raise serializers.ValidationError(msg)
     #     return content
+
+
+class CommandConfirmSerializer(serializers.Serializer):
+    session_id = serializers.UUIDField(required=True, allow_null=False)
+    cmd_filter_rule_id = serializers.UUIDField(required=True, allow_null=False)
+    run_command = serializers.CharField(required=True, allow_null=False)
+
+    def __init__(self, *args, **kwargs):
+        super().__init__(*args, **kwargs)
+        self.session = None
+        self.cmd_filter_rule = None
+
+    def validate_session_id(self, session_id):
+        self.session = self.validate_object_exist(Session, session_id)
+        return session_id
+
+    def validate_cmd_filter_rule_id(self, cmd_filter_rule_id):
+        self.cmd_filter_rule = self.validate_object_exist(CommandFilterRule, cmd_filter_rule_id)
+        return cmd_filter_rule_id
+
+    @staticmethod
+    def validate_object_exist(model, field_id):
+        with tmp_to_root_org():
+            obj = get_object_or_none(model, id=field_id)
+        if not obj:
+            error = '{} Model object does not exist'.format(model.__name__)
+            raise serializers.ValidationError(error)
+        return obj
+
+    @lazyproperty
+    def org(self):
+        return self.session.org

apps/assets/serializers/domain.py

@@ -48,13 +48,22 @@ class GatewaySerializer(AuthSerializerMixin, BulkOrgResourceModelSerializer):
     class Meta:
         model = Gateway
         list_serializer_class = AdaptedBulkListSerializer
-        fields = [
-            'id', 'name', 'ip', 'port', 'protocol', 'username', 'password',
-            'private_key', 'public_key', 'domain', 'is_active', 'date_created',
-            'date_updated', 'created_by', 'comment',
+        fields_mini = ['id', 'name']
+        fields_write_only = [
+            'password', 'private_key', 'public_key',
         ]
+        fields_small = fields_mini + fields_write_only + [
+            'username', 'ip', 'port', 'protocol',
+            'is_active',
+            'date_created', 'date_updated',
+            'created_by', 'comment',
+        ]
+        fields_fk = ['domain']
+        fields = fields_small + fields_fk
         extra_kwargs = {
-            'password': {'validators': [NoSpecialChars()]}
+            'password': {'write_only': True, 'validators': [NoSpecialChars()]},
+            'private_key': {"write_only": True},
+            'public_key': {"write_only": True},
         }
 
     def __init__(self, *args, **kwargs):
@@ -69,14 +78,12 @@ class GatewaySerializer(AuthSerializerMixin, BulkOrgResourceModelSerializer):
 
 
 class GatewayWithAuthSerializer(GatewaySerializer):
-    def get_field_names(self, declared_fields, info):
-        fields = super().get_field_names(declared_fields, info)
-        fields.extend(
-            ['password', 'private_key']
-        )
-        return fields
-
-
+    class Meta(GatewaySerializer.Meta):
+        extra_kwargs = {
+            'password': {'write_only': False, 'validators': [NoSpecialChars()]},
+            'private_key': {"write_only": False},
+            'public_key': {"write_only": False},
+        }
 
 
 class DomainWithGatewaySerializer(BulkOrgResourceModelSerializer):

apps/assets/serializers/gathered_user.py

@@ -10,11 +10,14 @@ from ..models import GatheredUser
 class GatheredUserSerializer(OrgResourceModelSerializerMixin):
     class Meta:
         model = GatheredUser
-        fields = [
-            'id', 'asset', 'hostname', 'ip', 'username',
-            'date_last_login', 'ip_last_login',
-            'present', 'date_created', 'date_updated'
+        fields_mini = ['id']
+        fields_small = fields_mini + [
+            'username', 'ip_last_login',
+            'present',
+            'date_last_login', 'date_created', 'date_updated'
         ]
+        fields_fk = ['asset', 'hostname', 'ip']
+        fields = fields_small + fields_fk
         read_only_fields = fields
         extra_kwargs = {
             'hostname': {'label': _("Hostname")},

apps/assets/serializers/label.py

@@ -1,6 +1,7 @@
 # -*- coding: utf-8 -*-
 #
 from rest_framework import serializers
+from django.utils.translation import ugettext_lazy as _
 
 from common.drf.serializers import AdaptedBulkListSerializer
 from orgs.mixins.serializers import BulkOrgResourceModelSerializer
@@ -9,16 +10,22 @@ from ..models import Label
 
 
 class LabelSerializer(BulkOrgResourceModelSerializer):
-    asset_count = serializers.SerializerMethodField()
+    asset_count = serializers.SerializerMethodField(label=_("Assets amount"))
+    category_display = serializers.ReadOnlyField(source='get_category_display', label=_('Category display'))
 
     class Meta:
         model = Label
-        fields = [
-            'id', 'name', 'value', 'category', 'is_active', 'comment',
-            'date_created', 'asset_count', 'assets', 'get_category_display'
+        fields_mini = ['id', 'name']
+        fields_small = fields_mini + [
+            'value', 'category', 'category_display',
+            'is_active',
+            'date_created',
+            'comment',
         ]
+        fields_m2m = ['asset_count', 'assets']
+        fields = fields_small + fields_m2m
         read_only_fields = (
-            'category', 'date_created', 'asset_count', 'get_category_display'
+            'category', 'date_created', 'asset_count',
         )
         extra_kwargs = {
             'assets': {'required': False}

apps/assets/serializers/system_user.py

@@ -26,16 +26,18 @@ class SystemUserSerializer(AuthSerializerMixin, BulkOrgResourceModelSerializer):
     class Meta:
         model = SystemUser
         list_serializer_class = AdaptedBulkListSerializer
-        fields = [
-            'id', 'name', 'username', 'protocol',
-            'password', 'public_key', 'private_key',
-            'login_mode', 'login_mode_display',
-            'priority', 'username_same_with_user',
-            'auto_push', 'cmd_filters', 'sudo', 'shell', 'comment',
-            'auto_generate_key', 'sftp_root', 'token',
-            'assets_amount', 'date_created', 'created_by',
-            'home', 'system_groups', 'ad_domain'
+        fields_mini = ['id', 'name', 'username']
+        fields_write_only = ['password', 'public_key', 'private_key']
+        fields_small = fields_mini + fields_write_only + [
+            'protocol', 'login_mode', 'login_mode_display', 'priority',
+            'sudo', 'shell', 'sftp_root', 'token',
+            'home', 'system_groups', 'ad_domain',
+            'username_same_with_user', 'auto_push', 'auto_generate_key',
+            'date_created', 'date_updated',
+            'comment', 'created_by',
         ]
+        fields_m2m = [ 'cmd_filters', 'assets_amount']
+        fields = fields_small + fields_m2m
         extra_kwargs = {
             'password': {"write_only": True},
             'public_key': {"write_only": True},
@@ -101,6 +103,12 @@ class SystemUserSerializer(AuthSerializerMixin, BulkOrgResourceModelSerializer):
             raise serializers.ValidationError(msg)
         return username
 
+    def validate_home(self, home):
+        username_same_with_user = self.initial_data.get("username_same_with_user")
+        if username_same_with_user:
+            return ''
+        return home
+
     def validate_sftp_root(self, value):
         if value in ['home', 'tmp']:
             return value
@@ -147,16 +155,18 @@ class SystemUserSerializer(AuthSerializerMixin, BulkOrgResourceModelSerializer):
 class SystemUserListSerializer(SystemUserSerializer):
 
     class Meta(SystemUserSerializer.Meta):
-        fields = [
-            'id', 'name', 'username', 'protocol',
-            'password', 'public_key', 'private_key',
-            'login_mode', 'login_mode_display',
-            'priority', "username_same_with_user",
-            'auto_push', 'sudo', 'shell', 'comment',
-            "assets_amount", 'home', 'system_groups',
-            'auto_generate_key', 'ad_domain',
-            'sftp_root',
+        fields_mini = ['id', 'name', 'username']
+        fields_write_only = ['password', 'public_key', 'private_key']
+        fields_small = fields_mini + fields_write_only + [
+            'protocol', 'login_mode', 'login_mode_display', 'priority',
+            'sudo', 'shell', 'home', 'system_groups',
+            'ad_domain', 'sftp_root',
+            "username_same_with_user", 'auto_push', 'auto_generate_key',
+            'date_created', 'date_updated',
+            'comment', 'created_by',
         ]
+        fields_m2m = ["assets_amount",]
+        fields = fields_small + fields_m2m
         extra_kwargs = {
             'password': {"write_only": True},
             'public_key': {"write_only": True},
@@ -177,15 +187,15 @@ class SystemUserListSerializer(SystemUserSerializer):
 
 class SystemUserWithAuthInfoSerializer(SystemUserSerializer):
     class Meta(SystemUserSerializer.Meta):
-        fields = [
-            'id', 'name', 'username', 'protocol',
-            'password', 'public_key', 'private_key',
-            'login_mode', 'login_mode_display',
-            'priority', 'username_same_with_user',
-            'auto_push', 'sudo', 'shell', 'comment',
-            'auto_generate_key', 'sftp_root', 'token',
-            'ad_domain',
+        fields_mini = ['id', 'name', 'username']
+        fields_write_only = ['password', 'public_key', 'private_key']
+        fields_small = fields_mini + fields_write_only + [
+            'protocol', 'login_mode', 'login_mode_display', 'priority',
+            'sudo', 'shell', 'ad_domain', 'sftp_root', 'token',
+            "username_same_with_user", 'auto_push', 'auto_generate_key',
+            'comment',
         ]
+        fields = fields_small
         extra_kwargs = {
             'nodes_amount': {'label': _('Node')},
             'assets_amount': {'label': _('Asset')},

apps/assets/signals_handler/__init__.py

@@ -0,0 +1,3 @@
+from .common import *
+from .node_assets_amount import *
+from .node_assets_mapping import *

apps/assets/signals_handler/common.py

@@ -1,21 +1,17 @@
 # -*- coding: utf-8 -*-
 #
-from operator import add, sub
-
-from assets.utils import is_asset_exists_in_node
 from django.db.models.signals import (
     post_save, m2m_changed, pre_delete, post_delete, pre_save
 )
-from django.db.models import Q, F
 from django.dispatch import receiver
 
 from common.exceptions import M2MReverseNotAllowed
-from common.const.signals import PRE_ADD, POST_ADD, POST_REMOVE, PRE_CLEAR, PRE_REMOVE
+from common.const.signals import POST_ADD, POST_REMOVE, PRE_REMOVE
 from common.utils import get_logger
 from common.decorator import on_transaction_commit
-from .models import Asset, SystemUser, Node, compute_parent_key
+from assets.models import Asset, SystemUser, Node
 from users.models import User
-from .tasks import (
+from assets.tasks import (
     update_assets_hardware_info_util,
     test_asset_connectivity_util,
     push_system_user_to_assets_manual,
@@ -23,7 +19,6 @@ from .tasks import (
     add_nodes_assets_to_system_users
 )
 
-
 logger = get_logger(__file__)
 
 
@@ -87,13 +82,13 @@ def on_system_user_assets_change(instance, action, model, pk_set, **kwargs):
         return
     logger.debug("System user assets change signal recv: {}".format(instance))
     if model == Asset:
-        system_users_id = [instance.id]
-        assets_id = pk_set
+        system_user_ids = [instance.id]
+        asset_ids = pk_set
     else:
-        system_users_id = pk_set
-        assets_id = [instance.id]
-    for system_user_id in system_users_id:
-        push_system_user_to_assets.delay(system_user_id, assets_id)
+        system_user_ids = pk_set
+        asset_ids = [instance.id]
+    for system_user_id in system_user_ids:
+        push_system_user_to_assets.delay(system_user_id, asset_ids)
 
 
 @receiver(m2m_changed, sender=SystemUser.users.through)
@@ -198,138 +193,11 @@ def on_asset_nodes_add(instance, action, reverse, pk_set, **kwargs):
                 systemuser_id=system_user_id,
                 asset_id=asset_id
             ))
-        push_system_user_to_assets.delay(system_user_id, asset_ids_to_push)
+        if asset_ids_to_push:
+            push_system_user_to_assets.delay(system_user_id, asset_ids_to_push)
     m2m_model.objects.bulk_create(to_create)
 
 
-def _update_node_assets_amount(node: Node, asset_pk_set: set, operator=add):
-    """
-    一个节点与多个资产关系变化时，更新计数
-
-    :param node: 节点实例
-    :param asset_pk_set: 资产的`id`集合, 内部不会修改该值
-    :param operator: 操作
-    * -> Node
-    # -> Asset
-
-           * [3]
-          / \
-         *   * [2]
-        /     \
-       *       * [1]
-      /       / \
-     *   [a] #  # [b]
-
-    """
-    # 获取节点[1]祖先节点的 `key` 含自己，也就是[1, 2, 3]节点的`key`
-    ancestor_keys = node.get_ancestor_keys(with_self=True)
-    ancestors = Node.objects.filter(key__in=ancestor_keys).order_by('-key')
-    to_update = []
-    for ancestor in ancestors:
-        # 迭代祖先节点的`key`，顺序是 [1] -> [2] -> [3]
-        # 查询该节点及其后代节点是否包含要操作的资产，将包含的从要操作的
-        # 资产集合中去掉，他们是重复节点，无论增加或删除都不会影响节点的资产数量
-
-        asset_pk_set -= set(Asset.objects.filter(
-            id__in=asset_pk_set
-        ).filter(
-            Q(nodes__key__istartswith=f'{ancestor.key}:') |
-            Q(nodes__key=ancestor.key)
-        ).distinct().values_list('id', flat=True))
-        if not asset_pk_set:
-            # 要操作的资产集合为空，说明都是重复资产，不用改变节点资产数量
-            # 而且既然它包含了，它的祖先节点肯定也包含了，所以祖先节点都不用
-            # 处理了
-            break
-        ancestor.assets_amount = operator(F('assets_amount'), len(asset_pk_set))
-        to_update.append(ancestor)
-    Node.objects.bulk_update(to_update, fields=('assets_amount', 'parent_key'))
-
-
-def _remove_ancestor_keys(ancestor_key, tree_set):
-    # 这里判断 `ancestor_key` 不能是空，防止数据错误导致的死循环
-    # 判断是否在集合里，来区分是否已被处理过
-    while ancestor_key and ancestor_key in tree_set:
-        tree_set.remove(ancestor_key)
-        ancestor_key = compute_parent_key(ancestor_key)
-
-
-def _update_nodes_asset_amount(node_keys, asset_pk, operator):
-    """
-    一个资产与多个节点关系变化时，更新计数
-
-    :param node_keys: 节点 id 的集合
-    :param asset_pk: 资产 id
-    :param operator: 操作
-    """
-
-    # 所有相关节点的祖先节点，组成一棵局部树
-    ancestor_keys = set()
-    for key in node_keys:
-        ancestor_keys.update(Node.get_node_ancestor_keys(key))
-
-    # 相关节点可能是其他相关节点的祖先节点，如果是从相关节点里干掉
-    node_keys -= ancestor_keys
-
-    to_update_keys = []
-    for key in node_keys:
-        # 遍历相关节点，处理它及其祖先节点
-        # 查询该节点是否包含待处理资产
-        exists = is_asset_exists_in_node(asset_pk, key)
-        parent_key = compute_parent_key(key)
-
-        if exists:
-            # 如果资产在该节点，那么他及其祖先节点都不用处理
-            _remove_ancestor_keys(parent_key, ancestor_keys)
-            continue
-        else:
-            # 不存在，要更新本节点
-            to_update_keys.append(key)
-            # 这里判断 `parent_key` 不能是空，防止数据错误导致的死循环
-            # 判断是否在集合里，来区分是否已被处理过
-            while parent_key and parent_key in ancestor_keys:
-                exists = is_asset_exists_in_node(asset_pk, parent_key)
-                if exists:
-                    _remove_ancestor_keys(parent_key, ancestor_keys)
-                    break
-                else:
-                    to_update_keys.append(parent_key)
-                    ancestor_keys.remove(parent_key)
-                    parent_key = compute_parent_key(parent_key)
-
-    Node.objects.filter(key__in=to_update_keys).update(
-        assets_amount=operator(F('assets_amount'), 1)
-    )
-
-
-@receiver(m2m_changed, sender=Asset.nodes.through)
-def update_nodes_assets_amount(action, instance, reverse, pk_set, **kwargs):
-    # 不允许 `pre_clear` ，因为该信号没有 `pk_set`
-    # [官网](https://docs.djangoproject.com/en/3.1/ref/signals/#m2m-changed)
-    refused = (PRE_CLEAR,)
-    if action in refused:
-        raise ValueError
-
-    mapper = {
-        PRE_ADD: add,
-        POST_REMOVE: sub
-    }
-    if action not in mapper:
-        return
-
-    operator = mapper[action]
-
-    if reverse:
-        node: Node = instance
-        asset_pk_set = set(pk_set)
-        _update_node_assets_amount(node, asset_pk_set, operator)
-    else:
-        asset_pk = instance.id
-        # 与资产直接关联的节点
-        node_keys = set(Node.objects.filter(id__in=pk_set).values_list('key', flat=True))
-        _update_nodes_asset_amount(node_keys, asset_pk, operator)
-
-
 RELATED_NODE_IDS = '_related_node_ids'
 
 

apps/assets/signals_handler/node_assets_amount.py

@@ -0,0 +1,160 @@
+# -*- coding: utf-8 -*-
+#
+from operator import add, sub
+from django.db.models import Q, F
+from django.dispatch import receiver
+from django.db.models.signals import (
+    m2m_changed
+)
+
+from orgs.utils import ensure_in_real_or_default_org, tmp_to_org
+from common.const.signals import PRE_ADD, POST_REMOVE, PRE_CLEAR
+from common.utils import get_logger
+from assets.models import Asset, Node, compute_parent_key
+from assets.locks import NodeTreeUpdateLock
+
+
+logger = get_logger(__file__)
+
+
+@receiver(m2m_changed, sender=Asset.nodes.through)
+def on_node_asset_change(sender, action, instance, reverse, pk_set, **kwargs):
+    # 不允许 `pre_clear` ，因为该信号没有 `pk_set`
+    # [官网](https://docs.djangoproject.com/en/3.1/ref/signals/#m2m-changed)
+    refused = (PRE_CLEAR,)
+    if action in refused:
+        raise ValueError
+
+    mapper = {
+        PRE_ADD: add,
+        POST_REMOVE: sub
+    }
+    if action not in mapper:
+        return
+
+    operator = mapper[action]
+
+    with tmp_to_org(instance.org):
+        if reverse:
+            node: Node = instance
+            asset_pk_set = set(pk_set)
+            NodeAssetsAmountUtils.update_node_assets_amount(node, asset_pk_set, operator)
+        else:
+            asset_pk = instance.id
+            # 与资产直接关联的节点
+            node_keys = set(Node.objects.filter(id__in=pk_set).values_list('key', flat=True))
+            NodeAssetsAmountUtils.update_nodes_asset_amount(node_keys, asset_pk, operator)
+
+
+class NodeAssetsAmountUtils:
+
+    @classmethod
+    def _remove_ancestor_keys(cls, ancestor_key, tree_set):
+        # 这里判断 `ancestor_key` 不能是空，防止数据错误导致的死循环
+        # 判断是否在集合里，来区分是否已被处理过
+        while ancestor_key and ancestor_key in tree_set:
+            tree_set.remove(ancestor_key)
+            ancestor_key = compute_parent_key(ancestor_key)
+
+    @classmethod
+    def _is_asset_exists_in_node(cls, asset_pk, node_key):
+        exists = Asset.objects.filter(
+            Q(nodes__key__istartswith=f'{node_key}:') | Q(nodes__key=node_key)
+        ).filter(id=asset_pk).exists()
+        return exists
+
+    @classmethod
+    @ensure_in_real_or_default_org
+    @NodeTreeUpdateLock()
+    def update_nodes_asset_amount(cls, node_keys, asset_pk, operator):
+        """
+        一个资产与多个节点关系变化时，更新计数
+
+        :param node_keys: 节点 id 的集合
+        :param asset_pk: 资产 id
+        :param operator: 操作
+        """
+
+        # 所有相关节点的祖先节点，组成一棵局部树
+        ancestor_keys = set()
+        for key in node_keys:
+            ancestor_keys.update(Node.get_node_ancestor_keys(key))
+
+        # 相关节点可能是其他相关节点的祖先节点，如果是从相关节点里干掉
+        node_keys -= ancestor_keys
+
+        to_update_keys = []
+        for key in node_keys:
+            # 遍历相关节点，处理它及其祖先节点
+            # 查询该节点是否包含待处理资产
+            exists = cls._is_asset_exists_in_node(asset_pk, key)
+            parent_key = compute_parent_key(key)
+
+            if exists:
+                # 如果资产在该节点，那么他及其祖先节点都不用处理
+                cls._remove_ancestor_keys(parent_key, ancestor_keys)
+                continue
+            else:
+                # 不存在，要更新本节点
+                to_update_keys.append(key)
+                # 这里判断 `parent_key` 不能是空，防止数据错误导致的死循环
+                # 判断是否在集合里，来区分是否已被处理过
+                while parent_key and parent_key in ancestor_keys:
+                    exists = cls._is_asset_exists_in_node(asset_pk, parent_key)
+                    if exists:
+                        cls._remove_ancestor_keys(parent_key, ancestor_keys)
+                        break
+                    else:
+                        to_update_keys.append(parent_key)
+                        ancestor_keys.remove(parent_key)
+                        parent_key = compute_parent_key(parent_key)
+
+        Node.objects.filter(key__in=to_update_keys).update(
+            assets_amount=operator(F('assets_amount'), 1)
+        )
+
+    @classmethod
+    @ensure_in_real_or_default_org
+    @NodeTreeUpdateLock()
+    def update_node_assets_amount(cls, node: Node, asset_pk_set: set, operator=add):
+        """
+        一个节点与多个资产关系变化时，更新计数
+
+        :param node: 节点实例
+        :param asset_pk_set: 资产的`id`集合, 内部不会修改该值
+        :param operator: 操作
+        * -> Node
+        # -> Asset
+
+               * [3]
+              / \
+             *   * [2]
+            /     \
+           *       * [1]
+          /       / \
+         *   [a] #  # [b]
+
+        """
+        # 获取节点[1]祖先节点的 `key` 含自己，也就是[1, 2, 3]节点的`key`
+        ancestor_keys = node.get_ancestor_keys(with_self=True)
+        ancestors = Node.objects.filter(key__in=ancestor_keys).order_by('-key')
+        to_update = []
+        for ancestor in ancestors:
+            # 迭代祖先节点的`key`，顺序是 [1] -> [2] -> [3]
+            # 查询该节点及其后代节点是否包含要操作的资产，将包含的从要操作的
+            # 资产集合中去掉，他们是重复节点，无论增加或删除都不会影响节点的资产数量
+
+            asset_pk_set -= set(Asset.objects.filter(
+                id__in=asset_pk_set
+            ).filter(
+                Q(nodes__key__istartswith=f'{ancestor.key}:') |
+                Q(nodes__key=ancestor.key)
+            ).distinct().values_list('id', flat=True))
+            if not asset_pk_set:
+                # 要操作的资产集合为空，说明都是重复资产，不用改变节点资产数量
+                # 而且既然它包含了，它的祖先节点肯定也包含了，所以祖先节点都不用
+                # 处理了
+                break
+            ancestor.assets_amount = operator(F('assets_amount'), len(asset_pk_set))
+            to_update.append(ancestor)
+        Node.objects.bulk_update(to_update, fields=('assets_amount', 'parent_key'))

apps/assets/signals_handler/node_assets_mapping.py

@@ -0,0 +1,100 @@
+# -*- coding: utf-8 -*-
+#
+import os
+import threading
+
+from django.db.models.signals import (
+    m2m_changed, post_save, post_delete
+)
+from django.dispatch import receiver
+from django.utils.functional import LazyObject
+
+from common.signals import django_ready
+from common.utils.connection import RedisPubSub
+from common.utils import get_logger
+from assets.models import Asset, Node
+from orgs.models import Organization
+
+
+logger = get_logger(__file__)
+
+# clear node assets mapping for memory
+# ------------------------------------
+
+
+def get_node_assets_mapping_for_memory_pub_sub():
+    return RedisPubSub('fm.node_all_asset_ids_memory_mapping')
+
+
+class NodeAssetsMappingForMemoryPubSub(LazyObject):
+    def _setup(self):
+        self._wrapped = get_node_assets_mapping_for_memory_pub_sub()
+
+
+node_assets_mapping_for_memory_pub_sub = NodeAssetsMappingForMemoryPubSub()
+
+
+def expire_node_assets_mapping_for_memory(org_id):
+    # 所有进程清除(自己的 memory 数据)
+    org_id = str(org_id)
+    root_org_id = Organization.ROOT_ID
+
+    # 当前进程清除(cache 数据)
+    logger.debug(
+        "Expire node assets id mapping from cache of org={}, pid={}"
+        "".format(org_id, os.getpid())
+    )
+    Node.expire_node_all_asset_ids_mapping_from_cache(org_id)
+    Node.expire_node_all_asset_ids_mapping_from_cache(root_org_id)
+
+    node_assets_mapping_for_memory_pub_sub.publish(org_id)
+    node_assets_mapping_for_memory_pub_sub.publish(root_org_id)
+
+
+@receiver(post_save, sender=Node)
+def on_node_post_create(sender, instance, created, update_fields, **kwargs):
+    if created:
+        need_expire = True
+    elif update_fields and 'key' in update_fields:
+        need_expire = True
+    else:
+        need_expire = False
+
+    if need_expire:
+        expire_node_assets_mapping_for_memory(instance.org_id)
+
+
+@receiver(post_delete, sender=Node)
+def on_node_post_delete(sender, instance, **kwargs):
+    expire_node_assets_mapping_for_memory(instance.org_id)
+
+
+@receiver(m2m_changed, sender=Asset.nodes.through)
+def on_node_asset_change(sender, instance, **kwargs):
+    expire_node_assets_mapping_for_memory(instance.org_id)
+
+
+@receiver(django_ready)
+def subscribe_node_assets_mapping_expire(sender, **kwargs):
+    logger.debug("Start subscribe for expire node assets id mapping from memory")
+
+    def keep_subscribe():
+        while True:
+            try:
+                subscribe = node_assets_mapping_for_memory_pub_sub.subscribe()
+                for message in subscribe.listen():
+                    if message["type"] != "message":
+                        continue
+                    org_id = message['data'].decode()
+                    Node.expire_node_all_asset_ids_mapping_from_memory(org_id)
+                    logger.debug(
+                        "Expire node assets id mapping from memory of org={}, pid={}"
+                        "".format(str(org_id), os.getpid())
+                    )
+            except Exception as e:
+                logger.exception(f'subscribe_node_assets_mapping_expire: {e}')
+                Node.expire_all_orgs_node_all_asset_ids_mapping_from_memory()
+
+    t = threading.Thread(target=keep_subscribe)
+    t.daemon = True
+    t.start()

apps/assets/tasks/common.py

@@ -12,6 +12,7 @@ __all__ = ['add_nodes_assets_to_system_users']
 @tmp_to_root_org()
 def add_nodes_assets_to_system_users(nodes_keys, system_users):
     from ..models import Node
-    assets = Node.get_nodes_all_assets(nodes_keys).values_list('id', flat=True)
+    nodes = Node.objects.filter(key__in=nodes_keys)
+    assets = Node.get_nodes_all_assets(*nodes)
     for system_user in system_users:
         system_user.assets.add(*tuple(assets))

apps/assets/tasks/gather_asset_users.py

@@ -141,7 +141,8 @@ def gather_asset_users(assets, task_name=None):
 
 @shared_task(queue="ansible")
 def gather_nodes_asset_users(nodes_key):
-    assets = Node.get_nodes_all_assets(nodes_key)
+    nodes = Node.objects.filter(key__in=nodes_key)
+    assets = Node.get_nodes_all_assets(*nodes)
     assets_groups_by_100 = [assets[i:i+100] for i in range(0, len(assets), 100)]
     for _assets in assets_groups_by_100:
         gather_asset_users(_assets)

apps/assets/tasks/nodes_amount.py

@@ -12,16 +12,24 @@ from common.utils import get_logger
 logger = get_logger(__file__)
 
 
-@shared_task(queue='celery_heavy_tasks')
-def check_node_assets_amount_task(org_id=Organization.ROOT_ID):
-    try:
-        with tmp_to_org(Organization.get_instance(org_id)):
-            check_node_assets_amount()
-    except AcquireFailed:
-        logger.error(_('The task of self-checking is already running and cannot be started repeatedly'))
+@shared_task
+def check_node_assets_amount_task(org_id=None):
+    if org_id is None:
+        orgs = Organization.objects.all()
+    else:
+        orgs = [Organization.get_instance(org_id)]
+
+    for org in orgs:
+        try:
+            with tmp_to_org(org):
+                check_node_assets_amount()
+        except AcquireFailed:
+            error = _('The task of self-checking is already running '
+                      'and cannot be started repeatedly')
+            logger.error(error)
 
 
 @register_as_period_task(crontab='0 2 * * *')
-@shared_task(queue='celery_heavy_tasks')
+@shared_task
 def check_node_assets_amount_period_task():
     check_node_assets_amount_task()

apps/assets/tasks/push_system_user.py

@@ -32,11 +32,19 @@ def _dump_args(args: dict):
 
 
 def get_push_unixlike_system_user_tasks(system_user, username=None):
+    comment = system_user.name
+
     if username is None:
         username = system_user.username
+
+    if system_user.username_same_with_user:
+        from users.models import User
+        user = User.objects.filter(username=username).only('name', 'username').first()
+        if user:
+            comment = f'{system_user.name}[{str(user)}]'
+
     password = system_user.password
     public_key = system_user.public_key
-    comment = system_user.name
 
     groups = _split_by_comma(system_user.system_groups)
 
@@ -48,6 +56,7 @@ def get_push_unixlike_system_user_tasks(system_user, username=None):
         'shell': system_user.shell or Empty,
         'state': 'present',
         'home': system_user.home or Empty,
+        'expires': -1,
         'groups': groups or Empty,
         'comment': comment
     }
@@ -225,18 +234,18 @@ def push_system_user_util(system_user, assets, task_name, username=None):
         print(_("Hosts count: {}").format(len(_assets)))
 
         id_asset_map = {_asset.id: _asset for _asset in _assets}
-        assets_id = id_asset_map.keys()
+        asset_ids = id_asset_map.keys()
         no_special_auth = []
         special_auth_set = set()
 
-        auth_books = AuthBook.objects.filter(username__in=usernames, asset_id__in=assets_id)
+        auth_books = AuthBook.objects.filter(username__in=usernames, asset_id__in=asset_ids)
 
         for auth_book in auth_books:
             special_auth_set.add((auth_book.username, auth_book.asset_id))
 
         for _username in usernames:
             no_special_assets = []
-            for asset_id in assets_id:
+            for asset_id in asset_ids:
                 if (_username, asset_id) not in special_auth_set:
                     no_special_assets.append(id_asset_map[asset_id])
             if no_special_assets:
@@ -281,12 +290,12 @@ def push_system_user_a_asset_manual(system_user, asset, username=None):
 
 @shared_task(queue="ansible")
 @tmp_to_root_org()
-def push_system_user_to_assets(system_user_id, assets_id, username=None):
+def push_system_user_to_assets(system_user_id, asset_ids, username=None):
     """
     推送系统用户到指定的若干资产上
     """
     system_user = SystemUser.objects.get(id=system_user_id)
-    assets = get_objects(Asset, assets_id)
+    assets = get_objects(Asset, asset_ids)
     task_name = _("Push system users to assets: {}").format(system_user.name)
 
     return push_system_user_util(system_user, assets, task_name, username=username)

apps/assets/tasks/system_user_connectivity.py

@@ -5,6 +5,7 @@ from collections import defaultdict
 from celery import shared_task
 from django.utils.translation import ugettext as _
 
+from assets.models import Asset
 from common.utils import get_logger
 from orgs.utils import tmp_to_org, org_aware_func
 from ..models import SystemUser
@@ -96,9 +97,12 @@ def test_system_user_connectivity_util(system_user, assets, task_name):
 
 @shared_task(queue="ansible")
 @org_aware_func("system_user")
-def test_system_user_connectivity_manual(system_user):
+def test_system_user_connectivity_manual(system_user, asset_ids=None):
     task_name = _("Test system user connectivity: {}").format(system_user)
-    assets = system_user.get_related_assets()
+    if asset_ids:
+        assets = Asset.objects.filter(id__in=asset_ids)
+    else:
+        assets = system_user.get_related_assets()
     test_system_user_connectivity_util(system_user, assets, task_name)
 
 

apps/assets/tasks/utils.py

@@ -25,10 +25,13 @@ def check_asset_can_run_ansible(asset):
 
 
 def check_system_user_can_run_ansible(system_user):
-    if not system_user.is_need_push():
-        msg = _("Push system user task skip, auto push not enable or "
-                "protocol is not ssh or rdp: {}").format(system_user.name)
-        logger.info(msg)
+    if not system_user.auto_push:
+        logger.warn(f'Push system user task skip, auto push not enable: system_user={system_user.name}')
+        return False
+    if not system_user.is_protocol_support_push:
+        logger.warn(f'Push system user task skip, protocol not support: '
+                    f'system_user={system_user.name} protocol={system_user.protocol} '
+                    f'support_protocol={system_user.SUPPORT_PUSH_PROTOCOLS}')
         return False
 
     # Push root as system user is dangerous
@@ -37,10 +40,6 @@ def check_system_user_can_run_ansible(system_user):
         logger.info(msg)
         return False
 
-    # if system_user.protocol != "ssh":
-    #     msg = _("System user protocol not ssh: {}".format(system_user))
-    #     logger.info(msg)
-    #     return False
     return True
 
 

apps/assets/tests/tree.py

@@ -0,0 +1,33 @@
+from assets.tree import Tree
+
+
+def test():
+    from orgs.models import Organization
+    from assets.models import Node, Asset
+    import time
+    Organization.objects.get(id='1863cf22-f666-474e-94aa-935fe175203c').change_to()
+
+    t1 = time.time()
+    nodes = list(Node.objects.exclude(key__startswith='-').only('id', 'key', 'parent_key'))
+    node_asset_id_pairs = Asset.nodes.through.objects.all().values_list('node_id', 'asset_id')
+    t2 = time.time()
+    node_asset_id_pairs = list(node_asset_id_pairs)
+    tree = Tree(nodes,  node_asset_id_pairs)
+    tree.build_tree()
+    tree.nodes = None
+    tree.node_asset_id_pairs = None
+    import pickle
+    d = pickle.dumps(tree)
+    print('------------', len(d))
+    return tree
+    tree.compute_tree_node_assets_amount()
+
+    print(f'校对算法准确性 ......')
+    for node in nodes:
+        tree_node = tree.key_tree_node_mapper[node.key]
+        if tree_node.assets_amount != node.assets_amount:
+            print(f'ERROR: {tree_node.assets_amount} {node.assets_amount}')
+        # print(f'OK {tree_node.asset_amount} {node.assets_amount}')
+
+    print(f'数据库时间: {t2 - t1}')
+    return tree

apps/assets/urls/api_urls.py

@@ -2,7 +2,6 @@
 from django.urls import path, re_path
 from rest_framework_nested import routers
 from rest_framework_bulk.routes import BulkRouter
-from django.db.transaction import non_atomic_requests
 
 from common import api as capi
 
@@ -57,13 +56,16 @@ urlpatterns = [
     path('nodes/children/', api.NodeChildrenApi.as_view(), name='node-children-2'),
     path('nodes/<uuid:pk>/children/add/', api.NodeAddChildrenApi.as_view(), name='node-add-children'),
     path('nodes/<uuid:pk>/assets/', api.NodeAssetsApi.as_view(), name='node-assets'),
-    path('nodes/<uuid:pk>/assets/add/', non_atomic_requests(api.NodeAddAssetsApi.as_view()), name='node-add-assets'),
-    path('nodes/<uuid:pk>/assets/replace/', non_atomic_requests(api.MoveAssetsToNodeApi.as_view()), name='node-replace-assets'),
-    path('nodes/<uuid:pk>/assets/remove/', non_atomic_requests(api.NodeRemoveAssetsApi.as_view()), name='node-remove-assets'),
+    path('nodes/<uuid:pk>/assets/add/', api.NodeAddAssetsApi.as_view(), name='node-add-assets'),
+    path('nodes/<uuid:pk>/assets/replace/', api.MoveAssetsToNodeApi.as_view(), name='node-replace-assets'),
+    path('nodes/<uuid:pk>/assets/remove/', api.NodeRemoveAssetsApi.as_view(), name='node-remove-assets'),
     path('nodes/<uuid:pk>/tasks/', api.NodeTaskCreateApi.as_view(), name='node-task-create'),
 
     path('gateways/<uuid:pk>/test-connective/', api.GatewayTestConnectionApi.as_view(), name='test-gateway-connective'),
 
+    path('cmd-filters/command-confirm/', api.CommandConfirmAPI.as_view(), name='command-confirm'),
+    path('cmd-filters/command-confirm/<uuid:pk>/status/', api.CommandConfirmStatusAPI.as_view(), name='command-confirm-status')
+
 ]
 
 old_version_urlpatterns = [

apps/assets/utils.py

@@ -1,41 +1,47 @@
 # ~*~ coding: utf-8 ~*~
 #
-import time
-
-from django.db.models import Q
-
-from common.utils import get_logger, dict_get_any, is_uuid, get_object_or_none
-from common.utils.lock import DistributedLock
+from collections import defaultdict
+from common.utils import get_logger, dict_get_any, is_uuid, get_object_or_none, timeit
 from common.http import is_true
-from .models import Asset, Node
+from common.struct import Stack
+from common.db.models import output_as_string
+from orgs.utils import ensure_in_real_or_default_org, current_org
 
+from .locks import NodeTreeUpdateLock
+from .models import Node, Asset
 
 logger = get_logger(__file__)
 
 
-@DistributedLock(name="assets.node.check_node_assets_amount", blocking=False)
+@NodeTreeUpdateLock()
+@ensure_in_real_or_default_org
 def check_node_assets_amount():
-    for node in Node.objects.all():
-        logger.info(f'Check node assets amount: {node}')
-        assets_amount = Asset.objects.filter(
-            Q(nodes__key__istartswith=f'{node.key}:') | Q(nodes=node)
-        ).distinct().count()
+    logger.info(f'Check node assets amount {current_org}')
+    nodes = list(Node.objects.all().only('id', 'key', 'assets_amount'))
+    nodeid_assetid_pairs = list(Asset.nodes.through.objects.all().values_list('node_id', 'asset_id'))
 
+    nodekey_assetids_mapper = defaultdict(set)
+    nodeid_nodekey_mapper = {}
+    for node in nodes:
+        nodeid_nodekey_mapper[node.id] = node.key
+
+    for nodeid, assetid in nodeid_assetid_pairs:
+        if nodeid not in nodeid_nodekey_mapper:
+            continue
+        nodekey = nodeid_nodekey_mapper[nodeid]
+        nodekey_assetids_mapper[nodekey].add(assetid)
+
+    util = NodeAssetsUtil(nodes, nodekey_assetids_mapper)
+    util.generate()
+
+    to_updates = []
+    for node in nodes:
+        assets_amount = util.get_assets_amount(node.key)
         if node.assets_amount != assets_amount:
-            logger.warn(f'Node wrong assets amount <Node:{node.key}> '
-                        f'{node.assets_amount} right is {assets_amount}')
+            logger.error(f'Node[{node.key}] assets amount error {node.assets_amount} != {assets_amount}')
             node.assets_amount = assets_amount
-            node.save()
-        # 防止自检程序给数据库的压力太大
-        time.sleep(0.1)
-
-
-def is_asset_exists_in_node(asset_pk, node_key):
-    return Asset.objects.filter(
-        id=asset_pk
-    ).filter(
-        Q(nodes__key__istartswith=f'{node_key}:') | Q(nodes__key=node_key)
-    ).exists()
+            to_updates.append(node)
+    Node.objects.bulk_update(to_updates, fields=('assets_amount',))
 
 
 def is_query_node_all_assets(request):
@@ -57,3 +63,77 @@ def get_node(request):
     else:
         node = get_object_or_none(Node, key=node_id)
     return node
+
+
+class NodeAssetsInfo:
+    __slots__ = ('key', 'assets_amount', 'assets')
+
+    def __init__(self, key, assets_amount, assets):
+        self.key = key
+        self.assets_amount = assets_amount
+        self.assets = assets
+
+    def __str__(self):
+        return self.key
+
+
+class NodeAssetsUtil:
+    def __init__(self, nodes, nodekey_assetsid_mapper):
+        """
+        :param nodes: 节点
+        :param nodekey_assetsid_mapper:  节点直接资产id的映射 {"key1": set(), "key2": set()}
+        """
+        self.nodes = nodes
+        # node_id --> set(asset_id1, asset_id2)
+        self.nodekey_assetsid_mapper = nodekey_assetsid_mapper
+        self.nodekey_assetsinfo_mapper = {}
+
+    @timeit
+    def generate(self):
+        # 准备排序好的资产信息数据
+        infos = []
+        for node in self.nodes:
+            assets = self.nodekey_assetsid_mapper.get(node.key, set())
+            info = NodeAssetsInfo(key=node.key, assets_amount=0, assets=assets)
+            infos.append(info)
+        infos = sorted(infos, key=lambda i: [int(i) for i in i.key.split(':')])
+        # 这个守卫需要添加一下，避免最后一个无法出栈
+        guarder = NodeAssetsInfo(key='', assets_amount=0, assets=set())
+        infos.append(guarder)
+
+        stack = Stack()
+        for info in infos:
+            # 如果栈顶的不是这个节点的父祖节点，那么可以出栈了，可以计算资产数量了
+            while stack.top and not info.key.startswith(f'{stack.top.key}:'):
+                pop_info = stack.pop()
+                pop_info.assets_amount = len(pop_info.assets)
+                self.nodekey_assetsinfo_mapper[pop_info.key] = pop_info
+                if not stack.top:
+                    continue
+                stack.top.assets.update(pop_info.assets)
+            stack.push(info)
+
+    def get_assets_by_key(self, key):
+        info = self.nodekey_assetsinfo_mapper[key]
+        return info['assets']
+
+    def get_assets_amount(self, key):
+        info = self.nodekey_assetsinfo_mapper[key]
+        return info.assets_amount
+
+    @classmethod
+    def test_it(cls):
+        from assets.models import Node, Asset
+
+        nodes = list(Node.objects.all())
+        nodes_assets = Asset.nodes.through.objects.all()\
+            .annotate(aid=output_as_string('asset_id'))\
+            .values_list('node__key', 'aid')
+
+        mapping = defaultdict(set)
+        for key, asset_id in nodes_assets:
+            mapping[key].add(asset_id)
+
+        util = cls(nodes, mapping)
+        util.generate()
+        return util

apps/audits/api.py

@@ -100,6 +100,8 @@ class CommandExecutionViewSet(ListModelMixin, OrgGenericViewSet):
 
     def get_queryset(self):
         queryset = super().get_queryset()
+        if current_org.is_root():
+            return queryset
         queryset = queryset.filter(run_as__org_id=current_org.org_id())
         return queryset
 

apps/audits/migrations/0012_auto_20210414_1443.py

@@ -0,0 +1,18 @@
+# Generated by Django 3.1 on 2021-04-14 06:43
+
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('audits', '0011_userloginlog_backend'),
+    ]
+
+    operations = [
+        migrations.AlterField(
+            model_name='userloginlog',
+            name='type',
+            field=models.CharField(choices=[('W', 'Web'), ('T', 'Terminal'), ('U', 'Unknown')], max_length=2, verbose_name='Login type'),
+        ),
+    ]

apps/audits/models.py

@@ -79,6 +79,7 @@ class UserLoginLog(models.Model):
     LOGIN_TYPE_CHOICE = (
         ('W', 'Web'),
         ('T', 'Terminal'),
+        ('U', 'Unknown'),
     )
 
     MFA_DISABLED = 0

apps/audits/serializers.py

@@ -16,10 +16,14 @@ class FTPLogSerializer(serializers.ModelSerializer):
 
     class Meta:
         model = models.FTPLog
-        fields = (
-            'id', 'user', 'remote_addr', 'asset', 'system_user', 'org_id',
-            'operate', 'filename', 'is_success', 'date_start', 'operate_display'
-        )
+        fields_mini = ['id']
+        fields_small = fields_mini + [
+            'user', 'remote_addr', 'asset', 'system_user', 'org_id',
+            'operate', 'filename', 'operate_display',
+            'is_success',
+            'date_start',
+        ]
+        fields = fields_small
 
 
 class UserLoginLogSerializer(serializers.ModelSerializer):
@@ -29,11 +33,14 @@ class UserLoginLogSerializer(serializers.ModelSerializer):
 
     class Meta:
         model = models.UserLoginLog
-        fields = (
-            'id', 'username', 'type', 'type_display', 'ip', 'city', 'user_agent',
-            'mfa', 'reason', 'status', 'status_display', 'datetime', 'mfa_display',
-            'backend'
-        )
+        fields_mini = ['id']
+        fields_small = fields_mini + [
+            'username', 'type', 'type_display', 'ip', 'city', 'user_agent',
+            'mfa', 'mfa_display', 'reason', 'backend',
+            'status', 'status_display',
+            'datetime',
+        ]
+        fields = fields_small
         extra_kwargs = {
             "user_agent": {'label': _('User agent')}
         }
@@ -42,10 +49,13 @@ class UserLoginLogSerializer(serializers.ModelSerializer):
 class OperateLogSerializer(serializers.ModelSerializer):
     class Meta:
         model = models.OperateLog
-        fields = (
-            'id', 'user', 'action', 'resource_type', 'resource',
-            'remote_addr', 'datetime', 'org_id'
-        )
+        fields_mini = ['id']
+        fields_small = fields_mini + [
+            'user', 'action', 'resource_type', 'resource', 'remote_addr',
+            'datetime',
+            'org_id'
+        ]
+        fields = fields_small
 
 
 class PasswordChangeLogSerializer(serializers.ModelSerializer):
@@ -64,6 +74,9 @@ class SessionAuditSerializer(serializers.ModelSerializer):
 
 class CommandExecutionSerializer(serializers.ModelSerializer):
     is_success = serializers.BooleanField(read_only=True, label=_('Is success'))
+    hosts_display = serializers.ListSerializer(
+        child=serializers.CharField(), source='hosts', read_only=True, label=_('Hosts for display')
+    )
 
     class Meta:
         model = CommandExecution
@@ -72,7 +85,7 @@ class CommandExecutionSerializer(serializers.ModelSerializer):
             'run_as', 'command', 'user', 'is_finished',
             'date_start', 'result', 'is_success', 'org_id'
         ]
-        fields = fields_small + ['hosts', 'run_as_display', 'user_display']
+        fields = fields_small + ['hosts', 'hosts_display', 'run_as_display', 'user_display']
         extra_kwargs = {
             'result': {'label': _('Result')},  # model 上的方法，只能在这修改
             'is_success': {'label': _('Is success')},

apps/audits/signals_handler.py

@@ -1,10 +1,11 @@
 # -*- coding: utf-8 -*-
 #
-
 from django.db.models.signals import post_save, post_delete
 from django.dispatch import receiver
+from django.conf import settings
 from django.db import transaction
 from django.utils import timezone
+from django.utils.functional import LazyObject
 from django.contrib.auth import BACKEND_SESSION_KEY
 from django.utils.translation import ugettext_lazy as _
 from rest_framework.renderers import JSONRenderer
@@ -26,25 +27,45 @@ json_render = JSONRenderer()
 
 
 MODELS_NEED_RECORD = (
-    'User', 'UserGroup', 'Asset', 'Node', 'AdminUser', 'SystemUser',
-    'Domain', 'Gateway', 'Organization', 'AssetPermission', 'CommandFilter',
-    'CommandFilterRule', 'License', 'Setting', 'Account', 'SyncInstanceTask',
-    'Platform', 'ChangeAuthPlan', 'GatherUserTask',
-    'RemoteApp', 'RemoteAppPermission', 'DatabaseApp', 'DatabaseAppPermission',
+    # users
+    'User', 'UserGroup',
+    # acls
+    'LoginACL', 'LoginAssetACL',
+    # assets
+    'Asset', 'Node', 'AdminUser', 'SystemUser', 'Domain', 'Gateway', 'CommandFilterRule',
+    'CommandFilter', 'Platform',
+    # applications
+    'Application',
+    # orgs
+    'Organization',
+    # settings
+    'Setting',
+    # perms
+    'AssetPermission', 'ApplicationPermission',
+    # xpack
+    'License', 'Account', 'SyncInstanceTask', 'ChangeAuthPlan', 'GatherUserTask',
 )
 
 
-LOGIN_BACKEND = {
-    'PublicKeyAuthBackend': _('SSH Key'),
-    'RadiusBackend': User.Source.radius.label,
-    'RadiusRealmBackend': User.Source.radius.label,
-    'LDAPAuthorizationBackend': User.Source.ldap.label,
-    'ModelBackend': _('Password'),
-    'SSOAuthentication': _('SSO'),
-    'CASBackend': User.Source.cas.label,
-    'OIDCAuthCodeBackend': User.Source.openid.label,
-    'OIDCAuthPasswordBackend': User.Source.openid.label,
-}
+class AuthBackendLabelMapping(LazyObject):
+    @staticmethod
+    def get_login_backends():
+        backend_label_mapping = {}
+        for source, backends in User.SOURCE_BACKEND_MAPPING.items():
+            for backend in backends:
+                backend_label_mapping[backend] = source.label
+        backend_label_mapping[settings.AUTH_BACKEND_PUBKEY] = _('SSH Key')
+        backend_label_mapping[settings.AUTH_BACKEND_MODEL] = _('Password')
+        backend_label_mapping[settings.AUTH_BACKEND_SSO] = _('SSO')
+        backend_label_mapping[settings.AUTH_BACKEND_WECOM] = _('WeCom')
+        backend_label_mapping[settings.AUTH_BACKEND_DINGTALK] = _('DingTalk')
+        return backend_label_mapping
+
+    def _setup(self):
+        self._wrapped = self.get_login_backends()
+
+
+AUTH_BACKEND_LABEL_MAPPING = AuthBackendLabelMapping()
 
 
 def create_operate_log(action, sender, resource):
@@ -70,6 +91,7 @@ def create_operate_log(action, sender, resource):
 
 @receiver(post_save)
 def on_object_created_or_update(sender, instance=None, created=False, update_fields=None, **kwargs):
+    # last_login 改变是最后登录日期, 每次登录都会改变
     if instance._meta.object_name == 'User' and \
             update_fields and 'last_login' in update_fields:
         return
@@ -125,21 +147,20 @@ def on_audits_log_create(sender, instance=None, **kwargs):
 
 
 def get_login_backend(request):
-    backend = request.session.get('auth_backend', '') or request.session.get(BACKEND_SESSION_KEY, '')
+    backend = request.session.get('auth_backend', '') or \
+              request.session.get(BACKEND_SESSION_KEY, '')
 
-    backend = backend.rsplit('.', maxsplit=1)[-1]
-    if backend in LOGIN_BACKEND:
-        return LOGIN_BACKEND[backend]
-    else:
-        logger.warn(f'LOGIN_BACKEND_NOT_FOUND: {backend}')
-        return ''
+    backend_label = AUTH_BACKEND_LABEL_MAPPING.get(backend, None)
+    if backend_label is None:
+        backend_label = ''
+    return backend_label
 
 
 def generate_data(username, request):
     user_agent = request.META.get('HTTP_USER_AGENT', '')
     login_ip = get_request_ip(request) or '0.0.0.0'
     if isinstance(request, Request):
-        login_type = request.META.get('HTTP_X_JMS_LOGIN_TYPE', '')
+        login_type = request.META.get('HTTP_X_JMS_LOGIN_TYPE', 'U')
     else:
         login_type = 'W'
 
@@ -147,7 +168,7 @@ def generate_data(username, request):
         'username': username,
         'ip': login_ip,
         'type': login_type,
-        'user_agent': user_agent,
+        'user_agent': user_agent[0:254],
         'datetime': timezone.now(),
         'backend': get_login_backend(request)
     }

apps/authentication/api/__init__.py

@@ -1,9 +1,12 @@
 # -*- coding: utf-8 -*-
 #
 
-from .auth import *
+from .connection_token import *
 from .token import *
 from .mfa import *
 from .access_key import *
 from .login_confirm import *
 from .sso import *
+from .wecom import *
+from .dingtalk import *
+from .password import *

apps/authentication/api/auth.py

@@ -1,55 +0,0 @@
-# -*- coding: utf-8 -*-
-#
-import uuid
-
-from django.core.cache import cache
-from django.shortcuts import get_object_or_404
-from rest_framework.response import Response
-from rest_framework.views import APIView
-
-from common.utils import get_logger
-from common.permissions import IsOrgAdminOrAppUser
-from orgs.mixins.api import RootOrgViewMixin
-from users.models import User
-from assets.models import Asset, SystemUser
-
-logger = get_logger(__name__)
-__all__ = [
-    'UserConnectionTokenApi',
-]
-
-
-class UserConnectionTokenApi(RootOrgViewMixin, APIView):
-    permission_classes = (IsOrgAdminOrAppUser,)
-
-    def post(self, request):
-        user_id = request.data.get('user', '')
-        asset_id = request.data.get('asset', '')
-        system_user_id = request.data.get('system_user', '')
-        token = str(uuid.uuid4())
-        user = get_object_or_404(User, id=user_id)
-        asset = get_object_or_404(Asset, id=asset_id)
-        system_user = get_object_or_404(SystemUser, id=system_user_id)
-        value = {
-            'user': user_id,
-            'username': user.username,
-            'asset': asset_id,
-            'hostname': asset.hostname,
-            'system_user': system_user_id,
-            'system_user_name': system_user.name
-        }
-        cache.set(token, value, timeout=20)
-        return Response({"token": token}, status=201)
-
-    def get(self, request):
-        token = request.query_params.get('token')
-        user_only = request.query_params.get('user-only', None)
-        value = cache.get(token, None)
-
-        if not value:
-            return Response('', status=404)
-
-        if not user_only:
-            return Response(value)
-        else:
-            return Response({'user': value['user']})

apps/authentication/api/connection_token.py

@@ -0,0 +1,245 @@
+# -*- coding: utf-8 -*-
+#
+import urllib.parse
+
+from django.conf import settings
+from django.core.cache import cache
+from django.shortcuts import get_object_or_404
+from django.http import HttpResponse
+from rest_framework.response import Response
+from rest_framework.viewsets import GenericViewSet
+from rest_framework.decorators import action
+from rest_framework.exceptions import PermissionDenied
+
+from common.utils import get_logger, random_string
+from common.drf.api import SerializerMixin2
+from common.permissions import IsSuperUserOrAppUser, IsValidUser, IsSuperUser
+
+from orgs.mixins.api import RootOrgViewMixin
+
+from ..serializers import (
+    ConnectionTokenSerializer, ConnectionTokenSecretSerializer,
+    RDPFileSerializer
+)
+
+logger = get_logger(__name__)
+__all__ = ['UserConnectionTokenViewSet']
+
+
+class UserConnectionTokenViewSet(RootOrgViewMixin, SerializerMixin2, GenericViewSet):
+    permission_classes = (IsSuperUserOrAppUser,)
+    serializer_classes = {
+        'default': ConnectionTokenSerializer,
+        'get_secret_detail': ConnectionTokenSecretSerializer,
+        'get_rdp_file': RDPFileSerializer
+    }
+    CACHE_KEY_PREFIX = 'CONNECTION_TOKEN_{}'
+
+    @staticmethod
+    def check_resource_permission(user, asset, application, system_user):
+        from perms.utils.asset import has_asset_system_permission
+        from perms.utils.application import has_application_system_permission
+        if asset and not has_asset_system_permission(user, asset, system_user):
+            error = f'User not has this asset and system user permission: ' \
+                    f'user={user.id} system_user={system_user.id} asset={asset.id}'
+            raise PermissionDenied(error)
+        if application and not has_application_system_permission(user, application, system_user):
+            error = f'User not has this application and system user permission: ' \
+                    f'user={user.id} system_user={system_user.id} application={application.id}'
+            raise PermissionDenied(error)
+        return True
+
+    def create_token(self, user, asset, application, system_user):
+        if not settings.CONNECTION_TOKEN_ENABLED:
+            raise PermissionDenied('Connection token disabled')
+        if not user:
+            user = self.request.user
+        if not self.request.user.is_superuser and user != self.request.user:
+            raise PermissionDenied('Only super user can create user token')
+        self.check_resource_permission(user, asset, application, system_user)
+        token = random_string(36)
+        value = {
+            'user': str(user.id),
+            'username': user.username,
+            'system_user': str(system_user.id),
+            'system_user_name': system_user.name
+        }
+
+        if asset:
+            value.update({
+                'type': 'asset',
+                'asset': str(asset.id),
+                'hostname': asset.hostname,
+            })
+        elif application:
+            value.update({
+                'type': 'application',
+                'application': application.id,
+                'application_name': str(application)
+            })
+
+        key = self.CACHE_KEY_PREFIX.format(token)
+        cache.set(key, value, timeout=30*60)
+        return token
+
+    def create(self, request, *args, **kwargs):
+        serializer = self.get_serializer(data=request.data)
+        serializer.is_valid(raise_exception=True)
+
+        asset = serializer.validated_data.get('asset')
+        application = serializer.validated_data.get('application')
+        system_user = serializer.validated_data['system_user']
+        user = serializer.validated_data.get('user')
+        token = self.create_token(user, asset, application, system_user)
+        return Response({"token": token}, status=201)
+
+    @action(methods=['POST', 'GET'], detail=False, url_path='rdp/file')
+    def get_rdp_file(self, request, *args, **kwargs):
+        options = {
+            'full address:s': '',
+            'username:s': '',
+            'screen mode id:i': '0',
+            'desktopwidth:i': '1280',
+            'desktopheight:i': '800',
+            'use multimon:i': '1',
+            'session bpp:i': '32',
+            'audiomode:i': '0',
+            'disable wallpaper:i': '0',
+            'disable full window drag:i': '0',
+            'disable menu anims:i': '0',
+            'disable themes:i': '0',
+            'alternate shell:s': '',
+            'shell working directory:s': '',
+            'authentication level:i': '2',
+            'connect to console:i': '0',
+            'disable cursor setting:i': '0',
+            'allow font smoothing:i': '1',
+            'allow desktop composition:i': '1',
+            'redirectprinters:i': '0',
+            'prompt for credentials on client:i': '0',
+            'autoreconnection enabled:i': '1',
+            'bookmarktype:i': '3',
+            'use redirection server name:i': '0',
+            # 'alternate shell:s:': '||MySQLWorkbench',
+            # 'remoteapplicationname:s': 'Firefox',
+            # 'remoteapplicationcmdline:s': '',
+        }
+        if self.request.method == 'GET':
+            data = self.request.query_params
+        else:
+            data = request.data
+        serializer = self.get_serializer(data=data)
+        serializer.is_valid(raise_exception=True)
+
+        asset = serializer.validated_data.get('asset')
+        application = serializer.validated_data.get('application')
+        system_user = serializer.validated_data['system_user']
+        user = serializer.validated_data.get('user')
+        height = serializer.validated_data.get('height')
+        width = serializer.validated_data.get('width')
+        token = self.create_token(user, asset, application, system_user)
+
+        # Todo: 上线后地址是 JumpServerAddr:3389
+        address = self.request.query_params.get('address') or '1.1.1.1'
+        options['full address:s'] = address
+        options['username:s'] = '{}|{}'.format(user.username, token)
+        options['desktopwidth:i'] = width
+        options['desktopheight:i'] = height
+        data = ''
+        for k, v in options.items():
+            data += f'{k}:{v}\n'
+        response = HttpResponse(data, content_type='application/octet-stream')
+        filename = "{}-{}-jumpserver.rdp".format(user.username, asset.hostname)
+        filename = urllib.parse.quote(filename)
+        response['Content-Disposition'] = 'attachment; filename*=UTF-8\'\'%s' % filename
+        return response
+
+    @staticmethod
+    def _get_application_secret_detail(value):
+        from applications.models import Application
+        from perms.models import Action
+        application = get_object_or_404(Application, id=value.get('application'))
+        gateway = None
+
+        if not application.category_remote_app:
+            actions = Action.NONE
+            remote_app = {}
+            asset = None
+            domain = application.domain
+        else:
+            remote_app = application.get_rdp_remote_app_setting()
+            actions = Action.CONNECT
+            asset = application.get_remote_app_asset()
+            domain = asset.domain
+
+        if domain and domain.has_gateway():
+            gateway = domain.random_gateway()
+
+        return {
+            'asset': asset,
+            'application': application,
+            'gateway': gateway,
+            'remote_app': remote_app,
+            'actions': actions
+        }
+
+    @staticmethod
+    def _get_asset_secret_detail(value, user, system_user):
+        from assets.models import Asset
+        from perms.utils.asset import get_asset_system_user_ids_with_actions_by_user
+        asset = get_object_or_404(Asset, id=value.get('asset'))
+        systemuserid_actions_mapper = get_asset_system_user_ids_with_actions_by_user(user, asset)
+        actions = systemuserid_actions_mapper.get(system_user.id, [])
+        gateway = None
+        if asset and asset.domain and asset.domain.has_gateway():
+            gateway = asset.domain.random_gateway()
+        return {
+            'asset': asset,
+            'application': None,
+            'gateway': gateway,
+            'remote_app': None,
+            'actions': actions,
+        }
+
+    @action(methods=['POST'], detail=False, permission_classes=[IsSuperUserOrAppUser], url_path='secret-info/detail')
+    def get_secret_detail(self, request, *args, **kwargs):
+        from users.models import User
+        from assets.models import SystemUser
+
+        token = request.data.get('token', '')
+        key = self.CACHE_KEY_PREFIX.format(token)
+        value = cache.get(key, None)
+        if not value:
+            return Response(status=404)
+        user = get_object_or_404(User, id=value.get('user'))
+        system_user = get_object_or_404(SystemUser, id=value.get('system_user'))
+        data = dict(user=user, system_user=system_user)
+
+        if value.get('type') == 'asset':
+            asset_detail = self._get_asset_secret_detail(value, user=user, system_user=system_user)
+            data['type'] = 'asset'
+            data.update(asset_detail)
+        else:
+            app_detail = self._get_application_secret_detail(value)
+            data['type'] = 'application'
+            data.update(app_detail)
+
+        serializer = self.get_serializer(data)
+        return Response(data=serializer.data, status=200)
+
+    def get_permissions(self):
+        if self.action in ["create", "get_rdp_file"]:
+            if self.request.data.get('user', None):
+                self.permission_classes = (IsSuperUser,)
+            else:
+                self.permission_classes = (IsValidUser,)
+        return super().get_permissions()
+
+    def get(self, request):
+        token = request.query_params.get('token')
+        key = self.CACHE_KEY_PREFIX.format(token)
+        value = cache.get(key, None)
+
+        if not value:
+            return Response('', status=404)
+        return Response(value)

apps/authentication/api/dingtalk.py

@@ -0,0 +1,35 @@
+from rest_framework.views import APIView
+from rest_framework.request import Request
+from rest_framework.response import Response
+
+from users.permissions import IsAuthPasswdTimeValid
+from users.models import User
+from common.utils import get_logger
+from common.permissions import IsOrgAdmin
+from common.mixins.api import RoleUserMixin, RoleAdminMixin
+from authentication import errors
+
+logger = get_logger(__file__)
+
+
+class DingTalkQRUnBindBase(APIView):
+    user: User
+
+    def post(self, request: Request, **kwargs):
+        user = self.user
+
+        if not user.dingtalk_id:
+            raise errors.DingTalkNotBound
+
+        user.dingtalk_id = None
+        user.save()
+        return Response()
+
+
+class DingTalkQRUnBindForUserApi(RoleUserMixin, DingTalkQRUnBindBase):
+    permission_classes = (IsAuthPasswdTimeValid,)
+
+
+class DingTalkQRUnBindForAdminApi(RoleAdminMixin, DingTalkQRUnBindBase):
+    user_id_url_kwarg = 'user_id'
+    permission_classes = (IsOrgAdmin,)

apps/authentication/api/login_confirm.py

@@ -3,10 +3,10 @@
 from rest_framework.generics import UpdateAPIView
 from rest_framework.response import Response
 from rest_framework.views import APIView
+from rest_framework.permissions import AllowAny
 from django.shortcuts import get_object_or_404
-from django.utils.translation import ugettext as _
 
-from common.utils import get_logger, get_object_or_none
+from common.utils import get_logger
 from common.permissions import IsOrgAdmin
 from ..models import LoginConfirmSetting
 from ..serializers import LoginConfirmSettingSerializer
@@ -32,7 +32,7 @@ class LoginConfirmSettingUpdateApi(UpdateAPIView):
 
 
 class TicketStatusApi(mixins.AuthMixin, APIView):
-    permission_classes = ()
+    permission_classes = (AllowAny,)
 
     def get(self, request, *args, **kwargs):
         try:

apps/authentication/api/mfa.py

@@ -29,7 +29,7 @@ class MFAChallengeApi(AuthMixin, CreateAPIView):
             if not valid:
                 self.request.session['auth_mfa'] = ''
                 raise errors.MFAFailedError(
-                    username=user.username, request=self.request
+                    username=user.username, request=self.request, ip=self.get_request_ip()
                 )
             else:
                 self.request.session['auth_mfa'] = '1'

apps/authentication/api/password.py

@@ -0,0 +1,26 @@
+from rest_framework.generics import CreateAPIView
+from rest_framework.response import Response
+
+from authentication.serializers import PasswordVerifySerializer
+from common.permissions import IsValidUser
+from authentication.mixins import authenticate
+from authentication.errors import PasswdInvalid
+from authentication.mixins import AuthMixin
+
+
+class UserPasswordVerifyApi(AuthMixin, CreateAPIView):
+    permission_classes = (IsValidUser,)
+    serializer_class = PasswordVerifySerializer
+
+    def create(self, request, *args, **kwargs):
+        serializer = self.get_serializer(data=request.data)
+        serializer.is_valid(raise_exception=True)
+        password = serializer.validated_data['password']
+        user = self.request.user
+
+        user = authenticate(request=request, username=user.username, password=password)
+        if not user:
+            raise PasswdInvalid
+
+        self.set_passwd_verify_on_session(user)
+        return Response()

apps/authentication/api/sso.py

@@ -7,6 +7,7 @@ from django.http.response import HttpResponseRedirect
 from rest_framework.decorators import action
 from rest_framework.response import Response
 from rest_framework.request import Request
+from rest_framework.permissions import AllowAny
 
 from common.utils.timezone import utcnow
 from common.const.http import POST, GET
@@ -31,6 +32,7 @@ class SSOViewSet(AuthMixin, JmsGenericViewSet):
         'login_url': SSOTokenSerializer,
         'login': EmptySerializer
     }
+    permission_classes = (IsSuperUser,)
 
     @action(methods=[POST], detail=False, permission_classes=[IsSuperUser], url_path='login-url')
     def login_url(self, request, *args, **kwargs):
@@ -54,7 +56,7 @@ class SSOViewSet(AuthMixin, JmsGenericViewSet):
         login_url = '%s?%s' % (reverse('api-auth:sso-login', external=True), urlencode(query))
         return Response(data={'login_url': login_url})
 
-    @action(methods=[GET], detail=False, filter_backends=[AuthKeyQueryDeclaration], permission_classes=[])
+    @action(methods=[GET], detail=False, filter_backends=[AuthKeyQueryDeclaration], permission_classes=[AllowAny])
     def login(self, request: Request, *args, **kwargs):
         """
         此接口违反了 `Restful` 的规范

apps/authentication/api/wecom.py

@@ -0,0 +1,35 @@
+from rest_framework.views import APIView
+from rest_framework.request import Request
+from rest_framework.response import Response
+
+from users.permissions import IsAuthPasswdTimeValid
+from users.models import User
+from common.utils import get_logger
+from common.permissions import IsOrgAdmin
+from common.mixins.api import RoleUserMixin, RoleAdminMixin
+from authentication import errors
+
+logger = get_logger(__file__)
+
+
+class WeComQRUnBindBase(APIView):
+    user: User
+
+    def post(self, request: Request, **kwargs):
+        user = self.user
+
+        if not user.wecom_id:
+            raise errors.WeComNotBound
+
+        user.wecom_id = None
+        user.save()
+        return Response()
+
+
+class WeComQRUnBindForUserApi(RoleUserMixin, WeComQRUnBindBase):
+    permission_classes = (IsAuthPasswdTimeValid,)
+
+
+class WeComQRUnBindForAdminApi(RoleAdminMixin, WeComQRUnBindBase):
+    user_id_url_kwarg = 'user_id'
+    permission_classes = (IsOrgAdmin,)

apps/authentication/backends/api.py

@@ -8,7 +8,7 @@ from django.core.cache import cache
 from django.utils.translation import ugettext as _
 from six import text_type
 from django.contrib.auth import get_user_model
-from django.contrib.auth.backends import ModelBackend
+from django.contrib.auth.backends import ModelBackend as DJModelBackend
 from rest_framework import HTTP_HEADER_ENCODING
 from rest_framework import authentication, exceptions
 from common.auth import signature
@@ -25,6 +25,11 @@ def get_request_date_header(request):
     return date
 
 
+class ModelBackend(DJModelBackend):
+    def user_can_authenticate(self, user):
+        return user.is_valid
+
+
 class AccessKeyAuthentication(authentication.BaseAuthentication):
     """App使用Access key进行签名认证, 目前签名算法比较简单,
     app注册或者手动建立后,会生成 access_key_id 和 access_key_secret,
@@ -205,3 +210,21 @@ class SSOAuthentication(ModelBackend):
 
     def authenticate(self, request, sso_token=None, **kwargs):
         pass
+
+
+class WeComAuthentication(ModelBackend):
+    """
+    什么也不做呀😺
+    """
+
+    def authenticate(self, request, **kwargs):
+        pass
+
+
+class DingTalkAuthentication(ModelBackend):
+    """
+    什么也不做呀😺
+    """
+
+    def authenticate(self, request, **kwargs):
+        pass

apps/authentication/backends/ldap.py

@@ -27,6 +27,39 @@ class LDAPAuthorizationBackend(LDAPBackend):
         is_valid = getattr(user, 'is_valid', None)
         return is_valid or is_valid is None
 
+    def get_or_build_user(self, username, ldap_user):
+        """
+                This must return a (User, built) 2-tuple for the given LDAP user.
+
+                username is the Django-friendly username of the user. ldap_user.dn is
+                the user's DN and ldap_user.attrs contains all of their LDAP
+                attributes.
+
+                The returned User object may be an unsaved model instance.
+
+                """
+        model = self.get_user_model()
+
+        if self.settings.USER_QUERY_FIELD:
+            query_field = self.settings.USER_QUERY_FIELD
+            query_value = ldap_user.attrs[self.settings.USER_ATTR_MAP[query_field]][0]
+            query_value = query_value.strip()
+            lookup = query_field
+        else:
+            query_field = model.USERNAME_FIELD
+            query_value = username.lower()
+            lookup = "{}__iexact".format(query_field)
+
+        try:
+            user = model.objects.get(**{lookup: query_value})
+        except model.DoesNotExist:
+            user = model(**{query_field: query_value})
+            built = True
+        else:
+            built = False
+
+        return (user, built)
+
     def pre_check(self, username, password):
         if not settings.AUTH_LDAP:
             error = 'Not enabled auth ldap'
@@ -128,6 +161,7 @@ class LDAPUser(_LDAPUser):
         for field, attr in self.settings.USER_ATTR_MAP.items():
             try:
                 value = self.attrs[attr][0]
+                value = value.strip()
                 if attr.lower() == 'useraccountcontrol' \
                         and field == 'is_active' and value:
                     value = int(value) & LDAP_AD_ACCOUNT_DISABLE \

apps/authentication/backends/radius.py

@@ -2,7 +2,7 @@
 #
 import traceback
 
-from django.contrib.auth import get_user_model, authenticate
+from django.contrib.auth import get_user_model
 from radiusauth.backends import RADIUSBackend, RADIUSRealmBackend
 from django.conf import settings
 

apps/authentication/errors.py

@@ -6,9 +6,7 @@ from django.conf import settings
 
 from common.exceptions import JMSException
 from .signals import post_auth_failed
-from users.utils import (
-    increase_login_failed_count, get_login_failed_count
-)
+from users.utils import LoginBlockUtil, MFABlockUtils
 
 reason_password_failed = 'password_failed'
 reason_password_decrypt_failed = 'password_decrypt_failed'
@@ -18,6 +16,10 @@ reason_user_not_exist = 'user_not_exist'
 reason_password_expired = 'password_expired'
 reason_user_invalid = 'user_invalid'
 reason_user_inactive = 'user_inactive'
+reason_user_expired = 'user_expired'
+reason_backend_not_match = 'backend_not_match'
+reason_acl_not_allow = 'acl_not_allow'
+only_local_users_are_allowed = 'only_local_users_are_allowed'
 
 reason_choices = {
     reason_password_failed: _('Username/password check failed'),
@@ -27,7 +29,11 @@ reason_choices = {
     reason_user_not_exist: _("Username does not exist"),
     reason_password_expired: _("Password expired"),
     reason_user_invalid: _('Disabled or expired'),
-    reason_user_inactive: _("This account is inactive.")
+    reason_user_inactive: _("This account is inactive."),
+    reason_user_expired: _("This account is expired"),
+    reason_backend_not_match: _("Auth backend not match"),
+    reason_acl_not_allow: _("ACL is not allowed"),
+    only_local_users_are_allowed: _("Only local users are allowed")
 }
 old_reason_choices = {
     '0': '-',
@@ -48,7 +54,15 @@ block_login_msg = _(
     "The account has been locked "
     "(please contact admin to unlock it or try again after {} minutes)"
 )
-mfa_failed_msg = _("MFA code invalid, or ntp sync server time")
+block_mfa_msg = _(
+    "The account has been locked "
+    "(please contact admin to unlock it or try again after {} minutes)"
+)
+mfa_failed_msg = _(
+    "MFA code invalid, or ntp sync server time, "
+    "You can also try {times_try} times "
+    "(The account will be temporarily locked for {block_time} minutes)"
+)
 
 mfa_required_msg = _("MFA required")
 mfa_unset_msg = _("MFA not set, please set it first")
@@ -76,7 +90,7 @@ class AuthFailedNeedBlockMixin:
 
     def __init__(self, *args, **kwargs):
         super().__init__(*args, **kwargs)
-        increase_login_failed_count(self.username, self.ip)
+        LoginBlockUtil(self.username, self.ip).incr_failed_count()
 
 
 class AuthFailedError(Exception):
@@ -103,13 +117,12 @@ class AuthFailedError(Exception):
 class CredentialError(AuthFailedNeedLogMixin, AuthFailedNeedBlockMixin, AuthFailedError):
     def __init__(self, error, username, ip, request):
         super().__init__(error=error, username=username, ip=ip, request=request)
-        times_up = settings.SECURITY_LOGIN_LIMIT_COUNT
-        times_failed = get_login_failed_count(username, ip)
-        times_try = int(times_up) - int(times_failed)
+        util = LoginBlockUtil(username, ip)
+        times_remainder = util.get_remainder_times()
         block_time = settings.SECURITY_LOGIN_LIMIT_TIME
 
         default_msg = invalid_login_msg.format(
-            times_try=times_try, block_time=block_time
+            times_try=times_remainder, block_time=block_time
         )
         if error == reason_password_failed:
             self.msg = default_msg
@@ -119,12 +132,32 @@ class CredentialError(AuthFailedNeedLogMixin, AuthFailedNeedBlockMixin, AuthFail
 
 class MFAFailedError(AuthFailedNeedLogMixin, AuthFailedError):
     error = reason_mfa_failed
-    msg = mfa_failed_msg
+    msg: str
 
-    def __init__(self, username, request):
+    def __init__(self, username, request, ip):
+        util = MFABlockUtils(username, ip)
+        util.incr_failed_count()
+
+        times_remainder = util.get_remainder_times()
+        block_time = settings.SECURITY_LOGIN_LIMIT_TIME
+
+        if times_remainder:
+            self.msg = mfa_failed_msg.format(
+                times_try=times_remainder, block_time=block_time
+            )
+        else:
+            self.msg = block_mfa_msg.format(settings.SECURITY_LOGIN_LIMIT_TIME)
         super().__init__(username=username, request=request)
 
 
+class BlockMFAError(AuthFailedNeedLogMixin, AuthFailedError):
+    error = 'block_mfa'
+
+    def __init__(self, username, request, ip):
+        self.msg = block_mfa_msg.format(settings.SECURITY_LOGIN_LIMIT_TIME)
+        super().__init__(username=username, request=request, ip=ip)
+
+
 class MFAUnsetError(AuthFailedNeedLogMixin, AuthFailedError):
     error = reason_mfa_unset
     msg = mfa_unset_msg
@@ -180,6 +213,28 @@ class MFARequiredError(NeedMoreInfoError):
         }
 
 
+class ACLError(AuthFailedNeedLogMixin, AuthFailedError):
+    msg = reason_acl_not_allow
+    error = 'acl_error'
+
+    def __init__(self, msg, **kwargs):
+        self.msg = msg
+        super().__init__(**kwargs)
+
+    def as_data(self):
+        return {
+            "error": reason_acl_not_allow,
+            "msg": self.msg
+        }
+
+
+class LoginIPNotAllowed(ACLError):
+    def __init__(self, username, request, **kwargs):
+        self.username = username
+        self.request = request
+        super().__init__(_("IP is not allowed"), **kwargs)
+
+
 class LoginConfirmBaseError(NeedMoreInfoError):
     def __init__(self, ticket_id, **kwargs):
         self.ticket_id = ticket_id
@@ -222,6 +277,15 @@ class PasswdTooSimple(JMSException):
         self.url = url
 
 
+class PasswdNeedUpdate(JMSException):
+    default_code = 'passwd_need_update'
+    default_detail = _('You should to change your password before login')
+
+    def __init__(self, url, *args, **kwargs):
+        super().__init__(*args, **kwargs)
+        self.url = url
+
+
 class PasswordRequireResetError(JMSException):
     default_code = 'passwd_has_expired'
     default_detail = _('Your password has expired, please reset before logging in')
@@ -229,3 +293,28 @@ class PasswordRequireResetError(JMSException):
     def __init__(self, url, *args, **kwargs):
         super().__init__(*args, **kwargs)
         self.url = url
+
+
+class WeComCodeInvalid(JMSException):
+    default_code = 'wecom_code_invalid'
+    default_detail = 'Code invalid, can not get user info'
+
+
+class WeComBindAlready(JMSException):
+    default_code = 'wecom_bind_already'
+    default_detail = 'WeCom already binded'
+
+
+class WeComNotBound(JMSException):
+    default_code = 'wecom_not_bound'
+    default_detail = 'WeCom is not bound'
+
+
+class DingTalkNotBound(JMSException):
+    default_code = 'dingtalk_not_bound'
+    default_detail = 'DingTalk is not bound'
+
+
+class PasswdInvalid(JMSException):
+    default_code = 'passwd_invalid'
+    default_detail = _('Your password is invalid')

apps/authentication/forms.py

@@ -8,11 +8,26 @@ from captcha.fields import CaptchaField, CaptchaTextInput
 
 
 class UserLoginForm(forms.Form):
-    username = forms.CharField(label=_('Username'), max_length=100)
+    days_auto_login = int(settings.SESSION_COOKIE_AGE / 3600 / 24)
+    disable_days_auto_login = settings.SESSION_EXPIRE_AT_BROWSER_CLOSE_FORCE or days_auto_login < 1
+
+    username = forms.CharField(
+        label=_('Username'), max_length=100,
+        widget=forms.TextInput(attrs={
+            'placeholder': _("Username"),
+            'autofocus': 'autofocus'
+        })
+    )
     password = forms.CharField(
         label=_('Password'), widget=forms.PasswordInput,
         max_length=1024, strip=False
     )
+    auto_login = forms.BooleanField(
+        label=_("{} days auto login").format(days_auto_login or 1),
+        required=False, initial=False, widget=forms.CheckboxInput(
+            attrs={'disabled': disable_days_auto_login}
+        )
+    )
 
     def confirm_login_allowed(self, user):
         if not user.is_staff:
@@ -35,8 +50,13 @@ class CaptchaMixin(forms.Form):
 
 
 class ChallengeMixin(forms.Form):
-    challenge = forms.CharField(label=_('MFA code'), max_length=6,
-                                required=False)
+    challenge = forms.CharField(
+        label=_('MFA code'), max_length=6, required=False,
+        widget=forms.TextInput(attrs={
+            'placeholder': _("MFA code"),
+            'style': 'width: 50%'
+        })
+    )
 
 
 def get_user_login_form_cls(*, captcha=False):

apps/authentication/mixins.py

@@ -1,19 +1,23 @@
 # -*- coding: utf-8 -*-
 #
+import inspect
 from urllib.parse import urlencode
 from functools import partial
 import time
 
+from django.core.cache import cache
 from django.conf import settings
-from django.contrib.auth import authenticate
-from django.shortcuts import reverse
-from django.contrib.auth import BACKEND_SESSION_KEY
-
-from common.utils import get_object_or_none, get_request_ip, get_logger
-from users.models import User
-from users.utils import (
-    is_block_login, clean_failed_count
+from django.contrib import auth
+from django.utils.translation import ugettext as _
+from django.contrib.auth import (
+    BACKEND_SESSION_KEY, _get_backends,
+    PermissionDenied, user_login_failed, _clean_credentials
 )
+from django.shortcuts import reverse, redirect
+
+from common.utils import get_object_or_none, get_request_ip, get_logger, bulk_get, FlashMessageUtil
+from users.models import User
+from users.utils import LoginBlockUtil, MFABlockUtils
 from . import errors
 from .utils import rsa_decrypt
 from .signals import post_auth_success, post_auth_failed
@@ -22,8 +26,64 @@ from .const import RSA_PRIVATE_KEY
 logger = get_logger(__name__)
 
 
+def check_backend_can_auth(username, backend_path, allowed_auth_backends):
+    if allowed_auth_backends is not None and backend_path not in allowed_auth_backends:
+        logger.debug('Skip user auth backend: {}, {} not in'.format(
+                username, backend_path, ','.join(allowed_auth_backends)
+            )
+        )
+        return False
+    return True
+
+
+def authenticate(request=None, **credentials):
+    """
+    If the given credentials are valid, return a User object.
+    """
+    username = credentials.get('username')
+    allowed_auth_backends = User.get_user_allowed_auth_backends(username)
+
+    for backend, backend_path in _get_backends(return_tuples=True):
+        # 预先检查，不浪费认证时间
+        if not check_backend_can_auth(username, backend_path, allowed_auth_backends):
+            continue
+
+        backend_signature = inspect.signature(backend.authenticate)
+        try:
+            backend_signature.bind(request, **credentials)
+        except TypeError:
+            # This backend doesn't accept these credentials as arguments. Try the next one.
+            continue
+        try:
+            user = backend.authenticate(request, **credentials)
+        except PermissionDenied:
+            # This backend says to stop in our tracks - this user should not be allowed in at all.
+            break
+        if user is None:
+            continue
+        # 如果是 None, 证明没有检查过, 需要再次检查
+        if allowed_auth_backends is None:
+            # 有些 authentication 参数中不带 username, 之后还要再检查
+            allowed_auth_backends = user.get_allowed_auth_backends()
+            if not check_backend_can_auth(user.username, backend_path, allowed_auth_backends):
+                continue
+
+        # Annotate the user object with the path of the backend.
+        user.backend = backend_path
+        return user
+
+    # The credentials supplied are invalid to all backends, fire signal
+    user_login_failed.send(sender=__name__, credentials=_clean_credentials(credentials), request=request)
+
+
+auth.authenticate = authenticate
+
+
 class AuthMixin:
     request = None
+    partial_credential_error = None
+
+    key_prefix_captcha = "_LOGIN_INVALID_{}"
 
     def get_user_from_session(self):
         if self.request.session.is_empty():
@@ -53,13 +113,9 @@ class AuthMixin:
         ip = ip or get_request_ip(self.request)
         return ip
 
-    def check_is_block(self, raise_exception=True):
-        if hasattr(self.request, 'data'):
-            username = self.request.data.get("username")
-        else:
-            username = self.request.POST.get("username")
+    def _check_is_block(self, username, raise_exception=True):
         ip = self.get_request_ip()
-        if is_block_login(username, ip):
+        if LoginBlockUtil(username, ip).is_block():
             logger.warn('Ip was blocked' + ': ' + username + ':' + ip)
             exception = errors.BlockLoginError(username=username, ip=ip)
             if raise_exception:
@@ -67,6 +123,13 @@ class AuthMixin:
             else:
                 return exception
 
+    def check_is_block(self, raise_exception=True):
+        if hasattr(self.request, 'data'):
+            username = self.request.data.get("username")
+        else:
+            username = self.request.POST.get("username")
+        self._check_is_block(username, raise_exception)
+
     def decrypt_passwd(self, raw_passwd):
         # 获取解密密钥，对密码进行解密
         rsa_private_key = self.request.session.get(RSA_PRIVATE_KEY)
@@ -75,80 +138,152 @@ class AuthMixin:
                 return rsa_decrypt(raw_passwd, rsa_private_key)
             except Exception as e:
                 logger.error(e, exc_info=True)
-                logger.error(f'Decrypt password faild: password[{raw_passwd}] rsa_private_key[{rsa_private_key}]')
+                logger.error(f'Decrypt password failed: password[{raw_passwd}] '
+                             f'rsa_private_key[{rsa_private_key}]')
                 return None
         return raw_passwd
 
-    def check_user_auth(self, decrypt_passwd=False):
-        self.check_is_block()
+    def raise_credential_error(self, error):
+        raise self.partial_credential_error(error=error)
+
+    def _set_partial_credential_error(self, username, ip, request):
+        self.partial_credential_error = partial(errors.CredentialError, username=username, ip=ip, request=request)
+
+    def get_auth_data(self, decrypt_passwd=False):
         request = self.request
         if hasattr(request, 'data'):
             data = request.data
         else:
             data = request.POST
-        username = data.get('username', '')
-        password = data.get('password', '')
-        challenge = data.get('challenge', '')
-        public_key = data.get('public_key', '')
-        ip = self.get_request_ip()
 
-        CredentialError = partial(errors.CredentialError, username=username, ip=ip, request=request)
+        items = ['username', 'password', 'challenge', 'public_key', 'auto_login']
+        username, password, challenge, public_key, auto_login = bulk_get(data, *items,  default='')
+        password = password + challenge.strip()
+        ip = self.get_request_ip()
+        self._set_partial_credential_error(username=username, ip=ip, request=request)
 
         if decrypt_passwd:
             password = self.decrypt_passwd(password)
             if not password:
-                raise CredentialError(error=errors.reason_password_decrypt_failed)
+                self.raise_credential_error(errors.reason_password_decrypt_failed)
+        return username, password, public_key, ip, auto_login
 
-        user = authenticate(request,
-                            username=username,
-                            password=password + challenge.strip(),
-                            public_key=public_key)
+    def _check_only_allow_exists_user_auth(self, username):
+        # 仅允许预先存在的用户认证
+        if settings.ONLY_ALLOW_EXIST_USER_AUTH:
+            exist = User.objects.filter(username=username).exists()
+            if not exist:
+                logger.error(f"Only allow exist user auth, login failed: {username}")
+                self.raise_credential_error(errors.reason_user_not_exist)
 
+    def _check_auth_user_is_valid(self, username, password, public_key):
+        user = authenticate(self.request, username=username, password=password, public_key=public_key)
         if not user:
-            raise CredentialError(error=errors.reason_password_failed)
+            self.raise_credential_error(errors.reason_password_failed)
         elif user.is_expired:
-            raise CredentialError(error=errors.reason_user_inactive)
+            self.raise_credential_error(errors.reason_user_expired)
         elif not user.is_active:
-            raise CredentialError(error=errors.reason_user_inactive)
+            self.raise_credential_error(errors.reason_user_inactive)
+        return user
 
+    def _check_login_acl(self, user, ip):
+        # ACL 限制用户登录
+        from acls.models import LoginACL
+        is_allowed = LoginACL.allow_user_to_login(user, ip)
+        if not is_allowed:
+            raise errors.LoginIPNotAllowed(username=user.username, request=self.request)
+
+    def set_login_failed_mark(self):
+        ip = self.get_request_ip()
+        cache.set(self.key_prefix_captcha.format(ip), 1, 3600)
+
+    def set_passwd_verify_on_session(self, user: User):
+        self.request.session['user_id'] = str(user.id)
+        self.request.session['auth_password'] = 1
+        self.request.session['auth_password_expired_at'] = time.time() + settings.AUTH_EXPIRED_SECONDS
+
+    def check_is_need_captcha(self):
+        # 最近有登录失败时需要填写验证码
+        ip = get_request_ip(self.request)
+        need = cache.get(self.key_prefix_captcha.format(ip))
+        return need
+
+    def check_user_auth(self, decrypt_passwd=False):
+        self.check_is_block()
+        request = self.request
+        username, password, public_key, ip, auto_login = self.get_auth_data(decrypt_passwd=decrypt_passwd)
+
+        self._check_only_allow_exists_user_auth(username)
+        user = self._check_auth_user_is_valid(username, password, public_key)
+        # 校验login-acl规则
+        self._check_login_acl(user, ip)
         self._check_password_require_reset_or_not(user)
         self._check_passwd_is_too_simple(user, password)
+        self._check_passwd_need_update(user)
+
+        LoginBlockUtil(username, ip).clean_failed_count()
+        request.session['auth_password'] = 1
+        request.session['user_id'] = str(user.id)
+        request.session['auto_login'] = auto_login
+        request.session['auth_backend'] = getattr(user, 'backend', settings.AUTH_BACKEND_MODEL)
+        return user
+
+    def _check_is_local_user(self, user: User):
+        if user.source != User.Source.local:
+            raise self.raise_credential_error(error=errors.only_local_users_are_allowed)
+
+    def check_oauth2_auth(self, user: User, auth_backend):
+        ip = self.get_request_ip()
+        request = self.request
+
+        self._set_partial_credential_error(user.username, ip, request)
+        self._check_is_local_user(user)
+        self._check_is_block(user.username)
+        self._check_login_acl(user, ip)
+
+        LoginBlockUtil(user.username, ip).clean_failed_count()
+        MFABlockUtils(user.username, ip).clean_failed_count()
 
-        clean_failed_count(username, ip)
         request.session['auth_password'] = 1
         request.session['user_id'] = str(user.id)
-        auth_backend = getattr(user, 'backend', 'django.contrib.auth.backends.ModelBackend')
         request.session['auth_backend'] = auth_backend
         return user
 
     @classmethod
-    def generate_reset_password_url_with_flash_msg(cls, user: User, flash_view_name):
+    def generate_reset_password_url_with_flash_msg(cls, user, message):
         reset_passwd_url = reverse('authentication:reset-password')
         query_str = urlencode({
             'token': user.generate_reset_token()
         })
         reset_passwd_url = f'{reset_passwd_url}?{query_str}'
 
-        flash_page_url = reverse(flash_view_name)
-        query_str = urlencode({
-            'redirect_url': reset_passwd_url
-        })
-        return f'{flash_page_url}?{query_str}'
+        message_data = {
+            'title': _('Please change your password'),
+            'message': message,
+            'interval': 3,
+            'redirect_url': reset_passwd_url,
+        }
+        return FlashMessageUtil.gen_message_url(message_data)
 
     @classmethod
     def _check_passwd_is_too_simple(cls, user: User, password):
         if user.is_superuser and password == 'admin':
-            url = cls.generate_reset_password_url_with_flash_msg(
-                user, 'authentication:passwd-too-simple-flash-msg'
-            )
+            message = _('Your password is too simple, please change it for security')
+            url = cls.generate_reset_password_url_with_flash_msg(user, message=message)
             raise errors.PasswdTooSimple(url)
 
+    @classmethod
+    def _check_passwd_need_update(cls, user: User):
+        if user.need_update_password:
+            message = _('You should to change your password before login')
+            url = cls.generate_reset_password_url_with_flash_msg(user, message)
+            raise errors.PasswdNeedUpdate(url)
+
     @classmethod
     def _check_password_require_reset_or_not(cls, user: User):
         if user.password_has_expired:
-            url = cls.generate_reset_password_url_with_flash_msg(
-                user, 'authentication:passwd-has-expired-flash-msg'
-            )
+            message = _('Your password has expired, please reset before logging in')
+            url = cls.generate_reset_password_url_with_flash_msg(user, message)
             raise errors.PasswordRequireResetError(url)
 
     def check_user_auth_if_need(self, decrypt_passwd=False):
@@ -170,15 +305,34 @@ class AuthMixin:
             raise errors.MFAUnsetError(user, self.request, url)
         raise errors.MFARequiredError()
 
+    def mark_mfa_ok(self):
+        self.request.session['auth_mfa'] = 1
+        self.request.session['auth_mfa_time'] = time.time()
+        self.request.session['auth_mfa_type'] = 'otp'
+
+    def check_mfa_is_block(self, username, ip, raise_exception=True):
+        if MFABlockUtils(username, ip).is_block():
+            logger.warn('Ip was blocked' + ': ' + username + ':' + ip)
+            exception = errors.BlockMFAError(username=username, request=self.request, ip=ip)
+            if raise_exception:
+                raise exception
+            else:
+                return exception
+
     def check_user_mfa(self, code):
         user = self.get_user_from_session()
+        ip = self.get_request_ip()
+        self.check_mfa_is_block(user.username, ip)
         ok = user.check_mfa(code)
         if ok:
-            self.request.session['auth_mfa'] = 1
-            self.request.session['auth_mfa_time'] = time.time()
-            self.request.session['auth_mfa_type'] = 'otp'
+            self.mark_mfa_ok()
             return
-        raise errors.MFAFailedError(username=user.username, request=self.request)
+
+        raise errors.MFAFailedError(
+            username=user.username,
+            request=self.request,
+            ip=ip
+        )
 
     def get_ticket(self):
         from tickets.models import Ticket
@@ -245,3 +399,10 @@ class AuthMixin:
                 sender=self.__class__, username=username,
                 request=self.request, reason=reason
             )
+
+    def redirect_to_guard_view(self):
+        guard_url = reverse('authentication:login-guard')
+        args = self.request.META.get('QUERY_STRING', '')
+        if args:
+            guard_url = "%s?%s" % (guard_url, args)
+        return redirect(guard_url)

apps/authentication/models.py

@@ -68,6 +68,7 @@ class LoginConfirmSetting(CommonModelMixin):
     def create_confirm_ticket(self, request=None):
         from tickets import const
         from tickets.models import Ticket
+        from orgs.models import Organization
         ticket_title = _('Login confirm') + ' {}'.format(self.user)
         ticket_meta = self.construct_confirm_ticket_meta(request)
         ticket_assignees = self.reviewers.all()
@@ -75,6 +76,7 @@ class LoginConfirmSetting(CommonModelMixin):
             'title': ticket_title,
             'type': const.TicketTypeChoices.login_confirm.value,
             'meta': ticket_meta,
+            'org_id': Organization.ROOT_ID,
         }
         ticket = Ticket.objects.create(**data)
         ticket.assignees.set(ticket_assignees)

apps/authentication/serializers.py

@@ -1,16 +1,23 @@
 # -*- coding: utf-8 -*-
 #
+from django.utils import timezone
 from rest_framework import serializers
 
 from common.utils import get_object_or_none
 from users.models import User
+from assets.models import Asset, SystemUser, Gateway
+from applications.models import Application
 from users.serializers import UserProfileSerializer
-from .models import AccessKey, LoginConfirmSetting, SSOToken
+from assets.serializers import ProtocolsField
+from perms.serializers.asset.permission import ActionsField
+from .models import AccessKey, LoginConfirmSetting
 
 
 __all__ = [
     'AccessKeySerializer', 'OtpVerifySerializer', 'BearerTokenSerializer',
     'MFAChallengeSerializer', 'LoginConfirmSettingSerializer', 'SSOTokenSerializer',
+    'ConnectionTokenSerializer', 'ConnectionTokenSecretSerializer', 'RDPFileSerializer',
+    'PasswordVerifySerializer',
 ]
 
 
@@ -25,6 +32,10 @@ class OtpVerifySerializer(serializers.Serializer):
     code = serializers.CharField(max_length=6, min_length=6)
 
 
+class PasswordVerifySerializer(serializers.Serializer):
+    password = serializers.CharField()
+
+
 class BearerTokenSerializer(serializers.Serializer):
     username = serializers.CharField(allow_null=True, required=False, write_only=True)
     password = serializers.CharField(write_only=True, allow_null=True,
@@ -40,6 +51,10 @@ class BearerTokenSerializer(serializers.Serializer):
     def get_keyword(obj):
         return 'Bearer'
 
+    def update_last_login(self, user):
+        user.last_login = timezone.now()
+        user.save(update_fields=['last_login'])
+
     def create(self, validated_data):
         request = self.context.get('request')
         if request.user and not request.user.is_anonymous:
@@ -52,6 +67,8 @@ class BearerTokenSerializer(serializers.Serializer):
                     "user id {} not exist".format(user_id)
                 )
         token, date_expired = user.create_bearer_token(request)
+        self.update_last_login(user)
+
         instance = {
             "token": token,
             "date_expired": date_expired,
@@ -82,3 +99,105 @@ class SSOTokenSerializer(serializers.Serializer):
     username = serializers.CharField(write_only=True)
     login_url = serializers.CharField(read_only=True)
     next = serializers.CharField(write_only=True, allow_blank=True, required=False, allow_null=True)
+
+
+class ConnectionTokenSerializer(serializers.Serializer):
+    user = serializers.CharField(max_length=128, required=False, allow_blank=True)
+    system_user = serializers.CharField(max_length=128, required=True)
+    asset = serializers.CharField(max_length=128, required=False)
+    application = serializers.CharField(max_length=128, required=False)
+
+    @staticmethod
+    def validate_user(user_id):
+        from users.models import User
+        user = User.objects.filter(id=user_id).first()
+        if user is None:
+            raise serializers.ValidationError('user id not exist')
+        return user
+
+    @staticmethod
+    def validate_system_user(system_user_id):
+        from assets.models import SystemUser
+        system_user = SystemUser.objects.filter(id=system_user_id).first()
+        if system_user is None:
+            raise serializers.ValidationError('system_user id not exist')
+        return system_user
+
+    @staticmethod
+    def validate_asset(asset_id):
+        from assets.models import Asset
+        asset = Asset.objects.filter(id=asset_id).first()
+        if asset is None:
+            raise serializers.ValidationError('asset id not exist')
+        return asset
+
+    @staticmethod
+    def validate_application(app_id):
+        from applications.models import Application
+        app = Application.objects.filter(id=app_id).first()
+        if app is None:
+            raise serializers.ValidationError('app id not exist')
+        return app
+
+    def validate(self, attrs):
+        asset = attrs.get('asset')
+        application = attrs.get('application')
+        if not asset and not application:
+            raise serializers.ValidationError('asset or application required')
+        if asset and application:
+            raise serializers.ValidationError('asset and application should only one')
+        return super().validate(attrs)
+
+
+class ConnectionTokenUserSerializer(serializers.ModelSerializer):
+    class Meta:
+        model = User
+        fields = ['id', 'name', 'username', 'email']
+
+
+class ConnectionTokenAssetSerializer(serializers.ModelSerializer):
+    protocols = ProtocolsField(label='Protocols', read_only=True)
+
+    class Meta:
+        model = Asset
+        fields = ['id', 'hostname', 'ip', 'protocols', 'org_id']
+
+
+class ConnectionTokenSystemUserSerializer(serializers.ModelSerializer):
+    class Meta:
+        model = SystemUser
+        fields = ['id', 'name', 'username', 'password', 'private_key']
+
+
+class ConnectionTokenGatewaySerializer(serializers.ModelSerializer):
+    class Meta:
+        model = Gateway
+        fields = ['id', 'ip', 'port', 'username', 'password', 'private_key']
+
+
+class ConnectionTokenRemoteAppSerializer(serializers.Serializer):
+    program = serializers.CharField()
+    working_directory = serializers.CharField()
+    parameters = serializers.CharField()
+
+
+class ConnectionTokenApplicationSerializer(serializers.ModelSerializer):
+    class Meta:
+        model = Application
+        fields = ['id', 'name', 'category', 'type']
+
+
+class ConnectionTokenSecretSerializer(serializers.Serializer):
+    type = serializers.ChoiceField(choices=[('application', 'Application'), ('asset', 'Asset')])
+    user = ConnectionTokenUserSerializer(read_only=True)
+    asset = ConnectionTokenAssetSerializer(read_only=True)
+    remote_app = ConnectionTokenRemoteAppSerializer(read_only=True)
+    application = ConnectionTokenApplicationSerializer(read_only=True)
+    system_user = ConnectionTokenSystemUserSerializer(read_only=True)
+    gateway = ConnectionTokenGatewaySerializer(read_only=True)
+    actions = ActionsField()
+
+
+class RDPFileSerializer(ConnectionTokenSerializer):
+    width = serializers.IntegerField(default=1280)
+    height = serializers.IntegerField(default=800)

apps/authentication/signals_handlers.py

@@ -1,6 +1,5 @@
 from importlib import import_module
 
-from django.contrib.auth import BACKEND_SESSION_KEY
 from django.conf import settings
 from django.contrib.auth import user_logged_in
 from django.core.cache import cache
@@ -24,18 +23,18 @@ def on_user_auth_login_success(sender, user, request, **kwargs):
 
 
 @receiver(openid_user_login_success)
-def on_oidc_user_login_success(sender, request, user, **kwargs):
-    request.session[BACKEND_SESSION_KEY] = 'OIDCAuthCodeBackend'
+def on_oidc_user_login_success(sender, request, user, create=False, **kwargs):
+    request.session['auth_backend'] = settings.AUTH_BACKEND_OIDC_CODE
     post_auth_success.send(sender, user=user, request=request)
 
 
 @receiver(openid_user_login_failed)
 def on_oidc_user_login_failed(sender, username, request, reason, **kwargs):
-    request.session[BACKEND_SESSION_KEY] = 'OIDCAuthCodeBackend'
+    request.session['auth_backend'] = settings.AUTH_BACKEND_OIDC_CODE
     post_auth_failed.send(sender, username=username, request=request, reason=reason)
 
 
 @receiver(cas_user_authenticated)
 def on_cas_user_login_success(sender, request, user, **kwargs):
-    request.session[BACKEND_SESSION_KEY] = 'CASBackend'
+    request.session['auth_backend'] = settings.AUTH_BACKEND_CAS
     post_auth_success.send(sender, user=user, request=request)

apps/authentication/templates/authentication/login.html

@@ -1,12 +1,11 @@
-{% load static %}
 {% load i18n %}
+{% load bootstrap3 %}
+{% load static %}
 <!DOCTYPE html>
 <html>
-<!--/*@thymesVar id="LoginConstants" type="com.fit2cloud.support.common.constants.LoginConstants"*/-->
-<!--/*@thymesVar id="message" type="java.lang.String"*/-->
 <head>
     <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
-        <link rel="shortcut icon" href="{{ FAVICON_URL }}" type="image/x-icon">
+    <link rel="shortcut icon" href="{{ FAVICON_URL }}" type="image/x-icon">
     <title>
         {{ JMS_TITLE }}
     </title>
@@ -16,6 +15,8 @@
     <link href="{% static 'css/font-awesome.min.css' %}" rel="stylesheet">
     <link href="{% static 'css/bootstrap-style.css' %}" rel="stylesheet">
     <link href="{% static 'css/login-style.css' %}" rel="stylesheet">
+    <link href="{% static 'css/style.css' %}" rel="stylesheet">
+    <link href="{% static 'css/jumpserver.css' %}" rel="stylesheet">
 
     <!-- scripts -->
     <script src="{% static 'js/jquery-3.1.1.min.js' %}"></script>
@@ -24,26 +25,54 @@
     <script src="{% static 'js/plugins/datatables/datatables.min.js' %}"></script>
 
     <style>
+        .login-content {
+            box-shadow: 0 5px 5px -3px rgb(0 0 0 / 20%), 0 8px 10px 1px rgb(0 0 0 / 14%), 0 3px 14px 2px rgb(0 0 0 / 12%);
+        }
 
-        .box-1{
+        .help-block {
+            margin: 0;
+            text-align: left;
+        }
+
+        form label {
+            color: #737373;
+            font-size: 13px;
+            font-weight: normal;
+        }
+
+        .hr-line-dashed {
+            border-top: 1px dashed #e7eaec;
+            color: #ffffff;
+            background-color: #ffffff;
+            height: 1px;
+            margin: 20px 0;
+        }
+
+        .login-content {
             height: 472px;
             width: 984px;
             margin-right: auto;
             margin-left: auto;
-            margin-top: calc((100vh - 470px)/2);
-
+            margin-top: calc((100vh - 470px) / 3);
         }
-        .box-2{
+        body {
+            background-color: #f2f2f2;
+            height: calc(100vh - (100vh - 470px) / 3);
+        }
+
+        .right-image-box {
             height: 100%;
             width: 50%;
             float: right;
         }
-        .box-3{
+
+        .left-form-box {
             text-align: center;
             background-color: white;
             height: 100%;
             width: 50%;
         }
+
         .captcha {
             float: right;
         }
@@ -56,136 +85,164 @@
             text-align: left;
         }
 
+        .form-group.has-error {
+            margin-bottom: 0;
+        }
+
+        .captch-field .has-error .help-block {
+            margin-top: -8px !important;
+        }
+
+        .no-captcha-challenge .form-group {
+            margin-bottom: 20px;
+        }
+
+        .jms-title {
+            padding: 40px 10px 10px;
+        }
+
+        .no-captcha-challenge .jms-title {
+            padding: 60px 10px 10px;
+        }
+
+        .no-captcha-challenge .welcome-message {
+            padding-top: 10px;
+        }
+
+        .radio, .checkbox {
+            margin: 0;
+        }
+
+        #github_star {
+            float: right;
+            margin: 10px 10px 0 0;
+        }
+        .more-login-item {
+            border-right: 1px dashed #dedede;
+            padding-left: 5px;
+            padding-right: 5px;
+        }
+
+        .more-login-item:last-child {
+            border: none;
+        }
+
     </style>
 </head>
 
-<body style="height: 100%;font-size: 13px">
-    <div>
-        <div class="box-1">
-            <div class="box-2">
-                <img src="{{ LOGIN_IMAGE_URL }}" style="height: 100%; width: 100%"/>
+<body>
+<div class="login-content ">
+    <div class="right-image-box">
+        <a href="{% if not XPACK_ENABLED %}https://github.com/jumpserver/jumpserver{% endif %}">
+            <img src="{{ LOGIN_IMAGE_URL }}" style="height: 100%; width: 100%"/>
+        </a>
+    </div>
+    <div class="left-form-box {% if not form.challenge and not form.captcha %} no-captcha-challenge {% endif %}">
+        <div style="background-color: white">
+            <div class="jms-title">
+                <span style="font-size: 21px;font-weight:400;color: #151515;letter-spacing: 0;">{{ JMS_TITLE }}</span>
             </div>
-            <div class="box-3">
-                <div style="background-color: white">
-                    {% if form.challenge %}
-                        <div style="margin-top: 20px;padding-top: 30px;padding-left: 20px;padding-right: 20px;height: 60px">
+            <div class="contact-form col-md-10 col-md-offset-1">
+                <form id="login-form" action="" method="post" role="form" novalidate="novalidate">
+                    {% csrf_token %}
+                    <div style="line-height: 17px;margin-bottom: 20px;color: #999999;">
+                    {% if form.errors %}
+                        <p class="red-fonts" style="color: red">
+                            {% if form.non_field_errors %}
+                                {{ form.non_field_errors.as_text }}
+                            {% endif %}
+                        </p>
                     {% else %}
-                        <div style="margin-top: 20px;padding-top: 40px;padding-left: 20px;padding-right: 20px;height: 80px">
+                        <p class="welcome-message">
+                            {% trans 'Welcome back, please enter username and password to login' %}
+                        </p>
                     {% endif %}
-                            <span style="font-size: 21px;font-weight:400;color: #151515;letter-spacing: 0;">{{ JMS_TITLE }}</span>
-                        </div>
-                    <div style="font-size: 12px;color: #999999;letter-spacing: 0;line-height: 18px;margin-top: 18px">
-                        {% trans 'Welcome back, please enter username and password to login' %}
                     </div>
-                    <div style="margin-bottom: 0px">
-                        <div>
-                            <div class="col-md-1"></div>
-                            <div class="contact-form col-md-10" style="margin-top: 0px;height: 35px">
-                                <form id="contact-form" action="" method="post" role="form" novalidate="novalidate">
-                                    {% csrf_token %}
-                                    {% if form.non_field_errors %}
-                                        {% if form.challenge %}
-                                            <div style="height: 50px;color: red;line-height: 17px;">
-                                        {% else %}
-                                            <div style="height: 70px;color: red;line-height: 17px;">
-                                        {% endif %}
-                                            <p class="red-fonts">{{ form.non_field_errors.as_text }}</p>
-                                        </div>
-                                    {% elif form.errors.captcha %}
-                                        <p class="red-fonts">{% trans 'Captcha invalid' %}</p>
-                                    {% else %}
-                                        <div style="height: 50px"></div>
-                                    {% endif %}
 
-                                    <div class="form-group">
-                                        <input type="text" class="form-control" name="{{ form.username.html_name }}" placeholder="{% trans 'Username' %}" required="" value="{% if form.username.value %}{{ form.username.value }}{% endif %}" style="height: 35px">
-                                        {% if form.errors.username %}
-                                            <div class="help-block field-error">
-                                                <p class="red-fonts">{{ form.errors.username.as_text }}</p>
-                                            </div>
-                                        {% endif %}
-                                    </div>
-                                    <div class="form-group">
-                                        <input type="password" class="form-control" id="password" placeholder="{% trans 'Password' %}" required="">
-                                        <input id="password-hidden" type="text" style="display:none" name="{{ form.password.html_name }}">
-                                        {% if form.errors.password %}
-                                            <div class="help-block field-error">
-                                                <p class="red-fonts">{{ form.errors.password.as_text }}</p>
-                                            </div>
-                                        {% endif %}
-                                    </div>
-                                    {% if form.challenge %}
-                                        <div class="form-group">
-                                            <input type="challenge" class="form-control" id="challenge" name="{{ form.challenge.html_name }}" placeholder="{% trans 'MFA code' %}" >
-                                            {% if form.errors.challenge %}
-                                                <div class="help-block field-error">
-                                                    <p class="red-fonts">{{ form.errors.challenge.as_text }}</p>
-                                                </div>
-                                            {% endif %}
-                                        </div>
-                                    {% endif %}
-                                    {% if form.captcha %}
-                                    <div class="form-group" style="height: 50px;margin-bottom: 0;font-size: 13px">
-                                        {{ form.captcha }}
-                                    </div>
-                                    {% else %}
-                                    <div class="form-group" style="height: 25px;margin-bottom: 0;font-size: 13px"></div>
-                                    {% endif %}
-                                    <div class="form-group" style="margin-top: 10px">
-                                        <button type="submit" class="btn btn-transparent" onclick="doLogin();return false;">{% trans 'Login' %}</button>
-                                    </div>
-
-                                    <div>
-                                        {% if AUTH_OPENID or AUTH_CAS %}
-                                            <div class="hr-line-dashed"></div>
-                                            <div style="display: inline-block; float: left">
-                                                <b class="text-muted text-left" style="margin-right: 10px">{% trans "More login options" %}</b>
-                                                {% if AUTH_OPENID %}
-                                                    <a href="{% url 'authentication:openid:login' %}">
-                                                        <i class="fa fa-openid"></i> {% trans 'OpenID' %}
-                                                    </a>
-                                                {% endif %}
-                                                {% if AUTH_CAS %}
-                                                    <a href="{% url 'authentication:cas:cas-login' %}">
-                                                        <i class="fa"><img src="{{ LOGIN_CAS_LOGO_URL }}" height="13" width="13"></i> {% trans 'CAS' %}
-                                                    </a>
-                                                {% endif %}
-                                            </div>
-                                            <div class="text-center" style="display: inline-block; float: right">
-                                        {% else %}
-                                            <div class="text-center" style="display: inline-block;">
-                                        {% endif %}
-                                            <a id="forgot_password" href="{% url 'authentication:forgot-password' %}">
-                                                <small>{% trans 'Forgot password' %}?</small>
-                                            </a>
-                                        </div>
-                                    </div>
-                                </form>
+                    {% bootstrap_field form.username show_label=False %}
+                    <div class="form-group">
+                        <input type="password" class="form-control" id="password" placeholder="{% trans 'Password' %}" required="">
+                        <input id="password-hidden" type="text" style="display:none" name="{{ form.password.html_name }}">
+                    </div>
+                    {% if form.challenge %}
+                        {% bootstrap_field form.challenge show_label=False %}
+                    {% elif form.captcha %}
+                        <div class="captch-field">
+                            {% bootstrap_field form.captcha show_label=False %}
+                        </div>
+                    {% endif %}
+                    <div class="form-group" style="padding-top: 5px; margin-bottom: 10px">
+                        <div class="row">
+                            <div class="col-md-6" style="text-align: left">
+                                {% if form.auto_login %}
+                                    {% bootstrap_field form.auto_login form_group_class='' %}
+                                {% endif %}
+                            </div>
+                            <div class="col-md-6">
+                                <a id="forgot_password" href="{{ forgot_password_url }}" style="float: right">
+                                    <small>{% trans 'Forgot password' %}?</small>
+                                </a>
                             </div>
-                            <div class="col-md-1"></div>
                         </div>
                     </div>
-                </div>
+
+                    <div class="form-group" style="">
+                        <button type="submit" class="btn btn-transparent" onclick="doLogin();return false;">{% trans 'Login' %}</button>
+                    </div>
+
+                    <div>
+                        {% if AUTH_OPENID or AUTH_CAS or AUTH_WECOM or AUTH_DINGTALK %}
+                            <div class="hr-line-dashed"></div>
+                            <div style="display: inline-block; float: left">
+                                <b class="text-muted text-left" >{% trans "More login options" %}</b>
+                                {% if AUTH_OPENID %}
+                                    <a href="{% url 'authentication:openid:login' %}" class="more-login-item">
+                                        <i class="fa fa-openid"></i> {% trans 'OpenID' %}
+                                    </a>
+                                {% endif %}
+                                {% if AUTH_CAS %}
+                                    <a href="{% url 'authentication:cas:cas-login' %}" class="more-login-item">
+                                        <i class="fa"><img src="{{ LOGIN_CAS_LOGO_URL }}" height="13" width="13"></i> {% trans 'CAS' %}
+                                    </a>
+                                {% endif %}
+                                {% if AUTH_WECOM %}
+                                    <a href="{% url 'authentication:wecom-qr-login' %}" class="more-login-item">
+                                        <i class="fa"><img src="{{ LOGIN_WECOM_LOGO_URL }}" height="13" width="13"></i> {% trans 'WeCom' %}
+                                    </a>
+                                {% endif %}
+                                {% if AUTH_DINGTALK %}
+                                    <a href="{% url 'authentication:dingtalk-qr-login' %}" class="more-login-item">
+                                        <i class="fa"><img src="{{ LOGIN_DINGTALK_LOGO_URL }}" height="13" width="13"></i> {% trans 'DingTalk' %}
+                                    </a>
+                                {% endif %}
+
+                            </div>
+                        {% else %}
+                            <div class="text-center" style="display: inline-block;">
+                        {% endif %}
+                        </div>
+                    </div>
+                </form>
             </div>
         </div>
     </div>
-    </div>
-
+</div>
 </body>
 <script type="text/javascript" src="/static/js/plugins/jsencrypt/jsencrypt.min.js"></script>
 <script>
-    function encryptLoginPassword(password, rsaPublicKey){
+    function encryptLoginPassword(password, rsaPublicKey) {
         var jsencrypt = new JSEncrypt(); //加密对象
         jsencrypt.setPublicKey(rsaPublicKey); // 设置密钥
         return jsencrypt.encrypt(password); //加密
     }
+
     function doLogin() {
         //公钥加密
         var rsaPublicKey = "{{ rsa_public_key }}"
-        var password =$('#password').val(); //明文密码
+        var password = $('#password').val(); //明文密码
         var passwordEncrypted = encryptLoginPassword(password, rsaPublicKey)
         $('#password-hidden').val(passwordEncrypted); //返回给密码输入input
-        $('#contact-form').submit();//post提交
+        $('#login-form').submit();//post提交
     }
 
     $(document).ready(function () {

apps/authentication/urls/api_urls.py

@@ -9,19 +9,24 @@ app_name = 'authentication'
 router = DefaultRouter()
 router.register('access-keys', api.AccessKeyViewSet, 'access-key')
 router.register('sso', api.SSOViewSet, 'sso')
+router.register('connection-token', api.UserConnectionTokenViewSet, 'connection-token')
 
 
 urlpatterns = [
     # path('token/', api.UserToken.as_view(), name='user-token'),
+    path('wecom/qr/unbind/', api.WeComQRUnBindForUserApi.as_view(), name='wecom-qr-unbind'),
+    path('wecom/qr/unbind/<uuid:user_id>/', api.WeComQRUnBindForAdminApi.as_view(), name='wecom-qr-unbind-for-admin'),
+
+    path('dingtalk/qr/unbind/', api.DingTalkQRUnBindForUserApi.as_view(), name='dingtalk-qr-unbind'),
+    path('dingtalk/qr/unbind/<uuid:user_id>/', api.DingTalkQRUnBindForAdminApi.as_view(), name='dingtalk-qr-unbind-for-admin'),
+
     path('auth/', api.TokenCreateApi.as_view(), name='user-auth'),
     path('tokens/', api.TokenCreateApi.as_view(), name='auth-token'),
     path('mfa/challenge/', api.MFAChallengeApi.as_view(), name='mfa-challenge'),
-    path('connection-token/',
-         api.UserConnectionTokenApi.as_view(), name='connection-token'),
     path('otp/verify/', api.UserOtpVerifyApi.as_view(), name='user-otp-verify'),
+    path('password/verify/', api.UserPasswordVerifyApi.as_view(), name='user-password-verify'),
     path('login-confirm-ticket/status/', api.TicketStatusApi.as_view(), name='login-confirm-ticket-status'),
     path('login-confirm-settings/<uuid:user_id>/', api.LoginConfirmSettingUpdateApi.as_view(), name='login-confirm-setting-update')
 ]
 
 urlpatterns += router.urls
-