1.5.7 Merge to dev (#3766)

* [Update] 暂存，优化解决不了问题

* [Update] 待续（小白）

* [Update] 修改asset user

* [Update] 计划再次更改

* [Update] 修改asset user

* [Update] 暂存与喜爱

* [Update] Add id in

* [Update] 阶段性完成ops task该做

* [Update] 修改asset user api

* [Update] 修改asset user 任务，查看认证等

* [Update] 基本完成asset user改造

* [Update] dynamic user only allow 1

* [Update] 修改asset user task

* [Update] 修改node admin user task api

* [Update] remove file header license

* [Update] 添加sftp root

* [Update] 暂存

* [Update] 暂存

* [Update] 修改翻译

* [Update] 修改系统用户改为同名后，用户名改为空

* [Update] 基本完成CAS调研

* [Update] 支持cas server

* [Update] 支持cas server

* [Update] 添加requirements

* [Update] 为方便调试添加mysql ipython到包中

* [Update] 添加huaweiyun翻译

* [Update] 增加下载session 录像

* [Update] 只有第一次通知replay离线的使用方法

* [Update] 暂存一下

* [Bugfix] 获取系统用户信息报错

* [Bugfix] 修改system user info

* [Update] 改成清理10天git status

* [Update] 修改celery日志保留时间

* [Update]修复部分pip包依赖的版本不兼容问题 (#3672)

* [Update] 修复用户更新页面会清空用户public_key的问题

* Fix broken dependencies

Co-authored-by: BaiJiangJie <32935519+BaiJiangJie@users.noreply.github.com>

* [Update] 修改获取系统用户auth info

* [Update] Remove log

* [Bugfix] 修复sftp home设置的bug

* [Update] 授权的系统用户添加sftp root

* [Update] 修改系统用户关联的用户

* [Update] 修改placeholder

* [Update] 优化获取授权的系统用户

* [Update] 修改tasks

* [Update] tree service update

* [Update] 暂存

* [Update] 基本完成用户授权树和资产树改造

* [Update] Dashbaord perf

* [update] Add huawei cloud sdk requirements

* [Updte] 优化dashboard页面

* [Update] system user auth info 添加id

* [Update] 修改系统用户serializer

* [Update] 优化api

* [Update] LDAP Test Util (#3720)

* [Update] LDAPTestUtil 1

* [Update] LDAPTestUtil 2

* [Update] LDAPTestUtil 3

* [Update] LDAPTestUtil 4

* [Update] LDAPTestUtil 5

* [Update] LDAPTestUtil 6

* [Update] LDAPTestUtil 7

* [Update] session 已添加is success，并且添加display serializer

* [Bugfix] 修复无法删除空节点的bug

* [Update] 命令记录分组织显示

* [Update] Session is_success 添加迁移文件

* [Update] 批量命令添加org_id

* [Update] 修复一些文案，修改不绑定MFA，不能ssh登录

* [Update] 修改replay api, 返回session信息

* [Update] 解决无效es导致访问命令记录页面失败的问题

* [Update] 拆分profile view

* [Update] 修改一个翻译

* [Update] 修改aysnc api框架

* [Update] 命令列表添加risk level

* [Update] 完成录像打包下载

* [Update] 更改登陆otp页面

* [Update] 修改command 存储redis_level

* [Update] 修改翻译

* [Update] 修改系统用户的用户列表字段

* [Update] 使用新logo和统一Jumpserver为JumpServer

* [Update] 优化cloud task

* [Update] 统一period task

* [Update] 统一period form serializer字段

* [Update] 修改period task

* [Update] 修改资产网关信息

* [Update] 用户授权资产树资产信息添加domain

* [Update] 修改翻译

* [Update] 测试可连接性

* 1.5.7 bai (#3764)

* [Update] 修复index页面Bug；修复测试资产用户可连接性问题；

* [Update] 修改测试资产用户可连接

* [Bugfix] 修复backends问题

* [Update] 修改marksafe依赖版本

* [Update] 修改测试资产用户可连接性

* [Update] 修改检测服务器性能时获取percent值

* [Update] 更新依赖boto3=1.12.14

Co-authored-by: Yanzhe Lee <lee.yanzhe@yanzhe.org>
Co-authored-by: BaiJiangJie <32935519+BaiJiangJie@users.noreply.github.com>
Co-authored-by: Bai <bugatti_it@163.com>

apps/assets/tasks/push_system_user.py

@@ -1,11 +1,13 @@
 # ~*~ coding: utf-8 ~*~
 
+from itertools import groupby
 from celery import shared_task
 from django.utils.translation import ugettext as _
 
 from common.utils import encrypt_password, get_logger
+from orgs.utils import tmp_to_org, org_aware_func
 from . import const
-from .utils import clean_hosts_by_protocol, clean_hosts
+from .utils import clean_ansible_task_hosts, group_asset_by_platform
 
 
 logger = get_logger(__file__)
@@ -15,31 +17,34 @@ __all__ = [
 ]
 
 
-def get_push_linux_system_user_tasks(system_user):
+def get_push_unixlike_system_user_tasks(system_user, username=None):
+    if username is None:
+        username = system_user.username
+    password = system_user.password
+    public_key = system_user.public_key
+
     tasks = [
         {
-            'name': 'Add user {}'.format(system_user.username),
+            'name': 'Add user {}'.format(username),
             'action': {
                 'module': 'user',
                 'args': 'name={} shell={} state=present'.format(
-                    system_user.username, system_user.shell,
+                    username, system_user.shell or '/bin/bash',
                 ),
             }
         },
         {
-            'name': 'Add group {}'.format(system_user.username),
+            'name': 'Add group {}'.format(username),
             'action': {
                 'module': 'group',
-                'args': 'name={} state=present'.format(
-                    system_user.username,
-                ),
+                'args': 'name={} state=present'.format(username),
             }
         },
         {
             'name': 'Check home dir exists',
             'action': {
                 'module': 'stat',
-                'args': 'path=/home/{}'.format(system_user.username)
+                'args': 'path=/home/{}'.format(username)
             },
             'register': 'home_existed'
         },
@@ -47,29 +52,29 @@ def get_push_linux_system_user_tasks(system_user):
             'name': "Set home dir permission",
             'action': {
                 'module': 'file',
-                'args': "path=/home/{0} owner={0} group={0} mode=700".format(system_user.username)
+                'args': "path=/home/{0} owner={0} group={0} mode=700".format(username)
             },
             'when': 'home_existed.stat.exists == true'
         }
     ]
-    if system_user.password:
+    if password:
         tasks.append({
-            'name': 'Set {} password'.format(system_user.username),
+            'name': 'Set {} password'.format(username),
             'action': {
                 'module': 'user',
                 'args': 'name={} shell={} state=present password={}'.format(
-                    system_user.username, system_user.shell,
-                    encrypt_password(system_user.password, salt="K3mIlKK"),
+                    username, system_user.shell,
+                    encrypt_password(password, salt="K3mIlKK"),
                 ),
             }
         })
-    if system_user.public_key:
+    if public_key:
         tasks.append({
-            'name': 'Set {} authorized key'.format(system_user.username),
+            'name': 'Set {} authorized key'.format(username),
             'action': {
                 'module': 'authorized_key',
                 'args': "user={} state=present key='{}'".format(
-                    system_user.username, system_user.public_key
+                    username, public_key
                 )
             }
         })
@@ -81,26 +86,27 @@ def get_push_linux_system_user_tasks(system_user):
             sudo_tmp.append(s.strip(','))
         sudo = ','.join(sudo_tmp)
         tasks.append({
-            'name': 'Set {} sudo setting'.format(system_user.username),
+            'name': 'Set {} sudo setting'.format(username),
             'action': {
                 'module': 'lineinfile',
                 'args': "dest=/etc/sudoers state=present regexp='^{0} ALL=' "
                         "line='{0} ALL=(ALL) NOPASSWD: {1}' "
-                        "validate='visudo -cf %s'".format(
-                    system_user.username, sudo,
-                )
+                        "validate='visudo -cf %s'".format(username, sudo)
             }
         })
 
     return tasks
 
 
-def get_push_windows_system_user_tasks(system_user):
+def get_push_windows_system_user_tasks(system_user, username=None):
+    if username is None:
+        username = system_user.username
+    password = system_user.password
     tasks = []
-    if not system_user.password:
+    if not password:
         return tasks
-    tasks.append({
-        'name': 'Add user {}'.format(system_user.username),
+    task = {
+        'name': 'Add user {}'.format(username),
         'action': {
             'module': 'win_user',
             'args': 'fullname={} '
@@ -112,78 +118,95 @@ def get_push_windows_system_user_tasks(system_user):
                     'password_never_expires=yes '
                     'groups="Users,Remote Desktop Users" '
                     'groups_action=add '
-                    ''.format(system_user.name,
-                              system_user.username,
-                              system_user.password),
+                    ''.format(username, username, password),
         }
-    })
+    }
+    print(task)
+    tasks.append(task)
     return tasks
 
 
-def get_push_system_user_tasks(host, system_user):
-    if host.is_unixlike():
-        tasks = get_push_linux_system_user_tasks(system_user)
-    elif host.is_windows():
-        tasks = get_push_windows_system_user_tasks(system_user)
-    else:
-        msg = _(
-            "The asset {} system platform {} does not "
-            "support run Ansible tasks".format(host.hostname, host.platform)
-        )
-        logger.info(msg)
-        tasks = []
+def get_push_system_user_tasks(system_user, platform="unixlike", username=None):
+    """
+    :param system_user:
+    :param platform:
+    :param username: 当动态时，近推送某个
+    :return:
+    """
+    get_task_map = {
+        "unixlike": get_push_unixlike_system_user_tasks,
+        "windows": get_push_windows_system_user_tasks,
+    }
+    get_tasks = get_task_map.get(platform, get_push_unixlike_system_user_tasks)
+    if not system_user.username_same_with_user:
+        return get_tasks(system_user)
+    tasks = []
+    # 仅推送这个username
+    if username is not None:
+        tasks.extend(get_tasks(system_user, username))
+        return tasks
+    users = system_user.users.all().values_list('username', flat=True)
+    print(_("System user is dynamic: {}").format(list(users)))
+    for _username in users:
+        tasks.extend(get_tasks(system_user, _username))
     return tasks
 
 
-@shared_task(queue="ansible")
-def push_system_user_util(system_user, assets, task_name):
+@org_aware_func("system_user")
+def push_system_user_util(system_user, assets, task_name, username=None):
     from ops.utils import update_or_create_ansible_task
-    if not system_user.is_need_push():
-        msg = _("Push system user task skip, auto push not enable or "
-                "protocol is not ssh or rdp: {}").format(system_user.name)
-        logger.info(msg)
-        return {}
-
-    # Set root as system user is dangerous
-    if system_user.username.lower() in ["root", "administrator"]:
-        msg = _("For security, do not push user {}".format(system_user.username))
-        logger.info(msg)
-        return {}
-
-    hosts = clean_hosts(assets)
+    hosts = clean_ansible_task_hosts(assets, system_user=system_user)
     if not hosts:
         return {}
 
-    hosts = clean_hosts_by_protocol(system_user, hosts)
-    if not hosts:
-        return {}
+    platform_hosts_map = {}
+    hosts_sorted = sorted(hosts, key=group_asset_by_platform)
+    platform_hosts = groupby(hosts_sorted, key=group_asset_by_platform)
+    for i in platform_hosts:
+        platform_hosts_map[i[0]] = list(i[1])
 
-    for host in hosts:
-        system_user.load_specific_asset_auth(host)
-        tasks = get_push_system_user_tasks(host, system_user)
-        if not tasks:
-            continue
+    def run_task(_tasks, _hosts):
+        if not _tasks:
+            return
         task, created = update_or_create_ansible_task(
-            task_name=task_name, hosts=[host], tasks=tasks, pattern='all',
+            task_name=task_name, hosts=_hosts, tasks=_tasks, pattern='all',
             options=const.TASK_OPTIONS, run_as_admin=True,
-            created_by=system_user.org_id,
         )
         task.run()
 
+    for platform, _hosts in platform_hosts_map.items():
+        if not _hosts:
+            continue
+        print(_("Start push system user for platform: [{}]").format(platform))
+        print(_("Hosts count: {}").format(len(_hosts)))
+
+        if not system_user.has_special_auth():
+            logger.debug("System user not has special auth")
+            tasks = get_push_system_user_tasks(system_user, platform, username=username)
+            run_task(tasks, _hosts)
+            continue
+
+        for _host in _hosts:
+            system_user.load_asset_special_auth(_host)
+            tasks = get_push_system_user_tasks(system_user, platform, username=username)
+            run_task(tasks, [_host])
+
 
 @shared_task(queue="ansible")
 def push_system_user_to_assets_manual(system_user):
-    assets = system_user.get_all_assets()
+    assets = system_user.get_related_assets()
     task_name = _("Push system users to assets: {}").format(system_user.name)
     return push_system_user_util(system_user, assets, task_name=task_name)
 
 
 @shared_task(queue="ansible")
-def push_system_user_a_asset_manual(system_user, asset):
-    task_name = _("Push system users to asset: {} => {}").format(
-        system_user.name, asset
+def push_system_user_a_asset_manual(system_user, asset, username=None):
+    if username is None:
+        username = system_user.username
+    task_name = _("Push system users to asset: {}({}) => {}").format(
+        system_user.name, username, asset
     )
-    return push_system_user_util(system_user, [asset], task_name=task_name)
+    return push_system_user_util(system_user, [asset], task_name=task_name, username=username)
 
 
 @shared_task(queue="ansible")
@@ -199,4 +222,4 @@ def push_system_user_to_assets(system_user, assets):
 # @after_app_shutdown_clean_periodic
 # def push_system_user_period():
 #     for system_user in SystemUser.objects.all():
-#         push_system_user_related_nodes(system_user)
+#         push_system_user_related_nodes(system_user)