refactor tree (重构&优化资产树/用户授权树加载速度) (#5548) (#5549)

* Bai reactor tree ( 重构获取完整资产树中节点下资产总数的逻辑) (#5548) * tree: v0.1 * tree: v0.2 * tree: v0.3 * tree: v0.4 * tree: 添加并发锁未请求到时的debug日志 * 以空间换时间的方式优化资产树 * Reactor tree togther v2 (#5576) * Bai reactor tree ( 重构获取完整资产树中节点下资产总数的逻辑) (#5548) * tree: v0.1 * tree: v0.2 * tree: v0.3 * tree: v0.4 * tree: 添加并发锁未请求到时的debug日志 * 以空间换时间的方式优化资产树 * 修改授权适配新方案 * 添加树处理工具 * 完成新的用户授权树计算以及修改一些信号 * 重构了获取资产的一些 api * 重构了一些节点的api * 整理了一些代码 * 完成了api 的重构 * 重构检查节点数量功能 * 完成重构授权树工具类 * api 添加强制刷新参数 * 整理一些信号 * 处理一些信号的问题 * 完成了信号的处理 * 重构了资产树相关的锁机制 * RebuildUserTreeTask 还得添加回来 * 优化下不能在root组织的检查函数 * 优化资产树变化时锁的使用 * 修改一些算法的小工具 * 资产树锁不再校验是否在具体组织里 * 整理了一些信号的位置 * 修复资产与节点关系维护的bug * 去掉一些调试代码 * 修复资产授权过期检查刷新授权树的 bug * 添加了可重入锁 * 添加一些计时，优化一些sql * 增加 union 查询的支持 * 尝试用 sql 解决节点资产数量问题 * 开始优化计算授权树节点资产数量不用冗余表 * 新代码能跑起来了，修复一下bug * 去掉 UserGrantedMappingNode 换成 UserAssetGrantedTreeNodeRelation * 修了些bug，做了些优化 * 优化QuerySetStage 执行逻辑 * 与小白的内存结合了 * 删掉老的表，迁移新的 assets_amount 字段 * 优化用户授权页面资产列表 count 慢 * 修复批量命令数量不对 * 修改获取非直接授权节点的 children 的逻辑 * 获取整棵树的节点 * 回退锁 * 整理迁移脚本 * 改变更新树策略 * perf: 修改一波缩进 * fix: 修改handler名称 * 修复授权树获取资产sql 泛滥 * 修复授权规则有效bug * 修复一些bug * 修复一些bug * 又修了一些小bug * 去掉了老的 get_nodes_all_assets * 修改一些写法 * Reactor tree togther b2 (#5570) * fix: 修改handler名称 * perf: 优化生成树 * perf: 去掉注释 * 优化了一些 * 重新生成迁移脚本 * 去掉周期检查节点资产数量的任务 * Pr@reactor tree togther guang@perf mapping (#5573) * fix: 修改handler名称 * perf: mapping 拆分出来 * 修改名称 * perf: 修改锁名 * perf: 去掉检查节点任务 * perf: 修改一下名称 * perf: 优化一波 Co-authored-by: Jiangjie.Bai <32935519+BaiJiangJie@users.noreply.github.com> Co-authored-by: Bai <bugatti_it@163.com> Co-authored-by: xinwen <coderWen@126.com> Co-authored-by: xinwen <coderWen@126.com> Co-authored-by: 老广 <ibuler@qq.com>
2025-09-05 01:11:57 +00:00 · 2021-02-05 13:29:29 +08:00
parent 709e7af953
commit 7cf6e54f01
49 changed files with 1829 additions and 1480 deletions
--- a/apps/perms/signals_handler/refresh_perms.py
+++ b/apps/perms/signals_handler/refresh_perms.py
@@ -0,0 +1,115 @@
+# -*- coding: utf-8 -*-
+#
+from django.db.models.signals import m2m_changed, pre_delete, pre_save, post_save
+from django.dispatch import receiver
+
+from users.models import User
+from assets.models import Asset
+from orgs.utils import current_org
+from common.utils import get_logger
+from common.exceptions import M2MReverseNotAllowed
+from common.const.signals import POST_ADD, POST_REMOVE, POST_CLEAR
+from perms.models import AssetPermission
+from perms.utils.asset.user_permission import UserGrantedTreeRefreshController
+
+
+logger = get_logger(__file__)
+
+
+@receiver(m2m_changed, sender=User.groups.through)
+def on_user_groups_change(sender, instance, action, reverse, pk_set, **kwargs):
+    if action.startswith('post'):
+        if reverse:
+            group_ids = [instance.id]
+            user_ids = pk_set
+        else:
+            group_ids = pk_set
+            user_ids = [instance.id]
+
+        exists = AssetPermission.user_groups.through.objects.filter(usergroup_id__in=group_ids).exists()
+        if exists:
+            org_ids = [current_org.id]
+            UserGrantedTreeRefreshController.add_need_refresh_orgs_for_users(org_ids, user_ids)
+
+
+@receiver([pre_delete], sender=AssetPermission)
+def on_asset_perm_pre_delete(sender, instance, **kwargs):
+    # 授权删除之前，查出所有相关用户
+    UserGrantedTreeRefreshController.add_need_refresh_by_asset_perm_ids([instance.id])
+
+
+@receiver([pre_save], sender=AssetPermission)
+def on_asset_perm_pre_save(sender, instance, **kwargs):
+    try:
+        old = AssetPermission.objects.get(id=instance.id)
+
+        if old.is_valid != instance.is_valid:
+            UserGrantedTreeRefreshController.add_need_refresh_by_asset_perm_ids([instance.id])
+    except AssetPermission.DoesNotExist:
+        pass
+
+
+@receiver([post_save], sender=AssetPermission)
+def on_asset_perm_post_save(sender, instance, created, **kwargs):
+    if created:
+        UserGrantedTreeRefreshController.add_need_refresh_by_asset_perm_ids([instance.id])
+
+
+def need_rebuild_mapping_node(action):
+    return action in (POST_REMOVE, POST_ADD, POST_CLEAR)
+
+
+@receiver(m2m_changed, sender=AssetPermission.nodes.through)
+def on_permission_nodes_changed(sender, instance, action, reverse, **kwargs):
+    if reverse:
+        raise M2MReverseNotAllowed
+
+    if need_rebuild_mapping_node(action):
+        UserGrantedTreeRefreshController.add_need_refresh_by_asset_perm_ids([instance.id])
+
+
+@receiver(m2m_changed, sender=AssetPermission.assets.through)
+def on_permission_assets_changed(sender, instance, action, reverse, pk_set, model, **kwargs):
+    if reverse:
+        raise M2MReverseNotAllowed
+
+    if need_rebuild_mapping_node(action):
+        UserGrantedTreeRefreshController.add_need_refresh_by_asset_perm_ids([instance.id])
+
+
+@receiver(m2m_changed, sender=AssetPermission.users.through)
+def on_asset_permission_users_changed(sender, action, reverse, pk_set, **kwargs):
+    if reverse:
+        raise M2MReverseNotAllowed
+
+    if need_rebuild_mapping_node(action):
+        UserGrantedTreeRefreshController.add_need_refresh_orgs_for_users(
+            [current_org.id], pk_set
+        )
+
+
+@receiver(m2m_changed, sender=AssetPermission.user_groups.through)
+def on_asset_permission_user_groups_changed(sender, action, pk_set, reverse, **kwargs):
+    if reverse:
+        raise M2MReverseNotAllowed
+
+    if need_rebuild_mapping_node(action):
+        user_ids = User.groups.through.objects.filter(usergroup_id__in=pk_set).distinct().values_list('user_id', flat=True)
+        UserGrantedTreeRefreshController.add_need_refresh_orgs_for_users(
+            [current_org.id], user_ids
+        )
+
+
+@receiver(m2m_changed, sender=Asset.nodes.through)
+def on_node_asset_change(action, instance, reverse, pk_set, **kwargs):
+    if not need_rebuild_mapping_node(action):
+        return
+
+    if reverse:
+        asset_pk_set = pk_set
+        node_pk_set = [instance.id]
+    else:
+        asset_pk_set = [instance.id]
+        node_pk_set = pk_set
+
+    UserGrantedTreeRefreshController.add_need_refresh_on_nodes_assets_relate_change(node_pk_set, asset_pk_set)