refactor tree (重构&优化资产树/用户授权树加载速度) (#5548) (#5549)

* Bai reactor tree ( 重构获取完整资产树中节点下资产总数的逻辑) (#5548)

* tree: v0.1

* tree: v0.2

* tree: v0.3

* tree: v0.4

* tree: 添加并发锁未请求到时的debug日志

* 以空间换时间的方式优化资产树

* Reactor tree togther v2 (#5576)

* Bai reactor tree ( 重构获取完整资产树中节点下资产总数的逻辑) (#5548)

* tree: v0.1

* tree: v0.2

* tree: v0.3

* tree: v0.4

* tree: 添加并发锁未请求到时的debug日志

* 以空间换时间的方式优化资产树

* 修改授权适配新方案

* 添加树处理工具

* 完成新的用户授权树计算以及修改一些信号

* 重构了获取资产的一些 api

* 重构了一些节点的api

* 整理了一些代码

* 完成了api 的重构

* 重构检查节点数量功能

* 完成重构授权树工具类

* api 添加强制刷新参数

* 整理一些信号

* 处理一些信号的问题

* 完成了信号的处理

* 重构了资产树相关的锁机制

* RebuildUserTreeTask 还得添加回来

* 优化下不能在root组织的检查函数

* 优化资产树变化时锁的使用

* 修改一些算法的小工具

* 资产树锁不再校验是否在具体组织里

* 整理了一些信号的位置

* 修复资产与节点关系维护的bug

* 去掉一些调试代码

* 修复资产授权过期检查刷新授权树的 bug

* 添加了可重入锁

* 添加一些计时,优化一些sql

* 增加 union 查询的支持

* 尝试用 sql 解决节点资产数量问题

* 开始优化计算授权树节点资产数量不用冗余表

* 新代码能跑起来了,修复一下bug

* 去掉 UserGrantedMappingNode 换成 UserAssetGrantedTreeNodeRelation

* 修了些bug,做了些优化

* 优化QuerySetStage 执行逻辑

* 与小白的内存结合了

* 删掉老的表,迁移新的 assets_amount 字段

* 优化用户授权页面资产列表 count 慢

* 修复批量命令数量不对

* 修改获取非直接授权节点的 children 的逻辑

* 获取整棵树的节点

* 回退锁

* 整理迁移脚本

* 改变更新树策略

* perf: 修改一波缩进

* fix: 修改handler名称

* 修复授权树获取资产sql 泛滥

* 修复授权规则有效bug

* 修复一些bug

* 修复一些bug

* 又修了一些小bug

* 去掉了老的 get_nodes_all_assets

* 修改一些写法

* Reactor tree togther b2 (#5570)

* fix: 修改handler名称

* perf: 优化生成树

* perf: 去掉注释

* 优化了一些

* 重新生成迁移脚本

* 去掉周期检查节点资产数量的任务

* Pr@reactor tree togther guang@perf mapping (#5573)

* fix: 修改handler名称

* perf: mapping 拆分出来

* 修改名称

* perf: 修改锁名

* perf: 去掉检查节点任务

* perf: 修改一下名称

* perf: 优化一波

Co-authored-by: Jiangjie.Bai <32935519+BaiJiangJie@users.noreply.github.com>
Co-authored-by: Bai <bugatti_it@163.com>
Co-authored-by: xinwen <coderWen@126.com>

Co-authored-by: xinwen <coderWen@126.com>
Co-authored-by: 老广 <ibuler@qq.com>
This commit is contained in:
Jiangjie.Bai
2021-02-05 13:29:29 +08:00
committed by GitHub
parent 709e7af953
commit 7cf6e54f01
49 changed files with 1829 additions and 1480 deletions

View File

@@ -1,7 +1,6 @@
# -*- coding: utf-8 -*-
#
import abc
from django.conf import settings
from rest_framework.generics import (
ListAPIView
)
@@ -10,16 +9,11 @@ from rest_framework.request import Request
from assets.api.mixin import SerializeToTreeNodeMixin
from common.utils import get_logger
from .mixin import ForAdminMixin, ForUserMixin, UserNodeGrantStatusDispatchMixin
from perms.hands import Node, User
from .mixin import RoleAdminMixin, RoleUserMixin
from perms.hands import User
from perms import serializers
from perms.utils.asset.user_permission import (
get_indirect_granted_node_children,
get_user_granted_nodes_list_via_mapping_node,
get_top_level_granted_nodes,
rebuild_user_tree_if_need, get_favorite_node,
get_ungrouped_node
)
from perms.utils.asset.user_permission import UserGrantedNodesQueryUtils
logger = get_logger(__name__)
@@ -61,7 +55,6 @@ class BaseGrantedNodeApi(_GrantedNodeStructApi, metaclass=abc.ABCMeta):
serializer_class = serializers.NodeGrantedSerializer
def list(self, request, *args, **kwargs):
rebuild_user_tree_if_need(request, self.user)
nodes = self.get_nodes()
serializer = self.get_serializer(nodes, many=True)
return Response(serializer.data)
@@ -73,7 +66,6 @@ class BaseNodeChildrenApi(NodeChildrenMixin, BaseGrantedNodeApi, metaclass=abc.A
class BaseGrantedNodeAsTreeApi(SerializeToTreeNodeMixin, _GrantedNodeStructApi, metaclass=abc.ABCMeta):
def list(self, request: Request, *args, **kwargs):
rebuild_user_tree_if_need(request, self.user)
nodes = self.get_nodes()
nodes = self.serialize_nodes(nodes, with_asset_amount=True)
return Response(data=nodes)
@@ -83,30 +75,16 @@ class BaseNodeChildrenAsTreeApi(NodeChildrenMixin, BaseGrantedNodeAsTreeApi, met
pass
class UserGrantedNodeChildrenMixin(UserNodeGrantStatusDispatchMixin):
class UserGrantedNodeChildrenMixin:
user: User
request: Request
def get_children(self):
user = self.user
key = self.request.query_params.get('key')
if not key:
nodes = list(get_top_level_granted_nodes(user))
else:
nodes = self.dispatch_get_data(key, user)
nodes = UserGrantedNodesQueryUtils(user).get_node_children(key)
return nodes
def get_data_on_node_direct_granted(self, key):
return Node.objects.filter(parent_key=key)
def get_data_on_node_indirect_granted(self, key):
nodes = get_indirect_granted_node_children(self.user, key)
return nodes
def get_data_on_node_not_granted(self, key):
return Node.objects.none()
class UserGrantedNodesMixin:
"""
@@ -115,41 +93,38 @@ class UserGrantedNodesMixin:
user: User
def get_nodes(self):
nodes = []
if settings.PERM_SINGLE_ASSET_TO_UNGROUP_NODE:
nodes.append(get_ungrouped_node(self.user))
nodes.append(get_favorite_node(self.user))
nodes.extend(get_user_granted_nodes_list_via_mapping_node(self.user))
utils = UserGrantedNodesQueryUtils(self.user)
nodes = utils.get_whole_tree_nodes()
return nodes
# ------------------------------------------
# 最终的 api
class UserGrantedNodeChildrenForAdminApi(ForAdminMixin, UserGrantedNodeChildrenMixin, BaseNodeChildrenApi):
class UserGrantedNodeChildrenForAdminApi(RoleAdminMixin, UserGrantedNodeChildrenMixin, BaseNodeChildrenApi):
pass
class MyGrantedNodeChildrenApi(ForUserMixin, UserGrantedNodeChildrenMixin, BaseNodeChildrenApi):
class MyGrantedNodeChildrenApi(RoleUserMixin, UserGrantedNodeChildrenMixin, BaseNodeChildrenApi):
pass
class UserGrantedNodeChildrenAsTreeForAdminApi(ForAdminMixin, UserGrantedNodeChildrenMixin, BaseNodeChildrenAsTreeApi):
class UserGrantedNodeChildrenAsTreeForAdminApi(RoleAdminMixin, UserGrantedNodeChildrenMixin, BaseNodeChildrenAsTreeApi):
pass
class MyGrantedNodeChildrenAsTreeApi(ForUserMixin, UserGrantedNodeChildrenMixin, BaseNodeChildrenAsTreeApi):
class MyGrantedNodeChildrenAsTreeApi(RoleUserMixin, UserGrantedNodeChildrenMixin, BaseNodeChildrenAsTreeApi):
pass
class UserGrantedNodesForAdminApi(ForAdminMixin, UserGrantedNodesMixin, BaseGrantedNodeApi):
class UserGrantedNodesForAdminApi(RoleAdminMixin, UserGrantedNodesMixin, BaseGrantedNodeApi):
pass
class MyGrantedNodesApi(ForUserMixin, UserGrantedNodesMixin, BaseGrantedNodeApi):
class MyGrantedNodesApi(RoleUserMixin, UserGrantedNodesMixin, BaseGrantedNodeApi):
pass
class MyGrantedNodesAsTreeApi(ForUserMixin, UserGrantedNodesMixin, BaseGrantedNodeAsTreeApi):
class MyGrantedNodesAsTreeApi(RoleUserMixin, UserGrantedNodesMixin, BaseGrantedNodeAsTreeApi):
pass
# ------------------------------------------