github/jumpserver
1
0
Fork 0
mirror of https://github.com/jumpserver/jumpserver.git synced 2025-09-19 10:26:27 +00:00
Code Issues Projects Releases Wiki Activity

refactor tree (重构&优化资产树/用户授权树加载速度) (#5548) (#5549)

Browse Source 
* Bai reactor tree ( 重构获取完整资产树中节点下资产总数的逻辑) (#5548)

* tree: v0.1

* tree: v0.2

* tree: v0.3

* tree: v0.4

* tree: 添加并发锁未请求到时的debug日志

* 以空间换时间的方式优化资产树

* Reactor tree togther v2 (#5576)

* Bai reactor tree ( 重构获取完整资产树中节点下资产总数的逻辑) (#5548)

* tree: v0.1

* tree: v0.2

* tree: v0.3

* tree: v0.4

* tree: 添加并发锁未请求到时的debug日志

* 以空间换时间的方式优化资产树

* 修改授权适配新方案

* 添加树处理工具

* 完成新的用户授权树计算以及修改一些信号

* 重构了获取资产的一些 api

* 重构了一些节点的api

* 整理了一些代码

* 完成了api 的重构

* 重构检查节点数量功能

* 完成重构授权树工具类

* api 添加强制刷新参数

* 整理一些信号

* 处理一些信号的问题

* 完成了信号的处理

* 重构了资产树相关的锁机制

* RebuildUserTreeTask 还得添加回来

* 优化下不能在root组织的检查函数

* 优化资产树变化时锁的使用

* 修改一些算法的小工具

* 资产树锁不再校验是否在具体组织里

* 整理了一些信号的位置

* 修复资产与节点关系维护的bug

* 去掉一些调试代码

* 修复资产授权过期检查刷新授权树的 bug

* 添加了可重入锁

* 添加一些计时，优化一些sql

* 增加 union 查询的支持

* 尝试用 sql 解决节点资产数量问题

* 开始优化计算授权树节点资产数量不用冗余表

* 新代码能跑起来了，修复一下bug

* 去掉 UserGrantedMappingNode 换成 UserAssetGrantedTreeNodeRelation

* 修了些bug，做了些优化

* 优化QuerySetStage 执行逻辑

* 与小白的内存结合了

* 删掉老的表，迁移新的 assets_amount 字段

* 优化用户授权页面资产列表 count 慢

* 修复批量命令数量不对

* 修改获取非直接授权节点的 children 的逻辑

* 获取整棵树的节点

* 回退锁

* 整理迁移脚本

* 改变更新树策略

* perf: 修改一波缩进

* fix: 修改handler名称

* 修复授权树获取资产sql 泛滥

* 修复授权规则有效bug

* 修复一些bug

* 修复一些bug

* 又修了一些小bug

* 去掉了老的 get_nodes_all_assets

* 修改一些写法

* Reactor tree togther b2 (#5570)

* fix: 修改handler名称

* perf: 优化生成树

* perf: 去掉注释

* 优化了一些

* 重新生成迁移脚本

* 去掉周期检查节点资产数量的任务

* Pr@reactor tree togther guang@perf mapping (#5573)

* fix: 修改handler名称

* perf: mapping 拆分出来

* 修改名称

* perf: 修改锁名

* perf: 去掉检查节点任务

* perf: 修改一下名称

* perf: 优化一波

Co-authored-by: Jiangjie.Bai <32935519+BaiJiangJie@users.noreply.github.com>
Co-authored-by: Bai <bugatti_it@163.com>
Co-authored-by: xinwen <coderWen@126.com>

Co-authored-by: xinwen <coderWen@126.com>
Co-authored-by: 老广 <ibuler@qq.com>
This commit is contained in:
Jiangjie.Bai
2021-02-05 13:29:29 +08:00
committed by GitHub
parent 709e7af953
commit 7cf6e54f01
49 changed files with 1829 additions and 1480 deletions
Download Patch File Download Diff File Expand all files Collapse all files

6
apps/perms/api/application/user_permission/common.py
View File

@@ -13,7 +13,7 @@ from applications.models import Application
from perms.utils.application.permission import (
get_application_system_users_id
)
from perms.api.asset.user_permission.mixin import ForAdminMixin, ForUserMixin
from perms.api.asset.user_permission.mixin import RoleAdminMixin, RoleUserMixin
from common.permissions import IsOrgAdminOrAppUser
from perms.hands import User, SystemUser
from perms import serializers
@@ -43,11 +43,11 @@ class GrantedApplicationSystemUsersMixin(ListAPIView):
return system_users
class UserGrantedApplicationSystemUsersApi(ForAdminMixin, GrantedApplicationSystemUsersMixin):
class UserGrantedApplicationSystemUsersApi(RoleAdminMixin, GrantedApplicationSystemUsersMixin):
pass
class MyGrantedApplicationSystemUsersApi(ForUserMixin, GrantedApplicationSystemUsersMixin):
class MyGrantedApplicationSystemUsersApi(RoleUserMixin, GrantedApplicationSystemUsersMixin):
pass

6
apps/perms/api/application/user_permission/user_permission_applications.py
View File

@@ -8,7 +8,7 @@ from applications.api.mixin import (
SerializeApplicationToTreeNodeMixin
)
from perms import serializers
from perms.api.asset.user_permission.mixin import ForAdminMixin, ForUserMixin
from perms.api.asset.user_permission.mixin import RoleAdminMixin, RoleUserMixin
from perms.utils.application.user_permission import (
get_user_granted_all_applications
)
@@ -34,11 +34,11 @@ class AllGrantedApplicationsMixin(CommonApiMixin, ListAPIView):
return queryset.only(*self.only_fields)
class UserAllGrantedApplicationsApi(ForAdminMixin, AllGrantedApplicationsMixin):
class UserAllGrantedApplicationsApi(RoleAdminMixin, AllGrantedApplicationsMixin):
pass
class MyAllGrantedApplicationsApi(ForUserMixin, AllGrantedApplicationsMixin):
class MyAllGrantedApplicationsApi(RoleUserMixin, AllGrantedApplicationsMixin):
pass

36
apps/perms/api/asset/user_permission/mixin.py
View File

@@ -4,37 +4,23 @@ from rest_framework.request import Request
from common.permissions import IsOrgAdminOrAppUser, IsValidUser
from common.utils import lazyproperty
from common.http import is_true
from orgs.utils import tmp_to_root_org
from users.models import User
from perms.models import UserGrantedMappingNode
from perms.utils.asset.user_permission import UserGrantedTreeRefreshController
class UserNodeGrantStatusDispatchMixin:
class PermBaseMixin:
user: User
@staticmethod
def get_mapping_node_by_key(key, user):
return UserGrantedMappingNode.objects.get(key=key, user=user)
def dispatch_get_data(self, key, user):
status = UserGrantedMappingNode.get_node_granted_status(key, user)
if status == UserGrantedMappingNode.GRANTED_DIRECT:
return self.get_data_on_node_direct_granted(key)
elif status == UserGrantedMappingNode.GRANTED_INDIRECT:
return self.get_data_on_node_indirect_granted(key)
else:
return self.get_data_on_node_not_granted(key)
def get_data_on_node_direct_granted(self, key):
raise NotImplementedError
def get_data_on_node_indirect_granted(self, key):
raise NotImplementedError
def get_data_on_node_not_granted(self, key):
raise NotImplementedError
def get(self, request, *args, **kwargs):
force = is_true(request.query_params.get('rebuild_tree'))
controller = UserGrantedTreeRefreshController(self.user)
controller.refresh_if_need(force)
return super().get(request, *args, **kwargs)
class ForAdminMixin:
class RoleAdminMixin(PermBaseMixin):
permission_classes = (IsOrgAdminOrAppUser,)
kwargs: dict
@@ -44,7 +30,7 @@ class ForAdminMixin:
return User.objects.get(id=user_id)
class ForUserMixin:
class RoleUserMixin(PermBaseMixin):
permission_classes = (IsValidUser,)
request: Request

156
apps/perms/api/asset/user_permission/user_permission_assets.py
View File

@@ -1,156 +0,0 @@
# -*- coding: utf-8 -*-
#
from perms.api.asset.user_permission.mixin import UserNodeGrantStatusDispatchMixin
from rest_framework.generics import ListAPIView
from rest_framework.response import Response
from rest_framework.request import Request
from django.conf import settings
from assets.api.mixin import SerializeToTreeNodeMixin
from common.utils import get_logger
from perms.pagination import GrantedAssetLimitOffsetPagination
from assets.models import Asset, Node, FavoriteAsset
from perms import serializers
from perms.utils.asset.user_permission import (
get_node_all_granted_assets, get_user_direct_granted_assets,
get_user_granted_all_assets
)
from .mixin import ForAdminMixin, ForUserMixin
logger = get_logger(__name__)
class UserDirectGrantedAssetsApi(ListAPIView):
"""
用户直接授权的资产的列表，也就是授权规则上直接授权的资产，并非是来自节点的
"""
serializer_class = serializers.AssetGrantedSerializer
only_fields = serializers.AssetGrantedSerializer.Meta.only_fields
filterset_fields = ['hostname', 'ip', 'id', 'comment']
search_fields = ['hostname', 'ip', 'comment']
def get_queryset(self):
if getattr(self, 'swagger_fake_view', False):
return Asset.objects.none()
user = self.user
assets = get_user_direct_granted_assets(user)\
.prefetch_related('platform')\
.only(*self.only_fields)
return assets
class UserFavoriteGrantedAssetsApi(ListAPIView):
serializer_class = serializers.AssetGrantedSerializer
only_fields = serializers.AssetGrantedSerializer.Meta.only_fields
filterset_fields = ['hostname', 'ip', 'id', 'comment']
search_fields = ['hostname', 'ip', 'comment']
def get_queryset(self):
if getattr(self, 'swagger_fake_view', False):
return Asset.objects.none()
user = self.user
assets = FavoriteAsset.get_user_favorite_assets(user)\
.prefetch_related('platform')\
.only(*self.only_fields)
return assets
class AssetsAsTreeMixin(SerializeToTreeNodeMixin):
"""
将 资产 序列化成树的结构返回
"""
def list(self, request: Request, *args, **kwargs):
queryset = self.filter_queryset(self.get_queryset())
if request.query_params.get('search'):
# 如果用户搜索的条件不精准，会导致返回大量的无意义数据。
# 这里限制一下返回数据的最大条数
queryset = queryset[:999]
data = self.serialize_assets(queryset, None)
return Response(data=data)
class UserDirectGrantedAssetsForAdminApi(ForAdminMixin, UserDirectGrantedAssetsApi):
pass
class MyDirectGrantedAssetsApi(ForUserMixin, UserDirectGrantedAssetsApi):
pass
class UserFavoriteGrantedAssetsForAdminApi(ForAdminMixin, UserFavoriteGrantedAssetsApi):
pass
class MyFavoriteGrantedAssetsApi(ForUserMixin, UserFavoriteGrantedAssetsApi):
pass
class UserDirectGrantedAssetsAsTreeForAdminApi(ForAdminMixin, AssetsAsTreeMixin, UserDirectGrantedAssetsApi):
pass
class MyUngroupAssetsAsTreeApi(ForUserMixin, AssetsAsTreeMixin, UserDirectGrantedAssetsApi):
def get_queryset(self):
queryset = super().get_queryset()
if not settings.PERM_SINGLE_ASSET_TO_UNGROUP_NODE:
queryset = queryset.none()
return queryset
class UserAllGrantedAssetsApi(ForAdminMixin, ListAPIView):
only_fields = serializers.AssetGrantedSerializer.Meta.only_fields
serializer_class = serializers.AssetGrantedSerializer
filterset_fields = ['hostname', 'ip', 'id', 'comment']
search_fields = ['hostname', 'ip', 'comment']
def get_queryset(self):
if getattr(self, 'swagger_fake_view', False):
return Asset.objects.none()
queryset = get_user_granted_all_assets(self.user)
queryset = queryset.prefetch_related('platform')
return queryset.only(*self.only_fields)
class MyAllGrantedAssetsApi(ForUserMixin, UserAllGrantedAssetsApi):
pass
class MyAllAssetsAsTreeApi(ForUserMixin, AssetsAsTreeMixin, UserAllGrantedAssetsApi):
search_fields = ['hostname', 'ip']
class UserGrantedNodeAssetsApi(UserNodeGrantStatusDispatchMixin, ListAPIView):
serializer_class = serializers.AssetGrantedSerializer
only_fields = serializers.AssetGrantedSerializer.Meta.only_fields
filterset_fields = ['hostname', 'ip', 'id', 'comment']
search_fields = ['hostname', 'ip', 'comment']
pagination_class = GrantedAssetLimitOffsetPagination
pagination_node: Node
def get_queryset(self):
if getattr(self, 'swagger_fake_view', False):
return Asset.objects.none()
node_id = self.kwargs.get("node_id")
node = Node.objects.get(id=node_id)
self.pagination_node = node
return self.dispatch_get_data(node.key, self.user)
def get_data_on_node_direct_granted(self, key):
# 如果这个节点是直接授权的(或者说祖先节点直接授权的), 获取下面的所有资产
return Node.get_node_all_assets_by_key_v2(key)
def get_data_on_node_indirect_granted(self, key):
self.pagination_node = self.get_mapping_node_by_key(key, self.user)
return get_node_all_granted_assets(self.user, key)
def get_data_on_node_not_granted(self, key):
return Asset.objects.none()
class UserGrantedNodeAssetsForAdminApi(ForAdminMixin, UserGrantedNodeAssetsApi):
pass
class MyGrantedNodeAssetsApi(ForUserMixin, UserGrantedNodeAssetsApi):
pass

1
apps/perms/api/asset/user_permission/user_permission_assets/__init__.py Normal file
View File

@@ -0,0 +1 @@
from .views import *

127
apps/perms/api/asset/user_permission/user_permission_assets/mixin.py Normal file
View File

@@ -0,0 +1,127 @@
from rest_framework.response import Response
from rest_framework.request import Request
from users.models import User
from assets.api.mixin import SerializeToTreeNodeMixin
from common.utils import get_logger
from perms.pagination import NodeGrantedAssetPagination, AllGrantedAssetPagination
from assets.models import Asset, Node
from perms import serializers
from perms.utils.asset.user_permission import UserGrantedAssetsQueryUtils, QuerySetStage
logger = get_logger(__name__)
# 获取数据的 ------------------------------------------------------------
class UserDirectGrantedAssetsQuerysetMixin:
only_fields = serializers.AssetGrantedSerializer.Meta.only_fields
user: User
def get_queryset(self):
if getattr(self, 'swagger_fake_view', False):
return Asset.objects.none()
user = self.user
assets = UserGrantedAssetsQueryUtils(user) \
.get_direct_granted_assets() \
.prefetch_related('platform') \
.only(*self.only_fields)
return assets
class UserAllGrantedAssetsQuerysetMixin:
only_fields = serializers.AssetGrantedSerializer.Meta.only_fields
pagination_class = AllGrantedAssetPagination
user: User
def get_union_queryset(self, qs_stage: QuerySetStage):
if getattr(self, 'swagger_fake_view', False):
return Asset.objects.none()
qs_stage.prefetch_related('platform').only(*self.only_fields)
queryset = UserGrantedAssetsQueryUtils(self.user) \
.get_all_granted_assets(qs_stage)
return queryset
class UserFavoriteGrantedAssetsMixin:
only_fields = serializers.AssetGrantedSerializer.Meta.only_fields
user: User
def get_union_queryset(self, qs_stage: QuerySetStage):
if getattr(self, 'swagger_fake_view', False):
return Asset.objects.none()
user = self.user
qs_stage.prefetch_related('platform').only(*self.only_fields)
utils = UserGrantedAssetsQueryUtils(user)
assets = utils.get_favorite_assets(qs_stage=qs_stage)
return assets
class UserGrantedNodeAssetsMixin:
only_fields = serializers.AssetGrantedSerializer.Meta.only_fields
pagination_class = NodeGrantedAssetPagination
pagination_node: Node
user: User
def get_union_queryset(self, qs_stage: QuerySetStage):
if getattr(self, 'swagger_fake_view', False):
return Asset.objects.none()
node_id = self.kwargs.get("node_id")
qs_stage.prefetch_related('platform').only(*self.only_fields)
node, assets = UserGrantedAssetsQueryUtils(self.user).get_node_all_assets(
node_id, qs_stage=qs_stage
)
self.pagination_node = node
return assets
# 控制格式的 ----------------------------------------------------
class AssetsUnionQuerysetMixin:
def get_queryset_union_prefer(self):
if hasattr(self, 'get_union_queryset'):
# 为了支持 union 查询
queryset = Asset.objects.all().distinct()
queryset = self.filter_queryset(queryset)
qs_stage = QuerySetStage()
qs_stage.and_with_queryset(queryset)
queryset = self.get_union_queryset(qs_stage)
else:
queryset = self.filter_queryset(self.get_queryset())
return queryset
class AssetsSerializerFormatMixin(AssetsUnionQuerysetMixin):
serializer_class = serializers.AssetGrantedSerializer
filterset_fields = ['hostname', 'ip', 'id', 'comment']
search_fields = ['hostname', 'ip', 'comment']
def list(self, request, *args, **kwargs):
queryset = self.get_queryset_union_prefer()
page = self.paginate_queryset(queryset)
if page is not None:
serializer = self.get_serializer(page, many=True)
return self.get_paginated_response(serializer.data)
serializer = self.get_serializer(queryset, many=True)
return Response(serializer.data)
class AssetsTreeFormatMixin(AssetsUnionQuerysetMixin, SerializeToTreeNodeMixin):
"""
将 资产 序列化成树的结构返回
"""
def list(self, request: Request, *args, **kwargs):
queryset = self.get_queryset_union_prefer()
if request.query_params.get('search'):
# 如果用户搜索的条件不精准，会导致返回大量的无意义数据。
# 这里限制一下返回数据的最大条数
queryset = queryset[:999]
data = self.serialize_assets(queryset, None)
return Response(data=data)
# def get_serializer_class(self):
# return EmptySerializer

99
apps/perms/api/asset/user_permission/user_permission_assets/views.py Normal file
View File

@@ -0,0 +1,99 @@
from rest_framework.generics import ListAPIView
from django.conf import settings
from common.utils import get_logger
from ..mixin import RoleAdminMixin, RoleUserMixin
from .mixin import (
UserAllGrantedAssetsQuerysetMixin, UserDirectGrantedAssetsQuerysetMixin, UserFavoriteGrantedAssetsMixin,
UserGrantedNodeAssetsMixin, AssetsSerializerFormatMixin, AssetsTreeFormatMixin,
)
__all__ = [
'UserDirectGrantedAssetsForAdminApi', 'MyDirectGrantedAssetsApi', 'UserFavoriteGrantedAssetsForAdminApi',
'MyFavoriteGrantedAssetsApi', 'UserDirectGrantedAssetsAsTreeForAdminApi', 'MyUngroupAssetsAsTreeApi',
'UserAllGrantedAssetsApi', 'MyAllGrantedAssetsApi', 'MyAllAssetsAsTreeApi', 'UserGrantedNodeAssetsForAdminApi',
'MyGrantedNodeAssetsApi',
]
logger = get_logger(__name__)
class UserDirectGrantedAssetsForAdminApi(UserDirectGrantedAssetsQuerysetMixin,
RoleAdminMixin,
AssetsSerializerFormatMixin,
ListAPIView):
pass
class MyDirectGrantedAssetsApi(UserDirectGrantedAssetsQuerysetMixin,
RoleUserMixin,
AssetsSerializerFormatMixin,
ListAPIView):
pass
class UserFavoriteGrantedAssetsForAdminApi(UserFavoriteGrantedAssetsMixin,
RoleAdminMixin,
AssetsSerializerFormatMixin,
ListAPIView):
pass
class MyFavoriteGrantedAssetsApi(UserFavoriteGrantedAssetsMixin,
RoleUserMixin,
AssetsSerializerFormatMixin,
ListAPIView):
pass
class UserDirectGrantedAssetsAsTreeForAdminApi(UserDirectGrantedAssetsQuerysetMixin,
RoleAdminMixin,
AssetsTreeFormatMixin,
ListAPIView):
pass
class MyUngroupAssetsAsTreeApi(UserDirectGrantedAssetsQuerysetMixin,
RoleUserMixin,
AssetsTreeFormatMixin,
ListAPIView):
def get_queryset(self):
queryset = super().get_queryset()
if not settings.PERM_SINGLE_ASSET_TO_UNGROUP_NODE:
queryset = queryset.none()
return queryset
class UserAllGrantedAssetsApi(UserAllGrantedAssetsQuerysetMixin,
RoleAdminMixin,
AssetsSerializerFormatMixin,
ListAPIView):
pass
class MyAllGrantedAssetsApi(UserAllGrantedAssetsQuerysetMixin,
RoleUserMixin,
AssetsSerializerFormatMixin,
ListAPIView):
pass
class MyAllAssetsAsTreeApi(UserAllGrantedAssetsQuerysetMixin,
RoleUserMixin,
AssetsTreeFormatMixin,
ListAPIView):
search_fields = ['hostname', 'ip']
class UserGrantedNodeAssetsForAdminApi(UserGrantedNodeAssetsMixin,
RoleAdminMixin,
AssetsSerializerFormatMixin,
ListAPIView):
pass
class MyGrantedNodeAssetsApi(UserGrantedNodeAssetsMixin,
RoleUserMixin,
AssetsSerializerFormatMixin,
ListAPIView):
pass

55
apps/perms/api/asset/user_permission/user_permission_nodes.py
View File

@@ -1,7 +1,6 @@
# -*- coding: utf-8 -*-
#
import abc
from django.conf import settings
from rest_framework.generics import (
ListAPIView
)
@@ -10,16 +9,11 @@ from rest_framework.request import Request
from assets.api.mixin import SerializeToTreeNodeMixin
from common.utils import get_logger
from .mixin import ForAdminMixin, ForUserMixin, UserNodeGrantStatusDispatchMixin
from perms.hands import Node, User
from .mixin import RoleAdminMixin, RoleUserMixin
from perms.hands import User
from perms import serializers
from perms.utils.asset.user_permission import (
get_indirect_granted_node_children,
get_user_granted_nodes_list_via_mapping_node,
get_top_level_granted_nodes,
rebuild_user_tree_if_need, get_favorite_node,
get_ungrouped_node
)
from perms.utils.asset.user_permission import UserGrantedNodesQueryUtils
logger = get_logger(__name__)
@@ -61,7 +55,6 @@ class BaseGrantedNodeApi(_GrantedNodeStructApi, metaclass=abc.ABCMeta):
serializer_class = serializers.NodeGrantedSerializer
def list(self, request, *args, **kwargs):
rebuild_user_tree_if_need(request, self.user)
nodes = self.get_nodes()
serializer = self.get_serializer(nodes, many=True)
return Response(serializer.data)
@@ -73,7 +66,6 @@ class BaseNodeChildrenApi(NodeChildrenMixin, BaseGrantedNodeApi, metaclass=abc.A
class BaseGrantedNodeAsTreeApi(SerializeToTreeNodeMixin, _GrantedNodeStructApi, metaclass=abc.ABCMeta):
def list(self, request: Request, *args, **kwargs):
rebuild_user_tree_if_need(request, self.user)
nodes = self.get_nodes()
nodes = self.serialize_nodes(nodes, with_asset_amount=True)
return Response(data=nodes)
@@ -83,30 +75,16 @@ class BaseNodeChildrenAsTreeApi(NodeChildrenMixin, BaseGrantedNodeAsTreeApi, met
pass
class UserGrantedNodeChildrenMixin(UserNodeGrantStatusDispatchMixin):
class UserGrantedNodeChildrenMixin:
user: User
request: Request
def get_children(self):
user = self.user
key = self.request.query_params.get('key')
if not key:
nodes = list(get_top_level_granted_nodes(user))
else:
nodes = self.dispatch_get_data(key, user)
nodes = UserGrantedNodesQueryUtils(user).get_node_children(key)
return nodes
def get_data_on_node_direct_granted(self, key):
return Node.objects.filter(parent_key=key)
def get_data_on_node_indirect_granted(self, key):
nodes = get_indirect_granted_node_children(self.user, key)
return nodes
def get_data_on_node_not_granted(self, key):
return Node.objects.none()
class UserGrantedNodesMixin:
"""
@@ -115,41 +93,38 @@ class UserGrantedNodesMixin:
user: User
def get_nodes(self):
nodes = []
if settings.PERM_SINGLE_ASSET_TO_UNGROUP_NODE:
nodes.append(get_ungrouped_node(self.user))
nodes.append(get_favorite_node(self.user))
nodes.extend(get_user_granted_nodes_list_via_mapping_node(self.user))
utils = UserGrantedNodesQueryUtils(self.user)
nodes = utils.get_whole_tree_nodes()
return nodes
# ------------------------------------------
# 最终的 api
class UserGrantedNodeChildrenForAdminApi(ForAdminMixin, UserGrantedNodeChildrenMixin, BaseNodeChildrenApi):
class UserGrantedNodeChildrenForAdminApi(RoleAdminMixin, UserGrantedNodeChildrenMixin, BaseNodeChildrenApi):
pass
class MyGrantedNodeChildrenApi(ForUserMixin, UserGrantedNodeChildrenMixin, BaseNodeChildrenApi):
class MyGrantedNodeChildrenApi(RoleUserMixin, UserGrantedNodeChildrenMixin, BaseNodeChildrenApi):
pass
class UserGrantedNodeChildrenAsTreeForAdminApi(ForAdminMixin, UserGrantedNodeChildrenMixin, BaseNodeChildrenAsTreeApi):
class UserGrantedNodeChildrenAsTreeForAdminApi(RoleAdminMixin, UserGrantedNodeChildrenMixin, BaseNodeChildrenAsTreeApi):
pass
class MyGrantedNodeChildrenAsTreeApi(ForUserMixin, UserGrantedNodeChildrenMixin, BaseNodeChildrenAsTreeApi):
class MyGrantedNodeChildrenAsTreeApi(RoleUserMixin, UserGrantedNodeChildrenMixin, BaseNodeChildrenAsTreeApi):
pass
class UserGrantedNodesForAdminApi(ForAdminMixin, UserGrantedNodesMixin, BaseGrantedNodeApi):
class UserGrantedNodesForAdminApi(RoleAdminMixin, UserGrantedNodesMixin, BaseGrantedNodeApi):
pass
class MyGrantedNodesApi(ForUserMixin, UserGrantedNodesMixin, BaseGrantedNodeApi):
class MyGrantedNodesApi(RoleUserMixin, UserGrantedNodesMixin, BaseGrantedNodeApi):
pass
class MyGrantedNodesAsTreeApi(ForUserMixin, UserGrantedNodesMixin, BaseGrantedNodeAsTreeApi):
class MyGrantedNodesAsTreeApi(RoleUserMixin, UserGrantedNodesMixin, BaseGrantedNodeAsTreeApi):
pass
# ------------------------------------------

184
apps/perms/api/asset/user_permission/user_permission_nodes_with_assets.py
View File

@@ -1,29 +1,23 @@
# -*- coding: utf-8 -*-
#
from itertools import chain
from rest_framework.generics import ListAPIView
from rest_framework.request import Request
from rest_framework.response import Response
from django.db.models import F, Value, CharField, Q
from django.db.models import F, Value, CharField
from django.conf import settings
from common.utils.common import timeit
from orgs.utils import tmp_to_root_org
from common.permissions import IsValidUser
from common.utils import get_logger, get_object_or_none
from .mixin import UserNodeGrantStatusDispatchMixin, ForUserMixin, ForAdminMixin
from .mixin import RoleUserMixin, RoleAdminMixin
from perms.utils.asset.user_permission import (
get_indirect_granted_node_children, UNGROUPED_NODE_KEY, FAVORITE_NODE_KEY,
get_user_direct_granted_assets, get_top_level_granted_nodes,
get_user_granted_nodes_list_via_mapping_node,
get_user_granted_all_assets, rebuild_user_tree_if_need,
get_user_all_assetpermissions_id, get_favorite_node,
get_ungrouped_node, compute_tmp_mapping_node_from_perm,
TMP_GRANTED_FIELD, count_direct_granted_node_assets,
count_node_all_granted_assets
UserGrantedTreeBuildUtils, get_user_all_asset_perm_ids,
UserGrantedNodesQueryUtils, UserGrantedAssetsQueryUtils,
QuerySetStage,
)
from perms.models import AssetPermission
from assets.models import Asset, FavoriteAsset
from perms.models import AssetPermission, PermNode
from assets.models import Asset
from assets.api import SerializeToTreeNodeMixin
from perms.hands import Node
@@ -33,76 +27,45 @@ logger = get_logger(__name__)
class MyGrantedNodesWithAssetsAsTreeApi(SerializeToTreeNodeMixin, ListAPIView):
permission_classes = (IsValidUser,)
def add_ungrouped_resource(self, data: list, user, asset_perms_id):
@timeit
def add_ungrouped_resource(self, data: list, nodes_query_utils, assets_query_utils):
if not settings.PERM_SINGLE_ASSET_TO_UNGROUP_NODE:
return
ungrouped_node = nodes_query_utils.get_ungrouped_node()
ungrouped_node = get_ungrouped_node(user, asset_perms_id=asset_perms_id)
direct_granted_assets = get_user_direct_granted_assets(
user, asset_perms_id=asset_perms_id
).annotate(
direct_granted_assets = assets_query_utils.get_direct_granted_assets().annotate(
parent_key=Value(ungrouped_node.key, output_field=CharField())
).prefetch_related('platform')
data.extend(self.serialize_nodes([ungrouped_node], with_asset_amount=True))
data.extend(self.serialize_assets(direct_granted_assets))
def add_favorite_resource(self, data: list, user, asset_perms_id):
favorite_node = get_favorite_node(user, asset_perms_id)
favorite_assets = FavoriteAsset.get_user_favorite_assets(
user, asset_perms_id=asset_perms_id
).annotate(
@timeit
def add_favorite_resource(self, data: list, nodes_query_utils, assets_query_utils):
favorite_node = nodes_query_utils.get_favorite_node()
qs_state = QuerySetStage().annotate(
parent_key=Value(favorite_node.key, output_field=CharField())
).prefetch_related('platform')
favorite_assets = assets_query_utils.get_favorite_assets(qs_stage=qs_state, only=())
data.extend(self.serialize_nodes([favorite_node], with_asset_amount=True))
data.extend(self.serialize_assets(favorite_assets))
@timeit
def add_node_filtered_by_system_user(self, data: list, user, asset_perms_id):
tmp_nodes = compute_tmp_mapping_node_from_perm(user, asset_perms_id=asset_perms_id)
granted_nodes_key = []
for _node in tmp_nodes:
_granted = getattr(_node, TMP_GRANTED_FIELD, False)
if not _granted:
if settings.PERM_SINGLE_ASSET_TO_UNGROUP_NODE:
assets_amount = count_direct_granted_node_assets(user, _node.key, asset_perms_id)
else:
assets_amount = count_node_all_granted_assets(user, _node.key, asset_perms_id)
_node.assets_amount = assets_amount
else:
granted_nodes_key.append(_node.key)
utils = UserGrantedTreeBuildUtils(user, asset_perms_id)
nodes = utils.get_whole_tree_nodes()
data.extend(self.serialize_nodes(nodes, with_asset_amount=True))
# 查询他们的子节点
q = Q()
for _key in granted_nodes_key:
q |= Q(key__startswith=f'{_key}:')
def add_assets(self, data: list, assets_query_utils: UserGrantedAssetsQueryUtils):
qs_stage = QuerySetStage().annotate(parent_key=F('nodes__key')).prefetch_related('platform')
if q:
descendant_nodes = Node.objects.filter(q).distinct()
else:
descendant_nodes = Node.objects.none()
data.extend(self.serialize_nodes(chain(tmp_nodes, descendant_nodes), with_asset_amount=True))
def add_assets(self, data: list, user, asset_perms_id):
if settings.PERM_SINGLE_ASSET_TO_UNGROUP_NODE:
all_assets = get_user_granted_all_assets(
user,
via_mapping_node=False,
include_direct_granted_assets=False,
asset_perms_id=asset_perms_id
)
all_assets = assets_query_utils.get_direct_granted_nodes_assets(qs_stage=qs_stage)
else:
all_assets = get_user_granted_all_assets(
user,
via_mapping_node=False,
include_direct_granted_assets=True,
asset_perms_id=asset_perms_id
)
all_assets = assets_query_utils.get_all_granted_assets(qs_stage=qs_stage)
all_assets = all_assets.annotate(
parent_key=F('nodes__key')
).prefetch_related('platform')
data.extend(self.serialize_assets(all_assets))
@tmp_to_root_org()
@@ -117,7 +80,7 @@ class MyGrantedNodesWithAssetsAsTreeApi(SerializeToTreeNodeMixin, ListAPIView):
user = request.user
data = []
asset_perms_id = get_user_all_assetpermissions_id(user)
asset_perms_id = get_user_all_asset_perm_ids(user)
system_user_id = request.query_params.get('system_user')
if system_user_id:
@@ -125,89 +88,72 @@ class MyGrantedNodesWithAssetsAsTreeApi(SerializeToTreeNodeMixin, ListAPIView):
id__in=asset_perms_id, system_users__id=system_user_id, actions__gt=0
).values_list('id', flat=True).distinct())
self.add_ungrouped_resource(data, user, asset_perms_id)
self.add_favorite_resource(data, user, asset_perms_id)
nodes_query_utils = UserGrantedNodesQueryUtils(user, asset_perms_id)
assets_query_utils = UserGrantedAssetsQueryUtils(user, asset_perms_id)
self.add_ungrouped_resource(data, nodes_query_utils, assets_query_utils)
self.add_favorite_resource(data, nodes_query_utils, assets_query_utils)
if system_user_id:
# 有系统用户筛选的需要重新计算树结构
self.add_node_filtered_by_system_user(data, user, asset_perms_id)
else:
rebuild_user_tree_if_need(request, user)
all_nodes = get_user_granted_nodes_list_via_mapping_node(user)
all_nodes = nodes_query_utils.get_whole_tree_nodes(with_special=False)
data.extend(self.serialize_nodes(all_nodes, with_asset_amount=True))
self.add_assets(data, user, asset_perms_id)
self.add_assets(data, assets_query_utils)
return Response(data=data)
class GrantedNodeChildrenWithAssetsAsTreeApiMixin(UserNodeGrantStatusDispatchMixin,
SerializeToTreeNodeMixin,
class GrantedNodeChildrenWithAssetsAsTreeApiMixin(SerializeToTreeNodeMixin,
ListAPIView):
"""
带资产的授权树
"""
user: None
def get_data_on_node_direct_granted(self, key):
nodes = Node.objects.filter(parent_key=key)
assets = Asset.org_objects.filter(nodes__key=key).distinct()
assets = assets.prefetch_related('platform')
return nodes, assets
def ensure_key(self):
key = self.request.query_params.get('key', None)
id = self.request.query_params.get('id', None)
def get_data_on_node_indirect_granted(self, key):
user = self.user
asset_perms_id = get_user_all_assetpermissions_id(user)
if key is not None:
return key
nodes = get_indirect_granted_node_children(user, key)
assets = Asset.org_objects.filter(
nodes__key=key,
).filter(
granted_by_permissions__id__in=asset_perms_id
).distinct()
assets = assets.prefetch_related('platform')
return nodes, assets
def get_data_on_node_not_granted(self, key):
return Node.objects.none(), Asset.objects.none()
def get_data(self, key, user):
assets, nodes = [], []
if not key:
root_nodes = get_top_level_granted_nodes(user)
nodes.extend(root_nodes)
elif key == UNGROUPED_NODE_KEY:
assets = get_user_direct_granted_assets(user)
assets = assets.prefetch_related('platform')
elif key == FAVORITE_NODE_KEY:
assets = FavoriteAsset.get_user_favorite_assets(user)
else:
nodes, assets = self.dispatch_get_data(key, user)
return nodes, assets
def id2key_if_have(self):
id = self.request.query_params.get('id')
if id is not None:
node = get_object_or_none(Node, id=id)
if node:
return node.key
node = get_object_or_none(Node, id=id)
if node:
return node.key
def list(self, request: Request, *args, **kwargs):
key = self.request.query_params.get('key')
if key is None:
key = self.id2key_if_have()
user = self.user
key = self.ensure_key()
nodes_query_utils = UserGrantedNodesQueryUtils(user)
assets_query_utils = UserGrantedAssetsQueryUtils(user)
nodes = PermNode.objects.none()
assets = Asset.objects.none()
if not key:
nodes = nodes_query_utils.get_top_level_nodes()
elif key == PermNode.UNGROUPED_NODE_KEY:
assets = assets_query_utils.get_ungroup_assets()
elif key == PermNode.FAVORITE_NODE_KEY:
assets = assets_query_utils.get_favorite_assets()
else:
nodes = nodes_query_utils.get_node_children(key)
assets = assets_query_utils.get_node_assets(key)
assets = assets.prefetch_related('platform')
user = self.user
rebuild_user_tree_if_need(request, user)
nodes, assets = self.get_data(key, user)
tree_nodes = self.serialize_nodes(nodes, with_asset_amount=True)
tree_assets = self.serialize_assets(assets, key)
return Response(data=[*tree_nodes, *tree_assets])
class UserGrantedNodeChildrenWithAssetsAsTreeApi(ForAdminMixin, GrantedNodeChildrenWithAssetsAsTreeApiMixin):
class UserGrantedNodeChildrenWithAssetsAsTreeApi(RoleAdminMixin, GrantedNodeChildrenWithAssetsAsTreeApiMixin):
pass
class MyGrantedNodeChildrenWithAssetsAsTreeApi(ForUserMixin, GrantedNodeChildrenWithAssetsAsTreeApiMixin):
class MyGrantedNodeChildrenWithAssetsAsTreeApi(RoleUserMixin, GrantedNodeChildrenWithAssetsAsTreeApiMixin):
pass

6
apps/perms/api/system_user_permission.py
View File

@@ -1,10 +1,10 @@
from rest_framework import generics
from django.db.models import Q
from django.utils.decorators import method_decorator
from assets.models import SystemUser
from common.permissions import IsValidUser
from orgs.utils import tmp_to_root_org
from perms.utils.asset.user_permission import get_user_all_asset_perm_ids
from .. import serializers
@@ -16,9 +16,9 @@ class SystemUserPermission(generics.ListAPIView):
def get_queryset(self):
user = self.request.user
asset_perms_id = get_user_all_asset_perm_ids(user)
queryset = SystemUser.objects.filter(
Q(granted_by_permissions__users=user) |
Q(granted_by_permissions__user_groups__users=user)
granted_by_permissions__id__in=asset_perms_id
).distinct()
return queryset

0
apps/perms/async_tasks/__init__.py
View File

47
apps/perms/async_tasks/mapping_node_task.py
View File

@@ -1,47 +0,0 @@
from django.utils.crypto import get_random_string
from perms.utils import rebuild_user_mapping_nodes_if_need_with_lock
from common.thread_pools import SingletonThreadPoolExecutor
from common.utils import get_logger
from perms.models import RebuildUserTreeTask
logger = get_logger(__name__)
class Executor(SingletonThreadPoolExecutor):
pass
executor = Executor()
def run_mapping_node_tasks():
failed_user_ids = []
ident = get_random_string()
logger.debug(f'[{ident}]mapping_node_tasks running')
while True:
task = RebuildUserTreeTask.objects.exclude(
user_id__in=failed_user_ids
).first()
if task is None:
break
user = task.user
try:
rebuild_user_mapping_nodes_if_need_with_lock(user)
except:
logger.exception(f'[{ident}]mapping_node_tasks_exception')
failed_user_ids.append(user.id)
logger.debug(f'[{ident}]mapping_node_tasks finished')
def submit_update_mapping_node_task():
executor.submit(run_mapping_node_tasks)
def submit_update_mapping_node_task_for_user(user):
executor.submit(rebuild_user_mapping_nodes_if_need_with_lock, user)

11
apps/perms/locks.py Normal file
View File

@@ -0,0 +1,11 @@
from common.utils.lock import DistributedLock
class UserGrantedTreeRebuildLock(DistributedLock):
name_template = 'perms.user.asset.node.tree.rebuid.<org_id:{org_id}>.<user_id:{user_id}>'
def __init__(self, org_id, user_id):
name = self.name_template.format(
org_id=org_id, user_id=user_id
)
super().__init__(name=name)

14
apps/perms/migrations/0014_build_users_perm_tree.py
View File

@@ -1,19 +1,6 @@
# Generated by Django 2.2.13 on 2020-08-21 08:20
from django.db import migrations
from perms.tasks import dispatch_mapping_node_tasks
def start_build_users_perm_tree_task(apps, schema_editor):
User = apps.get_model('users', 'User')
RebuildUserTreeTask = apps.get_model('perms', 'RebuildUserTreeTask')
user_ids = User.objects.all().values_list('id', flat=True).distinct()
RebuildUserTreeTask.objects.bulk_create(
[RebuildUserTreeTask(user_id=i) for i in user_ids]
)
dispatch_mapping_node_tasks.delay()
class Migration(migrations.Migration):
@@ -23,5 +10,4 @@ class Migration(migrations.Migration):
]
operations = [
migrations.RunPython(start_build_users_perm_tree_task)
]

65
apps/perms/migrations/0018_auto_20210204_1749.py Normal file
View File

@@ -0,0 +1,65 @@
# Generated by Django 3.1 on 2021-02-04 09:49
import assets.models.node
from django.conf import settings
from django.db import migrations, models
import django.db.models.deletion
class Migration(migrations.Migration):
dependencies = [
('assets', '0066_remove_node_assets_amount'),
migrations.swappable_dependency(settings.AUTH_USER_MODEL),
('perms', '0017_auto_20210104_0435'),
]
operations = [
migrations.CreateModel(
name='UserAssetGrantedTreeNodeRelation',
fields=[
('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
('created_by', models.CharField(blank=True, max_length=32, null=True, verbose_name='Created by')),
('updated_by', models.CharField(blank=True, max_length=32, null=True, verbose_name='Updated by')),
('date_created', models.DateTimeField(auto_now_add=True, null=True, verbose_name='Date created')),
('date_updated', models.DateTimeField(auto_now=True, verbose_name='Date updated')),
('org_id', models.CharField(blank=True, db_index=True, default='', max_length=36, verbose_name='Organization')),
('node_key', models.CharField(db_index=True, max_length=64, verbose_name='Key')),
('node_parent_key', models.CharField(db_index=True, default='', max_length=64, verbose_name='Parent key')),
('node_from', models.CharField(choices=[('granted', 'Direct node granted'), ('child', 'Have children node'), ('asset', 'Direct asset granted')], db_index=True, max_length=16)),
('node_assets_amount', models.IntegerField(default=0)),
('node', models.ForeignKey(db_constraint=False, default=None, on_delete=django.db.models.deletion.CASCADE, related_name='granted_node_rels', to='assets.node')),
('user', models.ForeignKey(db_constraint=False, on_delete=django.db.models.deletion.CASCADE, to=settings.AUTH_USER_MODEL)),
],
options={
'abstract': False,
},
bases=(assets.models.node.FamilyMixin, models.Model),
),
migrations.RemoveField(
model_name='usergrantedmappingnode',
name='node',
),
migrations.RemoveField(
model_name='usergrantedmappingnode',
name='user',
),
migrations.CreateModel(
name='PermNode',
fields=[
],
options={
'ordering': [],
'proxy': True,
'indexes': [],
'constraints': [],
},
bases=('assets.node',),
),
migrations.DeleteModel(
name='RebuildUserTreeTask',
),
migrations.DeleteModel(
name='UserGrantedMappingNode',
),
]

123
apps/perms/models/asset_permission.py
View File

@@ -2,7 +2,10 @@ import logging
from functools import reduce
from django.utils.translation import ugettext_lazy as _
from django.db.models import F
from common.db.models import ChoiceSet
from orgs.mixins.models import OrgModelMixin
from common.db import models
from common.utils import lazyproperty
from assets.models import Asset, SystemUser, Node, FamilyMixin
@@ -11,7 +14,7 @@ from .base import BasePermission
__all__ = [
'AssetPermission', 'Action', 'UserGrantedMappingNode', 'RebuildUserTreeTask',
'AssetPermission', 'Action', 'PermNode', 'UserAssetGrantedTreeNodeRelation',
]
# 使用场景
@@ -135,39 +138,109 @@ class AssetPermission(BasePermission):
from assets.models import Node
nodes_keys = self.nodes.all().values_list('key', flat=True)
assets_ids = set(self.assets.all().values_list('id', flat=True))
nodes_assets_ids = Node.get_nodes_all_assets_ids(nodes_keys)
nodes_assets_ids = Node.get_nodes_all_assets_ids_by_keys(nodes_keys)
assets_ids.update(nodes_assets_ids)
assets = Asset.objects.filter(id__in=assets_ids)
return assets
class UserAssetGrantedTreeNodeRelation(OrgModelMixin, FamilyMixin, models.JMSBaseModel):
class NodeFrom(ChoiceSet):
granted = 'granted', 'Direct node granted'
child = 'child', 'Have children node'
asset = 'asset', 'Direct asset granted'
class UserGrantedMappingNode(FamilyMixin, models.JMSBaseModel):
node = models.ForeignKey('assets.Node', default=None, on_delete=models.CASCADE,
db_constraint=False, null=True, related_name='mapping_nodes')
key = models.CharField(max_length=64, verbose_name=_("Key"), db_index=True) # '1:1:1:1'
user = models.ForeignKey('users.User', db_constraint=False, on_delete=models.CASCADE)
granted = models.BooleanField(default=False, db_index=True)
asset_granted = models.BooleanField(default=False, db_index=True)
parent_key = models.CharField(max_length=64, default='', verbose_name=_('Parent key'), db_index=True) # '1:1:1:1'
assets_amount = models.IntegerField(default=0)
node = models.ForeignKey('assets.Node', default=None, on_delete=models.CASCADE,
db_constraint=False, null=False, related_name='granted_node_rels')
node_key = models.CharField(max_length=64, verbose_name=_("Key"), db_index=True)
node_parent_key = models.CharField(max_length=64, default='', verbose_name=_('Parent key'), db_index=True)
node_from = models.CharField(choices=NodeFrom.choices, max_length=16, db_index=True)
node_assets_amount = models.IntegerField(default=0)
GRANTED_DIRECT = 1
GRANTED_INDIRECT = 2
GRANTED_NONE = 0
@property
def key(self):
return self.node_key
@property
def parent_key(self):
return self.node_parent_key
@classmethod
def get_node_granted_status(cls, key, user):
ancestor_keys = Node.get_node_ancestor_keys(key, with_self=True)
has_granted = UserGrantedMappingNode.objects.filter(
key__in=ancestor_keys, user=user
).values_list('granted', flat=True)
if not has_granted:
return cls.GRANTED_NONE
if any(list(has_granted)):
return cls.GRANTED_DIRECT
return cls.GRANTED_INDIRECT
def get_node_granted_status(cls, user, key):
ancestor_keys = set(cls.get_node_ancestor_keys(key, with_self=True))
ancestor_rel_nodes = cls.objects.filter(user=user, node_key__in=ancestor_keys)
for rel_node in ancestor_rel_nodes:
if rel_node.key == key:
return rel_node.node_from, rel_node
if rel_node.node_from == cls.NodeFrom.granted:
return cls.NodeFrom.granted, None
return '', None
class RebuildUserTreeTask(models.JMSBaseModel):
user = models.ForeignKey('users.User', on_delete=models.CASCADE, verbose_name=_('User'))
class PermNode(Node):
class Meta:
proxy = True
ordering = []
# 特殊节点
UNGROUPED_NODE_KEY = 'ungrouped'
UNGROUPED_NODE_VALUE = _('Ungrouped')
FAVORITE_NODE_KEY = 'favorite'
FAVORITE_NODE_VALUE = _('Favorite')
node_from = ''
granted_assets_amount = 0
# 提供可以设置 资产数量的字段
_assets_amount = None
annotate_granted_node_rel_fields = {
'granted_assets_amount': F('granted_node_rels__node_assets_amount'),
'node_from': F('granted_node_rels__node_from')
}
@property
def assets_amount(self):
_assets_amount = getattr(self, '_assets_amount')
if isinstance(_assets_amount, int):
return _assets_amount
return super().assets_amount
@assets_amount.setter
def assets_amount(self, value):
self._assets_amount = value
def use_granted_assets_amount(self):
self.assets_amount = self.granted_assets_amount
@classmethod
def get_ungrouped_node(cls, assets_amount):
return cls(
id=cls.UNGROUPED_NODE_KEY,
key=cls.UNGROUPED_NODE_KEY,
value=cls.UNGROUPED_NODE_VALUE,
assets_amount=assets_amount
)
@classmethod
def get_favorite_node(cls, assets_amount):
node = cls(
id=cls.FAVORITE_NODE_KEY,
key=cls.FAVORITE_NODE_KEY,
value=cls.FAVORITE_NODE_VALUE,
)
node.assets_amount = assets_amount
return node
def get_granted_status(self, user):
status, rel_node = UserAssetGrantedTreeNodeRelation.get_node_granted_status(user, self.key)
self.node_from = status
if rel_node:
self.granted_assets_amount = rel_node.node_assets_amount
return status
def save(self):
# 这是个只读 Model
raise NotImplementedError

38
apps/perms/pagination.py
View File

@@ -1,30 +1,54 @@
from rest_framework.pagination import LimitOffsetPagination
from rest_framework.request import Request
from django.db.models import Sum
from perms.models import UserAssetGrantedTreeNodeRelation
from common.utils import get_logger
logger = get_logger(__name__)
class GrantedAssetLimitOffsetPagination(LimitOffsetPagination):
class GrantedAssetPaginationBase(LimitOffsetPagination):
def paginate_queryset(self, queryset, request: Request, view=None):
self._request = request
self._view = view
self._user = request.user
return super().paginate_queryset(queryset, request, view=None)
def get_count(self, queryset):
exclude_query_params = {
self.limit_query_param,
self.offset_query_param,
'key', 'all', 'show_current_asset',
'cache_policy', 'display', 'draw'
'cache_policy', 'display', 'draw',
'order',
}
for k, v in self._request.query_params.items():
if k not in exclude_query_params and v is not None:
logger.warn(f'Not hit node.assets_amount because find a unknow query_param `{k}` -> {self._request.get_full_path()}')
return super().get_count(queryset)
return self.get_count_from_nodes(queryset)
def get_count_from_nodes(self, queryset):
raise NotImplementedError
class NodeGrantedAssetPagination(GrantedAssetPaginationBase):
def get_count_from_nodes(self, queryset):
node = getattr(self._view, 'pagination_node', None)
if node:
logger.debug(f'{self._request.get_full_path()} hit node.assets_amount[{node.assets_amount}]')
logger.debug(f'Hit node.assets_amount[{node.assets_amount}] -> {self._request.get_full_path()}')
return node.assets_amount
else:
logger.warn(f'Not hit node.assets_amount[{node}] because {self._view} not has `pagination_node` -> {self._request.get_full_path()}')
return super().get_count(queryset)
def paginate_queryset(self, queryset, request: Request, view=None):
self._request = request
self._view = view
return super().paginate_queryset(queryset, request, view=None)
class AllGrantedAssetPagination(GrantedAssetPaginationBase):
def get_count_from_nodes(self, queryset):
assets_amount = sum(UserAssetGrantedTreeNodeRelation.objects.filter(
user=self._user, node_parent_key=''
).values_list('node_assets_amount', flat=True))
logger.debug(f'Hit all assets amount {assets_amount} -> {self._request.get_full_path()}')
return assets_amount

2
apps/perms/signals_handler/__init__.py Normal file
View File

@@ -0,0 +1,2 @@
from . import common
from . import refresh_perms

88
apps/perms/signals_handler.py → apps/perms/signals_handler/common.py
View File

@@ -1,31 +1,22 @@
# -*- coding: utf-8 -*-
#
from django.db.models.signals import m2m_changed, pre_delete, pre_save
from django.db.models.signals import m2m_changed
from django.dispatch import receiver
from perms.tasks import create_rebuild_user_tree_task, \
create_rebuild_user_tree_task_by_related_nodes_or_assets
from users.models import User, UserGroup
from assets.models import Asset, SystemUser
from assets.models import SystemUser
from applications.models import Application
from common.utils import get_logger
from common.exceptions import M2MReverseNotAllowed
from common.const.signals import POST_ADD, POST_REMOVE, POST_CLEAR
from .models import AssetPermission, ApplicationPermission
from common.const.signals import POST_ADD
from perms.models import AssetPermission, ApplicationPermission
logger = get_logger(__file__)
def handle_rebuild_user_tree(instance, action, reverse, pk_set, **kwargs):
if action.startswith('post'):
if reverse:
create_rebuild_user_tree_task(pk_set)
else:
create_rebuild_user_tree_task([instance.id])
def handle_bind_groups_systemuser(instance, action, reverse, pk_set, **kwargs):
@receiver(m2m_changed, sender=User.groups.through)
def on_user_groups_change(sender, instance, action, reverse, pk_set, **kwargs):
"""
UserGroup 增加 User 时，增加的 User 需要与 UserGroup 关联的动态系统用户相关联
"""
@@ -47,53 +38,11 @@ def handle_bind_groups_systemuser(instance, action, reverse, pk_set, **kwargs):
system_user.users.add(*users_id)
@receiver(m2m_changed, sender=User.groups.through)
def on_user_groups_change(**kwargs):
handle_rebuild_user_tree(**kwargs)
handle_bind_groups_systemuser(**kwargs)
@receiver([pre_save], sender=AssetPermission)
def on_asset_perm_deactive(instance: AssetPermission, **kwargs):
try:
old = AssetPermission.objects.only('is_active').get(id=instance.id)
if instance.is_active != old.is_active:
create_rebuild_user_tree_task_by_asset_perm(instance)
except AssetPermission.DoesNotExist:
pass
@receiver([pre_delete], sender=AssetPermission)
def on_asset_permission_delete(instance, **kwargs):
# 授权删除之前，查出所有相关用户
create_rebuild_user_tree_task_by_asset_perm(instance)
def create_rebuild_user_tree_task_by_asset_perm(asset_perm: AssetPermission):
user_ids = set()
user_ids.update(
UserGroup.objects.filter(
assetpermissions=asset_perm, users__id__isnull=False
).distinct().values_list('users__id', flat=True)
)
user_ids.update(
User.objects.filter(assetpermissions=asset_perm).distinct().values_list('id', flat=True)
)
create_rebuild_user_tree_task(user_ids)
def need_rebuild_mapping_node(action):
return action in (POST_REMOVE, POST_ADD, POST_CLEAR)
@receiver(m2m_changed, sender=AssetPermission.nodes.through)
def on_permission_nodes_changed(instance, action, reverse, pk_set, model, **kwargs):
if reverse:
raise M2MReverseNotAllowed
if need_rebuild_mapping_node(action):
create_rebuild_user_tree_task_by_asset_perm(instance)
if action != POST_ADD:
return
logger.debug("Asset permission nodes change signal received")
@@ -110,9 +59,6 @@ def on_permission_assets_changed(instance, action, reverse, pk_set, model, **kwa
if reverse:
raise M2MReverseNotAllowed
if need_rebuild_mapping_node(action):
create_rebuild_user_tree_task_by_asset_perm(instance)
if action != POST_ADD:
return
logger.debug("Asset permission assets change signal received")
@@ -150,9 +96,6 @@ def on_asset_permission_users_changed(instance, action, reverse, pk_set, model,
if reverse:
raise M2MReverseNotAllowed
if need_rebuild_mapping_node(action):
create_rebuild_user_tree_task(pk_set)
if action != POST_ADD:
return
logger.debug("Asset permission users change signal received")
@@ -171,10 +114,6 @@ def on_asset_permission_user_groups_changed(instance, action, pk_set, model,
if reverse:
raise M2MReverseNotAllowed
if need_rebuild_mapping_node(action):
user_ids = User.objects.filter(groups__id__in=pk_set).distinct().values_list('id', flat=True)
create_rebuild_user_tree_task(user_ids)
if action != POST_ADD:
return
logger.debug("Asset permission user groups change signal received")
@@ -187,21 +126,6 @@ def on_asset_permission_user_groups_changed(instance, action, pk_set, model,
system_user.groups.add(*tuple(groups))
@receiver(m2m_changed, sender=Asset.nodes.through)
def on_node_asset_change(action, instance, reverse, pk_set, **kwargs):
if not need_rebuild_mapping_node(action):
return
if reverse:
asset_pk_set = pk_set
node_pk_set = [instance.id]
else:
asset_pk_set = [instance.id]
node_pk_set = pk_set
create_rebuild_user_tree_task_by_related_nodes_or_assets.delay(node_pk_set, asset_pk_set)
@receiver(m2m_changed, sender=ApplicationPermission.system_users.through)
def on_application_permission_system_users_changed(sender, instance: ApplicationPermission, action, reverse, pk_set, **kwargs):
if not instance.category_remote_app:

115
apps/perms/signals_handler/refresh_perms.py Normal file
View File

@@ -0,0 +1,115 @@
# -*- coding: utf-8 -*-
#
from django.db.models.signals import m2m_changed, pre_delete, pre_save, post_save
from django.dispatch import receiver
from users.models import User
from assets.models import Asset
from orgs.utils import current_org
from common.utils import get_logger
from common.exceptions import M2MReverseNotAllowed
from common.const.signals import POST_ADD, POST_REMOVE, POST_CLEAR
from perms.models import AssetPermission
from perms.utils.asset.user_permission import UserGrantedTreeRefreshController
logger = get_logger(__file__)
@receiver(m2m_changed, sender=User.groups.through)
def on_user_groups_change(sender, instance, action, reverse, pk_set, **kwargs):
if action.startswith('post'):
if reverse:
group_ids = [instance.id]
user_ids = pk_set
else:
group_ids = pk_set
user_ids = [instance.id]
exists = AssetPermission.user_groups.through.objects.filter(usergroup_id__in=group_ids).exists()
if exists:
org_ids = [current_org.id]
UserGrantedTreeRefreshController.add_need_refresh_orgs_for_users(org_ids, user_ids)
@receiver([pre_delete], sender=AssetPermission)
def on_asset_perm_pre_delete(sender, instance, **kwargs):
# 授权删除之前，查出所有相关用户
UserGrantedTreeRefreshController.add_need_refresh_by_asset_perm_ids([instance.id])
@receiver([pre_save], sender=AssetPermission)
def on_asset_perm_pre_save(sender, instance, **kwargs):
try:
old = AssetPermission.objects.get(id=instance.id)
if old.is_valid != instance.is_valid:
UserGrantedTreeRefreshController.add_need_refresh_by_asset_perm_ids([instance.id])
except AssetPermission.DoesNotExist:
pass
@receiver([post_save], sender=AssetPermission)
def on_asset_perm_post_save(sender, instance, created, **kwargs):
if created:
UserGrantedTreeRefreshController.add_need_refresh_by_asset_perm_ids([instance.id])
def need_rebuild_mapping_node(action):
return action in (POST_REMOVE, POST_ADD, POST_CLEAR)
@receiver(m2m_changed, sender=AssetPermission.nodes.through)
def on_permission_nodes_changed(sender, instance, action, reverse, **kwargs):
if reverse:
raise M2MReverseNotAllowed
if need_rebuild_mapping_node(action):
UserGrantedTreeRefreshController.add_need_refresh_by_asset_perm_ids([instance.id])
@receiver(m2m_changed, sender=AssetPermission.assets.through)
def on_permission_assets_changed(sender, instance, action, reverse, pk_set, model, **kwargs):
if reverse:
raise M2MReverseNotAllowed
if need_rebuild_mapping_node(action):
UserGrantedTreeRefreshController.add_need_refresh_by_asset_perm_ids([instance.id])
@receiver(m2m_changed, sender=AssetPermission.users.through)
def on_asset_permission_users_changed(sender, action, reverse, pk_set, **kwargs):
if reverse:
raise M2MReverseNotAllowed
if need_rebuild_mapping_node(action):
UserGrantedTreeRefreshController.add_need_refresh_orgs_for_users(
[current_org.id], pk_set
)
@receiver(m2m_changed, sender=AssetPermission.user_groups.through)
def on_asset_permission_user_groups_changed(sender, action, pk_set, reverse, **kwargs):
if reverse:
raise M2MReverseNotAllowed
if need_rebuild_mapping_node(action):
user_ids = User.groups.through.objects.filter(usergroup_id__in=pk_set).distinct().values_list('user_id', flat=True)
UserGrantedTreeRefreshController.add_need_refresh_orgs_for_users(
[current_org.id], user_ids
)
@receiver(m2m_changed, sender=Asset.nodes.through)
def on_node_asset_change(action, instance, reverse, pk_set, **kwargs):
if not need_rebuild_mapping_node(action):
return
if reverse:
asset_pk_set = pk_set
node_pk_set = [instance.id]
else:
asset_pk_set = [instance.id]
node_pk_set = pk_set
UserGrantedTreeRefreshController.add_need_refresh_on_nodes_assets_relate_change(node_pk_set, asset_pk_set)

90
apps/perms/tasks.py
View File

@@ -2,39 +2,18 @@
from __future__ import absolute_import, unicode_literals
from datetime import timedelta
from django.db import transaction
from django.db.models import Q
from django.db.transaction import atomic
from django.conf import settings
from celery import shared_task
from common.utils import get_logger
from common.utils.timezone import now, dt_formater, dt_parser
from users.models import User
from ops.celery.decorator import register_as_period_task
from assets.models import Node
from perms.models import RebuildUserTreeTask, AssetPermission
from perms.utils.asset.user_permission import rebuild_user_mapping_nodes_if_need_with_lock, lock
from perms.models import AssetPermission
from perms.utils.asset.user_permission import UserGrantedTreeRefreshController
logger = get_logger(__file__)
@shared_task(queue='node_tree')
def rebuild_user_mapping_nodes_celery_task(user_id):
user = User.objects.get(id=user_id)
try:
rebuild_user_mapping_nodes_if_need_with_lock(user)
except lock.SomeoneIsDoingThis:
pass
@shared_task(queue='node_tree')
def dispatch_mapping_node_tasks():
user_ids = RebuildUserTreeTask.objects.all().values_list('user_id', flat=True).distinct()
logger.info(f'>>> dispatch_mapping_node_tasks for users {list(user_ids)}')
for id in user_ids:
rebuild_user_mapping_nodes_celery_task.delay(id)
@register_as_period_task(interval=settings.PERM_EXPIRED_CHECK_PERIODIC)
@shared_task(queue='celery_check_asset_perm_expired')
@atomic()
@@ -60,66 +39,9 @@ def check_asset_permission_expired():
setting.value = dt_formater(end)
setting.save()
ids = AssetPermission.objects.filter(
asset_perm_ids = AssetPermission.objects.filter(
date_expired__gte=start, date_expired__lte=end
).distinct().values_list('id', flat=True)
logger.info(f'>>> checking {start} to {end} have {ids} expired')
dispatch_process_expired_asset_permission.delay(list(ids))
@shared_task(queue='node_tree')
def dispatch_process_expired_asset_permission(asset_perms_id):
user_ids = User.objects.filter(
Q(assetpermissions__id__in=asset_perms_id) |
Q(groups__assetpermissions__id__in=asset_perms_id)
).distinct().values_list('id', flat=True)
RebuildUserTreeTask.objects.bulk_create(
[RebuildUserTreeTask(user_id=user_id) for user_id in user_ids]
)
dispatch_mapping_node_tasks.delay()
def create_rebuild_user_tree_task(user_ids):
RebuildUserTreeTask.objects.bulk_create(
[RebuildUserTreeTask(user_id=i) for i in user_ids]
)
transaction.on_commit(dispatch_mapping_node_tasks.delay)
@shared_task(queue='node_tree')
def create_rebuild_user_tree_task_by_related_nodes_or_assets(node_ids, asset_ids):
node_ids = set(node_ids)
node_keys = set()
nodes = Node.objects.filter(id__in=node_ids)
for _node in nodes:
node_keys.update(_node.get_ancestor_keys())
node_ids.update(
Node.objects.filter(key__in=node_keys).values_list('id', flat=True)
)
asset_perms_id = set()
asset_perms_id.update(
AssetPermission.objects.filter(
assets__id__in=asset_ids
).values_list('id', flat=True).distinct()
)
asset_perms_id.update(
AssetPermission.objects.filter(
nodes__id__in=node_ids
).values_list('id', flat=True).distinct()
)
user_ids = set()
user_ids.update(
User.objects.filter(
assetpermissions__id__in=asset_perms_id
).distinct().values_list('id', flat=True)
)
user_ids.update(
User.objects.filter(
groups__assetpermissions__id__in=asset_perms_id
).distinct().values_list('id', flat=True)
)
create_rebuild_user_tree_task(user_ids)
asset_perm_ids = list(asset_perm_ids)
logger.info(f'>>> checking {start} to {end} have {asset_perm_ids} expired')
UserGrantedTreeRefreshController.add_need_refresh_by_asset_perm_ids_cross_orgs(asset_perm_ids)

1122
apps/perms/utils/asset/user_permission.py
View File

File diff suppressed because it is too large Load Diff