Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

智能扫描重构遗留问题处理-Phase 1 #2523

Closed
8 tasks done
iwanghc opened this issue Aug 6, 2024 · 9 comments
Closed
8 tasks done

智能扫描重构遗留问题处理-Phase 1 #2523

iwanghc opened this issue Aug 6, 2024 · 9 comments
Assignees
@taolx0 @iwanghc
Labels
bug Something isn't working need_update_doc 需要修改用户手册 verified the issue all pr is verified
Milestone
v3.2408.0

Comments

@iwanghc
Copy link
Collaborator

iwanghc commented Aug 6, 2024 •
edited
Loading

原始issue：https://github.com/actiontech/sqle-ee/issues/1576

需求描述（Describe）

  • 扫描任务跑增量同步时，获取到不需要同步的任务问题修复
  • mysql慢日志采集时，start_time字段未保存问题修复
  • mysql慢日志新增“启动任务时拉取慢日志时间范围”参数
  • 智能扫描自定义参数间的联动
  • TBase数据源扫描任务概览新增连接信息字段 @taolx0
  • 扫描任务详情表格数据加载异常 @LZS911
  • 智能扫描配置以及概览表格操作列按钮前端权限控制 @LZS911
  • 创建智能扫描任务时数据源获取需要根据权限进行筛选 @LZS911
@ColdWaterLW ColdWaterLW assigned iwanghc and unassigned ColdWaterLW Aug 6, 2024
@ColdWaterLW ColdWaterLW added this to the v3.2408.0 milestone Aug 6, 2024
This was referenced Aug 6, 2024
Merged
Merged
@taolx0 taolx0 assigned taolx0 and iwanghc and unassigned iwanghc Aug 7, 2024
@taolx0 taolx0 mentioned this issue Aug 7, 2024
Merged
5 tasks
@taolx0 taolx0 added not_compatible not compatible old version need_update_doc 需要修改用户手册 labels Aug 7, 2024
@taolx0
Copy link
Collaborator

taolx0 commented Aug 7, 2024 •
edited
Loading

TBase数据源扫描任务概览新增连接信息字段

方案

开放TBase连接信息分类

影响面

scanner上传命令行发生变更,需要更新scanner

@iwanghc iwanghc mentioned this issue Aug 12, 2024
Merged
5 tasks
@iwanghc
Copy link
Collaborator Author

iwanghc commented Aug 12, 2024 •
edited by LordofAvernus
Loading

验证Oracle TOP SQL任务问题

  • 创建oracle扫描任务时获取不到TOP SQL任务类型
    oracle扫描任务获取的instance type错误

验证DM TOP SQL任务问题

  • DM智能扫描任务TOP SQL采集sql失败
    修复达梦采集sql时，获取不到实例

@iwanghc iwanghc mentioned this issue Aug 13, 2024
Merged
5 tasks
@iwanghc
Copy link
Collaborator Author

iwanghc commented Aug 13, 2024

验证SQL管控的问题

sql管控获取sql分析时报错
image

@iwanghc iwanghc mentioned this issue Aug 15, 2024
Merged
5 tasks
@iwanghc
Copy link
Collaborator Author

iwanghc commented Aug 19, 2024 •
edited
Loading

智能扫描动态参数联动问题方案

背景：

慢日志类型参数“采集周期”、“启动任务时拉取慢日志时间范围”仅对 mysql.slow_log 表有效，所以需要在当采集来源选择slow.log文件采集时隐藏这两个输入框。

方案一：

后端修改GetAuditPlanMetas接口层的ParamsRes参数结构，调整ParamsRes为父子结构，或ParamsRes新增字段依赖需要联动的key与value
优点：返回的结构清晰
缺点：不易做成通用，有一定局限性，多参数依赖的话只能处理与的联动关系。
影响面：较小，如果仅对mysql慢日志参数特殊处理的话不影响其他扫描任务类型。

方案二：

调整params包Param结构体，新增字段依赖需要联动的key与value（接口层也同步新增字段）
优点：可以做成通用
缺点：有一定局限性，多参数依赖的话只能处理与的联动关系。

方案三：

前端对mysql慢日志的参数做特殊处理
优点：成本小，后端不需要调整
缺点：处理后续新增的扫描任务类型或以后其他需要联动的参数
影响面：不影响其他
image

最终方案

前端mysql慢日志的参数做特殊处理
原因：后端处理通用性不强，后续若出现稍复杂的依赖关系不易处理，目前仅有mysql慢日志参数需联动，且成本不高，所以由前端处理（隐藏参数也需要提交，因后端有对参数数量有做校验，后端需补充参数默认值，以默认值提交即可）

@iwanghc
Copy link
Collaborator Author

iwanghc commented Aug 19, 2024

扫描任务操作按钮根据权限显示和隐藏

1、创建编辑页面，可选数据源根据权限显隐【前端】
2、页面增删改和立即审核按钮根据权限显隐【前端】
3、实例扫描任务接口返回instance id，支持前端获取权限【后端】

@iwanghc iwanghc mentioned this issue Aug 19, 2024
Merged
5 tasks
@LZS911
Copy link
Collaborator

LZS911 commented Aug 20, 2024

扫描任务详情表格数据加载异常

例：
image
此时为初次通过SQL优先级筛选，数据正常。

清楚筛选后再次筛选，表格数据出现异常
image
可以明显发现表格数量多于接口返回的数据。

@LZS911 LZS911 mentioned this issue Aug 20, 2024
Merged
@iwanghc iwanghc mentioned this issue Aug 21, 2024
Merged
5 tasks
@LZS911 LZS911 mentioned this issue Aug 21, 2024
Merged
@iwanghc iwanghc mentioned this issue Aug 22, 2024
Closed
5 tasks
@iwanghc
Copy link
Collaborator Author

iwanghc commented Aug 23, 2024

关联Comment：#2523 (comment)
实例扫描任务列表instance_id出现精度丢失
解决：返回转成string返给前端

@iwanghc iwanghc mentioned this issue Aug 23, 2024
Merged
5 tasks
@LZS911 LZS911 mentioned this issue Aug 23, 2024
Merged
@actiontech actiontech deleted a comment from LordofAvernus Aug 30, 2024
@iwanghc iwanghc closed this as completed Aug 30, 2024
@iwanghc
Copy link
Collaborator Author

iwanghc commented Aug 30, 2024

关联遗留问题：#2572

@iwanghc iwanghc mentioned this issue Aug 30, 2024
Open
12 tasks
@iwanghc iwanghc changed the title 智能扫描重构遗留问题处理 智能扫描重构遗留问题处理v1 Aug 30, 2024
@iwanghc iwanghc changed the title 智能扫描重构遗留问题处理v1 智能扫描重构遗留问题处理-Phase 1 Aug 30, 2024
@iwanghc iwanghc added the bug Something isn't working label Aug 30, 2024
@hasa1K
Copy link
Contributor

hasa1K commented Aug 30, 2024 •
edited
Loading

测试

mysql慢日志采集时，start_time字段未保存问题修复
image
start_time字段有值，符合预期
扫描任务没有增量同步
image

扫描任务跑增量同步时，获取到不需要同步的任务问题修复
停用慢日志扫描任务
image

数据库中新增慢日志
image

sql_manage_records表没有新增慢日志
image

TBase数据源扫描任务概览新增连接信息字段
新增TBase慢日志扫描任务
image
新增连接信息
image

智能扫描自定义参数间的联动
mysql慢日志选择scanner时，没有其他选择
image

mysql慢日志选择mysql.slow_log时，出现采集周期以及采集时间选项
image

扫描任务详情表格数据加载异常
初始时
image
筛选高优先级sql，sql数量少了一条
image
清除筛选选项后，再进行筛选
image
没有出现数据异常现实的问题，符合预期

验证创建智能扫描任务时数据源获取需要根据权限进行筛选
配置成员与权限
image
image
无法看到他人创建的扫描任务
image
只能为对应数据源创建扫描任务
image
创建成功
image

为user2用户配置查看他人创建的智能扫描任务权限
image
image
可以看到其他人创建的智能扫描
image
没有创建扫描任务按钮
image

一开始只拉取24h内的慢日志
img_v3_02eb_d3c73d3e-2faf-44d6-b766-6454012231bg
24h内的慢日志是空的
img_v3_02eb_c4fd2961-7c36-48dc-8c38-796cfa68be8g
修改这个参数，拉取2400个小时内的慢日志
img_v3_02eb_5e2c719b-0285-4e0d-a88d-1479d546598g
成功采集到数据
image

@hasa1K hasa1K added the verified the issue all pr is verified label Sep 2, 2024
@LordofAvernus LordofAvernus removed the not_compatible not compatible old version label Sep 3, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@taolx0 taolx0

@iwanghc iwanghc

Labels
bug Something isn't working need_update_doc 需要修改用户手册 verified the issue all pr is verified
Projects
None yet
Milestone
v3.2408.0
Development

No branches or pull requests
6 participants
@LordofAvernus @ColdWaterLW @LZS911 @hasa1K @taolx0 @iwanghc