feat: support npm search command like npmio #513

Beace · 2023-06-13T16:41:18Z

找个合适的 eggjs es 插件，或者手撸个，看社区的几个版本都比较低
HTTP Server 新增 API /-/v1/search?text=react&size=20&from=0&quality=0.65&popularity=0.98&maintenance=0.5，第一版不一定能 qpm 都支持，先支持现有的下载量数据，即 popularity 的参考数据
监听相关的 metadata 变更的 event，同步写入、删除增量的 ES 数据，ES 有较强抗压能力，这块直接做成同步就好
考虑可能同步也会丢部分数据（stream 不稳定时），HTTP Server 再追加一个手动同步 ES 的接口，传包名同步触发写 ES 即可
提供全量一次性的初始化同步 ES 脚本
setting/mapping 参考 https://github.com/npms-io/npms-analyzer/blob/master/config/elasticsearch/npms.json5

Beace · 2023-06-13T16:47:48Z

预期实现效果

elrrrrrrr · 2023-06-14T01:56:35Z

找个合适的 eggjs es 插件，或者手撸个，看社区的几个版本都比较低
插件需要实现什么功能呀？

需要添加一个配置项，默认不开启。

另外可能还需要类似 Auth，提供一个 adapter ，便于企业内自定义实现。

fengmk2 · 2023-06-15T01:35:50Z

https://registry.npmjs.com/-/v1/search?text=react&size=20&from=0&quality=0.65&popularity=0.98&maintenance=0.5

https://github.com/npm/registry/blob/master/docs/REGISTRY-API.md#get-v1search
search api 有一个官方接口文档

Beace · 2023-06-17T12:04:40Z

https://github.com/Beace/eggjs-elasticsearch 初步完成 egg es插件，采用 es8，最新版本

Beace · 2023-06-28T12:44:21Z

实测插件可用

import { Client as ElasticsearchClient } from '@elastic/elasticsearch';

class A {
  @Inject()
  private readonly elasticsearch: ElasticsearchClient;
}

socket-security · 2023-06-28T12:45:56Z

New dependencies detected. Learn more about Socket for GitHub ↗︎

Packages	Version	New capabilities	Transitives	Size	Publisher
eggjs-elasticsearch	0.0.6	None	`+4`	2.83 MB	beace
@elastic/elasticsearch-mock	2.0.0	eval, network	`+8`	472 kB	delvedor
@elastic/elasticsearch	8.9.0	None	`+3`	2.83 MB	joshmock

codecov · 2023-06-28T12:49:01Z

Codecov Report

Merging #513 (c6ff174) into master (10d7a84) will decrease coverage by 0.12%.
Report is 1 commits behind head on master.
The diff coverage is 95.14%.

@@            Coverage Diff             @@
##           master     #513      +/-   ##
==========================================
- Coverage   97.05%   96.93%   -0.12%     
==========================================
  Files         175      180       +5     
  Lines       16792    17344     +552     
  Branches     2212     2249      +37     
==========================================
+ Hits        16297    16813     +516     
- Misses        495      531      +36

Files Changed	Coverage Δ
config/config.default.ts	`84.61% <33.33%> (-3.99%)`	⬇️
app/core/service/PackageSearchService.ts	`93.51% <93.51%> (ø)`
app/core/event/SyncESPackage.ts	`96.80% <96.80%> (ø)`
app/common/PackageUtil.ts	`97.85% <100.00%> (+0.23%)`	⬆️
app/common/typing.ts	`100.00% <100.00%> (ø)`
app/core/service/PackageSyncerService.ts	`98.21% <100.00%> (+<0.01%)`	⬆️
app/infra/SearchAdapter.ts	`100.00% <100.00%> (ø)`
app/port/config.ts	`100.00% <100.00%> (ø)`
...port/controller/package/SearchPackageController.ts	`100.00% <100.00%> (ø)`
app/port/typebox.ts	`98.96% <100.00%> (+0.06%)`	⬆️
... and 4 more

... and 2 files with indirect coverage changes

fengmk2 · 2023-07-23T14:53:33Z

@Beace 继续搞完？写一下使用文档，我加一个 es 服务。

Beace · 2023-07-23T14:54:48Z

@Beace 继续搞完？写一下使用文档，我加一个 es 服务。

就这两天会搞完

Beace · 2023-08-02T16:53:15Z

还差点，周末赶完...

fengmk2 · 2023-08-14T02:34:42Z

@Beace 加油，新版 web 就等这个搜索能力了。

Beace · 2023-08-14T03:24:07Z

快了快了

Beace · 2023-08-16T13:17:03Z

已达可用状态，待补充单测

app/port/controller/package/SearchPackageController.ts

app/common/PackageUtil.ts

config/config.default.ts

app/port/controller/package/SearchPackageController.ts

fengmk2 · 2023-08-30T15:21:53Z

@Beace 有个反馈，修复一下我就可以部署了。es 已经申请好了。

Beace · 2023-08-30T15:23:18Z

@Beace 有个反馈，修复一下我就可以部署了。es 已经申请好了。

稍等我半个小时，还在路上

fengmk2

+1 单测修复一下

Beace · 2023-08-30T17:12:15Z

+1 单测修复一下

单侧好诡异，没看出来

+1 单测修复一下

有点迷，@tegg/tegg-plugin 更新啥了，把 getEggObject 类型更新没了

Beace · 2023-08-30T18:38:01Z

master 最新分支上的 tsc 都跪了....

fengmk2 · 2023-08-31T00:37:18Z

@Beace 我来看看

fengmk2 · 2023-08-31T00:41:13Z

ets 挂了

fengmk2 · 2023-08-31T00:50:27Z

#578

fengmk2 · 2023-08-31T01:22:18Z

@Beace 合并 master 了，有类型定义错误

[skip ci] ## [3.43.0](v3.42.2...v3.43.0) (2023-09-01) ### Features * support npm search command like npmio ([#513](#513)) ([7f85848](7f85848))

fengmk2 · 2023-09-01T02:20:27Z

r.cnpmjs.org 验证成功

npm search colors --registry http://localhost:7001
NAME                      | DESCRIPTION          | AUTHOR          | DATE       | VERSION  | KEYWORDS
colors                    | get colors in your…  | =undefined      | 2019-09-22 | 1.4.0    |

fengmk2 · 2023-09-01T02:20:50Z

https://r.cnpmjs.org/-/v1/search?text=colors

fengmk2 · 2023-09-01T02:24:33Z

author 识别有问题

npm search npm --registry http://localhost:7001
NAME                      | DESCRIPTION          | AUTHOR          | DATE       | VERSION  | KEYWORDS                                    
npm                       | a package manager…   | =undefined…     | 2023-08-31 | 10.0.0   | install modules package manager package.json
renovate                  | Automated…           | =undefined…     | 2023-09-01 | 36.78.4  | automated azure bazel bitbucket buildkite dependencies dependency docker gitea github gitlab
@pulumi/kubernetes        | [![Build…            | =undefined…     | 2023-08-23 | 4.1.1    | pulumi kubernetes category/cloud kind/native
@xlunar/uni-view          | ## Project setup…    | =undefined      | 2023-09-01 | 0.1.0    |

fengmk2 · 2023-09-01T08:52:05Z

npmmirror 也开启了

npm search antd --registry=https://registry.npmmirror.com
NAME                      | DESCRIPTION          | AUTHOR          | DATE       | VERSION  | KEYWORDS                                   
@orca-fe/antd-plus        | Transformer…         | =nicokam…       | 2023-09-01 | 0.3.3    | react antd antd-plus orca-fe orca orca-team
@ecoding/components.antd  | > TODO: description  | =w3cwhy         | 2023-09-01 | 0.3.0    | 
zat-design-pro-component  | 基于 antd@4 版本开发 | =wanchongyang…  | 2023-09-01 | 3.0.20-… | 
@magicbe/design           | React + Antd Drag…   | =teacher-wang   | 2023-09-01 | 0.0.49   | Component UI React
@qiankun-fix/element-ui   | @qiankun-fix/antd…   | =robinwm        | 2023-09-01 | 0.7.8    | qiankun 微前端 element-UI vue
@magicbe/render           | React + Antd render… | =teacher-wang   | 2023-09-01 | 0.0.23   | Component UI React
@hbwow/custom-suspense    | ## 1.x 版本基于…     | =hbwow          | 2023-09-01 | 0.0.3    | component frontend react react-component

@Beace

### 改动原因在今年 4 月份，npm cli 对 search 做了[一些改动]，在 4 月份以前(npm cli < `v10.6.0`)，采用的是 `maintainers` 字段，而在 npm/cli#7407 后，改为使用: `publisher` 字段。导致当 npm 版本大于等于 `v10.6.0` 后，search 结果中的 `author` 将变成 `???`，如图： ![CleanShot 2024-12-18 at 16 46 58@2x](https://github.com/user-attachments/assets/4b97bf63-78cb-4720-8c00-469eeff3e92f) 预期的结果应该为: ![CleanShot 2024-12-18 at 16 47 33@2x](https://github.com/user-attachments/assets/2d7fca89-4383-42bd-8b83-66257efe65e8) ### 技术细节说明当前改动没有为 `es` 增加新的索引，原因是处于以下考虑: 1. es 的 `mapping` 一旦创建，就无法修改（虽然使用了 `dynamic: true` 但无法细粒度的进行控制） 2. 源数据中的 `_npmUser` 已经有相关信息了，没有必要为此浪费额外的磁盘空间 3. 如果想对以前的数据进行更新会比较麻烦，性价比较低 npm cli 老版本: https://github.com/npm/cli/pull/7407/files#diff-4bc15933c685fc9a9ce8be0c13a2f067f5e2b3334bacd6664bdfa7ddc46aedb6L58 npm cli 新版本: https://github.com/npm/cli/pull/7407/files#diff-4bc15933c685fc9a9ce8be0c13a2f067f5e2b3334bacd6664bdfa7ddc46aedb6R162 ### 其他相关 PR: #513 PTAL @Beace @fengmk2 @elrrrrrrr [一些改动]: npm/cli#7407 Signed-off-by: Kevin Cui <bh@bugs.cc>

Beace changed the title ~~feat: support npm search command like npmio~~ WIP feat: support npm search command like npmio Jun 13, 2023

Beace changed the title ~~WIP feat: support npm search command like npmio~~ [WIP] feat: support npm search command like npmio Jun 13, 2023

Beace added the enhancement New feature or request label Jun 13, 2023

Beace self-assigned this Jun 13, 2023

fengmk2 requested review from fengmk2 and elrrrrrrr June 15, 2023 01:32

feat: support npm search command like npmio

dadd39d

Beace force-pushed the feat-search branch from 7d534dd to dadd39d Compare June 28, 2023 12:43

fengmk2 linked an issue Jul 23, 2023 that may be closed by this pull request

提供搜索接口: GET·/-/v1/search #510

Closed

Beace added 2 commits August 3, 2023 00:32

feat: add search service

94d1c58

feat: add es search service and documents

e7c2977

feat: finished the search api and config doc

dbe774a

Beace force-pushed the feat-search branch from 12d843a to dbe774a Compare August 16, 2023 13:16

Merge branch 'master' into feat-search

d13fc6e

Beace changed the title ~~[WIP] feat: support npm search command like npmio~~ feat: support npm search command like npmio Aug 17, 2023

elrrrrrrr reviewed Aug 22, 2023

View reviewed changes

app/port/controller/package/SearchPackageController.ts Outdated Show resolved Hide resolved

elrrrrrrr reviewed Aug 22, 2023

View reviewed changes

app/common/PackageUtil.ts Show resolved Hide resolved

fengmk2 approved these changes Aug 30, 2023

View reviewed changes

fengmk2 reviewed Aug 30, 2023

View reviewed changes

config/config.default.ts Outdated Show resolved Hide resolved

fengmk2 reviewed Aug 30, 2023

View reviewed changes

app/port/controller/package/SearchPackageController.ts Show resolved Hide resolved

Beace added 3 commits August 30, 2023 23:41

fix: fix json typo and es config use env

4440c38

feat: es delete add admin access

65757b6

Merge branch 'master' into feat-search

6da659b

fengmk2 approved these changes Aug 30, 2023

View reviewed changes

fengmk2 linked an issue Aug 30, 2023 that may be closed by this pull request

搜索可以看看 Search as a Service #266

Closed

fengmk2 mentioned this pull request Aug 31, 2023

ets 运行挂了 eggjs/egg#5256

Closed

Merge branch 'master' into feat-search

f6d0cf0

elrrrrrrr force-pushed the feat-search branch from b67d944 to 11d9bca Compare August 31, 2023 17:58

fix: ci

c6ff174

elrrrrrrr force-pushed the feat-search branch from 11d9bca to c6ff174 Compare August 31, 2023 18:00

elrrrrrrr requested a review from fengmk2 August 31, 2023 18:42

fengmk2 merged commit 7f85848 into cnpm:master Sep 1, 2023

fengmk2 pushed a commit that referenced this pull request Sep 1, 2023

Release 3.43.0

941b277

[skip ci] ## [3.43.0](v3.42.2...v3.43.0) (2023-09-01) ### Features * support npm search command like npmio ([#513](#513)) ([7f85848](7f85848))

BlackHole1 mentioned this pull request Dec 18, 2024

fix(search): author is ??? in result #741

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: support npm search command like npmio #513

feat: support npm search command like npmio #513

Beace commented Jun 13, 2023 •

edited

Loading

Beace commented Jun 13, 2023

elrrrrrrr commented Jun 14, 2023

fengmk2 commented Jun 15, 2023

Beace commented Jun 17, 2023

Beace commented Jun 28, 2023

socket-security bot commented Jun 28, 2023 •

edited

Loading

codecov bot commented Jun 28, 2023 •

edited

Loading

fengmk2 commented Jul 23, 2023

Beace commented Jul 23, 2023

Beace commented Aug 2, 2023

fengmk2 commented Aug 14, 2023

Beace commented Aug 14, 2023

Beace commented Aug 16, 2023 •

edited

Loading

fengmk2 commented Aug 30, 2023

Beace commented Aug 30, 2023

fengmk2 left a comment

Beace commented Aug 30, 2023

Beace commented Aug 30, 2023

fengmk2 commented Aug 31, 2023

fengmk2 commented Aug 31, 2023

fengmk2 commented Aug 31, 2023

fengmk2 commented Aug 31, 2023

fengmk2 commented Sep 1, 2023

fengmk2 commented Sep 1, 2023

fengmk2 commented Sep 1, 2023

fengmk2 commented Sep 1, 2023

feat: support npm search command like npmio #513

feat: support npm search command like npmio #513

Conversation

Beace commented Jun 13, 2023 • edited Loading

Beace commented Jun 13, 2023

elrrrrrrr commented Jun 14, 2023

fengmk2 commented Jun 15, 2023

Beace commented Jun 17, 2023

Beace commented Jun 28, 2023

socket-security bot commented Jun 28, 2023 • edited Loading

codecov bot commented Jun 28, 2023 • edited Loading

Codecov Report

fengmk2 commented Jul 23, 2023

Beace commented Jul 23, 2023

Beace commented Aug 2, 2023

fengmk2 commented Aug 14, 2023

Beace commented Aug 14, 2023

Beace commented Aug 16, 2023 • edited Loading

fengmk2 commented Aug 30, 2023

Beace commented Aug 30, 2023

fengmk2 left a comment

Choose a reason for hiding this comment

Beace commented Aug 30, 2023

Beace commented Aug 30, 2023

fengmk2 commented Aug 31, 2023

fengmk2 commented Aug 31, 2023

fengmk2 commented Aug 31, 2023

fengmk2 commented Aug 31, 2023

fengmk2 commented Sep 1, 2023

fengmk2 commented Sep 1, 2023

fengmk2 commented Sep 1, 2023

fengmk2 commented Sep 1, 2023

Beace commented Jun 13, 2023 •

edited

Loading

socket-security bot commented Jun 28, 2023 •

edited

Loading

codecov bot commented Jun 28, 2023 •

edited

Loading

Beace commented Aug 16, 2023 •

edited

Loading