Notification branch fixes for issued arised during Q&A (#225)

* Update squid.yaml fix: squid.yml version numbering * Release PR: Orion v3.0.1 (#185) * fix: processor's overlay bug (#170) * atlas homepage query speed optimisation (#177) * fix: gitignored src/model/generated * add index on video.video_relevance field * fix typo in developer guide docs * regenerate db/migrations/*-Data.js file * Fix/unblock graphql (#183) * Make session optional and apply changes for auth handles * Create new middleware for users and use it on some resolvers * Remove remaining auth guards * Small docs change * Add middleware for channel report * Prettier * docs: 📝 update changelog (#184) --------- Co-authored-by: Zeeshan Akram <37098720+zeeshanakram3@users.noreply.github.com> Co-authored-by: WRadoslaw <92513933+WRadoslaw@users.noreply.github.com> * fix: 🚑 fix accountId global counter not being migrated (#188) * docs: 📝 add back up guide (#196) * mark 'VideoDeletedByModerator' & 'ChannelDeletedByModerator' events deprecated (#193) * fix: 🐛 encoding issues with member handle (#214) * fix: 🐛 encoding issues with member handle * fix: 🐛 pr issues * fix: 🐛 refactor and update changelog * docs: update version number * fix: 🚑 account counter (#215) * Orion v3.0.3 (#224) * Remove n+1 problem for StorageDataObject (#209) * Remove n+1 problem for StorageDataObject * Inform user about incorrect query * fix: ✨ add global counter migratino to account for 303 (#217) * Orion 303 final touches (#223) * feat: ✨ add CORS origins for atlas local testing * chore: 📝 adds 303 release notes * Fix typos (#176) * fix typos * fix typos * fix typos * fix typos * fix typos * fix typo * fix typos * fix typos * fix typos * fix typos * fix typos * fix typos --------- Co-authored-by: WRadoslaw <92513933+WRadoslaw@users.noreply.github.com> Co-authored-by: omahs <73983677+omahs@users.noreply.github.com> * fix(notifications): uninitialized access fields * fix(notifications): 🐛 add channelId to new auction * Orion v3.0.4 - Release (#230) * 🧮 Aggregate channel payments (#222) * Fix channels `cumulativeRewardClaimed_DESC` order * Aggregate channel payments * Make `cumulativeRewardClaimed` and `cumulativeRewardPaid` non null * Generate db migrations * Rename the `cumulativeReward` field * Re-generate db migrations * Bump Orion's version * feat: ⚡ add migration to version 3.0.4 for account id counter (#228) --------- Co-authored-by: Theophile Sandoz <theophile.sandoz@gmail.com> * Orion v3.1.0 (#238) * Add granular permissions support for Gateway operator users (#231) * Add granular permissions support for Gateway operator users * fix lint issues * revert docker-compose port change * mark 'grantPermissions' & 'revokePermissions' input fields are non-nullable & return new permissions instead of boolean * Set Channel Weight (`setChannelsWeights`) mutation (#232) * Add granular permissions support for Gateway operator users * fix lint issues * add mutation to set channel weight/bias for homepage video relevance * revert docker-compose port change * mark 'grantPermissions' & 'revokePermissions' input fields are non-nullable & return new permissions instead of boolean * bump package version * update global migration counter map * bumped package version & updated CHANGELOG --------- Co-authored-by: Ignazio Bovo <ignazio@jsgenesis.com> * Postgres performance improvements (#235) * add index in video.createdAt field * add pg_stat_extenstion extenstion for queries stats * docs: ✨ changelog and fix data-js (#237) --------- Co-authored-by: Zeeshan Akram <37098720+zeeshanakram3@users.noreply.github.com> * Hotfix/3.1.0 (#240) * Orion v3.1.0 (#238) * Add granular permissions support for Gateway operator users (#231) * Add granular permissions support for Gateway operator users * fix lint issues * revert docker-compose port change * mark 'grantPermissions' & 'revokePermissions' input fields are non-nullable & return new permissions instead of boolean * Set Channel Weight (`setChannelsWeights`) mutation (#232) * Add granular permissions support for Gateway operator users * fix lint issues * add mutation to set channel weight/bias for homepage video relevance * revert docker-compose port change * mark 'grantPermissions' & 'revokePermissions' input fields are non-nullable & return new permissions instead of boolean * bump package version * update global migration counter map * bumped package version & updated CHANGELOG --------- Co-authored-by: Ignazio Bovo <ignazio@jsgenesis.com> * Postgres performance improvements (#235) * add index in video.createdAt field * add pg_stat_extenstion extenstion for queries stats * docs: ✨ changelog and fix data-js (#237) --------- Co-authored-by: Zeeshan Akram <37098720+zeeshanakram3@users.noreply.github.com> * fix: 🚑 hotfix default value for channel weight no matter what --------- Co-authored-by: Zeeshan Akram <37098720+zeeshanakram3@users.noreply.github.com> * fix: ✅ add test for comment reply and metadata * fix: ✨ solve conflicts * test: ✅ add test for channel verification + refactoring * chore: 🚨 linter * fix: 💚 add 10 seconds sleep time for CI checks * fix: 🐛 channel excluded title * docs: 📝 typo --------- Co-authored-by: Zeeshan Akram <37098720+zeeshanakram3@users.noreply.github.com> Co-authored-by: WRadoslaw <92513933+WRadoslaw@users.noreply.github.com> Co-authored-by: omahs <73983677+omahs@users.noreply.github.com> Co-authored-by: Theophile Sandoz <theophile.sandoz@gmail.com>
Joystream · Nov 17, 2023 · 4ee063c · 4ee063c
1 parent d709e3a
commit 4ee063c
Show file tree

Hide file tree

Showing 29 changed files with 553 additions and 191 deletions.
diff --git a/.env b/.env
@@ -38,9 +38,10 @@ VIDEO_RELEVANCE_VIEWS_TICK=50
 #   views weight,
 #   comments weight,
 #   rections weights,
-#   [joystream creation weight, YT creation weight]
+#   [joystream creation weight, YT creation weight],
+#   Default channel weight/bias
 # ]
-RELEVANCE_WEIGHTS="[1, 0.03, 0.3, 0.5, [7,3]]"
+RELEVANCE_WEIGHTS="[1, 0.03, 0.3, 0.5, [7,3], 1]"
 MAX_CACHED_ENTITIES=1000
 APP_PRIVATE_KEY=this-is-not-so-secret-change-it
 SESSION_EXPIRY_AFTER_INACTIVITY_MINUTES=60

diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -1,6 +1,17 @@
-# 3.1.0
+# 3.2.0
 This release adds notifications to the orion infrastructure...
 
+# 3.1.0
+
+### Entities
+- Adds `User.permission` to the `User` entity, this however doesn't require migration logic.
+- Adds `Channel.channelWeights` in order to boost channel relevance. This value can be set via the `setChannelWeights` mutation
+### Resolvers
+- Adds supports for new permissions model for gateway operator users. Now the root user can assign/revoke operator permission/s to users using `grantPermissions` & `revokePermissions` mutations
+- Adds new `setChannelWeights` operator mutation to set weight/bias for any channel/s which will be used to calculate the Atlas homepage video relevance scores
+### Performance
+- Adds `Video.createdAt` as index in order to speed up Atlas home page queries 
+
 # 3.0.4
 
 ### Misc

diff --git a/db/migrations/1699537206830-Data.js → db/migrations/1700052773258-Data.js b/db/migrations/1699537206830-Data.js → db/migrations/1700052773258-Data.js
diff --git a/db/migrations/2200000000000-Operator.js b/db/migrations/2200000000000-Operator.js
@@ -19,9 +19,13 @@ module.exports = class Operator2300000000000 {
       INSERT INTO "admin"."user" ("id", "is_root")
       VALUES ('${process.env.OPERATOR_SECRET || randomAsHex(32)}', true);
     `)
+
+    // Create pg_stat_statements extension for analyzing query stats
+    await db.query(`CREATE EXTENSION pg_stat_statements`)
   }
 
   async down(db) {
     await db.query(`DELETE FROM "admin"."user" WHERE "is_root" = true;`)
+    await db.query(`DROP EXTENSION pg_stat_statements`)
   }
 }
diff --git a/db/postgres.conf b/db/postgres.conf
@@ -3,4 +3,6 @@ log_statement = all
 autovacuum_analyze_scale_factor = 0.01
 shared_buffers=2GB
 jit=off
-random_page_cost=1.0
+random_page_cost=1.0
+shared_preload_libraries = 'pg_stat_statements'
+pg_stat_statements.track = all
diff --git a/docker-compose.yml b/docker-compose.yml
@@ -10,8 +10,7 @@ services:
       POSTGRES_DB: squid
       POSTGRES_PASSWORD: squid
     ports:
-      - '127.0.0.1:${DB_PORT}:${DB_PORT}'
-      - '[::1]:${DB_PORT}:${DB_PORT}'
+      - '${DB_PORT}:${DB_PORT}'
     command: ['postgres', '-c', 'config_file=/etc/postgresql/postgresql.conf', '-p', '${DB_PORT}']
     shm_size: 1g
     volumes:

diff --git a/package-lock.json b/package-lock.json
diff --git a/package.json b/package.json
@@ -1,6 +1,6 @@
 {
   "name": "orion",
-  "version": "3.1.0",
+  "version": "3.2.0",
   "engines": {
     "node": ">=16"
   },

diff --git a/schema/auth.graphql b/schema/auth.graphql
@@ -1,10 +1,28 @@
+enum OperatorPermission {
+  GRANT_OPERATOR_PERMISSIONS
+  REVOKE_OPERATOR_PERMISSIONS
+  SET_VIDEO_WEIGHTS
+  SET_CHANNEL_WEIGHTS
+  SET_KILL_SWITCH
+  SET_VIDEO_VIEW_PER_USER_TIME_LIMIT
+  SET_VIDEO_HERO
+  SET_CATEGORY_FEATURED_VIDEOS
+  SET_SUPPORTED_CATEGORIES
+  SET_FEATURED_NFTS
+  EXCLUDE_CONTENT
+  RESTORE_CONTENT
+}
+
 type User @entity {
   "Unique identifier (32-byte string, securely random)"
   id: ID!
 
   "Whether the user has root (gateway operator) privileges"
   isRoot: Boolean!
 
+  "List of all the gateway operator permissions that this user has"
+  permissions: [OperatorPermission!]
+
   "The account associated with the user (if any)"
   account: Account @derivedFrom(field: "user")
 

diff --git a/schema/channels.graphql b/schema/channels.graphql
@@ -3,7 +3,7 @@ type Channel @entity {
   id: ID!
 
   "Timestamp of the block the channel was created at"
-  createdAt: DateTime!
+  createdAt: DateTime! @index
 
   "Current member-owner of the channel (if owned by a member)"
   ownerMember: Membership
@@ -65,6 +65,9 @@ type Channel @entity {
   "Cumulative rewards paid to this channel"
   cumulativeReward: BigInt!
 
+  "Weight/Bias of the channel affecting video relevance in the Homepage"
+  channelWeight: Float
+
   "Channel Ypp Status: either unverified , verified or suspended"
   yppStatus: ChannelYppStatus!
 }

diff --git a/schema/notifications.graphql b/schema/notifications.graphql
@@ -125,11 +125,13 @@ type NftFeaturedOnMarketPlace @variant {
 }
 
 type ChannelVerified @variant {
+  "no data needed as recipient is channel"
   phantom: Int
 }
 
 type ChannelExcluded @variant {
-  phantom: Int
+  "title for the channel used for notification text"
+  channelTitle: String!
 }
 
 type VideoExcluded @variant {

diff --git a/src/mail-scheduler/index.ts b/src/mail-scheduler/index.ts
@@ -1,6 +1,3 @@
-import { config as dontenvConfig } from 'dotenv'
-import path from 'path'
-
 import { ConfigVariable, config } from '../utils/config'
 import { EmailDeliveryAttempt, NotificationEmailDelivery } from '../model'
 import { EntityManager } from 'typeorm'

diff --git a/src/mappings/content/commentsAndReactions.ts b/src/mappings/content/commentsAndReactions.ts
@@ -513,11 +513,13 @@ export async function processCreateCommentMessage(
         videoId: video.id,
         videoTitle: parseVideoTitle(video),
         memberHandle: await memberHandleById(overlay, memberId),
+        memberId,
       }
+      const memberRecipientId = parentComment.authorId || undefined
       await addNotification(
         overlay,
         authorAccount,
-        new MemberRecipient({ membership: comment.authorId }),
+        new MemberRecipient({ membership: memberRecipientId }),
         new CommentReply(notificationData),
         event
       )

diff --git a/src/mappings/content/nft.ts b/src/mappings/content/nft.ts
@@ -50,7 +50,6 @@ import {
   NftOfferedEventData,
   Account,
   MemberRecipient,
-  AuctionTypeOpen,
 } from '../../model'
 import { addNftActivity, addNftHistoryEntry, genericEventFields } from '../utils'
 import { SubstrateBlock, assertNotNull } from '@subsquid/substrate-processor'
@@ -91,6 +90,7 @@ export async function processOpenAuctionStartedEvent({
   if (video.channelId) {
     const channelTitle = await getChannelTitleById(overlay, video.channelId)
     const notificationData = new NewAuction({
+      channelId: video.channelId,
       channelTitle,
       videoId: video.id,
       videoTitle: parseVideoTitle(video),
@@ -138,6 +138,7 @@ export async function processEnglishAuctionStartedEvent({
   if (video.channelId) {
     const channelTitle = await getChannelTitleById(overlay, video.channelId)
     const notificationData = new NewAuction({
+      channelId: video.channelId,
       channelTitle,
       videoId: video.id,
       videoTitle: parseVideoTitle(video),
@@ -547,6 +548,7 @@ export async function processNftSellOrderMadeEvent({
   if (video?.channelId) {
     const channelTitle = await getChannelTitleById(overlay, video.channelId)
     const notificationData = new NewNftOnSale({
+      channelId: video.channelId,
       channelTitle,
       videoId: video.id,
       videoTitle: parseVideoTitle(video),

diff --git a/src/mappings/content/utils.ts b/src/mappings/content/utils.ts
@@ -866,11 +866,11 @@ export function computeRoyalty(royaltyPct: number, price: bigint): bigint {
 }
 
 export function parseChannelTitle(channel: Flat<Channel>): string {
-  return channel.title || '??'
+  return channel.title || FALLBACK_CHANNEL_TITLE
 }
 
 export function parseVideoTitle(video: Flat<Video>): string {
-  return video.title || '??'
+  return video.title || FALLBACK_VIDEO_TITLE
 }
 
 export async function memberHandleById(
@@ -886,3 +886,6 @@ export async function getChannelTitleById(overlay: EntityManagerOverlay, channel
   const channel = await overlay.getRepository(Channel).getByIdOrFail(channelId)
   return parseChannelTitle(channel)
 }
+
+export const FALLBACK_CHANNEL_TITLE = '??'
+export const FALLBACK_VIDEO_TITLE = '??'
diff --git a/src/server-extension/check.ts b/src/server-extension/check.ts
@@ -1,8 +1,8 @@
 import { RequestCheckFunction } from '@subsquid/graphql-server/lib/check'
-import { Context as OpenreaderContext } from '@subsquid/openreader/lib/context'
 import { TypeormOpenreaderContext } from '@subsquid/graphql-server/lib/typeorm'
-import { AuthContext, authenticate } from '../utils/auth'
+import { Context as OpenreaderContext } from '@subsquid/openreader/lib/context'
 import { UnauthorizedError } from 'type-graphql'
+import { AuthContext, authenticate } from '../utils/auth'
 
 export type Context = OpenreaderContext & AuthContext