[Nara] add CRT token 'channelId' to amm burn/mint and sale mint notif…

…ication events (#314) * Add is short field to video entity (#301) * add isShort field to video entity * regenerate db migrations * remove @joystream/metadata-protobuf patch from assets/patches * fix lint issue * Disable both in Appp and eail notifications for video posted events (#299) * bump package version and update CHANGELOG (#302) * bump package version and update CHANGELOG * change release version * Simple public homefeed query and mutation (#304) * update graphql schema * add partial index on 'video.include_in_home_feed' field * update video view definition to only include public videos * regenerate migrations * add dumbPublicFeedVideos custom query * add setPublicFeedVideos mutation * fix lint issue * add arg to skip video IDs * revert: update video view definition to only include public videos * add feat. to unset public feed videos * address requested change * bump package version and update CHANGELOG * Update `nara` from `master` (#300) * Adds mappings for `ChannelAssetsDeletedByModerator` & `VideoAssetsDeletedByModerator` events (#199) * mark 'VideoDeletedByModerator' & 'ChannelDeletedByModerator' events deprecated * Implements mappings for 'Content.VideoAssetsDeletedByModerator and 'Content.ChannelAssetsDeletedByModerator' runtime events * remove unused import * Nara/crt update (#244) * feat: build orion * feat: start generating schema * fix: extra entities * fixup! * fix: continue implementing design specs * fix: review and fix foreign key relationships * fix: formatting * fix: generation errors * fix: add comment * fix: relations * fix: final review * fixup! * fix: add ending blocks * fix: generate type & set typegen to ipv4 * fix: add support for event backward compatibility * feat: start adding mappings * fix: continue with mappnigs * feat: init sale * feat: patronage decreased to & fixed build * feat: claim patronage event * feat: tokens bought on amm * feat: tokens sold on amm * fix: add relation between sales and vesting schedules * feat: add Tokens sold on sale vente * feat: update upcoming sale * feat: revenue share issued * feat: member joined whitelist * feat: amm deactivated * feat: burned token * feat: transfer policy changed to permissionless * feat: sale finalized * feat: finish mappings * fix: review * fix: remove cascade deletions * fix: renaming & formatting * fixup! * fixup! * fix: patched protobuf packages with token proto * feat: update metadata and add event handler scheleton * feat: token metadata * feat: sale metadata * fix: review comments * fix: formatting * fix: revenue * Revert "fix: revenue" This reverts commit 0821abe. * fix: token status after sale * fix: fixmes * fix: formatting * fix: funds accounting during sale * fix: amount accounting * fix: linter * fix: review * fix: review 2 * fix: review * fix: linter * feat: migration for new db scheam * fix: update event versions * fix: patch types with crt_release types * fix: patch types * fix: generate all events versions since mainnet * fix: temp fix after event version generation * fix: event versioning * fix: add migration * fix: mignations * fix: solve channel not being added * fix: add id to TokenChannel * fix: non-nullable deleted field set * fix: format * feat: creator token init sale re enabling * feat: re enable sale init code * fix: update types * fix: amm id * fix: id computation for revenue share * fix: amm id computation for token * fix: issuer transfer accounting * fix: amm tx id * fix: destination accounting * feat: minor fix on holder transfer processing * fix: re-enable metadata * fix: metadata parsing * fix: post reword cleanup * fix: format * fix: silence ci checks * fix: event version * fix: address PR changes I edited all the entity that have a composite index like TokenAccount so that they have a synthetic ID and an optionally unique @index * fix: add hidden entities conditions * fix: add extra fields to token in order to keep track of ongoing status * fix: build errors * fix: adapt mapping to new token fields * fix: format * feat: add trailer video entity this is required so we can simply make trailer video hidden if video is hidden * fix: linter * chore: prettier * fix: from PR review * fix: vesting schedule schema & mappings I have replaced the vesting schedule back to the original schema with: - VestingSchedule: holding vesting schedule information such being amount agnostic - VestedAccount: contains information regarded to a vested account, the goal is to mimic the runtime logic * fix: burning from vesting * patch: metadata-protobuf package * patch: metadata-protobuf package * fix: generate migrations * fix: purchase token on sale * Update schema/token.graphql Co-authored-by: Leszek Wiesner <leszek@jsgenesis.com> * Update schema/token.graphql Co-authored-by: Leszek Wiesner <leszek@jsgenesis.com> * fix: address PR * fix: hidden entities * fix: migration ok * feat: add extra check for migrations * fix: docker network * fix: format * fix: remove unrequired constraint * fix: 🐛 post rebase fixes * feat: 🎨 add metadata processing for issue token * feat(crt-v1): ✨ chain metadata for v 2003 * fix(crt-v1): 🚑 comment out view element for orion playgroud * fix(crt-v1): 🎨 add playground config variable to .env * feat: ✅ add tests * fix(crt-v1): 📦 packages and patches * fix(crt-v1): ✅ update entity id used and other minor fixes * fix(crt-v1): ✅ update entity id used and other minor fixes * test(crt-v1): 🐛 misc fixes to have tests working * test(crt-v1): 🐛 misc fixes to have tests working * fix(crt-v1): 🐛 metadata and trailer video * feat(crt-v1): 🎨 update types * fix(crt-v1): ✨ Add correct Ratio denomination (Permill) * update with master * fix: 🐛 metadata not being set * fix: 🐛 parameters order * test: 🧪 fixing integration tests * test(crt-v1): 🧪 fix integration tests * feat(crt-v1): ✨ last price for token and recovered field for rev share part * feat: ✨ add resolver for dividend amount * feat(crt-v1): ✨ start adding channel fields for trackingtotal revenue * feat(crt-v1): ✨ add utils for royalty computation * feat(crt-v1): ✨ cumulative revenue on channel * feat(crt-v1): ✨ add resolver for transferrable amount * fix(crt-v1): ✨ add `acquiredAt` to pinpoint latest vesting schedule for account * Token metadata processing update * Prettier * chore(crt-v1): ⚡ dbgen * fix(crt-v1): 🧪 fix integration tests * fix(crt-v1): 🐛 missing fields in token sale vesting source * test(crt-v1): 🧪 test for transferrable balance amount * fix(crt-v1): 🐛 transferrable amount * test: 🧪 update tests after resolver fix * fix: 🐛 error on vesting schedules array * fix: 🎨 CI fixes * docs: update gitignore * fix: 🚨 prettier * build: 📌 chai depnedencies --------- Co-authored-by: Leszek Wiesner <leszek@jsgenesis.com> Co-authored-by: WRadoslaw <r.wyszynski00@gmail.com> * Clear benefits even if not passed (#282) * 🤑 Fix revenue share dividend estimation (#297) * Fix on revenue share dividend estimation * Fix type on result * 🛕 Historical revenue share participants (#286) * New field for revenue share * Set potential revenue share particitants at the time of start * fix: .gitignore not working * fix lint issues * re-generate db migrations * commit register.html.mst file * fix: notifications integration test --------- Co-authored-by: Ignazio Bovo <ignazio@jsgenesis.com> Co-authored-by: Leszek Wiesner <leszek@jsgenesis.com> Co-authored-by: WRadoslaw <r.wyszynski00@gmail.com> Co-authored-by: WRadoslaw <92513933+WRadoslaw@users.noreply.github.com> * Revert "Update `nara` from `master` (#300)" (#306) This reverts commit 887427c. * generate auth api docs and types * add is short derived field to video entity (#310) * add is shirt derived field to video entity * add indices on is short fields * fix: video language detection fix (#309) * fix: video language detection fix * address requested changes * fix: predictVideoLanguage function * fix: include max 1 video per channel in homepage videos (#313) * fix: include max 1 video per channel in homepage videos * update setOrionLanguage Migration script * format updateVideoRelevanceValue SQL query * fix: use UTC midnight epoch instead of current epoch to calculate video relevance score * bump package version and update CHANGELOG * fix: lint bug * add CRT token 'channelId' to amm burn/mint and sale mint notification events --------- Co-authored-by: Ignazio Bovo <ignazio@jsgenesis.com> Co-authored-by: Leszek Wiesner <leszek@jsgenesis.com> Co-authored-by: WRadoslaw <r.wyszynski00@gmail.com> Co-authored-by: WRadoslaw <92513933+WRadoslaw@users.noreply.github.com>
Joystream · Mar 12, 2024 · 134ff7b · 134ff7b
1 parent ab5d974
commit 134ff7b
Show file tree

Hide file tree

Showing 10 changed files with 181 additions and 108 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -1,3 +1,17 @@
+# 3.7.0
+
+## Schema changes
+- Added `isShortDerived` field to `Video` entity indicating whether a video is a short format, vertical video or not. This field is computed in the mappings based on the video dimensions and duration when `isShort` is not set in the metadata.
+
+## Misc
+
+- update `setOrionLanguage` Custom migration script.
+
+## Bug Fixes:
+- Added fix to improve the accuracy of `Video.orionLanguage` field by reworking the `predictVideoLanguage` function in `src/utils/language.ts`
+- Use UTC midnight epoch instead of current epoch to calculate video relevance score in `VideoRelevanceManager`
+
+
 # 3.6.0
 
 ## Schema changes

diff --git a/package-lock.json b/package-lock.json
diff --git a/package.json b/package.json
@@ -1,6 +1,6 @@
 {
   "name": "orion",
-  "version": "3.6.0",
+  "version": "3.7.0",
   "engines": {
     "node": ">=16"
   },

diff --git a/schema/notifications.graphql b/schema/notifications.graphql
@@ -457,6 +457,9 @@ type CreatorTokenMarketStarted @variant {
 }
 
 type CreatorTokenMarketMint @variant {
+  "channel title for notification avatar"
+  channelId: String!
+
   "handle of member that minted tokens"
   minterHandle: String!
 
@@ -474,6 +477,9 @@ type CreatorTokenMarketMint @variant {
 }
 
 type CreatorTokenMarketBurn @variant {
+  "channel title for notification avatar"
+  channelId: String!
+
   "handle of member that burned tokens"
   burnerHandle: String!
 
@@ -491,6 +497,9 @@ type CreatorTokenMarketBurn @variant {
 }
 
 type CreatorTokenSaleMint @variant {
+  "channel title for notification avatar"
+  channelId: String!
+
   "handle of member that minted tokens"
   minterHandle: String!
 

diff --git a/src/mappings/content/video.ts b/src/mappings/content/video.ts
@@ -17,7 +17,7 @@ import {
   VideoViewEvent,
 } from '../../model'
 import { EventHandlerContext } from '../../utils/events'
-import { predictLanguage } from '../../utils/language'
+import { predictVideoLanguage } from '../../utils/language'
 import {
   deserializeMetadata,
   genericEventFields,
@@ -122,8 +122,10 @@ export async function processVideoCreatedEvent({
     }
   }
 
-  const languageText = [video.title ?? '', video.description ?? ''].join(' ')
-  video.orionLanguage = predictLanguage(languageText)
+  video.orionLanguage = predictVideoLanguage({
+    title: video.title ?? '',
+    description: video.description ?? '',
+  })
 
   channel.totalVideosCreated += 1
 
@@ -190,8 +192,10 @@ export async function processVideoUpdatedEvent({
     )
   }
 
-  const languageText = [video.title ?? '', video.description ?? ''].join(' ')
-  video.orionLanguage = predictLanguage(languageText)
+  video.orionLanguage = predictVideoLanguage({
+    title: video.title ?? '',
+    description: video.description ?? '',
+  })
 
   if (autoIssueNft) {
     await processNft(overlay, block, indexInBlock, extrinsicHash, video, contentActor, autoIssueNft)

diff --git a/src/mappings/token/index.ts b/src/mappings/token/index.ts
@@ -410,6 +410,7 @@ export async function processTokensBoughtOnAmmEvent({
 
   const minter = await overlay.getRepository(Membership).getById(memberId.toString())
   const notificationData = new CreatorTokenMarketMint({
+    channelId: tokenChannel.channelId,
     tokenSymbol: parseCreatorTokenSymbol(token),
     mintedTokenAmount: crtMinted,
     minterHandle: minter?.handle ?? 'Someone',
@@ -474,6 +475,7 @@ export async function processTokensSoldOnAmmEvent({
 
   const burnerMember = await overlay.getRepository(Membership).getById(memberId.toString())
   const notificationData = new CreatorTokenMarketBurn({
+    channelId: tokenChannel.channelId,
     tokenSymbol: parseCreatorTokenSymbol(token),
     burnedTokenAmount: crtBurned,
     burnerHandle: burnerMember?.handle ?? 'Someone',
@@ -547,6 +549,7 @@ export async function processTokensPurchasedOnSaleEvent({
 
   const minterMember = await overlay.getRepository(Membership).getById(memberId.toString())
   const notificationData = new CreatorTokenSaleMint({
+    channelId: tokenChannel.channelId,
     tokenSymbol: parseCreatorTokenSymbol(token),
     mintedTokenAmount: amountPurchased,
     minterHandle: minterMember?.handle ?? 'Someone',

diff --git a/src/utils/VideoRelevanceManager.ts b/src/utils/VideoRelevanceManager.ts
@@ -24,7 +24,7 @@ export class VideoRelevanceManager {
   }: VideoRelevanceManagerLoops): Promise<void> {
     const em = await globalEm
 
-    this.updateScheduledLoop(em, scheduledUpdateLoopTime)
+    this.updateLoop(em, scheduledUpdateLoopTime)
       .then(() => {
         /* Do nothing */
       })
@@ -33,7 +33,7 @@ export class VideoRelevanceManager {
         process.exit(-1)
       })
 
-    this.updateFullUpdateLoop(em, fullUpdateLoopTime)
+    this.updateLoop(em, fullUpdateLoopTime)
       .then(() => {
         /* Do nothing */
       })
@@ -74,52 +74,72 @@ export class VideoRelevanceManager {
 
     await em.query(`
         WITH videos_with_weight AS (
-        SELECT 
-        video.id as videoId,
-        channel.id as channelId,
-        (ROUND((
-        (extract(epoch from now()) - ${wtEpoch})
-        / ${NEWNESS_SECONDS_DIVIDER} * ${newnessWeight * -1} 
-        + (views_num * ${viewsWeight}) 
-        + (comments_count * ${commentsWeight}) 
-        + (reactions_count * ${reactionsWeight})) 
-        * COALESCE(channel.channel_weight, ${channelWeight}),2)) as videoRelevance
-        FROM video
-        INNER JOIN channel  ON video.channel_id = channel.id
-        ${
-          forceUpdateAll
-            ? ''
-            : `WHERE video.channel_id in (${[...this.channelsToUpdate.values()]
-                .map((id) => `'${id}'`)
-                .join(', ')})`
-        }
-        ORDER BY video.id),
-        
+          SELECT 
+            video.id as videoId,
+            channel.id as channelId,
+            (ROUND((
+            (extract(epoch FROM date_trunc('day', now() at time zone 'UTC')) - ${wtEpoch})
+            / ${NEWNESS_SECONDS_DIVIDER} * ${newnessWeight * -1} 
+            + (views_num * ${viewsWeight}) 
+            + (comments_count * ${commentsWeight}) 
+            + (reactions_count * ${reactionsWeight})) 
+            * COALESCE(channel.channel_weight, ${channelWeight}), 2)) as videoRelevance
+          FROM 
+            video
+            INNER JOIN channel  ON video.channel_id = channel.id
+          ${
+            forceUpdateAll
+              ? ''
+              : `WHERE video.channel_id in (${[...this.channelsToUpdate.values()]
+                  .map((id) => `'${id}'`)
+                  .join(', ')})`
+          }
+          ORDER BY 
+            video.id
+        ),
+    
         top_channel_score as (
-        SELECT 
-        channel.id as channelId,
-        MAX(videoCte.videoRelevance) as maxChannelRelevance
-        FROM channel
-        INNER JOIN videos_with_weight as videoCte on videoCte.channelId = channel.id
-        GROUP BY channel.id)
-        
-        UPDATE video
-        SET video_relevance = COALESCE(topChannelVideo.maxChannelRelevance, 1) 
-        FROM videos_with_weight as videoCte
-        LEFT JOIN top_channel_score as topChannelVideo on topChannelVideo.channelId = videoCte.channelId and topChannelVideo.maxChannelRelevance = videoCte.videoRelevance
-        WHERE video.id = videoCte.videoId;
-        `)
-    this.channelsToUpdate.clear()
-  }
+          SELECT
+            channel.id as channelId,
+            MAX(videos_with_weight.videoRelevance) as maxChannelRelevance
+          FROM
+            channel
+            INNER JOIN videos_with_weight on videos_with_weight.channelId = channel.id
+          GROUP BY
+            channel.id
+        ),
 
-  private async updateScheduledLoop(em: EntityManager, intervalMs: number): Promise<void> {
-    while (true) {
-      await this.updateVideoRelevanceValue(em)
-      await new Promise((resolve) => setTimeout(resolve, intervalMs))
-    }
+        ranked_videos AS (
+          SELECT
+            videos_with_weight.videoId,
+            topChannelVideo.maxChannelRelevance,
+            ROW_NUMBER() OVER (
+              PARTITION BY videos_with_weight.channelId
+              ORDER BY
+                videos_with_weight.videoRelevance DESC,
+                videos_with_weight.videoId
+            ) as rank
+          FROM
+            videos_with_weight
+            LEFT JOIN top_channel_score as topChannelVideo ON videos_with_weight.channelId = topChannelVideo.channelId
+        )
+
+        UPDATE
+          video
+        SET
+          video_relevance = CASE
+            WHEN ranked_videos.rank = 1 THEN ranked_videos.maxChannelRelevance
+            ELSE 1
+          END
+        FROM
+          ranked_videos
+        WHERE
+          video.id = ranked_videos.videoId;
+      `)
+    this.channelsToUpdate.clear()
   }
 
-  private async updateFullUpdateLoop(em: EntityManager, intervalMs: number): Promise<void> {
+  private async updateLoop(em: EntityManager, intervalMs: number): Promise<void> {
     while (true) {
       await this.updateVideoRelevanceValue(em)
       await new Promise((resolve) => setTimeout(resolve, intervalMs))

diff --git a/src/utils/auth.ts b/src/utils/auth.ts
@@ -1,17 +1,17 @@
+import { createLogger } from '@subsquid/logger'
 import { Request, Response } from 'express'
 import { EntityManager, FindOptionsWhere, IsNull, MoreThan } from 'typeorm'
 import { Account, Session } from '../model'
 import {
   CachedSessionData,
-  sessionCache,
-  SESSION_CACHE_MINIMUM_TTL,
   SESSION_CACHE_EXPIRY_TTL_MARGIN,
+  SESSION_CACHE_MINIMUM_TTL,
+  sessionCache,
 } from './cache'
-import { config, ConfigVariable } from './config'
-import { getUserAgentData } from './http'
-import { createLogger } from '@subsquid/logger'
-import { globalEm } from './globalEm'
+import { ConfigVariable, config } from './config'
 import { uniqueId } from './crypto'
+import { globalEm } from './globalEm'
+import { getUserAgentData } from './http'
 
 const authLogger = createLogger('authentication')
 

diff --git a/src/utils/customMigrations/setOrionLanguage.ts b/src/utils/customMigrations/setOrionLanguage.ts
@@ -1,50 +1,56 @@
-import { createLogger } from '@subsquid/logger'
-import { IsNull } from 'typeorm'
-import { Video } from '../../model'
+import { EntityManager } from 'typeorm'
 import { globalEm } from '../globalEm'
-import { predictLanguage } from '../language'
-
-const logger = createLogger('setOrionLanguage')
-
-async function setOrionLanguage() {
-  const em = await globalEm
+import { predictVideoLanguage } from '../language'
+
+async function detectVideoLanguage() {
+  const em: EntityManager = await globalEm
+  const videos: any[] = await em.query(`
+    SELECT id, title, description
+    FROM admin.video
+  `)
+
+  // Temporary storage for batch update data
+  const updates: any[] = []
+
+  for (const [i, video] of videos.entries()) {
+    const orionLanguage = predictVideoLanguage({
+      title: video.title,
+      description: video.description,
+    })
 
-  const batchSize = 10000
-  let offset = 0
-  let hasMore = true
+    // Instead of updating immediately, push the update data into the array
+    updates.push({ orionLanguage, id: video.id })
+    console.log(i)
+  }
 
-  while (hasMore) {
-    const videos = await em.find(Video, {
-      where: { orionLanguage: IsNull() },
-      order: { id: 'ASC' },
-      take: batchSize,
-      skip: offset,
-    })
+  // Define batch size
+  const batchSize = 1000 // Adjust the batch size based on your database and network performance
 
-    if (videos.length === 0) {
-      hasMore = false
-    } else {
-      const updates = videos.map((video) => {
-        const languageText = [video.title ?? '', video.description ?? ''].join(' ')
-        video.orionLanguage = predictLanguage(languageText)
-        return video
-      })
+  for (let i = 0; i < updates.length; i += batchSize) {
+    const batch = updates.slice(i, i + batchSize)
 
-      // Save all updates in a single transaction
-      await em.transaction(async (transactionalEntityManager) => {
-        await transactionalEntityManager.save(updates)
-      })
+    // Prepare the query and parameters for batch update
+    const query = `
+      UPDATE admin.video AS v SET
+        orion_language = c.orion_language
+      FROM (VALUES ${batch
+        .map((_, idx) => `($${idx * 2 + 1}, $${idx * 2 + 2})`)
+        .join(',')}) AS c(orion_language, id)
+      WHERE c.id = v.id;
+    `
 
-      logger.info(`Updated ${updates.length} videos.`)
+    const queryParams = batch.flatMap((update) => [update.orionLanguage, update.id])
 
-      offset += videos.length // Prepare the offset for the next batch
-    }
+    // Execute batch update
+    await em.query(query, queryParams)
   }
+
+  console.log(`Updated languages for ${videos.length} videos`)
 }
 
-setOrionLanguage()
-  .then(() => logger.info('Update process completed.'))
+detectVideoLanguage()
+  .then(() => console.log('Update process completed.'))
   .catch(() => {
-    logger.error('process failed')
+    console.error('process failed')
     process.exit(1)
   })