Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Set Channel Weight (setChannelsWeights) mutation #232

Merged
7 changes: 4 additions & 3 deletions .env
Original file line number Diff line number Diff line change
Expand Up @@ -38,9 +38,10 @@ VIDEO_RELEVANCE_VIEWS_TICK=50
# views weight,
# comments weight,
# rections weights,
# [joystream creation weight, YT creation weight]
# [joystream creation weight, YT creation weight],
# Default channel weight/bias
# ]
RELEVANCE_WEIGHTS="[1, 0.03, 0.3, 0.5, [7,3]]"
RELEVANCE_WEIGHTS="[1, 0.03, 0.3, 0.5, [7,3], 1]"
MAX_CACHED_ENTITIES=1000
APP_PRIVATE_KEY=this-is-not-so-secret-change-it
SESSION_EXPIRY_AFTER_INACTIVITY_MINUTES=60
Expand All @@ -59,4 +60,4 @@ SENDGRID_FROM_EMAIL=gateway@example.com

# Debug settings
SQD_DEBUG=api:*
OPENAPI_PLAYGROUND=true
OPENAPI_PLAYGROUND=true
5 changes: 5 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,8 @@
# 3.1.0

- Adds supports for new permissions model for gateway operator users. Now the root user can assign/revoke operator permission/s to users using `grantPermissions` & `revokePermissions` mutations
- Adds new `setVideoWeights` operator mutation to set weight/bias for any channel/s which will be used to calculate the Atlas homepage video relevance scores

# 3.0.4

### Misc
Expand Down
415 changes: 415 additions & 0 deletions db/migrations/1699504891472-Data.js

Large diffs are not rendered by default.

4 changes: 2 additions & 2 deletions package-lock.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion package.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "orion",
"version": "3.0.4",
"version": "3.1.0",
"engines": {
"node": ">=16"
},
Expand Down
1 change: 1 addition & 0 deletions schema/auth.graphql
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@ enum OperatorPermission {
GRANT_OPERATOR_PERMISSIONS
REVOKE_OPERATOR_PERMISSIONS
SET_VIDEO_WEIGHTS
SET_CHANNEL_WEIGHTS
SET_KILL_SWITCH
SET_VIDEO_VIEW_PER_USER_TIME_LIMIT
SET_VIDEO_HERO
Expand Down
3 changes: 3 additions & 0 deletions schema/channels.graphql
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,9 @@ type Channel @entity {

"Cumulative rewards paid to this channel"
cumulativeReward: BigInt!

"Weight/Bias of the channel affecting video relevance in the Homepage"
channelWeight: Float
}

type BannedMember @entity @index(fields: ["member", "channel"], unique: true) {
Expand Down
39 changes: 38 additions & 1 deletion src/server-extension/resolvers/AdminResolver/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -10,9 +10,10 @@ import { getResolveTree } from '@subsquid/openreader/lib/util/resolve-tree'
import { GraphQLResolveInfo } from 'graphql'
import 'reflect-metadata'
import { Args, Ctx, Info, Mutation, Query, Resolver, UseMiddleware } from 'type-graphql'
import { EntityManager, In, Not } from 'typeorm'
import { EntityManager, In, Not, UpdateResult } from 'typeorm'
import { videoRelevanceManager } from '../../../mappings/utils'
import {
Channel,
OperatorPermission,
User,
Video,
Expand All @@ -27,6 +28,7 @@ import { OperatorOnly } from '../middleware'
import { model } from '../model'
import {
AppActionSignatureInput,
ChannelWeight,
ExcludableContentType,
ExcludeContentArgs,
ExcludeContentResult,
Expand All @@ -39,6 +41,7 @@ import {
RevokeOperatorPermissionsInput,
SetCategoryFeaturedVideosArgs,
SetCategoryFeaturedVideosResult,
SetChannelsWeightsArgs,
SetFeaturedNftsInput,
SetFeaturedNftsResult,
SetKillSwitchInput,
Expand Down Expand Up @@ -105,13 +108,47 @@ export class AdminResolver {
args.commentsWeight,
args.reactionsWeight,
[args.joysteamTimestampSubWeight, args.ytTimestampSubWeight],
args.defaultChannelWeight,
],
em
)
await videoRelevanceManager.updateVideoRelevanceValue(em, true)
return { isApplied: true }
}

@UseMiddleware(OperatorOnly(OperatorPermission.SET_CHANNEL_WEIGHTS))
@Mutation(() => [ChannelWeight])
async setChannelsWeights(@Args() { inputs }: SetChannelsWeightsArgs): Promise<ChannelWeight[]> {
const em = await this.em()

const results: ChannelWeight[] = []

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was wondering that now that we have permissions in place it might happens that there can be more than one user with the SET_CHANNEL_WEIGHTS permission and they might try to call this mutation at the same time.
Should we consider locking for this (and possibly other mutations)?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I get your point but I don't think it's really necessary to lock the records, since locks have their own overhead, and they put all the future transactions affecting the record in idle state. which could have a drastic effect on performance (e.g. If we use lock in this mutation, then all the read/write operations by other mappings/queries/operations will be blocked until this operation is completed). Also, this specific transaction is only about updating a single field of the record, and even if there is any concurrent transaction affecting the record, at worst the channel weight update would fail, which I think is fine (as you would see that in the mutation response).

Another important thing is that maybe we want to review all the pessimistic_write lock usage in queries/mutations and see if it's really required.

// Process each SetChannelWeightInput
for (const weightInput of inputs) {
const { channelId, weight } = weightInput

// Update the channel weight in the database
const updateResult: UpdateResult = await em.transaction(
ignazio-bovo marked this conversation as resolved.
Show resolved Hide resolved
async (transactionalEntityManager) => {
return transactionalEntityManager
.createQueryBuilder()
.update(Channel)
.set({ channelWeight: weight })
.where('id = :id', { id: channelId })
.execute()
}
)

// Push the result into the results array
results.push({
channelId,
isApplied: !!updateResult.affected,
})
}

return results
}

@UseMiddleware(OperatorOnly(OperatorPermission.SET_KILL_SWITCH))
@Mutation(() => KillSwitch)
async setKillSwitch(@Args() args: SetKillSwitchInput): Promise<KillSwitch> {
Expand Down
27 changes: 27 additions & 0 deletions src/server-extension/resolvers/AdminResolver/types.ts
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,9 @@ export class SetVideoWeightsInput {

@Field(() => Float, { nullable: false })
ytTimestampSubWeight!: number

@Field(() => Float, { nullable: false })
defaultChannelWeight!: number
}

@ObjectType()
Expand All @@ -29,6 +32,30 @@ export class VideoWeights {
isApplied!: boolean
}

@InputType()
export class ChannelWeightInput {
@Field(() => String, { nullable: false })
channelId!: string

@Field(() => Float, { nullable: false })
weight!: number
}

@ArgsType()
export class SetChannelsWeightsArgs {
@Field(() => [ChannelWeightInput], { nullable: false })
inputs!: ChannelWeightInput[]
}

@ObjectType()
export class ChannelWeight {
@Field(() => String, { nullable: false })
channelId!: string

@Field(() => Boolean, { nullable: false })
isApplied!: boolean
}

@ArgsType()
export class SetKillSwitchInput {
@Field(() => Boolean, { nullable: false })
Expand Down
20 changes: 13 additions & 7 deletions src/utils/VideoRelevanceManager.ts
Original file line number Diff line number Diff line change
Expand Up @@ -31,21 +31,25 @@ export class VideoRelevanceManager {
commentsWeight,
reactionsWeight,
[joystreamTimestampWeight, ytTimestampWeight] = [7, 3],
defaultChannelWeight,
] = await config.get(ConfigVariable.RelevanceWeights, em)
await em.query(`
WITH weighted_timestamp AS (
SELECT
id,
"video"."id",
(
extract(epoch from created_at)*${joystreamTimestampWeight} +
COALESCE(extract(epoch from published_before_joystream), extract(epoch from created_at))*${ytTimestampWeight}
) / ${joystreamTimestampWeight + ytTimestampWeight} as wtEpoch
extract(epoch from video.created_at)*${joystreamTimestampWeight} +
COALESCE(extract(epoch from video.published_before_joystream), extract(epoch from video.created_at))*${ytTimestampWeight}
) / ${joystreamTimestampWeight} + ${ytTimestampWeight} as wtEpoch,
"channel"."channel_weight" as CW
FROM
"video"
INNER JOIN
"channel" ON "video"."channel_id" = "channel"."id"
${
forceUpdateAll
? ''
: `WHERE "id" IN (${[...this.videosToUpdate.values()]
: `WHERE "video"."id" IN (${[...this.videosToUpdate.values()]
.map((id) => `'${id}'`)
.join(', ')})`
}
Expand All @@ -54,10 +58,12 @@ export class VideoRelevanceManager {
"video"
SET
"video_relevance" = ROUND(
(extract(epoch from now()) - wtEpoch) / (60 * 60 * 24) * ${newnessWeight * -1} +
(
(extract(epoch from now()) - wtEpoch) / ${NEWNESS_SECONDS_DIVIDER} * ${newnessWeight * -1} +
(views_num * ${viewsWeight}) +
(comments_count * ${commentsWeight}) +
(reactions_count * ${reactionsWeight}),
(reactions_count * ${reactionsWeight})
) * COALESCE(CW, ${defaultChannelWeight}),
2)
FROM
weighted_timestamp
Expand Down
3 changes: 2 additions & 1 deletion src/utils/config.ts
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,8 @@ export const configVariables = {
[ConfigVariable.KillSwitch]: boolType,
[ConfigVariable.VideoViewPerUserTimeLimit]: intType,
[ConfigVariable.VideoRelevanceViewsTick]: intType,
[ConfigVariable.RelevanceWeights]: jsonType<[number, number, number, number, [number, number]]>(),
[ConfigVariable.RelevanceWeights]:
jsonType<[number, number, number, number, [number, number], number]>(),
[ConfigVariable.AppPrivateKey]: stringType,
[ConfigVariable.SessionMaxDurationHours]: intType,
[ConfigVariable.SessionExpiryAfterInactivityMinutes]: intType,
Expand Down
11 changes: 6 additions & 5 deletions src/utils/offchainState.ts
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
import { EntityManager } from 'typeorm'
import fs from 'fs'
import path from 'path'
import { createLogger } from '@subsquid/logger'
import assert from 'assert'
import { uniqueId } from './crypto'
import fs from 'fs'
import path from 'path'
import { EntityManager } from 'typeorm'
import { NextEntityId } from '../model'
import { uniqueId } from './crypto'

const DEFAULT_EXPORT_PATH = path.resolve(__dirname, '../../db/export/export.json')

Expand All @@ -22,7 +22,7 @@ const exportedStateMap = {
User: true,
Account: true,
Token: true,
Channel: ['is_excluded', 'video_views_num', 'follows_num'],
Channel: ['is_excluded', 'video_views_num', 'follows_num', 'channel_weight'],
Video: ['is_excluded', 'views_num'],
Comment: ['is_excluded'],
OwnedNft: ['is_featured'],
Expand Down Expand Up @@ -80,6 +80,7 @@ export class OffchainState {
'3.0.2': ['Account'],
'3.0.3': ['Account'],
'3.0.4': ['Account'],
'3.1.0': ['Account'],
}

private migrations: Migrations = {
Expand Down