Orion v2 (#49) (#101)

* Initial commit

* Initial orion draft

* Hide private data

* Fix extendedChannels query

* docker-compose db config adjustments

* Fix ExtendedBasicChannelFields

* Add `video` view

* Storage, Membership and ChannelCreated mappings

* Channel events processing

* Video mappings + some generic fixes

* Mappings: Comments, reactions + some generic fixes

* Cleanup

* NFT mappings, some post-testing fixes

* Video views mutation

* Channel follows + video views improvements

* Channel/video reports

* Category filtering: Part 1

* State subscription

* Category filtering: Part 2 + some refactorization

* Category filtering: Part 3 - add missing views and operator auth

* mostViewedVideosConnection custom query + generic custom queries fixes

* Kill switch

* mostRecentChannels query

* channelNftCollectors query

* extendedVideoCategories query

* Fix AND-OR queries and "eq: null"

* Docker setup improvements

* Video hero

* Category featured videos

* Entity caching (overlay) refactorization

* Prettier

* Documentation, setup improvements, cleanup

* FIX: Save NextEntityIds

* Excludable content

* - Allow limiting the total number of cached entities
- Deployment manifest fixes
- Custom ARCHIVE_GATEWAY_URL instead of HOST/PORT
- Updated dependencies
- `processAssets` bug fix

* setVideoViewPerIpTimeLimit operator mutation

* Ephesus mappings WIP

* Fix conflicts with local Joystream docker setup

* Metadata-protobuf patch location fix

* Ephesus Atlas queries

* Update CHANGELOG

* Linting/formatting fixes, dead code removal

* Apps metaprotocol mappings

* Open auction topBid fix

* Fix metadata-protobuf patch

* Add signAppActionCommitment endpoint

* Tests: Basic data comparison script + bug fixes

* Remove channel-owned apps

* Remove Ephesus scope

* Apps as member-owned initial rework

* Fixes after compareState vs mainnet

* Remove lead-owned apps, fix typeorm-codegen patch

* AppAction support (Joystream/joystream#4631)

* Update CHANGELOG based on tests

* Allow controlling "is_censored" content exclusion via env

* buildExtendedChannelsQuery fix (activeVideosCount_gt: 0)

* Benchmarking script

* Benchmarking results

* Event queries optimalizations, fix gitignored v1 and v2 schemas

* Adjust autovacuum_analyze_scale_factor

* Fix entity cache cleanup

* Add index on event's `inExtrinsic` field

* Benchmarking round 2

* Fix req.ip reverse proxy issue, allow conditionally displaying excluded content in results

* Operator queries

* Update docs

* Prettier format

* Add CI checks

* Remove issue template, fix "workflows" location

* Disallow app deletion

* Update `generateAppActionCommitment` from `@joystream/js`

* Fix channel follows, video views and reports after operator queries

* Fix #82

* Comment notifications: Don't notify the author about their own comment

* Introduce a mechanism to preserve views, follows, reports and config data when updating the processor

* mostViewedVideosConnection missing params fix

* Fix assets encoding (AppAction)

* Assets url resolving

* Remove accidently commited file

* Fix Aquarium deployment

* Update CHANGELOG

* Improve caching speed and add more logs

* Latest apps-related updated (generateAppActionCommitment, mappings)

* Add Request entity

* Add mutation to request featured nft

* Add new flag for nft

* Add protected mutation to edit featured NFTs

* Small fixes

* Regenerating migration

* Hiding request table

* Process member banning/unbanning

* Fix: Remove banned members when channel is removed

* Update migration

* State queries: Re-enable apps

* Views: Add missing chanel/video related entities

* Excluding/restoring comments: Update:
- parentComment.repliesCount,
- parentComment.reactionsAndRepliesCount,
- video.commentsCount

* Operator queries: 401 Unauthorized, remove `x-display-hidden-entities` (operator just sees all hidden entities by default)

* Update changelog

* Simplify processor state subscription

* Prevent the same comment from being "excluded"/"restored" multiple times

* Small adjustments to docs and compareState script

* Orion v1 migration scripts

* New logic for comment exclusion

* Fix: Reference to `yarn` in package.json

* Increase interval of ProcessorStateRetriever from 100 ms to 1000 ms

* Dependencies update: Fix npm audit issues

* Remove cached gitignore files

* CR fixes

* Delete index.ts

* @subsquid/graphql-server `forbidUnknownValues` patch

* Asset resolution: Return multiple urls

* Update changelog

* FIX: Events cleanup on deleteVideo

---------

Co-authored-by: WRadoslaw <r.wyszynski00@gmail.com>

.dockerignore

@@ -1,5 +1,4 @@
-*.log
-node_modules/
-dist/
-Dockerfile
-docker-compose.yml
+/.git
+/node_modules
+/lib
+/*Versions.jsonl

.env

@@ -0,0 +1,32 @@
+# LOCAL ENVIRONMENT
+
+# Db config
+DB_NAME=squid
+DB_PASS=squid
+DB_PORT=23798
+# Processor service host
+PROCESSOR_HOST=localhost
+# Processor service prometheus port
+PROCESSOR_PROMETHEUS_PORT=3337
+# Graphql server port
+GQL_PORT=4350
+
+# Archive gateway url
+ARCHIVE_GATEWAY_URL=${CUSTOM_ARCHIVE_GATEWAY_URL:-http://localhost:8888/graphql}
+
+# Default config values
+SUPPORT_NO_CATEGORY_VIDEOS=true
+SUPPORT_NEW_CATEGORIES=true
+KILL_SWITCH_ON=false
+VIDEO_VIEW_PER_IP_TIME_LIMIT=86400 # 86400 seconds = 24 hours
+# Operator API secret
+OPERATOR_SECRET=this-is-not-so-secret-change-it
+# Processor configuration
+MAX_CACHED_ENTITIES=1000
+APP_PRIVATE_KEY=this-is-not-so-secret-change-it
+
+# Adjust accordingly with the number of trusted (used) reverse proxies!
+TRUSTED_REVERSE_PROXIES=1
+
+# Debug settings
+SQD_DEBUG=api:*

.eslintignore

@@ -0,0 +1,6 @@
+# Any build artifacts/JS
+lib
+**/*.js
+# Autogenerated stuff
+src/types
+**/generated

.eslintrc.js

@@ -1,47 +1,34 @@
 module.exports = {
-  extends: ['eslint:recommended', 'plugin:@typescript-eslint/recommended', 'plugin:jest/recommended', 'prettier'],
   env: {
-    node: true,
     es6: true,
-    jest: true,
+  },
+  globals: {
+    Atomics: 'readonly',
+    SharedArrayBuffer: 'readonly',
   },
   parser: '@typescript-eslint/parser',
-  plugins: ['@typescript-eslint', 'jest'],
+  parserOptions: {
+    ecmaVersion: 2019,
+    sourceType: 'module',
+    project: ['./tsconfig.json', './scripts/tsconfig.json'],
+  },
+  extends: [
+    'standard',
+    'eslint:recommended',
+    'plugin:@typescript-eslint/recommended',
+    'plugin:prettier/recommended',
+    'prettier',
+  ],
   rules: {
-    '@typescript-eslint/explicit-module-boundary-types': 'off',
-    '@typescript-eslint/no-non-null-assertion': 'off',
-    '@typescript-eslint/naming-convention': [
+    'no-console': 'off',
+    'no-useless-constructor': 'off',
+    '@typescript-eslint/no-floating-promises': [
       'error',
       {
-        selector: 'default',
-        format: ['camelCase'],
-      },
-      {
-        selector: 'variable',
-        format: ['camelCase', 'UPPER_CASE', 'PascalCase'],
-      },
-      {
-        selector: 'property',
-        format: [], // Don't force format of object properties, so they can be ie.: { "Some thing": 123 }, { some_thing: 123 } etc.
-      },
-      {
-        selector: 'accessor',
-        format: ['camelCase', 'snake_case'],
-      },
-      {
-        selector: 'enumMember',
-        format: null,
-      },
-      {
-        selector: 'typeLike',
-        format: [],
-        custom: { regex: '^([A-Z][a-z0-9]*_?)+', match: true }, // combined PascalCase and snake_case to allow ie. OpeningType_Worker
-      },
-      {
-        selector: 'classMethod',
-        modifiers: ['static'],
-        format: ['PascalCase'],
+        ignoreVoid: true,
       },
     ],
+    '@typescript-eslint/no-misused-promises': 'error',
   },
+  plugins: ['standard', '@typescript-eslint', 'prettier'],
 }

.github/workflows/checks.yml

@@ -2,8 +2,8 @@ name: Checks
 on: [push, pull_request]
 
 jobs:
-  lint:
-    name: Linting
+  checks:
+    name: Checks
     runs-on: ${{ matrix.os }}
     strategy:
       matrix:
@@ -16,7 +16,7 @@ jobs:
         uses: actions/setup-node@v1
         with:
           node-version: ${{matrix.node-version}}
-      - name: Install modules
-        run: yarn install --frozen-lockfile
-      - name: Run ESLint
-        run: yarn lint
+      - name: Install npm packages
+        run: npm ci
+      - name: Run checks
+        run: npm run checks

.gitignore

@@ -1,20 +1,12 @@
-yarn-debug.log*
-yarn-error.log*
-
-node_modules/
-dist/
-
-globalConfig.json
-
-.idea
-db/
-db.zip
-.env
-
-.pnp.*
-.yarn/*
-!.yarn/patches
-!.yarn/plugins
-!.yarn/releases
-!.yarn/sdks
-!.yarn/versions
+/node_modules
+/lib
+
+/**Versions.json
+/**Versions.jsonl
+
+# IDE files
+/.idea
+src/model/generated
+/schema.graphql
+/db/persisted
+/scripts/orion-v1-migration/data

.npmrc

@@ -0,0 +1 @@
+engine-strict=true

.prettierignore

@@ -0,0 +1,10 @@
+# Any build artifacts
+lib
+# MD files (for now)
+**/*.md
+# Autogenerated stuff
+src/types
+src/model/generated
+db/migrations/*.js
+schema.graphql
+/scripts/orion-v1-migration/data

.prettierrc.js

@@ -1,4 +1,10 @@
 module.exports = {
-  ...require('@joystream/prettier-config'),
-  printWidth: 120
+  singleQuote: true,
+  arrowParens: 'always',
+  useTabs: false,
+  tabWidth: 2,
+  semi: false,
+  trailingComma: 'es5',
+  quoteProps: 'preserve',
+  printWidth: 100,
 }

.vscode/settings.json

@@ -0,0 +1,5 @@
+{
+  "editor.codeActionsOnSave": {
+    "source.fixAll.eslint": true
+  }
+}