integration failed, official bluesky and official bluesky-social/ozone

[itaru2622]

I'm trying to use https://github.com/bluesky-social/ozone to moderate posts in official bluesky(bsky.app).

I exactly followed instruction of https://github.com/bluesky-social/ozone/blob/main/HOSTING.md,
but it is not working as expected.

In my trial,

• my ozone service can receive report and succeed UI operation as below figure
• but when I reload the post in browser (bsky.app), appearance of post is NOT changed. It may be because labels field is empty array as below json.
• NOTE: the figure shows ozone received reports twice for the same post. in first reporting, UI assigns !hide. in second reporting, UI assigns more labels, because no appearance changed.

how can get result as expected?
how can debug and find out where is not working?

# snippet of xrpc/app.bsky.feed.getPostThread, collected by browser's developer tool.
{
    "thread": {
        "$type": "app.bsky.feed.defs#threadViewPost",
        "post": {
            "uri": "at://did:plc:...35jlijge46mr5zkrcsd/app.bsky.feed.post/3kt5nkik3nc2j",
            "cid": "bafyreianmw4w2iww5aoe3v7bljw27r6v4w4j3cevrhxjpqsvthubzi3cx4",
            "author": { ...valid records      },
            "record": {
                "$type": "app.bsky.feed.post",
                "createdAt": "2024-05-23T10:54:42.821Z",
                "langs": [  "en"   ],
                "text": "testing label"
            },
            "replyCount": 0,
            "repostCount": 0,
            "likeCount": 0,
            "indexedAt": "2024-05-23T10:54:42.821Z",
            "viewer": {},
            "labels": []
        },
        "replies": []
    }
}

[bnewbold]

• have you subscribed to the labeled from the account you are testing with? labels are not included unless the user subscribes, resulting in additional HTTP headers getting set on the request
• if you are self-hosting the PDS, I believe you need to be using a version from the past couple months, to ensure that the labeler HTTP header is passed through
• a common issue with self-hosting Ozone is WebSockets not working correctly with some reverse-proxies (like nginx). can you confirm that you can connect to your Ozone instance and subscribe to the label feed over HTTPS? eg wsdump "wss://ozone.EXAMPLE.COM/xrpc/com.atproto.label.subscribeLabels?cursor=0"

[itaru2622]

@bnewbold thank you for your response.

have you subscribed to the labeled from the account you are testing with?

yes, I did.

if you are self-hosting the PDS, I believe you need to be using a version from the past couple months, to ensure that the labeler HTTP header is passed through

I'm using official PDS (bsky.social) at current test.
I'm also planning to test ozone with self-hosting PDS. so I want to know more detail about version to use, later.

a common issue with self-hosting Ozone is WebSockets not working correctly... wss://ozone.EXAMPLE.COM/xrpc/com.atproto.label.subscribeLabels?cursor=0

ok, my current issue may be caused by this. please let me know about below:

• who does subscribe the above endpoint in real scenario? does it appview(bsky)?
• how the subscriber(component of the above) knows existence of ozone.EXAMPLE.COM and also it needs to subscribe ?
  I want to know the triggers which subscriber knows existence and starts subscribing to ozone.EXAMPLE.COM.

[bnewbold]

The appview does the subscription.

It knows about the existence of moderation services via the declaration record that gets created in an account repo. The appview consumes to the full firehose and notices new declaration records, and updates to that record.

IIRC, the record indicates a service DID, and the DID document indicates a service endpoint (HTTPS URL) to the moderation service backend. The appview will resolve all of that and continuously try to connect to the labeler service.

[itaru2622]

@bnewbold Thank you for respose.

I re-created bluesky account for my ozone service and tested again, then it becomes better but still not perfect, unfortunately.
I tested two cases with the same configuration :

# test environment is the same as before, excepts re-creating ozone service account.

case 1) my ozone service assigns system-defined-label ( !hide or !warn) to the specific post: 
            => succeeded (it works as expected. the view of post also changed).
case 2) my ozone service assigns myown label (test-label31) to the specific post: 
            => failure. (the view of post is NOT changed. even the post has lebels field with my label definition.

in both cases:
- I used bsky.app via chrome browser on PC as client.
- whenever I assigned labels, ozone services sent events to xrpc/com.atproto.label.subscribeLabels, checked with subscribing by myself.

any idea what is wrong?
I think there is something wrong with providing view according to the assigned labels and user settings.
but I'm not sure where it is (appview or social-app).
belows are configuration and the logs:

myown label defined in ozone was below:

{
  "labelValues": [    "label-test31"  ],
  "labelValueDefinitions": [
    {
      "locales": [
        {
          "lang": "en",
          "name": "label-test31 Label Display Name",
          "description": "label-test31 Label description."
        }
      ],
      "blurs": "none",
      "severity": "inform",
      "adultOnly": false,
      "identifier": "label-test31",
      "defaultSetting": "warn"
    }
  ]
}

and my testing account subscribes label and set preference as below:

then,
in test case 1, the view of post got changed (i.e: 'blocked').
but in test case 2, the view is not changed even the post has labels field and having my label val as below:

# result of app.bsky.feed.getPostThread

{
    "thread": {
        "$type": "app.bsky.feed.defs#threadViewPost",
        "post": {
            "uri": "at://did:plc:...35jlijge46mr5zkrcsd/app.bsky.feed.post/3kth6huxfiz26",
            "cid": "bafyreia7kgwk2ahgfszcruomfzqsfdo54hsb5md6dabmieh46ys24szida",
            "author": { ... valid record     },
            "record": {
                "$type": "app.bsky.feed.post",
                "createdAt": "2024-05-27T05:51:25.292Z",
                "langs": [  "en"   ],
                "text": "testing label"
            },
            "replyCount": 0,
            "repostCount": 0,
            "likeCount": 0,
            "indexedAt": "2024-05-27T05:51:25.292Z",
            "viewer": {},
            "labels": [
                {
                    "cid": "bafyreia7kgwk2ahgfszcruomfzqsfdo54hsb5md6dabmieh46ys24szida",
                    "cts": "2024-05-27T06:34:10.050Z",
                    "src": "did:plc:2j7ysr2rkucc26jsbtzyilvg",
                    "uri": "at://did:plc:...35jlijge46mr5zkrcsd/app.bsky.feed.post/3kth6huxfiz26",
                    "val": "label-test31",
                    "ver": 1
                }
            ]
        },
        "replies": []
    }

the view of post on browser is below, with developer-tools introspect:

in test case1, the value of labels was below:

            "labels": [
                {
                    "cid": "bafyreia7kgwk2ahgfszcruomfzqsfdo54hsb5md6dabmieh46ys24szida",
                    "cts": "2024-05-27T07:18:10.139Z",
                    "src": "did:plc:2j7ysr2rkucc26jsbtzyilvg",
                    "uri": "at://did:plc:...35jlijge46mr5zkrcsd/app.bsky.feed.post/3kth6huxfiz26",
                    "val": "!hide",
                    "ver": 1
                }
            ]

even I changes some different value for label(changing blurs and severity), the results are the same.
i.e. the post has non-empty-labels field but the views are not changed.

[bnewbold]

Can you try with a label that has no numeric digits, only Latin alphabetic characters (a-z)?

The protocol spec for the label field is relatively flexible, but we strongly recommend limiting the characters set for now, and it is possible the bsky client or appview is being very strict and filtering out label-test31 somewhere in the stack.

https://atproto.com/specs/label#recommended-string-syntax

[itaru2622]

@bnewbold thank you, I confirmed that my own label can change the view of post by remove numeric from label string.

It is better the above syntax is more noticeable to avoid troubles. There is no describing about syntax in hosting.md nor in configuring page.

[bnewbold]

created a tracking ticket here: bluesky-social/ozone#126

[itaru2622]

@bnewbold as I mentioned above post, I succeeded to test ozone with official bluesky (bsky.app, .social, /network} thank you.

now, I am planing to start testing ozone with my entired self-hosting bluesky (all components, plc,relay, appview, pds, social-app are already selfhosted and working excepts moderation).

Before starting it, I would like to clarify the following points you mentioned before:

if you are self-hosting the PDS, I believe you need to be using a version from the past couple months, to ensure that the labeler HTTP header is passed through

Which PDS version should I use to ensure the above? which header needs to be present?

confirm that you can connect to your Ozone instance and subscribe to the label feed over HTTPS? eg wsdump "wss://ozone.EXAMPLE.COM/xrpc/com.atproto.label.subscribeLabels?cursor=0"
It(appview) knows about the existence of moderation services via the declaration record that gets created in an account repo. The appview consumes to the full firehose and notices new declaration records, and updates to that record.
IIRC, the record indicates a service DID, and the DID document indicates a service endpoint (HTTPS URL) to the moderation service backend. The appview will resolve all of that and continuously try to connect to the labeler service.

I found the code appview subscribs repos. at bsky/src/data-plane/server/indexing as below,
but could not figure out where appview subscribes ozone's label assignment(xrpc/com.atproto.label.subscribeLabels). could you point it out?

atproto/packages/bsky/src/data-plane/server/indexing/index.ts

Line 161 in ca0ca08

const { api } = new AtpAgent({ service: pds })

atproto/packages/bsky/src/data-plane/server/indexing/index.ts

Line 70 in ca0ca08

labeler: Labeler.makePlugin(this.db, this.background),

In this thread, I want to keep focusing on ozone x official bluesky (source). I will create another thread when needed.

[itaru2622]

@bnewbold after some investigation, I found that

there is no insertInto('label') statement in bsky data-plain, even though data-plain trying to get record from label table.
Is it as expected? I guess it is NOT since many test codes insert record into label table in bsky.
I'm glad if you confirm it's a bug or not. when it is a bug could you please fix it or propose workaround. I will post an issue if needed.

# lot of test codes insert records into label table in bsky (i.e data-plane)

$ grep -R -n "insertInto('label')" atproto/packages/*
atproto/packages/bsky/tests/query-labels.test.ts:76:      .insertInto('label')
atproto/packages/bsky/tests/views/takedown-labels.test.ts:107:    await network.bsky.db.db.insertInto('label').values(labels).execute()
atproto/packages/bsky/tests/views/timeline.test.ts:259:    .insertInto('label')
atproto/packages/bsky/tests/label-hydration.test.ts:182:      .insertInto('label')
atproto/packages/dev-env/src/seed/basic.ts:179:    .insertInto('label')
atproto/packages/dev-env/src/mock/index.ts:531:    .insertInto('label')
atproto/packages/ozone/tests/get-lists.test.ts:92:      .insertInto('label')
atproto/packages/ozone/tests/3p-labeler.test.ts:97:    await network.bsky.db.db.insertInto('label').values(labelEntries).execute()
atproto/packages/ozone/src/mod-service/index.ts:895:      .insertInto('label')
atproto/packages/pds/tests/seeds/basic.ts:161:    .insertInto('label')

# code inserting record into label or labeler table.
$ grep -R -n    insertInto        atproto/packages/bsky/src/data-plane/* |   grep label
atproto/packages/bsky/src/data-plane/server/indexing/plugins/labeler.ts:23:    .insertInto('labeler')
!!! NONE FOR insertInto('label')

# code getting record from label or labeler table.
$ grep -R -n    selectFrom         atproto/packages/bsky/*   |   grep label
atproto/packages/bsky/src/data-plane/server/routes/profile.ts:29:     .selectFrom('labeler')
atproto/packages/bsky/src/data-plane/server/routes/labels.ts:18:      .selectFrom('label')  // !! result should be always empty.

# codes inserting record into any table of data-plane
$ grep -R -n    insertInto    atproto/packages/bsky/src/data-plane/
atproto/packages/bsky/src/data-plane/bsync/index.ts:44:            .insertInto('mute')
atproto/packages/bsky/src/data-plane/bsync/index.ts:54:            .insertInto('list_mute')
atproto/packages/bsky/src/data-plane/server/db/migrations/20230608T155101190Z-algo-whats-hot-view.ts:15:    .insertInto('view_param')
atproto/packages/bsky/src/data-plane/server/routes/mutes.ts:113:      .insertInto('mute')
atproto/packages/bsky/src/data-plane/server/routes/mutes.ts:142:      .insertInto('list_mute')
atproto/packages/bsky/src/data-plane/server/routes/moderation.ts:58:      .insertInto('blob_takedown')
atproto/packages/bsky/src/data-plane/server/routes/notifs.ts:106:      .insertInto('actor_state')
atproto/packages/bsky/src/data-plane/server/indexing/plugins/list.ts:22:    .insertInto('list')
atproto/packages/bsky/src/data-plane/server/indexing/plugins/labeler.ts:23:    .insertInto('labeler')
atproto/packages/bsky/src/data-plane/server/indexing/plugins/feed-generator.ts:22:    .insertInto('feed_generator')
atproto/packages/bsky/src/data-plane/server/indexing/plugins/profile.ts:22:    .insertInto('profile')
atproto/packages/bsky/src/data-plane/server/indexing/plugins/repost.ts:33:      .insertInto('repost')
atproto/packages/bsky/src/data-plane/server/indexing/plugins/repost.ts:39:      .insertInto('feed_item')
atproto/packages/bsky/src/data-plane/server/indexing/plugins/repost.ts:117:    .insertInto('post_agg')
atproto/packages/bsky/src/data-plane/server/indexing/plugins/like.ts:23:    .insertInto('like')
atproto/packages/bsky/src/data-plane/server/indexing/plugins/like.ts:94:    .insertInto('post_agg')
atproto/packages/bsky/src/data-plane/server/indexing/plugins/post.ts:88:      .insertInto('post')
atproto/packages/bsky/src/data-plane/server/indexing/plugins/post.ts:94:      .insertInto('feed_item')
atproto/packages/bsky/src/data-plane/server/indexing/plugins/post.ts:157:      await db.insertInto('post_embed_image').values(imagesEmbed).execute()
atproto/packages/bsky/src/data-plane/server/indexing/plugins/post.ts:168:      await db.insertInto('post_embed_external').values(externalEmbed).execute()
atproto/packages/bsky/src/data-plane/server/indexing/plugins/post.ts:177:      await db.insertInto('post_embed_record').values(recordEmbed).execute()
atproto/packages/bsky/src/data-plane/server/indexing/plugins/post.ts:361:        .insertInto('post_agg')
atproto/packages/bsky/src/data-plane/server/indexing/plugins/post.ts:381:    .insertInto('profile_agg')
atproto/packages/bsky/src/data-plane/server/indexing/plugins/follow.ts:23:    .insertInto('follow')
atproto/packages/bsky/src/data-plane/server/indexing/plugins/follow.ts:88:    .insertInto('profile_agg')
atproto/packages/bsky/src/data-plane/server/indexing/plugins/follow.ts:102:    .insertInto('profile_agg')
atproto/packages/bsky/src/data-plane/server/indexing/plugins/list-block.ts:22:    .insertInto('list_block')
atproto/packages/bsky/src/data-plane/server/indexing/plugins/thread-gate.ts:28:    .insertInto('thread_gate')
atproto/packages/bsky/src/data-plane/server/indexing/plugins/list-item.ts:29:    .insertInto('list_item')
atproto/packages/bsky/src/data-plane/server/indexing/plugins/block.ts:22:    .insertInto('actor_block')
atproto/packages/bsky/src/data-plane/server/indexing/processor.ts:73:      .insertInto('record')
atproto/packages/bsky/src/data-plane/server/indexing/processor.ts:101:        .insertInto('duplicate_record')
atproto/packages/bsky/src/data-plane/server/indexing/processor.ts:250:        await db.db.insertInto('notification').values(chunk).execute()
atproto/packages/bsky/src/data-plane/server/indexing/index.ts:147:      .insertInto('actor')
atproto/packages/bsky/src/data-plane/server/indexing/index.ts:230:      .insertInto('actor_sync')

[bnewbold]

@devinivy is this maybe a thing with this variant of the dataplane not ingesting labels correctly?

[itaru2622]

@bnewbold @devinivy

when I tested with ozone with entire selfhosting env(selhosting PLC, relay, appview, PDS),
even my ozone publishing events to /xrpc/com.atproto.label.subscribeLabels and my appview stores labeler records in dataplane labeler table, there are no record in my dataplane label table.
it may cause the result that the post has empty labels field and its view is not changed on my social-app.

the above is the reason I want to confirm if current published appview(data-plane) code is correct(as expected).

[itaru2622]

@bnewbold @devinivy

after some more investigation of atproto repo,
I found the source to subscribe com.atproto.sync.subscribeRepos and store those messages, in below dev-env.

atproto/packages/dev-env/src/bsky.ts

Lines 91 to 96 in c5d36d5

	const sub = new bsky.RepoSubscription({
	service: cfg.repoProvider,
	db,
	idResolver: dataplane.idResolver,
	background: new BackgroundQueue(db),
	})

the above RepoSubscription instance subscribes com.atproto.sync.subscribeRepos and store messages with dataplane indexingservice.
but there is no similar mechanism for ozone's com.atproto.label.subscribeLabels in current published codes.

I believe that current officially hosted appview has the code to subscribe and store ozone's activity (label assignments) when it detects new ozone service joined. we are glad if you share the codes to us in this repository, for avoiding double development.

[bnewbold]

Yes, I think you are correct: we don't have label subscription working in the TypeScript implementation of the appview indexer.

We'll try to get this added, but i'm not sure what the timeline for that will be.

Thanks for investigating and reporting this!

[itaru2622]

created issue as #2552 about lacks of subscribeLabels