Occasional ValueError when using CAR.from_bytes

[that-damn-cat]

Using the example code for CAR decoding firehose commits...

https://atproto.blue/en/latest/atproto_firehose/index.html

from atproto import FirehoseSubscribeReposClient, parse_subscribe_repos_message
from atproto import CAR, models

firehoseClient = FirehoseSubscribeReposClient()

def on_message_handler(message) -> None:
    commit = parse_subscribe_repos_message(message)
    # we need to be sure that it's commit message with .blocks inside
    if not isinstance(commit, models.ComAtprotoSyncSubscribeRepos.Commit):
        return

    car = CAR.from_bytes(commit.blocks)

# Start the firehose
firehoseClient.start(on_message_handler)

Will occasionally result in the following error:

ValueError: Failed to read CAR header. Invalid uvarint

With this traceback...

Traceback (most recent call last):
  File "[...]\.venv\Lib\site-packages\atproto_firehose\client.py", line 141, in _process_message_frame
    self._on_message_callback(frame)
  File "[...]\firehoseScrape.py", line 12, in on_message_handler
    car = CAR.from_bytes(commit.blocks)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "[...]\.venv\Lib\site-packages\atproto_core\car\car.py", line 51, in from_bytes
    header, blocks = libipld.decode_car(data)
                     ^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: Failed to read CAR header. Invalid uvarint

[that-damn-cat]

Happens about once every million or so commits from the firehose, so it's fairly rare. I've seen it 3 times so far.

Set up a catch in code for it and am waiting for it to happen again so I can share the blocks that cause it.

[MarshalX]

My guess is CAR.from_bytes(None). blocks sometimes are empty somehow...

[emilyhunt]

My guess is CAR.from_bytes(None). blocks sometimes are empty somehow...

Yeah, I think this is the issue. I think it can be a blank binary string sometimes - although it's out of spec for the AT Protocol, sometimes commits have empty blocks.

I handle if for the astronomy feeds with an extra check you could put before CAR.from_bytes(commit.blocks), like:

if not commit.blocks:
    return
car = CAR.from_bytes(commit.blocks)

[MarshalX]

sounds like we need to fix the examples :)

[that-damn-cat]

Yep, that's what I ended up catching was an empty block.