Fix: Chain writer failed due to TypeError
#549
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
hoh
force-pushed
the
hoh-fix-packing-string-in-memory
branch
from
fa567ea
to
19d2c51
Compare
nesitor
previously approved these changes
Jan 31, 2024
hoh
force-pushed
the
hoh-fix-packing-string-in-memory
branch
from
16c11f2
to
ab29a70
Compare
hoh
force-pushed
the
hoh-fix-packing-string-in-memory
branch
from
04f815c
to
eb1ce1f
Compare
Solution: Use `bytes` in memory everywhere. Not elegant but fixes the problem. --- The Chain writer task for Chain.ETH failed with the following stacktrace: ``` aleph-pyaleph-1 | 2024-01-29 14:09:32 [ERROR] aleph.chains.connector: Chain writer task for Chain.ETH failed, relaunching in 10 seconds. aleph-pyaleph-1 | Traceback (most recent call last): aleph-pyaleph-1 | File "/opt/pyaleph/src/aleph/chains/connector.py", line 63, in chain_writer_task aleph-pyaleph-1 | await connector.packer(config) aleph-pyaleph-1 | File "/opt/pyaleph/src/aleph/chains/ethereum.py", line 351, in packer aleph-pyaleph-1 | await self.chain_data_service.prepare_sync_event_payload( aleph-pyaleph-1 | File "/opt/pyaleph/src/aleph/chains/chain_data_service.py", line 75, in prepare_sync_event_payload aleph-pyaleph-1 | ipfs_cid = await self.storage_service.add_file( aleph-pyaleph-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ aleph-pyaleph-1 | File "/opt/pyaleph/src/aleph/storage.py", line 293, in add_file aleph-pyaleph-1 | await self.add_file_content_to_local_storage( aleph-pyaleph-1 | File "/opt/pyaleph/src/aleph/storage.py", line 269, in add_file_content_to_local_storage aleph-pyaleph-1 | await self.storage_engine.write(filename=file_hash, content=file_content) aleph-pyaleph-1 | File "/opt/pyaleph/src/aleph/services/storage/fileystem_engine.py", line 26, in write aleph-pyaleph-1 | file_path.write_bytes(content) aleph-pyaleph-1 | File "/usr/lib/python3.11/pathlib.py", line 1066, in write_bytes aleph-pyaleph-1 | view = memoryview(data) aleph-pyaleph-1 | ^^^^^^^^^^^^^^^^ aleph-pyaleph-1 | TypeError: memoryview: a bytes-like object is required, not 'str' ``` It appears that a `StringIO` was passed to `aleph.storage.add_file(...)`, resulting in `file_content` being of type `str`, where the underlying function `aleph.services.storage.fileystem_engine.FileSystemStorageEngine.write` expects the type `bytes`. Solution: 1. Encode the `archive_content` in UTF-8 before adding it in the storage service. 2. Modify the type annotations accordingly 3. Modify the signature of `aleph.storage.StorageService.add_file` to expect `bytes` instead of any `IO` type.
hoh
force-pushed
the
hoh-fix-packing-string-in-memory
branch
from
eb1ce1f
to
1213beb
Compare
MHHukiewitz
approved these changes
Feb 14, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The Chain writer task for Chain.ETH failed with the following stacktrace:
It appears that a
StringIOwas passed toaleph.storage.add_file(...), resulting infile_contentbeing of typestr, where the underlying functionaleph.services.storage.fileystem_engine.FileSystemStorageEngine.writeexpects the typebytes.Solution:
archive_contentin UTF-8 before adding it in the storage service.aleph.storage.StorageService.add_fileto expectBytesIOinstead of anyIOtype.