-
Notifications
You must be signed in to change notification settings - Fork 765
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Failed to upload artifact #270
Comments
Pushing up for facing the exact issue |
Facing same issue here as well. Was working previously on Nov 12th but not now. |
We are experiencing the same issue. Re-running the same workflow that was successful yesterday afternoon now gives this error. Here is the console output for our workflow:
|
Same here. Job was working on Dec 4th, 2021, but now fails for no obvious reason, There were no changes to the job itself. |
I believe this is a problem introduced in the 2.3.0 release (the timing checks out for us). We pegged our action to 2.2.4 and everything is working again. The issue in 2.3.0 could be related to artifacts containing local git repository files as indicated in #281. |
confirmed. When the action is set to @v2.2.4. the upload is working again. Most recent update to v2.3 should either be fixed or reverted. |
Had the exact same issue. Worked yesterday. Broken today. Updating to explicitly use 2.2.4 worked perfectly. |
The bug could be related to the number of files. A Job using @v2 works when uploading only a single file. Jobs uploading multiple files fail. This would also match the description in #281. The failure described there might not be related to .gitkeep being a repository file but just to .gitkeep being the second file. |
File that fails "2021-12-09_09h46m37s_webdriver_replay_sequence_1.log" is an empty log 0kb 7/12/2021
9/12/2021
|
The same observed in my repo https://github.com/metanorma/packed-mn/runs/4461130168?check_suite_focus=true
|
Getting the same issue appear for us (https://github.com/autodesk-forks/MaterialX/runs/4462734963?check_suite_focus=true). This appears this is the common error that comes back from the link given in the log:
|
Same error observed here, but here the enviroment is windows-latest
Also confirming, using v. @v2.2.4. and the upload works again |
I'm running my builds on linux, macos, and windows:
and the above upload error happens only for windows. |
This is a workaround for actions/upload-artifact#270
seeing similar issue with |
I have an assumption that it can happen for 0 bytes files |
We've just encountered the issue mentioned by @azqa in this particular instance when using The exact output is:
The corresponding It baffles me that this particular download fails when our other similar jobs have passed. I wonder if this is due to the length of the artifact name; that was the only difference amongst our jobs. |
Same here. Running v3. |
Downgraded from v3 to v2, while I'm seeing fewer failures still see those on v2 ( https://github.com/metabase/metabase/runs/5789943574?check_suite_focus=true |
any news? |
* Move linting and unit tests from CircleCI -> GHA Allows us to run unit tests without charge. Also opens the door to more interesting integrations (such as linting failures showing up as PR comments). Integration tests will be migrated in a later PR. * Remove CircleCI config for lint & unit tests These will run in github now, so we no longer need to spend money to run them in circle ci. * Fix issue uploading artifact in v3 Works around actions/upload-artifact#270 * Revert "Remove CircleCI config for lint & unit tests" This reverts commit 20139b3. Co-authored-by: Albin Severinson <albin@severinson.org>
* fix Altinity#311 * fix Altinity#312 * fix https://github.com/Altinity/clickhouse-backup/runs/4385266807 * fix wrong amd64 `libc` dependency * change default skip_tables pattern to exclude INFORMATION_SCHEMA database for clickhouse 21.11+ * actualize GET /backup/actions, and fix config.go `CLICKHOUSE_SKIP_TABLES` definition * add COS_DEBUG separate setting, wait details in Altinity#316 * try to resolve Altinity#317 * Allow using OIDC token for AWS credentials * update ReadMe.md add notes about INFORMATION_SCHEMA.* * fix Altinity#220, allow total_bytes as uint64 fetching fix allocations for `tableMetadataForDownload` fix getTableSizeFromParts behavior only for required tables fix Error handling on some suggested cases * fix Altinity#331, corner case when `Table` and `Database` have the same name. update clickhouse-go to 1.5.1 * fix Altinity#331 * add SFTP_DEBUG to try debug Altinity#335 * fix bug, recursuve=>recursive * BackUPList use 'recursive=true', and other codes do not change, hope this can pass CI * Force recursive equals true locally * Reset recursive flag to false * fix Altinity#111 * add inner Interface for COS * properly fix for recursive delimiter, fix Altinity#338 * Fix bug about metadata.json, we should check the file name first, instead of appending metadata.json arbitrary * add ability to restore schema ON CLUSTER, fix Altinity#145 * fix bug about clickhouse-backup list remote which shows no backups info, clickhouse-backup create_remote which will not delete the right backups * fix `Address: NULL pointer` when DROP TABLE ... ON CLUSTER, fix Altinity#145 * try to fix `TestServerAPI` https://github.com/Altinity/clickhouse-backup/runs/4727526265 * try to fix `TestServerAPI` https://github.com/Altinity/clickhouse-backup/runs/4727754542 * Give up using metaDataFilePath variable * fix bug * Add support encrypted disk (include s3 encrypted disks), fix [Altinity#260](Altinity#260) add 21.12 to test matrix fix FTP MkDirAll behavior fix `restore --rm` behavior for 20.12+ for tables which have dependent objects (like dictionary) * try to fix failed build https://github.com/Altinity/clickhouse-backup/runs/4749276032 * add S3 only disks check for 21.8+ * fix Altinity#304 * fix Altinity#309 * try return GCP_TESTS back * fix run GCP_TESTS * fix run GCP_TESTS, again * split build-artifacts and build-test-artifacts * try to fix https://github.com/Altinity/clickhouse-backup/runs/4757549891 * debug workflows/build.yaml * debug workflows/build.yaml * debug workflows/build.yaml * final download atrifacts for workflows/build.yaml * fix build docker https://github.com/Altinity/clickhouse-backup/runs/4758167628 * fix integration_tests https://github.com/AlexAkulov/clickhouse-backup/runs/4758357087 * Improve list remote speed via local metadata cache, fix Altinity#318 * try to fix https://github.com/Altinity/clickhouse-backup/runs/4763790332 * fix test after fail https://github.com/Altinity/clickhouse-backup/runs/4764141333 * fix concurrency MkDirAll for FTP remote storage, improve `invalid compression_format` error message * fix TestLongListRemote * Clean code, do not name variables so sloppily, names should be meaningful * Update clickhouse.go Change partitions => part * Not change Files filed in json file * Code should be placed in proper position * Update server.go * fix bug * Invoke SoftSelect should begin with ch. * fix error, clickhouse.common.TablePathEncode => common.TablePathEncode * refine code * try to commit * fix bug * Remove unused codes * Use NewReplacer * Add `CLICKHOUSE_IGNORE_NOT_EXISTS_ERROR_DURING_FREEZE`, fix Altinity#319 * fix test fail https://github.com/Altinity/clickhouse-backup/runs/4825973411?check_suite_focus=true * run only TestSkipNotExistsTable on Github actions * try to fix TestSkipNotExistsTable * try to fix TestSkipNotExistsTable * try to fix TestSkipNotExistsTable, for ClickHouse version v1.x * try to fix TestSkipNotExistsTable, for ClickHouse version v1.x * add microseconds to log, try to fix TestSkipNotExistsTable, for ClickHouse version v20.8 * add microseconds to log, try to fix TestSkipNotExistsTable, for ClickHouse version v20.8 * fix connectWithWait, some versions of clickhouse accept connections during process /entrypoint-initdb.d, need wait to continue * add TestProjections * rename dropAllDatabases to more mental and clear name * skip TestSkipNotExistsTable * Support specified partition backup (Altinity#356) * Support specify partition during backup create Authored-by: wangzhen <wangzhen@growingio.com> * fix PROJECTION restore Altinity#320 * fix TestProjection fail after https://github.com/Altinity/clickhouse-backup/actions/runs/1712868840 * switch to `altinity-qa-test` bucket in GCS test * update github.com/mholt/archiver/v3 and github.com/ClickHouse/clickhouse-go to latest version, remove old github.com/mholt/archiver usage * fix `How to convert MergeTree to ReplicatedMergeTree` instruction * fix `FTP` connection usage in MkDirAll * optimize ftp.go connection pool * Add `UPLOAD_BY_PART` config settings for improve upload/download concurrency fix Altinity#324 * try debug https://github.com/AlexAkulov/clickhouse-backup/runs/4920777422 * try debug https://github.com/AlexAkulov/clickhouse-backup/runs/4920777422 * fix vsFTPd 500 OOPS: vsf_sysutil_bind, maximum number of attempts to find a listening port exceeded, fix https://github.com/AlexAkulov/clickhouse-backup/runs/4921182982 * try to fix race condition in GCP https://github.com/AlexAkulov/clickhouse-backup/runs/4924432841 * update clickhouse-go to 1.5.3, properly handle `--schema` parameter for show local backup size after `download` * add `Database not exists` corner case for `IgnoreNotExistsErrorDuringFreeze` option * prepare release 1.3.0 - Add implementation `--diff-from-remote` for `upload` command and properly handle `required` on download command, fix Altinity#289 - properly `REMOTE_STORAGE=none` error handle, fix Altinity#375 - Add support for `--partitions` on create, upload, download, restore CLI commands and API endpoint fix Altinity#378, properly implementation of Altinity#356 - Add `print-config` cli command fix Altinity#366 - API Server optimization for speed of `last_backup_size_remote` metric calculation to make it async during REST API startup and after download/upload, fix Altinity#309 - Improve `list remote` speed via local metadata cache in `$TEMP/.clickhouse-backup.$REMOTE_STORAGE`, fix Altinity#318 - fix Altinity#375, properly `REMOTE_STORAGE=none` error handle - fix Altinity#379, will try to clean `shadow` if `create` fail during `moveShadow` - more precise calculation backup size during `upload`, for backups created with `--partitions`, fix bug after Altinity#356 - fix `restore --rm` behavior for 20.12+ for tables which have dependent objects (like dictionary) - fix concurrency by `FTP` creation directories during upload, reduce connection pool usage - properly handle `--schema` parameter for show local backup size after `download` - add ClickHouse 22.1 instead of 21.12 to test matrix * fix build https://github.com/Altinity/clickhouse-backup/runs/5033550335 * Add `API_ALLOW_PARALLEL` to support multiple parallel execution calls for, WARNING, control command names don't try to execute multiple same commands and be careful, it could allocate much memory during upload / download, fix Altinity#332 * apt-get update too slow today on github ;( * fix TestLongListRemote * fix Altinity#340, properly handle errors on S3 during Walk() and delete old backup * Add TestFlows tests to GH workflow (Altinity#5) * add actions tests * Update test.yaml * update * update * update * update * update * update * update * update * update * update * update * update * update * update * update * update * update * update * update * updated * added config.yml * added config.yml * update * updated files * added tests for views * updated tests * updated * fixed snapshots * updated tests in response to @Slach * upload new stuff * rerun * fix * fix * remove file * added requirements * fix fails * ReRun actions * Moved credentials * added secrets * ReRun actions * Edited test.yaml * Edited test.yaml * ReRun actions * removed TE flag * update * update * update * fix type * update * try to reanimate ugly github actions and ugly python tests * try to reanimate ugly config_rbac.py * fix Altinity#300 fix WINDOW VIEW restore fix restore for different compression_format than backup created fix most of xfail in regression.py merge test.yaml and build.yaml in github actions Try to add experimental support for backup `MaterializedMySQL` and `MaterializedPostgeSQL` tables, restore MySQL tables not impossible now without replace `table_name.json` to `Engine=MergeTree`, PostgreSQL not supported now, see ClickHouse/ClickHouse#32902 * return format back * fix build.yaml after https://github.com/Altinity/clickhouse-backup/actions/runs/1800312966 * fix build.yaml after https://github.com/Altinity/clickhouse-backup/actions/runs/1800312966 * build fixes after https://github.com/Altinity/clickhouse-backup/runs/5079597138 * build fixes after https://github.com/Altinity/clickhouse-backup/runs/5079630559 * build fixes after https://github.com/Altinity/clickhouse-backup/runs/5079669062 * fix tfs report * fix upload artifact for tfs report * fix upload artifact for clickhouse logs, remove unused BackupOptions * suuka * fix upload `clickhouse-logs` artifacts and tfs `report.html` * fix upload `clickhouse-logs` artifacts * fix upload `clickhouse-logs` artifacts, fix tfs reports * fix tfs reports * change retention to allow upload-artifacts work * fix ChangeLog.md * skip gcs and aws remote storage tests if secrets not set * remove short output * increase timeout to allow download images during pull * remove upload `tesflows-clickhouse-logs` artifacts to avoid 500 error * fix upload_release_assets action for properly support arm64 * switch to mantainable `softprops/action-gh-release` * fix Unexpected input(s) 'release_name' * move internal, config, util into `pkg` refactoring * updated test requirements * refactoring `filesystemhelper.Chown` remove unnecessary getter/setter, try to reproduce access denied for Altinity#388 (comment) * resolve Altinity#390, for 1.2.3 hotfix branch * backport 1.3.x Dockerfile and Makefile to allow 1.2.3 docker ARM support * fix Altinity#387 (comment), improve documentation related to memory and CPU usage * fix Altinity#388, improve restore ON CLUSTER for VIEW with TO clause * fix Altinity#388, improve restore ATTACH ... VIEW ... ON CLUSTER, GCS golang sdk updated to latest * fix Altinity#385, properly handle multiple incremental backup sequences + `BACKUPS_TO_KEEP_REMOTE` * fix Altinity#392, correct download for recursive sequence of diff backups when `DOWNLOAD_BY_PART` true fix integration_test.go, add RUN_ADVANCED_TESTS environment, fix minio_nodelete.sh * try to reduce upload artifact jobs, look actions/upload-artifact#171 and https://github.com/Altinity/clickhouse-backup/runs/5229552384?check_suite_focus=true * try to docker-compose up from first time https://github.com/AlexAkulov/clickhouse-backup/runs/5231510719?check_suite_focus=true * disable telemetry for GCS related to googleapis/google-cloud-go#5664 * update aws-sdk-go and GCS storage SDK * DROP DATABASE didn't clean S3 files, DROP TABLE clean! * - fix Altinity#406, properly handle `path` for S3, GCS for case when it begin from "/" * fix getTablesWithSkip * fix Altinity#409 * cherry pick release.yaml from 1.3.x to 1.2.x * fix Altinity#409, for 1.3.x avoid delete partially uploaded backups via `backups_keep_remote` option * Updated requirements file * fix Altinity#409, for 1.3.x avoid delete partially uploaded backups via `backups_keep_remote` option * fix testflows test * fix testflows test * restore tests after update minio * Fix incorrect in progress check on the example of Kubernetes CronJob * removeOldBackup error log from fatal to warning, to avoid race-condition deletion during multi-shard backup * switch to golang 1.18 Signed-off-by: Slach <bloodjazman@gmail.com> * add 22.3 to test matrix, fix Altinity#422, avoid cache broken (partially uploaded) remote backup metadata. * add 22.3 to test matrix * fix Altinity#404, switch to 22.3 by default Signed-off-by: Slach <bloodjazman@gmail.com> * fix Altinity#404, update to archiver/v4, properly support context during upload / download and correct error handler, reduce `SELECT * system.disks` calls Signed-off-by: Slach <bloodjazman@gmail.com> * cleanup ChangeLog.md, finally before 1.3.2 release Signed-off-by: Slach <bloodjazman@gmail.com> * continue fix Altinity#404 Signed-off-by: Slach <bloodjazman@gmail.com> * continue fix Altinity#404, properly calculate max_parts_count Signed-off-by: Slach <bloodjazman@gmail.com> * continue fix Altinity#404, properly calculate max_parts_count Signed-off-by: Slach <bloodjazman@gmail.com> * add multithreading GZIP implementation Signed-off-by: Slach <bloodjazman@gmail.com> * add multithreading GZIP implementation Signed-off-by: Slach <bloodjazman@gmail.com> * add multithreading GZIP implementation Signed-off-by: Slach <bloodjazman@gmail.com> * Updated Testflows README.md * add `S3_ALLOW_MULTIPART_DOWNLOAD` to config, to improve download speed, fix Altinity#431 Signed-off-by: Slach <bloodjazman@gmail.com> * fix snapshot after change default config Signed-off-by: Slach <bloodjazman@gmail.com> * fix testflows healthcheck for slow internet connection during `clickhouse_backup` start Signed-off-by: Slach <bloodjazman@gmail.com> * fix snapshot after change defaultConfig Signed-off-by: Slach <bloodjazman@gmail.com> * - add support backup/restore user defined functions https://clickhouse.com/docs/en/sql-reference/statements/create/function, fix Altinity#420 Signed-off-by: Slach <bloodjazman@gmail.com> * Updated README.md in testflows tests * remove unnecessary SQL query for calculateMaxSize, refactoring test to allow restoreRBAC with restart on 21.8 (strange bug, clickhouse stuck after try to run too much distributed DDL queries from ZK), update LastBackupSize metric during API call /list/remote, add healthcheck to docker-compose in integration tests Signed-off-by: Slach <bloodjazman@gmail.com> * try to fix GitHub actions Signed-off-by: Slach <bloodjazman@gmail.com> * try to fix GitHub actions, WTF, why testflows failed? Signed-off-by: Slach <bloodjazman@gmail.com> * add `clickhouse_backup_number_backups_remote`, `clickhouse_backup_number_backups_local`, `clickhouse_backup_number_backups_remote_expected`,`clickhouse_backup_number_backups_local_expected` prometheus metric, fix Altinity#437 Signed-off-by: Slach <bloodjazman@gmail.com> * add ability to apply `system.macros` values to `path` field in all types of `remote_storage`, fix Altinity#438 Signed-off-by: Slach <bloodjazman@gmail.com> * use all disks for upload and download for mutli-disk volumes in parallel when `upload_by_part: true` fix Altinity#400 Signed-off-by: Slach <bloodjazman@gmail.com> * fix wrong warning for .gz, .bz2, .br archive extensions during download, fix Altinity#441 Signed-off-by: Slach <bloodjazman@gmail.com> * fix Altinity#441, again ;( Signed-off-by: Slach <bloodjazman@gmail.com> * try to improve strange parts long tail during test Signed-off-by: Slach <bloodjazman@gmail.com> * update actions/download-artifact@v3 and actions/upload-artifact@v2, after actions fail Signed-off-by: Slach <bloodjazman@gmail.com> * downgrade actions/upload-artifact@v2.2.4, actions/upload-artifact#270, after actions fail https://github.com/AlexAkulov/clickhouse-backup/runs/6481819375 Signed-off-by: Slach <bloodjazman@gmail.com> * fix upload data go routines wait, expect improve upload speed the same as 1.3.2 Signed-off-by: Slach <bloodjazman@gmail.com> * prepare 1.4.1 Signed-off-by: Slach <bloodjazman@gmail.com> * Fix typo in Example.md * Set default value for max_parts_count in Azure config * fix `--partitions` parameter parsing, fix Altinity#425 Signed-off-by: Slach <bloodjazman@gmail.com> * remove unnecessary logs, fix release.yaml to mark properly tag in GitHub release Signed-off-by: Slach <bloodjazman@gmail.com> * add `API_INTEGRATION_TABLES_HOST` option to allow use DNS name in integration tables system.backup_list, system.backup_actions Signed-off-by: Slach <bloodjazman@gmail.com> * add `API_INTEGRATION_TABLES_HOST` fix for tesflows fails Signed-off-by: Slach <bloodjazman@gmail.com> * fix `upload_by_part: false` max file size calculation, fix Altinity#454 * upgrade actions/upload-artifact@v3, actions/upload-artifact#270, after actions fail https://github.com/Altinity/clickhouse-backup/runs/6962550621 * [clickhouse-backup] fixes on top of upstream * upstream versions Co-authored-by: Slach <bloodjazman@gmail.com> Co-authored-by: Vilmos Nebehaj <vilmos@sprig.com> Co-authored-by: Eugene Klimov <eklimov@altinity.com> Co-authored-by: root <root@SLACH-MINI.localdomain> Co-authored-by: wangzhen <wangzhen@growingio.com> Co-authored-by: W <wangzhenaaa7@gmail.com> Co-authored-by: Andrey Zvonov <32552679+zvonand@users.noreply.github.com> Co-authored-by: zvonand <azvonov@altinity.com> Co-authored-by: benbiti <wangshouben@hotmail.com> Co-authored-by: Vitaliis <vsviderskyi@altinity.com> Co-authored-by: Toan Nguyen <hgiasac@gmail.com> Co-authored-by: Guido Iaquinti <guido@posthog.com> Co-authored-by: ricoberger <mail@ricoberger.de>
Second edit: probably because I'm trying upload an artifact of the same name twice https://github.com/ThatXliner/skootils/runs/7374236761?check_suite_focus=true Does downgrading help? Or is this just flaky Edit: The logs mention
|
it looks completelly flaky ;( and github team do nothing for it ;( |
It so annoying ;(
|
Just to add another voice, our organization has also been experiencing this issue. It happens intermittently, and also appears to happen more frequently when it must upload multiple chunks. |
I am also running into this problem on v3, like @curena-contrast said the artifact size seems to affect how often it happens. I have a matrix pipeline building several bundles, some are <1mb and some are >200mb. The ones <1mb don't fail and the ones >200mb always fails I have seen some reports from people having a very similar error: this seems related to the code not properly handling a 408 HTTP response (Request Timeout) from the servers and retrying to upload the chunk Switching to v2 seems to fix it, or at the very least it triggers less often? It can be hard to tell |
Checking in here, also experiencing this with v3 today. |
I had this problem and the solution for me was to introduce a period of waiting.
The code was correct and worked prior to the introduction to docker. |
Getting the same error sporadically
with commit 83fd05a in step
|
Can this be caused by parallel (competing?) attempts from jobs running under matrix strategy? |
- name: Upload testflows logs
uses: actions/upload-artifact@v3
with:
name: testflows-logs-and-reports-${{ matrix.clickhouse }}-${{ github.run_id }}
path: |
test/testflows/*.log
test/testflows/*.log.txt
test/testflows/clickhouse_backup/_instances/**/*.log
test/testflows/*.html
retention-days: 7 and it doesn't work ;( |
V4 upload-artifact has released today! Recommend switching over. https://github.blog/changelog/2023-12-14-github-actions-artifacts-v4-is-now-generally-available/ v4 is a complete rewrite of the artifact actions with a new backend. v1-v3 uploads sometimes would hit 100% or close to 100% and things would just stop and fail due to mysterious reasons. Sometimes there would also be a small amount of transient errors like 500s or 400s that you see. v4 is all around more reliable, simpler and a host of issues described in this issue should no longer happen. If there are any similar issues with v4 then please open up new issues |
What happened?
https://github.com/darwinia-network/darwinia/runs/4295287117?check_suite_focus=true
What did you expect to happen?
Upload successful.
How can we reproduce it?
Not sure.
Anything else we need to know?
No response
What version of the action are you using?
v2
What are your runner environments?
linux
Are you on GitHub Enterprise Server? If so, what version?
No response
The text was updated successfully, but these errors were encountered: