Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Github actions run out of disk space while building ASAN #1497

Closed
empiredan opened this issue May 29, 2023 · 0 comments
Closed

Github actions run out of disk space while building ASAN #1497

empiredan opened this issue May 29, 2023 · 0 comments

Comments

@empiredan
Copy link
Contributor

empiredan commented May 29, 2023

Previously Build Release and Build with jemalloc failed due to running out of disk space (see #1485). Recently, Build ASAN also failed due to the same reason (Unhandled exception. System.IO.IOException: No space left on device):

image

This means we have to spare more space. And actually CMakeFiles occupied much disk space. By running find ./build/latest/src/ -name '*CMakeFiles*' -type d -exec du -csh "{}" + we found that typically it could consume 3.4GB:

8.0K	./build/latest/src/CMakeFiles
25M	./build/latest/src/aio/CMakeFiles
9.4M	./build/latest/src/aio/test/CMakeFiles
19M	./build/latest/src/base/CMakeFiles
3.6M	./build/latest/src/base/test/CMakeFiles
9.4M	./build/latest/src/block_service/CMakeFiles
9.1M	./build/latest/src/block_service/fds/CMakeFiles
11M	./build/latest/src/block_service/hdfs/CMakeFiles
11M	./build/latest/src/block_service/local/CMakeFiles
35M	./build/latest/src/block_service/test/CMakeFiles
45M	./build/latest/src/client/CMakeFiles
6.9M	./build/latest/src/client/test/CMakeFiles
29M	./build/latest/src/client_lib/CMakeFiles
80M	./build/latest/src/common/CMakeFiles
68M	./build/latest/src/common/test/CMakeFiles
16M	./build/latest/src/failure_detector/CMakeFiles
16M	./build/latest/src/failure_detector/test/CMakeFiles
8.0K	./build/latest/src/geo/CMakeFiles
7.6M	./build/latest/src/geo/lib/CMakeFiles
12M	./build/latest/src/geo/test/CMakeFiles
2.4M	./build/latest/src/geo/bench/CMakeFiles
26M	./build/latest/src/http/CMakeFiles
11M	./build/latest/src/http/test/CMakeFiles
306M	./build/latest/src/meta/CMakeFiles
294M	./build/latest/src/meta/test/CMakeFiles
19M	./build/latest/src/meta/test/balancer_simulator/CMakeFiles
8.9M	./build/latest/src/meta/test/meta_state/CMakeFiles
45M	./build/latest/src/nfs/CMakeFiles
6.9M	./build/latest/src/nfs/test/CMakeFiles
17M	./build/latest/src/perf_counter/CMakeFiles
12M	./build/latest/src/perf_counter/test/CMakeFiles
8.0K	./build/latest/src/redis_protocol/CMakeFiles
7.4M	./build/latest/src/redis_protocol/proxy/CMakeFiles
17M	./build/latest/src/redis_protocol/proxy_lib/CMakeFiles
8.0M	./build/latest/src/redis_protocol/proxy_ut/CMakeFiles
6.1M	./build/latest/src/remote_cmd/CMakeFiles
411M	./build/latest/src/replica/CMakeFiles
90M	./build/latest/src/replica/duplication/test/CMakeFiles
13M	./build/latest/src/replica/backup/test/CMakeFiles
14M	./build/latest/src/replica/bulk_load/test/CMakeFiles
15M	./build/latest/src/replica/split/test/CMakeFiles
8.0K	./build/latest/src/replica/storage/CMakeFiles
22M	./build/latest/src/replica/storage/simple_kv/CMakeFiles
74M	./build/latest/src/replica/storage/simple_kv/test/CMakeFiles
123M	./build/latest/src/replica/test/CMakeFiles
11M	./build/latest/src/reporter/CMakeFiles
113M	./build/latest/src/runtime/CMakeFiles
131M	./build/latest/src/runtime/test/CMakeFiles
78M	./build/latest/src/runtime/rpc/CMakeFiles
73M	./build/latest/src/runtime/task/CMakeFiles
57M	./build/latest/src/runtime/security/CMakeFiles
19M	./build/latest/src/runtime/ranger/CMakeFiles
344K	./build/latest/src/sample/CMakeFiles
213M	./build/latest/src/server/CMakeFiles
345M	./build/latest/src/server/test/CMakeFiles
105M	./build/latest/src/shell/CMakeFiles
2.9M	./build/latest/src/test_util/CMakeFiles
4.6M	./build/latest/src/test/bench_test/CMakeFiles
8.0K	./build/latest/src/test/function_test/CMakeFiles
13M	./build/latest/src/test/function_test/utils/CMakeFiles
7.5M	./build/latest/src/test/function_test/backup_restore_test/CMakeFiles
64M	./build/latest/src/test/function_test/base_api_test/CMakeFiles
5.6M	./build/latest/src/test/function_test/bulk_load_test/CMakeFiles
3.5M	./build/latest/src/test/function_test/detect_hotspot_test/CMakeFiles
6.0M	./build/latest/src/test/function_test/partition_split_test/CMakeFiles
5.3M	./build/latest/src/test/function_test/recovery_test/CMakeFiles
8.1M	./build/latest/src/test/function_test/restore_test/CMakeFiles
6.4M	./build/latest/src/test/function_test/throttle_test/CMakeFiles
24M	./build/latest/src/test/kill_test/CMakeFiles
4.7M	./build/latest/src/test/pressure_test/CMakeFiles
6.3M	./build/latest/src/tools/CMakeFiles
87M	./build/latest/src/utils/CMakeFiles
1.3M	./build/latest/src/utils/long_adder_bench/CMakeFiles
94M	./build/latest/src/utils/test/CMakeFiles
5.4M	./build/latest/src/utils/test/nth_element_bench/CMakeFiles
16M	./build/latest/src/zookeeper/CMakeFiles
7.9M	./build/latest/src/zookeeper/test/CMakeFiles
3.4G	total

Therefore we could drop CMakeFiles directories to spare more disk space.

empiredan added a commit that referenced this issue May 30, 2023
…out of disk space (#1498)

#1497

After `Build Release` and `Build with jemalloc` had failed due to running out of 
disk space (resolved by #1486)
for the branch `migrate-metrics-dev`, Github actions `Build ASAN` failed too due
to the same reason for the same branch.

It was found that `CMakeFiles` directories had occupied much disk space, about
3.4 GB. Once these directories were cleared, more space would be spared.

This problem is also tracked by another PR:
        #1491.
empiredan added a commit that referenced this issue May 31, 2023
…units (#1491)

#1490

For the enum classes, such as metric type and unit, in addition to the declaration
of all enumerators, extra code is needed to implement `enum_to_string` for each
enumerator. However, the "extra code" tended to be forgotten, just as what has
been said in the above issue #1490.

Therefore, we should find a way that could share all the enumerators between
the declaration and `enum_to_string`. Each enumerator would be written only
once, and there is no need to remember to add `enum_to_string` for each
enumerator.

We could implement the "sharing" by function-like macros. They have the names
of all the enumerators, and accept an argument to perform custom actions over
the names to implement the "sharing". Both metric types and units would be
refactored in this way.

While building ASAN Github actions failed due to running out of disk space.
This problem is resolved by dropping all directories of `CMakeFiles`, which is
also tracked another issue: 
        #1497
empiredan added a commit that referenced this issue Jun 5, 2023
…units (#1491)

#1490

For the enum classes, such as metric type and unit, in addition to the declaration
of all enumerators, extra code is needed to implement `enum_to_string` for each
enumerator. However, the "extra code" tended to be forgotten, just as what has
been said in the above issue #1490.

Therefore, we should find a way that could share all the enumerators between
the declaration and `enum_to_string`. Each enumerator would be written only
once, and there is no need to remember to add `enum_to_string` for each
enumerator.

We could implement the "sharing" by function-like macros. They have the names
of all the enumerators, and accept an argument to perform custom actions over
the names to implement the "sharing". Both metric types and units would be
refactored in this way.

While building ASAN Github actions failed due to running out of disk space.
This problem is resolved by dropping all directories of `CMakeFiles`, which is
also tracked another issue: 
        #1497
empiredan added a commit that referenced this issue Jun 21, 2023
…units (#1491)

#1490

For the enum classes, such as metric type and unit, in addition to the declaration
of all enumerators, extra code is needed to implement `enum_to_string` for each
enumerator. However, the "extra code" tended to be forgotten, just as what has
been said in the above issue #1490.

Therefore, we should find a way that could share all the enumerators between
the declaration and `enum_to_string`. Each enumerator would be written only
once, and there is no need to remember to add `enum_to_string` for each
enumerator.

We could implement the "sharing" by function-like macros. They have the names
of all the enumerators, and accept an argument to perform custom actions over
the names to implement the "sharing". Both metric types and units would be
refactored in this way.

While building ASAN Github actions failed due to running out of disk space.
This problem is resolved by dropping all directories of `CMakeFiles`, which is
also tracked another issue: 
        #1497
empiredan added a commit that referenced this issue Aug 9, 2023
…units (#1491)

#1490

For the enum classes, such as metric type and unit, in addition to the declaration
of all enumerators, extra code is needed to implement `enum_to_string` for each
enumerator. However, the "extra code" tended to be forgotten, just as what has
been said in the above issue #1490.

Therefore, we should find a way that could share all the enumerators between
the declaration and `enum_to_string`. Each enumerator would be written only
once, and there is no need to remember to add `enum_to_string` for each
enumerator.

We could implement the "sharing" by function-like macros. They have the names
of all the enumerators, and accept an argument to perform custom actions over
the names to implement the "sharing". Both metric types and units would be
refactored in this way.

While building ASAN Github actions failed due to running out of disk space.
This problem is resolved by dropping all directories of `CMakeFiles`, which is
also tracked another issue: 
        #1497
empiredan added a commit that referenced this issue Aug 11, 2023
…units (#1491)

#1490

For the enum classes, such as metric type and unit, in addition to the declaration
of all enumerators, extra code is needed to implement `enum_to_string` for each
enumerator. However, the "extra code" tended to be forgotten, just as what has
been said in the above issue #1490.

Therefore, we should find a way that could share all the enumerators between
the declaration and `enum_to_string`. Each enumerator would be written only
once, and there is no need to remember to add `enum_to_string` for each
enumerator.

We could implement the "sharing" by function-like macros. They have the names
of all the enumerators, and accept an argument to perform custom actions over
the names to implement the "sharing". Both metric types and units would be
refactored in this way.

While building ASAN Github actions failed due to running out of disk space.
This problem is resolved by dropping all directories of `CMakeFiles`, which is
also tracked another issue: 
        #1497
empiredan added a commit to empiredan/pegasus that referenced this issue Dec 6, 2023
…units (apache#1491)

apache#1490

For the enum classes, such as metric type and unit, in addition to the declaration
of all enumerators, extra code is needed to implement `enum_to_string` for each
enumerator. However, the "extra code" tended to be forgotten, just as what has
been said in the above issue apache#1490.

Therefore, we should find a way that could share all the enumerators between
the declaration and `enum_to_string`. Each enumerator would be written only
once, and there is no need to remember to add `enum_to_string` for each
enumerator.

We could implement the "sharing" by function-like macros. They have the names
of all the enumerators, and accept an argument to perform custom actions over
the names to implement the "sharing". Both metric types and units would be
refactored in this way.

While building ASAN Github actions failed due to running out of disk space.
This problem is resolved by dropping all directories of `CMakeFiles`, which is
also tracked another issue: 
        apache#1497
empiredan added a commit that referenced this issue Dec 11, 2023
…units (#1491)

#1490

For the enum classes, such as metric type and unit, in addition to the declaration
of all enumerators, extra code is needed to implement `enum_to_string` for each
enumerator. However, the "extra code" tended to be forgotten, just as what has
been said in the above issue #1490.

Therefore, we should find a way that could share all the enumerators between
the declaration and `enum_to_string`. Each enumerator would be written only
once, and there is no need to remember to add `enum_to_string` for each
enumerator.

We could implement the "sharing" by function-like macros. They have the names
of all the enumerators, and accept an argument to perform custom actions over
the names to implement the "sharing". Both metric types and units would be
refactored in this way.

While building ASAN Github actions failed due to running out of disk space.
This problem is resolved by dropping all directories of `CMakeFiles`, which is
also tracked another issue: 
        #1497
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants