Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix metric reporting error #18689

Open
wants to merge 1,157 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1157 commits
Select commit Hold shift + click to select a range
59474ed
Update versions to 305-SNAPSHOT
Xenorith Sep 11, 2023
3f16569
Enable sortpom after release and sort
Xenorith Sep 11, 2023
466fc05
Enable tests after release
Xenorith Sep 11, 2023
6165c0d
Remove hostname from metrics key
beinan Sep 11, 2023
5ad6e21
[DOCFIX] Update bin/alluxio usage format
Xenorith Sep 11, 2023
3c41091
Create DoraWorkerServiceHandlerTest.java
thu-david Sep 12, 2023
71f5708
Make fuse max reader concurrency configurable
JiamingMai Sep 12, 2023
fc7971c
Fix typo
JiamingMai Sep 12, 2023
c897a1b
[DOCFIX] Update bin/alluxio usage in ufs/compute
Xenorith Sep 12, 2023
6c97d6f
Support accessing OSS with proxy
Jackson-Wang-7 Sep 13, 2023
1f146a9
Fix default value of proxy host
Jackson-Wang-7 Sep 13, 2023
7328331
Extract path resolution in DoraCacheFS into static utility
jiacheliu3 Sep 13, 2023
7fc1040
[DOCFIX] Fix check-docs errMsg bug
Zijian-Zhu Sep 14, 2023
876d57f
Move path resolution logic and fix a bug
jiacheliu3 Sep 14, 2023
1fe33f7
Improve dora stream error handling
elega Sep 14, 2023
959c527
Add JSON-style reports
twalluxio Sep 14, 2023
9f34c2f
Refactor the launch in CLI to be more flexible
Kai-Zhang Sep 15, 2023
84dc05f
Write to UFS when failed to write to Alluxio Worker
JiamingMai Sep 15, 2023
6a9f5fd
Improve distributed load
elega Sep 15, 2023
100eb4f
Recover the ufs uri support
beinan Sep 15, 2023
f05deb3
[DOCFIX] Update docker documentation
ssz1997 Sep 15, 2023
8bba797
Add a test to monitor the distribution of vnode to physical node
yuzhu Sep 15, 2023
6f35645
Add CLI tools for debugging and analyzing caching issues
JiamingMai Sep 17, 2023
86308c3
Parse JSON-style reports in golang side
twalluxio Sep 18, 2023
838893d
Add datePredicate
ssyssy Sep 18, 2023
ddba020
Add throughput distribution and coarse datapoints for StressWorkerBench
twalluxio Sep 18, 2023
71fd8c4
Add Unit Test for OSS, OBS and GCS
Sep 18, 2023
db4fc2b
Fix datePredicate
jja725 Sep 19, 2023
dd863fe
Generate user cli doc
Xenorith Sep 19, 2023
d737d56
Fix the broken PagedDoraWorkerTest
YichuanSun Sep 19, 2023
814fad4
Fix `NoClassDefFoundError` UnderFileSystemFactory not loaded
dbw9580 Sep 19, 2023
4b9ab67
return an all zero fsstat
huanghua78 Sep 20, 2023
796b66f
Add back journal checkpoint command
Xenorith Sep 20, 2023
e21a7a3
[DOCFIX] Fix typo in generate docs cmd
Xenorith Sep 20, 2023
7b37a86
Enable config the virtual nodes num per node
JiamingMai Sep 20, 2023
765048c
Fix GCSV2UnderFileSystem initialization
jja725 Sep 21, 2023
6f707e1
Stop fuse web server
huanghua78 Sep 21, 2023
2e18cd2
Only stop processes belong to the current user
secfree Sep 21, 2023
d056937
Fix typo in ServiceDiscoveryRecipe
voddle Sep 21, 2023
3a82f8e
Fix the bug of using a terminated executor in S3 UFS
Jackson-Wang-7 Sep 22, 2023
82bd57b
Add new worker identity
dbw9580 Sep 22, 2023
bc4f95b
[DOCFIX] Describe how the user cli docs are generated
Xenorith Sep 22, 2023
325582e
Split the fuse stream close() method
elega Sep 22, 2023
4e2a494
Refactor the collectInfo main for future use
Kai-Zhang Sep 22, 2023
be6974c
Improve the random read behavior in StressWorkerBench
voddle Sep 22, 2023
ba7fedf
Parse JSON-style reports in golang side
twalluxio Sep 22, 2023
e3d7c80
Update collectAlluxioInfo and other bin/alluxio cmd strings
Xenorith Sep 22, 2023
e12f626
Refactor build logic for usability
tieujason330 Sep 22, 2023
c75317f
Fix last modified time is 0 in UfsFileIterable
jja725 Sep 22, 2023
e68c6d8
Fix and re-enable UfsFileIn/OutStreamTest
voddle Sep 23, 2023
189ca24
Enable static consistent hash ring
JiamingMai Sep 24, 2023
fe55fc9
Add Read Metrics
huanghua78 Sep 25, 2023
3dd49e4
Add Dora read fallback to UFS logics
LuQQiu Sep 25, 2023
a2d8410
Close Etcd client when process exits
jiacheliu3 Sep 25, 2023
9bfdc07
Fix read position in PagedDoraWorker
dbw9580 Sep 25, 2023
fc70ad2
Improve resource deallocation in FUSE
jiacheliu3 Sep 25, 2023
d8c0ea8
Move json formatting in info as util SKIPCI
Xenorith Sep 25, 2023
c11e4cb
Remove log
elega Sep 25, 2023
8d0c0ff
Remove master host check
jja725 Sep 25, 2023
9ab7391
Add test for DoraMetaManager
voddle Sep 26, 2023
527ffe8
Modify StressWorkerBench: calculate elapsed time in nanoseconds
twalluxio Sep 26, 2023
926f393
Add manual exit at the end of FUSE life cycle
jiacheliu3 Sep 26, 2023
dc0849a
Add project structure and template for native components
dbw9580 Sep 26, 2023
dec36be
Make copy & move return list of tasks
jja725 Sep 26, 2023
9047c62
Create a init test for cluster
thu-david Sep 27, 2023
cde8e4c
Fix check-cached
elega Sep 27, 2023
535d47b
[DOCFIX] Remove deprecated sections from journal docs
Xenorith Sep 27, 2023
b5e7b87
Modify go dependencies and code to be compatible with go1.15
Xenorith Sep 27, 2023
4777915
Add user/password support for etcd
lucyge2022 Oct 2, 2023
063a11a
Add membership module doc
lucyge2022 Oct 4, 2023
c394d65
Fail directly if sudo needs password
secfree Oct 4, 2023
9a88be2
Fix FileOutStreamIntegrationTest
voddle Oct 7, 2023
95046f7
Add UT to DefaultPageMetaStore.commitFile
YichuanSun Oct 7, 2023
28db253
Improve list files RESTful API providing more useful info
JiamingMai Oct 8, 2023
d4a39b5
Fix the UfsBaseFileSystemTest
YichuanSun Oct 8, 2023
8329244
Disable UFS fallback in DoraFileSystemIntegrationTest
YichuanSun Oct 8, 2023
cb74679
Fix Integration tests framework and DoraFileSystemIntegrationTest
YichuanSun Oct 9, 2023
1a4b2a9
Fix reconnect logic of servicediscoveryrecipe
lucyge2022 Oct 9, 2023
ceeeefe
Avoid creating FileSystem instance many times in HttpServerHandler
JiamingMai Oct 9, 2023
ee0f966
Remove MigrateIntegrationTest and PersistIntegrationTest
jasondrogba Oct 10, 2023
447a444
Fix FileSystemBlockLocationIntegrationTest
voddle Oct 10, 2023
bf997fe
Support set/get xattr
Jackson-Wang-7 Oct 10, 2023
75e31dc
Remove a few deprecated integration tests
YichuanSun Oct 10, 2023
3470e17
Support StressWorkerBench using consistent hash policy
twalluxio Oct 10, 2023
ba61600
Fix FileInStreamIntegrationTest
voddle Oct 10, 2023
b7a0a77
Add a property key to control if alluxio write the xattr to UFS
Jackson-Wang-7 Oct 10, 2023
a002643
Add UT of PagedFileReader
YichuanSun Oct 10, 2023
6ede390
Use linked hash set to keep sequence when picking multiple worker
jja725 Oct 10, 2023
f7ee2ad
Add ser/de support for JSON or WorkerIdentity
dbw9580 Oct 11, 2023
9ef97ec
Refactor handling of java opts env vars in cli
Xenorith Oct 11, 2023
ba20fd0
Use new worker id on worker and client
dbw9580 Oct 13, 2023
20965a0
Check load meta option at worker
secfree Oct 13, 2023
d8a12c7
[SMALLFIX] Enable WorkerMetadataSyncIntegrationTest
YichuanSun Oct 16, 2023
fae6922
[SMALLFIX] Enable one UT case in OutStreamTest
YichuanSun Oct 16, 2023
8e169df
Fix URIStatus notserializable bug
singer-bin Oct 16, 2023
7c3b462
Add RESTful API for distributed load
JiamingMai Oct 17, 2023
b597c61
Change CosNUnderFileSystemFactory implementation UnderFileSystemFactory
alluxio-bot Oct 17, 2023
1bdb1dd
test force push to main
apc999 Oct 17, 2023
b660e74
Revert "Add RESTful API for distributed load"
jja725 Oct 17, 2023
24a32d4
Load big file
jja725 Oct 18, 2023
f3494dc
Remove test file
apc999 Oct 18, 2023
7d222bb
Load ufs path directly
jja725 Oct 19, 2023
b0dc312
Create locationCommandTest and fix some IntegretionTest bugs
thu-david Oct 19, 2023
cd7f2e1
Fix load command ufs string
jja725 Oct 19, 2023
077c372
Fix FileSystemShellUtilsTest
YichuanSun Oct 19, 2023
b5616c0
[SMALLFIX] Remove OldLoadCommandIntegrationTest
YichuanSun Oct 19, 2023
fe98953
[SMALLFIX] Delete JobServiceFaultToleranceShellTest
YichuanSun Oct 19, 2023
908b7d6
Add name to artifacts metadata
tieujason330 Oct 19, 2023
62cc17a
Implement HDFS list status iterator
JiamingMai Oct 20, 2023
47b7667
Add RESTful API for distributed load
JiamingMai Oct 20, 2023
3c851d7
Add multi-dimensional metrics system
Kai-Zhang Oct 20, 2023
013c9e1
Change UfsManager key to not rely on properties
jiacheliu3 Oct 23, 2023
3a47fa2
Use new UfsManager.get() avoid creating conf instances every time
jiacheliu3 Oct 23, 2023
6bfbbd6
[SMALLFIX] Remove UnsetTtlTest
YichuanSun Oct 23, 2023
0884fc3
Remove restriction in overwriting worker info within join logic
lucyge2022 Oct 23, 2023
dc12c11
Improve cold read performance by prefetching data on workers
elega Oct 24, 2023
e78b66d
Add prefetch cache policy
elega Oct 24, 2023
5b0e661
Add getStatus RESTful API
JiamingMai Oct 25, 2023
08b9456
Fix MemoryPageStore error on commit() and clear cache on close()
jiacheliu3 Oct 25, 2023
e830663
Bring back integration tests
YichuanSun Oct 25, 2023
bb8e2cf
Remove WorkerFuseIntegrationTest
YichuanSun Oct 26, 2023
6185ba0
Delete useless UnderFileSystemAlluxioTest
YichuanSun Oct 26, 2023
7a5734f
Update worker api support for load multi replicas
jja725 Oct 26, 2023
53c49f7
Add Rust toolchain to CI
dbw9580 Oct 26, 2023
d0ad98f
Add regx pattern file filter for distributed load
JiamingMai Oct 26, 2023
8f41c65
Create DoraLsCommandIntegrationTest
thu-david Oct 26, 2023
961ed17
[SMALLFIX] Extract create UfsBaseFileSystem for extension
jiacheliu3 Oct 27, 2023
14efca9
Fix a buffer memory leak in NettyReadHandlerStateMachine
jiacheliu3 Oct 27, 2023
d3bfd8c
[SMALLFIX] Remove LeaderCommandIntegrationTest.java
YichuanSun Oct 27, 2023
779ece2
[SMALLFIX] Remove NeedsSyncCommandIntegrationTest.java
YichuanSun Oct 27, 2023
37b33dc
[SMALLFIX] Recover HelpCommandIntegrationTest
YichuanSun Oct 27, 2023
eecf2a5
Add rust spdk library
linzs148 Oct 30, 2023
32f675f
Update Load replica job
jja725 Oct 30, 2023
660e00d
Fix the time check logic for judging stale client channels to be inac…
liuxiaohu9527 Oct 31, 2023
ec4868d
Fix isAllPageCached
elega Oct 31, 2023
2114bea
Change the metrics registry and some metrics
Kai-Zhang Nov 1, 2023
ccf20e4
Move job related operations to DoraCacheFileSystem
elega Nov 1, 2023
0e83207
Refactor PagedDoraWorker by injecting MetaManager and UfsManager
jiacheliu3 Nov 1, 2023
ad3baa1
Add remote only policy to StressWorkerBench
twalluxio Nov 2, 2023
b2df584
Fix the bug that command line doesn't support regx file filter
JiamingMai Nov 2, 2023
d8342f7
Fix page store TTL metrics
beinan Nov 3, 2023
24ff6b5
Fix file mtime display when ufs mtime is not null
lfyzjck Nov 3, 2023
ea6af23
Disable xatter in PagedDoraWorker test
voddle Nov 6, 2023
c7c0820
Fix UT readMultipleFiles in LocalCacheFileInStreamTest
jasondrogba Nov 6, 2023
d766ae4
Improve distributed load
elega Nov 6, 2023
741563b
[SMALLFIX] Allow extension on `UfsManager.tryUfsFileSystem()`
jiacheliu3 Nov 6, 2023
bad173c
Fix COS/OSS/OBS iterative listing
elega Nov 6, 2023
912731c
Print warning messages for stale open handle instead of removing it
huanghua78 Nov 7, 2023
8d3ab67
Fix cosn ufs cannot find class
maobaolong Nov 7, 2023
00a527d
Change logging level in NettyChannelPool
apc999 Nov 7, 2023
723c720
Add a configurable DLQ capacity of avoid OOM
elega Nov 7, 2023
d28559a
Fix the issue to get a tag of a non-existed in OSS/COS
Jackson-Wang-7 Nov 7, 2023
420e560
Fix NPE
elega Nov 7, 2023
7d15de7
Fix chaotic chinese characters with golang cli commands
twalluxio Nov 7, 2023
bd99175
[SMALLFIX] Remove unused objects initialize
lfyzjck Nov 7, 2023
246c7ee
Improve UpdateChecker
apc999 Nov 7, 2023
18c2190
Fix page store TTL metrics (#18363)
beinan Nov 8, 2023
5474513
Fix StressWorkerBench with remote worker policy
twalluxio Nov 8, 2023
502b0dd
Fix integration tests again
Xenorith Nov 8, 2023
615a6d8
cherry pick empty because Tencent Cloud EMR doc have been removed in …
wheatxiong Jul 10, 2023
8eba127
Close client when heartbeat executor closed
maobaolong Jul 17, 2023
704a460
Fix a deadlock in FileSystemContext
apc999 Sep 5, 2023
6be7125
Fix local cache identifier to solve the local cache consistent issue …
secfree Jul 3, 2023
c453859
Support getFileBlockLocation in LocalCacheFileSystem
maobaolong Jul 24, 2023
1e283df
Support read bytebuffer for non ByteBufferReadable input stream
maobaolong Sep 8, 2023
d0d6b0c
[SMALLFIX] Fix a checkstyle issue from resolving a conflict
jiacheliu3 Nov 8, 2023
dd2d0f9
Use correct audit log configuration in proxy
Haoning-Sun Nov 4, 2023
f474bd7
Support Mkdir/CreateFile with configured default umask in HDFS API
Jackson-Wang-7 Oct 10, 2023
b64b0cf
cherry pick without resolving conflicts: Get User/Group/Permission of…
Jackson-Wang-7 Oct 11, 2023
a5d33f9
resolve cherry-pick conflicts
jiacheliu3 Nov 8, 2023
3350541
[SMALLFIX] Fix import order on OSSUnderFileSystem for checkstyle
jiacheliu3 Nov 8, 2023
c9760b8
[SMALLFIX] Fix broken ChmodCommandIntegrationTest
YichuanSun Nov 8, 2023
6dc2fae
Use generic Exception to catch all exceptions in getStatus()
huanghua78 Nov 8, 2023
c2f4cd2
Merge master-2.x commits 2023/07/01~2023/11/08 into main
alluxio-bot Nov 8, 2023
1c279c6
Fix the NPE in COSUnderFileSystem
Jackson-Wang-7 Nov 8, 2023
b0f803a
Update version to 305
Xenorith Nov 8, 2023
d07ca76
Update version to 306-SNAPSHOT
Xenorith Nov 8, 2023
2e1f95d
Remove malformed empty page files on access
dbw9580 Nov 9, 2023
69ca11b
Replace the FOLDER_SUFFIX from "_$folder$" to "/" in OSS UFS
Jackson-Wang-7 Nov 9, 2023
f7225f4
Fix stress worker bench remote only policy not working
twalluxio Nov 9, 2023
1852423
Change file exist exception to runtime exception in PagedDoraWorker
voddle Nov 9, 2023
d9cbbf1
Translate Introduction
xiong-make Nov 10, 2023
5421aa4
Improve page loading performance
elega Nov 10, 2023
328ee8c
Allow rename() to overwrite, fix an uncaught ex and reenable UT
voddle Nov 10, 2023
ba02104
[SMALLFIX] Remove invalid Property WORKER_FUSE_ENABLED
YichuanSun Nov 11, 2023
08547a5
Add call points of the new metrics
Kai-Zhang Nov 13, 2023
f6e4260
Update README.md
computerscienceiscool Nov 13, 2023
feb0116
Clean up fuse unmount workflow
lucyge2022 Nov 14, 2023
3a29f8d
Fix the issue when mount an empty oss bucket
Jackson-Wang-7 Nov 14, 2023
d2d29df
Fix the bug that HdfsUfsStatusIterator returns wrong path
JiamingMai Nov 17, 2023
f7eed03
Fix the bug that HDFS iterator doesn't work
JiamingMai Nov 20, 2023
24e4d3d
Fix duplicated class and resources in presto mvn build
beinan Nov 29, 2023
50fd48f
Update version to 306
alluxio-bot Nov 29, 2023
cacc1cc
Update version to 307-SNAPSHOT
alluxio-bot Nov 29, 2023
15b05ba
Add WorkerClusterView
dbw9580 Dec 1, 2023
0db1ff7
Fix the shading issue for javax security
beinan Dec 4, 2023
7223e80
Update version to 307
alluxio-bot Dec 5, 2023
e6ac3b8
Update version to 308-SNAPSHOT
alluxio-bot Dec 5, 2023
4ab2acf
Make proxy transfer buffer size configurable
ddrid Dec 6, 2023
73d1746
Make WorkerInfo enum (and some small refactors)
dbw9580 Dec 8, 2023
22af1b1
Fix to disable worker identity reuse for registration unless in k8s
lucyge2022 Dec 9, 2023
9eae1e9
Use cluster view to replace list of workers
dbw9580 Dec 12, 2023
9902e8a
Fix outdated worker address info returned by consistent hash policy
dbw9580 Dec 14, 2023
4e3a722
Improve rest load api response JSON
JiamingMai Dec 14, 2023
4e39eda
Fix alluxio-fuse script for finding fuse pid
lucyge2022 Dec 14, 2023
b5a2895
Support get page with RESTful API by specifying offset and length
JiamingMai Dec 21, 2023
726f812
Support write page with RESTful API
JiamingMai Jan 4, 2024
9ef7552
Add fake numbers for statfs
huanghua78 Jan 8, 2024
b9de24c
Support Various Kinds of Consistent Hash
Jan 10, 2024
7d8f317
[DOCFIX] Update kerberos configuration 'alluxio.hadoop.kerberos.keyta…
QianyongY Jan 11, 2024
e4da0e4
Create metadata directory in initiateMultipartUpload method
Haoning-Sun Jan 11, 2024
4f07b99
Fix entrypoint
ssz1997 Jan 13, 2024
6f2a527
Fix the issue of S3 range read
007DXR Jan 19, 2024
97f325a
Fix negative size if specified filesize over Integer.MAX_VALUE for st…
maobaolong Jan 19, 2024
055b375
Add ServiceRegistry membership manager which only include active workers
jja725 Jan 25, 2024
262b2b3
Add http server port in the worker net address
ChunxuTang Jan 31, 2024
05e089b
Update version to 308
alluxio-bot Feb 1, 2024
e9a32c9
Update version to 309-SNAPSHOT
alluxio-bot Feb 1, 2024
07ec4b9
Fix DoraLoadCommandIntegrationTest
secfree Feb 1, 2024
3d13072
Support nullable offset and length in get page RESTful API
JiamingMai Feb 2, 2024
2f9bb6a
Update version to 309
alluxio-bot Feb 13, 2024
59bfee0
Update version to 310-SNAPSHOT
alluxio-bot Feb 13, 2024
c34921d
Change default value of virtual node count per worker
jja725 Feb 15, 2024
a3ecbc7
Improve the local cache fallback behavior for corrupted pages
beinan Feb 23, 2024
7429c25
Fix runtime stats when cache hit
beinan Feb 23, 2024
fa279a9
Update version to 310
alluxio-bot Feb 23, 2024
2d75444
Update version to 311-SNAPSHOT
alluxio-bot Feb 23, 2024
de7e92b
Fix Unrecognized error
jja725 Feb 24, 2024
1bdd219
Add decommission capability and removenode cmd
lucyge2022 Feb 29, 2024
81bc9dd
Fix code injection admin
lucyge2022 Mar 4, 2024
0fb4043
Fix libjnifuse.so build
lucyge2022 Mar 5, 2024
1b92066
Update version to 311
alluxio-bot Mar 11, 2024
85663ed
Update version to 312-SNAPSHOT
alluxio-bot Mar 11, 2024
aac8a61
Support running on early-access JDK
wendigo Mar 22, 2024
fd5478e
Fix etcd calls exception handling and adjust failure detection timeout
lucyge2022 Mar 28, 2024
4371afa
Fix local cache ttl
jja725 Apr 8, 2024
6039267
Update version to 312
alluxio-bot Apr 8, 2024
6e5a618
Update version to 313-SNAPSHOT
alluxio-bot Apr 8, 2024
36b0b16
Name threads used by LocalCacheManager
findepi Apr 11, 2024
b6dbd59
Update the logic of Alluxio clearing expired metadata
Apr 25, 2024
541f247
Add http getPage cache hit ratio
LuQQiu Apr 30, 2024
0d6b57e
Install netcat in Dockerfile
ssz1997 May 6, 2024
cf5ebda
Add http server health check
ssz1997 May 10, 2024
db91042
Add health check RESTful API
JiamingMai May 10, 2024
40ebdd0
Fix metrics when delete non existing page
jja725 Jun 7, 2024
19aa8f0
Update version to 313
alluxio-bot Jun 7, 2024
efed93c
Update version to 314-SNAPSHOT
alluxio-bot Jun 7, 2024
05d2537
Add back shaded netty within grpc
jja725 Jul 1, 2024
6d99656
Install ucx on dockerfile jdk8
LuQQiu Jul 25, 2024
d04d14d
Fix metric reporting error
Sep 6, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
The diff you're trying to view is too large. We only load the first 3000 changed files.
15 changes: 7 additions & 8 deletions .github/workflows/checkstyle.yml
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
name: Checkstyle, Findbugs, Doc Check, etc.
name: Checkstyle, Findbugs, Doc Check, SortPom, etc.

on: [pull_request]

Expand All @@ -8,40 +8,39 @@ jobs:

steps:
- name: checkout repo
uses: actions/checkout@v2
uses: actions/checkout@v3

- name: Use Node.js ${{ matrix.node-version }}
uses: actions/setup-node@v1
with:
node-version: '10.11.0'

- name: Cache local Maven repository
uses: actions/cache@v2
uses: actions/cache@v3
with:
path: ~/.m2/repository
key: ${{ runner.os }}-maven-${{ hashFiles('**/pom.xml') }}
restore-keys: |
${{ runner.os }}-maven-

- name: Cache local Go modules
uses: actions/cache@v2
uses: actions/cache@v3
with:
path: ~/go/pkg/mod
key: ${{ runner.os }}-gomod-${{ hashFiles('**/go.mod') }}

- name: Execute license check, checkstyle, findbugs
- name: Execute license check, checkstyle, findbugs, sortpom
run: |
mkdir -p ~/.m2
ALLUXIO_DOCKER_NO_TTY=true \
ALLUXIO_DOCKER_GIT_CLEAN=true \
ALLUXIO_DOCKER_MVN_RUNTOEND=true \
ALLUXIO_CHECKSTYLE=true \
dev/github/run_docker.sh "\"-Dtest=${{ matrix.modules }}\"" -pl tests
dev/github/run_docker.sh
timeout-minutes: 60

- name: Archive artifacts
continue-on-error: true
uses: actions/upload-artifact@v2
uses: actions/upload-artifact@v3
if: always()
with:
name: artifact
Expand Down
61 changes: 0 additions & 61 deletions .github/workflows/fuse_integration_tests.yml

This file was deleted.

66 changes: 66 additions & 0 deletions .github/workflows/java8_container_tests.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
name: Java 8 Container Tests

# This build runs integration tests using testcontainers
# The user executing the test command must have access to a running docker daemon

on: [pull_request]

jobs:
build:
name: "modules: "

strategy:
fail-fast: false
matrix:
modules:
- >-
alluxio.membership.**

runs-on: ubuntu-latest
if: "!contains(github.event.pull_request.title, 'DOCFIX') &&
!contains(github.event.pull_request.title, 'SKIPCI')"

steps:
- name: checkout repo
uses: actions/checkout@v3

- name: Use Node.js ${{ matrix.node-version }}
uses: actions/setup-node@v1
with:
node-version: '10.11.0'

- name: Cache local Maven repository
uses: actions/cache@v3
with:
path: ~/.m2/repository
key: ${{ runner.os }}-maven-java11-${{ hashFiles('**/pom.xml') }}

- name: Cache local Go modules
uses: actions/cache@v3
with:
path: ~/go/pkg/mod
key: ${{ runner.os }}-gomod-${{ hashFiles('**/go.mod') }}

- name: Run tests
id: test0
run: |
mkdir -p ~/.m2
ALLUXIO_DOCKER_NO_TTY=true \
ALLUXIO_DOCKER_GIT_CLEAN=true \
ALLUXIO_DOCKER_ID=0 \
ALLUXIO_DOCKER_MVN_PROJECT_LIST=dora/tests/testcontainers \
ALLUXIO_DOCKER_MVN_TESTS=${{ matrix.modules }} \
dev/github/run_docker.sh
timeout-minutes: 60

- name: Archive artifacts
continue-on-error: true
uses: actions/upload-artifact@v3
if: always()
with:
name: artifact
path: |
**/target/surefire-reports/*
**/target/artifacts/*
**/target/logs/*
retention-days: 7
32 changes: 17 additions & 15 deletions .github/workflows/java8_integration_tests.yml
Original file line number Diff line number Diff line change
@@ -1,5 +1,11 @@
name: Java 8 Integration Tests

# This build runs the default integration tests in the dora/tests module
# It runs all the tests besides those categorized as:
# - Fault tolerance = alluxio.server.ft
# - Web UI = alluxio.web
# - FUSE = alluxio.client.fuse

on: [pull_request]

jobs:
Expand All @@ -11,15 +17,11 @@ jobs:
matrix:
modules:
- >-
alluxio.client.cli.**,!alluxio.client.cli.fs.**
- >-
alluxio.client.cli.fs.**
- >-
alluxio.client.fs.**,!alluxio.client.fs.concurrent.**,!alluxio.client.fs.io.**
alluxio.client.cli.**
- >-
alluxio.client.fs.concurrent.**,alluxio.client.fs.io.**
alluxio.client.fs.**
- >-
alluxio.client.**,!alluxio.client.fs.**,!alluxio.client.cli.**,!alluxio.client.fuse.**
alluxio.client.**,!alluxio.client.fs.**,!alluxio.client.cli.**
- >-
alluxio.job.**,alluxio.master.**,alluxio.stress.**
- >-
Expand All @@ -31,21 +33,21 @@ jobs:

steps:
- name: checkout repo
uses: actions/checkout@v2
uses: actions/checkout@v3

- name: Use Node.js ${{ matrix.node-version }}
uses: actions/setup-node@v1
with:
node-version: '10.11.0'

- name: Cache local Maven repository
uses: actions/cache@v2
uses: actions/cache@v3
with:
path: ~/.m2/repository
key: ${{ runner.os }}-maven-java8-${{ hashFiles('**/pom.xml') }}
key: ${{ runner.os }}-maven-java11-${{ hashFiles('**/pom.xml') }}

- name: Cache local Go modules
uses: actions/cache@v2
uses: actions/cache@v3
with:
path: ~/go/pkg/mod
key: ${{ runner.os }}-gomod-${{ hashFiles('**/go.mod') }}
Expand All @@ -56,14 +58,14 @@ jobs:
mkdir -p ~/.m2
ALLUXIO_DOCKER_NO_TTY=true \
ALLUXIO_DOCKER_GIT_CLEAN=true \
ALLUXIO_DOCKER_MVN_RUNTOEND=true \
ALLUXIO_DOCKER_MVN_PROJECT_LIST=!webui,!shaded/client,!integration/tools/hms,!integration/yarn,!assembly/client,!assembly/server,!table/server/underdb/glue,!underfs/hdfs,!underfs/ozone,!underfs/adl,!underfs/abfs,!underfs/cosn,!underfs/wasb,!underfs/cos,!underfs/kodo,!underfs/oss,!underfs/swift \
dev/github/run_docker.sh "\"-Dtest=${{ matrix.modules }}\"" -pl tests
ALLUXIO_DOCKER_MVN_PROJECT_LIST=dora/tests/integration \
ALLUXIO_DOCKER_MVN_TESTS=${{ matrix.modules }} \
dev/github/run_docker.sh
timeout-minutes: 60

- name: Archive artifacts
continue-on-error: true
uses: actions/upload-artifact@v2
uses: actions/upload-artifact@v3
if: always()
with:
name: artifact
Expand Down
21 changes: 13 additions & 8 deletions .github/workflows/java8_integration_tests_ft.yml
Original file line number Diff line number Diff line change
@@ -1,5 +1,10 @@
name: Java 8 Fault Tolerant Integration Tests

# This build runs the fault tolerance specific integration tests in the dora/tests module
# The key differences between this build and the default integration test build are
# - Only runs integration tests with prefix alluxio.server.ft
# - Sets fork count to 1 to prevent flakey errors due to running concurrent tests; default count is 2

on: [pull_request]

jobs:
Expand All @@ -21,21 +26,21 @@ jobs:

steps:
- name: checkout repo
uses: actions/checkout@v2
uses: actions/checkout@v3

- name: Use Node.js ${{ matrix.node-version }}
uses: actions/setup-node@v1
with:
node-version: '10.11.0'

- name: Cache local Maven repository
uses: actions/cache@v2
uses: actions/cache@v3
with:
path: ~/.m2/repository
key: ${{ runner.os }}-maven-java8-${{ hashFiles('**/pom.xml') }}
key: ${{ runner.os }}-maven-java11-${{ hashFiles('**/pom.xml') }}

- name: Cache local Go modules
uses: actions/cache@v2
uses: actions/cache@v3
with:
path: ~/go/pkg/mod
key: ${{ runner.os }}-gomod-${{ hashFiles('**/go.mod') }}
Expand All @@ -48,14 +53,14 @@ jobs:
ALLUXIO_DOCKER_FORK_COUNT=1 \
ALLUXIO_DOCKER_NO_TTY=true \
ALLUXIO_DOCKER_GIT_CLEAN=true \
ALLUXIO_DOCKER_MVN_RUNTOEND=true \
ALLUXIO_DOCKER_MVN_PROJECT_LIST=!webui,!shaded/client,!integration/tools/hms,!integration/yarn,!assembly/client,!assembly/server,!table/server/underdb/glue,!underfs/hdfs,!underfs/ozone,!underfs/adl,!underfs/abfs,!underfs/cosn,!underfs/wasb,!underfs/cos,!underfs/kodo,!underfs/oss,!underfs/swift \
dev/github/run_docker.sh "\"-Dtest=${{ matrix.modules }}\"" -pl tests
ALLUXIO_DOCKER_MVN_PROJECT_LIST=dora/tests/integration \
ALLUXIO_DOCKER_MVN_TESTS=${{ matrix.modules }} \
dev/github/run_docker.sh
timeout-minutes: 60

- name: Archive artifacts
continue-on-error: true
uses: actions/upload-artifact@v2
uses: actions/upload-artifact@v3
if: always()
with:
name: artifact
Expand Down
23 changes: 14 additions & 9 deletions .github/workflows/java8_integration_tests_webui.yml
Original file line number Diff line number Diff line change
@@ -1,5 +1,10 @@
name: Java 8 Integration Tests (w/ webui)

# This build runs the integration tests in the dora/tests module that require the web UI
# The key differences between this build and the default integration test build are
# - Only runs integration tests with prefix alluxio.web
# - Adds webui to the project list

on: [pull_request]

jobs:
Expand All @@ -19,21 +24,21 @@ jobs:

steps:
- name: checkout repo
uses: actions/checkout@v2
uses: actions/checkout@v3

- name: Use Node.js ${{ matrix.node-version }}
uses: actions/setup-node@v1
with:
node-version: '10.11.0'

- name: Cache local Maven repository
uses: actions/cache@v2
uses: actions/cache@v3
with:
path: ~/.m2/repository
key: ${{ runner.os }}-maven-java8-${{ hashFiles('**/pom.xml') }}
key: ${{ runner.os }}-maven-java11-${{ hashFiles('**/pom.xml') }}

- name: Cache local Go modules
uses: actions/cache@v2
uses: actions/cache@v3
with:
path: ~/go/pkg/mod
key: ${{ runner.os }}-gomod-${{ hashFiles('**/go.mod') }}
Expand All @@ -44,14 +49,14 @@ jobs:
mkdir -p ~/.m2
ALLUXIO_DOCKER_NO_TTY=true \
ALLUXIO_DOCKER_GIT_CLEAN=true \
ALLUXIO_DOCKER_MVN_RUNTOEND=true \
ALLUXIO_DOCKER_MVN_PROJECT_LIST=!shaded/client,!integration/tools/hms,!integration/yarn,!assembly/client,!assembly/server \
dev/github/run_docker.sh "\"-Dtest=${{ matrix.modules }}\"" -pl tests
ALLUXIO_DOCKER_MVN_PROJECT_LIST=dora/tests/integration,webui \
ALLUXIO_DOCKER_MVN_TESTS=${{ matrix.modules }} \
dev/github/run_docker.sh
timeout-minutes: 60

- name: Archive artifacts
continue-on-error: true
uses: actions/upload-artifact@v2
uses: actions/upload-artifact@v3
if: always()
with:
name: artifact
Expand Down
Loading