-
Notifications
You must be signed in to change notification settings - Fork 2.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support Hash Algorithms Benchmark #18539
base: main
Are you sure you want to change the base?
Commits on Sep 5, 2023
-
[DOCFIX] Update journal command doc
### What changes are proposed in this pull request? Please outline the changes and how this PR fixes the issue. ### Why are the changes needed? Please clarify why the changes are needed. For instance, 1. If you propose a new API, clarify the use case for a new API. 2. If you fix a bug, describe the bug. ### Does this PR introduce any user facing changes? Please list the user-facing changes introduced by your change, including 1. change in user-facing APIs 2. addition or removal of property keys 3. webui pr-link: Alluxio#18113 change-id: cid-586e7941f2f4b737f0b0ebe7c15cb13d182e6d0c
Configuration menu - View commit details
-
Copy full SHA for 8bc9a9e - Browse repository at this point
Copy the full SHA 8bc9a9eView commit details
Commits on Sep 6, 2023
-
Return FileNotFound when a file is being create/write
### What changes are proposed in this pull request? Return FileNotFound when a file is being create/write. No need to refresh its metadata. Metadata will be refreshed when file is closed. ### Why are the changes needed? If a file is being written to, and another getStatus() comes for such file, this file probably has not been created in UFS. At that time, the getStatus() thinks that this file does not exists. So the worker tries to remove its metadata and its data from cache. ERROR logs were generated in such case. This patch is to handle this correctly and remove such ERROR logs: ``` 2023-09-05 17:16:32,802 ERROR LocalCacheManager - Failed to delete page PageId{FileId=f4b0d1f405d7419069129a1262eae9329e9c19ed2584fc3fe27656a698344047, PageIndex=0} (isTemporary: false) from pageStore. alluxio.exception.PageNotFoundException: /Volumes/ramdisk/LOCAL/1048576/824/f4b0d1f405d7419069129a1262eae9329e9c19ed2584fc3fe27656a698344047/0 at alluxio.client.file.cache.store.LocalPageStore.delete(LocalPageStore.java:135) at alluxio.client.file.cache.LocalCacheManager.deletePage(LocalCacheManager.java:889) at alluxio.client.file.cache.LocalCacheManager.delete(LocalCacheManager.java:694) at alluxio.client.file.cache.LocalCacheManager.delete(LocalCacheManager.java:706) at alluxio.client.file.cache.LocalCacheManager.lambda$deleteFile$5(LocalCacheManager.java:839) at java.lang.Iterable.forEach(Iterable.java:75) at alluxio.client.file.cache.LocalCacheManager.deleteFile(LocalCacheManager.java:839) at alluxio.client.file.cache.NoExceptionCacheManager.deleteFile(NoExceptionCacheManager.java:209) at alluxio.worker.dora.DoraMetaManager.invalidateCachedFile(DoraMetaManager.java:337) at alluxio.worker.dora.DoraMetaManager.removeFromMetaStore(DoraMetaManager.java:178) at alluxio.worker.dora.DoraMetaManager.loadFromUfs(DoraMetaManager.java:131) at alluxio.worker.dora.PagedDoraWorker.getGrpcFileInfo(PagedDoraWorker.java:392) at alluxio.worker.grpc.DoraWorkerClientServiceHandler.getStatus(DoraWorkerClientServiceHandler.java:201) at alluxio.grpc.BlockWorkerGrpc$MethodHandlers.invoke(BlockWorkerGrpc.java:1589) at io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:182) at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:355) at io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:867) at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) ``` ### Does this PR introduce any user facing changes? No pr-link: Alluxio#18108 change-id: cid-47d15fcde921a539a6e3c64e7ac25fc400f26034
Configuration menu - View commit details
-
Copy full SHA for 0e3201e - Browse repository at this point
Copy the full SHA 0e3201eView commit details -
Add isReadOnly() for FuseFileStream
### What changes are proposed in this pull request? Add isReadOnly() for FuseFileStream and implement this interface in various Streams. ### Why are the changes needed? This interface is used to determine if the Stream is only. This is needed to determine if a real data flush is needed. ### Does this PR introduce any user facing changes? N/A pr-link: Alluxio#18114 change-id: cid-6a60e9e917bb0f8a439d76b51beaaba2b723dde4
Configuration menu - View commit details
-
Copy full SHA for 55a0e48 - Browse repository at this point
Copy the full SHA 55a0e48View commit details -
Run launch-process-bash script
Script `launch-process` was renamed to `launch-process-bash`. Update the name of the script in `entrypoint.sh` pr-link: Alluxio#18115 change-id: cid-7053532e9fac93ef3f987e1525ae7260ac3a8dc0
Configuration menu - View commit details
-
Copy full SHA for f2355ca - Browse repository at this point
Copy the full SHA f2355caView commit details
Commits on Sep 7, 2023
-
Add a config to turn on/off the sdk cache fallback
### What changes are proposed in this pull request? Add a config to turn on/off the sdk cache fallback ### Why are the changes needed? the read fallback should be turn off by default to avoid retry storm to UFS ### Does this PR introduce any user facing changes? Addition of property keys pr-link: Alluxio#18099 change-id: cid-2c893a617ac5e90086db98610434db6d763c1fe9
Configuration menu - View commit details
-
Copy full SHA for bab3538 - Browse repository at this point
Copy the full SHA bab3538View commit details -
Add dora worker metrics at http requests
One more place to count the cache hit requests pr-link: Alluxio#17878 change-id: cid-0cb55b7ca627832ce2b53e80ea58cfd6a2730bfa
Configuration menu - View commit details
-
Copy full SHA for bdd3b61 - Browse repository at this point
Copy the full SHA bdd3b61View commit details -
Fix the array out of bound exception in presto
### What changes are proposed in this pull request? Fix the array out of bound exception in presto ### Why are the changes needed? Because we saw earlier that the LocalCacheManager swallows all exceptions, if there is an error accessing one of the cached files, the exception will be ignored. However, the offset may have changed during the failed read attempt in LocalCacheManager. When LocalCacheManager returns -1 on error and tries to reread from the lower layer storage, the offset could be out of bounds. Regarding why this issue only appears in versions 2.9.3 and later, it's because the offset became a member variable of our target buffer starting from 2.9.3. In earlier versions it was always a local variable, so there was no compounding of offset errors. ### Does this PR introduce any user facing changes? no pr-link: Alluxio#18098 change-id: cid-c7b949d9146847b14ba672df56e21bc4bc5ad705
Configuration menu - View commit details
-
Copy full SHA for b8a8826 - Browse repository at this point
Copy the full SHA b8a8826View commit details -
Add dora worker metadata metrics
Count the hit and miss of ListStatus and GetFileInfo in worker pr-link: Alluxio#17848 change-id: cid-97c8efb81088d7e9b636ef927fcbc62a166fe085
Configuration menu - View commit details
-
Copy full SHA for 3d2eb7e - Browse repository at this point
Copy the full SHA 3d2eb7eView commit details
Commits on Sep 8, 2023
-
Change metric type of LocalCacheState from counter to gauge
### What changes are proposed in this pull request? Change metric type of LocalCacheState from counter to gauge. ### Why are the changes needed? The type of LocalCacheState metric is not reasonable since the value of this metric is a enum type not an increasable value ### Does this PR introduce any user facing changes? No pr-link: Alluxio#18070 change-id: cid-61037d3580e2f25f782e1c876ced0ecad5b312a4
Configuration menu - View commit details
-
Copy full SHA for 6a0e1c5 - Browse repository at this point
Copy the full SHA 6a0e1c5View commit details -
Change name of NoopMembershipManager
### What changes are proposed in this pull request? Change name of NoopMembershipManager to MasterMembershipManager. ### Why are the changes needed? N/A ### Does this PR introduce any user facing changes? N/A pr-link: Alluxio#18097 change-id: cid-9f39e3b1a9b8cdad51e49bbb5d12e86d21b07e23
Configuration menu - View commit details
-
Copy full SHA for 1422109 - Browse repository at this point
Copy the full SHA 1422109View commit details -
Revert "Change metric type of LocalCacheState from counter to gauge"
Reverts Alluxio#18070 revert per request on backward compatibility pr-link: Alluxio#18119 change-id: cid-aeecbff3bc9d6bff41eec25405e87a2f2b79079f
Configuration menu - View commit details
-
Copy full SHA for 72de4e9 - Browse repository at this point
Copy the full SHA 72de4e9View commit details -
### What changes are proposed in this pull request? Refactor the Netty read handler of worker. ### Why are the changes needed? The previous implementation creates a state machine per read request, instead of per channel. This implies that if two read requests are sent over the same channel, the worker would possibly use one channel to send data of different files or regions. This can lead to data corruption. This PR proposes to use a state machine per channel, and handles channel events throughout the whole lifecycle of the channel. Things like a faulty client sending a second request over the same channel before the first request is completed, is handled gracefully with a client error. The state transitions look like the following: ![graph(3)](https://github.com/Alluxio/alluxio/assets/6999708/8088f14c-6224-4af4-929a-d6d3e0b8b2ef) ### Does this PR introduce any user facing changes? No. ### Tests Tests have been done with basic Alluxio CLI tools, as well as automated PrestoDB and TPC-DS tests. pr-link: Alluxio#17479 change-id: cid-bb8b2c70f0bf0bd84e73d9e858bf6e80706427aa
Configuration menu - View commit details
-
Copy full SHA for 9f683b3 - Browse repository at this point
Copy the full SHA 9f683b3View commit details -
Refactor Netty read handler to allow subclassing. pr-link: Alluxio#18120 change-id: cid-d0d4125459644d8e92c04dd9decdb622f14cbdb6
Configuration menu - View commit details
-
Copy full SHA for dc79cd1 - Browse repository at this point
Copy the full SHA dc79cd1View commit details -
Implement per-thread cache context
### What changes are proposed in this pull request? Implement per-thread cache context ### Why are the changes needed? Enable the fine-grained cache admission ### Does this PR introduce any user facing changes? no pr-link: Alluxio#18029 change-id: cid-839bc71b2df158a4aeaedf22c5c7fb40dfd769e8
Configuration menu - View commit details
-
Copy full SHA for 65f27f5 - Browse repository at this point
Copy the full SHA 65f27f5View commit details -
[DOCFIX] Remove outdated limitation in QuickStart doc
Fuse is supported on K8s. Remove the outdated limitation. pr-link: Alluxio#18122 change-id: cid-99cacf64405867a720a5304ab5604b778c0b2127
Configuration menu - View commit details
-
Copy full SHA for 7e9691b - Browse repository at this point
Copy the full SHA 7e9691bView commit details
Commits on Sep 9, 2023
-
Change Benchmark's
runClass
toexec class
### What changes are proposed in this pull request? Change `runClass` in Benchmark to `exec class` ### Why are the changes needed? Since new alluxio cli change from `runClass` to `exec class`, related benchmark code should be change as well to keep it work ### Does this PR introduce any user facing changes? no pr-link: Alluxio#18079 change-id: cid-2e8b23a7fc272ed0266c81f5e816442fdb9cd25b
Configuration menu - View commit details
-
Copy full SHA for 9bb6518 - Browse repository at this point
Copy the full SHA 9bb6518View commit details -
`uname -m` on arm == `arm64` pr-link: Alluxio#18123 change-id: cid-c10f5760052ff7cc3757843df86710200d70b090
Configuration menu - View commit details
-
Copy full SHA for 2360819 - Browse repository at this point
Copy the full SHA 2360819View commit details
Commits on Sep 10, 2023
-
Configuration menu - View commit details
-
Copy full SHA for db51058 - Browse repository at this point
Copy the full SHA db51058View commit details -
Configuration menu - View commit details
-
Copy full SHA for 5fb603d - Browse repository at this point
Copy the full SHA 5fb603dView commit details
Commits on Sep 11, 2023
-
Configuration menu - View commit details
-
Copy full SHA for 3020c34 - Browse repository at this point
Copy the full SHA 3020c34View commit details -
Configuration menu - View commit details
-
Copy full SHA for ea6ac3a - Browse repository at this point
Copy the full SHA ea6ac3aView commit details -
Configuration menu - View commit details
-
Copy full SHA for 87263d2 - Browse repository at this point
Copy the full SHA 87263d2View commit details -
Configuration menu - View commit details
-
Copy full SHA for 568acf6 - Browse repository at this point
Copy the full SHA 568acf6View commit details -
Configuration menu - View commit details
-
Copy full SHA for 59474ed - Browse repository at this point
Copy the full SHA 59474edView commit details -
Configuration menu - View commit details
-
Copy full SHA for 3f16569 - Browse repository at this point
Copy the full SHA 3f16569View commit details -
Configuration menu - View commit details
-
Copy full SHA for 466fc05 - Browse repository at this point
Copy the full SHA 466fc05View commit details -
Remove hostname from metrics key
### What changes are proposed in this pull request? Remove hostname from metrics key ### Why are the changes needed? For easy aggregation on prometheus and grafana side ### Does this PR introduce any user facing changes? Add a flag to disable this for compatibility pr-link: Alluxio#18121 change-id: cid-ba6c2f9fae625747192044168fce7dc026c66b9c
Configuration menu - View commit details
-
Copy full SHA for 6165c0d - Browse repository at this point
Copy the full SHA 6165c0dView commit details -
[DOCFIX] Update bin/alluxio usage format
besides the User-CLI.md doc, update other doc files that refer to `bin/alluxio` commands - remove docs on path config - remove starting/stopping job master/worker from contributor docs pr-link: Alluxio#18128 change-id: cid-fc71dd493b16ef3aeeb0b1b190941c43b9af9cab
Configuration menu - View commit details
-
Copy full SHA for 5ad6e21 - Browse repository at this point
Copy the full SHA 5ad6e21View commit details
Commits on Sep 12, 2023
-
Create DoraWorkerServiceHandlerTest.java
What changes are proposed in this pull request? I have created a test and create a liststatus test for its function. Why are the changes needed? Please clarify why the changes are needed. For instance, add a unit test for DoraWorkerClientServiceHandler. Does this PR introduce any user facing changes? No. pr-link: Alluxio#18059 change-id: cid-b82706a4419700f017584f3e5579d2ef3410aeb3
Configuration menu - View commit details
-
Copy full SHA for 3c41091 - Browse repository at this point
Copy the full SHA 3c41091View commit details -
Make fuse max reader concurrency configurable
Make fuse max reader concurrency configurable. The default value was 64 and it was unchangeable. pr-link: Alluxio#18129 change-id: cid-9c55821622329bd1e608da2e7445e8ab591df38a
Configuration menu - View commit details
-
Copy full SHA for 71f5708 - Browse repository at this point
Copy the full SHA 71f5708View commit details -
Fix typo from alluxio.max.fuse.reader.concurrency to alluxio.fuse.max.reader.concurrency pr-link: Alluxio#18134 change-id: cid-434086cf6ba9e9f8d173e3417fc8518963dfa102
Configuration menu - View commit details
-
Copy full SHA for fc7971c - Browse repository at this point
Copy the full SHA fc7971cView commit details -
[DOCFIX] Update bin/alluxio usage in ufs/compute
update usages of bin/alluxio, bin/alluxio-start.sh and bin/alluxio-stop.sh to their new counterparts simplify section of CephFS.md and remove sections related to mounting. the ufs must be configured as the root mount via alluxio-site.properties. pr-link: Alluxio#18136 change-id: cid-fa7d0eec00c8fb136680ef6d5a2c7ee78571d123
Configuration menu - View commit details
-
Copy full SHA for c897a1b - Browse repository at this point
Copy the full SHA c897a1bView commit details
Commits on Sep 13, 2023
-
Support accessing OSS with proxy
### What changes are proposed in this pull request? Support accessing OSS through proxy by configuring alluxio properties or system properties. ### Why are the changes needed? When accessing OSS through a proxy, the OSS client cannot recognize the proxy configuration in system property and environment variables. So it has to proactively set proxy-related configurations in the configuration. ### Does this PR introduce any user facing changes? Please list the user-facing changes introduced by your change, including 1. change in user-facing APIs 2. addition or removal of property keys 3. webui pr-link: Alluxio#18139 change-id: cid-5e30dfd90747d4a1aafe9b2ff985331f05fefec6
Configuration menu - View commit details
-
Copy full SHA for 6c97d6f - Browse repository at this point
Copy the full SHA 6c97d6fView commit details -
Fix default value of proxy host
### What changes are proposed in this pull request? If don't set oss.proxy.host, the default value should be NULL ### Why are the changes needed? If don't set oss.proxy.host, the default value should be NULL ### Does this PR introduce any user facing changes? Please list the user-facing changes introduced by your change, including 1. change in user-facing APIs 2. addition or removal of property keys 3. webui pr-link: Alluxio#18142 change-id: cid-bce2790e583445c4ba6720d2f0a64551fb19de20
Configuration menu - View commit details
-
Copy full SHA for 1f146a9 - Browse repository at this point
Copy the full SHA 1f146a9View commit details -
Extract path resolution in DoraCacheFS into static utility
### What changes are proposed in this pull request? In this change the path conversion logic is extracted to static utility methods for code reuse (because other classes may use the same path resolution logic). The method names are slightly improved, to distinguish the member methods in `DoraCacheFileSystem` (may be inherited) from the static utility methods. pr-link: Alluxio#18140 change-id: cid-557fa148f6daa41f0b296132e5a2ecae6c5d6c22
Configuration menu - View commit details
-
Copy full SHA for 7328331 - Browse repository at this point
Copy the full SHA 7328331View commit details
Commits on Sep 14, 2023
-
[DOCFIX] Fix check-docs errMsg bug
pr-link: Alluxio#18138 change-id: cid-e9f862385bdfe6cc5e6938eb49907055449deb4a
Configuration menu - View commit details
-
Copy full SHA for 7fc1040 - Browse repository at this point
Copy the full SHA 7fc1040View commit details -
Move path resolution logic and fix a bug
### What changes are proposed in this pull request? 1. Move the path resolution logic from `DoraCacheFileSystem` to `PathUtils` where it makes more sense 2. Fix the alluxioPathToUfsPath resolution by handling the case where the ufs path may have no matching alluxio path, making the util method more generic pr-link: Alluxio#18146 change-id: cid-ebace1efcf58e385bbf71b599e4b5a15a2199f7e
Configuration menu - View commit details
-
Copy full SHA for 876d57f - Browse repository at this point
Copy the full SHA 876d57fView commit details -
Improve dora stream error handling
### What changes are proposed in this pull request? Please outline the changes and how this PR fixes the issue. ### Why are the changes needed? Better error handling when a stream is closed. Log more and don't swallow errors ### Does this PR introduce any user facing changes? N/A pr-link: Alluxio#18145 change-id: cid-b1146421ea5bc51e9a5eea5bf11ce1a5d8466912
Configuration menu - View commit details
-
Copy full SHA for 1fe33f7 - Browse repository at this point
Copy the full SHA 1fe33f7View commit details -
Use jackson JSON as a standard format for reports. Output example for `bin/alluxio info report summary`: ``` { "mSafeMode":false, "mZookeeper":false, "mRaftJournal":true, "version":"304-SNAPSHOT", "uptime":"0 day(s), 0 hour(s), 3 minute(s), and 7 second(s)", "rpcPort":19998, "webPort":19999, "masterAddress":"Ec2Cluster-masters-0:19998", "masterVersions":[ { "version":"304-SNAPSHOT", "state":"PRIMARY", "host":"Ec2Cluster-masters-0", "port":19998 } ], "started":"08-24-2023 06:43:32:856", "zookeeperAddress":[ ], "raftJournalAddress":[ "Ec2Cluster-masters-0:19200" ], "liveWorkers":2, "lostWorkers":0, "freeCapacity":"2048.00MB", "totalCapacityOnTiers":{ "MEM":"2048.00MB" }, "usedCapacityOnTiers":{ "MEM":"0B" } } ``` pr-link: Alluxio#18047 change-id: cid-bf6d54f47390a4d2bd84e4baac2ea2863d4638e1
Configuration menu - View commit details
-
Copy full SHA for 959c527 - Browse repository at this point
Copy the full SHA 959c527View commit details
Commits on Sep 15, 2023
-
Refactor the launch in CLI to be more flexible
### What changes are proposed in this pull request? refactor the cli code ### Why are the changes needed? make the code more flexible and easy for adding more functions in a cleaner way ### Does this PR introduce any user facing changes? nope pr-link: Alluxio#18152 change-id: cid-d8f937075174d913daf32387d781161096f03345
Configuration menu - View commit details
-
Copy full SHA for 9f34c2f - Browse repository at this point
Copy the full SHA 9f34c2fView commit details -
Write to UFS when failed to write to Alluxio Worker
1. Use `addSuppressed` instead of creating a new exception to throw the original exception. 2. Go ahead to write data to UFS when it encountered exception during the time writing data to Alluxio Worker. pr-link: Alluxio#18017 change-id: cid-9337252b71e40fced28fb1598ea88eed56c69229
Configuration menu - View commit details
-
Copy full SHA for 84dc05f - Browse repository at this point
Copy the full SHA 84dc05fView commit details -
### What changes are proposed in this pull request? Improve distributed load 1. Configurable job failure criteria 2. Configuration to determine if the load job should be restored from journal or not 3. Add an option to skip existing fully loaded file 4. Add retry count for failed files 5. Bug fixing ### Why are the changes needed? To enhance the distributed load tool ### Does this PR introduce any user facing changes? Yes. The skip-if-exists option is added to the distributed load cli. pr-link: Alluxio#18153 change-id: cid-5644da1c09bd6ee48f628552f51cb570de581b93
Configuration menu - View commit details
-
Copy full SHA for 6a9f5fd - Browse repository at this point
Copy the full SHA 6a9f5fdView commit details -
### What changes are proposed in this pull request? Recover the ufs uri support ### Why are the changes needed? Ufs uri should be the first class citizen in dora ### Does this PR introduce any user facing changes? No pr-link: Alluxio#18135 change-id: cid-a35fe39bdf69879ef113fc97737a35ebf6d8b29a
Configuration menu - View commit details
-
Copy full SHA for 100eb4f - Browse repository at this point
Copy the full SHA 100eb4fView commit details -
[DOCFIX] Update docker documentation
Add docker doc back pr-link: Alluxio#18130 change-id: cid-6038dce4ae1821f0e7ccf0e2e874bed5d312057d
Configuration menu - View commit details
-
Copy full SHA for f05deb3 - Browse repository at this point
Copy the full SHA f05deb3View commit details -
Add a test to monitor the distribution of vnode to physical node
### What changes are proposed in this pull request? Add a test so we can monitor the vnode distribution is not too uneven. This test calculates the standard deviation over mean on the collection of virtual nodes assigned to physical nodes. It arbitrarily bounds it at 0.25, but ideally this number should get smaller over time as we improve hashing algorithm and use better ways to assign virtual nodes to physical nodes. ### Why are the changes needed? We may change hashing algorithm and virtual node assignment in the future, this will provide guidance and catch errors. ### Does this PR introduce any user facing changes? No. pr-link: Alluxio#18147 change-id: cid-152d8edc9b65ef59967d5985849feeb471a6650d
Configuration menu - View commit details
-
Copy full SHA for 8bba797 - Browse repository at this point
Copy the full SHA 8bba797View commit details
Commits on Sep 17, 2023
-
Add CLI tools for debugging and analyzing caching issues
Add the following CLI tools for debugging and analyzing caching issues: 1. checkCaching. Checks if files under a path have been cached in alluxio. 2. location. Displays the list of hosts storing the specified file. 3. consistentHash. This command is for checking whether the consistent hash ring is changed or not. pr-link: Alluxio#18151 change-id: cid-c89b98da70a5270070d873bdcfce1aa9b23cf083
Configuration menu - View commit details
-
Copy full SHA for 6f35645 - Browse repository at this point
Copy the full SHA 6f35645View commit details
Commits on Sep 18, 2023
-
Parse JSON-style reports in golang side
Update the Golang side commands to be able to use this output: 1. Return either the yaml or json (default) output to the console. 2. Users can define the format they want with `--format` flag, like `bin/alluxio info report --format yaml` 3. In JSON format, print properties in a fixed, easy-to-read order 4. In YAML format, print properties alphabetically (since YAML specification regards property order non-significant) Before: ``` {"safeMode":false,"masterVersions":[{"version":"304-SNAPSHOT","host":"localhost","port":19998,"state":"PRIMARY"}],"masterAddress":"localhost:19998","zookeeperAddress":[],"useZookeeper":false,"raftJournalAddress":["localhost:19200"],"useRaftJournal":true,"liveWorkers":1,"lostWorkers":0,"freeCapacity":"1024.00MB","totalCapacityOnTiers":{"MEM":"1024.00MB"},"usedCapacityOnTiers":{"MEM":"0B"},"version":"304-SNAPSHOT","webPort":19999,"started":"09-15-2023 15:54:56:635","uptime":"0 day(s), 0 hour(s), 26 minute(s), and 37 second(s)","rpcPort":19998} ``` After (in JSON): ``` { "rpcPort": 19998, "started": "09-15-2023 15:54:56:635", "uptime": "0 day(s), 0 hour(s), 55 minute(s), and 31 second(s)", "safeMode": false, "version": "304-SNAPSHOT", "webPort": 19999, "masterVersions": [ { "version": "304-SNAPSHOT", "host": "localhost", "port": 19998, "state": "PRIMARY" } ], "masterAddress": "localhost:19998", "zookeeperAddress": [], "useZookeeper": false, "raftJournalAddress": [ "localhost:19200" ], "useRaftJournal": true, "liveWorkers": 1, "lostWorkers": 0, "freeCapacity": "1024.00MB", "totalCapacityOnTiers": { "MEM": "1024.00MB" }, "usedCapacityOnTiers": { "MEM": "0B" } } ``` After (in YAML): ``` freeCapacity: 1024.00MB liveWorkers: 1 lostWorkers: 0 masterAddress: localhost:19998 masterVersions: - host: localhost port: 19998 state: PRIMARY version: 304-SNAPSHOT raftJournalAddress: - localhost:19200 rpcPort: 19998 safeMode: false started: 09-15-2023 15:54:56:635 totalCapacityOnTiers: MEM: 1024.00MB uptime: 0 day(s), 1 hour(s), 1 minute(s), and 36 second(s) useRaftJournal: true useZookeeper: false usedCapacityOnTiers: MEM: 0B version: 304-SNAPSHOT webPort: 19999 zookeeperAddress: [] ``` pr-link: Alluxio#18159 change-id: cid-deb6e74552de9afcf45391c6c230a9fe00785e37
Configuration menu - View commit details
-
Copy full SHA for 86308c3 - Browse repository at this point
Copy the full SHA 86308c3View commit details -
### What changes are proposed in this pull request? Add datePredicate, i.e.: lastModifiedDate(2000/01/01, 2023/09/01) ### Why are the changes needed? Customer requirement. ### Does this PR introduce any user facing changes? na pr-link: Alluxio#18155 change-id: cid-7e1a7b7d208747807b87502c9da854ddf0b8c7fc
Configuration menu - View commit details
-
Copy full SHA for 838893d - Browse repository at this point
Copy the full SHA 838893dView commit details -
Add throughput distribution and coarse datapoints for StressWorkerBench
For random reads, bytes read per file is not a constant any more. In spite of existing duration distribution, need a throughput distribution for better understanding of reading performance. Also, when duration too long, grpc will receive huge size of output data. Should aggregate data points to transfer more datapoints with limited output size. Group datapoints by threads and time slices: Example: ``` nodeResults: { worker-0: { dataPoints: [ data: [ { // worker 0, thread 0, slice 0 count: 1, iobytes: 33554432, }, { // worker 0, thread 0, other slices … } ], [ // worker 0, other workers … ] ] throughputPercentiles: […] }, worker-1: { // other workers … } } ``` Slice time with `--slice-size` flag, e.g. `--slice-size 1s`. pr-link: Alluxio#18149 change-id: cid-ec8ed5a4f9eeaa86b1d86b6b449db4647d584823
Configuration menu - View commit details
-
Copy full SHA for ddba020 - Browse repository at this point
Copy the full SHA ddba020View commit details -
Add Unit Test for OSS, OBS and GCS
### What changes are proposed in this pull request? Add Unit Test for OSS, OBS and GCS ### Why are the changes needed? Unit test is important for improving functions of Alluxio. ### Does this PR introduce any user facing changes? No pr-link: Alluxio#17985 change-id: cid-c757b8249c62e2ccf0483cb99436e33d351358a1
Zihao Zhao authoredSep 18, 2023 Configuration menu - View commit details
-
Copy full SHA for 71fd8c4 - Browse repository at this point
Copy the full SHA 71fd8c4View commit details
Commits on Sep 19, 2023
-
### What changes are proposed in this pull request? Fix datePredicate so it would respect the interval specified in the policy. Polish tests ### Why are the changes needed? bug fix ### Does this PR introduce any user facing changes? na pr-link: Alluxio#18167 change-id: cid-b035b5cc31aa70a2f83de2c3f84ba49ed75f9fb5
Configuration menu - View commit details
-
Copy full SHA for db4fc2b - Browse repository at this point
Copy the full SHA db4fc2bView commit details -
- Add a new CLI command that iterates through the command tree and generates a markdown file based on each command's definition and description - Migrate all the content in the previous User-CLI.md into the corresponding commands in golang code, mainly updating their `Long` and `Examples` fields - Run `bin/alluxio generate user-cli` to write the generated content directly into `docs/en/operation/User-CLI.md` pr-link: Alluxio#18144 change-id: cid-9b29f273efef9693e1b0b303c62cc19602d77acc
Configuration menu - View commit details
-
Copy full SHA for dd863fe - Browse repository at this point
Copy the full SHA dd863feView commit details -
Fix the broken PagedDoraWorkerTest
### What changes are proposed in this pull request? Fix the broken PagedDoraWorkerTest. ### Why are the changes needed? The old test is.broken, I just fix it. ### Does this PR introduce any user facing changes? no. pr-link: Alluxio#18150 change-id: cid-a142297c4f08780189e4321abc8e99ff512091ec
Configuration menu - View commit details
-
Copy full SHA for d737d56 - Browse repository at this point
Copy the full SHA d737d56View commit details -
Fix
NoClassDefFoundError
UnderFileSystemFactory not loaded### What changes are proposed in this pull request? Fix UFS not loaded in some cases. ### Why are the changes needed? Fix the following error when loading `UnderFileSystemFactory`: ``` java.lang.NoClassDefFoundError: alluxio/underfs/UnderFileSystemFactory at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:756) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) at java.net.URLClassLoader.defineClass(URLClassLoader.java:468) at java.net.URLClassLoader.access$100(URLClassLoader.java:74) at java.net.URLClassLoader$1.run(URLClassLoader.java:369) at java.net.URLClassLoader$1.run(URLClassLoader.java:363) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:362) at alluxio.extensions.ExtensionsClassLoader.findClass(ExtensionsClassLoader.java:73) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at alluxio.extensions.ExtensionsClassLoader.loadClass(ExtensionsClassLoader.java:82) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:348) at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:370) at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404) at java.util.ServiceLoader$1.next(ServiceLoader.java:480) at alluxio.extensions.ExtensionFactoryRegistry.scan(ExtensionFactoryRegistry.java:248) at alluxio.extensions.ExtensionFactoryRegistry.scanLibs(ExtensionFactoryRegistry.java:229) at alluxio.extensions.ExtensionFactoryRegistry.findAllWithRecorder(ExtensionFactoryRegistry.java:149) at alluxio.underfs.UnderFileSystemFactoryRegistry.findAllWithRecorder(UnderFileSystemFactoryRegistry.java:115) at alluxio.underfs.UnderFileSystem$Factory.createWithRecorder(UnderFileSystem.java:111) at alluxio.underfs.UnderFileSystem$Factory.create(UnderFileSystem.java:93) ``` This was because when `Thread.currentThread().getContextClassLoader()` is null, it indicates that the class loader is actually the system class loader. If using `null` for the default class loader, it will not be able to resolve `UnderFileSystemFactory` which was already loaded by the system class loader. ### Does this PR introduce any user facing changes? No. pr-link: Alluxio#18169 change-id: cid-d79281b3fb14e894e4ff21fe34222ef9e04f05d4
Configuration menu - View commit details
-
Copy full SHA for 814fad4 - Browse repository at this point
Copy the full SHA 814fad4View commit details
Commits on Sep 20, 2023
-
Alluxio does not have a valid Block Info at this moment. ### What changes are proposed in this pull request? return an all zero block info for fsstat ### Why are the changes needed? No valid block info is managed by Alluxio at this moment. Alluxio does not have 'master' node, and no block info is stored/managed by other components. This may change in future, e.g. statfs() can gather the block info from all worker nodes. ### Does this PR introduce any user facing changes? N/A pr-link: Alluxio#18172 change-id: cid-eda9fb4ad68fa91d6d766285f6de211d5f02df56
Configuration menu - View commit details
-
Copy full SHA for 4b9ab67 - Browse repository at this point
Copy the full SHA 4b9ab67View commit details -
Add back journal checkpoint command
pr-link: Alluxio#18174 change-id: cid-996586c993d3920cd91c4663a8771ced840fe80f
Configuration menu - View commit details
-
Copy full SHA for 796b66f - Browse repository at this point
Copy the full SHA 796b66fView commit details -
[DOCFIX] Fix typo in generate docs cmd
pr-link: Alluxio#18177 change-id: cid-9e895702ab3f800a47fed97ce2c1a756abb47819
Configuration menu - View commit details
-
Copy full SHA for e21a7a3 - Browse repository at this point
Copy the full SHA e21a7a3View commit details -
Enable config the virtual nodes num per node
The configuration `alluxio.user.consistent.hash.virtual.node.count` is for total virtual nodes of all workers. We need to adjust the configuration according to the number of worker nodes. However, it is difficult to determine how many virtual nodes are enough to avoid the data skew issue. Based on this background, a better way to solve the data skew issue is to allow user to set the virtual nodes number for a worker node, but not the total virtual nodes of all worker nodes. This is what this PR does. pr-link: Alluxio#18179 change-id: cid-fbb4c9ebe13f5c0cce4ac08f9fe1789d97cdf209
Configuration menu - View commit details
-
Copy full SHA for 7b37a86 - Browse repository at this point
Copy the full SHA 7b37a86View commit details
Commits on Sep 21, 2023
-
Fix GCSV2UnderFileSystem initialization
### What changes are proposed in this pull request? Fix GCSV2UnderFileSystem initialization ### Why are the changes needed? not able to create ufs instance if not set access key, which should not be a requirement for gcs v2 ### Does this PR introduce any user facing changes? na pr-link: Alluxio#18184 change-id: cid-c736612c8ccc2d5d8bd9ecbd9f862942625e8156
Configuration menu - View commit details
-
Copy full SHA for 765048c - Browse repository at this point
Copy the full SHA 765048cView commit details -
And refactor fuse start code into startCommon() and stopCommon() ### What changes are proposed in this pull request? Stop Fuse web server when exit. ### Why are the changes needed? Fuse web server must be stopped, otherwise the main thread is waiting for it forever. ### Does this PR introduce any user facing changes? N/A pr-link: Alluxio#18190 change-id: cid-cdad62f25870a6a1e1984127834402045b9763d4
Configuration menu - View commit details
-
Copy full SHA for 6f707e1 - Browse repository at this point
Copy the full SHA 6f707e1View commit details -
Only stop processes belong to the current user
### What changes are proposed in this pull request? Only stop processes belong to the current user while running `./bin/alluxio process start` or `./bin/alluxio process stop` ### Why are the changes needed? Without this PR, it will try to stop all processes match the class name even they are not started by the current user. Like the below case ``` $ ./bin/alluxio process stop local WARN[2023-09-21T14:57:31+08:00] No process to stop because could not find running process matching alluxio.worker.AlluxioWorker INFO[2023-09-21T14:57:31+08:00] Found 1 running process(es) matching alluxio.master.AlluxioMaster INFO[2023-09-21T14:57:31+08:00] Successfully killed 0 process(es) ERRO[2023-09-21T14:57:31+08:00] Failed to kill 1 process(es): ERRO[2023-09-21T14:57:31+08:00] Failed to kill process: error sending TERM signal to process for alluxio.master.AlluxioMaster with pid 17055 --- at /tmp/alluxio1717887242/cli/src/alluxio.org/cli/env/process.go:175 (BaseProcess.Stop.func1) --- Caused by: operation not permitted ``` ### Does this PR introduce any user facing changes? NO pr-link: Alluxio#18187 change-id: cid-ee45c89e93fff4bd8c58dba3e81cd1fa4de52368
Configuration menu - View commit details
-
Copy full SHA for 2e18cd2 - Browse repository at this point
Copy the full SHA 2e18cd2View commit details -
Fix typo in ServiceDiscoveryRecipe
### What changes are proposed in this pull request? Fixed a typo ### Why are the changes needed? ### Does this PR introduce any user facing changes? No pr-link: Alluxio#18191 change-id: cid-67a503376ed89395ebccd674644394e5bc02522b
2Configuration menu - View commit details
-
Copy full SHA for d056937 - Browse repository at this point
Copy the full SHA d056937View commit details
Commits on Sep 22, 2023
-
Fix the bug of using a terminated executor in S3 UFS
### What changes are proposed in this pull request? Don't call the ufs.close() in PageFileReader.close() ### Why are the changes needed? 1. we should not close the ufs instance, because we always try to reuse it for the same ufs mount point. unless we don't reuse the instance of the ufs instance. 2. S3UnderFileSystem.close() will just shut down the executor held by the instance. when other threads reuse the executor, it will cause issues. ### Does this PR introduce any user facing changes? no pr-link: Alluxio#18188 change-id: cid-2d97d2ba48bb8dc333cb8d2dddd7990dad5d372b
Configuration menu - View commit details
-
Copy full SHA for 3a82f8e - Browse repository at this point
Copy the full SHA 3a82f8eView commit details -
### What changes are proposed in this pull request? 1. Add an abstract representation of a worker's identity. The new identity is the same as a `long` based identity that used to be generated by the master. It decouples the concept of an identity and the actual implementation of an identity. It also includes a version field that allows backward compatible evolution of the worker identifier. 2. A new implementation for worker identities that uses UUID. A worker will generate its own identity instead of acquiring one from the master, unless the user explicitly specifies one in the configuration. The generated identity is then persisted in the worker's working directory, and reloaded at the next worker restart. ### Why are the changes needed? For Dora workers, the identities are not generated by a centralized service, but by workers themselves. Therefore, we need something that provides randomness and avoids collision between workers, e.g. a UUID. The V1 parser implementation included in this PR uses a UUID as the worker identity. ### Does this PR introduce any user facing changes? Yes, a new configuration item `alluxio.worker.identity.uuid` is added that allows a user to specify the worker identity. pr-link: Alluxio#17820 change-id: cid-3200e78fa24be0126d4d49370f0a266f8c766e12
Configuration menu - View commit details
-
Copy full SHA for 82bd57b - Browse repository at this point
Copy the full SHA 82bd57bView commit details -
[DOCFIX] Describe how the user cli docs are generated
dev docs for generating user cli documentation pr-link: Alluxio#18192 change-id: cid-ee3ad14a4a284ddf2040850f4d590e1ccb70cb69
Configuration menu - View commit details
-
Copy full SHA for bc4f95b - Browse repository at this point
Copy the full SHA bc4f95bView commit details -
Split the fuse stream close() method
### What changes are proposed in this pull request? Split the fuse stream close() method into two individual methods: 1. Close the fuse stream 2. Release the lock Currently no behavior changes on the close function as the function will call these two functions to achieve the same functionality as before. ### Why are the changes needed? When we handle some fuse operations, we sometimes want to close the stream only but want to keep the lock there, to avoid inconsistency. ### Does this PR introduce any user facing changes? Please list the user-facing changes introduced by your change, including 1. change in user-facing APIs 3. addition or removal of property keys 4. webui pr-link: Alluxio#18180 change-id: cid-93e1f5c6c2448536321cbcbb528471b032c4c01f
Configuration menu - View commit details
-
Copy full SHA for 325582e - Browse repository at this point
Copy the full SHA 325582eView commit details -
Refactor the collectInfo main for future use
### What changes are proposed in this pull request? Refactor some of the code in java and go to make the CLI more flexible for the future ### Why are the changes needed? There might be more commands added in the future ### Does this PR introduce any user facing changes? nope pr-link: Alluxio#18197 change-id: cid-00c52688dd60bc0a451cfbff932b1da329549993
Configuration menu - View commit details
-
Copy full SHA for 4e2a494 - Browse repository at this point
Copy the full SHA 4e2a494View commit details -
Improve the random read behavior in StressWorkerBench
### What changes are proposed in this pull request? Improve the randomness of StressWorkerBench random read test, now each thread throw dice every time it trying to do a random read. ### Why are the changes needed? In previous StressWorkerBench random read test each thread read from same offset and same length everytime, this cause low randomness. ### Does this PR introduce any user facing changes? no pr-link: Alluxio#18000 change-id: cid-974afb58c022061e8a6d8d6894415a8bd292b764
Configuration menu - View commit details
-
Copy full SHA for be6974c - Browse repository at this point
Copy the full SHA be6974cView commit details -
Parse JSON-style reports in golang side
Cherry-pick of existing commit. orig-pr: Alluxio#18159 orig-commit: Alluxio/alluxio@86308c3 orig-commit-author: twalluxio <139105284+twalluxio@users.noreply.github.com> pr-link: Alluxio#18166 change-id: cid-deb6e74552de9afcf45391c6c230a9fe00785e37
Configuration menu - View commit details
-
Copy full SHA for ba7fedf - Browse repository at this point
Copy the full SHA ba7fedfView commit details -
Update collectAlluxioInfo and other bin/alluxio cmd strings
collectAlluxioInfo calls bin/alluxio commands and collects their output. fix the commands after the golang CLI refactor pr-link: Alluxio#18186 change-id: cid-23c99f152405d04dcdd9bbea3953b00e23ee3882
Configuration menu - View commit details
-
Copy full SHA for e3d7c80 - Browse repository at this point
Copy the full SHA e3d7c80View commit details -
Refactor build logic for usability
### What changes are proposed in this pull request? - made some funcs/consts public - tarball artifacts list lib/plugin modules in `metadata` field ### Why are the changes needed? ### Does this PR introduce any user facing changes? no pr-link: Alluxio#18195 change-id: cid-7f7738ef1f2d4e9907dd5d9e047870f048cdd269
Configuration menu - View commit details
-
Copy full SHA for e12f626 - Browse repository at this point
Copy the full SHA e12f626View commit details -
Fix last modified time is 0 in UfsFileIterable
### What changes are proposed in this pull request? Fix last modified time is 0 in UfsFileIterable ### Why are the changes needed? bug fix ### Does this PR introduce any user facing changes? na pr-link: Alluxio#18200 change-id: cid-e3fc548f65b642f8b8cb62d3b94f26366e33e4a7
Configuration menu - View commit details
-
Copy full SHA for c75317f - Browse repository at this point
Copy the full SHA c75317fView commit details
Commits on Sep 23, 2023
-
Fix and re-enable UfsFileIn/OutStreamTest
### What changes are proposed in this pull request? UfsFileIn/OutStream Test runnable now ### Why are the changes needed? Need UfsFileIn/OutStream Test ### Does this PR introduce any user facing changes? No pr-link: Alluxio#18170 change-id: cid-2de71fa1d5bc6189c03f6fd496f55c5c53fd778c
Configuration menu - View commit details
-
Copy full SHA for e68c6d8 - Browse repository at this point
Copy the full SHA e68c6d8View commit details
Commits on Sep 24, 2023
-
Enable static consistent hash ring
By default, we build a dynamic consistent hash with the live worker list that comes from master or ETCD. Sometimes we want to build a static consistent hash ring to make sure we won't write data to other worker node when a worker node is offline temporarily (especially when other worker nodes are running out of disk space). This PR provides allows us to build a static consistent hash ring by setting `alluxio.user.dynamic.consistent.hash.ring.enabled=false`. In this case, client will read from UFS if the worker where the specified file locate is down. pr-link: Alluxio#18183 change-id: cid-ccd37436391d62aa7c6a9942c80e2d3c8ffdb170
Configuration menu - View commit details
-
Copy full SHA for 189ca24 - Browse repository at this point
Copy the full SHA 189ca24View commit details
Commits on Sep 25, 2023
-
### What changes are proposed in this pull request? Add client read metrics ### Why are the changes needed? These metrics are helpful. ### Does this PR introduce any user facing changes? N/A pr-link: Alluxio#18160 change-id: cid-08d84f39da7aec0ca21e1d57731930a67693dbeb
Configuration menu - View commit details
-
Copy full SHA for fe55fc9 - Browse repository at this point
Copy the full SHA fe55fc9View commit details -
Add Dora read fallback to UFS logics
Support dora read to fallback to UFS if any error occur during read. pr-link: Alluxio#18154 change-id: cid-755c5bdbe4edf9abf32a7d700d17617cbb8a78a1
Configuration menu - View commit details
-
Copy full SHA for 3dd49e4 - Browse repository at this point
Copy the full SHA 3dd49e4View commit details -
Close Etcd client when process exits
### What changes are proposed in this pull request? 1. Since EtcdClient is a singleton, close it in the jvm shutdownHook 2. Close related resources within EtcdClient pr-link: Alluxio#18204 change-id: cid-b00c40829136c88dd6a6e1c0731b1a5ee3647cff
Configuration menu - View commit details
-
Copy full SHA for a2d8410 - Browse repository at this point
Copy the full SHA a2d8410View commit details -
Fix read position in PagedDoraWorker
### What changes are proposed in this pull request? 1. The length of the read is incorrectly calculated. Should not have anything to do with the position of the reader. 2. `mPos` should not be updated for the `read` method, as it's a positioned read operation (has an `offset` argument). ### Why are the changes needed? bug fix ### Does this PR introduce any user facing changes? No. pr-link: Alluxio#18205 change-id: cid-eb56603cad6c462e799793ca1086d3638209af00
Configuration menu - View commit details
-
Copy full SHA for 9bfdc07 - Browse repository at this point
Copy the full SHA 9bfdc07View commit details -
Improve resource deallocation in FUSE
### What changes are proposed in this pull request? This improves resource deallocation in two ways: 1. Close FileSystemContext 2. Close an optional thread pool on normal exit (before it is only destroy on error) ### Why are the changes needed? Resource deallocation pr-link: Alluxio#18206 change-id: cid-03ec1f712460172dd8d61a34fc65b95484243a3b
Configuration menu - View commit details
-
Copy full SHA for fc70ad2 - Browse repository at this point
Copy the full SHA fc70ad2View commit details -
Move json formatting in info as util SKIPCI
move code into separate package pr-link: Alluxio#18207 change-id: cid-91860a65af36b70f98e8913f0710e6a295d14008
Configuration menu - View commit details
-
Copy full SHA for d8c0ea8 - Browse repository at this point
Copy the full SHA d8c0ea8View commit details -
### What changes are proposed in this pull request? Please outline the changes and how this PR fixes the issue. ### Why are the changes needed? Please clarify why the changes are needed. For instance, 1. If you propose a new API, clarify the use case for a new API. 2. If you fix a bug, describe the bug. ### Does this PR introduce any user facing changes? Please list the user-facing changes introduced by your change, including 1. change in user-facing APIs 2. addition or removal of property keys 3. webui pr-link: Alluxio#18209 change-id: cid-3d147d628a6be3a0e654eec338ea76d2fb5f25cf
Configuration menu - View commit details
-
Copy full SHA for c11e4cb - Browse repository at this point
Copy the full SHA c11e4cbView commit details -
### What changes are proposed in this pull request? remove master host check ### Why are the changes needed? master host should not be a hard requirement right now. ### Does this PR introduce any user facing changes? na pr-link: Alluxio#18201 change-id: cid-7cfc7a688e925137a1e9d62e4af0a07298f447d4
Configuration menu - View commit details
-
Copy full SHA for 8d0c0ff - Browse repository at this point
Copy the full SHA 8d0c0ffView commit details
Commits on Sep 26, 2023
-
Add DoraMetaManagerTest. pr-link: Alluxio#18127 change-id: cid-7c8f875ce957e85597c53bc5db611f1d533596ee
Configuration menu - View commit details
-
Copy full SHA for 9ab7391 - Browse repository at this point
Copy the full SHA 9ab7391View commit details -
Modify StressWorkerBench: calculate elapsed time in nanoseconds
For now StressWorkerBench uses milliseconds as the smallest unit of recording. In random read, if file is very small, it is possible to have a duration <1ms. Since instant throughput = bytes read / duration, it will lead to a divide by zero error. This PR replace millisecond-level record of a file read with a nanosecond-level approach. It also makes the following updates on StressWorkerBench: - record file read time within `applyOperation()` function, instead of before entering or after exiting it - wrap metrics about the `applyOperation()` function to a private class, instead of returning bytes read only - remove getter and setters for WorkerBenchDataPoint, since all elements inside is public - standardize the unit of output throughput, now unit is MB/s pr-link: Alluxio#18175 change-id: cid-a606ca5812f75d8116ecaf7da45da9dedd78ab55
Configuration menu - View commit details
-
Copy full SHA for 527ffe8 - Browse repository at this point
Copy the full SHA 527ffe8View commit details -
Add manual exit at the end of FUSE life cycle
### What changes are proposed in this pull request? Added a manual double-safety `System.exit()` at the end of FUSE lifecycle, to be absolutely sure we exit the JVM and terminate all non-daemon & daemon threads. Before this change, after we `alluxio-fuse umount` or `kill (not -9)`, the FUSE process may fail to quit. One reason is a non-daemon thread dangling around: ``` # This is a non daemon thread in AlluxioEtcdClient "vert.x-eventloop-thread-0" Alluxio#21 prio=5 os_prio=31 cpu=482.23ms elapsed=722.11s tid=0x00007fde79251800 nid=0x8c03 runnable [0x00007000102ae000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.KQueue.poll(java.base@11.0.11/Native Method) at sun.nio.ch.KQueueSelectorImpl.doSelect(java.base@11.0.11/KQueueSelectorImpl.java:122) at sun.nio.ch.SelectorImpl.lockAndDoSelect(java.base@11.0.11/SelectorImpl.java:124) - locked <0x00000007c0603938> (a io.netty.channel.nio.SelectedSelectionKeySet) - locked <0x00000007c06038d8> (a sun.nio.ch.KQueueSelectorImpl) at sun.nio.ch.SelectorImpl.select(java.base@11.0.11/SelectorImpl.java:136) at io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(java.base@11.0.11/Thread.java:829) ``` The JVM shutdownHook executes after the LAST non-daemon thread dies or when `System.exit()` triggers. So if some libraries we use or some thread pools of ours accidentally introduce non-daemon threads, `System.exit()` is our last resort to trigger shutdownHook. FUSE does handle signals and exit, but there are some corner cases which don't seem to trigger that successfully. So this PR serves as the last resort. https://github.com/Alluxio/alluxio/blob/527ffe8105568757e05dc81dff1e28a8078e9a9e/dora/integration/fuse/src/main/java/alluxio/fuse/FuseSignalHandler.java#L52 ### Why are the changes needed? See above ### Does this PR introduce any user facing changes? No pr-link: Alluxio#18211 change-id: cid-adeaeadfaa5fb4013a62ea8191005b93c0ca69ad
Configuration menu - View commit details
-
Copy full SHA for 926f393 - Browse repository at this point
Copy the full SHA 926f393View commit details -
Add project structure and template for native components
### What changes are proposed in this pull request? Prepare project structure for native components. Add a maven archetype for Rust-based native components. The `native` module is behind a profile `native-components` and is deactivated by default. ### Why are the changes needed? Add support for native components ### Does this PR introduce any user facing changes? No. pr-link: Alluxio#18164 change-id: cid-530dd723b84d69cef22b08f27eae7e4b4874995a
Configuration menu - View commit details
-
Copy full SHA for dc0849a - Browse repository at this point
Copy the full SHA dc0849aView commit details -
Make copy & move return list of tasks
### What changes are proposed in this pull request? Make copy & move return list of tasks ### Why are the changes needed? make worker busy ### Does this PR introduce any user facing changes? na pr-link: Alluxio#18215 change-id: cid-d0c3abe26ec4cee3cdbd93e2a99ffb319b187524
Configuration menu - View commit details
-
Copy full SHA for dec36be - Browse repository at this point
Copy the full SHA dec36beView commit details
Commits on Sep 27, 2023
-
Create a init test for cluster
### What changes are proposed in this pull request? I have created a init test to check whether the workers has already run and connected to the leader. ### Why are the changes needed? Because the version has updated to 3.0 but we still do not have the correct init test for the cluster. ### Does this PR introduce any user facing changes? Yes, users may still not run test when they first run the cluster because i must communicate with leader to decide which args will be used. pr-link: Alluxio#18148 change-id: cid-74a018569e1ba13c55573ba9e1713bf9dd8e3f2d
Configuration menu - View commit details
-
Copy full SHA for 9047c62 - Browse repository at this point
Copy the full SHA 9047c62View commit details -
### What changes are proposed in this pull request? Please outline the changes and how this PR fixes the issue. ### Why are the changes needed? Please clarify why the changes are needed. For instance, 1. If you propose a new API, clarify the use case for a new API. 2. If you fix a bug, describe the bug. ### Does this PR introduce any user facing changes? Please list the user-facing changes introduced by your change, including 1. change in user-facing APIs 2. addition or removal of property keys 3. webui pr-link: Alluxio#18223 change-id: cid-375b97fb3beb2714a75dbb227c15978e18c2909a
Configuration menu - View commit details
-
Copy full SHA for cde8e4c - Browse repository at this point
Copy the full SHA cde8e4cView commit details -
[DOCFIX] Remove deprecated sections from journal docs
also update `init format` command to be able to format journal on all masters but avoid formatting workers pr-link: Alluxio#18124 change-id: cid-b8fd99ec5c6e6b7dbaa1c9a22414fad2a33caf50
Configuration menu - View commit details
-
Copy full SHA for 535d47b - Browse repository at this point
Copy the full SHA 535d47bView commit details -
Modify go dependencies and code to be compatible with go1.15
- downgrade viper dependency to be compatible - remove any native golang calls introduced in 16+ - update version check for cli compilation script - add license check skip flag in tarball build pr-link: Alluxio#18218 change-id: cid-bc73e4d4a8c6c78a189f213dbb11ddd4b77e5454
Configuration menu - View commit details
-
Copy full SHA for b5e7b87 - Browse repository at this point
Copy the full SHA b5e7b87View commit details
Commits on Oct 2, 2023
-
Add user/password support for etcd
### What changes are proposed in this pull request? Allow AlluxioEtcdClient to use username/password to talk to etcd if target etcd server has authentication enabled. ### Why are the changes needed? as above. ### Does this PR introduce any user facing changes? N/A pr-link: Alluxio#18193 change-id: cid-4d2644b1bf1d600a8ff4dcffa18cbbfa8f0f6ad0
Configuration menu - View commit details
-
Copy full SHA for 4777915 - Browse repository at this point
Copy the full SHA 4777915View commit details
Commits on Oct 4, 2023
-
### What changes are proposed in this pull request? add membership module doc ### Why are the changes needed? N/A ### Does this PR introduce any user facing changes? N/A pr-link: Alluxio#18233 change-id: cid-ffea2ab2522a5c374e373f9e1c73dce36a5f38eb
Configuration menu - View commit details
-
Copy full SHA for 063a11a - Browse repository at this point
Copy the full SHA 063a11aView commit details -
Fail directly if sudo needs password
### What changes are proposed in this pull request? Fail directly if sudo needs password. ### Why are the changes needed? Fix Alluxio#18194 ### Does this PR introduce any user facing changes? NO pr-link: Alluxio#18219 change-id: cid-ad2566d5d69cdbe77dcb35302c5268c9ead4f848
Configuration menu - View commit details
-
Copy full SHA for c394d65 - Browse repository at this point
Copy the full SHA c394d65View commit details
Commits on Oct 7, 2023
-
Fix FileOutStreamIntegrationTest
Removed an unsupported test. Added new test for `FileOutStream.write(int)` Fixed `AbstractFileOutIntegrationTest` pr-link: Alluxio#18189 change-id: cid-a0a3c2ab1f1eee9bc24240e695082ef17badeff0
Configuration menu - View commit details
-
Copy full SHA for 9a88be2 - Browse repository at this point
Copy the full SHA 9a88be2View commit details -
Add UT to DefaultPageMetaStore.commitFile
### What changes are proposed in this pull request? Add unit test for DefaultPageMetaStore.commitFile() method, and rename some variables to more accurate names. ### Why are the changes needed? Improve the test coverage. ### Does this PR introduce any user facing changes? No. pr-link: Alluxio#18162 change-id: cid-e7de0d5b474d36b67c85f0ed0a49202a37dc7194
Configuration menu - View commit details
-
Copy full SHA for 95046f7 - Browse repository at this point
Copy the full SHA 95046f7View commit details
Commits on Oct 8, 2023
-
Improve list files RESTful API providing more useful info
Improve list files RESTful API providing more useful info. This PR add the following info to the response JSON: 1. Formatted file size 2. Alluxio path 3. UFS path 4. Last modification timestamp pr-link: Alluxio#18245 change-id: cid-b7260c0ae964b31cfe67eead92f693b40b6688fc
Configuration menu - View commit details
-
Copy full SHA for 28db253 - Browse repository at this point
Copy the full SHA 28db253View commit details -
### What changes are proposed in this pull request? The old UfsBaseFileSystem cannot run, this pr fix it. ### Why are the changes needed? Improve code quality. ### Does this PR introduce any user facing changes? no. pr-link: Alluxio#18163 change-id: cid-7b19138cb9db0d8b02613d96af9e6f33bc807b5d
Configuration menu - View commit details
-
Copy full SHA for d4a39b5 - Browse repository at this point
Copy the full SHA d4a39b5View commit details -
Disable UFS fallback in DoraFileSystemIntegrationTest
### What changes are proposed in this pull request? Set PropertyKey.DORA_CLIENT_UFS_FALLBACK_ENABLED to false. ### Why are the changes needed? As we mentioned in [here](Alluxio#18213). When we want to test the DoraFileSystem, we have to disable the Ufs Fallback, otherwise the methods like `createFile` `getStatus` will fall back to Ufs, while we want to test DoraFileSystem. **This is a important change, so it is necessary to open a new PR for it.** ### Does this PR introduce any user facing changes? No. pr-link: Alluxio#18244 change-id: cid-d81c5b5298fb9e7eb98fc5edde236f6fda05e0f0
Configuration menu - View commit details
-
Copy full SHA for 8329244 - Browse repository at this point
Copy the full SHA 8329244View commit details
Commits on Oct 9, 2023
-
Fix Integration tests framework and DoraFileSystemIntegrationTest
### What changes are proposed in this pull request? The LocalAlluxioCluster used in integration tests is broken when Ufs fallback is disable, this PR is used to fix it. The old master and worker liveness detection mechanism is invalid in ALluxio 3.0 arch, this PR also fix it. ### Why are the changes needed? Fix broken tests. ### Does this PR introduce any user facing changes? No. pr-link: Alluxio#18213 change-id: cid-6e7d3cf73c80aa45f67007f90605a78654924614
Configuration menu - View commit details
-
Copy full SHA for cb74679 - Browse repository at this point
Copy the full SHA cb74679View commit details -
Fix reconnect logic of servicediscoveryrecipe
### What changes are proposed in this pull request? when etcd lease keepalive client returned with onCompleted or onError, the lease might not directly get expired at the time, therefore we always new a lease to resume keepalive without explicitly checking isleaseexpired ### Why are the changes needed? when etcd got induced network unstableness such as pkg loss, there's race condition in our reconnection logic from ServiceDiscoveryRecipe, where we directly check isLeaseExpired when we got terminal callback from keepalive client, but at the time the lease isn't expired yet. Hence we stop trying to create new lease to keepalive. ### Does this PR introduce any user facing changes? No pr-link: Alluxio#18217 change-id: cid-bdfc1d37b359080585ac92b5daa9fac31598ea9b
Configuration menu - View commit details
-
Copy full SHA for 1a4b2a9 - Browse repository at this point
Copy the full SHA 1a4b2a9View commit details -
Avoid creating FileSystem instance many times in HttpServerHandler
Avoid creating `FileSystem` instance many times in `HttpServerHandler`, and call the `close()` method when the handler is removed. pr-link: Alluxio#18248 change-id: cid-9c4f1369a9549d58b8191214547046cd1e188d48
Configuration menu - View commit details
-
Copy full SHA for ceeeefe - Browse repository at this point
Copy the full SHA ceeeefeView commit details
Commits on Oct 10, 2023
-
Remove MigrateIntegrationTest and PersistIntegrationTest
### What changes are proposed in this pull request? Remove MigrateIntegrationTest and PersistIntegrationTest. ### Why are the changes needed? MigrateIntegrationTest and PersistIntegrationTest are related to the job service, which no longer exists in the Dora architecture. Therefore, MigrateIntegrationTest and PersistIntegrationTest can be removed. ### Does this PR introduce any user facing changes? pr-link: Alluxio#18247 change-id: cid-f38d5660f6b7c91b8407e0740aec04d2cccb8b89
Configuration menu - View commit details
-
Copy full SHA for ee0f966 - Browse repository at this point
Copy the full SHA ee0f966View commit details -
Fix FileSystemBlockLocationIntegrationTest
### What changes are proposed in this pull request? Fixed one test case in FileSystemBlockLocationIntegrationTest ### Why are the changes needed? Logic of getBlockLocation in DoraCacheFileSystem has chagned, so related IntegrationTest need be change as well ### Does this PR introduce any user facing changes? no pr-link: Alluxio#18243 change-id: cid-39521af14dafcb91d5368a25091c1031651c6dc6
Configuration menu - View commit details
-
Copy full SHA for 447a444 - Browse repository at this point
Copy the full SHA 447a444View commit details -
### What changes are proposed in this pull request? Supports setXattr API of Alluxio FileSystem, it will set the corresponding attributes to UFS. If UFS is a filesystem, it's set through the setXattr interface of UFS. if UFS is object storage, it's set through setTagging API. ### Why are the changes needed? Based on the 3.x architecture, recover support for the setXattr interface. There are still many places where the interface is used, and it should not be supported at present. pr-link: Alluxio#18053 change-id: cid-4f8c96c88cced4374d54b6387bfc618b47d2a423
Configuration menu - View commit details
-
Copy full SHA for bf997fe - Browse repository at this point
Copy the full SHA bf997feView commit details -
Remove a few deprecated integration tests
### What changes are proposed in this pull request? Remove BackupCommandStateLockingIntegrationTest, ReportEmbeddedJournalIntegrationTest, and MetricsCommandIntegrationTest ### Why are the changes needed? Remove invalid tests. ### Does this PR introduce any user facing changes? no. pr-link: Alluxio#18251 change-id: cid-fe71cc37517fc3ad1eb3f5dba4b05f5503afaa5e
Configuration menu - View commit details
-
Copy full SHA for 75e31dc - Browse repository at this point
Copy the full SHA 75e31dcView commit details -
Support StressWorkerBench using consistent hash policy
### What changes are proposed in this pull request? This PR adds a `--mode` option for StressWorkerBench. This option allows user to choose from a range of file read policies. Possible option values are `hash`(default) and `local-only`. If use `hash`, the config `alluxio.client.file.dora.ConsistentHashPolicy` is set to `alluxio.client.file.dora.ConsistentHashPolicy`. If use `local-only`, that config is set to `alluxio.client.file.dora.LocalWorkerPolicy`. The benchmark will parse this option, and choose the right policy. Also, it will print a log about the policy the user is using in both the benchmark and the policy factory. ### Why are the changes needed? In previous versions, the default policy is `local-only`. However, in Alluxio, the default policy is `hash`, using `alluxio.client.file.dora.ConsistentHashPolicy`. If user wants to use the `local-only` policy, add `--mode local-only` to the end of the benchmark command. pr-link: Alluxio#18246 change-id: cid-29676a12710b9c815f3e55f84eb5e4226fea9ad7
Configuration menu - View commit details
-
Copy full SHA for 3470e17 - Browse repository at this point
Copy the full SHA 3470e17View commit details -
Fix FileInStreamIntegrationTest
Fix `FileInStreamIntegrationTest`, removed unnecessary test cases pr-link: Alluxio#18178 change-id: cid-dac1c4c2e4e403bbf4ad3cbdde9eaf43ce20e046
Configuration menu - View commit details
-
Copy full SHA for ba61600 - Browse repository at this point
Copy the full SHA ba61600View commit details -
Add a property key to control if alluxio write the xattr to UFS
### What changes are proposed in this pull request? Add a property key to control if alluxio write the xattr to UFS ### Why are the changes needed? The current S3 mock doesn't support setTagging/getTagging. And in some cases, we don't want to write xAttr to UFS. So Add a switch to disable it. ### Does this PR introduce any user facing changes? pr-link: Alluxio#18255 change-id: cid-ccc2971f9bbb5cdd5abf31dc9fd845a1f26097b5
Configuration menu - View commit details
-
Copy full SHA for b7a0a77 - Browse repository at this point
Copy the full SHA b7a0a77View commit details -
### What changes are proposed in this pull request? Fix a bug of PagedFileReader. The PagedFileReader.read() method should use mFileSize, instead of mPos. Then add unit tests of PagedFileReader. ### Why are the changes needed? improve code coverage ### Does this PR introduce any user facing changes? no. ![image](https://github.com/Alluxio/alluxio/assets/30886176/49f3d75b-2b9d-49e7-a089-b19ba027c7d1) pr-link: Alluxio#18107 change-id: cid-0b929b7a1a8c9e32bb53be861e41f12b64d16183
Configuration menu - View commit details
-
Copy full SHA for a002643 - Browse repository at this point
Copy the full SHA a002643View commit details -
Use linked hash set to keep sequence when picking multiple worker
### What changes are proposed in this pull request? Use linked hash set to keep sequence when picking multiple worker ### Why are the changes needed? When loading multiple replica into Alluxio, we want deterministic sequence so we can decide which worker to load data and which worker the client can talk to. ### Does this PR introduce any user facing changes? na pr-link: Alluxio#18235 change-id: cid-1193f29eda77cd82af8dea4bc59914e97d1f8fa3
Configuration menu - View commit details
-
Copy full SHA for 6ede390 - Browse repository at this point
Copy the full SHA 6ede390View commit details
Commits on Oct 11, 2023
-
Add ser/de support for JSON or WorkerIdentity
### What changes are proposed in this pull request? Allow `WorkerIdentity` to be de/serialized to JSON and Java serialization format. ### Why are the changes needed? To be able to be embedded in `WorkerInfo` and other data objects that need to be de/serialized to JSON. ### Does this PR introduce any user facing changes? No. pr-link: Alluxio#18224 change-id: cid-9bd20ac6b1934feee10d130633e27323750efd17
Configuration menu - View commit details
-
Copy full SHA for f7ee2ad - Browse repository at this point
Copy the full SHA f7ee2adView commit details -
Refactor handling of java opts env vars in cli
address Alluxio#18249 refactor how java opt env vars are handled. previously they were treating as strings and then using a split operation to separate the single string into cmd line arguments. now they are handled as `[]string` immediately after parsing from env to properly handle any leading or trailing whitespaces. pr-link: Alluxio#18252 change-id: cid-44c4e14465432b0a5d9d8e84b488e9cd13c1e11b
Configuration menu - View commit details
-
Copy full SHA for 9ef97ec - Browse repository at this point
Copy the full SHA 9ef97ecView commit details
Commits on Oct 13, 2023
-
Use new worker id on worker and client
### What changes are proposed in this pull request? Use new worker identity for worker registration and client-side cluster management. ### Why are the changes needed? To allow workers to use generated IDs that are not assigned by the master. To allow clients to build its view of the cluster using more efficient and reliable worker identity implementation. ### Does this PR introduce any user facing changes? The worker identity is now persisted in etcd, if the worker uses etcd-based registration. pr-link: Alluxio#18210 change-id: cid-b044e5648329715bd59172a6fd39b406ef1cdad3
Configuration menu - View commit details
-
Copy full SHA for ba20fd0 - Browse repository at this point
Copy the full SHA ba20fd0View commit details -
Check load meta option at worker
### What changes are proposed in this pull request? Check the `-f, --load-metadata` option at the worker side. ### Why are the changes needed? The [doc](https://docs.alluxio.io/os/user/edge/en/operation/User-CLI.html#fs-ls) has ![2023-10-03_12-03-58](https://github.com/Alluxio/alluxio/assets/7149512/e839b2e5-ef2a-4834-b421-b414353572f4) and the "fs ls" command has the following option ``` -f, --load-metadata Force load metadata for immediate children in a directory ``` But this option is only set at the client side and is not checked at the worker side. So setting it has no use. Please check the following example ``` $ ./bin/alluxio fs ls / -rw-rw-r-- test test 1366 10-02-2023 17:39:54:284 FILE /file.01 -rw-rw-r-- test test 1366 10-02-2023 17:41:37:496 FILE /file.02 -rw-rw-r-- test test 1366 10-03-2023 11:04:18:177 FILE /file.03 $ ./bin/alluxio fs ls -f / -rw-rw-r-- test test 1366 10-02-2023 17:39:54:284 FILE /file.01 -rw-rw-r-- test test 1366 10-02-2023 17:41:37:496 FILE /file.02 -rw-rw-r-- test test 1366 10-03-2023 11:04:18:177 FILE /file.03 $ ./bin/alluxio fs ls -Dalluxio.user.file.metadata.sync.interval=0 / -rw-rw-r-- test test 1366 10-02-2023 17:39:54:284 FILE /file.01 -rw-rw-r-- test test 1366 10-02-2023 17:41:37:496 FILE /file.02 -rw-rw-r-- test test 1366 10-03-2023 11:04:18:177 FILE /file.03 -rw-rw-r-- test test 1366 10-03-2023 11:09:33:642 FILE /file.04 -rw-rw-r-- test test 1366 10-03-2023 11:12:57:668 FILE /file.05 ``` This PR checks it at the worker side and makes it matches the doc. ### Does this PR introduce any user facing changes? NO pr-link: Alluxio#18238 change-id: cid-d8e77340861e2b78e8908012b4fe262421202909
Configuration menu - View commit details
-
Copy full SHA for 20965a0 - Browse repository at this point
Copy the full SHA 20965a0View commit details
Commits on Oct 16, 2023
-
[SMALLFIX] Enable WorkerMetadataSyncIntegrationTest
### What changes are proposed in this pull request? Remove the @ignore annotations. It works now. ### Why are the changes needed? improve code quality. ### Does this PR introduce any user facing changes? no. pr-link: Alluxio#18264 change-id: cid-8124098803c0b05968d67b5ef5c31eed18b69187
Configuration menu - View commit details
-
Copy full SHA for d8a12c7 - Browse repository at this point
Copy the full SHA d8a12c7View commit details -
[SMALLFIX] Enable one UT case in OutStreamTest
### What changes are proposed in this pull request? A comment in OutStreamTest is useless and need to be removed. ### Why are the changes needed? Improve code quality. ### Does this PR introduce any user facing changes? No. pr-link: Alluxio#18272 change-id: cid-602cf0523ed6fbedc30ff448dfb4682006599baa
Configuration menu - View commit details
-
Copy full SHA for fae6922 - Browse repository at this point
Copy the full SHA fae6922View commit details -
Fix URIStatus notserializable bug
### What changes are proposed in this pull request? fix object notserializable bug ### Why are the changes needed? about issue: Alluxio#18265 ### Does this PR introduce any user facing changes? Please list the user-facing changes introduced by your change, including 1. change in user-facing APIs 2. addition or removal of property keys 3. webui pr-link: Alluxio#18266 change-id: cid-6c74a4a16b470de4a8d7f5a36d2ca87f11d7bb56
Configuration menu - View commit details
-
Copy full SHA for 8e169df - Browse repository at this point
Copy the full SHA 8e169dfView commit details
Commits on Oct 17, 2023
-
Add RESTful API for distributed load
Add RESTful API for distributed load. ### Usage: **SUBMIT:** description: submit a load job example: `http://localhost:28080/v1/load?path=/&opType=submit&partialListing=false&&verify=true&bandwidth=1000&loadMetadataOnly=false&verbose=true&skipIfExists=true` **STOP:** description: stop the load job example: http://localhost:28080/v1/load?path=/&opType=stop **PROGRESS:** description: get the progress of the load job example: `http://localhost:28080/v1/load?path=/&opType=progress&progressFormat=text&verbose=true` pr-link: Alluxio#18254 change-id: cid-8a2e4f6747d7ba6cb8c25032cc13ce4ec719da8f
Configuration menu - View commit details
-
Copy full SHA for 7c3b462 - Browse repository at this point
Copy the full SHA 7c3b462View commit details -
Change CosNUnderFileSystemFactory implementation UnderFileSystemFactory
Cherry-pick of existing commit. orig-pr: Alluxio#18143 orig-commit: Alluxio/alluxio@057804e orig-commit-author: gp1314 <814085234@qq.com> pr-link: Alluxio#18279 change-id: cid-4a80f6cdeae5b9bdb9e956c36838403ee6ce7c46
Configuration menu - View commit details
-
Copy full SHA for b597c61 - Browse repository at this point
Copy the full SHA b597c61View commit details -
Configuration menu - View commit details
-
Copy full SHA for 1bdb1dd - Browse repository at this point
Copy the full SHA 1bdb1ddView commit details -
Revert "Add RESTful API for distributed load"
This reverts commit 7c3b462. ### What changes are proposed in this pull request? Please outline the changes and how this PR fixes the issue. ### Why are the changes needed? Please clarify why the changes are needed. For instance, 1. If you propose a new API, clarify the use case for a new API. 2. If you fix a bug, describe the bug. ### Does this PR introduce any user facing changes? Please list the user-facing changes introduced by your change, including 1. change in user-facing APIs 2. addition or removal of property keys 3. webui pr-link: Alluxio#18284 change-id: cid-ae7e13a05237624a55adba85d2204fa1332b608a
Configuration menu - View commit details
-
Copy full SHA for b660e74 - Browse repository at this point
Copy the full SHA b660e74View commit details
Commits on Oct 18, 2023
-
### What changes are proposed in this pull request? Please outline the changes and how this PR fixes the issue. ### Why are the changes needed? Please clarify why the changes are needed. For instance, 1. If you propose a new API, clarify the use case for a new API. 2. If you fix a bug, describe the bug. ### Does this PR introduce any user facing changes? Please list the user-facing changes introduced by your change, including 1. change in user-facing APIs 2. addition or removal of property keys 3. webui pr-link: Alluxio#18285 change-id: cid-675c904a992ef44cd62d1daa51d4672234093483
Configuration menu - View commit details
-
Copy full SHA for 24a32d4 - Browse repository at this point
Copy the full SHA 24a32d4View commit details -
### What changes are proposed in this pull request? Remove a previously added file due to branch cleanup ### Why are the changes needed? Remove a previously added file due to branch cleanup ### Does this PR introduce any user facing changes? N/A pr-link: Alluxio#18283 change-id: cid-e84a3275f32b33c427277ba1ed2f57f6ee4a9927
Configuration menu - View commit details
-
Copy full SHA for f3494dc - Browse repository at this point
Copy the full SHA f3494dcView commit details
Commits on Oct 19, 2023
-
### What changes are proposed in this pull request? Load ufs path directly ### Why are the changes needed? Previously we concatenate root with relative path, now we directly pass ufs path So we can have multiple ufs support ### Does this PR introduce any user facing changes? load have to input full ufs path pr-link: Alluxio#18289 change-id: cid-3d9c4bad7b68af142ca7985954418bfef0f34b0b
Configuration menu - View commit details
-
Copy full SHA for 7d222bb - Browse repository at this point
Copy the full SHA 7d222bbView commit details -
Create locationCommandTest and fix some IntegretionTest bugs
### What changes are proposed in this pull request? I have created DoraFileLocationIntegretionTest and fix the bugs in IntegretionTest. ### Why are the changes needed? Because we just added the user command but donnot know whether they can be used correctly. ### Does this PR introduce any user facing changes? User can added other IntegretionTests due to the correct infra. pr-link: Alluxio#18294 change-id: cid-7832b08a4a8bf6bffcb8eaba4bd18c1761f7ce65
Configuration menu - View commit details
-
Copy full SHA for b0dc312 - Browse repository at this point
Copy the full SHA b0dc312View commit details -
### What changes are proposed in this pull request? Fix load command ufs string ### Why are the changes needed? bug fix ### Does this PR introduce any user facing changes? na pr-link: Alluxio#18297 change-id: cid-200e7eaf2ea7206725abbd802a2f2f8d9afc897e
Configuration menu - View commit details
-
Copy full SHA for cd7f2e1 - Browse repository at this point
Copy the full SHA cd7f2e1View commit details -
### What changes are proposed in this pull request? The loadCommands of FileSystemShellUtilsTest now works. ### Why are the changes needed? Improve code quality. ### Does this PR introduce any user facing changes? No. pr-link: Alluxio#18278 change-id: cid-3697d62cd123bdf832dece2b4b844f05133258a5
Configuration menu - View commit details
-
Copy full SHA for 077c372 - Browse repository at this point
Copy the full SHA 077c372View commit details -
[SMALLFIX] Remove OldLoadCommandIntegrationTest
### What changes are proposed in this pull request? Remove invalid integration test OldLoadCommandIntegrationTest. ### Why are the changes needed? Improve code quality. ### Does this PR introduce any user facing changes? No. pr-link: Alluxio#18290 change-id: cid-c86a3cb9e11971b6015af19a895009f1ff9f07eb
Configuration menu - View commit details
-
Copy full SHA for b5616c0 - Browse repository at this point
Copy the full SHA b5616c0View commit details -
[SMALLFIX] Delete JobServiceFaultToleranceShellTest
### What changes are proposed in this pull request? Delete JobServiceFaultToleranceShellTest, it makes no sense now. ### Why are the changes needed? Improve code quality. ### Does this PR introduce any user facing changes? No. pr-link: Alluxio#18280 change-id: cid-ae9f816c9219544c2b1c2c97b6fa33429d6835ad
Configuration menu - View commit details
-
Copy full SHA for fe98953 - Browse repository at this point
Copy the full SHA fe98953View commit details -
Add name to artifacts metadata
### What changes are proposed in this pull request? add name to metadata ### Why are the changes needed? Needed to identify docker artifacts ### Does this PR introduce any user facing changes? no pr-link: Alluxio#18302 change-id: cid-014522dd2772b44b9fe7a4d703c9e86d827ab44b
Configuration menu - View commit details
-
Copy full SHA for 908b7d6 - Browse repository at this point
Copy the full SHA 908b7d6View commit details
Commits on Oct 20, 2023
-
Implement HDFS list status iterator
When there are a lot of files in HDFS, it takes a large amount of time and memory to complete a `listStatus` request. Moreover, sometimes OOM occurs. This PR provides an iterator for the HDFS under file system to list files. pr-link: Alluxio#18295 change-id: cid-11019e8f163210c7664f3f2b6ddf3bae27e8ee8c
Configuration menu - View commit details
-
Copy full SHA for 62cc17a - Browse repository at this point
Copy the full SHA 62cc17aView commit details -
Add RESTful API for distributed load
Add RESTful API for distributed load. ### Usage: **SUBMIT:** description: submit a load job example: `http://localhost:28080/v1/load?path=s3a://jiamingmai-test/&opType=submit&partialListing=false&&verify=true&bandwidth=1000&loadMetadataOnly=false&verbose=true&skipIfExists=true` **STOP:** description: stop the load job example: `http://localhost:28080/v1/load?path=s3a://jiamingmai-test/&opType=stop` **PROGRESS:** description: get the progress of the load job example: `http://localhost:28080/v1/load?path=/&opType=progress&progressFormat=text&verbose=true` pr-link: Alluxio#18291 change-id: cid-3a4f1e6bb66dd6e6f518aaeb9e26bf17eda698de
Configuration menu - View commit details
-
Copy full SHA for 47b7667 - Browse repository at this point
Copy the full SHA 47b7667View commit details -
Add multi-dimensional metrics system
### What changes are proposed in this pull request? Introduce a new tool to add multi-dimensional metrics more easily. The new class holds all the metrics and exposes them in Prometheus type. ### Why are the changes needed? Add following metrics: - alluxio_data_access - alluxio_ufs_data_access - alluxio_cached_data_read - alluxio_meta_operation - alluxio_cached_storage - alluxio_cached_capacity - alluxio_cached_evicted_data - alluxio_cached_readable_storage ### Does this PR introduce any user facing changes? No. The new metrics will expose on a new URL which the webui is not currently using. pr-link: Alluxio#18307 change-id: cid-ed60de4c62f089c9ef6f95a99efc1bd383241ee0
Configuration menu - View commit details
-
Copy full SHA for 3c851d7 - Browse repository at this point
Copy the full SHA 3c851d7View commit details
Commits on Oct 23, 2023
-
Change UfsManager key to not rely on properties
### What changes are proposed in this pull request? In the UfsManager the UFS properties are not a prerequisite for locating an UFS instance. By not hashing on properties, the `get()` does not need to resolve the target UFS property keys, which saves cost. Before: 15.32% CPU in a worker stress test ![image](https://github.com/Alluxio/alluxio/assets/14806853/53039841-d133-4a10-99f8-f73b8458a746) After: <1% CPU in the same worker stress test ![image](https://github.com/Alluxio/alluxio/assets/14806853/f8233952-b4e8-4da1-be6d-f8518ad223b9) pr-link: Alluxio#18182 change-id: cid-7b48a9bd6d8446efe9f22d699462c25a9550683a
Configuration menu - View commit details
-
Copy full SHA for 013c9e1 - Browse repository at this point
Copy the full SHA 013c9e1View commit details -
Use new UfsManager.get() avoid creating conf instances every time
### What changes are proposed in this pull request? PR Alluxio#18182 was merged with changes conflicting to Alluxio#18285 which CI was not able to catch. So `main` is not able to compile after merging `18182`. This small change fixes that, by adapting new code in 18285 to interface changes in 18182. pr-link: Alluxio#18315 change-id: cid-2b5a7fe4b88b9ea4fe47b76e8cc0eed007120e48
Configuration menu - View commit details
-
Copy full SHA for 3a47fa2 - Browse repository at this point
Copy the full SHA 3a47fa2View commit details -
[SMALLFIX] Remove UnsetTtlTest
### What changes are proposed in this pull request? Remove UnsetTtlTest which is outdated now. ### Why are the changes needed? Improve code quality. ### Does this PR introduce any user facing changes? No. pr-link: Alluxio#18292 change-id: cid-3de116636374ad12962793e7ea19ba1170870db7
Configuration menu - View commit details
-
Copy full SHA for 6bfbbd6 - Browse repository at this point
Copy the full SHA 6bfbbd6View commit details -
Remove restriction in overwriting worker info within join logic
### What changes are proposed in this pull request? Now with worker id can be assumed from a different worker instance whether on a different pod in k8s or a different host machine for baremetal. The creation onto the persisted ring path : /DHT/DefaultAlluxioCluster/AUTHORIZED/ should not bail if a different value is seen. ### Why are the changes needed? to enable rejoin of a worker bearing same worker id but with different host or other WorkerInfo fileds. ### Does this PR introduce any user facing changes? No pr-link: Alluxio#18275 change-id: cid-51322e010e0d51ae4f81268c2bb607b568f08c46
Configuration menu - View commit details
-
Copy full SHA for 0884fc3 - Browse repository at this point
Copy the full SHA 0884fc3View commit details
Commits on Oct 24, 2023
-
Improve cold read performance by prefetching data on workers
### What changes are proposed in this pull request? In the position reader, preloading pages on workers to improve the cold read performance. ### Why are the changes needed? To improve the cold read performance. ### Does this PR introduce any user facing changes? N/A pr-link: Alluxio#18317 change-id: cid-97e28711cf7f7b3ce60da346c737505a365d3238
Configuration menu - View commit details
-
Copy full SHA for dc12c11 - Browse repository at this point
Copy the full SHA dc12c11View commit details -
Origin @dbw9580 ### What changes are proposed in this pull request? Add an prefetch cache policy that does not reset the sliding window when a cache read misses. Fuse sometimes create read requests that are out of the order and this helps the prefetch keep stable. pr-link: Alluxio#18318 change-id: cid-b0a6331fac06bb743724e8b68005e257b89aa64d
Configuration menu - View commit details
-
Copy full SHA for e78b66d - Browse repository at this point
Copy the full SHA e78b66dView commit details
Commits on Oct 25, 2023
-
Add getStatus RESTful API. **Example:** Get the specified directory/file information by the following request: `curl -X GET http://localhost:28080/v1/info?path=/tpcds-data` The response JSON looks like: `[ { "mType": "directory", "mName": "tpcds-data", "mPath": "/tpcds-data", "mUfsPath": "s3a://jiamingmai-test/tpcds-data", "mLastModificationTimeMs": 0, "mLength": 0, "mHumanReadableFileSize": "0B" } ]` <img width="597" alt="image" src="https://github.com/Alluxio/alluxio/assets/6129818/b38d644b-11da-4206-937b-6d61fd6b3a6c"> pr-link: Alluxio#18312 change-id: cid-6e563a21372e9fe1867a36d9b311e246b49c459e
Configuration menu - View commit details
-
Copy full SHA for 5b0e661 - Browse repository at this point
Copy the full SHA 5b0e661View commit details -
Fix MemoryPageStore error on commit() and clear cache on close()
### What changes are proposed in this pull request? This change mainly adds two fixes to the `MemoryPageStore`: 1. Add a noop implementation to commit() so it does nothing instead of throwing `UnsupportedOperationException` 2. Clears the cache on close explicitly pr-link: Alluxio#18322 change-id: cid-5e7656e1eca363ed2de40cd48025f464c41a2584
Configuration menu - View commit details
-
Copy full SHA for 08b9456 - Browse repository at this point
Copy the full SHA 08b9456View commit details -
### What changes are proposed in this pull request? All integration tests do not run now, this PR fixs it. ### Why are the changes needed? Fix bug. ### Does this PR introduce any user facing changes? No. pr-link: Alluxio#18313 change-id: cid-e3f71f5b817624c45b8a3fc2baae2b08b99cd109
Configuration menu - View commit details
-
Copy full SHA for e830663 - Browse repository at this point
Copy the full SHA e830663View commit details
Commits on Oct 26, 2023
-
Remove WorkerFuseIntegrationTest
### What changes are proposed in this pull request? The `WORKER_FUSE_ENABLED ` key is not used, so this test is outdated. ### Why are the changes needed? Improve code quality. ### Does this PR introduce any user facing changes? no. pr-link: Alluxio#18268 change-id: cid-8acc857afc5fff384d65bff748ad1c6b07a13e87
Configuration menu - View commit details
-
Copy full SHA for bb8e2cf - Browse repository at this point
Copy the full SHA bb8e2cfView commit details -
Delete useless UnderFileSystemAlluxioTest
### What changes are proposed in this pull request? Delete invalid UnderFileSystemAlluxioTest for changed codebase. ### Why are the changes needed? Improve code quality. ### Does this PR introduce any user facing changes? No. pr-link: Alluxio#18277 change-id: cid-8814028d7725c927c7973917f1da6d22e1f827db
Configuration menu - View commit details
-
Copy full SHA for 6185ba0 - Browse repository at this point
Copy the full SHA 6185ba0View commit details -
Update worker api support for load multi replicas
### What changes are proposed in this pull request? Update worker api support for load multi replicas ### Why are the changes needed? part of PR to support load multi replicas ### Does this PR introduce any user facing changes? na pr-link: Alluxio#18296 change-id: cid-0213f2aba669b7687ac42cf932cdcec911d397a4
Configuration menu - View commit details
-
Copy full SHA for 7a5734f - Browse repository at this point
Copy the full SHA 7a5734fView commit details -
### What changes are proposed in this pull request? Add Rust toolchain to the Docker image used for CI. ### Why are the changes needed? Allow CI to compile and run Rust code. ### Does this PR introduce any user facing changes? No. pr-link: Alluxio#18319 change-id: cid-6912fe3659bb0044a6980d9ac2f32f79101efc44
Configuration menu - View commit details
-
Copy full SHA for 53c49f7 - Browse repository at this point
Copy the full SHA 53c49f7View commit details -
Add regx pattern file filter for distributed load
Add regx pattern file filter for distributed load. **Example:** The following request allows us to load the files under `/test-load` directory with "hello" prefix: `curl -X GET http://localhost:28080/v1/load?path=s3a://jiamingmai-test/test-load&opType=submit&verbose=true&fileFilterRegx=^hello.*` pr-link: Alluxio#18311 change-id: cid-4ec2bfe58bfba413f6d2925f5b3937bd6f5c2eb1
Configuration menu - View commit details
-
Copy full SHA for d0ad98f - Browse repository at this point
Copy the full SHA d0ad98fView commit details -
Create DoraLsCommandIntegrationTest
### What changes are proposed in this pull request? I have created a new file named DoraLsCommandIntegrationTest and it can test the 'ls' command whether it can correctly run.Also, I revised the base class that can create byte files in alluxio.What's more, I have added a new test named DoraMkdirCommandIntegrationTest which it can test cli 'mkdir'. ### Why are the changes needed? 1、New Test does not have a function that can create file in alluxio. 2、Add a IntegrationTest. ### Does this PR introduce any user facing changes? No. pr-link: Alluxio#18325 change-id: cid-067ec742deab39af294d089dec932b69c0362682
Configuration menu - View commit details
-
Copy full SHA for 8f41c65 - Browse repository at this point
Copy the full SHA 8f41c65View commit details
Commits on Oct 27, 2023
-
[SMALLFIX] Extract create UfsBaseFileSystem for extension
### What changes are proposed in this pull request? Extract the logic to create `UfsBaseFileSystem` into `FileSystemContext`, for possible extension. This change is functionally a refactor that changes nothing. pr-link: Alluxio#18333 change-id: cid-105be187f763f83680f64bd963646ce32eb58493
Configuration menu - View commit details
-
Copy full SHA for 961ed17 - Browse repository at this point
Copy the full SHA 961ed17View commit details -
Fix a buffer memory leak in NettyReadHandlerStateMachine
### What changes are proposed in this pull request? resolves Alluxio#18324 Disclaimer: I might have monkey-typed this fix but I still do not know anything about buffer ref counting. This fix does NOT make me the owner of this state machine. pr-link: Alluxio#18323 change-id: cid-eb5bde353c08d3d9bdd39da5b9caf13681bae495
Configuration menu - View commit details
-
Copy full SHA for 14efca9 - Browse repository at this point
Copy the full SHA 14efca9View commit details -
[SMALLFIX] Remove LeaderCommandIntegrationTest.java
### What changes are proposed in this pull request? Remove invalid LeaderCommandIntegrationTest.java, the command is deleted. ### Why are the changes needed? improve code quality. ### Does this PR introduce any user facing changes? no. pr-link: Alluxio#18287 change-id: cid-43a00b7ec27735262bfa45b7b37331389d82a881
Configuration menu - View commit details
-
Copy full SHA for d3bfd8c - Browse repository at this point
Copy the full SHA d3bfd8cView commit details -
[SMALLFIX] Remove NeedsSyncCommandIntegrationTest.java
### What changes are proposed in this pull request? Remove NeedsSyncCommandIntegrationTest, which command is deleted. ### Why are the changes needed? Improve code quality. ### Does this PR introduce any user facing changes? No. pr-link: Alluxio#18288 change-id: cid-0c6040e1804065067032384e6046c0d47ccf8312
Configuration menu - View commit details
-
Copy full SHA for 779ece2 - Browse repository at this point
Copy the full SHA 779ece2View commit details -
[SMALLFIX] Recover HelpCommandIntegrationTest
### What changes are proposed in this pull request? HelpCommandIntegrationTest works now. ### Why are the changes needed? Improve code quality. ### Does this PR introduce any user facing changes? No. pr-link: Alluxio#18293 change-id: cid-ccb04b9ede68fae2342d6a25e443b85ccd8f990c
Configuration menu - View commit details
-
Copy full SHA for 37b33dc - Browse repository at this point
Copy the full SHA 37b33dcView commit details
Commits on Oct 30, 2023
-
### What changes are proposed in this pull request? Add rust spdk library and design structures for cache. ### Why are the changes needed? For NVMe SSD cache requirement. ### Does this PR introduce any user facing changes? No. ### Benchmark Result <table style="text-align:center;"> <tbody> <tr> <th rowspan="2">block size</th> <th colspan="3">time consumption</th> <th rowspan="2">throughput</th> </tr> <tr> <th>millisecond</th> <th>microsecond</th> <th>nanosecond</th> </tr> <tr> <td>512B</td> <td>10ms</td> <td>10194us</td> <td>10194421ns</td> <td>0.048MB/s</td> </tr> <tr> <td>1KB</td> <td>13ms</td> <td>13472us</td> <td>13472304ns</td> <td>0.072MB/s</td> </tr> <tr> <td>4KB</td> <td>9ms</td> <td>9242us</td> <td>9242424ns</td> <td>0.423MB/s</td> </tr> <tr> <td>16KB</td> <td>8ms</td> <td>8585us</td> <td>8585361ns</td> <td>1.820MB/s</td> </tr> <tr> <td>64KB</td> <td>11ms</td> <td>11030us</td> <td>11030930ns</td> <td>5.666MB/s</td> </tr> <tr> <td>256KB</td> <td>15ms</td> <td>15962us</td> <td>15962353ns</td> <td>15.662MB/s</td> </tr> <tr> <td>1MB</td> <td>13ms</td> <td>13059us</td> <td>13059113ns</td> <td>76.575MB/s</td> </tr> <tr> <td><b>4MB</b></td> <td><b>28ms</b></td> <td><b>28930us</b></td> <td><b>28930274ns</b></td> <td><b>138.264MB/s</b></td> </tr> <tr> <td><b>16MB</b></td> <td><b>79ms</b></td> <td><b>79423us</b></td> <td><b>79423390ns</b></td> <td><b>201.452MB/s</b></td> </tr> <tr> <td>64MB</td> <td>308ms</td> <td>308856us</td> <td>308856745ns</td> <td>207.216MB/s</td> </tr> <tr> <td>256MB</td> <td>1218ms</td> <td>1218323us</td> <td>1218323252ns</td> <td>210.125MB/s</td> </tr> <tr> <td>1GB</td> <td>5056ms</td> <td>5056277us</td> <td>5056277683ns</td> <td>202.521MB/s</td> </tr> </tbody> </table> pr-link: Alluxio#18231 change-id: cid-92ee56270bc5bb237ecc0df78c2974e1051bc543
Configuration menu - View commit details
-
Copy full SHA for eecf2a5 - Browse repository at this point
Copy the full SHA eecf2a5View commit details -
### What changes are proposed in this pull request? Update load job to adopt multi replicas ### Why are the changes needed? part of PR to support loading multi replicas ### Does this PR introduce any user facing changes? new load option `replicas` pr-link: Alluxio#18320 change-id: cid-7d01ca19a28faf4c7773cbf5c355dd6cf070728f
Configuration menu - View commit details
-
Copy full SHA for 32f675f - Browse repository at this point
Copy the full SHA 32f675fView commit details
Commits on Oct 31, 2023
-
Fix the time check logic for judging stale client channels to be inac…
### What changes are proposed in this pull request? Fix bug involved by Alluxio#18332. Alter the time judgment logic for judging whether stale client channels are inactive. Using the LocaTime object cannot correctly judge whether a channel client is inactive, because a LocalTime plus or minus time offset only changes the hour, minute, second attribute value, and It will not affect the date, you actually need to use the LocalDateTime object instead. ### Why are the changes needed? Please clarify why the changes are needed. For instance, In the code, the LocaTime class is used to determine that a client channel is inactive. The LocalTime object adds or subtracts the time offset. It only changes the hour, minute and second attribute value and does not affect the date. In fact, you need to use the LocalDateTime object. In other words, the three-day certification cycle judgment should be based on date and time, not just time. ### Does this PR introduce any user facing changes? Please list the user-facing changes introduced by your change, including None pr-link: Alluxio#18340 change-id: cid-5b69e0c87d3bad8556ae27d491f3e0dc567378b9
Configuration menu - View commit details
-
Copy full SHA for 660e00d - Browse repository at this point
Copy the full SHA 660e00dView commit details -
### What changes are proposed in this pull request? Fix a bug when file size < 1 page, alluxio considers all pages are cached no matter if it really caches it or not. ### Why are the changes needed? Bug fixing ### Does this PR introduce any user facing changes? N/A pr-link: Alluxio#18347 change-id: cid-da4bdce6615e4b1d9777a98ab335bf36e503d102
Configuration menu - View commit details
-
Copy full SHA for ec4868d - Browse repository at this point
Copy the full SHA ec4868dView commit details
Commits on Nov 1, 2023
-
Change the metrics registry and some metrics
### What changes are proposed in this pull request? Add some metrics and change the registry to expose the metrics more accurately. And add the call point of the capacity-related metrics. ### Why are the changes needed? Use default registry will expose all the metrics and some of them are not meaningful for all components. ### Does this PR introduce any user facing changes? no pr-link: Alluxio#18350 change-id: cid-646ee1f3e41171d2147df29fccaf5a3476b66033
Configuration menu - View commit details
-
Copy full SHA for 2114bea - Browse repository at this point
Copy the full SHA 2114beaView commit details -
Move job related operations to DoraCacheFileSystem
### What changes are proposed in this pull request? Please outline the changes and how this PR fixes the issue. ### Why are the changes needed? Please clarify why the changes are needed. For instance, 1. If you propose a new API, clarify the use case for a new API. 2. If you fix a bug, describe the bug. ### Does this PR introduce any user facing changes? Please list the user-facing changes introduced by your change, including 1. change in user-facing APIs 2. addition or removal of property keys 3. webui pr-link: Alluxio#18352 change-id: cid-b5f5b695a1c32a65ff3b303cb8227889bee5d81c
Configuration menu - View commit details
-
Copy full SHA for ccf20e4 - Browse repository at this point
Copy the full SHA ccf20e4View commit details -
Refactor PagedDoraWorker by injecting MetaManager and UfsManager
### What changes are proposed in this pull request? 1. If an object is created inside `PagedDoraWorker` constructor, extract that creation to before the constructor and use dependency injection to inject it to the worker object. This doesn't change any creation logic, just a refactor to better adapt to dependency injection flavor. 2. There is a circular dependency between `MetaManager` and `PagedDoraWorker`. This change removes that cycle. Now we create one, then create the other. Before, we create one and in the construction, we let `this` ref escape and create the other. Some methods are either moved or changed to `static`. 3. By adapting to dependency injection, we rely on `UfsManager` interface instead of `DoraUfsManager` implementation. Some method signatures are extracted to the interface level. 4. A few other small refactors to get rid of some downcasts and variable scope changes. Reasons are attached in comments on this PR. ### Why are the changes needed? Improve code quality and extensibility. ### Does this PR introduce any user facing changes? No. All refactor changes are small and equivalent to existing code. So nothing should break. pr-link: Alluxio#18181 change-id: cid-4f9e9bc770b12253188bb541dd456ef3cd889c2b
Configuration menu - View commit details
-
Copy full SHA for 0e83207 - Browse repository at this point
Copy the full SHA 0e83207View commit details
Commits on Nov 2, 2023
-
Add remote only policy to StressWorkerBench
### What changes are proposed in this pull request? Add one `RemoteOnlyPolicy` implementation for testing. This is usable for reading all files from remote nodes, rather than the local node itself. Generally, this policy keeps a thread-safe list including all workers. When one thread reads, the round-robin list returns all available workers after a roulette (putting the first element in the list to the end) , and choose the first remote worker to read from. We also added available options of `--mode` in StressWorkerBench to use the new remote only policy. ### Why are the changes needed? The new policy is for internal testing where all test clients find the remote worker for IO. This policy should not be used in real deployments because if all clients find remote worker, overall throughput can be quite low due to bandwidth restrictions. ### Does this PR introduce any user facing changes? No, RemoteOnlyPolicy should only be used in internal testing pr-link: Alluxio#18273 change-id: cid-ce534382a1ebd86230296475f4e2d3c6dd862033
Configuration menu - View commit details
-
Copy full SHA for ad3baa1 - Browse repository at this point
Copy the full SHA ad3baa1View commit details -
Fix the bug that command line doesn't support regx file filter
Fix the bug that command line doesn't support regx file filter pr-link: Alluxio#18359 change-id: cid-c2ec9a5394a1ad776e31251a9c113f4115cb651d
Configuration menu - View commit details
-
Copy full SHA for b2df584 - Browse repository at this point
Copy the full SHA b2df584View commit details
Commits on Nov 3, 2023
-
Configuration menu - View commit details
-
Copy full SHA for d8342f7 - Browse repository at this point
Copy the full SHA d8342f7View commit details -
Fix file mtime display when ufs mtime is not null
### What changes are proposed in this pull request? Alluxio#18357 ### Why are the changes needed? mtime display error when ufs is hdfs ![企业微信截图_9ff2cbaf-4e24-4cbc-ae11-6a5ace7812f1](https://github.com/Alluxio/alluxio/assets/800042/606cbcb0-b07f-4f91-8919-b1fe1b9e6dcc) after this commit: ![企业微信截图_8c7ee012-851c-425e-a4c1-d22f0d39984b](https://github.com/Alluxio/alluxio/assets/800042/e05bfed5-c929-43c1-a633-6b6dee3f10ef) ### Does this PR introduce any user facing changes? Please list the user-facing changes introduced by your change, including 1. change in user-facing APIs. no 2. addition or removal of property keys. no 3. webui. no pr-link: Alluxio#18358 change-id: cid-54cc12a64383e77e5cd52b7ac89976714a93612b
Configuration menu - View commit details
-
Copy full SHA for 24ff6b5 - Browse repository at this point
Copy the full SHA 24ff6b5View commit details
Commits on Nov 6, 2023
-
Disable xatter in PagedDoraWorker test
### What changes are proposed in this pull request? XATTR config added to PagedDoraWorkerTest ### Why are the changes needed? LocalUFS does not support xattr ### Does this PR introduce any user facing changes? no pr-link: Alluxio#18339 change-id: cid-76ff63355ee62dea9514666cb8556dc84a83b8b2
Configuration menu - View commit details
-
Copy full SHA for ea6af23 - Browse repository at this point
Copy the full SHA ea6af23View commit details -
Fix UT readMultipleFiles in LocalCacheFileInStreamTest
### What changes are proposed in this pull request? Added a modification to the code in the LocalCacheFileInStreamTest class.Specifically, the code change addresses an issue with the readMultipleFiles unit test in the LocalCacheFileInStreamTest. The proposed change to fix this issue is to add the setUfsPath method within the generateURIStatus method and set it to be identical to the Path. This ensures that the UfsPath is properly set for each file, allowing the readMultipleFiles test to pass successfully. ### Why are the changes needed? The changes in this pull request are needed because they address a specific issue with the LocalCacheFileInStreamTest class's readMultipleFiles unit test. The readMultipleFiles unit test was failing because it relied on the correct calculation of FieldId in Dora, which is based on the UfsPath. However, in this test, the UfsPath for files was not being set and remained empty, leading to all files having the same FieldId after hash calculation. The absence of a proper UfsPath for files in the test class caused multiple files to share the same FieldId, causing them to read data from a single file instead of their respective files. To resolve this issue, the proposed solution is to set the UfsPath within the generateURIStatus method to be identical to the Path. This ensures that each file has a distinct and properly set UfsPath, allowing the readMultipleFiles test to pass without affecting other unit tests. ### Does this PR introduce any user facing changes? Please list the user-facing changes introduced by your change, including 1. change in user-facing APIs 2. addition or removal of property keys 3. webui pr-link: Alluxio#18168 change-id: cid-422d5b4eb8bd94b044dbae9181dfc91810e300ed
Configuration menu - View commit details
-
Copy full SHA for c7c0820 - Browse repository at this point
Copy the full SHA c7c0820View commit details -
### What changes are proposed in this pull request? Features: 1. Retry in the end of the load process only once 2. Persist failed file list once the load job is done Improvements: 1. Only keep the most recent 1000 errroes to save memory 2. Create a LoadSubTaskIterator and refactor the task generation progress 3. Add an option to skip fetching ufs status again to speed up the retry (deafaulted on) 4. --progress displays the elapsed time 5. Add a metric to record error attribution 6. --progresss displays subtask level metrics additionally 7. fast fail the task when hash ring moves 8. --progress --verbose displays information about the subtask (instead of the file name only) Bug fixes: 1. Fix the wrong behavior where the distributed load still gives all workers when dynamic hash ring is disabled 2. Fix the incorrect display of the loading progress report ![image](https://github.com/Alluxio/alluxio/assets/6771554/ff2c9ccb-0eed-472d-98cc-9cc53def37ba) pr-link: Alluxio#18362 change-id: cid-7fb3f6f9230c04a01f7f466b32726c906e191ee2
Configuration menu - View commit details
-
Copy full SHA for d766ae4 - Browse repository at this point
Copy the full SHA d766ae4View commit details -
[SMALLFIX] Allow extension on
UfsManager.tryUfsFileSystem()
### What changes are proposed in this pull request? Change the method visibility to `protected` to allow overriding this method. ### Why are the changes needed? I can override this method to a noop so the `UfsManager` does not connect to the real UFS instance. pr-link: Alluxio#18374 change-id: cid-8869941af37e8eda89b37f1637afd21743a6c61e
Configuration menu - View commit details
-
Copy full SHA for 741563b - Browse repository at this point
Copy the full SHA 741563bView commit details -
Fix COS/OSS/OBS iterative listing
### What changes are proposed in this pull request? Please outline the changes and how this PR fixes the issue. ### Why are the changes needed? Please clarify why the changes are needed. For instance, 1. If you propose a new API, clarify the use case for a new API. 2. If you fix a bug, describe the bug. ### Does this PR introduce any user facing changes? Please list the user-facing changes introduced by your change, including 1. change in user-facing APIs 2. addition or removal of property keys 3. webui pr-link: Alluxio#18383 change-id: cid-ccd3bcc9e11d88274603ec058e091e97ea136448
Configuration menu - View commit details
-
Copy full SHA for bad173c - Browse repository at this point
Copy the full SHA bad173cView commit details
Commits on Nov 7, 2023
-
Print warning messages for stale open handle instead of removing it
### What changes are proposed in this pull request? Checking all open handles periodically. if some handle is stale (inactive for more than 24 hours), please warning messages. Every time an open handle is used, its last access time is updated. In future, we may need to handle this case instead of printing warning messages. ### Why are the changes needed? If an open handle is inactive for a long time, it may be caused by stuck, or by disconnected client. So a warning message is generated. Originally the open handle was closed in such case. But if a write is really slow or paused there, we are not going to close this handle. ### Does this PR introduce any user facing changes? N/A. pr-link: Alluxio#18267 change-id: cid-5aa996931b6ae28513fbd00a63427116c17b4da6
Configuration menu - View commit details
-
Copy full SHA for 912731c - Browse repository at this point
Copy the full SHA 912731cView commit details -
Fix cosn ufs cannot find class
Fix bug involved by Alluxio#17024 After Alluxio#18143, HdfsUnderFileSystemFactory is no longer included in COSN jar therefore no need to exclude. Same PR as Alluxio#18303 but pushing to the main branch pr-link: Alluxio#18336 change-id: cid-c79454f41157781a29fb04d6067ea8cf16520977
Configuration menu - View commit details
-
Copy full SHA for 8d3ab67 - Browse repository at this point
Copy the full SHA 8d3ab67View commit details -
Change logging level in NettyChannelPool
### What changes are proposed in this pull request? Reduce unnecessary logging to help debugging ### Why are the changes needed? Noisy logging like `2023-09-11 20:02:30,750 INFO NettyChannelPool - Channel closed` ### Does this PR introduce any user facing changes? N/A pr-link: Alluxio#18203 change-id: cid-d7c3842e0c76f0527a83345257b1ca2388927da9
Configuration menu - View commit details
-
Copy full SHA for 00a527d - Browse repository at this point
Copy the full SHA 00a527dView commit details -
Add a configurable DLQ capacity of avoid OOM
### What changes are proposed in this pull request? 1. Add a property to limit the DLQ capacity in distributed load 2. Fix a position reader bug where the prefetch size will converge at 2 * read length - 1 pr-link: Alluxio#18387 change-id: cid-2a20dd7c85de56926cd3a6bbc67c3ed6e8c14299
Configuration menu - View commit details
-
Copy full SHA for 723c720 - Browse repository at this point
Copy the full SHA 723c720View commit details -
Fix the issue to get a tag of a non-existed in OSS/COS
### What changes are proposed in this pull request? Add exception handling for OSS and COS ufs in getting object tags and other apis. ### Why are the changes needed? When handling exceptions, we need to identify exceptions that can be allowed, and other errors need to be converted to an Alluxio exception. ### Does this PR introduce any user facing changes? Please list the user-facing changes introduced by your change, including 1. change in user-facing APIs 2. addition or removal of property keys 3. webui pr-link: Alluxio#18388 change-id: cid-2eb458f3ec6955321981dff350dacc9f33104c1b
Configuration menu - View commit details
-
Copy full SHA for d28559a - Browse repository at this point
Copy the full SHA d28559aView commit details -
### What changes are proposed in this pull request? Please outline the changes and how this PR fixes the issue. ### Why are the changes needed? Please clarify why the changes are needed. For instance, 1. If you propose a new API, clarify the use case for a new API. 2. If you fix a bug, describe the bug. ### Does this PR introduce any user facing changes? Please list the user-facing changes introduced by your change, including 1. change in user-facing APIs 2. addition or removal of property keys 3. webui pr-link: Alluxio#18391 change-id: cid-24fcdc50fc0b6560f77c0f2f31ebadd78000bac8
Configuration menu - View commit details
-
Copy full SHA for 420e560 - Browse repository at this point
Copy the full SHA 420e560View commit details -
Fix chaotic chinese characters with golang cli commands
When debugging load command, it is found that `./bin/alluxio` CLI can't process Chinese characters correctly. There must be something wrong with the CLI encoding or decoding. It seems that the Chinese characters are encoded incorrectly. This happens because the construction of CLI commands does not initialize environmental variables correctly. OS-level environmental variables are ignored, so the program uses a false encoder. This pull request fixes this problem. pr-link: Alluxio#18386 change-id: cid-a15fb106b4a6cd77087cdbba82e426f1ba1e4b4c
Configuration menu - View commit details
-
Copy full SHA for 7d15de7 - Browse repository at this point
Copy the full SHA 7d15de7View commit details -
[SMALLFIX] Remove unused objects initialize
### What changes are proposed in this pull request? ### Why are the changes needed? `objects` is initialized by empty array but we never used this default value. ### Does this PR introduce any user facing changes? Please list the user-facing changes introduced by your change, including 1. change in user-facing APIs. no 2. addition or removal of property keys. no 3. webui. no pr-link: Alluxio#18380 change-id: cid-f16cc464ebc647d4c9be79b2151fb06c4ec71ed4
Configuration menu - View commit details
-
Copy full SHA for bd99175 - Browse repository at this point
Copy the full SHA bd99175View commit details -
### What changes are proposed in this pull request? Improve UpdateChecker ### Why are the changes needed? 1. remove irrelevant master features 2. turn on fuse update check pr-link: Alluxio#18101 change-id: cid-8f6801fedda5f7710d111939e50cfd5f4372b7e5
Configuration menu - View commit details
-
Copy full SHA for 246c7ee - Browse repository at this point
Copy the full SHA 246c7eeView commit details
Commits on Nov 8, 2023
-
Configuration menu - View commit details
-
Copy full SHA for 18c2190 - Browse repository at this point
Copy the full SHA 18c2190View commit details -
Fix StressWorkerBench with remote worker policy
This pull request fixes a bug on stress worker bench. 1. When using the remote worker policy, running `findNextWorker()`, multiple threads increment `mNextIndex`, leading to possible array out of bounds errors when getting worker info. 2. If use remote worker policy and the cluster size is 1, no workers can be found in `findNextWorker()`, resulting in errors. This fix turns `findNextWorker()` into a synchronized function, which can avoid data corruption. In addition, this policy can only be used when cluster size > 1. pr-link: Alluxio#18377 change-id: cid-2e391549b5d81ca3f1ff60f7a6c7eb59e99cc736
Configuration menu - View commit details
-
Copy full SHA for 5474513 - Browse repository at this point
Copy the full SHA 5474513View commit details -
a previous PR Alluxio#18313 restored the integration tests but caused the unit tests to also run with the integration tests. the issue was correctly identified as an issue with the maven project list but the underlying problem was that the project name was not correctly set; `dora/tests` is not a module. it was previously a module until `dora/tests/testcontainers` was introduced, thus separating testcontainer tests vs integration tests. this fix updates the previous project list with the correct module name pr-link: Alluxio#18392 change-id: cid-5f899f224abfbea4b7608d72cc2d38fed6449836
Configuration menu - View commit details
-
Copy full SHA for 502b0dd - Browse repository at this point
Copy the full SHA 502b0ddView commit details -
cherry pick empty because Tencent Cloud EMR doc have been removed in …
…3.x [DOCFIX] Fix doc errors for Tencent Cloud EMR Please outline the changes and how this PR fixes the issue. Tencent Cloud EMR has updated some version information and needs to be synchronized to community documents Please list the user-facing changes introduced by your change, including 1. change in user-facing APIs 2. addition or removal of property keys 3. webui pr-link: Alluxio#17747 change-id: cid-73d26a59de4f1023081ed0ffaf91cffcc2f8db0e
Configuration menu - View commit details
-
Copy full SHA for 615a6d8 - Browse repository at this point
Copy the full SHA 615a6d8View commit details -
Close client when heartbeat executor closed
### What changes are proposed in this pull request? Please outline the changes and how this PR fixes the issue. ### Why are the changes needed? Please clarify why the changes are needed. For instance, 1. If you propose a new API, clarify the use case for a new API. 2. If you fix a bug, describe the bug. ### Does this PR introduce any user facing changes? Please list the user-facing changes introduced by your change, including 1. change in user-facing APIs 2. addition or removal of property keys 3. webui pr-link: Alluxio#17785 change-id: cid-657e47ef7657714a572a1f9811bbcc524bb8cfcc
Configuration menu - View commit details
-
Copy full SHA for 8eba127 - Browse repository at this point
Copy the full SHA 8eba127View commit details -
Fix a deadlock in FileSystemContext
### What changes are proposed in this pull request? Fix a deadlock. ### Why are the changes needed? On the Alluxio client, there is a monitor lock of `FileSystemContext` to sync individual client-to-master operations. One application thread can successfully hold the monitor lock (`alluxio.client.file.FileSystemContext`) in order to get worker list, but blocked by another “lock” `FileSystemContextReinitializer` (waiting for latch on on-going RPCs down to zero) for acquiring the block master client to really connect to master process (waiting for other callers involving `FileSystemContextReinitializer` to finish). On the other hand, another heartbeat thread "config-hash-master-heartbeat-0" is awaking periodically to sync with the master process to fetch the latest configuration. This thread detected the conf update and thus entered `FileSystemContextReinitializer` (bumping latch) but was blocked by waiting for the monitor lock of `alluxio.client.file.FileSystemContext` in order to get the master address. This PR moves `getMasterAddress` outside `reinit` block to avoid holding the `Reinitializer` object and wait for the monitor object of `FileSystemContext`. ### Does this PR introduce any user facing changes? No pr-link: Alluxio#18109 change-id: cid-0fab540f7dbef3661acb44e7472425add163b019
Configuration menu - View commit details
-
Copy full SHA for 704a460 - Browse repository at this point
Copy the full SHA 704a460View commit details -
Fix local cache identifier to solve the local cache consistent issue …
…and resolve a trivial conflict Fix local cache identifier If not including `modification time`, it may have consistency issues. NO pr-link: Alluxio#17514 change-id: cid-c9c2e91e53d4a95d4f3a165918fc23c02f623891
Configuration menu - View commit details
-
Copy full SHA for 6be7125 - Browse repository at this point
Copy the full SHA 6be7125View commit details -
Support getFileBlockLocation in LocalCacheFileSystem
### What changes are proposed in this pull request? Delegate `getFileBlockLocation` to external file system in `LocalCacheFileSystem`. ### Why are the changes needed? Otherwise, `LocalCacheFileSystem` inherits the default behavior of `org.apache.hadoop.fs.FileSystem` which returns `localhost` only. ### Does this PR introduce any user facing changes? No. pr-link: Alluxio#17672 change-id: cid-eb545dbd8ed42001d074fecfb9c8d6b118a559c1
Configuration menu - View commit details
-
Copy full SHA for c453859 - Browse repository at this point
Copy the full SHA c453859View commit details -
Support read bytebuffer for non ByteBufferReadable input stream
### What changes are proposed in this pull request? Support read bytebuffer for non ByteBufferReadable input stream ### Why are the changes needed? Without this changes, a non ByteBufferReadable input stream can throw exception when the bytebuffer apis are called. ### Does this PR introduce any user facing changes? No pr-link: Alluxio#17982 change-id: cid-1363dbbeca327cbb7b4d39cea1afbb8c51905483
Configuration menu - View commit details
-
Copy full SHA for 1e283df - Browse repository at this point
Copy the full SHA 1e283dfView commit details -
Configuration menu - View commit details
-
Copy full SHA for d0d6b0c - Browse repository at this point
Copy the full SHA d0d6b0cView commit details -
Use correct audit log configuration in proxy
### What changes are proposed in this pull request? The v1 and v2 versions of proxy use uniform judgment conditions. pr-link: Alluxio#18371 change-id: cid-1842a0b96b5233f9a456e1aee7dbd95e2bd2d5e0
Configuration menu - View commit details
-
Copy full SHA for dd2d0f9 - Browse repository at this point
Copy the full SHA dd2d0f9View commit details -
Support Mkdir/CreateFile with configured default umask in HDFS API
### What changes are proposed in this pull request? Support creating directories and files with default permissions based on configuration propertykey in HDFS API. ### Why are the changes needed? Alluxio Hdfs api hasn't the corresponding implementation of Mkdir without permission parameter. If it does not carry permission, the umask property in the configuration item shall prevail. ### Does this PR introduce any user facing changes? pr-link: Alluxio#18253 change-id: cid-6ddd2243bac00ebbdbdff1e731036c0d3d6228c8
Configuration menu - View commit details
-
Copy full SHA for f474bd7 - Browse repository at this point
Copy the full SHA f474bd7View commit details -
cherry pick without resolving conflicts: Get User/Group/Permission of…
… existed objects in OSS ufs Using the user of the OSS bucket to represent the user of the object loaded from OSS. Add the mapping way from OSS username to the custom username. Add the default permission mode for existing files loaded from OSS. the object loaded from OSS ufs will be null user, null group, and 777 by default. pr-link: Alluxio#18262 change-id: cid-7e56861c9def695876cde32d1e0aa453b512470d
Configuration menu - View commit details
-
Copy full SHA for b64b0cf - Browse repository at this point
Copy the full SHA b64b0cfView commit details -
Configuration menu - View commit details
-
Copy full SHA for a5d33f9 - Browse repository at this point
Copy the full SHA a5d33f9View commit details -
Configuration menu - View commit details
-
Copy full SHA for 3350541 - Browse repository at this point
Copy the full SHA 3350541View commit details -
[SMALLFIX] Fix broken ChmodCommandIntegrationTest
### What changes are proposed in this pull request? The old one is ignored, now bring it back. ### Why are the changes needed? Improve code quality. ### Does this PR introduce any user facing changes? No. pr-link: Alluxio#18376 change-id: cid-2cdb7611d32e50fc16044192edf1122f4a3cd646
Configuration menu - View commit details
-
Copy full SHA for c9760b8 - Browse repository at this point
Copy the full SHA c9760b8View commit details -
Use generic Exception to catch all exceptions in getStatus()
Also print file path in getStatus() exception on client side ### What changes are proposed in this pull request? Use generic Exception to catch all exceptions in getStatus(), just like that in all other handlers. File path will be included in log message for getStatus(). ### Why are the changes needed? Sometimes, exceptions are thrown in under file system and/or other third-party libraries. We don't know the exact type of the specific exceptions. If they are not caught, client will report unknown exceptions in logs. This is misleading. ### Does this PR introduce any user facing changes? Client will show logs with detailed exception information if there is any. For getStatus(), file path will also be shown in log. pr-link: Alluxio#18398 change-id: cid-b6acee3dd0c3e2e03e66bbf195cbda1e95f6bf64
Configuration menu - View commit details
-
Copy full SHA for 6dc2fae - Browse repository at this point
Copy the full SHA 6dc2faeView commit details -
Merge master-2.x commits 2023/07/01~2023/11/08 into main
### What changes are proposed in this pull request? Merge missing commits from master-2.x to main. The commits in 2023/07/01~2023/11/08 from Alluxio/alluxio@main...master-2.x will be included by this PR. We do this merge to catch missing fixes from `master-2.x` and catch the train before `main` cuts a release. Alluxio#17747 is not cherry picked because tencent cloud EMR doc is removed Alluxio#17755 is not cherry picked because DistLoadCliRunner has been removed in 3.x Alluxio#17758 is not cherry picked because MonoBlockStore has been removed in 3.x Alluxio#17641 is not cherry picked because the PR has already been in main Alluxio#17781 is not cherry picked because the PR has already been in main Alluxio#17722 is not cherry picked because the alluxio-fuse command has been changed a lot Alluxio#17489 is not cherry picked because audit log on master is no longer in 3.x Alluxio#17865 is not cherry picked because replication on job service is no longer in 3.x Alluxio#17858 is not cherry picked because it is already in main Alluxio#18090 is not cherry picked because generate-tarball has been rewritten in 3.x Alluxio#18091 is not cherry picked because the change is already in main Alluxio#17474 is not cherry picked because reconfiguration feature is not defined in 3.x Alluxio#17735 is not cherry picked because MonoBlockStore is no longer in 3.x Alluxio#18133 is not cherry picked because the issue is about master metadata and no longer relevant in 3.x Alluxio#17910 is not cherry picked because I prefer to do that manually Alluxio#17983 is not cherry picked because the web UI has been reworked Alluxio#17984 is not cherry picked because Mount/Unmount commands have been reworked in 3.x Alluxio#18103 is not cherry picked because worker cache metrics have been reworked in 3.x Alluxio#18185 is not cherry picked because the report command has been reworked in 3.x Alluxio#18222 is not cherry picked because Mount/Unmount operations have been reworked in 3.x Alluxio#18143 is not cherry picked because the change is already in main Alluxio#18303 is not cherry picked because the change is already in main Alluxio#18208 is not cherry picked because cache metrics have been reworked in 3.x Alluxio#17002 is not cherry picked because the owner has been notified separately Alluxio#18334 is not cherry picked because the bash scripts have been reworked in 3.x Alluxio#18326 is not cherry picked because the owner has been notified separately pr-link: Alluxio#18397 change-id: cid-dbf8cbb2d9e721a5a0a1e5028a3c9577438a2ac0
Configuration menu - View commit details
-
Copy full SHA for c2f4cd2 - Browse repository at this point
Copy the full SHA c2f4cd2View commit details -
Fix the NPE in COSUnderFileSystem
### What changes are proposed in this pull request? meta.getLastModified() could be null, and should avoid the NPE. the parameter is nullable, so give it a null in this case. ### Why are the changes needed? meta.getLastModified() could be null, and should avoid the NPE. the parameter is nullable, so give it a null in this case. ### Does this PR introduce any user facing changes? Please list the user-facing changes introduced by your change, including 1. change in user-facing APIs 2. addition or removal of property keys 3. webui pr-link: Alluxio#18403 change-id: cid-9d3c16f8c2b403dff668f52649432a5d7ca8f1a3
Configuration menu - View commit details
-
Copy full SHA for 1c279c6 - Browse repository at this point
Copy the full SHA 1c279c6View commit details -
Configuration menu - View commit details
-
Copy full SHA for b0f803a - Browse repository at this point
Copy the full SHA b0f803aView commit details -
Configuration menu - View commit details
-
Copy full SHA for d07ca76 - Browse repository at this point
Copy the full SHA d07ca76View commit details
Commits on Nov 9, 2023
-
Remove malformed empty page files on access
### What changes are proposed in this pull request? Remove empty pages when they are accessed by the get page methods. Empty pages except for the vert first page of a file are most likely malformed and are a result of a failed put attempt. ### Why are the changes needed? Reading empty pages causes 0 bytes to be read, and in some cases causing an infinite loop in the caller. When used with `getDataFileChannel`, it results in the channel to be created multiple times for a given empty page, because the channels are always 0 sized. ### Does this PR introduce any user facing changes? No. pr-link: Alluxio#18401 change-id: cid-fff8f48af3814e3c71317377aa0b065599c06a57
Configuration menu - View commit details
-
Copy full SHA for 2e1f95d - Browse repository at this point
Copy the full SHA 2e1f95dView commit details -
Replace the FOLDER_SUFFIX from "_$folder$" to "/" in OSS UFS
### What changes are proposed in this pull request? Replace the FOLDER_SUFFIX from "_$folder$" to "/" in OSS UFS ### Why are the changes needed? We don't want to use '_$ Folder $" to represent the directory. We want to use a more general "/" instead to avoid some issues ### Does this PR introduce any user facing changes? No pr-link: Alluxio#18408 change-id: cid-d16023b954bcf0ad9f37c9f629c5d0799443af8f
Configuration menu - View commit details
-
Copy full SHA for 69ca11b - Browse repository at this point
Copy the full SHA 69ca11bView commit details -
Fix stress worker bench remote only policy not working
PR Alluxio#18377 removes setting property `alluxio.user.worker.selection.policy` by mistake. This will cause remote only policy not working. This PR fixes this problem. pr-link: Alluxio#18407 change-id: cid-a53dce950a07cdd661b3237731037a0e9efb052c
Configuration menu - View commit details
-
Copy full SHA for f7225f4 - Browse repository at this point
Copy the full SHA f7225f4View commit details -
Change file exist exception to runtime exception in PagedDoraWorker
### What changes are proposed in this pull request? The `RuntimeException(FileAlreadyExistException)` changed to `AlreadyExistException()` in `PagedDoraWorker` ### Why are the changes needed? `RuntimeException(FileAlreadyExistException)` seems won't be caught in `DoraWorkerClientServiceHandler` when convertin it to `AlluxioRuntimeException`, which willl cause the loss status code ### Does this PR introduce any user facing changes? no pr-link: Alluxio#18337 change-id: cid-9a520c49579847bae6da21302f484ba713eeb4d9
Configuration menu - View commit details
-
Copy full SHA for 1852423 - Browse repository at this point
Copy the full SHA 1852423View commit details
Commits on Nov 10, 2023
-
Introduction Chapter translation pr-link: Alluxio#18176 change-id: cid-c31d1ab3c4365f0bdb0d0b17418c92885a109276
Configuration menu - View commit details
-
Copy full SHA for d9cbbf1 - Browse repository at this point
Copy the full SHA d9cbbf1View commit details -
Improve page loading performance
### What changes are proposed in this pull request? Improve page loading performance by loading the page form UFS directly, instead of creating a reader. ### Why are the changes needed? To improve the data loading performance & reduce the memory consumption ### Does this PR introduce any user facing changes? N/A pr-link: Alluxio#18389 change-id: cid-46e89dcfd22dcbbb4090df7f6c8861408ef8cb67
Configuration menu - View commit details
-
Copy full SHA for 5421aa4 - Browse repository at this point
Copy the full SHA 5421aa4View commit details -
Allow rename() to overwrite, fix an uncaught ex and reenable UT
### What changes are proposed in this pull request? As the title states, this PR does a few things: 1. Allow rename() to overwrite an existing path, if specified in the option 2. The method `rename` in `AbstractFileSystem`, now will process the input path and rerun `rename` method when caught `AlluxioException` or `AlluxioRuntimeException`. Instead of log the exception and return end. 3. Also add some path checks during the rename 4. Reenable `FileSystemRenameIntegrationTest` by adding configs to `LocalAlluxioClusterResource` and reenable UT cases. pr-link: Alluxio#18263 change-id: cid-2870bf87fea8a3b2419e5b10a05423ff2dede6a2
Configuration menu - View commit details
-
Copy full SHA for 328ee8c - Browse repository at this point
Copy the full SHA 328ee8cView commit details
Commits on Nov 11, 2023
-
[SMALLFIX] Remove invalid Property WORKER_FUSE_ENABLED
### What changes are proposed in this pull request? Now WORKER_FUSE_ENABLED is invalid. ### Why are the changes needed? improve code quality. ### Does this PR introduce any user facing changes? no. pr-link: Alluxio#18349 change-id: cid-8064a8ef19179c31cdce01a833847ff530c1f776
Configuration menu - View commit details
-
Copy full SHA for ba02104 - Browse repository at this point
Copy the full SHA ba02104View commit details
Commits on Nov 13, 2023
-
Add call points of the new metrics
### What changes are proposed in this pull request? Add call points of the new metrics ### Why are the changes needed? The new metrics are already defined in the previous pr, and need to be called in this pr. ### Does this PR introduce any user facing changes? no pr-link: Alluxio#18351 change-id: cid-1b35a87f41a1f836005c368881378329f4c77b25
Configuration menu - View commit details
-
Copy full SHA for 08547a5 - Browse repository at this point
Copy the full SHA 08547a5View commit details -
Fixed a single spelling error. ### What changes are proposed in this pull request? Please outline the changes and how this PR fixes the issue. ### Why are the changes needed? Please clarify why the changes are needed. For instance, 1. If you propose a new API, clarify the use case for a new API. 2. If you fix a bug, describe the bug. ### Does this PR introduce any user facing changes? Please list the user-facing changes introduced by your change, including 1. change in user-facing APIs 2. addition or removal of property keys 3. webui pr-link: Alluxio#18310 change-id: cid-77c21de69cf13a6c94be418adf2be95e0f347d42
Configuration menu - View commit details
-
Copy full SHA for f6e4260 - Browse repository at this point
Copy the full SHA f6e4260View commit details
Commits on Nov 14, 2023
-
Clean up fuse unmount workflow
### What changes are proposed in this pull request? clean up unmount of fuse ### Why are the changes needed? alluxio-fuse may cause AlluxioFuse continues to hang, and also the current flow to unmount is scattered and all over the place. ### Does this PR introduce any user facing changes? no pr-link: Alluxio#18241 change-id: cid-0246bd831466396697efe1977547d365ad63ba3d
Configuration menu - View commit details
-
Copy full SHA for feb0116 - Browse repository at this point
Copy the full SHA feb0116View commit details -
Fix the issue when mount an empty oss bucket
### What changes are proposed in this pull request? Please outline the changes and how this PR fixes the issue. ### Why are the changes needed? Please clarify why the changes are needed. For instance, 1. If you propose a new API, clarify the use case for a new API. 2. If you fix a bug, describe the bug. ### Does this PR introduce any user facing changes? Please list the user-facing changes introduced by your change, including 1. change in user-facing APIs 2. addition or removal of property keys 3. webui pr-link: Alluxio#18412 change-id: cid-fa4b35605d8f97d79c159140613aacf27d3c1836
Configuration menu - View commit details
-
Copy full SHA for 3a29f8d - Browse repository at this point
Copy the full SHA 3a29f8dView commit details
Commits on Nov 17, 2023
-
Fix the bug that HdfsUfsStatusIterator returns wrong path
Fix the bug that `HdfsUfsStatusIterator` returns wrong path (the getName() method of `UfsStatus` should return a relative path, but not just the file name). pr-link: Alluxio#18424 change-id: cid-8b15fbd345f26562a06c5d9f52bdbc90a53b2039
Configuration menu - View commit details
-
Copy full SHA for d2d29df - Browse repository at this point
Copy the full SHA d2d29dfView commit details
Commits on Nov 20, 2023
-
Fix the bug that HDFS iterator doesn't work
Fix the bug that HDFS iterator doesn't work. pr-link: Alluxio#18429 change-id: cid-dbb5f1a246bce706f7b3b5d179ad158218dccf83
Configuration menu - View commit details
-
Copy full SHA for f7eed03 - Browse repository at this point
Copy the full SHA f7eed03View commit details
Commits on Nov 29, 2023
-
Fix duplicated class and resources in presto mvn build
### What changes are proposed in this pull request? Shade all the class under javax ### Why are the changes needed? We need this change to fix the maven build error in presto ### Does this PR introduce any user facing changes? no pr-link: Alluxio#18450 change-id: cid-91aa8f5a00867b0f44165eb486fa168cc710504c
Configuration menu - View commit details
-
Copy full SHA for 24e4d3d - Browse repository at this point
Copy the full SHA 24e4d3dView commit details -
Configuration menu - View commit details
-
Copy full SHA for 50fd48f - Browse repository at this point
Copy the full SHA 50fd48fView commit details -
Configuration menu - View commit details
-
Copy full SHA for cacc1cc - Browse repository at this point
Copy the full SHA cacc1ccView commit details
Commits on Dec 1, 2023
-
### What changes are proposed in this pull request? Add a `WorkerClusterView` to allow more efficiently indexing and filtering workers by different criteria. ### Why are the changes needed? The current return type `List<WorkerInfo>` of the methods `getAllWorkers` etc. of `MembershipManager` does not allow efficiently indexing and filtering workers, e.g. getting a worker by it ID, finding all lost workers. This is partially motivated by the review comments in Alluxio#18434 (comment) ### Does this PR introduce any user facing changes? No. pr-link: Alluxio#18441 change-id: cid-e28beb96752cc45cdad973b98f2d689ce1581b0f
Configuration menu - View commit details
-
Copy full SHA for 15b05ba - Browse repository at this point
Copy the full SHA 15b05baView commit details
Commits on Dec 4, 2023
-
Fix the shading issue for javax security
### What changes are proposed in this pull request? Fix the shading issue for javax security ### Why are the changes needed? you might see class not found in javax's security module ### Does this PR introduce any user facing changes? no pr-link: Alluxio#18455 change-id: cid-d04ba461930b7abba2613810e8d7d2ae77d7e3be
Configuration menu - View commit details
-
Copy full SHA for 0db1ff7 - Browse repository at this point
Copy the full SHA 0db1ff7View commit details
Commits on Dec 5, 2023
-
Configuration menu - View commit details
-
Copy full SHA for 7223e80 - Browse repository at this point
Copy the full SHA 7223e80View commit details -
Configuration menu - View commit details
-
Copy full SHA for e6ac3b8 - Browse repository at this point
Copy the full SHA e6ac3b8View commit details
Commits on Dec 6, 2023
-
Make proxy transfer buffer size configurable
### What changes are proposed in this pull request? Make the buffer size in s3 proxy configurable, which is currently 8KB and unchangeable. pr-link: Alluxio#18459 change-id: cid-7ebb81819b6f13222496a341cbef743cf50bda7b
Configuration menu - View commit details
-
Copy full SHA for 4ab2acf - Browse repository at this point
Copy the full SHA 4ab2acfView commit details
Commits on Dec 8, 2023
-
Make WorkerInfo enum (and some small refactors)
### What changes are proposed in this pull request? 1. Move `WorkerState` enum from master package to the wire package. 2. Add a new worker state `UNRECOGNIZED` and use it as the default state. 3. Make worker state in `WorkerInfo` an enum. 4. Add copy constructors to `WorkerInfo` and `WorkerNetAddress`. ### Why are the changes needed? 1. Make sure the state of worker can be enumerated. 2. Allow safely copying mutable `WorkerInfo` and `WorkerNetAddress` objects. ### Does this PR introduce any user facing changes? No. pr-link: Alluxio#18460 change-id: cid-8daf9c1e3ebe8e862a9b0dabb669c80918f5b8b0
Configuration menu - View commit details
-
Copy full SHA for 73d1746 - Browse repository at this point
Copy the full SHA 73d1746View commit details
Commits on Dec 9, 2023
-
Fix to disable worker identity reuse for registration unless in k8s
### What changes are proposed in this pull request? Put back the restriction of reuse worker identity for non-k8s env deployment. ### Why are the changes needed? Worker identity gets reused by wrong deployment behaviors such as copy conf/ over for new worker setup, as opposed to k8s deployment is thru operator / automation, bare metal deployment has no way of prevention, thus putting back the restriction for non-k8s deployment. ### Does this PR introduce any user facing changes? No. pr-link: Alluxio#18454 change-id: cid-295ea352895b16c2a5f0a23fa790c9f42a5e3881
Configuration menu - View commit details
-
Copy full SHA for 22af1b1 - Browse repository at this point
Copy the full SHA 22af1b1View commit details
Commits on Dec 12, 2023
-
Use cluster view to replace list of workers
### What changes are proposed in this pull request? Replace `List<BlockWorkerInfo>` with `WorkerClusterView` in APIs. Important APIs that are changed: 1. `FileSystemContext.getCachedWorkers` now returns `WorkerClusterView` 2. `WorkerLocationPolicy.getPreferredWorkers` (as well as all its implementations) now accepts a `WorkerClusterView` as the first argument (but still returns `List<BlockWorkerInfo>` as the returned list must be ordered) APIs that are using `List<BlockWorkerInfo>` (or `List<WorkerInfo>`) but *not* migrated to `WorkerClusterView`: 1. `alluxio.master.scheduler.WorkerProvider.getWorkerInfos` returns `List<WorkerInfo>`. 2. Job service related APIs, e.g. `alluxio.job.plan.PlanDefinition.selectExecutors` Notable behavior change: Now `EtcdMembershipManager` assigns the correct state (`LIVE` or `LOST`) for all workers in its `WorkerInfo` struct. Before this change, this information is not available and the state defaults to `UNRECOGNIZED`. ### Why are the changes needed? Allow more efficient indexing and filtering workers by worker ID. ### Does this PR introduce any user facing changes? No. pr-link: Alluxio#18451 change-id: cid-5052d2faa506f4de6e4b0df7062c5def3e09df1c
Configuration menu - View commit details
-
Copy full SHA for 9eae1e9 - Browse repository at this point
Copy the full SHA 9eae1e9View commit details
Commits on Dec 14, 2023
-
Fix outdated worker address info returned by consistent hash policy
### What changes are proposed in this pull request? Fix outdated worker address info returned by consistent hash policy Summary of changes: 1. `ConsistentHashProvider` only concerns about `WorkerIdentity` when building the hash ring. Therefore, the APIs have been limited to accept and return `WorkerIdentity`s. 2. `ConsistentHashProvider.refresh` now accepts a set of worker identities instead of a list, as the order does not matter. 3. Added a test to cover the bug fix. ### Why are the changes needed? Fix a bug where the consistent hash provider caches the `BlockWorkerInfo` of all workers, and when a worker changes its network addresses but its ID stays the same, the hash provider won't update the worker's info. A client will continue to use the outdated network address. The fix is to make the hash provider to only consider `WorkerIdentity`s, and let the client to figure out the worker's address with the ID provided by the consistent hash provider. ### Does this PR introduce any user facing changes? No. pr-link: Alluxio#18434 change-id: cid-93f1601d846385f314c79a556c8705d3983a1199
Configuration menu - View commit details
-
Copy full SHA for 9902e8a - Browse repository at this point
Copy the full SHA 9902e8aView commit details -
Improve rest load api response JSON
This PR improves RESTful load api, use JSON format as the response content. ## Examples: ### Submit Job Example: ``` // 20231211174310 // http://localhost:28080/v1/load?path=hdfs://node01:8020/testRoot/testDirectory2&opType=submit&verbose=true { "success": true, "jobId": "0dbc0f47-580b-420f-b50e-d08a170746c8", "path": "hdfs://node01:8020/testRoot/testDirectory2", "message": "Load 'hdfs://node01:8020/testRoot/testDirectory2' is successfully submitted. JobId: 0dbc0f47-580b-420f-b50e-d08a170746c8\n" } ``` ### Get Job Progress Example: ``` // 20231211180110 // http://localhost:28080/v1/load?path=hdfs://node01:8020/testRoot/testDirectory2&opType=progress&verbose=true { "jobState": "RUNNING", "path": "hdfs://node01:8020/testRoot/testDirectory2", "message": "Progress for loading path 'hdfs://node01:8020/testRoot/testDirectory2':\n\tSettings:\tbandwidth: unlimited\tverify: false\tmetadata-only: false\n\tTime Elapsed: 00:00:03\n\tJob State: RUNNING\n\tStage: RETRYING\n\tInodes Scanned: 4\n\tInodes Processed: 4\n\tBytes Loaded: 0B out of 0B\n\tThroughput: 0B/s\n\tFile Failure rate: 0.00%\n\tSubtask Failure rate: 0.00%\n\tFiles Failed: 0\n\tRecent failed subtasks: \n\tRecent retrying subtasks: \n\tSubtask Retry rate: 0.00%\n\tSubtasks on Retry Dead Letter Queue: 0\n", "respProperties": { "Files Failed": "0", "Recent failed subtasks": "", "Subtask Retry rate": "0.00%", "Throughput": "0B/s", "File Failure rate": "0.00%", "Subtasks on Retry Dead Letter Queue": "0", "Time Elapsed": "00", "Bytes Loaded": "0B out of 0B", "Stage": "RETRYING", "Inodes Scanned": "4", "Inodes Processed": "4", "Recent retrying subtasks": "", "Subtask Failure rate": "0.00%", "Settings": "bandwidth", "Job State": "RUNNING", "Progress for loading path 'hdfs": "//node01" } } ``` ``` // 20231211174358 // http://localhost:28080/v1/load?path=hdfs://node01:8020/testRoot/testDirectory2&opType=progress&verbose=true { "jobState": "SUCCEEDED", "path": "hdfs://node01:8020/testRoot/testDirectory2", "message": "Progress for loading path 'hdfs://node01:8020/testRoot/testDirectory2':\n\tSettings:\tbandwidth: unlimited\tverify: false\tmetadata-only: false\n\tTime Elapsed: 00:00:16\n\tJob State: SUCCEEDED\n\tInodes Scanned: 4\n\tInodes Processed: 4\n\tBytes Loaded: 0B out of 0B\n\tThroughput: 0B/s\n\tFile Failure rate: 0.00%\n\tSubtask Failure rate: 0.00%\n\tFiles Failed: 0\n\tRecent failed subtasks: \n\tRecent retrying subtasks: \n\tSubtask Retry rate: 0.00%\n\tSubtasks on Retry Dead Letter Queue: 0\n", "respProperties": { "Files Failed": "0", "Recent failed subtasks": "", "Subtask Retry rate": "0.00%", "Throughput": "0B/s", "File Failure rate": "0.00%", "Subtasks on Retry Dead Letter Queue": "0", "Time Elapsed": "00", "Bytes Loaded": "0B out of 0B", "Inodes Scanned": "4", "Inodes Processed": "4", "Recent retrying subtasks": "", "Subtask Failure rate": "0.00%", "Settings": "bandwidth", "Job State": "SUCCEEDED", "Progress for loading path 'hdfs": "//node01" } } ``` ### Stop Job Example: ``` // 20231211180219 // http://localhost:28080/v1/load?path=hdfs://node01:8020/testRoot/testDirectory2&opType=stop&verbose=true { "success": true, "path": "hdfs://node01:8020/testRoot/testDirectory2", "message": "Load 'hdfs://node01:8020/testRoot/testDirectory2' is successfully stopped.\n" } ``` ``` // 20231211180153 // http://localhost:28080/v1/load?path=hdfs://node01:8020/testRoot/testDirectory2&opType=stop&verbose=true { "success": false, "path": "hdfs://node01:8020/testRoot/testDirectory2", "message": "Cannot find load job for path hdfs://node01:8020/testRoot/testDirectory2, it might have already been stopped or finished\n" } ``` pr-link: Alluxio#18464 change-id: cid-1fff9a23457064ab71534909449c60a6b0123f22
Configuration menu - View commit details
-
Copy full SHA for 4e3a722 - Browse repository at this point
Copy the full SHA 4e3a722View commit details -
Fix alluxio-fuse script for finding fuse pid
### What changes are proposed in this pull request? Remove additional white space in alluxio-fuse script ### Why are the changes needed? alluxio-fuse unmount <mnt_point> is unable to find the pid of AlluxioFuse process because the grep content isn't correct. ### Does this PR introduce any user facing changes? No pr-link: Alluxio#18465 change-id: cid-3e70b0c8edbaa0ba50d744fd6155b0d494a243f9
Configuration menu - View commit details
-
Copy full SHA for 4e39eda - Browse repository at this point
Copy the full SHA 4e39edaView commit details
Commits on Dec 21, 2023
-
Support get page with RESTful API by specifying offset and length
Support get page with RESTful API by specifying offset and length. PAGE_URL_FORMAT = ( "[http://{worker_host}:{http_port}/v1/file/{path_id}/page/{page_index}?offset=100&length=1024](http://{worker_host}:%7Bhttp_port%7D/v1/file/%7Bpath_id%7D/page/%7Bpage_index%7D?offset=100&length=1024)" ) pr-link: Alluxio#18474 change-id: cid-ba5b0c5050843ccc5642950beadc8a0b049948be
Configuration menu - View commit details
-
Copy full SHA for b5a2895 - Browse repository at this point
Copy the full SHA b5a2895View commit details
Commits on Jan 4, 2024
-
Support write page with RESTful API
Support write page with RESTful API ### Usage ``` HTTP Method: POST Request URL: http://localhost:28080/v1/file/<fileId>/page/<pageIndex> HTTP Body: <page bytes> ``` pr-link: Alluxio#18481 change-id: cid-cab175a007bfcaf294e89adbe47531419036a245
Configuration menu - View commit details
-
Copy full SHA for 726f812 - Browse repository at this point
Copy the full SHA 726f812View commit details
Commits on Jan 8, 2024
-
``` $ df -h /mnt/fuse/ Filesystem Size Used Avail Use% Mounted on alluxio-fuse 910T 0 910T 0% /mnt/fuse ``` ### What changes are proposed in this pull request? Add fake numbers for statfs ### Why are the changes needed? Some application checks the available space in a file system before continuing to do file operations. ### Does this PR introduce any user facing changes? A fake number (1 Petabytes) is provided to statfs. This number does not reflect real available storage space. pr-link: Alluxio#18482 change-id: cid-9f60d185393b616be02bf8f473b2026f2047f28c
Configuration menu - View commit details
-
Copy full SHA for 9ef7552 - Browse repository at this point
Copy the full SHA 9ef7552View commit details
Commits on Jan 10, 2024
-
Support Various Kinds of Consistent Hash
### What changes are proposed in this pull request? Add Ketama Hashing, Jump Consistent Hashing, Maglev Hashing, and Multi Probe Hashing. ### Why are the changes needed? Now alluxio's user worker selection policy is Consistent Hash Policy. It bings too much time cost, and it is not enough uniform, and not strictly consistent. Ketama: https://github.com/RJ/ketama Jump Consistent Hashing: https://arxiv.org/pdf/1406.2294.pdf Maglev Hashing: https://static.googleusercontent.com/media/research.google.com/zh-CN//pubs/archive/44824.pdf Multi Probe Hasing: https://arxiv.org/pdf/1505.00062.pdf We strongly recommend using Maglev Hashing for User Worker Selection Policy. Under most situation, it has the minimum time cost, and it is the most uniform and balanced hashing policy. ### Does this PR introduce any user facing changes? `alluxio.user.worker.selection.policy` has the following values: `CONSISTENT`, `JUMP`, `KETAMA`, `MAGLEV`, `MULTI_PROBE`, `LOCAL`, `REMOTE_ONLY`, corresponding to consistent hash policy, maglev hash policy, ketama hash policy, maglev hash policy, multi-probe respectively hash policy, local worker policy, remote only policy. The current default value is `CONSISTENT`. We recommend using Maglev Hash, which has the best hash consistency and is the least time-consuming. That is to say, set the value of `alluxio.user.worker.selection.policy` to `MAGLEV`. We will also consider setting this as the default value in the future. **Ketama Hasing** `alluxio.user.ketama.hash.replicas`: This is the value of replicas in the ketama hashing algorithm. When workers changes, it will guarantee the hash table is changed only in a minimal. The value of replicas should be X times the physical nodes in the cluster, where X is a balance between efficiency and cost. **Jump Consistent Hashing** None. **Maglev Hashing** `alluxio.user.maglev.hash.lookup.size`: This is the size of the lookup table in the maglev hashing algorithm. It must be a prime number. In the maglev hashing, it will generate a lookup table for workers. The bigger the size of the lookup table, the smaller the variance of this hashing algorithm will be. But bigger look up table will consume more time and memory. **Multi Probe Hashing** `alluxio.user.multi.probe.hash.probe.num`: This is the number of probes in the multi-probe hashing algorithm. In the multi-probe hashing algorithm, the bigger the number of probes, the smaller the variance of this hashing algorithm will be. But more probes will consume more time and memory. pr-link: Alluxio#17817 change-id: cid-bad21c6e5ad83eb3da15a8960ba372b14c67b081
Zihao Zhao authoredJan 10, 2024 Configuration menu - View commit details
-
Copy full SHA for b9de24c - Browse repository at this point
Copy the full SHA b9de24cView commit details
Commits on Jan 11, 2024
-
[DOCFIX] Update kerberos configuration 'alluxio.hadoop.kerberos.keyta…
…b.login.autorenewal' in HDFS docs ### What changes are proposed in this pull request? Update the correct kerberos configuration 'alluxio.hadoop.kerberos.keytab.login.autorenewal' in HDFS docs to avoid user confusion. ### Why are the changes needed? Fix Alluxio#18486 pr-link: Alluxio#18487 change-id: cid-b8f08e2f67e5f10aa1426de7629b8e268e339433
Configuration menu - View commit details
-
Copy full SHA for 7d8f317 - Browse repository at this point
Copy the full SHA 7d8f317View commit details -
Create metadata directory in initiateMultipartUpload method
### What changes are proposed in this pull request? Create metadata directory in initiateMultipartUpload method. ### Why are the changes needed? Each request calls the initialization method of the handler and sends an `exists` request to the master, which is unnecessary. pr-link: Alluxio#18462 change-id: cid-3efd076d7eb33cc063609fa1e1003e3aff480be6
Configuration menu - View commit details
-
Copy full SHA for e4da0e4 - Browse repository at this point
Copy the full SHA e4da0e4View commit details
Commits on Jan 13, 2024
-
`format` commands don't exist in cli anymore, so as job services. Delete some entrypoint code. Solves Alluxio#18466 pr-link: Alluxio#18490 change-id: cid-0ee0b45a012a29df4a793107d7f8cce4ca98fc99
Configuration menu - View commit details
-
Copy full SHA for 4f07b99 - Browse repository at this point
Copy the full SHA 4f07b99View commit details
Commits on Jan 19, 2024
-
Fix the issue of S3 range read
### What changes are proposed in this pull request? fix the issue of S3 range read. ### Why are the changes needed? there is a bug when setting S3 range offset. ### Does this PR introduce any user facing changes? user can try the following cmd to validate S3 range read. `aws --endpoint http://localhost:39999/api/v1/s3 s3api get-object --range bytes=10-20 --bucket [bucket-name] --key=[key-name] [output-file]` pr-link: Alluxio#18484 change-id: cid-b5fd9832a9900fba1105bb494a96f315b20f507d
Configuration menu - View commit details
-
Copy full SHA for 6f2a527 - Browse repository at this point
Copy the full SHA 6f2a527View commit details -
Fix negative size if specified filesize over Integer.MAX_VALUE for st…
…ressWorkerBench ### What changes are proposed in this pull request? Change to use long type store file size. ### Why are the changes needed? Without this PR, we cannot specified a filesize lagger than Integer.MAX_VALUE. ### Does this PR introduce any user facing changes? No pr-link: Alluxio#18492 change-id: cid-2b816d1f2cbc9ebcf888b06802eb682fb76d55c2
Configuration menu - View commit details
-
Copy full SHA for 97f325a - Browse repository at this point
Copy the full SHA 97f325aView commit details
Commits on Jan 25, 2024
-
Add ServiceRegistry membership manager which only include active workers
### What changes are proposed in this pull request? Add etcd membership manager only include active workers For python client it would need to only look at active worker list ### Why are the changes needed? We only need active workers list in big tech env. If the worker is down, we don't care and just reshard small portion of data to other workers. ### Does this PR introduce any user facing changes? na pr-link: Alluxio#18495 change-id: cid-70ec6f27539f5f47b99be1ce3ff85cb9c117c3bf
Configuration menu - View commit details
-
Copy full SHA for 055b375 - Browse repository at this point
Copy the full SHA 055b375View commit details
Commits on Jan 31, 2024
-
Add http server port in the worker net address
### What changes are proposed in this pull request? This PR enables registering the worker's HTTP server's port in the etcd. This helps to find worker's restful APIs from the Python client. ### Why are the changes needed? Alluxio Python client (e.g. in ML use cases) needs to connect to the worker's REST APIs. But as the http server port isn't included in the worker's information in the etcd, the client fails to find the API endpoint. ### Does this PR introduce any user facing changes? No. pr-link: Alluxio#18499 change-id: cid-1cf7e0bdc7cc0c9702949bc313de5583d9cc2fb8
Configuration menu - View commit details
-
Copy full SHA for 262b2b3 - Browse repository at this point
Copy the full SHA 262b2b3View commit details
Commits on Feb 1, 2024
-
Configuration menu - View commit details
-
Copy full SHA for 05e089b - Browse repository at this point
Copy the full SHA 05e089bView commit details -
Configuration menu - View commit details
-
Copy full SHA for e9a32c9 - Browse repository at this point
Copy the full SHA e9a32c9View commit details -
Fix DoraLoadCommandIntegrationTest
### What changes are proposed in this pull request? Fix DoraLoadCommandIntegrationTest. ### Why are the changes needed? Without this PR, DoraLoadCommandIntegrationTest may fail with the following exception ``` [ERROR] alluxio.client.cli.fs.command.DoraLoadCommandIntegrationTest.testCommand Time elapsed: 7.995 s <<< FAILURE! java.lang.AssertionError at org.junit.Assert.fail(Assert.java:87) at org.junit.Assert.assertTrue(Assert.java:42) at org.junit.Assert.assertTrue(Assert.java:53) at alluxio.client.cli.fs.command.DoraLoadCommandIntegrationTest.testCommand(DoraLoadCommandIntegrationTest.java:107) ``` The cause is that - The second job is using the same path as the first job - While submitting the second job, the second job is still in "cleaning" state. As the following log shows ``` 2024-01-31 09:52:33,057 [master-rpc-executor-TPE-thread-494] WARN scheduler.Scheduler (Scheduler.java:submitJob) - There's concurrent submit while job is still in cleaning state ``` - "progress" returns the progress of the first job instead of the second one The PR changes to use a different path for the second job, which avoids this issue. ### Does this PR introduce any user facing changes? NO pr-link: Alluxio#18504 change-id: cid-331ba5508e86e8161006073d452ab1ba6230473a
Configuration menu - View commit details
-
Copy full SHA for 07ec4b9 - Browse repository at this point
Copy the full SHA 07ec4b9View commit details
Commits on Feb 2, 2024
-
Support nullable offset and length in get page RESTful API
Fix the bug that the get page RESTful API doesn't support nullable offset and length. pr-link: Alluxio#18506 change-id: cid-85eab5152e501b97bc9b4678e92b0d8e665a95ce
Configuration menu - View commit details
-
Copy full SHA for 3d13072 - Browse repository at this point
Copy the full SHA 3d13072View commit details
Commits on Feb 13, 2024
-
Configuration menu - View commit details
-
Copy full SHA for 2f9bb6a - Browse repository at this point
Copy the full SHA 2f9bb6aView commit details -
Configuration menu - View commit details
-
Copy full SHA for 59bfee0 - Browse repository at this point
Copy the full SHA 59bfee0View commit details
Commits on Feb 15, 2024
-
Change default value of virtual node count per worker
### What changes are proposed in this pull request? 2000 is too much ### Why are the changes needed? 2000 is too much ### Does this PR introduce any user facing changes? na pr-link: Alluxio#18516 change-id: cid-98762f4a176f30b7a83399183aef6f11d5113132
Configuration menu - View commit details
-
Copy full SHA for c34921d - Browse repository at this point
Copy the full SHA c34921dView commit details
Commits on Mar 6, 2024
-
Support Hash Algorithms Benchmark
zhaozihao committedMar 6, 2024 Configuration menu - View commit details
-
Copy full SHA for f89e5f5 - Browse repository at this point
Copy the full SHA f89e5f5View commit details -
Support Hash Algorithms Benchmark
zhaozihao committedMar 6, 2024 Configuration menu - View commit details
-
Copy full SHA for dff2efe - Browse repository at this point
Copy the full SHA dff2efeView commit details
Commits on Mar 7, 2024
-
Support Hash Algorithms Benchmark
zhaozihao committedMar 7, 2024 Configuration menu - View commit details
-
Copy full SHA for bd3b6f1 - Browse repository at this point
Copy the full SHA bd3b6f1View commit details
Commits on Mar 8, 2024
-
Support Hash Algorithms Benchmark
zhaozihao committedMar 8, 2024 Configuration menu - View commit details
-
Copy full SHA for 717013a - Browse repository at this point
Copy the full SHA 717013aView commit details