Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support Hash Algorithms Benchmark #18539

Open
wants to merge 1,134 commits into
base: main
Choose a base branch
from
This pull request is big! We’re only showing the most recent 250 commits.

Commits on Sep 5, 2023

  1. [DOCFIX] Update journal command doc

    ### What changes are proposed in this pull request?
    
    Please outline the changes and how this PR fixes the issue.
    
    ### Why are the changes needed?
    
    Please clarify why the changes are needed. For instance,
      1. If you propose a new API, clarify the use case for a new API.
      2. If you fix a bug, describe the bug.
    
    ### Does this PR introduce any user facing changes?
    
    Please list the user-facing changes introduced by your change, including
      1. change in user-facing APIs
      2. addition or removal of property keys
      3. webui
    
    			pr-link: Alluxio#18113
    			change-id: cid-586e7941f2f4b737f0b0ebe7c15cb13d182e6d0c
    jja725 authored Sep 5, 2023
    Configuration menu
    Copy the full SHA
    8bc9a9e View commit details
    Browse the repository at this point in the history

Commits on Sep 6, 2023

  1. Return FileNotFound when a file is being create/write

    ### What changes are proposed in this pull request?
    
    Return FileNotFound when a file is being create/write. No need to refresh its metadata.
    Metadata will be refreshed when file is closed.
    
    ### Why are the changes needed?
    
    If a file is being written to, and another getStatus() comes for such file, 
    this file probably has not been created in UFS. At that time,
    the getStatus() thinks that this file does not exists. So the worker tries to remove its metadata and its data from cache.
    ERROR logs were generated in such case.
    
    This patch is to handle this correctly and remove such ERROR logs:
    ```
    2023-09-05 17:16:32,802 ERROR LocalCacheManager - Failed to delete page PageId{FileId=f4b0d1f405d7419069129a1262eae9329e9c19ed2584fc3fe27656a698344047, PageIndex=0} (isTemporary: false) from pageStore.
    alluxio.exception.PageNotFoundException: /Volumes/ramdisk/LOCAL/1048576/824/f4b0d1f405d7419069129a1262eae9329e9c19ed2584fc3fe27656a698344047/0
    	at alluxio.client.file.cache.store.LocalPageStore.delete(LocalPageStore.java:135)
    	at alluxio.client.file.cache.LocalCacheManager.deletePage(LocalCacheManager.java:889)
    	at alluxio.client.file.cache.LocalCacheManager.delete(LocalCacheManager.java:694)
    	at alluxio.client.file.cache.LocalCacheManager.delete(LocalCacheManager.java:706)
    	at alluxio.client.file.cache.LocalCacheManager.lambda$deleteFile$5(LocalCacheManager.java:839)
    	at java.lang.Iterable.forEach(Iterable.java:75)
    	at alluxio.client.file.cache.LocalCacheManager.deleteFile(LocalCacheManager.java:839)
    	at alluxio.client.file.cache.NoExceptionCacheManager.deleteFile(NoExceptionCacheManager.java:209)
    	at alluxio.worker.dora.DoraMetaManager.invalidateCachedFile(DoraMetaManager.java:337)
    	at alluxio.worker.dora.DoraMetaManager.removeFromMetaStore(DoraMetaManager.java:178)
    	at alluxio.worker.dora.DoraMetaManager.loadFromUfs(DoraMetaManager.java:131)
    	at alluxio.worker.dora.PagedDoraWorker.getGrpcFileInfo(PagedDoraWorker.java:392)
    	at alluxio.worker.grpc.DoraWorkerClientServiceHandler.getStatus(DoraWorkerClientServiceHandler.java:201)
    	at alluxio.grpc.BlockWorkerGrpc$MethodHandlers.invoke(BlockWorkerGrpc.java:1589)
    	at io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:182)
    	at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:355)
    	at io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:867)
    	at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
    	at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
    	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    	at java.lang.Thread.run(Thread.java:750)
    ```
    
    ### Does this PR introduce any user facing changes?
    No
    			pr-link: Alluxio#18108
    			change-id: cid-47d15fcde921a539a6e3c64e7ac25fc400f26034
    huanghua78 authored Sep 6, 2023
    Configuration menu
    Copy the full SHA
    0e3201e View commit details
    Browse the repository at this point in the history
  2. Add isReadOnly() for FuseFileStream

    ### What changes are proposed in this pull request?
    
    Add isReadOnly() for FuseFileStream and implement this interface in various Streams.
    
    ### Why are the changes needed?
    
    This interface is used to determine if the Stream is only. This is needed to determine if a real data flush is needed.
    
    ### Does this PR introduce any user facing changes?
    
    N/A
    			pr-link: Alluxio#18114
    			change-id: cid-6a60e9e917bb0f8a439d76b51beaaba2b723dde4
    huanghua78 authored Sep 6, 2023
    Configuration menu
    Copy the full SHA
    55a0e48 View commit details
    Browse the repository at this point in the history
  3. Run launch-process-bash script

    Script `launch-process` was renamed to `launch-process-bash`. Update the name of the script in `entrypoint.sh`
    			pr-link: Alluxio#18115
    			change-id: cid-7053532e9fac93ef3f987e1525ae7260ac3a8dc0
    ssz1997 authored Sep 6, 2023
    Configuration menu
    Copy the full SHA
    f2355ca View commit details
    Browse the repository at this point in the history

Commits on Sep 7, 2023

  1. Add a config to turn on/off the sdk cache fallback

    ### What changes are proposed in this pull request?
    
    Add a config to turn on/off the sdk cache fallback
    
    ### Why are the changes needed?
    
    the read fallback should be turn off by default to avoid retry storm to UFS
    
    ### Does this PR introduce any user facing changes?
    
    Addition  of property keys
    
    
    			pr-link: Alluxio#18099
    			change-id: cid-2c893a617ac5e90086db98610434db6d763c1fe9
    beinan authored Sep 7, 2023
    Configuration menu
    Copy the full SHA
    bab3538 View commit details
    Browse the repository at this point in the history
  2. Add dora worker metrics at http requests

    One more place to count the cache hit requests
    			pr-link: Alluxio#17878
    			change-id: cid-0cb55b7ca627832ce2b53e80ea58cfd6a2730bfa
    ssz1997 authored Sep 7, 2023
    Configuration menu
    Copy the full SHA
    bdd3b61 View commit details
    Browse the repository at this point in the history
  3. Fix the array out of bound exception in presto

    ### What changes are proposed in this pull request?
    Fix the array out of bound exception in presto
    
    ### Why are the changes needed?
    
    Because we saw earlier that the LocalCacheManager swallows all exceptions, if there is an error accessing one of the cached files, the exception will be ignored. However, the offset may have changed during the failed read attempt in LocalCacheManager. When LocalCacheManager returns -1 on error and tries to reread from the lower layer storage, the offset could be out of bounds.
    
    Regarding why this issue only appears in versions 2.9.3 and later, it's because the offset became a member variable of our target buffer starting from 2.9.3. In earlier versions it was always a local variable, so there was no compounding of offset errors.
    
    ### Does this PR introduce any user facing changes?
    
    no
    
    			pr-link: Alluxio#18098
    			change-id: cid-c7b949d9146847b14ba672df56e21bc4bc5ad705
    beinan authored Sep 7, 2023
    Configuration menu
    Copy the full SHA
    b8a8826 View commit details
    Browse the repository at this point in the history
  4. Add dora worker metadata metrics

    Count the hit and miss of ListStatus and GetFileInfo in worker
    			pr-link: Alluxio#17848
    			change-id: cid-97c8efb81088d7e9b636ef927fcbc62a166fe085
    ssz1997 authored Sep 7, 2023
    Configuration menu
    Copy the full SHA
    3d2eb7e View commit details
    Browse the repository at this point in the history

Commits on Sep 8, 2023

  1. Change metric type of LocalCacheState from counter to gauge

    ### What changes are proposed in this pull request?
    
    Change metric type of LocalCacheState from counter to gauge.
    
    ### Why are the changes needed?
    
    The type of LocalCacheState metric is not reasonable since the value of this metric is a enum type not an increasable value
    
    ### Does this PR introduce any user facing changes?
    
    No
    
    			pr-link: Alluxio#18070
    			change-id: cid-61037d3580e2f25f782e1c876ced0ecad5b312a4
    maobaolong authored Sep 8, 2023
    Configuration menu
    Copy the full SHA
    6a0e1c5 View commit details
    Browse the repository at this point in the history
  2. Change name of NoopMembershipManager

    ### What changes are proposed in this pull request?
    
    Change name of NoopMembershipManager to MasterMembershipManager.
    
    ### Why are the changes needed?
    
    N/A
    
    ### Does this PR introduce any user facing changes?
    
    N/A
    
    			pr-link: Alluxio#18097
    			change-id: cid-9f39e3b1a9b8cdad51e49bbb5d12e86d21b07e23
    lucyge2022 authored Sep 8, 2023
    Configuration menu
    Copy the full SHA
    1422109 View commit details
    Browse the repository at this point in the history
  3. Revert "Change metric type of LocalCacheState from counter to gauge"

    Reverts Alluxio#18070
    revert per request on backward compatibility
    			pr-link: Alluxio#18119
    			change-id: cid-aeecbff3bc9d6bff41eec25405e87a2f2b79079f
    jiacheliu3 authored Sep 8, 2023
    Configuration menu
    Copy the full SHA
    72de4e9 View commit details
    Browse the repository at this point in the history
  4. Refactor Netty reader handler

    ### What changes are proposed in this pull request?
    
    Refactor the Netty read handler of worker.
    
    ### Why are the changes needed?
    
    The previous implementation creates a state machine per read request, instead of per channel. This implies that if two read requests are sent over the same channel, the worker would possibly use one channel to send data of different files or regions. This can lead to data corruption.
    
    This PR proposes to use a state machine per channel, and handles channel events throughout the whole lifecycle of the channel. Things like a faulty client sending a second request over the same channel before the first request is completed, is handled gracefully with a client error.
    
    The state transitions look like the following:
    ![graph(3)](https://github.com/Alluxio/alluxio/assets/6999708/8088f14c-6224-4af4-929a-d6d3e0b8b2ef)
    
    
    ### Does this PR introduce any user facing changes?
    
    No.
    
    ### Tests
    
    Tests have been done with basic Alluxio CLI tools, as well as automated PrestoDB and TPC-DS tests.
    
    
    
    			pr-link: Alluxio#17479
    			change-id: cid-bb8b2c70f0bf0bd84e73d9e858bf6e80706427aa
    dbw9580 authored Sep 8, 2023
    Configuration menu
    Copy the full SHA
    9f683b3 View commit details
    Browse the repository at this point in the history
  5. Refactor Netty read handler

    Refactor Netty read handler to allow subclassing.
    
    			pr-link: Alluxio#18120
    			change-id: cid-d0d4125459644d8e92c04dd9decdb622f14cbdb6
    dbw9580 authored Sep 8, 2023
    Configuration menu
    Copy the full SHA
    dc79cd1 View commit details
    Browse the repository at this point in the history
  6. Implement per-thread cache context

    ### What changes are proposed in this pull request?
    Implement per-thread cache context
    
    ### Why are the changes needed?
    Enable the fine-grained cache admission 
    
    ### Does this PR introduce any user facing changes?
    no
    
    			pr-link: Alluxio#18029
    			change-id: cid-839bc71b2df158a4aeaedf22c5c7fb40dfd769e8
    beinan authored Sep 8, 2023
    Configuration menu
    Copy the full SHA
    65f27f5 View commit details
    Browse the repository at this point in the history
  7. [DOCFIX] Remove outdated limitation in QuickStart doc

    Fuse is supported on K8s. Remove the outdated limitation.
    			pr-link: Alluxio#18122
    			change-id: cid-99cacf64405867a720a5304ab5604b778c0b2127
    ssz1997 authored Sep 8, 2023
    Configuration menu
    Copy the full SHA
    7e9691b View commit details
    Browse the repository at this point in the history

Commits on Sep 9, 2023

  1. Change Benchmark's runClass to exec class

    ### What changes are proposed in this pull request?
    
    Change `runClass` in Benchmark to `exec class`
    
    ### Why are the changes needed?
    
    Since new alluxio cli change from `runClass` to `exec class`, related benchmark code should be change as well to keep it work
    
    ### Does this PR introduce any user facing changes?
    
    no
    
    			pr-link: Alluxio#18079
    			change-id: cid-2e8b23a7fc272ed0266c81f5e816442fdb9cd25b
    voddle authored Sep 9, 2023
    Configuration menu
    Copy the full SHA
    9bb6518 View commit details
    Browse the repository at this point in the history
  2. Fix cli binary name for arm

    `uname -m` on arm == `arm64`
    			pr-link: Alluxio#18123
    			change-id: cid-c10f5760052ff7cc3757843df86710200d70b090
    Xenorith authored Sep 9, 2023
    Configuration menu
    Copy the full SHA
    2360819 View commit details
    Browse the repository at this point in the history

Commits on Sep 10, 2023

  1. Skip tests for release

    ZacBlanco authored and beinan committed Sep 10, 2023
    Configuration menu
    Copy the full SHA
    db51058 View commit details
    Browse the repository at this point in the history
  2. Skip sortpom check for release

    beinan committed Sep 10, 2023
    Configuration menu
    Copy the full SHA
    5fb603d View commit details
    Browse the repository at this point in the history

Commits on Sep 11, 2023

  1. Configuration menu
    Copy the full SHA
    3020c34 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    ea6ac3a View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    87263d2 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    568acf6 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    59474ed View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    3f16569 View commit details
    Browse the repository at this point in the history
  7. Enable tests after release

    Xenorith committed Sep 11, 2023
    Configuration menu
    Copy the full SHA
    466fc05 View commit details
    Browse the repository at this point in the history
  8. Remove hostname from metrics key

    ### What changes are proposed in this pull request?
    
    Remove hostname from metrics key
    
    ### Why are the changes needed?
    
    For easy aggregation on prometheus and grafana side
    
    ### Does this PR introduce any user facing changes?
    
    Add a flag to disable this for compatibility
    
    			pr-link: Alluxio#18121
    			change-id: cid-ba6c2f9fae625747192044168fce7dc026c66b9c
    beinan authored Sep 11, 2023
    Configuration menu
    Copy the full SHA
    6165c0d View commit details
    Browse the repository at this point in the history
  9. [DOCFIX] Update bin/alluxio usage format

    besides the User-CLI.md doc, update other doc files that refer to `bin/alluxio` commands
    - remove docs on path config
    - remove starting/stopping job master/worker from contributor docs
    			pr-link: Alluxio#18128
    			change-id: cid-fc71dd493b16ef3aeeb0b1b190941c43b9af9cab
    Xenorith authored Sep 11, 2023
    Configuration menu
    Copy the full SHA
    5ad6e21 View commit details
    Browse the repository at this point in the history

Commits on Sep 12, 2023

  1. Create DoraWorkerServiceHandlerTest.java

    What changes are proposed in this pull request?
    I have created a test and create a liststatus test for its function.
    
    Why are the changes needed?
    Please clarify why the changes are needed. For instance,
    
    add a unit test for DoraWorkerClientServiceHandler.
    Does this PR introduce any user facing changes?
    No.
    			pr-link: Alluxio#18059
    			change-id: cid-b82706a4419700f017584f3e5579d2ef3410aeb3
    thu-david authored Sep 12, 2023
    Configuration menu
    Copy the full SHA
    3c41091 View commit details
    Browse the repository at this point in the history
  2. Make fuse max reader concurrency configurable

    Make fuse max reader concurrency configurable. The default value was 64 and it was unchangeable.
    			pr-link: Alluxio#18129
    			change-id: cid-9c55821622329bd1e608da2e7445e8ab591df38a
    JiamingMai authored Sep 12, 2023
    Configuration menu
    Copy the full SHA
    71f5708 View commit details
    Browse the repository at this point in the history
  3. Fix typo

    Fix typo from alluxio.max.fuse.reader.concurrency to alluxio.fuse.max.reader.concurrency
    			pr-link: Alluxio#18134
    			change-id: cid-434086cf6ba9e9f8d173e3417fc8518963dfa102
    JiamingMai authored Sep 12, 2023
    Configuration menu
    Copy the full SHA
    fc7971c View commit details
    Browse the repository at this point in the history
  4. [DOCFIX] Update bin/alluxio usage in ufs/compute

    update usages of bin/alluxio, bin/alluxio-start.sh and bin/alluxio-stop.sh to their new counterparts
    
    simplify section of CephFS.md and remove sections related to mounting. the ufs must be configured as the root mount via alluxio-site.properties.
    			pr-link: Alluxio#18136
    			change-id: cid-fa7d0eec00c8fb136680ef6d5a2c7ee78571d123
    Xenorith authored Sep 12, 2023
    Configuration menu
    Copy the full SHA
    c897a1b View commit details
    Browse the repository at this point in the history

Commits on Sep 13, 2023

  1. Support accessing OSS with proxy

    ### What changes are proposed in this pull request?
    
    Support accessing OSS through proxy by configuring alluxio properties or system properties.
    
    ### Why are the changes needed?
    
    When accessing OSS through a proxy, the OSS client cannot recognize the proxy configuration in system property and environment variables. So it has to proactively set proxy-related configurations in the configuration.
    
    ### Does this PR introduce any user facing changes?
    
    Please list the user-facing changes introduced by your change, including
      1. change in user-facing APIs
      2. addition or removal of property keys
      3. webui
    
    			pr-link: Alluxio#18139
    			change-id: cid-5e30dfd90747d4a1aafe9b2ff985331f05fefec6
    Jackson-Wang-7 authored Sep 13, 2023
    Configuration menu
    Copy the full SHA
    6c97d6f View commit details
    Browse the repository at this point in the history
  2. Fix default value of proxy host

    ### What changes are proposed in this pull request?
    
    If don't set oss.proxy.host, the default value should be NULL
    
    ### Why are the changes needed?
    
    If don't set oss.proxy.host, the default value should be NULL
    
    ### Does this PR introduce any user facing changes?
    
    Please list the user-facing changes introduced by your change, including
      1. change in user-facing APIs
      2. addition or removal of property keys
      3. webui
    
    			pr-link: Alluxio#18142
    			change-id: cid-bce2790e583445c4ba6720d2f0a64551fb19de20
    Jackson-Wang-7 authored Sep 13, 2023
    Configuration menu
    Copy the full SHA
    1f146a9 View commit details
    Browse the repository at this point in the history
  3. Extract path resolution in DoraCacheFS into static utility

    ### What changes are proposed in this pull request?
    
    In this change the path conversion logic is extracted to static utility methods for code reuse (because other classes may use the same path resolution logic). 
    
    The method names are slightly improved, to distinguish the member methods in `DoraCacheFileSystem` (may be inherited) from the static utility methods.
    			pr-link: Alluxio#18140
    			change-id: cid-557fa148f6daa41f0b296132e5a2ecae6c5d6c22
    jiacheliu3 authored Sep 13, 2023
    Configuration menu
    Copy the full SHA
    7328331 View commit details
    Browse the repository at this point in the history

Commits on Sep 14, 2023

  1. [DOCFIX] Fix check-docs errMsg bug

    			pr-link: Alluxio#18138
    			change-id: cid-e9f862385bdfe6cc5e6938eb49907055449deb4a
    Zijian-Zhu authored Sep 14, 2023
    Configuration menu
    Copy the full SHA
    7fc1040 View commit details
    Browse the repository at this point in the history
  2. Move path resolution logic and fix a bug

    ### What changes are proposed in this pull request?
    1. Move the path resolution logic from `DoraCacheFileSystem` to `PathUtils` where it makes more sense
    2. Fix the alluxioPathToUfsPath resolution by handling the case where the ufs path may have no matching alluxio path, making the util method more generic
    			pr-link: Alluxio#18146
    			change-id: cid-ebace1efcf58e385bbf71b599e4b5a15a2199f7e
    jiacheliu3 authored Sep 14, 2023
    Configuration menu
    Copy the full SHA
    876d57f View commit details
    Browse the repository at this point in the history
  3. Improve dora stream error handling

    ### What changes are proposed in this pull request?
    
    Please outline the changes and how this PR fixes the issue.
    
    ### Why are the changes needed?
    
    Better error handling when a stream is closed. Log more and don't swallow errors
    
    ### Does this PR introduce any user facing changes?
    
    N/A
    			pr-link: Alluxio#18145
    			change-id: cid-b1146421ea5bc51e9a5eea5bf11ce1a5d8466912
    elega authored Sep 14, 2023
    Configuration menu
    Copy the full SHA
    1fe33f7 View commit details
    Browse the repository at this point in the history
  4. Add JSON-style reports

    Use jackson JSON as a standard format for reports.
    
    Output example for `bin/alluxio info report summary`:
    ```
    {
        "mSafeMode":false,
        "mZookeeper":false,
        "mRaftJournal":true,
        "version":"304-SNAPSHOT",
        "uptime":"0 day(s), 0 hour(s), 3 minute(s), and 7 second(s)",
        "rpcPort":19998,
        "webPort":19999,
        "masterAddress":"Ec2Cluster-masters-0:19998",
        "masterVersions":[
            {
                "version":"304-SNAPSHOT",
                "state":"PRIMARY",
                "host":"Ec2Cluster-masters-0",
                "port":19998
            }
        ],
        "started":"08-24-2023 06:43:32:856",
        "zookeeperAddress":[
    
        ],
        "raftJournalAddress":[
            "Ec2Cluster-masters-0:19200"
        ],
        "liveWorkers":2,
        "lostWorkers":0,
        "freeCapacity":"2048.00MB",
        "totalCapacityOnTiers":{
            "MEM":"2048.00MB"
        },
        "usedCapacityOnTiers":{
            "MEM":"0B"
        }
    }
    ```
    			pr-link: Alluxio#18047
    			change-id: cid-bf6d54f47390a4d2bd84e4baac2ea2863d4638e1
    twalluxio authored Sep 14, 2023
    Configuration menu
    Copy the full SHA
    959c527 View commit details
    Browse the repository at this point in the history

Commits on Sep 15, 2023

  1. Refactor the launch in CLI to be more flexible

    ### What changes are proposed in this pull request?
    refactor the cli code
    
    ### Why are the changes needed?
    make the code more flexible and easy for adding more functions in a cleaner way
    
    ### Does this PR introduce any user facing changes?
    nope
    
    			pr-link: Alluxio#18152
    			change-id: cid-d8f937075174d913daf32387d781161096f03345
    Kai-Zhang authored Sep 15, 2023
    Configuration menu
    Copy the full SHA
    9f34c2f View commit details
    Browse the repository at this point in the history
  2. Write to UFS when failed to write to Alluxio Worker

    1. Use `addSuppressed` instead of creating a new exception to throw the original exception.
    2. Go ahead to write data to UFS when it encountered exception during the time writing data to Alluxio Worker.
    
    			pr-link: Alluxio#18017
    			change-id: cid-9337252b71e40fced28fb1598ea88eed56c69229
    JiamingMai authored Sep 15, 2023
    Configuration menu
    Copy the full SHA
    84dc05f View commit details
    Browse the repository at this point in the history
  3. Improve distributed load

    ### What changes are proposed in this pull request?
    
    Improve distributed load
    1. Configurable job failure criteria 
    2. Configuration to determine if the load job should be restored from journal or not
    3. Add an option to skip existing fully loaded file 
    4. Add retry count for failed files 
    5. Bug fixing
    
    ### Why are the changes needed?
    
    To enhance the distributed load tool
    
    ### Does this PR introduce any user facing changes?
    
    Yes. The skip-if-exists option is added to the distributed load cli.
    			pr-link: Alluxio#18153
    			change-id: cid-5644da1c09bd6ee48f628552f51cb570de581b93
    elega authored Sep 15, 2023
    Configuration menu
    Copy the full SHA
    6a9f5fd View commit details
    Browse the repository at this point in the history
  4. Recover the ufs uri support

    ### What changes are proposed in this pull request?
    
    Recover the ufs uri support
    
    ### Why are the changes needed?
    
    Ufs uri should be the first class citizen in dora
    
    ### Does this PR introduce any user facing changes?
    
    No
    
    			pr-link: Alluxio#18135
    			change-id: cid-a35fe39bdf69879ef113fc97737a35ebf6d8b29a
    beinan authored Sep 15, 2023
    Configuration menu
    Copy the full SHA
    100eb4f View commit details
    Browse the repository at this point in the history
  5. [DOCFIX] Update docker documentation

    Add docker doc back
    			pr-link: Alluxio#18130
    			change-id: cid-6038dce4ae1821f0e7ccf0e2e874bed5d312057d
    ssz1997 authored Sep 15, 2023
    Configuration menu
    Copy the full SHA
    f05deb3 View commit details
    Browse the repository at this point in the history
  6. Add a test to monitor the distribution of vnode to physical node

    ### What changes are proposed in this pull request?
    
    Add a test so we can monitor the vnode distribution is not too uneven. 
    
    This test calculates the standard deviation over mean on the collection of virtual nodes assigned to physical nodes. It arbitrarily bounds it at 0.25, but ideally this number should get smaller over time as we improve hashing algorithm
    and use better ways to assign virtual nodes to physical nodes.
    
    
    ### Why are the changes needed?
    
    We may change hashing algorithm and virtual node assignment in the future, this will provide guidance and catch errors. 
    
    ### Does this PR introduce any user facing changes?
    No. 
    
    			pr-link: Alluxio#18147
    			change-id: cid-152d8edc9b65ef59967d5985849feeb471a6650d
    yuzhu authored Sep 15, 2023
    Configuration menu
    Copy the full SHA
    8bba797 View commit details
    Browse the repository at this point in the history

Commits on Sep 17, 2023

  1. Add CLI tools for debugging and analyzing caching issues

    Add the following CLI tools for debugging and analyzing caching issues:
    1. checkCaching. Checks if files under a path have been cached in alluxio.
    2. location. Displays the list of hosts storing the specified file.
    3. consistentHash. This command is for checking whether the consistent hash ring is changed or not.
    			pr-link: Alluxio#18151
    			change-id: cid-c89b98da70a5270070d873bdcfce1aa9b23cf083
    JiamingMai authored Sep 17, 2023
    Configuration menu
    Copy the full SHA
    6f35645 View commit details
    Browse the repository at this point in the history

Commits on Sep 18, 2023

  1. Parse JSON-style reports in golang side

    Update the Golang side commands to be able to use this output:
    1. Return either the yaml or json (default) output to the console.
    2. Users can define the format they want with `--format` flag, like `bin/alluxio info report --format yaml`
    3. In JSON format, print properties in a fixed, easy-to-read order
    4. In YAML format, print properties alphabetically (since YAML specification regards property order non-significant)
    
    Before:
    ```
    {"safeMode":false,"masterVersions":[{"version":"304-SNAPSHOT","host":"localhost","port":19998,"state":"PRIMARY"}],"masterAddress":"localhost:19998","zookeeperAddress":[],"useZookeeper":false,"raftJournalAddress":["localhost:19200"],"useRaftJournal":true,"liveWorkers":1,"lostWorkers":0,"freeCapacity":"1024.00MB","totalCapacityOnTiers":{"MEM":"1024.00MB"},"usedCapacityOnTiers":{"MEM":"0B"},"version":"304-SNAPSHOT","webPort":19999,"started":"09-15-2023 15:54:56:635","uptime":"0 day(s), 0 hour(s), 26 minute(s), and 37 second(s)","rpcPort":19998}
    ```
    
    After (in JSON):
    ```
    {
        "rpcPort": 19998,
        "started": "09-15-2023 15:54:56:635",
        "uptime": "0 day(s), 0 hour(s), 55 minute(s), and 31 second(s)",
        "safeMode": false,
        "version": "304-SNAPSHOT",
        "webPort": 19999,
        "masterVersions": [
            {
                "version": "304-SNAPSHOT",
                "host": "localhost",
                "port": 19998,
                "state": "PRIMARY"
            }
        ],
        "masterAddress": "localhost:19998",
        "zookeeperAddress": [],
        "useZookeeper": false,
        "raftJournalAddress": [
            "localhost:19200"
        ],
        "useRaftJournal": true,
        "liveWorkers": 1,
        "lostWorkers": 0,
        "freeCapacity": "1024.00MB",
        "totalCapacityOnTiers": {
            "MEM": "1024.00MB"
        },
        "usedCapacityOnTiers": {
            "MEM": "0B"
        }
    }
    ```
    
    After (in YAML):
    ```
    freeCapacity: 1024.00MB
    liveWorkers: 1
    lostWorkers: 0
    masterAddress: localhost:19998
    masterVersions:
        - host: localhost
          port: 19998
          state: PRIMARY
          version: 304-SNAPSHOT
    raftJournalAddress:
        - localhost:19200
    rpcPort: 19998
    safeMode: false
    started: 09-15-2023 15:54:56:635
    totalCapacityOnTiers:
        MEM: 1024.00MB
    uptime: 0 day(s), 1 hour(s), 1 minute(s), and 36 second(s)
    useRaftJournal: true
    useZookeeper: false
    usedCapacityOnTiers:
        MEM: 0B
    version: 304-SNAPSHOT
    webPort: 19999
    zookeeperAddress: []
    ```
    			pr-link: Alluxio#18159
    			change-id: cid-deb6e74552de9afcf45391c6c230a9fe00785e37
    twalluxio authored Sep 18, 2023
    Configuration menu
    Copy the full SHA
    86308c3 View commit details
    Browse the repository at this point in the history
  2. Add datePredicate

    ### What changes are proposed in this pull request?
    
    Add datePredicate, i.e.:
    lastModifiedDate(2000/01/01, 2023/09/01)
    
    ### Why are the changes needed?
    
    Customer requirement.
    
    ### Does this PR introduce any user facing changes?
    
    na
    
    			pr-link: Alluxio#18155
    			change-id: cid-7e1a7b7d208747807b87502c9da854ddf0b8c7fc
    ssyssy authored Sep 18, 2023
    Configuration menu
    Copy the full SHA
    838893d View commit details
    Browse the repository at this point in the history
  3. Add throughput distribution and coarse datapoints for StressWorkerBench

    For random reads, bytes read per file is not a constant any more. In spite of existing duration distribution, need a throughput distribution for better understanding of reading performance.
    
    Also, when duration too long, grpc will receive huge size of output data. Should aggregate data points to transfer more datapoints with limited output size.
    
    Group datapoints by threads and time slices:
    Example:
    ```
    nodeResults: {
      worker-0: {
        dataPoints: [
          data: [
            { // worker 0, thread 0, slice 0
              count: 1,
              iobytes: 33554432,
            }, { // worker 0, thread 0, other slices
              …
            }
          ], [ // worker 0, other workers
            …
          ]
        ]
        throughputPercentiles: […]
      },
      worker-1: { // other workers
        …
      }
    }
    ```
    
    Slice time with `--slice-size` flag, e.g. `--slice-size 1s`.
    			pr-link: Alluxio#18149
    			change-id: cid-ec8ed5a4f9eeaa86b1d86b6b449db4647d584823
    twalluxio authored Sep 18, 2023
    Configuration menu
    Copy the full SHA
    ddba020 View commit details
    Browse the repository at this point in the history
  4. Add Unit Test for OSS, OBS and GCS

    ### What changes are proposed in this pull request?
    
    Add Unit Test for OSS, OBS and GCS
    
    ### Why are the changes needed?
    
    Unit test is important for improving functions of Alluxio.
    
    ### Does this PR introduce any user facing changes?
    
    No
    
    			pr-link: Alluxio#17985
    			change-id: cid-c757b8249c62e2ccf0483cb99436e33d351358a1
    Zihao Zhao authored Sep 18, 2023
    Configuration menu
    Copy the full SHA
    71fd8c4 View commit details
    Browse the repository at this point in the history

Commits on Sep 19, 2023

  1. Fix datePredicate

    ### What changes are proposed in this pull request?
    
    Fix datePredicate so it would respect the interval specified in the policy.
    Polish tests
    ### Why are the changes needed?
    
    bug fix
    
    ### Does this PR introduce any user facing changes?
    na
    
    			pr-link: Alluxio#18167
    			change-id: cid-b035b5cc31aa70a2f83de2c3f84ba49ed75f9fb5
    jja725 authored Sep 19, 2023
    Configuration menu
    Copy the full SHA
    db4fc2b View commit details
    Browse the repository at this point in the history
  2. Generate user cli doc

    - Add a new CLI command that iterates through the command tree and generates a markdown file based on each command's definition and description
    - Migrate all the content in the previous User-CLI.md into the corresponding commands in golang code, mainly updating their `Long` and `Examples` fields
    - Run `bin/alluxio generate user-cli` to write the generated content directly into `docs/en/operation/User-CLI.md`
    			pr-link: Alluxio#18144
    			change-id: cid-9b29f273efef9693e1b0b303c62cc19602d77acc
    Xenorith authored Sep 19, 2023
    Configuration menu
    Copy the full SHA
    dd863fe View commit details
    Browse the repository at this point in the history
  3. Fix the broken PagedDoraWorkerTest

    ### What changes are proposed in this pull request?
    
    Fix the broken PagedDoraWorkerTest.
    
    ### Why are the changes needed?
    
    The old test is.broken, I just fix it.
    
    ### Does this PR introduce any user facing changes?
    
    no.
    
    			pr-link: Alluxio#18150
    			change-id: cid-a142297c4f08780189e4321abc8e99ff512091ec
    YichuanSun authored Sep 19, 2023
    Configuration menu
    Copy the full SHA
    d737d56 View commit details
    Browse the repository at this point in the history
  4. Fix NoClassDefFoundError UnderFileSystemFactory not loaded

    ### What changes are proposed in this pull request?
    
    Fix UFS not loaded in some cases.
    
    ### Why are the changes needed?
    
    Fix the following error when loading `UnderFileSystemFactory`:
    
    ```
    java.lang.NoClassDefFoundError: alluxio/underfs/UnderFileSystemFactory
    	at java.lang.ClassLoader.defineClass1(Native Method)
    	at java.lang.ClassLoader.defineClass(ClassLoader.java:756)
    	at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
    	at java.net.URLClassLoader.defineClass(URLClassLoader.java:468)
    	at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
    	at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
    	at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
    	at java.security.AccessController.doPrivileged(Native Method)
    	at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
    	at alluxio.extensions.ExtensionsClassLoader.findClass(ExtensionsClassLoader.java:73)
    	at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
    	at alluxio.extensions.ExtensionsClassLoader.loadClass(ExtensionsClassLoader.java:82)
    	at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
    	at java.lang.Class.forName0(Native Method)
    	at java.lang.Class.forName(Class.java:348)
    	at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:370)
    	at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
    	at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
    	at alluxio.extensions.ExtensionFactoryRegistry.scan(ExtensionFactoryRegistry.java:248)
    	at alluxio.extensions.ExtensionFactoryRegistry.scanLibs(ExtensionFactoryRegistry.java:229)
    	at alluxio.extensions.ExtensionFactoryRegistry.findAllWithRecorder(ExtensionFactoryRegistry.java:149)
    	at alluxio.underfs.UnderFileSystemFactoryRegistry.findAllWithRecorder(UnderFileSystemFactoryRegistry.java:115)
    	at alluxio.underfs.UnderFileSystem$Factory.createWithRecorder(UnderFileSystem.java:111)
    	at alluxio.underfs.UnderFileSystem$Factory.create(UnderFileSystem.java:93)
    ```
    
    This was because when `Thread.currentThread().getContextClassLoader()` is null, it indicates that the class loader is actually the system class loader. If using `null` for the default class loader, it will not be able to resolve `UnderFileSystemFactory` which was already loaded by the system class loader.
    
    ### Does this PR introduce any user facing changes?
    No.
    			pr-link: Alluxio#18169
    			change-id: cid-d79281b3fb14e894e4ff21fe34222ef9e04f05d4
    dbw9580 authored Sep 19, 2023
    Configuration menu
    Copy the full SHA
    814fad4 View commit details
    Browse the repository at this point in the history

Commits on Sep 20, 2023

  1. return an all zero fsstat

    Alluxio does not have a valid Block Info at this moment.
    
    ### What changes are proposed in this pull request?
    
    return an all zero block info for fsstat
    
    ### Why are the changes needed?
    
    No valid block info is managed by Alluxio at this moment.
    Alluxio does not have 'master' node, and no block info is stored/managed by other components.
    
    This may change in future, e.g. statfs() can gather the block info from all worker nodes.
    
    ### Does this PR introduce any user facing changes?
    
    N/A
    
    			pr-link: Alluxio#18172
    			change-id: cid-eda9fb4ad68fa91d6d766285f6de211d5f02df56
    huanghua78 authored Sep 20, 2023
    Configuration menu
    Copy the full SHA
    4b9ab67 View commit details
    Browse the repository at this point in the history
  2. Add back journal checkpoint command

    			pr-link: Alluxio#18174
    			change-id: cid-996586c993d3920cd91c4663a8771ced840fe80f
    Xenorith authored Sep 20, 2023
    Configuration menu
    Copy the full SHA
    796b66f View commit details
    Browse the repository at this point in the history
  3. [DOCFIX] Fix typo in generate docs cmd

    			pr-link: Alluxio#18177
    			change-id: cid-9e895702ab3f800a47fed97ce2c1a756abb47819
    Xenorith authored Sep 20, 2023
    Configuration menu
    Copy the full SHA
    e21a7a3 View commit details
    Browse the repository at this point in the history
  4. Enable config the virtual nodes num per node

    The configuration `alluxio.user.consistent.hash.virtual.node.count` is for total virtual nodes of all workers. We need to adjust the configuration according to the number of worker nodes. However, it is difficult to determine how many virtual nodes are enough to avoid the data skew issue.
    
    Based on this background, a better way to solve the data skew issue is to allow user to set the virtual nodes number for a worker node, but not the total virtual nodes of all worker nodes. This is what this PR does.
    			pr-link: Alluxio#18179
    			change-id: cid-fbb4c9ebe13f5c0cce4ac08f9fe1789d97cdf209
    JiamingMai authored Sep 20, 2023
    Configuration menu
    Copy the full SHA
    7b37a86 View commit details
    Browse the repository at this point in the history

Commits on Sep 21, 2023

  1. Fix GCSV2UnderFileSystem initialization

    ### What changes are proposed in this pull request?
    
    Fix GCSV2UnderFileSystem initialization
    
    ### Why are the changes needed?
    
    not able to create ufs instance if not set access key, which should not be a requirement for gcs v2
    
    ### Does this PR introduce any user facing changes?
    
    na
    
    			pr-link: Alluxio#18184
    			change-id: cid-c736612c8ccc2d5d8bd9ecbd9f862942625e8156
    jja725 authored Sep 21, 2023
    Configuration menu
    Copy the full SHA
    765048c View commit details
    Browse the repository at this point in the history
  2. Stop fuse web server

    And refactor fuse start code into startCommon() and stopCommon()
    
    ### What changes are proposed in this pull request?
    
    Stop Fuse web server when exit.
    
    ### Why are the changes needed?
    
    Fuse web server must be stopped, otherwise the main thread is waiting for it forever.
    
    ### Does this PR introduce any user facing changes?
    
    N/A
    
    			pr-link: Alluxio#18190
    			change-id: cid-cdad62f25870a6a1e1984127834402045b9763d4
    huanghua78 authored Sep 21, 2023
    Configuration menu
    Copy the full SHA
    6f707e1 View commit details
    Browse the repository at this point in the history
  3. Only stop processes belong to the current user

    ### What changes are proposed in this pull request?
    
    Only stop processes belong to the current user while running `./bin/alluxio process start` or `./bin/alluxio process stop`
    
    ### Why are the changes needed?
    
    Without this PR, it will try to stop all processes match the class name even they are not started by the current user. Like the below case
    
    ```
    $ ./bin/alluxio process stop local
    WARN[2023-09-21T14:57:31+08:00] No process to stop because could not find running process matching alluxio.worker.AlluxioWorker
    INFO[2023-09-21T14:57:31+08:00] Found 1 running process(es) matching alluxio.master.AlluxioMaster
    INFO[2023-09-21T14:57:31+08:00] Successfully killed 0 process(es)
    ERRO[2023-09-21T14:57:31+08:00] Failed to kill 1 process(es):
    ERRO[2023-09-21T14:57:31+08:00] Failed to kill process: error sending TERM signal to process for alluxio.master.AlluxioMaster with pid 17055
     --- at /tmp/alluxio1717887242/cli/src/alluxio.org/cli/env/process.go:175 (BaseProcess.Stop.func1) ---
    Caused by: operation not permitted
    ```
    
    ### Does this PR introduce any user facing changes?
    
    NO
    
    			pr-link: Alluxio#18187
    			change-id: cid-ee45c89e93fff4bd8c58dba3e81cd1fa4de52368
    secfree authored Sep 21, 2023
    Configuration menu
    Copy the full SHA
    2e18cd2 View commit details
    Browse the repository at this point in the history
  4. Fix typo in ServiceDiscoveryRecipe

    ### What changes are proposed in this pull request?
    Fixed a typo
    
    ### Why are the changes needed?
    
    ### Does this PR introduce any user facing changes?
    No
    
    			pr-link: Alluxio#18191
    			change-id: cid-67a503376ed89395ebccd674644394e5bc02522b
    voddle authored Sep 21, 2023
    2 Configuration menu
    Copy the full SHA
    d056937 View commit details
    Browse the repository at this point in the history

Commits on Sep 22, 2023

  1. Fix the bug of using a terminated executor in S3 UFS

    ### What changes are proposed in this pull request?
    Don't call the ufs.close() in PageFileReader.close()
    
    ### Why are the changes needed?
    1. we should not close the ufs instance, because we always try to reuse it for the same ufs mount point. unless we don't reuse the instance of the ufs instance.
    2. S3UnderFileSystem.close() will just shut down the executor held by the instance. when other threads reuse the executor, it will cause issues.
    
    ### Does this PR introduce any user facing changes?
    no
    
    			pr-link: Alluxio#18188
    			change-id: cid-2d97d2ba48bb8dc333cb8d2dddd7990dad5d372b
    Jackson-Wang-7 authored Sep 22, 2023
    Configuration menu
    Copy the full SHA
    3a82f8e View commit details
    Browse the repository at this point in the history
  2. Add new worker identity

    ### What changes are proposed in this pull request?
    
    1. Add an abstract representation of a worker's identity.
    The new identity is the same as a `long` based identity that used to be generated by the master. It decouples the concept of an identity and the actual implementation of an identity. It also includes a version field that allows backward compatible evolution of the worker identifier.
    
    2. A new implementation for worker identities that uses UUID.
    A worker will generate its own identity instead of acquiring one from the master, unless the user explicitly specifies one in the configuration. The generated identity is then persisted in the worker's working directory, and reloaded at the next worker restart.
    
    ### Why are the changes needed?
    
    For Dora workers, the identities are not generated by a centralized service, but by workers themselves. Therefore, we need something that provides randomness and avoids collision between workers, e.g. a UUID. The V1 parser implementation included in this PR uses a UUID as the worker identity.
    
    ### Does this PR introduce any user facing changes?
    
    Yes, a new configuration item `alluxio.worker.identity.uuid` is added that allows a user to specify the worker identity.
    
    			pr-link: Alluxio#17820
    			change-id: cid-3200e78fa24be0126d4d49370f0a266f8c766e12
    dbw9580 authored Sep 22, 2023
    Configuration menu
    Copy the full SHA
    82bd57b View commit details
    Browse the repository at this point in the history
  3. [DOCFIX] Describe how the user cli docs are generated

    dev docs for generating user cli documentation
    			pr-link: Alluxio#18192
    			change-id: cid-ee3ad14a4a284ddf2040850f4d590e1ccb70cb69
    Xenorith authored Sep 22, 2023
    Configuration menu
    Copy the full SHA
    bc4f95b View commit details
    Browse the repository at this point in the history
  4. Split the fuse stream close() method

    ### What changes are proposed in this pull request?
    
    Split the fuse stream close() method into two individual methods:
    1. Close the fuse stream
    2. Release the lock
    
    Currently no behavior changes on the close function as the function will call these two functions to achieve the same functionality as before. 
    
    ### Why are the changes needed?
    
    When we handle some fuse operations, we sometimes want to close the stream only but want to keep the lock there, to avoid inconsistency.  
    
    ### Does this PR introduce any user facing changes?
    
    Please list the user-facing changes introduced by your change, including
      1. change in user-facing APIs
      3. addition or removal of property keys
      4. webui
    
    			pr-link: Alluxio#18180
    			change-id: cid-93e1f5c6c2448536321cbcbb528471b032c4c01f
    elega authored Sep 22, 2023
    Configuration menu
    Copy the full SHA
    325582e View commit details
    Browse the repository at this point in the history
  5. Refactor the collectInfo main for future use

    ### What changes are proposed in this pull request?
    Refactor some of the code in java and go to make the CLI more flexible for the future
    
    ### Why are the changes needed?
    There might be more commands added in the future
    
    ### Does this PR introduce any user facing changes?
    nope
    			pr-link: Alluxio#18197
    			change-id: cid-00c52688dd60bc0a451cfbff932b1da329549993
    Kai-Zhang authored Sep 22, 2023
    Configuration menu
    Copy the full SHA
    4e2a494 View commit details
    Browse the repository at this point in the history
  6. Improve the random read behavior in StressWorkerBench

    ### What changes are proposed in this pull request?
    
    Improve the randomness of StressWorkerBench random read test, now each thread throw dice every time it trying to do a random read.
    
    ### Why are the changes needed?
    
    In previous StressWorkerBench random read test each thread read from same offset and same length everytime, this cause low randomness.
    
    ### Does this PR introduce any user facing changes?
    
    no
    			pr-link: Alluxio#18000
    			change-id: cid-974afb58c022061e8a6d8d6894415a8bd292b764
    voddle authored Sep 22, 2023
    Configuration menu
    Copy the full SHA
    be6974c View commit details
    Browse the repository at this point in the history
  7. Parse JSON-style reports in golang side

    Cherry-pick of existing commit.
    orig-pr: Alluxio#18159
    orig-commit: Alluxio/alluxio@86308c3
    orig-commit-author: twalluxio <139105284+twalluxio@users.noreply.github.com>
    
    			pr-link: Alluxio#18166
    			change-id: cid-deb6e74552de9afcf45391c6c230a9fe00785e37
    twalluxio authored Sep 22, 2023
    Configuration menu
    Copy the full SHA
    ba7fedf View commit details
    Browse the repository at this point in the history
  8. Update collectAlluxioInfo and other bin/alluxio cmd strings

    collectAlluxioInfo calls bin/alluxio commands and collects their output. fix the commands after the golang CLI refactor
    			pr-link: Alluxio#18186
    			change-id: cid-23c99f152405d04dcdd9bbea3953b00e23ee3882
    Xenorith authored Sep 22, 2023
    Configuration menu
    Copy the full SHA
    e3d7c80 View commit details
    Browse the repository at this point in the history
  9. Refactor build logic for usability

    ### What changes are proposed in this pull request?
    
    - made some funcs/consts public
    - tarball artifacts list lib/plugin modules in `metadata` field
    
    ### Why are the changes needed?
    
    
    ### Does this PR introduce any user facing changes?
    
    no
    			pr-link: Alluxio#18195
    			change-id: cid-7f7738ef1f2d4e9907dd5d9e047870f048cdd269
    tieujason330 authored Sep 22, 2023
    Configuration menu
    Copy the full SHA
    e12f626 View commit details
    Browse the repository at this point in the history
  10. Fix last modified time is 0 in UfsFileIterable

    ### What changes are proposed in this pull request?
    
    Fix last modified time is 0 in UfsFileIterable
    
    ### Why are the changes needed?
    
    bug fix
    
    ### Does this PR introduce any user facing changes?
    na
    
    
    			pr-link: Alluxio#18200
    			change-id: cid-e3fc548f65b642f8b8cb62d3b94f26366e33e4a7
    jja725 authored Sep 22, 2023
    Configuration menu
    Copy the full SHA
    c75317f View commit details
    Browse the repository at this point in the history

Commits on Sep 23, 2023

  1. Fix and re-enable UfsFileIn/OutStreamTest

    ### What changes are proposed in this pull request?
      UfsFileIn/OutStream Test runnable now
    
    
    ### Why are the changes needed?
      Need UfsFileIn/OutStream Test
    
    ### Does this PR introduce any user facing changes?
      No
    
    
    
    			pr-link: Alluxio#18170
    			change-id: cid-2de71fa1d5bc6189c03f6fd496f55c5c53fd778c
    voddle authored Sep 23, 2023
    Configuration menu
    Copy the full SHA
    e68c6d8 View commit details
    Browse the repository at this point in the history

Commits on Sep 24, 2023

  1. Enable static consistent hash ring

    By default, we build a dynamic consistent hash with the live worker list that comes from master or ETCD. Sometimes we want to build a static consistent hash ring to make sure we won't write data to other worker node when a worker node is offline temporarily (especially when other worker nodes are running out of disk space).
    
    This PR provides allows us to build a static consistent hash ring by setting `alluxio.user.dynamic.consistent.hash.ring.enabled=false`. In this case, client will read from UFS if the worker where the specified file locate is down.
    			pr-link: Alluxio#18183
    			change-id: cid-ccd37436391d62aa7c6a9942c80e2d3c8ffdb170
    JiamingMai authored Sep 24, 2023
    Configuration menu
    Copy the full SHA
    189ca24 View commit details
    Browse the repository at this point in the history

Commits on Sep 25, 2023

  1. Add Read Metrics

    ### What changes are proposed in this pull request?
    
    Add client read metrics
    
    ### Why are the changes needed?
    
    These metrics are helpful.
    
    ### Does this PR introduce any user facing changes?
    
    N/A
    
    			pr-link: Alluxio#18160
    			change-id: cid-08d84f39da7aec0ca21e1d57731930a67693dbeb
    huanghua78 authored Sep 25, 2023
    Configuration menu
    Copy the full SHA
    fe55fc9 View commit details
    Browse the repository at this point in the history
  2. Add Dora read fallback to UFS logics

    Support dora read to fallback to UFS if any error occur during read.
    
    			pr-link: Alluxio#18154
    			change-id: cid-755c5bdbe4edf9abf32a7d700d17617cbb8a78a1
    LuQQiu authored Sep 25, 2023
    Configuration menu
    Copy the full SHA
    3dd49e4 View commit details
    Browse the repository at this point in the history
  3. Close Etcd client when process exits

    ### What changes are proposed in this pull request?
    
    1. Since EtcdClient is a singleton, close it in the jvm shutdownHook
    2. Close related resources within EtcdClient
    
    			pr-link: Alluxio#18204
    			change-id: cid-b00c40829136c88dd6a6e1c0731b1a5ee3647cff
    jiacheliu3 authored Sep 25, 2023
    Configuration menu
    Copy the full SHA
    a2d8410 View commit details
    Browse the repository at this point in the history
  4. Fix read position in PagedDoraWorker

    ### What changes are proposed in this pull request?
    
    1. The length of the read is incorrectly calculated. Should not have anything to do with the position of the reader.
    2. `mPos` should not be updated for the `read` method, as it's a positioned read operation (has an `offset` argument).
    
    ### Why are the changes needed?
    
    bug fix
    
    ### Does this PR introduce any user facing changes?
    No.
    
    			pr-link: Alluxio#18205
    			change-id: cid-eb56603cad6c462e799793ca1086d3638209af00
    dbw9580 authored Sep 25, 2023
    Configuration menu
    Copy the full SHA
    9bfdc07 View commit details
    Browse the repository at this point in the history
  5. Improve resource deallocation in FUSE

    ### What changes are proposed in this pull request?
    
    This improves resource deallocation in two ways:
    1. Close FileSystemContext
    2. Close an optional thread pool on normal exit (before it is only destroy on error)
    
    ### Why are the changes needed?
    
    Resource deallocation
    			pr-link: Alluxio#18206
    			change-id: cid-03ec1f712460172dd8d61a34fc65b95484243a3b
    jiacheliu3 authored Sep 25, 2023
    Configuration menu
    Copy the full SHA
    fc70ad2 View commit details
    Browse the repository at this point in the history
  6. Move json formatting in info as util SKIPCI

    move code into separate package
    			pr-link: Alluxio#18207
    			change-id: cid-91860a65af36b70f98e8913f0710e6a295d14008
    Xenorith authored Sep 25, 2023
    Configuration menu
    Copy the full SHA
    d8c0ea8 View commit details
    Browse the repository at this point in the history
  7. Remove log

    ### What changes are proposed in this pull request?
    
    Please outline the changes and how this PR fixes the issue.
    
    ### Why are the changes needed?
    
    Please clarify why the changes are needed. For instance,
      1. If you propose a new API, clarify the use case for a new API.
      2. If you fix a bug, describe the bug.
    
    ### Does this PR introduce any user facing changes?
    
    Please list the user-facing changes introduced by your change, including
      1. change in user-facing APIs
      2. addition or removal of property keys
      3. webui
    
    			pr-link: Alluxio#18209
    			change-id: cid-3d147d628a6be3a0e654eec338ea76d2fb5f25cf
    elega authored Sep 25, 2023
    Configuration menu
    Copy the full SHA
    c11e4cb View commit details
    Browse the repository at this point in the history
  8. Remove master host check

    ### What changes are proposed in this pull request?
    
    remove master host check
    
    ### Why are the changes needed?
    master host should not be a hard requirement right now.
    
    ### Does this PR introduce any user facing changes?
    na
    
    			pr-link: Alluxio#18201
    			change-id: cid-7cfc7a688e925137a1e9d62e4af0a07298f447d4
    jja725 authored Sep 25, 2023
    Configuration menu
    Copy the full SHA
    8d0c0ff View commit details
    Browse the repository at this point in the history

Commits on Sep 26, 2023

  1. Add test for DoraMetaManager

    Add DoraMetaManagerTest.
    			pr-link: Alluxio#18127
    			change-id: cid-7c8f875ce957e85597c53bc5db611f1d533596ee
    voddle authored Sep 26, 2023
    Configuration menu
    Copy the full SHA
    9ab7391 View commit details
    Browse the repository at this point in the history
  2. Modify StressWorkerBench: calculate elapsed time in nanoseconds

    For now StressWorkerBench uses milliseconds as the smallest unit of recording. In random read, if file is very small, it is possible to have a duration <1ms. Since instant throughput = bytes read / duration, it will lead to a divide by zero error.
    This PR replace millisecond-level record of a file read with a nanosecond-level approach. It also makes the following updates on StressWorkerBench:
    - record file read time within `applyOperation()` function, instead of before entering or after exiting it
    - wrap metrics about the `applyOperation()` function to a private class, instead of returning bytes read only
    - remove getter and setters for WorkerBenchDataPoint, since all elements inside is public
    - standardize the unit of output throughput, now unit is MB/s
    			pr-link: Alluxio#18175
    			change-id: cid-a606ca5812f75d8116ecaf7da45da9dedd78ab55
    twalluxio authored Sep 26, 2023
    Configuration menu
    Copy the full SHA
    527ffe8 View commit details
    Browse the repository at this point in the history
  3. Add manual exit at the end of FUSE life cycle

    ### What changes are proposed in this pull request?
    
    Added a manual double-safety `System.exit()` at the end of FUSE lifecycle, to be absolutely sure we exit the JVM and terminate all non-daemon & daemon threads.
    
    Before this change, after we `alluxio-fuse umount` or `kill (not -9)`, the FUSE process may fail to quit. One reason is a non-daemon thread dangling around:
    ```
    # This is a non daemon thread in AlluxioEtcdClient
    "vert.x-eventloop-thread-0" Alluxio#21 prio=5 os_prio=31 cpu=482.23ms elapsed=722.11s tid=0x00007fde79251800 nid=0x8c03 runnable  [0x00007000102ae000]
       java.lang.Thread.State: RUNNABLE
       at sun.nio.ch.KQueue.poll(java.base@11.0.11/Native Method)
       at sun.nio.ch.KQueueSelectorImpl.doSelect(java.base@11.0.11/KQueueSelectorImpl.java:122)
       at sun.nio.ch.SelectorImpl.lockAndDoSelect(java.base@11.0.11/SelectorImpl.java:124)
       - locked <0x00000007c0603938> (a io.netty.channel.nio.SelectedSelectionKeySet)
       - locked <0x00000007c06038d8> (a sun.nio.ch.KQueueSelectorImpl)
       at sun.nio.ch.SelectorImpl.select(java.base@11.0.11/SelectorImpl.java:136)
       at io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62)
       at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883)
       at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526)
       at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
       at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
       at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
       at java.lang.Thread.run(java.base@11.0.11/Thread.java:829)
    ```
    
    The JVM shutdownHook executes after the LAST non-daemon thread dies or when `System.exit()` triggers. So if some libraries we use or some thread pools of ours accidentally introduce non-daemon threads, `System.exit()` is our last resort to trigger shutdownHook.
    
    FUSE does handle signals and exit, but there are some corner cases which don't seem to trigger that successfully. So this PR serves as the last resort.
    https://github.com/Alluxio/alluxio/blob/527ffe8105568757e05dc81dff1e28a8078e9a9e/dora/integration/fuse/src/main/java/alluxio/fuse/FuseSignalHandler.java#L52
    
    ### Why are the changes needed?
    
    See above
    
    ### Does this PR introduce any user facing changes?
    
    No
    
    			pr-link: Alluxio#18211
    			change-id: cid-adeaeadfaa5fb4013a62ea8191005b93c0ca69ad
    jiacheliu3 authored Sep 26, 2023
    Configuration menu
    Copy the full SHA
    926f393 View commit details
    Browse the repository at this point in the history
  4. Add project structure and template for native components

    ### What changes are proposed in this pull request?
    
    Prepare project structure for native components.
    Add a maven archetype for Rust-based native components.
    The `native` module is behind a profile `native-components` and is deactivated by default.
    
    ### Why are the changes needed?
    
    Add support for native components
    
    ### Does this PR introduce any user facing changes?
    
    No.
    
    			pr-link: Alluxio#18164
    			change-id: cid-530dd723b84d69cef22b08f27eae7e4b4874995a
    dbw9580 authored Sep 26, 2023
    Configuration menu
    Copy the full SHA
    dc0849a View commit details
    Browse the repository at this point in the history
  5. Make copy & move return list of tasks

    ### What changes are proposed in this pull request?
    
    Make copy & move return list of tasks
    
    ### Why are the changes needed?
    make worker busy
    ### Does this PR introduce any user facing changes?
    na
    
    			pr-link: Alluxio#18215
    			change-id: cid-d0c3abe26ec4cee3cdbd93e2a99ffb319b187524
    jja725 authored Sep 26, 2023
    Configuration menu
    Copy the full SHA
    dec36be View commit details
    Browse the repository at this point in the history

Commits on Sep 27, 2023

  1. Create a init test for cluster

    ### What changes are proposed in this pull request?
    
    I have created a init test to check whether the workers has already run and connected to the leader.
    
    ### Why are the changes needed?
    
    Because the version has updated to 3.0 but we still do not have the correct init test for the cluster.
    
    ### Does this PR introduce any user facing changes?
    
    Yes, users may still not run test when they first run the cluster because i must communicate with leader to decide which args will be used.
    
    			pr-link: Alluxio#18148
    			change-id: cid-74a018569e1ba13c55573ba9e1713bf9dd8e3f2d
    thu-david authored Sep 27, 2023
    Configuration menu
    Copy the full SHA
    9047c62 View commit details
    Browse the repository at this point in the history
  2. Fix check-cached

    ### What changes are proposed in this pull request?
    
    Please outline the changes and how this PR fixes the issue.
    
    ### Why are the changes needed?
    
    Please clarify why the changes are needed. For instance,
      1. If you propose a new API, clarify the use case for a new API.
      2. If you fix a bug, describe the bug.
    
    ### Does this PR introduce any user facing changes?
    
    Please list the user-facing changes introduced by your change, including
      1. change in user-facing APIs
      2. addition or removal of property keys
      3. webui
    
    			pr-link: Alluxio#18223
    			change-id: cid-375b97fb3beb2714a75dbb227c15978e18c2909a
    elega authored Sep 27, 2023
    Configuration menu
    Copy the full SHA
    cde8e4c View commit details
    Browse the repository at this point in the history
  3. [DOCFIX] Remove deprecated sections from journal docs

    also update `init format` command to be able to format journal on all masters but avoid formatting workers
    			pr-link: Alluxio#18124
    			change-id: cid-b8fd99ec5c6e6b7dbaa1c9a22414fad2a33caf50
    Xenorith authored Sep 27, 2023
    Configuration menu
    Copy the full SHA
    535d47b View commit details
    Browse the repository at this point in the history
  4. Modify go dependencies and code to be compatible with go1.15

    - downgrade viper dependency to be compatible
    - remove any native golang calls introduced in 16+
    - update version check for cli compilation script
    - add license check skip flag in tarball build
    			pr-link: Alluxio#18218
    			change-id: cid-bc73e4d4a8c6c78a189f213dbb11ddd4b77e5454
    Xenorith authored Sep 27, 2023
    Configuration menu
    Copy the full SHA
    b5e7b87 View commit details
    Browse the repository at this point in the history

Commits on Oct 2, 2023

  1. Add user/password support for etcd

    ### What changes are proposed in this pull request?
    
    Allow AlluxioEtcdClient to use username/password to talk to etcd if target etcd server has authentication enabled.
    
    ### Why are the changes needed?
    
    as above.
    
    ### Does this PR introduce any user facing changes?
    
    N/A
    
    			pr-link: Alluxio#18193
    			change-id: cid-4d2644b1bf1d600a8ff4dcffa18cbbfa8f0f6ad0
    lucyge2022 authored Oct 2, 2023
    Configuration menu
    Copy the full SHA
    4777915 View commit details
    Browse the repository at this point in the history

Commits on Oct 4, 2023

  1. Add membership module doc

    ### What changes are proposed in this pull request?
    
    add membership module doc
    
    ### Why are the changes needed?
    
    N/A
    
    ### Does this PR introduce any user facing changes?
    
    N/A
    
    			pr-link: Alluxio#18233
    			change-id: cid-ffea2ab2522a5c374e373f9e1c73dce36a5f38eb
    lucyge2022 authored Oct 4, 2023
    Configuration menu
    Copy the full SHA
    063a11a View commit details
    Browse the repository at this point in the history
  2. Fail directly if sudo needs password

    ### What changes are proposed in this pull request?
    
    Fail directly if sudo needs password.
    
    ### Why are the changes needed?
    
    Fix Alluxio#18194
    
    ### Does this PR introduce any user facing changes?
    
    NO
    
    			pr-link: Alluxio#18219
    			change-id: cid-ad2566d5d69cdbe77dcb35302c5268c9ead4f848
    secfree authored Oct 4, 2023
    Configuration menu
    Copy the full SHA
    c394d65 View commit details
    Browse the repository at this point in the history

Commits on Oct 7, 2023

  1. Fix FileOutStreamIntegrationTest

    Removed an unsupported test.
    Added new test for `FileOutStream.write(int)`
    Fixed `AbstractFileOutIntegrationTest`
    
    			pr-link: Alluxio#18189
    			change-id: cid-a0a3c2ab1f1eee9bc24240e695082ef17badeff0
    voddle authored Oct 7, 2023
    Configuration menu
    Copy the full SHA
    9a88be2 View commit details
    Browse the repository at this point in the history
  2. Add UT to DefaultPageMetaStore.commitFile

    ### What changes are proposed in this pull request?
    
    Add unit test for DefaultPageMetaStore.commitFile() method, and rename some variables to more accurate names. 
    
    ### Why are the changes needed?
    
    Improve the test coverage.
    
    ### Does this PR introduce any user facing changes?
    
    No.
    			pr-link: Alluxio#18162
    			change-id: cid-e7de0d5b474d36b67c85f0ed0a49202a37dc7194
    YichuanSun authored Oct 7, 2023
    Configuration menu
    Copy the full SHA
    95046f7 View commit details
    Browse the repository at this point in the history

Commits on Oct 8, 2023

  1. Improve list files RESTful API providing more useful info

    Improve list files RESTful API providing more useful info. This PR add the following info to the response JSON:
    1. Formatted file size
    2. Alluxio path
    3. UFS path
    4. Last modification timestamp
    			pr-link: Alluxio#18245
    			change-id: cid-b7260c0ae964b31cfe67eead92f693b40b6688fc
    JiamingMai authored Oct 8, 2023
    Configuration menu
    Copy the full SHA
    28db253 View commit details
    Browse the repository at this point in the history
  2. Fix the UfsBaseFileSystemTest

    ### What changes are proposed in this pull request?
    
    The old UfsBaseFileSystem cannot run, this pr fix it.
    
    ### Why are the changes needed?
    
    Improve code quality.
    
    ### Does this PR introduce any user facing changes?
    
    no.
    
    			pr-link: Alluxio#18163
    			change-id: cid-7b19138cb9db0d8b02613d96af9e6f33bc807b5d
    YichuanSun authored Oct 8, 2023
    Configuration menu
    Copy the full SHA
    d4a39b5 View commit details
    Browse the repository at this point in the history
  3. Disable UFS fallback in DoraFileSystemIntegrationTest

    ### What changes are proposed in this pull request?
    
    Set PropertyKey.DORA_CLIENT_UFS_FALLBACK_ENABLED to false.
    
    ### Why are the changes needed?
    
    As we mentioned in [here](Alluxio#18213).
    When we want to test the DoraFileSystem, we have to disable the Ufs Fallback, otherwise the methods like `createFile`  `getStatus` will fall back to Ufs, while we want to test DoraFileSystem.
    
    **This is a important change, so it is necessary to open a new PR for it.**
    
    ### Does this PR introduce any user facing changes?
    
    No.
    
    			pr-link: Alluxio#18244
    			change-id: cid-d81c5b5298fb9e7eb98fc5edde236f6fda05e0f0
    YichuanSun authored Oct 8, 2023
    Configuration menu
    Copy the full SHA
    8329244 View commit details
    Browse the repository at this point in the history

Commits on Oct 9, 2023

  1. Fix Integration tests framework and DoraFileSystemIntegrationTest

    ### What changes are proposed in this pull request?
    
    The LocalAlluxioCluster used in integration tests is broken when Ufs fallback is disable, this PR is used to fix it. The old master and worker liveness detection mechanism is invalid in ALluxio 3.0 arch, this PR also fix it.
    
    ### Why are the changes needed?
    
    Fix broken tests.
    
    ### Does this PR introduce any user facing changes?
    
    No.
    
    			pr-link: Alluxio#18213
    			change-id: cid-6e7d3cf73c80aa45f67007f90605a78654924614
    YichuanSun authored Oct 9, 2023
    Configuration menu
    Copy the full SHA
    cb74679 View commit details
    Browse the repository at this point in the history
  2. Fix reconnect logic of servicediscoveryrecipe

    ### What changes are proposed in this pull request?
    
    when etcd lease keepalive client returned with onCompleted or onError, the lease might not directly get expired at the time, therefore we always new a lease to resume keepalive without explicitly checking isleaseexpired
    
    ### Why are the changes needed?
    
    when etcd got induced network unstableness such as pkg loss, there's race condition in our reconnection logic from ServiceDiscoveryRecipe, where we directly check isLeaseExpired when we got terminal callback from keepalive client, but at the time the lease isn't expired yet. Hence we stop trying to create new lease to keepalive.
    
    ### Does this PR introduce any user facing changes?
    
    No
    
    			pr-link: Alluxio#18217
    			change-id: cid-bdfc1d37b359080585ac92b5daa9fac31598ea9b
    lucyge2022 authored Oct 9, 2023
    Configuration menu
    Copy the full SHA
    1a4b2a9 View commit details
    Browse the repository at this point in the history
  3. Avoid creating FileSystem instance many times in HttpServerHandler

    Avoid creating `FileSystem` instance many times in `HttpServerHandler`, and call the `close()` method when the handler is removed.
    			pr-link: Alluxio#18248
    			change-id: cid-9c4f1369a9549d58b8191214547046cd1e188d48
    JiamingMai authored Oct 9, 2023
    Configuration menu
    Copy the full SHA
    ceeeefe View commit details
    Browse the repository at this point in the history

Commits on Oct 10, 2023

  1. Remove MigrateIntegrationTest and PersistIntegrationTest

    ### What changes are proposed in this pull request?
    
    Remove MigrateIntegrationTest and PersistIntegrationTest.
    
    ### Why are the changes needed?
    
    MigrateIntegrationTest and PersistIntegrationTest are related to the job service, which no longer exists in the Dora architecture. Therefore, MigrateIntegrationTest and PersistIntegrationTest can be removed.
    
    ### Does this PR introduce any user facing changes?
    
    
    			pr-link: Alluxio#18247
    			change-id: cid-f38d5660f6b7c91b8407e0740aec04d2cccb8b89
    jasondrogba authored Oct 10, 2023
    Configuration menu
    Copy the full SHA
    ee0f966 View commit details
    Browse the repository at this point in the history
  2. Fix FileSystemBlockLocationIntegrationTest

    ### What changes are proposed in this pull request?
    
    Fixed one test case in FileSystemBlockLocationIntegrationTest
    
    ### Why are the changes needed?
    
    Logic of getBlockLocation in DoraCacheFileSystem has chagned, so related IntegrationTest need be change as well
    
    ### Does this PR introduce any user facing changes?
    
    no
    			pr-link: Alluxio#18243
    			change-id: cid-39521af14dafcb91d5368a25091c1031651c6dc6
    voddle authored Oct 10, 2023
    Configuration menu
    Copy the full SHA
    447a444 View commit details
    Browse the repository at this point in the history
  3. Support set/get xattr

    ### What changes are proposed in this pull request?
    
    Supports setXattr API of Alluxio FileSystem, it will set the corresponding attributes to UFS. If UFS is a filesystem, it's set through the setXattr interface of UFS. if UFS is object storage, it's set through setTagging API.
    
    ### Why are the changes needed?
    Based on the 3.x architecture, recover support for the setXattr interface. There are still many places where the interface is used, and it should not be supported at present.
    
    
    			pr-link: Alluxio#18053
    			change-id: cid-4f8c96c88cced4374d54b6387bfc618b47d2a423
    Jackson-Wang-7 authored Oct 10, 2023
    Configuration menu
    Copy the full SHA
    bf997fe View commit details
    Browse the repository at this point in the history
  4. Remove a few deprecated integration tests

    ### What changes are proposed in this pull request?
    
    Remove BackupCommandStateLockingIntegrationTest, ReportEmbeddedJournalIntegrationTest, and 
     MetricsCommandIntegrationTest
    
    ### Why are the changes needed?
    
    Remove invalid tests.
    
    ### Does this PR introduce any user facing changes?
    
    no.
    
    			pr-link: Alluxio#18251
    			change-id: cid-fe71cc37517fc3ad1eb3f5dba4b05f5503afaa5e
    YichuanSun authored Oct 10, 2023
    Configuration menu
    Copy the full SHA
    75e31dc View commit details
    Browse the repository at this point in the history
  5. Support StressWorkerBench using consistent hash policy

    ### What changes are proposed in this pull request?
    This PR adds a `--mode` option for StressWorkerBench. This option allows user to choose from a range of file read policies. Possible option values are `hash`(default) and `local-only`.
    
    If use `hash`, the config `alluxio.client.file.dora.ConsistentHashPolicy` is set to `alluxio.client.file.dora.ConsistentHashPolicy`.
    
    If use `local-only`, that config is set to `alluxio.client.file.dora.LocalWorkerPolicy`.
    
    The benchmark will parse this option, and choose the right policy. Also, it will print a log about the policy the user is using in both the benchmark and the policy factory.
    
    ### Why are the changes needed?
    In previous versions, the default policy is `local-only`.
    
    However, in Alluxio, the default policy is `hash`, using `alluxio.client.file.dora.ConsistentHashPolicy`.
    
    If user wants to use the `local-only` policy, add `--mode local-only` to the end of the benchmark command.
    			pr-link: Alluxio#18246
    			change-id: cid-29676a12710b9c815f3e55f84eb5e4226fea9ad7
    twalluxio authored Oct 10, 2023
    Configuration menu
    Copy the full SHA
    3470e17 View commit details
    Browse the repository at this point in the history
  6. Fix FileInStreamIntegrationTest

    Fix `FileInStreamIntegrationTest`, removed unnecessary test cases
    			pr-link: Alluxio#18178
    			change-id: cid-dac1c4c2e4e403bbf4ad3cbdde9eaf43ce20e046
    voddle authored Oct 10, 2023
    Configuration menu
    Copy the full SHA
    ba61600 View commit details
    Browse the repository at this point in the history
  7. Add a property key to control if alluxio write the xattr to UFS

    ### What changes are proposed in this pull request?
    
    Add a property key to control if alluxio write the xattr to UFS
    
    ### Why are the changes needed?
    The current S3 mock doesn't support setTagging/getTagging. And in some cases, we don't want to write xAttr to UFS. So Add a switch to disable it.
    
    ### Does this PR introduce any user facing changes?
    
    			pr-link: Alluxio#18255
    			change-id: cid-ccc2971f9bbb5cdd5abf31dc9fd845a1f26097b5
    Jackson-Wang-7 authored Oct 10, 2023
    Configuration menu
    Copy the full SHA
    b7a0a77 View commit details
    Browse the repository at this point in the history
  8. Add UT of PagedFileReader

    ### What changes are proposed in this pull request?
    
    Fix a bug of PagedFileReader. The PagedFileReader.read() method should use mFileSize, instead of mPos.
    
    Then add unit tests of PagedFileReader.
    
    ### Why are the changes needed?
    
    improve code coverage
    
    ### Does this PR introduce any user facing changes?
    
    no.
    
    ![image](https://github.com/Alluxio/alluxio/assets/30886176/49f3d75b-2b9d-49e7-a089-b19ba027c7d1)
    
    
    
    			pr-link: Alluxio#18107
    			change-id: cid-0b929b7a1a8c9e32bb53be861e41f12b64d16183
    YichuanSun authored Oct 10, 2023
    Configuration menu
    Copy the full SHA
    a002643 View commit details
    Browse the repository at this point in the history
  9. Use linked hash set to keep sequence when picking multiple worker

    ### What changes are proposed in this pull request?
    
    Use linked hash set to keep sequence when picking multiple worker
    
    ### Why are the changes needed?
    When loading multiple replica into Alluxio, we want deterministic sequence so we can decide which worker to load data and which worker the client can talk to.
    
    ### Does this PR introduce any user facing changes?
    
    na
    
    			pr-link: Alluxio#18235
    			change-id: cid-1193f29eda77cd82af8dea4bc59914e97d1f8fa3
    jja725 authored Oct 10, 2023
    Configuration menu
    Copy the full SHA
    6ede390 View commit details
    Browse the repository at this point in the history

Commits on Oct 11, 2023

  1. Add ser/de support for JSON or WorkerIdentity

    ### What changes are proposed in this pull request?
    
    Allow `WorkerIdentity` to be de/serialized to JSON and Java serialization format.
    
    ### Why are the changes needed?
    
    To be able to be embedded in `WorkerInfo` and other data objects that need to be de/serialized to JSON.
    
    ### Does this PR introduce any user facing changes?
    
    No.
    
    			pr-link: Alluxio#18224
    			change-id: cid-9bd20ac6b1934feee10d130633e27323750efd17
    dbw9580 authored Oct 11, 2023
    Configuration menu
    Copy the full SHA
    f7ee2ad View commit details
    Browse the repository at this point in the history
  2. Refactor handling of java opts env vars in cli

    address Alluxio#18249
    
    refactor how java opt env vars are handled. previously they were treating as strings and then using a split operation to separate the single string into cmd line arguments. now they are handled as `[]string` immediately after parsing from env to properly handle any leading or trailing whitespaces.
    			pr-link: Alluxio#18252
    			change-id: cid-44c4e14465432b0a5d9d8e84b488e9cd13c1e11b
    Xenorith authored Oct 11, 2023
    Configuration menu
    Copy the full SHA
    9ef97ec View commit details
    Browse the repository at this point in the history

Commits on Oct 13, 2023

  1. Use new worker id on worker and client

    ### What changes are proposed in this pull request?
    
    Use new worker identity for worker registration and client-side cluster management.
    
    ### Why are the changes needed?
    
    To allow workers to use generated IDs that are not assigned by the master.
    To allow clients to build its view of the cluster using more efficient and reliable worker identity implementation.
    
    ### Does this PR introduce any user facing changes?
    
    The worker identity is now persisted in etcd, if the worker uses etcd-based registration.
    
    			pr-link: Alluxio#18210
    			change-id: cid-b044e5648329715bd59172a6fd39b406ef1cdad3
    dbw9580 authored Oct 13, 2023
    Configuration menu
    Copy the full SHA
    ba20fd0 View commit details
    Browse the repository at this point in the history
  2. Check load meta option at worker

    ### What changes are proposed in this pull request?
    
    Check the `-f, --load-metadata` option at the worker side.
    
    ### Why are the changes needed?
    
    The [doc](https://docs.alluxio.io/os/user/edge/en/operation/User-CLI.html#fs-ls) has
    
    ![2023-10-03_12-03-58](https://github.com/Alluxio/alluxio/assets/7149512/e839b2e5-ef2a-4834-b421-b414353572f4)
    
    and the "fs ls" command has the following option
    
    ```
     -f, --load-metadata       Force load metadata for immediate children in a directory
    ```
    
    But this option is only set at the client side and is not checked at the worker side. So setting it has no use. Please check the following example
    
    ```
    $ ./bin/alluxio fs ls /
    -rw-rw-r--  test test              1366                 10-02-2023 17:39:54:284 FILE /file.01
    -rw-rw-r--  test test              1366                 10-02-2023 17:41:37:496 FILE /file.02
    -rw-rw-r--  test test              1366                 10-03-2023 11:04:18:177 FILE /file.03
    
    $ ./bin/alluxio fs ls -f /
    -rw-rw-r--   test test             1366                 10-02-2023 17:39:54:284 FILE /file.01
    -rw-rw-r--   test test             1366                 10-02-2023 17:41:37:496 FILE /file.02
    -rw-rw-r--   test test             1366                 10-03-2023 11:04:18:177 FILE /file.03
    
    $ ./bin/alluxio fs ls -Dalluxio.user.file.metadata.sync.interval=0 /
    -rw-rw-r--   test test             1366                 10-02-2023 17:39:54:284 FILE /file.01
    -rw-rw-r--   test test             1366                 10-02-2023 17:41:37:496 FILE /file.02
    -rw-rw-r--   test test             1366                 10-03-2023 11:04:18:177 FILE /file.03
    -rw-rw-r--   test test             1366                 10-03-2023 11:09:33:642 FILE /file.04
    -rw-rw-r--   test test             1366                 10-03-2023 11:12:57:668 FILE /file.05
    ```
    
    This PR checks it at the worker side and makes it matches the doc.
    
    ### Does this PR introduce any user facing changes?
    
    NO
    
    			pr-link: Alluxio#18238
    			change-id: cid-d8e77340861e2b78e8908012b4fe262421202909
    secfree authored Oct 13, 2023
    Configuration menu
    Copy the full SHA
    20965a0 View commit details
    Browse the repository at this point in the history

Commits on Oct 16, 2023

  1. [SMALLFIX] Enable WorkerMetadataSyncIntegrationTest

    ### What changes are proposed in this pull request?
    
    Remove the @ignore annotations. It works now.
    
    ### Why are the changes needed?
    
    improve code quality.
    
    ### Does this PR introduce any user facing changes?
    
    no.
    
    			pr-link: Alluxio#18264
    			change-id: cid-8124098803c0b05968d67b5ef5c31eed18b69187
    YichuanSun authored Oct 16, 2023
    Configuration menu
    Copy the full SHA
    d8a12c7 View commit details
    Browse the repository at this point in the history
  2. [SMALLFIX] Enable one UT case in OutStreamTest

    ### What changes are proposed in this pull request?
    
    A comment in OutStreamTest is useless and need to be removed.
    
    ### Why are the changes needed?
    
    Improve code quality.
    
    ### Does this PR introduce any user facing changes?
    
    No.
    
    			pr-link: Alluxio#18272
    			change-id: cid-602cf0523ed6fbedc30ff448dfb4682006599baa
    YichuanSun authored Oct 16, 2023
    Configuration menu
    Copy the full SHA
    fae6922 View commit details
    Browse the repository at this point in the history
  3. Fix URIStatus notserializable bug

    ### What changes are proposed in this pull request?
    
    fix object notserializable bug
    
    ### Why are the changes needed?
    
    about issue: Alluxio#18265
    
    ### Does this PR introduce any user facing changes?
    
    Please list the user-facing changes introduced by your change, including
      1. change in user-facing APIs
      2. addition or removal of property keys
      3. webui
    
    			pr-link: Alluxio#18266
    			change-id: cid-6c74a4a16b470de4a8d7f5a36d2ca87f11d7bb56
    singer-bin authored Oct 16, 2023
    Configuration menu
    Copy the full SHA
    8e169df View commit details
    Browse the repository at this point in the history

Commits on Oct 17, 2023

  1. Add RESTful API for distributed load

    Add RESTful API for distributed load.
    
    ### Usage:
    **SUBMIT:**
    description: submit a load job
    example:
    `http://localhost:28080/v1/load?path=/&opType=submit&partialListing=false&&verify=true&bandwidth=1000&loadMetadataOnly=false&verbose=true&skipIfExists=true`
    
    **STOP:**
    description: stop the load job
    example:
    http://localhost:28080/v1/load?path=/&opType=stop
    
    **PROGRESS:**
    description: get the progress of the load job
    example:
    `http://localhost:28080/v1/load?path=/&opType=progress&progressFormat=text&verbose=true`
    			pr-link: Alluxio#18254
    			change-id: cid-8a2e4f6747d7ba6cb8c25032cc13ce4ec719da8f
    JiamingMai authored Oct 17, 2023
    Configuration menu
    Copy the full SHA
    7c3b462 View commit details
    Browse the repository at this point in the history
  2. Change CosNUnderFileSystemFactory implementation UnderFileSystemFactory

    Cherry-pick of existing commit.
    orig-pr: Alluxio#18143
    orig-commit: Alluxio/alluxio@057804e
    orig-commit-author: gp1314 <814085234@qq.com>
    
    			pr-link: Alluxio#18279
    			change-id: cid-4a80f6cdeae5b9bdb9e956c36838403ee6ce7c46
    alluxio-bot authored Oct 17, 2023
    Configuration menu
    Copy the full SHA
    b597c61 View commit details
    Browse the repository at this point in the history
  3. test force push to main

    apc999 committed Oct 17, 2023
    Configuration menu
    Copy the full SHA
    1bdb1dd View commit details
    Browse the repository at this point in the history
  4. Revert "Add RESTful API for distributed load"

    This reverts commit 7c3b462.
    
    ### What changes are proposed in this pull request?
    
    Please outline the changes and how this PR fixes the issue.
    
    ### Why are the changes needed?
    
    Please clarify why the changes are needed. For instance,
      1. If you propose a new API, clarify the use case for a new API.
      2. If you fix a bug, describe the bug.
    
    ### Does this PR introduce any user facing changes?
    
    Please list the user-facing changes introduced by your change, including
      1. change in user-facing APIs
      2. addition or removal of property keys
      3. webui
    
    			pr-link: Alluxio#18284
    			change-id: cid-ae7e13a05237624a55adba85d2204fa1332b608a
    jja725 authored Oct 17, 2023
    Configuration menu
    Copy the full SHA
    b660e74 View commit details
    Browse the repository at this point in the history

Commits on Oct 18, 2023

  1. Load big file

    ### What changes are proposed in this pull request?
    
    Please outline the changes and how this PR fixes the issue.
    
    ### Why are the changes needed?
    
    Please clarify why the changes are needed. For instance,
      1. If you propose a new API, clarify the use case for a new API.
      2. If you fix a bug, describe the bug.
    
    ### Does this PR introduce any user facing changes?
    
    Please list the user-facing changes introduced by your change, including
      1. change in user-facing APIs
      2. addition or removal of property keys
      3. webui
    
    			pr-link: Alluxio#18285
    			change-id: cid-675c904a992ef44cd62d1daa51d4672234093483
    jja725 authored Oct 18, 2023
    Configuration menu
    Copy the full SHA
    24a32d4 View commit details
    Browse the repository at this point in the history
  2. Remove test file

    ### What changes are proposed in this pull request?
    
    Remove a previously added file due to branch cleanup
    
    ### Why are the changes needed?
    
    Remove a previously added file due to branch cleanup
    
    ### Does this PR introduce any user facing changes?
    
    N/A
    			pr-link: Alluxio#18283
    			change-id: cid-e84a3275f32b33c427277ba1ed2f57f6ee4a9927
    apc999 authored Oct 18, 2023
    Configuration menu
    Copy the full SHA
    f3494dc View commit details
    Browse the repository at this point in the history

Commits on Oct 19, 2023

  1. Load ufs path directly

    ### What changes are proposed in this pull request?
    Load ufs path directly
    
    ### Why are the changes needed?
    Previously we concatenate root with relative path, now we directly pass ufs path So we can have multiple ufs support
    
    ### Does this PR introduce any user facing changes?
    load have to input full ufs path
    
    			pr-link: Alluxio#18289
    			change-id: cid-3d9c4bad7b68af142ca7985954418bfef0f34b0b
    jja725 authored Oct 19, 2023
    Configuration menu
    Copy the full SHA
    7d222bb View commit details
    Browse the repository at this point in the history
  2. Create locationCommandTest and fix some IntegretionTest bugs

    ### What changes are proposed in this pull request?
    
    I have created DoraFileLocationIntegretionTest and fix the bugs in IntegretionTest.
    
    ### Why are the changes needed?
    
    Because we just added the user command but donnot know whether they can be used correctly.
    
    ### Does this PR introduce any user facing changes?
    
    User can added other IntegretionTests due to the correct infra.
    
    			pr-link: Alluxio#18294
    			change-id: cid-7832b08a4a8bf6bffcb8eaba4bd18c1761f7ce65
    thu-david authored Oct 19, 2023
    Configuration menu
    Copy the full SHA
    b0dc312 View commit details
    Browse the repository at this point in the history
  3. Fix load command ufs string

    ### What changes are proposed in this pull request?
    
    Fix load command ufs string
    
    ### Why are the changes needed?
    bug fix
    
    ### Does this PR introduce any user facing changes?
    na
    
    			pr-link: Alluxio#18297
    			change-id: cid-200e7eaf2ea7206725abbd802a2f2f8d9afc897e
    jja725 authored Oct 19, 2023
    Configuration menu
    Copy the full SHA
    cd7f2e1 View commit details
    Browse the repository at this point in the history
  4. Fix FileSystemShellUtilsTest

    ### What changes are proposed in this pull request?
    
    The loadCommands of FileSystemShellUtilsTest now works.
    
    ### Why are the changes needed?
    
    Improve code quality.
    
    ### Does this PR introduce any user facing changes?
    
    No.
    
    			pr-link: Alluxio#18278
    			change-id: cid-3697d62cd123bdf832dece2b4b844f05133258a5
    YichuanSun authored Oct 19, 2023
    Configuration menu
    Copy the full SHA
    077c372 View commit details
    Browse the repository at this point in the history
  5. [SMALLFIX] Remove OldLoadCommandIntegrationTest

    ### What changes are proposed in this pull request?
    
    Remove invalid integration test OldLoadCommandIntegrationTest.
    
    ### Why are the changes needed?
    
    Improve code quality.
    
    ### Does this PR introduce any user facing changes?
    
    No.
    			pr-link: Alluxio#18290
    			change-id: cid-c86a3cb9e11971b6015af19a895009f1ff9f07eb
    YichuanSun authored Oct 19, 2023
    Configuration menu
    Copy the full SHA
    b5616c0 View commit details
    Browse the repository at this point in the history
  6. [SMALLFIX] Delete JobServiceFaultToleranceShellTest

    ### What changes are proposed in this pull request?
    
    Delete JobServiceFaultToleranceShellTest, it makes no sense now.
    
    ### Why are the changes needed?
    
    Improve code quality.
    
    ### Does this PR introduce any user facing changes?
    
    No.
    
    			pr-link: Alluxio#18280
    			change-id: cid-ae9f816c9219544c2b1c2c97b6fa33429d6835ad
    YichuanSun authored Oct 19, 2023
    Configuration menu
    Copy the full SHA
    fe98953 View commit details
    Browse the repository at this point in the history
  7. Add name to artifacts metadata

    ### What changes are proposed in this pull request?
    add name to metadata
    
    ### Why are the changes needed?
    Needed to identify docker artifacts
    
    ### Does this PR introduce any user facing changes?
    no
    
    			pr-link: Alluxio#18302
    			change-id: cid-014522dd2772b44b9fe7a4d703c9e86d827ab44b
    tieujason330 authored Oct 19, 2023
    Configuration menu
    Copy the full SHA
    908b7d6 View commit details
    Browse the repository at this point in the history

Commits on Oct 20, 2023

  1. Implement HDFS list status iterator

    When there are a lot of files in HDFS, it takes a large amount of time and memory to complete a `listStatus` request. Moreover, sometimes OOM occurs. This PR provides an iterator for the HDFS under file system to list files.
    			pr-link: Alluxio#18295
    			change-id: cid-11019e8f163210c7664f3f2b6ddf3bae27e8ee8c
    JiamingMai authored Oct 20, 2023
    Configuration menu
    Copy the full SHA
    62cc17a View commit details
    Browse the repository at this point in the history
  2. Add RESTful API for distributed load

    Add RESTful API for distributed load.
    
    ### Usage:
    **SUBMIT:**
    description: submit a load job
    example:
    `http://localhost:28080/v1/load?path=s3a://jiamingmai-test/&opType=submit&partialListing=false&&verify=true&bandwidth=1000&loadMetadataOnly=false&verbose=true&skipIfExists=true`
    
    **STOP:**
    description: stop the load job
    example:
    `http://localhost:28080/v1/load?path=s3a://jiamingmai-test/&opType=stop`
    
    **PROGRESS:**
    description: get the progress of the load job
    example:
    `http://localhost:28080/v1/load?path=/&opType=progress&progressFormat=text&verbose=true`
    			pr-link: Alluxio#18291
    			change-id: cid-3a4f1e6bb66dd6e6f518aaeb9e26bf17eda698de
    JiamingMai authored Oct 20, 2023
    Configuration menu
    Copy the full SHA
    47b7667 View commit details
    Browse the repository at this point in the history
  3. Add multi-dimensional metrics system

    ### What changes are proposed in this pull request?
    Introduce a new tool to add multi-dimensional metrics more easily. The new class holds all the metrics and exposes them in Prometheus type.
    
    ### Why are the changes needed?
    Add following metrics:
    - alluxio_data_access
    - alluxio_ufs_data_access
    - alluxio_cached_data_read
    - alluxio_meta_operation
    - alluxio_cached_storage
    - alluxio_cached_capacity
    - alluxio_cached_evicted_data
    - alluxio_cached_readable_storage
    
    ### Does this PR introduce any user facing changes?
    No. The new metrics will expose on a new URL which the webui is not currently using.
    
    			pr-link: Alluxio#18307
    			change-id: cid-ed60de4c62f089c9ef6f95a99efc1bd383241ee0
    Kai-Zhang authored Oct 20, 2023
    Configuration menu
    Copy the full SHA
    3c851d7 View commit details
    Browse the repository at this point in the history

Commits on Oct 23, 2023

  1. Change UfsManager key to not rely on properties

    ### What changes are proposed in this pull request?
    
    In the UfsManager the UFS properties are not a prerequisite for locating an UFS instance. By not hashing on properties, the `get()` does not need to resolve the target UFS property keys, which saves cost.
    
    Before: 15.32% CPU in a worker stress test
    ![image](https://github.com/Alluxio/alluxio/assets/14806853/53039841-d133-4a10-99f8-f73b8458a746)
    
    After: <1% CPU in the same worker stress test
    ![image](https://github.com/Alluxio/alluxio/assets/14806853/f8233952-b4e8-4da1-be6d-f8518ad223b9)
    
    
    			pr-link: Alluxio#18182
    			change-id: cid-7b48a9bd6d8446efe9f22d699462c25a9550683a
    jiacheliu3 authored Oct 23, 2023
    Configuration menu
    Copy the full SHA
    013c9e1 View commit details
    Browse the repository at this point in the history
  2. Use new UfsManager.get() avoid creating conf instances every time

    ### What changes are proposed in this pull request?
    
    PR Alluxio#18182 was merged with changes conflicting to Alluxio#18285 which CI was not able to catch. So `main` is not able to compile after merging `18182`. This small change fixes that, by adapting new code in 18285 to interface changes in 18182.
    
    			pr-link: Alluxio#18315
    			change-id: cid-2b5a7fe4b88b9ea4fe47b76e8cc0eed007120e48
    jiacheliu3 authored Oct 23, 2023
    Configuration menu
    Copy the full SHA
    3a47fa2 View commit details
    Browse the repository at this point in the history
  3. [SMALLFIX] Remove UnsetTtlTest

    ### What changes are proposed in this pull request?
    
    Remove UnsetTtlTest which is outdated now.
    
    ### Why are the changes needed?
    
    Improve code quality.
    
    ### Does this PR introduce any user facing changes?
    
    No.
    
    			pr-link: Alluxio#18292
    			change-id: cid-3de116636374ad12962793e7ea19ba1170870db7
    YichuanSun authored Oct 23, 2023
    Configuration menu
    Copy the full SHA
    6bfbbd6 View commit details
    Browse the repository at this point in the history
  4. Remove restriction in overwriting worker info within join logic

    ### What changes are proposed in this pull request?
    
    Now with worker id can be assumed from a different worker instance whether on a different pod in k8s or a different host machine for baremetal. The creation onto the persisted ring path : /DHT/DefaultAlluxioCluster/AUTHORIZED/ should not bail if a different value is seen.
    
    ### Why are the changes needed?
    
    to enable rejoin of a worker bearing same worker id but with different host or other WorkerInfo fileds.
    
    ### Does this PR introduce any user facing changes?
    
    No
    
    			pr-link: Alluxio#18275
    			change-id: cid-51322e010e0d51ae4f81268c2bb607b568f08c46
    lucyge2022 authored Oct 23, 2023
    Configuration menu
    Copy the full SHA
    0884fc3 View commit details
    Browse the repository at this point in the history

Commits on Oct 24, 2023

  1. Improve cold read performance by prefetching data on workers

    ### What changes are proposed in this pull request?
    
    In the position reader, preloading pages on workers to improve the cold read performance.
    
    ### Why are the changes needed?
    
    To improve the cold read performance.
    
    ### Does this PR introduce any user facing changes?
    
    N/A
    			pr-link: Alluxio#18317
    			change-id: cid-97e28711cf7f7b3ce60da346c737505a365d3238
    elega authored Oct 24, 2023
    Configuration menu
    Copy the full SHA
    dc12c11 View commit details
    Browse the repository at this point in the history
  2. Add prefetch cache policy

    Origin @dbw9580 
    
    ### What changes are proposed in this pull request?
    
    Add an prefetch cache policy that does not reset the sliding window when a cache read misses. Fuse sometimes create read requests that are out of the order and this helps the prefetch keep stable.
    
    
    			pr-link: Alluxio#18318
    			change-id: cid-b0a6331fac06bb743724e8b68005e257b89aa64d
    elega authored Oct 24, 2023
    Configuration menu
    Copy the full SHA
    e78b66d View commit details
    Browse the repository at this point in the history

Commits on Oct 25, 2023

  1. Add getStatus RESTful API

    Add getStatus RESTful API.
    
    **Example:**
    Get the specified directory/file information by the following request:
    `curl -X GET http://localhost:28080/v1/info?path=/tpcds-data`
    
    The response JSON looks like:
    `[
      {
        "mType": "directory",
        "mName": "tpcds-data",
        "mPath": "/tpcds-data",
        "mUfsPath": "s3a://jiamingmai-test/tpcds-data",
        "mLastModificationTimeMs": 0,
        "mLength": 0,
        "mHumanReadableFileSize": "0B"
      }
    ]`
    
    <img width="597" alt="image" src="https://github.com/Alluxio/alluxio/assets/6129818/b38d644b-11da-4206-937b-6d61fd6b3a6c">
    
    			pr-link: Alluxio#18312
    			change-id: cid-6e563a21372e9fe1867a36d9b311e246b49c459e
    JiamingMai authored Oct 25, 2023
    Configuration menu
    Copy the full SHA
    5b0e661 View commit details
    Browse the repository at this point in the history
  2. Fix MemoryPageStore error on commit() and clear cache on close()

    ### What changes are proposed in this pull request?
    
    This change mainly adds two fixes to the `MemoryPageStore`:
    1. Add a noop implementation to commit() so it does nothing instead of throwing `UnsupportedOperationException`
    2. Clears the cache on close explicitly
    
    			pr-link: Alluxio#18322
    			change-id: cid-5e7656e1eca363ed2de40cd48025f464c41a2584
    jiacheliu3 authored Oct 25, 2023
    Configuration menu
    Copy the full SHA
    08b9456 View commit details
    Browse the repository at this point in the history
  3. Bring back integration tests

    ### What changes are proposed in this pull request?
    
    All integration tests do not run now, this PR fixs it.
    
    ### Why are the changes needed?
    
    Fix bug.
    
    ### Does this PR introduce any user facing changes?
    
    No.
    			pr-link: Alluxio#18313
    			change-id: cid-e3f71f5b817624c45b8a3fc2baae2b08b99cd109
    YichuanSun authored Oct 25, 2023
    Configuration menu
    Copy the full SHA
    e830663 View commit details
    Browse the repository at this point in the history

Commits on Oct 26, 2023

  1. Remove WorkerFuseIntegrationTest

    ### What changes are proposed in this pull request?
    
    The `WORKER_FUSE_ENABLED ` key is not used, so this test is outdated.
    
    ### Why are the changes needed?
    
    Improve code quality.
    
    ### Does this PR introduce any user facing changes?
    no.
    
    			pr-link: Alluxio#18268
    			change-id: cid-8acc857afc5fff384d65bff748ad1c6b07a13e87
    YichuanSun authored Oct 26, 2023
    Configuration menu
    Copy the full SHA
    bb8e2cf View commit details
    Browse the repository at this point in the history
  2. Delete useless UnderFileSystemAlluxioTest

    ### What changes are proposed in this pull request?
    
    Delete invalid UnderFileSystemAlluxioTest for changed codebase.
    
    ### Why are the changes needed?
    
    Improve code quality.
    
    ### Does this PR introduce any user facing changes?
    
    No.
    			pr-link: Alluxio#18277
    			change-id: cid-8814028d7725c927c7973917f1da6d22e1f827db
    YichuanSun authored Oct 26, 2023
    Configuration menu
    Copy the full SHA
    6185ba0 View commit details
    Browse the repository at this point in the history
  3. Update worker api support for load multi replicas

    ### What changes are proposed in this pull request?
    Update worker api support for load multi replicas
    
    ### Why are the changes needed?
    part of PR to support load multi replicas
    
    ### Does this PR introduce any user facing changes?
    na
    
    			pr-link: Alluxio#18296
    			change-id: cid-0213f2aba669b7687ac42cf932cdcec911d397a4
    jja725 authored Oct 26, 2023
    Configuration menu
    Copy the full SHA
    7a5734f View commit details
    Browse the repository at this point in the history
  4. Add Rust toolchain to CI

    ### What changes are proposed in this pull request?
    
    Add Rust toolchain to the Docker image used for CI.
    
    ### Why are the changes needed?
    
    Allow CI to compile and run Rust code.
    
    ### Does this PR introduce any user facing changes?
    
    No.
    
    			pr-link: Alluxio#18319
    			change-id: cid-6912fe3659bb0044a6980d9ac2f32f79101efc44
    dbw9580 authored Oct 26, 2023
    Configuration menu
    Copy the full SHA
    53c49f7 View commit details
    Browse the repository at this point in the history
  5. Add regx pattern file filter for distributed load

    Add regx pattern file filter for distributed load.
    
    **Example:**
    The following request allows us to load the files under `/test-load` directory with "hello" prefix:
    `curl -X GET http://localhost:28080/v1/load?path=s3a://jiamingmai-test/test-load&opType=submit&verbose=true&fileFilterRegx=^hello.*`
    			pr-link: Alluxio#18311
    			change-id: cid-4ec2bfe58bfba413f6d2925f5b3937bd6f5c2eb1
    JiamingMai authored Oct 26, 2023
    Configuration menu
    Copy the full SHA
    d0ad98f View commit details
    Browse the repository at this point in the history
  6. Create DoraLsCommandIntegrationTest

    ### What changes are proposed in this pull request?
    
    I have created a new file named DoraLsCommandIntegrationTest and it can test the 'ls' command whether it can correctly run.Also, I revised the base class that can create byte files in alluxio.What's more, I have added a new test named DoraMkdirCommandIntegrationTest which it can test cli 'mkdir'.
     
    ### Why are the changes needed?
    
    1、New Test does not have a function that can create file in alluxio.
    2、Add a IntegrationTest.
    
    ### Does this PR introduce any user facing changes?
    
    No.
    
    			pr-link: Alluxio#18325
    			change-id: cid-067ec742deab39af294d089dec932b69c0362682
    thu-david authored Oct 26, 2023
    Configuration menu
    Copy the full SHA
    8f41c65 View commit details
    Browse the repository at this point in the history

Commits on Oct 27, 2023

  1. [SMALLFIX] Extract create UfsBaseFileSystem for extension

    ### What changes are proposed in this pull request?
    
    Extract the logic to create `UfsBaseFileSystem` into `FileSystemContext`, for possible extension. This change is functionally a refactor that changes nothing.
    			pr-link: Alluxio#18333
    			change-id: cid-105be187f763f83680f64bd963646ce32eb58493
    jiacheliu3 authored Oct 27, 2023
    Configuration menu
    Copy the full SHA
    961ed17 View commit details
    Browse the repository at this point in the history
  2. Fix a buffer memory leak in NettyReadHandlerStateMachine

    ### What changes are proposed in this pull request?
    
    resolves Alluxio#18324
    
    Disclaimer: I might have monkey-typed this fix but I still do not know anything about buffer ref counting. This fix does NOT make me the owner of this state machine.
    
    			pr-link: Alluxio#18323
    			change-id: cid-eb5bde353c08d3d9bdd39da5b9caf13681bae495
    jiacheliu3 authored Oct 27, 2023
    Configuration menu
    Copy the full SHA
    14efca9 View commit details
    Browse the repository at this point in the history
  3. [SMALLFIX] Remove LeaderCommandIntegrationTest.java

    ### What changes are proposed in this pull request?
    
    Remove invalid LeaderCommandIntegrationTest.java, the command is deleted.
    
    ### Why are the changes needed?
    
    improve code quality.
    
    ### Does this PR introduce any user facing changes?
    
    no.
    
    			pr-link: Alluxio#18287
    			change-id: cid-43a00b7ec27735262bfa45b7b37331389d82a881
    YichuanSun authored Oct 27, 2023
    Configuration menu
    Copy the full SHA
    d3bfd8c View commit details
    Browse the repository at this point in the history
  4. [SMALLFIX] Remove NeedsSyncCommandIntegrationTest.java

    ### What changes are proposed in this pull request?
    Remove NeedsSyncCommandIntegrationTest, which command is deleted.
    
    ### Why are the changes needed?
    
    Improve code quality.
    
    ### Does this PR introduce any user facing changes?
    
    No.
    
    			pr-link: Alluxio#18288
    			change-id: cid-0c6040e1804065067032384e6046c0d47ccf8312
    YichuanSun authored Oct 27, 2023
    Configuration menu
    Copy the full SHA
    779ece2 View commit details
    Browse the repository at this point in the history
  5. [SMALLFIX] Recover HelpCommandIntegrationTest

    ### What changes are proposed in this pull request?
    
    HelpCommandIntegrationTest works now.
    
    ### Why are the changes needed?
    
    Improve code quality.
    
    ### Does this PR introduce any user facing changes?
    
    No.
    
    			pr-link: Alluxio#18293
    			change-id: cid-ccb04b9ede68fae2342d6a25e443b85ccd8f990c
    YichuanSun authored Oct 27, 2023
    Configuration menu
    Copy the full SHA
    37b33dc View commit details
    Browse the repository at this point in the history

Commits on Oct 30, 2023

  1. Add rust spdk library

    ### What changes are proposed in this pull request?
    
    Add rust spdk library and design structures for cache.
    
    ### Why are the changes needed?
    
    For NVMe SSD cache requirement.
    
    ### Does this PR introduce any user facing changes?
    
    No.
    
    ### Benchmark Result
    
    <table style="text-align:center;">
    <tbody>
      <tr>
        <th rowspan="2">block size</th>
        <th colspan="3">time consumption</th>
        <th rowspan="2">throughput</th>
      </tr>
      <tr>
        <th>millisecond</th>
        <th>microsecond</th>
        <th>nanosecond</th>
      </tr>
      <tr>
        <td>512B</td>
        <td>10ms</td>
        <td>10194us</td>
        <td>10194421ns</td>
        <td>0.048MB/s</td>
      </tr>
      <tr>
        <td>1KB</td>
        <td>13ms</td>
        <td>13472us</td>
        <td>13472304ns</td>
        <td>0.072MB/s</td>
      </tr>
      <tr>
        <td>4KB</td>
        <td>9ms</td>
        <td>9242us</td>
        <td>9242424ns</td>
        <td>0.423MB/s</td>
      </tr>
      <tr>
        <td>16KB</td>
        <td>8ms</td>
        <td>8585us</td>
        <td>8585361ns</td>
        <td>1.820MB/s</td>
      </tr>
      <tr>
        <td>64KB</td>
        <td>11ms</td>
        <td>11030us</td>
        <td>11030930ns</td>
        <td>5.666MB/s</td>
      </tr>
      <tr>
        <td>256KB</td>
        <td>15ms</td>
        <td>15962us</td>
        <td>15962353ns</td>
        <td>15.662MB/s</td>
      </tr>
      <tr>
        <td>1MB</td>
        <td>13ms</td>
        <td>13059us</td>
        <td>13059113ns</td>
        <td>76.575MB/s</td>
      </tr>
      <tr>
        <td><b>4MB</b></td>
        <td><b>28ms</b></td>
        <td><b>28930us</b></td>
        <td><b>28930274ns</b></td>
        <td><b>138.264MB/s</b></td>
      </tr>
      <tr>
        <td><b>16MB</b></td>
        <td><b>79ms</b></td>
        <td><b>79423us</b></td>
        <td><b>79423390ns</b></td>
        <td><b>201.452MB/s</b></td>
      </tr>
      <tr>
        <td>64MB</td>
        <td>308ms</td>
        <td>308856us</td>
        <td>308856745ns</td>
        <td>207.216MB/s</td>
      </tr>
      <tr>
        <td>256MB</td>
        <td>1218ms</td>
        <td>1218323us</td>
        <td>1218323252ns</td>
        <td>210.125MB/s</td>
      </tr>
      <tr>
        <td>1GB</td>
        <td>5056ms</td>
        <td>5056277us</td>
        <td>5056277683ns</td>
        <td>202.521MB/s</td>
      </tr>
    </tbody>
    </table>
    
    
    
    			pr-link: Alluxio#18231
    			change-id: cid-92ee56270bc5bb237ecc0df78c2974e1051bc543
    linzs148 authored Oct 30, 2023
    Configuration menu
    Copy the full SHA
    eecf2a5 View commit details
    Browse the repository at this point in the history
  2. Update Load replica job

    ### What changes are proposed in this pull request?
    
    Update load job to adopt multi replicas
    
    ### Why are the changes needed?
    part of PR to support loading multi replicas
    
    ### Does this PR introduce any user facing changes?
    new load option `replicas`
    
    			pr-link: Alluxio#18320
    			change-id: cid-7d01ca19a28faf4c7773cbf5c355dd6cf070728f
    jja725 authored Oct 30, 2023
    Configuration menu
    Copy the full SHA
    32f675f View commit details
    Browse the repository at this point in the history

Commits on Oct 31, 2023

  1. Fix the time check logic for judging stale client channels to be inac…

    ### What changes are proposed in this pull request?
    
    Fix bug involved by Alluxio#18332.
    Alter the time judgment logic for judging whether stale client channels are inactive. Using the LocaTime object cannot correctly judge whether a channel client is inactive, because a LocalTime plus or minus time offset only changes the hour, minute, second attribute value, and It will not affect the date, you actually need to use the LocalDateTime object instead.
    ### Why are the changes needed?
    
    Please clarify why the changes are needed. For instance,
    In the code, the LocaTime class is used to determine that a client channel is inactive. The LocalTime object adds or subtracts the time offset. It only changes the hour, minute and second attribute value and does not affect the date. In fact, you need to use the LocalDateTime object.  In other words, the three-day certification cycle judgment should be based on date and time, not just time.
    
    ### Does this PR introduce any user facing changes?
    
    Please list the user-facing changes introduced by your change, including
    None
    
    			pr-link: Alluxio#18340
    			change-id: cid-5b69e0c87d3bad8556ae27d491f3e0dc567378b9
    liuxiaohu9527 authored Oct 31, 2023
    Configuration menu
    Copy the full SHA
    660e00d View commit details
    Browse the repository at this point in the history
  2. Fix isAllPageCached

    ### What changes are proposed in this pull request?
    
    Fix a bug when file size < 1 page, alluxio considers all pages are cached no matter if it really caches it or not.
    
    ### Why are the changes needed?
    
    Bug fixing 
    
    ### Does this PR introduce any user facing changes?
    
    N/A
    			pr-link: Alluxio#18347
    			change-id: cid-da4bdce6615e4b1d9777a98ab335bf36e503d102
    elega authored Oct 31, 2023
    Configuration menu
    Copy the full SHA
    ec4868d View commit details
    Browse the repository at this point in the history

Commits on Nov 1, 2023

  1. Change the metrics registry and some metrics

    ### What changes are proposed in this pull request?
    Add some metrics and change the registry to expose the metrics more accurately.
    And add the call point of the capacity-related metrics.
    
    ### Why are the changes needed?
    Use default registry will expose all the metrics and some of them are not meaningful for all components.
    
    ### Does this PR introduce any user facing changes?
    no
    
    			pr-link: Alluxio#18350
    			change-id: cid-646ee1f3e41171d2147df29fccaf5a3476b66033
    Kai-Zhang authored Nov 1, 2023
    Configuration menu
    Copy the full SHA
    2114bea View commit details
    Browse the repository at this point in the history
  2. Move job related operations to DoraCacheFileSystem

    ### What changes are proposed in this pull request?
    
    Please outline the changes and how this PR fixes the issue.
    
    ### Why are the changes needed?
    
    Please clarify why the changes are needed. For instance,
      1. If you propose a new API, clarify the use case for a new API.
      2. If you fix a bug, describe the bug.
    
    ### Does this PR introduce any user facing changes?
    
    Please list the user-facing changes introduced by your change, including
      1. change in user-facing APIs
      2. addition or removal of property keys
      3. webui
    
    			pr-link: Alluxio#18352
    			change-id: cid-b5f5b695a1c32a65ff3b303cb8227889bee5d81c
    elega authored Nov 1, 2023
    Configuration menu
    Copy the full SHA
    ccf20e4 View commit details
    Browse the repository at this point in the history
  3. Refactor PagedDoraWorker by injecting MetaManager and UfsManager

    ### What changes are proposed in this pull request?
    
    1. If an object is created inside `PagedDoraWorker` constructor, extract that creation to before the constructor and use dependency injection to inject it to the worker object. This doesn't change any creation logic, just a refactor to better adapt to dependency injection flavor.
    2. There is a circular dependency between `MetaManager` and `PagedDoraWorker`. This change removes that cycle. Now we create one, then create the other. Before, we create one and in the construction, we let `this` ref escape and create the other. Some methods are either moved or changed to `static`.
    3. By adapting to dependency injection, we rely on `UfsManager` interface instead of `DoraUfsManager` implementation. Some method signatures are extracted to the interface level.
    4. A few other small refactors to get rid of some downcasts and variable scope changes. Reasons are attached in comments on this PR.
    
    ### Why are the changes needed?
    
    Improve code quality and extensibility.
    
    ### Does this PR introduce any user facing changes?
    
    No. All refactor changes are small and equivalent to existing code. So nothing should break.
    
    			pr-link: Alluxio#18181
    			change-id: cid-4f9e9bc770b12253188bb541dd456ef3cd889c2b
    jiacheliu3 authored Nov 1, 2023
    Configuration menu
    Copy the full SHA
    0e83207 View commit details
    Browse the repository at this point in the history

Commits on Nov 2, 2023

  1. Add remote only policy to StressWorkerBench

    ### What changes are proposed in this pull request?
    
    Add one `RemoteOnlyPolicy` implementation for testing. This is usable for reading all files from remote nodes, rather than the local node itself.
    
    Generally, this policy keeps a thread-safe list including all workers. When one thread reads, the round-robin list returns all available workers after a roulette (putting the first element in the list to the end) , and choose the first remote worker to read from.
    
    We also added available options of `--mode` in StressWorkerBench to use the new remote only policy.
    
    ### Why are the changes needed?
    
    The new policy is for internal testing where all test clients find the remote worker for IO. This policy should not be used in real deployments because if all clients find remote worker, overall throughput can be quite low due to bandwidth restrictions.
    
    ### Does this PR introduce any user facing changes?
    
    No, RemoteOnlyPolicy should only be used in internal testing
    			pr-link: Alluxio#18273
    			change-id: cid-ce534382a1ebd86230296475f4e2d3c6dd862033
    twalluxio authored Nov 2, 2023
    Configuration menu
    Copy the full SHA
    ad3baa1 View commit details
    Browse the repository at this point in the history
  2. Fix the bug that command line doesn't support regx file filter

    Fix the bug that command line doesn't support regx file filter
    			pr-link: Alluxio#18359
    			change-id: cid-c2ec9a5394a1ad776e31251a9c113f4115cb651d
    JiamingMai authored Nov 2, 2023
    Configuration menu
    Copy the full SHA
    b2df584 View commit details
    Browse the repository at this point in the history

Commits on Nov 3, 2023

  1. Fix page store TTL metrics

    beinan committed Nov 3, 2023
    Configuration menu
    Copy the full SHA
    d8342f7 View commit details
    Browse the repository at this point in the history
  2. Fix file mtime display when ufs mtime is not null

    ### What changes are proposed in this pull request?
    
    Alluxio#18357
    
    ### Why are the changes needed?
    
    mtime display error when ufs is hdfs
    ![企业微信截图_9ff2cbaf-4e24-4cbc-ae11-6a5ace7812f1](https://github.com/Alluxio/alluxio/assets/800042/606cbcb0-b07f-4f91-8919-b1fe1b9e6dcc)
    
    after this commit:
    
    ![企业微信截图_8c7ee012-851c-425e-a4c1-d22f0d39984b](https://github.com/Alluxio/alluxio/assets/800042/e05bfed5-c929-43c1-a633-6b6dee3f10ef)
    
    
    
    ### Does this PR introduce any user facing changes?
    
    Please list the user-facing changes introduced by your change, including
      1. change in user-facing APIs. no
      2. addition or removal of property keys. no
      3. webui. no
    
    			pr-link: Alluxio#18358
    			change-id: cid-54cc12a64383e77e5cd52b7ac89976714a93612b
    lfyzjck authored Nov 3, 2023
    Configuration menu
    Copy the full SHA
    24ff6b5 View commit details
    Browse the repository at this point in the history

Commits on Nov 6, 2023

  1. Disable xatter in PagedDoraWorker test

    ### What changes are proposed in this pull request?
    
    XATTR config added to PagedDoraWorkerTest
    
    ### Why are the changes needed?
    
    LocalUFS does not support xattr
    
    ### Does this PR introduce any user facing changes?
    
    no
    
    			pr-link: Alluxio#18339
    			change-id: cid-76ff63355ee62dea9514666cb8556dc84a83b8b2
    voddle authored Nov 6, 2023
    Configuration menu
    Copy the full SHA
    ea6af23 View commit details
    Browse the repository at this point in the history
  2. Fix UT readMultipleFiles in LocalCacheFileInStreamTest

    ### What changes are proposed in this pull request?
    
    Added a modification to the code in the LocalCacheFileInStreamTest class.Specifically, the code change addresses an issue with the readMultipleFiles unit test in the LocalCacheFileInStreamTest.
    The proposed change to fix this issue is to add the setUfsPath method within the generateURIStatus method and set it to be identical to the Path. This ensures that the UfsPath is properly set for each file, allowing the readMultipleFiles test to pass successfully.
    
    ### Why are the changes needed?
    
    The changes in this pull request are needed because they address a specific issue with the LocalCacheFileInStreamTest class's readMultipleFiles unit test.
    The readMultipleFiles unit test was failing because it relied on the correct calculation of FieldId in Dora, which is based on the UfsPath. However, in this test, the UfsPath for files was not being set and remained empty, leading to all files having the same FieldId after hash calculation.
    The absence of a proper UfsPath for files in the test class caused multiple files to share the same FieldId, causing them to read data from a single file instead of their respective files.
    To resolve this issue, the proposed solution is to set the UfsPath within the generateURIStatus method to be identical to the Path. This ensures that each file has a distinct and properly set UfsPath, allowing the readMultipleFiles test to pass without affecting other unit tests.
    
    ### Does this PR introduce any user facing changes?
    
    Please list the user-facing changes introduced by your change, including
      1. change in user-facing APIs
      2. addition or removal of property keys
      3. webui
    
    			pr-link: Alluxio#18168
    			change-id: cid-422d5b4eb8bd94b044dbae9181dfc91810e300ed
    jasondrogba authored Nov 6, 2023
    Configuration menu
    Copy the full SHA
    c7c0820 View commit details
    Browse the repository at this point in the history
  3. Improve distributed load

    ### What changes are proposed in this pull request?
    
    Features:
    1. Retry in the end of the load process only once
    2. Persist failed file list once the load job is done
    
    Improvements:
    1. Only keep the most recent 1000 errroes to save memory
    2. Create a LoadSubTaskIterator and refactor the task generation progress
    3. Add an option to skip fetching ufs status again to speed up the retry (deafaulted on)
    4. --progress displays the elapsed time
    5. Add a metric to record error attribution
    6. --progresss displays subtask level metrics additionally
    7. fast fail the task when hash ring moves
    8. --progress --verbose displays information about the subtask (instead of the file name only)
    
    Bug fixes:
    1. Fix the wrong behavior where the distributed load still gives all workers when dynamic hash ring is disabled
    2. Fix the incorrect display of the loading progress report
    
    ![image](https://github.com/Alluxio/alluxio/assets/6771554/ff2c9ccb-0eed-472d-98cc-9cc53def37ba)
    
    
    			pr-link: Alluxio#18362
    			change-id: cid-7fb3f6f9230c04a01f7f466b32726c906e191ee2
    elega authored Nov 6, 2023
    Configuration menu
    Copy the full SHA
    d766ae4 View commit details
    Browse the repository at this point in the history
  4. [SMALLFIX] Allow extension on UfsManager.tryUfsFileSystem()

    ### What changes are proposed in this pull request?
    
    Change the method visibility to `protected` to allow overriding this method.
    
    ### Why are the changes needed?
    
    I can override this method to a noop so the `UfsManager` does not connect to the real UFS instance.
    
    			pr-link: Alluxio#18374
    			change-id: cid-8869941af37e8eda89b37f1637afd21743a6c61e
    jiacheliu3 authored Nov 6, 2023
    Configuration menu
    Copy the full SHA
    741563b View commit details
    Browse the repository at this point in the history
  5. Fix COS/OSS/OBS iterative listing

    ### What changes are proposed in this pull request?
    
    Please outline the changes and how this PR fixes the issue.
    
    ### Why are the changes needed?
    
    Please clarify why the changes are needed. For instance,
      1. If you propose a new API, clarify the use case for a new API.
      2. If you fix a bug, describe the bug.
    
    ### Does this PR introduce any user facing changes?
    
    Please list the user-facing changes introduced by your change, including
      1. change in user-facing APIs
      2. addition or removal of property keys
      3. webui
    
    			pr-link: Alluxio#18383
    			change-id: cid-ccd3bcc9e11d88274603ec058e091e97ea136448
    elega authored Nov 6, 2023
    Configuration menu
    Copy the full SHA
    bad173c View commit details
    Browse the repository at this point in the history

Commits on Nov 7, 2023

  1. Print warning messages for stale open handle instead of removing it

    ### What changes are proposed in this pull request?
    
    Checking all open handles periodically. 
    if some handle is stale (inactive for more than 24 hours), please warning messages.
    Every time an open handle is used, its last access time is updated.
    
    In future, we may need to handle this case instead of printing warning messages.
    
    ### Why are the changes needed?
    
    If an open handle is inactive for a long time, it may be caused by stuck, or by disconnected client.
    So a warning message is generated. Originally the open handle was closed in such case.
    But if a write is really slow or paused there, we are not going to close this handle.
    
    ### Does this PR introduce any user facing changes?
    
    N/A.
    
    			pr-link: Alluxio#18267
    			change-id: cid-5aa996931b6ae28513fbd00a63427116c17b4da6
    huanghua78 authored Nov 7, 2023
    Configuration menu
    Copy the full SHA
    912731c View commit details
    Browse the repository at this point in the history
  2. Fix cosn ufs cannot find class

    Fix bug involved by Alluxio#17024
    
    After Alluxio#18143, HdfsUnderFileSystemFactory is no longer included in COSN jar therefore no need to exclude.
    
    Same PR as Alluxio#18303 but pushing to the main branch
    			pr-link: Alluxio#18336
    			change-id: cid-c79454f41157781a29fb04d6067ea8cf16520977
    maobaolong authored Nov 7, 2023
    Configuration menu
    Copy the full SHA
    8d3ab67 View commit details
    Browse the repository at this point in the history
  3. Change logging level in NettyChannelPool

    ### What changes are proposed in this pull request?
    
    Reduce unnecessary logging to help debugging
    
    ### Why are the changes needed?
    
    Noisy logging like `2023-09-11 20:02:30,750 INFO  NettyChannelPool - Channel closed`
    
    ### Does this PR introduce any user facing changes?
    
    N/A
    
    			pr-link: Alluxio#18203
    			change-id: cid-d7c3842e0c76f0527a83345257b1ca2388927da9
    apc999 authored Nov 7, 2023
    Configuration menu
    Copy the full SHA
    00a527d View commit details
    Browse the repository at this point in the history
  4. Add a configurable DLQ capacity of avoid OOM

    ### What changes are proposed in this pull request?
    
    1. Add a property to limit the DLQ capacity in distributed load
    2. Fix a position reader bug where the prefetch size will converge at 2 * read length - 1
    
    			pr-link: Alluxio#18387
    			change-id: cid-2a20dd7c85de56926cd3a6bbc67c3ed6e8c14299
    elega authored Nov 7, 2023
    Configuration menu
    Copy the full SHA
    723c720 View commit details
    Browse the repository at this point in the history
  5. Fix the issue to get a tag of a non-existed in OSS/COS

    ### What changes are proposed in this pull request?
    
    Add exception handling for OSS and COS ufs in getting object tags and other apis.
    
    ### Why are the changes needed?
    When handling exceptions, we need to identify exceptions that can be allowed, and other errors need to be converted to an Alluxio exception.
    
    ### Does this PR introduce any user facing changes?
    
    Please list the user-facing changes introduced by your change, including
      1. change in user-facing APIs
      2. addition or removal of property keys
      3. webui
    
    			pr-link: Alluxio#18388
    			change-id: cid-2eb458f3ec6955321981dff350dacc9f33104c1b
    Jackson-Wang-7 authored Nov 7, 2023
    Configuration menu
    Copy the full SHA
    d28559a View commit details
    Browse the repository at this point in the history
  6. Fix NPE

    ### What changes are proposed in this pull request?
    
    Please outline the changes and how this PR fixes the issue.
    
    ### Why are the changes needed?
    
    Please clarify why the changes are needed. For instance,
      1. If you propose a new API, clarify the use case for a new API.
      2. If you fix a bug, describe the bug.
    
    ### Does this PR introduce any user facing changes?
    
    Please list the user-facing changes introduced by your change, including
      1. change in user-facing APIs
      2. addition or removal of property keys
      3. webui
    
    			pr-link: Alluxio#18391
    			change-id: cid-24fcdc50fc0b6560f77c0f2f31ebadd78000bac8
    elega authored Nov 7, 2023
    Configuration menu
    Copy the full SHA
    420e560 View commit details
    Browse the repository at this point in the history
  7. Fix chaotic chinese characters with golang cli commands

    When debugging load command, it is found that `./bin/alluxio` CLI can't process Chinese characters correctly. There must be something wrong with the CLI encoding or decoding. It seems that the Chinese characters are encoded incorrectly.
    
    This happens because the construction of CLI commands does not initialize environmental variables correctly. OS-level environmental variables are ignored, so the program uses a false encoder.
    
    This pull request fixes this problem.
    			pr-link: Alluxio#18386
    			change-id: cid-a15fb106b4a6cd77087cdbba82e426f1ba1e4b4c
    twalluxio authored Nov 7, 2023
    Configuration menu
    Copy the full SHA
    7d15de7 View commit details
    Browse the repository at this point in the history
  8. [SMALLFIX] Remove unused objects initialize

    ### What changes are proposed in this pull request?
    
    
    
    ### Why are the changes needed?
    
    `objects` is initialized by empty array but we never used this default value.
    
    ### Does this PR introduce any user facing changes?
    
    Please list the user-facing changes introduced by your change, including
      1. change in user-facing APIs. no
      2. addition or removal of property keys. no
      3. webui. no
    
    			pr-link: Alluxio#18380
    			change-id: cid-f16cc464ebc647d4c9be79b2151fb06c4ec71ed4
    lfyzjck authored Nov 7, 2023
    Configuration menu
    Copy the full SHA
    bd99175 View commit details
    Browse the repository at this point in the history
  9. Improve UpdateChecker

    ### What changes are proposed in this pull request?
    
    Improve UpdateChecker
    
    ### Why are the changes needed?
    
      1. remove irrelevant master features 
      2. turn on fuse update check
    
    
    			pr-link: Alluxio#18101
    			change-id: cid-8f6801fedda5f7710d111939e50cfd5f4372b7e5
    apc999 authored Nov 7, 2023
    Configuration menu
    Copy the full SHA
    246c7ee View commit details
    Browse the repository at this point in the history

Commits on Nov 8, 2023

  1. Configuration menu
    Copy the full SHA
    18c2190 View commit details
    Browse the repository at this point in the history
  2. Fix StressWorkerBench with remote worker policy

    This pull request fixes a bug on stress worker bench.
    
    1. When using the remote worker policy, running `findNextWorker()`, multiple threads increment `mNextIndex`, leading to possible array out of bounds errors when getting worker info.
    2. If use remote worker policy and the cluster size is 1, no workers can be found in `findNextWorker()`, resulting in errors.
    
    This fix turns `findNextWorker()` into a synchronized function, which can avoid data corruption. In addition, this policy can only be used when cluster size > 1.
    			pr-link: Alluxio#18377
    			change-id: cid-2e391549b5d81ca3f1ff60f7a6c7eb59e99cc736
    twalluxio authored Nov 8, 2023
    Configuration menu
    Copy the full SHA
    5474513 View commit details
    Browse the repository at this point in the history
  3. Fix integration tests again

    a previous PR Alluxio#18313 restored the integration tests but caused the unit tests to also run with the integration tests.
    
    the issue was correctly identified as an issue with the maven project list but the underlying problem was that the project name was not correctly set; `dora/tests` is not a module. it was previously a module until `dora/tests/testcontainers` was introduced, thus separating testcontainer tests vs integration tests.
    
    this fix updates the previous project list with the correct module name
    			pr-link: Alluxio#18392
    			change-id: cid-5f899f224abfbea4b7608d72cc2d38fed6449836
    Xenorith authored Nov 8, 2023
    Configuration menu
    Copy the full SHA
    502b0dd View commit details
    Browse the repository at this point in the history
  4. cherry pick empty because Tencent Cloud EMR doc have been removed in …

    …3.x [DOCFIX] Fix doc errors for Tencent Cloud EMR
    
    Please outline the changes and how this PR fixes the issue.
    
    Tencent Cloud EMR has updated some version information and needs to be synchronized to community documents
    
    Please list the user-facing changes introduced by your change, including
      1. change in user-facing APIs
      2. addition or removal of property keys
      3. webui
    
    			pr-link: Alluxio#17747
    			change-id: cid-73d26a59de4f1023081ed0ffaf91cffcc2f8db0e
    wheatxiong authored and jiacheliu3 committed Nov 8, 2023
    Configuration menu
    Copy the full SHA
    615a6d8 View commit details
    Browse the repository at this point in the history
  5. Close client when heartbeat executor closed

    ### What changes are proposed in this pull request?
    
    Please outline the changes and how this PR fixes the issue.
    
    ### Why are the changes needed?
    
    Please clarify why the changes are needed. For instance,
      1. If you propose a new API, clarify the use case for a new API.
      2. If you fix a bug, describe the bug.
    
    ### Does this PR introduce any user facing changes?
    
    Please list the user-facing changes introduced by your change, including
      1. change in user-facing APIs
      2. addition or removal of property keys
      3. webui
    
    			pr-link: Alluxio#17785
    			change-id: cid-657e47ef7657714a572a1f9811bbcc524bb8cfcc
    maobaolong authored and jiacheliu3 committed Nov 8, 2023
    Configuration menu
    Copy the full SHA
    8eba127 View commit details
    Browse the repository at this point in the history
  6. Fix a deadlock in FileSystemContext

    ### What changes are proposed in this pull request?
    
    Fix a deadlock.
    
    ### Why are the changes needed?
    
    On the Alluxio client, there is a monitor lock of `FileSystemContext` to sync individual client-to-master operations.
    
    One application thread can successfully hold the monitor lock (`alluxio.client.file.FileSystemContext`)  in order to get worker list, but blocked by another “lock” `FileSystemContextReinitializer`  (waiting for latch on on-going RPCs down to zero) for acquiring the block master client to really connect to master process  (waiting for other callers involving `FileSystemContextReinitializer` to finish). 
    
    On the other hand, another heartbeat thread "config-hash-master-heartbeat-0" is awaking periodically to sync with the master process to fetch the latest configuration. This thread detected the conf update and thus entered `FileSystemContextReinitializer` (bumping latch) but was blocked by waiting for the monitor lock of  `alluxio.client.file.FileSystemContext` in order to get the master address.
    
    This PR moves `getMasterAddress` outside `reinit` block to avoid holding the `Reinitializer` object and wait for the monitor object of `FileSystemContext`.
    
    ### Does this PR introduce any user facing changes?
    
    No
    			pr-link: Alluxio#18109
    			change-id: cid-0fab540f7dbef3661acb44e7472425add163b019
    apc999 authored and jiacheliu3 committed Nov 8, 2023
    Configuration menu
    Copy the full SHA
    704a460 View commit details
    Browse the repository at this point in the history
  7. Fix local cache identifier to solve the local cache consistent issue …

    …and resolve a trivial conflict
    
    Fix local cache identifier
    
    If not including `modification time`, it may have consistency issues.
    
    NO
    
    			pr-link: Alluxio#17514
    			change-id: cid-c9c2e91e53d4a95d4f3a165918fc23c02f623891
    secfree authored and jiacheliu3 committed Nov 8, 2023
    Configuration menu
    Copy the full SHA
    6be7125 View commit details
    Browse the repository at this point in the history
  8. Support getFileBlockLocation in LocalCacheFileSystem

    ### What changes are proposed in this pull request?
    
    Delegate `getFileBlockLocation` to external file system in `LocalCacheFileSystem`.
    
    ### Why are the changes needed?
    
    Otherwise, `LocalCacheFileSystem` inherits the default behavior of `org.apache.hadoop.fs.FileSystem` which returns `localhost` only. 
    
    ### Does this PR introduce any user facing changes?
    
    No.
    
    			pr-link: Alluxio#17672
    			change-id: cid-eb545dbd8ed42001d074fecfb9c8d6b118a559c1
    maobaolong authored and jiacheliu3 committed Nov 8, 2023
    Configuration menu
    Copy the full SHA
    c453859 View commit details
    Browse the repository at this point in the history
  9. Support read bytebuffer for non ByteBufferReadable input stream

    ### What changes are proposed in this pull request?
    
    Support read bytebuffer for non ByteBufferReadable input stream
    
    ### Why are the changes needed?
    
    Without this changes, a non ByteBufferReadable input stream can throw exception when the bytebuffer apis are called.
    
    ### Does this PR introduce any user facing changes?
    
    No
    
    			pr-link: Alluxio#17982
    			change-id: cid-1363dbbeca327cbb7b4d39cea1afbb8c51905483
    maobaolong authored and jiacheliu3 committed Nov 8, 2023
    Configuration menu
    Copy the full SHA
    1e283df View commit details
    Browse the repository at this point in the history
  10. Configuration menu
    Copy the full SHA
    d0d6b0c View commit details
    Browse the repository at this point in the history
  11. Use correct audit log configuration in proxy

    ### What changes are proposed in this pull request?
    
    The v1 and v2 versions of proxy use uniform judgment conditions.
    
    
    			pr-link: Alluxio#18371
    			change-id: cid-1842a0b96b5233f9a456e1aee7dbd95e2bd2d5e0
    Haoning-Sun authored and jiacheliu3 committed Nov 8, 2023
    Configuration menu
    Copy the full SHA
    dd2d0f9 View commit details
    Browse the repository at this point in the history
  12. Support Mkdir/CreateFile with configured default umask in HDFS API

    ### What changes are proposed in this pull request?
    Support creating directories and files with default permissions based on configuration propertykey in HDFS API.
    
    ### Why are the changes needed?
    Alluxio Hdfs api hasn't the corresponding implementation of Mkdir without permission parameter. If it does not carry permission, the umask property in the configuration item shall prevail.
    
    ### Does this PR introduce any user facing changes?
    
    
    			pr-link: Alluxio#18253
    			change-id: cid-6ddd2243bac00ebbdbdff1e731036c0d3d6228c8
    Jackson-Wang-7 authored and jiacheliu3 committed Nov 8, 2023
    Configuration menu
    Copy the full SHA
    f474bd7 View commit details
    Browse the repository at this point in the history
  13. cherry pick without resolving conflicts: Get User/Group/Permission of…

    … existed objects in OSS ufs
    
    Using the user of the OSS bucket to represent the user of the object loaded from OSS.
    Add the mapping way from OSS username to the custom username.
    Add the default permission mode for existing files loaded from OSS.
    
    the object loaded from OSS ufs will be null user, null group, and 777 by default.
    
    			pr-link: Alluxio#18262
    			change-id: cid-7e56861c9def695876cde32d1e0aa453b512470d
    Jackson-Wang-7 authored and jiacheliu3 committed Nov 8, 2023
    Configuration menu
    Copy the full SHA
    b64b0cf View commit details
    Browse the repository at this point in the history
  14. Configuration menu
    Copy the full SHA
    a5d33f9 View commit details
    Browse the repository at this point in the history
  15. Configuration menu
    Copy the full SHA
    3350541 View commit details
    Browse the repository at this point in the history
  16. [SMALLFIX] Fix broken ChmodCommandIntegrationTest

    ### What changes are proposed in this pull request?
    
    The old one is ignored, now bring it back.
    
    ### Why are the changes needed?
    
    Improve code quality.
    
    ### Does this PR introduce any user facing changes?
    
    No.
    
    			pr-link: Alluxio#18376
    			change-id: cid-2cdb7611d32e50fc16044192edf1122f4a3cd646
    YichuanSun authored Nov 8, 2023
    Configuration menu
    Copy the full SHA
    c9760b8 View commit details
    Browse the repository at this point in the history
  17. Use generic Exception to catch all exceptions in getStatus()

    Also print file path in getStatus() exception on client side
    
    ### What changes are proposed in this pull request?
    
    Use generic Exception to catch all exceptions in getStatus(), just like that in all other handlers.
    File path will be included in log message for getStatus().
    
    ### Why are the changes needed?
    
    Sometimes, exceptions are thrown in under file system and/or other third-party libraries.
    We don't know the exact type of the specific exceptions. If they are not caught, client will
    report unknown exceptions in logs. This is misleading. 
    
    ### Does this PR introduce any user facing changes?
    
    Client will show logs with detailed exception information if there is any.
    For getStatus(), file path will also be shown in log.
    
    			pr-link: Alluxio#18398
    			change-id: cid-b6acee3dd0c3e2e03e66bbf195cbda1e95f6bf64
    huanghua78 authored Nov 8, 2023
    Configuration menu
    Copy the full SHA
    6dc2fae View commit details
    Browse the repository at this point in the history
  18. Merge master-2.x commits 2023/07/01~2023/11/08 into main

    ### What changes are proposed in this pull request?
    Merge missing commits from master-2.x to main. The commits in 2023/07/01~2023/11/08 from Alluxio/alluxio@main...master-2.x will be included by this PR.
    
    We do this merge to catch missing fixes from `master-2.x` and catch the train before `main` cuts a release.
    
    Alluxio#17747 is not cherry picked because tencent cloud EMR doc is removed
    Alluxio#17755 is not cherry picked because DistLoadCliRunner has been removed in 3.x
    Alluxio#17758 is not cherry picked because MonoBlockStore has been removed in 3.x
    Alluxio#17641 is not cherry picked because the PR has already been in main
    Alluxio#17781 is not cherry picked because the PR has already been in main
    Alluxio#17722 is not cherry picked because the alluxio-fuse command has been changed a lot
    Alluxio#17489 is not cherry picked because audit log on master is no longer in 3.x
    Alluxio#17865 is not cherry picked because replication on job service is no longer in 3.x
    Alluxio#17858 is not cherry picked because it is already in main
    Alluxio#18090 is not cherry picked because generate-tarball has been rewritten in 3.x
    Alluxio#18091 is not cherry picked because the change is already in main
    Alluxio#17474 is not cherry picked because reconfiguration feature is not defined in 3.x
    Alluxio#17735 is not cherry picked because MonoBlockStore is no longer in 3.x
    Alluxio#18133 is not cherry picked because the issue is about master metadata and no longer relevant in 3.x
    Alluxio#17910 is not cherry picked because I prefer to do that manually
    Alluxio#17983 is not cherry picked because the web UI has been reworked
    Alluxio#17984 is not cherry picked because Mount/Unmount commands have been reworked in 3.x
    Alluxio#18103 is not cherry picked because worker cache metrics have been reworked in 3.x
    Alluxio#18185 is not cherry picked because the report command has been reworked in 3.x
    Alluxio#18222 is not cherry picked because Mount/Unmount operations have been reworked in 3.x
    Alluxio#18143 is not cherry picked because the change is already in main
    Alluxio#18303 is not cherry picked because the change is already in main
    Alluxio#18208 is not cherry picked because cache metrics have been reworked in 3.x
    Alluxio#17002 is not cherry picked because the owner has been notified separately
    Alluxio#18334 is not cherry picked because the bash scripts have been reworked in 3.x
    Alluxio#18326 is not cherry picked because the owner has been notified separately
    
    			pr-link: Alluxio#18397
    			change-id: cid-dbf8cbb2d9e721a5a0a1e5028a3c9577438a2ac0
    alluxio-bot authored Nov 8, 2023
    Configuration menu
    Copy the full SHA
    c2f4cd2 View commit details
    Browse the repository at this point in the history
  19. Fix the NPE in COSUnderFileSystem

    ### What changes are proposed in this pull request?
    
    meta.getLastModified() could be null, and should avoid the NPE. the parameter is nullable, so give it a null in this case.
    
    ### Why are the changes needed?
    
    meta.getLastModified() could be null, and should avoid the NPE. the parameter is nullable, so give it a null in this case.
    
    ### Does this PR introduce any user facing changes?
    
    Please list the user-facing changes introduced by your change, including
      1. change in user-facing APIs
      2. addition or removal of property keys
      3. webui
    
    			pr-link: Alluxio#18403
    			change-id: cid-9d3c16f8c2b403dff668f52649432a5d7ca8f1a3
    Jackson-Wang-7 authored Nov 8, 2023
    Configuration menu
    Copy the full SHA
    1c279c6 View commit details
    Browse the repository at this point in the history
  20. Update version to 305

    Xenorith committed Nov 8, 2023
    Configuration menu
    Copy the full SHA
    b0f803a View commit details
    Browse the repository at this point in the history
  21. Configuration menu
    Copy the full SHA
    d07ca76 View commit details
    Browse the repository at this point in the history

Commits on Nov 9, 2023

  1. Remove malformed empty page files on access

    ### What changes are proposed in this pull request?
    
    Remove empty pages when they are accessed by the get page methods. Empty pages except for the vert first page of a file are most likely malformed and are a result of a failed put attempt.
    
    ### Why are the changes needed?
    
    Reading empty pages causes 0 bytes to be read, and in some cases causing an infinite loop in the caller. When used with `getDataFileChannel`, it results in the channel to be created multiple times for a given empty page, because the channels are always 0 sized.
    
    ### Does this PR introduce any user facing changes?
    
    No.
    
    			pr-link: Alluxio#18401
    			change-id: cid-fff8f48af3814e3c71317377aa0b065599c06a57
    dbw9580 authored Nov 9, 2023
    Configuration menu
    Copy the full SHA
    2e1f95d View commit details
    Browse the repository at this point in the history
  2. Replace the FOLDER_SUFFIX from "_$folder$" to "/" in OSS UFS

    ### What changes are proposed in this pull request?
    Replace the FOLDER_SUFFIX from "_$folder$" to "/" in OSS UFS
    
    ### Why are the changes needed?
    We don't want to use '_$ Folder $" to represent the directory. We want to use a more general "/" instead to avoid some issues
    
    
    ### Does this PR introduce any user facing changes?
    No
    
    			pr-link: Alluxio#18408
    			change-id: cid-d16023b954bcf0ad9f37c9f629c5d0799443af8f
    Jackson-Wang-7 authored Nov 9, 2023
    Configuration menu
    Copy the full SHA
    69ca11b View commit details
    Browse the repository at this point in the history
  3. Fix stress worker bench remote only policy not working

    PR Alluxio#18377 removes setting property `alluxio.user.worker.selection.policy` by mistake.
    
    This will cause remote only policy not working.
    
    This PR fixes this problem.
    			pr-link: Alluxio#18407
    			change-id: cid-a53dce950a07cdd661b3237731037a0e9efb052c
    twalluxio authored Nov 9, 2023
    Configuration menu
    Copy the full SHA
    f7225f4 View commit details
    Browse the repository at this point in the history
  4. Change file exist exception to runtime exception in PagedDoraWorker

    ### What changes are proposed in this pull request?
    
    The `RuntimeException(FileAlreadyExistException)` changed to `AlreadyExistException()` in `PagedDoraWorker`
    
    ### Why are the changes needed?
    
    `RuntimeException(FileAlreadyExistException)` seems won't be caught in `DoraWorkerClientServiceHandler` when convertin it to `AlluxioRuntimeException`, which willl cause the loss status code
    
    ### Does this PR introduce any user facing changes?
    
    no
    
    			pr-link: Alluxio#18337
    			change-id: cid-9a520c49579847bae6da21302f484ba713eeb4d9
    voddle authored Nov 9, 2023
    Configuration menu
    Copy the full SHA
    1852423 View commit details
    Browse the repository at this point in the history

Commits on Nov 10, 2023

  1. Translate Introduction

    Introduction Chapter translation
    			pr-link: Alluxio#18176
    			change-id: cid-c31d1ab3c4365f0bdb0d0b17418c92885a109276
    xiong-make authored Nov 10, 2023
    Configuration menu
    Copy the full SHA
    d9cbbf1 View commit details
    Browse the repository at this point in the history
  2. Improve page loading performance

    ### What changes are proposed in this pull request?
    
    Improve page loading performance by loading the page form UFS directly, instead of creating a reader.
    
    ### Why are the changes needed?
    
    To improve the data loading performance & reduce the memory consumption 
    
    ### Does this PR introduce any user facing changes?
    
    N/A
    			pr-link: Alluxio#18389
    			change-id: cid-46e89dcfd22dcbbb4090df7f6c8861408ef8cb67
    elega authored Nov 10, 2023
    Configuration menu
    Copy the full SHA
    5421aa4 View commit details
    Browse the repository at this point in the history
  3. Allow rename() to overwrite, fix an uncaught ex and reenable UT

    ### What changes are proposed in this pull request?
    
    As the title states, this PR does a few things:
    
    1. Allow rename() to overwrite an existing path, if specified in the option
    2. The method `rename` in `AbstractFileSystem`, now will process the input path and rerun `rename` method when caught `AlluxioException` or `AlluxioRuntimeException`. Instead of log the exception and return end.
    3. Also add some path checks during the rename
    4. Reenable `FileSystemRenameIntegrationTest` by adding configs to `LocalAlluxioClusterResource` and reenable UT cases.
    
    			pr-link: Alluxio#18263
    			change-id: cid-2870bf87fea8a3b2419e5b10a05423ff2dede6a2
    voddle authored Nov 10, 2023
    Configuration menu
    Copy the full SHA
    328ee8c View commit details
    Browse the repository at this point in the history

Commits on Nov 11, 2023

  1. [SMALLFIX] Remove invalid Property WORKER_FUSE_ENABLED

    ### What changes are proposed in this pull request?
    
    Now WORKER_FUSE_ENABLED is invalid.
    
    ### Why are the changes needed?
    
    improve code quality.
    
    ### Does this PR introduce any user facing changes?
    
    no.
    
    			pr-link: Alluxio#18349
    			change-id: cid-8064a8ef19179c31cdce01a833847ff530c1f776
    YichuanSun authored Nov 11, 2023
    Configuration menu
    Copy the full SHA
    ba02104 View commit details
    Browse the repository at this point in the history

Commits on Nov 13, 2023

  1. Add call points of the new metrics

    ### What changes are proposed in this pull request?
    Add call points of the new metrics
    
    ### Why are the changes needed?
    The new metrics are already defined in the previous pr, and need to be called in this pr.
    
    ### Does this PR introduce any user facing changes?
    no
    
    			pr-link: Alluxio#18351
    			change-id: cid-1b35a87f41a1f836005c368881378329f4c77b25
    Kai-Zhang authored Nov 13, 2023
    Configuration menu
    Copy the full SHA
    08547a5 View commit details
    Browse the repository at this point in the history
  2. Update README.md

    Fixed a single spelling error.
    
    ### What changes are proposed in this pull request?
    
    Please outline the changes and how this PR fixes the issue.
    
    ### Why are the changes needed?
    
    Please clarify why the changes are needed. For instance,
      1. If you propose a new API, clarify the use case for a new API.
      2. If you fix a bug, describe the bug.
    
    ### Does this PR introduce any user facing changes?
    
    Please list the user-facing changes introduced by your change, including
      1. change in user-facing APIs
      2. addition or removal of property keys
      3. webui
    
    			pr-link: Alluxio#18310
    			change-id: cid-77c21de69cf13a6c94be418adf2be95e0f347d42
    computerscienceiscool authored Nov 13, 2023
    Configuration menu
    Copy the full SHA
    f6e4260 View commit details
    Browse the repository at this point in the history

Commits on Nov 14, 2023

  1. Clean up fuse unmount workflow

    ### What changes are proposed in this pull request?
    
    clean up unmount of fuse
    
    ### Why are the changes needed?
    
    alluxio-fuse  may cause AlluxioFuse continues to hang, and also the current flow to unmount is scattered and all over the place.
    
    ### Does this PR introduce any user facing changes?
    
    no
    
    			pr-link: Alluxio#18241
    			change-id: cid-0246bd831466396697efe1977547d365ad63ba3d
    lucyge2022 authored Nov 14, 2023
    Configuration menu
    Copy the full SHA
    feb0116 View commit details
    Browse the repository at this point in the history
  2. Fix the issue when mount an empty oss bucket

    ### What changes are proposed in this pull request?
    
    Please outline the changes and how this PR fixes the issue.
    
    ### Why are the changes needed?
    
    Please clarify why the changes are needed. For instance,
      1. If you propose a new API, clarify the use case for a new API.
      2. If you fix a bug, describe the bug.
    
    ### Does this PR introduce any user facing changes?
    
    Please list the user-facing changes introduced by your change, including
      1. change in user-facing APIs
      2. addition or removal of property keys
      3. webui
    
    			pr-link: Alluxio#18412
    			change-id: cid-fa4b35605d8f97d79c159140613aacf27d3c1836
    Jackson-Wang-7 authored Nov 14, 2023
    Configuration menu
    Copy the full SHA
    3a29f8d View commit details
    Browse the repository at this point in the history

Commits on Nov 17, 2023

  1. Fix the bug that HdfsUfsStatusIterator returns wrong path

    Fix the bug that `HdfsUfsStatusIterator` returns wrong path (the getName() method of `UfsStatus` should return a relative path, but not just the file name).
    			pr-link: Alluxio#18424
    			change-id: cid-8b15fbd345f26562a06c5d9f52bdbc90a53b2039
    JiamingMai authored Nov 17, 2023
    Configuration menu
    Copy the full SHA
    d2d29df View commit details
    Browse the repository at this point in the history

Commits on Nov 20, 2023

  1. Fix the bug that HDFS iterator doesn't work

    Fix the bug that HDFS iterator doesn't work.
    			pr-link: Alluxio#18429
    			change-id: cid-dbb5f1a246bce706f7b3b5d179ad158218dccf83
    JiamingMai authored Nov 20, 2023
    Configuration menu
    Copy the full SHA
    f7eed03 View commit details
    Browse the repository at this point in the history

Commits on Nov 29, 2023

  1. Fix duplicated class and resources in presto mvn build

    ### What changes are proposed in this pull request?
    
    Shade all the class under javax
    
    ### Why are the changes needed?
    
    We need this change to fix the maven build error in presto
    
    ### Does this PR introduce any user facing changes?
    
    no
    
    			pr-link: Alluxio#18450
    			change-id: cid-91aa8f5a00867b0f44165eb486fa168cc710504c
    beinan authored Nov 29, 2023
    Configuration menu
    Copy the full SHA
    24e4d3d View commit details
    Browse the repository at this point in the history
  2. Update version to 306

    alluxio-bot committed Nov 29, 2023
    Configuration menu
    Copy the full SHA
    50fd48f View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    cacc1cc View commit details
    Browse the repository at this point in the history

Commits on Dec 1, 2023

  1. Add WorkerClusterView

    ### What changes are proposed in this pull request?
    
    Add a `WorkerClusterView` to allow more efficiently indexing and filtering workers by different criteria.
    
    ### Why are the changes needed?
    
    The current return type `List<WorkerInfo>` of the methods `getAllWorkers` etc. of `MembershipManager` does not allow efficiently indexing and filtering workers, e.g. getting a worker by it ID, finding all lost workers.
    
    This is partially motivated by the review comments in Alluxio#18434 (comment)
    
    ### Does this PR introduce any user facing changes?
    
    No.
    
    
    			pr-link: Alluxio#18441
    			change-id: cid-e28beb96752cc45cdad973b98f2d689ce1581b0f
    dbw9580 authored Dec 1, 2023
    Configuration menu
    Copy the full SHA
    15b05ba View commit details
    Browse the repository at this point in the history

Commits on Dec 4, 2023

  1. Fix the shading issue for javax security

    ### What changes are proposed in this pull request?
    
    Fix the shading issue for javax security
    
    ### Why are the changes needed?
    you might see class not found in javax's security module
    
    ### Does this PR introduce any user facing changes?
    no
    
    			pr-link: Alluxio#18455
    			change-id: cid-d04ba461930b7abba2613810e8d7d2ae77d7e3be
    beinan authored Dec 4, 2023
    Configuration menu
    Copy the full SHA
    0db1ff7 View commit details
    Browse the repository at this point in the history

Commits on Dec 5, 2023

  1. Update version to 307

    alluxio-bot committed Dec 5, 2023
    Configuration menu
    Copy the full SHA
    7223e80 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    e6ac3b8 View commit details
    Browse the repository at this point in the history

Commits on Dec 6, 2023

  1. Make proxy transfer buffer size configurable

    ### What changes are proposed in this pull request?
    
    Make the buffer size in s3 proxy configurable, which is currently 8KB and unchangeable.
    			pr-link: Alluxio#18459
    			change-id: cid-7ebb81819b6f13222496a341cbef743cf50bda7b
    ddrid authored Dec 6, 2023
    Configuration menu
    Copy the full SHA
    4ab2acf View commit details
    Browse the repository at this point in the history

Commits on Dec 8, 2023

  1. Make WorkerInfo enum (and some small refactors)

    ### What changes are proposed in this pull request?
    
    1. Move `WorkerState` enum from master package to the wire package.
    2. Add a new worker state `UNRECOGNIZED` and use it as the default state.
    3. Make worker state in `WorkerInfo` an enum.
    4. Add copy constructors to `WorkerInfo` and `WorkerNetAddress`.
    
    ### Why are the changes needed?
    
    1. Make sure the state of worker can be enumerated.
    2. Allow safely copying mutable `WorkerInfo` and `WorkerNetAddress` objects.
    
    ### Does this PR introduce any user facing changes?
    
    No.
    
    			pr-link: Alluxio#18460
    			change-id: cid-8daf9c1e3ebe8e862a9b0dabb669c80918f5b8b0
    dbw9580 authored Dec 8, 2023
    Configuration menu
    Copy the full SHA
    73d1746 View commit details
    Browse the repository at this point in the history

Commits on Dec 9, 2023

  1. Fix to disable worker identity reuse for registration unless in k8s

    ### What changes are proposed in this pull request?
    
    Put back the restriction of reuse worker identity for non-k8s env deployment.
    
    ### Why are the changes needed?
    
    Worker identity gets reused by wrong deployment behaviors such as copy conf/ over for new worker setup, as opposed to k8s deployment is thru operator / automation, bare metal deployment has no way of prevention, thus putting back the restriction for non-k8s deployment.
    
    ### Does this PR introduce any user facing changes?
    
    No.
    
    			pr-link: Alluxio#18454
    			change-id: cid-295ea352895b16c2a5f0a23fa790c9f42a5e3881
    lucyge2022 authored Dec 9, 2023
    Configuration menu
    Copy the full SHA
    22af1b1 View commit details
    Browse the repository at this point in the history

Commits on Dec 12, 2023

  1. Use cluster view to replace list of workers

    ### What changes are proposed in this pull request?
    
    Replace `List<BlockWorkerInfo>` with `WorkerClusterView` in APIs.
    
    Important APIs that are changed:
    
    1. `FileSystemContext.getCachedWorkers` now returns `WorkerClusterView`
    2. `WorkerLocationPolicy.getPreferredWorkers` (as well as all its implementations) now accepts a `WorkerClusterView` as the first argument (but still returns `List<BlockWorkerInfo>` as the returned list must be ordered)
    
    APIs that are using `List<BlockWorkerInfo>` (or `List<WorkerInfo>`) but *not* migrated to `WorkerClusterView`:
    
    1. `alluxio.master.scheduler.WorkerProvider.getWorkerInfos` returns `List<WorkerInfo>`.
    2. Job service related APIs, e.g. `alluxio.job.plan.PlanDefinition.selectExecutors`
    
    Notable behavior change:
    
    Now `EtcdMembershipManager` assigns the correct state (`LIVE` or `LOST`) for all workers in its  `WorkerInfo` struct. Before this change, this information is not available and the state defaults to `UNRECOGNIZED`.
    
    ### Why are the changes needed?
    
    Allow more efficient indexing and filtering workers by worker ID.
    
    ### Does this PR introduce any user facing changes?
    
    No.
    
    			pr-link: Alluxio#18451
    			change-id: cid-5052d2faa506f4de6e4b0df7062c5def3e09df1c
    dbw9580 authored Dec 12, 2023
    Configuration menu
    Copy the full SHA
    9eae1e9 View commit details
    Browse the repository at this point in the history

Commits on Dec 14, 2023

  1. Fix outdated worker address info returned by consistent hash policy

    ### What changes are proposed in this pull request?
    
    Fix outdated worker address info returned by consistent hash policy
    
    Summary of changes:
    
    1. `ConsistentHashProvider` only concerns about `WorkerIdentity` when building the hash ring. Therefore, the APIs have been limited to accept and return `WorkerIdentity`s.
    2. `ConsistentHashProvider.refresh` now accepts a set of worker identities instead of a list, as the order does not matter.
    3. Added a test to cover the bug fix.
    
    ### Why are the changes needed?
    
    Fix a bug where the consistent hash provider caches the `BlockWorkerInfo` of all workers, and when a worker changes its network addresses but its ID stays the same, the hash provider won't update the worker's info. A client will continue to use the outdated network address.
    
    The fix is to make the hash provider to only consider `WorkerIdentity`s, and let the client to figure out the worker's address with the ID provided by the consistent hash provider.
    
    ### Does this PR introduce any user facing changes?
    
    No.
    
    			pr-link: Alluxio#18434
    			change-id: cid-93f1601d846385f314c79a556c8705d3983a1199
    dbw9580 authored Dec 14, 2023
    Configuration menu
    Copy the full SHA
    9902e8a View commit details
    Browse the repository at this point in the history
  2. Improve rest load api response JSON

    This PR improves RESTful load api, use JSON format as the response content.
    
    ## Examples:
    ### Submit Job Example:
    ```
    // 20231211174310
    // http://localhost:28080/v1/load?path=hdfs://node01:8020/testRoot/testDirectory2&opType=submit&verbose=true
    {
      "success": true,
      "jobId": "0dbc0f47-580b-420f-b50e-d08a170746c8",
      "path": "hdfs://node01:8020/testRoot/testDirectory2",
      "message": "Load 'hdfs://node01:8020/testRoot/testDirectory2' is successfully submitted. JobId: 0dbc0f47-580b-420f-b50e-d08a170746c8\n"
    }
    ```
    ### Get Job Progress Example:
    ```
    // 20231211180110
    // http://localhost:28080/v1/load?path=hdfs://node01:8020/testRoot/testDirectory2&opType=progress&verbose=true
    
    {
      "jobState": "RUNNING",
      "path": "hdfs://node01:8020/testRoot/testDirectory2",
      "message": "Progress for loading path 'hdfs://node01:8020/testRoot/testDirectory2':\n\tSettings:\tbandwidth: unlimited\tverify: false\tmetadata-only: false\n\tTime Elapsed: 00:00:03\n\tJob State: RUNNING\n\tStage: RETRYING\n\tInodes Scanned: 4\n\tInodes Processed: 4\n\tBytes Loaded: 0B out of 0B\n\tThroughput: 0B/s\n\tFile Failure rate: 0.00%\n\tSubtask Failure rate: 0.00%\n\tFiles Failed: 0\n\tRecent failed subtasks: \n\tRecent retrying subtasks: \n\tSubtask Retry rate: 0.00%\n\tSubtasks on Retry Dead Letter Queue: 0\n",
      "respProperties": {
        "Files Failed": "0",
        "Recent failed subtasks": "",
        "Subtask Retry rate": "0.00%",
        "Throughput": "0B/s",
        "File Failure rate": "0.00%",
        "Subtasks on Retry Dead Letter Queue": "0",
        "Time Elapsed": "00",
        "Bytes Loaded": "0B out of 0B",
        "Stage": "RETRYING",
        "Inodes Scanned": "4",
        "Inodes Processed": "4",
        "Recent retrying subtasks": "",
        "Subtask Failure rate": "0.00%",
        "Settings": "bandwidth",
        "Job State": "RUNNING",
        "Progress for loading path 'hdfs": "//node01"
      }
    }
    ```
    
    ```
    // 20231211174358
    // http://localhost:28080/v1/load?path=hdfs://node01:8020/testRoot/testDirectory2&opType=progress&verbose=true
    
    {
      "jobState": "SUCCEEDED",
      "path": "hdfs://node01:8020/testRoot/testDirectory2",
      "message": "Progress for loading path 'hdfs://node01:8020/testRoot/testDirectory2':\n\tSettings:\tbandwidth: unlimited\tverify: false\tmetadata-only: false\n\tTime Elapsed: 00:00:16\n\tJob State: SUCCEEDED\n\tInodes Scanned: 4\n\tInodes Processed: 4\n\tBytes Loaded: 0B out of 0B\n\tThroughput: 0B/s\n\tFile Failure rate: 0.00%\n\tSubtask Failure rate: 0.00%\n\tFiles Failed: 0\n\tRecent failed subtasks: \n\tRecent retrying subtasks: \n\tSubtask Retry rate: 0.00%\n\tSubtasks on Retry Dead Letter Queue: 0\n",
      "respProperties": {
        "Files Failed": "0",
        "Recent failed subtasks": "",
        "Subtask Retry rate": "0.00%",
        "Throughput": "0B/s",
        "File Failure rate": "0.00%",
        "Subtasks on Retry Dead Letter Queue": "0",
        "Time Elapsed": "00",
        "Bytes Loaded": "0B out of 0B",
        "Inodes Scanned": "4",
        "Inodes Processed": "4",
        "Recent retrying subtasks": "",
        "Subtask Failure rate": "0.00%",
        "Settings": "bandwidth",
        "Job State": "SUCCEEDED",
        "Progress for loading path 'hdfs": "//node01"
      }
    }
    ```
    ### Stop Job Example:
    ```
    // 20231211180219
    // http://localhost:28080/v1/load?path=hdfs://node01:8020/testRoot/testDirectory2&opType=stop&verbose=true
    
    {
      "success": true,
      "path": "hdfs://node01:8020/testRoot/testDirectory2",
      "message": "Load 'hdfs://node01:8020/testRoot/testDirectory2' is successfully stopped.\n"
    }
    ```
    
    ```
    // 20231211180153
    // http://localhost:28080/v1/load?path=hdfs://node01:8020/testRoot/testDirectory2&opType=stop&verbose=true
    
    {
      "success": false,
      "path": "hdfs://node01:8020/testRoot/testDirectory2",
      "message": "Cannot find load job for path hdfs://node01:8020/testRoot/testDirectory2, it might have already been stopped or finished\n"
    }
    ```
    
    			pr-link: Alluxio#18464
    			change-id: cid-1fff9a23457064ab71534909449c60a6b0123f22
    JiamingMai authored Dec 14, 2023
    Configuration menu
    Copy the full SHA
    4e3a722 View commit details
    Browse the repository at this point in the history
  3. Fix alluxio-fuse script for finding fuse pid

    ### What changes are proposed in this pull request?
    
    Remove additional white space in alluxio-fuse script
    
    ### Why are the changes needed?
    
    alluxio-fuse unmount <mnt_point> is unable to find the pid of AlluxioFuse process because the grep content isn't correct.
    
    ### Does this PR introduce any user facing changes?
    
    No
    
    			pr-link: Alluxio#18465
    			change-id: cid-3e70b0c8edbaa0ba50d744fd6155b0d494a243f9
    lucyge2022 authored Dec 14, 2023
    Configuration menu
    Copy the full SHA
    4e39eda View commit details
    Browse the repository at this point in the history

Commits on Dec 21, 2023

  1. Support get page with RESTful API by specifying offset and length

    Support get page with RESTful API by specifying offset and length.
    
    PAGE_URL_FORMAT = (
    "[http://{worker_host}:{http_port}/v1/file/{path_id}/page/{page_index}?offset=100&length=1024](http://{worker_host}:%7Bhttp_port%7D/v1/file/%7Bpath_id%7D/page/%7Bpage_index%7D?offset=100&length=1024)"
    )
    			pr-link: Alluxio#18474
    			change-id: cid-ba5b0c5050843ccc5642950beadc8a0b049948be
    JiamingMai authored Dec 21, 2023
    Configuration menu
    Copy the full SHA
    b5a2895 View commit details
    Browse the repository at this point in the history

Commits on Jan 4, 2024

  1. Support write page with RESTful API

    Support write page with RESTful API
    
    ### Usage
    ```
    HTTP Method: POST
    Request URL: http://localhost:28080/v1/file/<fileId>/page/<pageIndex>
    HTTP Body: <page bytes>
    ```
    			pr-link: Alluxio#18481
    			change-id: cid-cab175a007bfcaf294e89adbe47531419036a245
    JiamingMai authored Jan 4, 2024
    Configuration menu
    Copy the full SHA
    726f812 View commit details
    Browse the repository at this point in the history

Commits on Jan 8, 2024

  1. Add fake numbers for statfs

    ```
    $ df -h /mnt/fuse/
    Filesystem      Size  Used Avail Use% Mounted on
    alluxio-fuse    910T     0  910T   0% /mnt/fuse
    ```
    
    ### What changes are proposed in this pull request?
    
    Add fake numbers for statfs
    
    ### Why are the changes needed?
    
    Some application checks the available space in a file system before continuing to do file operations.
    
    ### Does this PR introduce any user facing changes?
    
    A fake number (1 Petabytes) is provided to statfs. This number does not reflect real available storage space.
    
    			pr-link: Alluxio#18482
    			change-id: cid-9f60d185393b616be02bf8f473b2026f2047f28c
    huanghua78 authored Jan 8, 2024
    Configuration menu
    Copy the full SHA
    9ef7552 View commit details
    Browse the repository at this point in the history

Commits on Jan 10, 2024

  1. Support Various Kinds of Consistent Hash

    ### What changes are proposed in this pull request?
    
    Add Ketama Hashing, Jump Consistent Hashing, Maglev Hashing, and Multi Probe Hashing.
    
    ### Why are the changes needed?
    
    Now alluxio's user worker selection policy is Consistent Hash Policy.  It bings too much time cost, and it is not enough uniform, and not strictly consistent.
    
    Ketama: https://github.com/RJ/ketama
    Jump Consistent Hashing: https://arxiv.org/pdf/1406.2294.pdf
    Maglev Hashing: https://static.googleusercontent.com/media/research.google.com/zh-CN//pubs/archive/44824.pdf
    Multi Probe Hasing: https://arxiv.org/pdf/1505.00062.pdf
    
    We strongly recommend using Maglev Hashing for User Worker Selection Policy. Under most situation, it has the minimum time cost, and it is the most uniform and balanced hashing policy.
    
    ### Does this PR introduce any user facing changes?
    
    `alluxio.user.worker.selection.policy` has the following values: `CONSISTENT`, `JUMP`, `KETAMA`, `MAGLEV`, `MULTI_PROBE`, `LOCAL`, `REMOTE_ONLY`, corresponding to consistent hash policy, maglev hash policy, ketama hash policy, maglev hash policy, multi-probe respectively hash policy, local worker policy, remote only policy.
    
    The current default value is `CONSISTENT`.
    
    We recommend using Maglev Hash, which has the best hash consistency and is the least time-consuming. That is to say, set the value of `alluxio.user.worker.selection.policy` to `MAGLEV`. We will also consider setting this as the default value in the future.
    
    **Ketama Hasing**
    `alluxio.user.ketama.hash.replicas`: This is the value of replicas in the ketama hashing algorithm. When workers changes, it will guarantee the hash table is changed only in a minimal. The value of replicas should be X times the physical nodes in the cluster, where X is a balance between efficiency and cost.
    
    **Jump Consistent Hashing**
    None.
    
    **Maglev Hashing**
    `alluxio.user.maglev.hash.lookup.size`: This is the size of the lookup table in the maglev hashing algorithm. It must be a prime number. In the maglev hashing, it will generate a lookup table for workers. The bigger the size of the lookup table, the smaller the variance of this hashing algorithm will be. But bigger look up table will consume more time and memory.
    
    **Multi Probe Hashing**
    `alluxio.user.multi.probe.hash.probe.num`: This is the number of probes in the multi-probe hashing algorithm. In the multi-probe hashing algorithm, the bigger the number of probes, the smaller the variance of this hashing algorithm will be. But more probes will consume more time and memory.
    
    
    			pr-link: Alluxio#17817
    			change-id: cid-bad21c6e5ad83eb3da15a8960ba372b14c67b081
    Zihao Zhao authored Jan 10, 2024
    Configuration menu
    Copy the full SHA
    b9de24c View commit details
    Browse the repository at this point in the history

Commits on Jan 11, 2024

  1. [DOCFIX] Update kerberos configuration 'alluxio.hadoop.kerberos.keyta…

    …b.login.autorenewal' in HDFS docs
    
    
    
    ### What changes are proposed in this pull request?
    
    Update the correct kerberos configuration 'alluxio.hadoop.kerberos.keytab.login.autorenewal' in HDFS docs to avoid user confusion.
    
    ### Why are the changes needed?
    
    Fix Alluxio#18486 
    
    
    
    			pr-link: Alluxio#18487
    			change-id: cid-b8f08e2f67e5f10aa1426de7629b8e268e339433
    QianyongY authored Jan 11, 2024
    Configuration menu
    Copy the full SHA
    7d8f317 View commit details
    Browse the repository at this point in the history
  2. Create metadata directory in initiateMultipartUpload method

    ### What changes are proposed in this pull request?
    
    Create metadata directory in initiateMultipartUpload method.
    
    ### Why are the changes needed?
    
    Each request calls the initialization method of the handler and sends an `exists` request to the master, which is unnecessary.
    
    
    			pr-link: Alluxio#18462
    			change-id: cid-3efd076d7eb33cc063609fa1e1003e3aff480be6
    Haoning-Sun authored Jan 11, 2024
    Configuration menu
    Copy the full SHA
    e4da0e4 View commit details
    Browse the repository at this point in the history

Commits on Jan 13, 2024

  1. Fix entrypoint

    `format` commands don't exist in cli anymore, so as job services. Delete some entrypoint code.
    
    Solves Alluxio#18466
    			pr-link: Alluxio#18490
    			change-id: cid-0ee0b45a012a29df4a793107d7f8cce4ca98fc99
    ssz1997 authored Jan 13, 2024
    Configuration menu
    Copy the full SHA
    4f07b99 View commit details
    Browse the repository at this point in the history

Commits on Jan 19, 2024

  1. Fix the issue of S3 range read

    ### What changes are proposed in this pull request?
    
    fix the issue of S3 range read. 
    
    ### Why are the changes needed?
    
    there is a bug when setting  S3 range offset.
    
    ### Does this PR introduce any user facing changes?
    
    user can try the following cmd to validate S3 range read.
    `aws --endpoint http://localhost:39999/api/v1/s3 s3api get-object --range bytes=10-20 --bucket [bucket-name] --key=[key-name] [output-file]`
    
    
    
    			pr-link: Alluxio#18484
    			change-id: cid-b5fd9832a9900fba1105bb494a96f315b20f507d
    007DXR authored Jan 19, 2024
    Configuration menu
    Copy the full SHA
    6f2a527 View commit details
    Browse the repository at this point in the history
  2. Fix negative size if specified filesize over Integer.MAX_VALUE for st…

    …ressWorkerBench
    
    ### What changes are proposed in this pull request?
    
    Change to use long type store file size.
    
    ### Why are the changes needed?
    
    Without this PR, we cannot specified a filesize lagger than Integer.MAX_VALUE.
    
    ### Does this PR introduce any user facing changes?
    
    No
    
    			pr-link: Alluxio#18492
    			change-id: cid-2b816d1f2cbc9ebcf888b06802eb682fb76d55c2
    maobaolong authored Jan 19, 2024
    Configuration menu
    Copy the full SHA
    97f325a View commit details
    Browse the repository at this point in the history

Commits on Jan 25, 2024

  1. Add ServiceRegistry membership manager which only include active workers

    ### What changes are proposed in this pull request?
    Add etcd membership manager only include active workers
    For python client it would need to only look at active worker list 
    ### Why are the changes needed?
    We only need active workers list in big tech env. If the worker is down, we don't care and just reshard small portion of data to other workers.
    
    ### Does this PR introduce any user facing changes?
    
    na
    
    			pr-link: Alluxio#18495
    			change-id: cid-70ec6f27539f5f47b99be1ce3ff85cb9c117c3bf
    jja725 authored Jan 25, 2024
    Configuration menu
    Copy the full SHA
    055b375 View commit details
    Browse the repository at this point in the history

Commits on Jan 31, 2024

  1. Add http server port in the worker net address

    ### What changes are proposed in this pull request?
    
    This PR enables registering the worker's HTTP server's port in the etcd. This helps to find worker's restful APIs from the Python client.
    
    ### Why are the changes needed?
    
    Alluxio Python client (e.g. in ML use cases) needs to connect to the worker's REST APIs. But as the http server port isn't included in the worker's information in the etcd, the client fails to find the API endpoint.
    
    ### Does this PR introduce any user facing changes?
    
    No.
    
    			pr-link: Alluxio#18499
    			change-id: cid-1cf7e0bdc7cc0c9702949bc313de5583d9cc2fb8
    ChunxuTang authored Jan 31, 2024
    Configuration menu
    Copy the full SHA
    262b2b3 View commit details
    Browse the repository at this point in the history

Commits on Feb 1, 2024

  1. Update version to 308

    alluxio-bot committed Feb 1, 2024
    Configuration menu
    Copy the full SHA
    05e089b View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    e9a32c9 View commit details
    Browse the repository at this point in the history
  3. Fix DoraLoadCommandIntegrationTest

    ### What changes are proposed in this pull request?
    
    Fix DoraLoadCommandIntegrationTest.
    
    ### Why are the changes needed?
    
    Without this PR, DoraLoadCommandIntegrationTest may fail with the following exception
    
    ```
    [ERROR] alluxio.client.cli.fs.command.DoraLoadCommandIntegrationTest.testCommand  Time elapsed: 7.995 s  <<< FAILURE!
    java.lang.AssertionError
            at org.junit.Assert.fail(Assert.java:87)
            at org.junit.Assert.assertTrue(Assert.java:42)
            at org.junit.Assert.assertTrue(Assert.java:53)
            at alluxio.client.cli.fs.command.DoraLoadCommandIntegrationTest.testCommand(DoraLoadCommandIntegrationTest.java:107)
    ```
    
    The cause is that
    - The second job is using the same path as the first job
    - While submitting the second job, the second job is still in "cleaning" state. As the following log shows
    
        ```
        2024-01-31 09:52:33,057 [master-rpc-executor-TPE-thread-494] WARN  scheduler.Scheduler (Scheduler.java:submitJob) - There's concurrent submit while job is still in cleaning state
        ```
    
    - "progress" returns the progress of the first job instead of the second one
    
    The PR changes to use a different path for the second job, which avoids this issue.
    
    ### Does this PR introduce any user facing changes?
    
    NO
    
    			pr-link: Alluxio#18504
    			change-id: cid-331ba5508e86e8161006073d452ab1ba6230473a
    secfree authored Feb 1, 2024
    Configuration menu
    Copy the full SHA
    07ec4b9 View commit details
    Browse the repository at this point in the history

Commits on Feb 2, 2024

  1. Support nullable offset and length in get page RESTful API

    Fix the bug that the get page RESTful API doesn't support nullable offset and length.
    			pr-link: Alluxio#18506
    			change-id: cid-85eab5152e501b97bc9b4678e92b0d8e665a95ce
    JiamingMai authored Feb 2, 2024
    Configuration menu
    Copy the full SHA
    3d13072 View commit details
    Browse the repository at this point in the history

Commits on Feb 13, 2024

  1. Update version to 309

    alluxio-bot committed Feb 13, 2024
    Configuration menu
    Copy the full SHA
    2f9bb6a View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    59bfee0 View commit details
    Browse the repository at this point in the history

Commits on Feb 15, 2024

  1. Change default value of virtual node count per worker

    ### What changes are proposed in this pull request?
    
    2000 is too much
    
    ### Why are the changes needed?
    
    2000 is too much
    
    ### Does this PR introduce any user facing changes?
    
    na
    
    			pr-link: Alluxio#18516
    			change-id: cid-98762f4a176f30b7a83399183aef6f11d5113132
    jja725 authored Feb 15, 2024
    Configuration menu
    Copy the full SHA
    c34921d View commit details
    Browse the repository at this point in the history

Commits on Mar 6, 2024

  1. Support Hash Algorithms Benchmark

    zhaozihao committed Mar 6, 2024
    Configuration menu
    Copy the full SHA
    f89e5f5 View commit details
    Browse the repository at this point in the history
  2. Support Hash Algorithms Benchmark

    zhaozihao committed Mar 6, 2024
    Configuration menu
    Copy the full SHA
    dff2efe View commit details
    Browse the repository at this point in the history

Commits on Mar 7, 2024

  1. Support Hash Algorithms Benchmark

    zhaozihao committed Mar 7, 2024
    Configuration menu
    Copy the full SHA
    bd3b6f1 View commit details
    Browse the repository at this point in the history

Commits on Mar 8, 2024

  1. Support Hash Algorithms Benchmark

    zhaozihao committed Mar 8, 2024
    Configuration menu
    Copy the full SHA
    717013a View commit details
    Browse the repository at this point in the history