[core] fifo worker killing policy

[clarng]

Why are these changes needed?

For long-living, memory leaking actors, it is more desirable to kill oldest task that is leaking the most. This avoid the situation where we constantly kill actor, which may lead to side effects where we generate a lot of log files, or trigger increased memory consumption in gcs / dashboard

Related issue number

https://github.com/anyscale/product/issues/18727
https://github.com/anyscale/product/issues/18728

Checks

• I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
  • I've added any new APIs to the API Reference. For example, if I added a
    method in Tune, I've added it in doc/source/tune/api/ under the
    corresponding .rst file.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[scv119]

is this the only change comparing to LIFO?

[clarng]

yes, there is room to refactor, we need to figure out the right long term solution and productize that code

[cadedaniel]

This is failing on master, even though it passes in this PR. I'm not sure why...

[cadedaniel]

cc @jjyao , what do you think about this? chen and clarence are out today

=================================== FAILURES ===================================
_________________ test_one_actor_max_fifo_kill_previous_actor __________________
shutdown_only = None
    @pytest.mark.skipif(
        sys.platform != "linux" and sys.platform != "linux2",
        reason="memory monitor only on linux currently",
    )
    def test_one_actor_max_fifo_kill_previous_actor(shutdown_only):
        with ray.init(
            _system_config={
                "worker_killing_policy": "retriable_fifo",
                "memory_usage_threshold": 0.4,
            },
        ):
            bytes_to_alloc = get_additional_bytes_to_reach_memory_usage_pct(0.3)
    
            first_actor = Leaker.options(name="first_actor").remote()
            ray.get(first_actor.allocate.remote(bytes_to_alloc))
    
            actors = ray.util.list_named_actors()
            assert len(actors) == 1
            assert "first_actor" in actors
    
            second_actor = Leaker.options(name="second_actor").remote()
            ray.get(second_actor.allocate.remote(bytes_to_alloc))
    
            actors = ray.util.list_named_actors()
>           assert len(actors) == 1
E           assert 2 == 1
E             +2
E             -1
python/ray/tests/test_memory_pressure.py:540: AssertionError

[clarng]

Will take a look

On Mon, Mar 20, 2023 at 1:12 PM Cade Daniel ***@***.***> wrote: cc @jjyao <https://github.com/jjyao> , what do you think about this? chen and clarence are out today =================================== FAILURES ===================================_________________ test_one_actor_max_fifo_kill_previous_actor __________________shutdown_only = None @pytest.mark.skipif( sys.platform != "linux" and sys.platform != "linux2", reason="memory monitor only on linux currently", ) def test_one_actor_max_fifo_kill_previous_actor(shutdown_only): with ray.init( _system_config={ "worker_killing_policy": "retriable_fifo", "memory_usage_threshold": 0.4, }, ): bytes_to_alloc = get_additional_bytes_to_reach_memory_usage_pct(0.3) first_actor = Leaker.options(name="first_actor").remote() ray.get(first_actor.allocate.remote(bytes_to_alloc)) actors = ray.util.list_named_actors() assert len(actors) == 1 assert "first_actor" in actors second_actor = Leaker.options(name="second_actor").remote() ray.get(second_actor.allocate.remote(bytes_to_alloc)) actors = ray.util.list_named_actors()> assert len(actors) == 1E assert 2 == 1E +2E -1python/ray/tests/test_memory_pressure.py:540: AssertionError — Reply to this email directly, view it on GitHub <#33430 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFOFZ54X4XV3KNOOORM63HLW5CTZRANCNFSM6AAAAAAV7IXLGA> . You are receiving this because you authored the thread.Message ID: ***@***.***>

-- Best, Clarence

[cadedaniel]

gunna revert now