Why does the resource-groups configuration in the cluster not take effect

[xsk123456]

rules.json configuration
{
"rootGroups": [
{
"name": "global",
"softMemoryLimit": "10%",
"hardConcurrencyLimit": 100,
"maxQueued": 1000,
"schedulingPolicy": "weighted",
"jmxExport": true,
"subGroups": [
{
"name": "hive",
"softMemoryLimit": "100%",
"hardConcurrencyLimit": 45,
"maxQueued": 100,
"schedulingWeight": 1,
"jmxExport": true,
"subGroups": [
{
"name": "hh_xx2_001",
"softMemoryLimit": "1%",
"hardConcurrencyLimit": 5,
"maxQueued": 1
},
{
"name": "hadoop",
"softMemoryLimit": "10%",
"hardConcurrencyLimit": 5,
"maxQueued": 100
},
{
"name": "hh_xx2_002",
"softMemoryLimit": "100%",
"hardConcurrencyLimit": 5,
"maxQueued": 100
},
{
"name": "hh_xx2_003",
"softMemoryLimit": "50%",
"hardConcurrencyLimit": 5,
"maxQueued": 100
}
]
}
]
}
],
"selectors": [
{
"group": "global.hive"
}
]
}
The softmemorylimit of subgroups is not valid！
I want to be able to specified as an absolute value or as a percentage of the cluster’s memory.

[wikke]

similar problem to me.

  {
    "rootGroups": [
      {
        "name": "global",
        "maxQueued": 100,
        "softMemoryLimit": "100%",
        "hardConcurrencyLimit": 4,
        "schedulingPolicy": "weighted_fair",
        "subGroups": [
          {
            "name": "dam",
            "maxQueued": 100,
            "softMemoryLimit": "4GB",
            "hardConcurrencyLimit": 2,
            "schedulingWeight": 2
          },
          {
            "name": "default",
            "maxQueued": 100,
            "softMemoryLimit": "100%",
            "hardConcurrencyLimit": 8,
            "schedulingWeight": 8
          }
       ]
     }
    ],
    "selectors": [
     {
       "user": "dam",
       "group": "global.dam"
     },
     {
       "group": "global.default"
     }
    ]
  }

I want dam user with global.dam resource group take no more than 4GB memory, but query exceeded, reaching 13GB!!!

[endlesscyl]

similar problem to me.

  {
    "rootGroups": [
      {
        "name": "global",
        "maxQueued": 100,
        "softMemoryLimit": "100%",
        "hardConcurrencyLimit": 4,
        "schedulingPolicy": "weighted_fair",
        "subGroups": [
          {
            "name": "dam",
            "maxQueued": 100,
            "softMemoryLimit": "4GB",
            "hardConcurrencyLimit": 2,
            "schedulingWeight": 2
          },
          {
            "name": "default",
            "maxQueued": 100,
            "softMemoryLimit": "100%",
            "hardConcurrencyLimit": 8,
            "schedulingWeight": 8
          }
       ]
     }
    ],
    "selectors": [
     {
       "user": "dam",
       "group": "global.dam"
     },
     {
       "group": "global.default"
     }
    ]
  }

I want dam user with global.dam resource group take no more than 4GB memory, but query exceeded, reaching 13GB!!!

---

I have the same problem, have you solved it?

[jack-o-hara]

I'm also seeing this issue where softMemoryLimit is not respected with the same setup as @endlesscyl

In the Trino docs it is described as:

• softMemoryLimit (required): maximum amount of distributed memory this group may use, before new queries become queued. May be specified as an absolute value (i.e. 1GB) or as a percentage (i.e. 10%) of the cluster’s memory.

In the Presto docs it is described as:

• softMemoryLimit: A memory limit that, if exceeded, prevents a group from scheduling new queries until the memory usage value falls below the softMemoryLimit. This can be either an absolute value or a percentage of total cluster memory.

Is this an issue with documentation on the Trino side?

[hashhar]

Resource groups in either Trino or Presto are only admission control tools. They don't control resource usage once query starts running. The limits only apply to next queries - not already executing ones.

Note the doc wording:

before new queries become queued.

For already executing ones the limits are via query_max_memory session property for example - which you can apply to specific resource groups using the Session Property Manager.

[A-little-bit-of-data]

Hello, that means that the query_max_memory parameter is used to manage the running query, and the resource groups are used to set the next query. How can I set the query_max_memory session parameter for each user? Because in resource groups, only the softMemoryLimit parameter can be set for the user to control the memory, but this is not very practical.
Is there any way to directly set the parameter of how much or how much memory a user can occupy in the cluster? Is it a hard indicator?

[lordicecream]

Resource groups in either Trino or Presto are only admission control tools. They don't control resource usage once query starts running. The limits only apply to next queries - not already executing ones.

Note the doc wording:

before new queries become queued.

For already executing ones the limits are via query_max_memory session property for example - which you can apply to specific resource groups using the Session Property Manager.

@hashhar but in the Trino documentation it is not mentioned that query_max_memory is also a session property...it's just a config property...also if I want some queries to be able to utilize only a max amount of memory without killing them, since it's acceptable that those queries from that source will run slowly, is there any way we can implement that?

[hashhar]

memory is not an "optional" resource. You cannot make things run at all without using more memory.

I'd suggest to either look into scaling cluster size dynamically (to increase available memory) or to enforce stricter memory usage limits.