Lazy rollover causes writes to fail if writer lacks manage index privilege

[axw]

Elasticsearch Version

8.13.0-SNAPSHOT

Installed Plugins

No response

Java Version

bundled

OS Version

N/A

Problem Description

After a lazy rollover, document writes will start failing if the writer lacks index management privileges. This means that lazy rollover is not transparent to writers, and requires additional privileges of writers when compared to synchronous rollover.

Steps to Reproduce

ELASTICSEARCH_URL=http://admin:changeme@localhost:9200

# Create a data stream.
curl -XPUT $ELASTICSEARCH_URL/_data_stream/logs-rollover-bug

# Rollover the data stream lazily.
curl -XPOST $ELASTICSEARCH_URL/logs-rollover-bug/_rollover?lazy

# Create an API Key that can only auto_configure (create on index) data streams, and create documents. This matches what Fleet configures for Elastic Agent.
API_KEY=$(curl -XPOST -H Content-Type:application/json $ELASTICSEARCH_URL/_security/api_key --data-binary '{
  "name": "writer-key",
  "expiration": "1d",   
  "role_descriptors": { 
    "writer": {
      "indices": [
        {
          "names": ["logs-*"],
          "privileges": ["auto_configure", "create_doc"]
        }
      ]
    }
  }
}' | jq -r .encoded)

# Attempt to index a document. This will fail with an error about the `indices:admin/rollover` action.
curl -XPOST -H "Authorization: ApiKey $API_KEY" -H Content-Type:application/json $ELASTICSEARCH_URL/logs-rollover-bug/_doc --data '{"@timestamp": "2024-01-25"}'

Logs (if relevant)

{"error":{"root_cause":[{"type":"security_exception","reason":"action [indices:admin/rollover] is unauthorized for API key id [pdmdP40BivS0Pj5zQ7AE] of user [admin] on indices [logs-rollover-bug], this action is granted by the index privileges [manage_follow_index,manage,all]"}],"type":"security_exception","reason":"action [indices:admin/rollover] is unauthorized for API key id [pdmdP40BivS0Pj5zQ7AE] of user [admin] on indices [logs-rollover-bug], this action is granted by the index privileges [manage_follow_index,manage,all]"},"status":403}

[elasticsearchmachine]

Pinging @elastic/es-data-management (Team:Data Management)

[kpollich]

Hello @elastic/es-data-management - would we be able to get a sense for the level of priority on this ask? Is this likely to land in 8.13 or 8.14? We'd like to determine when we can un-revert elastic/kibana#174790 to make Fleet's rollover process lazy. This will greatly improve Fleet's stability when it comes to integration upgrades, so we're eager to get that change in, but it's blocked by this issue on the ES side.

Thanks for taking a look!

[mattc58]

Hey @kpollich we'll be doing this for 8.13.0. Thanks for the ping on it.