Indices with index.blocks.read_only set to true return OK (HTTP-200) on _bulk requests

[PereBal]

Bug report

Elasticsearch version (bin/elasticsearch --version): 7.5.1

Plugins installed: []

JVM version (java -version):
openjdk version "13.0.1" 2019-10-15
OpenJDK Runtime Environment AdoptOpenJDK (build 13.0.1+9)
OpenJDK 64-Bit Server VM AdoptOpenJDK (build 13.0.1+9, mixed mode, sharing)

OS version (uname -a if on a Unix-like system):
Linux deathstar 4.15.0-88-generic #88-Ubuntu SMP Tue Feb 11 20:11:34 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Current Behavior:
When index.blocks.read_only is set to true, _bulk requests to that index return OK 200

Expected Behavior:
Elasticsearch returns FORBIDDEN 403 when a _bulk request is performed against an index with index.blocks.read_only set to true as they do when cluster.blocks.read_only is set or what single document inserts do in both scenarios.

Steps to reproduce:

# Start a cluster
$ docker run --rm --hostname=$HOSTNAME --env 'node.name=deathstar' --env 'cluster.name=docker-cluster' --env 'discovery.type=single-node' --env 'xpack.security.enabled=true' --env 'ELASTIC_PASSWORD=demo' --net=host -it docker.elastic.co/elasticsearch/elasticsearch:7.5.1 &

# Create an index
$ curl -XPUT -su elastic:demo localhost:9200/demo | jq           
{
  "acknowledged": true,
  "shards_acknowledged": true,
  "index": "demo"
}

# Mark as read_only
$ curl -XPUT -svu elastic:demo localhost:9200/demo/_settings -H'content-type:application/json' -d '{"index.blocks.read_only": true}' | jq                                                                 
{
  "acknowledged": true
}

# Make a _bulk request (buggy behavior)
$ echo '{"index": {"_index": "demo"}}\n{"a":1}' \
   | curl -XPUT -vsu elastic:demo localhost:9200/_bulk -H'content-type:application/x-ndjson' --data-binary @- | jq          
...
< HTTP/1.1 200 OK
< content-type: application/json; charset=UTF-8
< content-length: 235
< 
{ [235 bytes data]
* Connection #0 to host localhost left intact
{
  "took": 0,
  "errors": true,
  "items": [
    {
      "index": {
        "_index": "demo",
        "_type": "_doc",
        "_id": "EX_Lm3ABtEtWcnD73zAf",
        "status": 403,
        "error": {
          "type": "cluster_block_exception",
          "reason": "index [demo] blocked by: [FORBIDDEN/5/index read-only (api)];"
        }
      }
    }
  ]
}

# Block writes cluster wide
$ curl -XPUT -su elastic:demo localhost:9200/_cluster/settings -H'content-type:application/json' -d '{"transient": {"cluster.blocks.read_only": true}}' | jq 
{
  "acknowledged": true,
  "persistent": {},
  "transient": {
    "cluster": {
      "blocks": {
        "read_only": "true"
      }
    }
  }
}

# Try to make a _bulk request ( expected behavior in this situation )
$ echo '{"index": {"_index": "demo"}}\n{"a":1}' \
   | curl -XPUT -vsu elastic:demo localhost:9200/_bulk -H'content-type:application/x-ndjson' --data-binary @- | jq                          
...
< HTTP/1.1 403 Forbidden
< content-type: application/json; charset=UTF-8
< content-length: 231
< 
{ [231 bytes data]
* Connection #0 to host localhost left intact
{
  "error": {
    "root_cause": [
      {
        "type": "cluster_block_exception",
        "reason": "blocked by: [FORBIDDEN/6/cluster read-only (api)];"
      }
    ],
    "type": "cluster_block_exception",
    "reason": "blocked by: [FORBIDDEN/6/cluster read-only (api)];"
  },
  "status": 403
}

Impact: This kind of behaviors make managing elasticsearch unnecessarily harder. While it is tricky to decide which status code needs to be returned on a _bulk response, it does make sense to return the worst possible one and between 200 and 403, 403 seems like a clear candidate.

[DaveCTurner]

There is no HTTP response code that means exactly the right thing here, so every choice is a compromise. We have as a team settled on 200 OK for bulk requests successfully processed by the coordinating node, even if some or all of the individual items do not succeed (see e.g. #28522 (comment) and #29169 (comment)). We recognise that this means that clients cannot use the status code of the overall response to react appropriately to failures of bulk requests, but there are too many cases where a different status code would also result in the wrong reaction. Choosing the "worst possible" code doesn't really work, even where it is well-defined. It turns out to be simpler for all concerned if we uniformly ask clients always to look at the status of each item in a bulk response as we do today.