Connect taskConfigs appears to choke on large config

[slominskir]

If I try to add a large set of channels (25,500) the Connector will run (status RUNNING), but the tasks array will be empty (no task will be started). I'm not seeing any errors in the log file either. The workaround is to use a patch code branch config-from-file that writes/reads the configuration from a file. Using a smaller config works too.

See:
https://stackoverflow.com/questions/72296071/kafka-connect-tasks-empty-with-large-config

[slominskir]

I've added a Docker compose file that reproduces the issue. Launch with:

git clone https://github.com/JeffersonLab/epics2kafka
cd epics2kafka
docker compose -f test-large.yml up

Observe the DEBUG task start() is never called. Check the Connect status in another terminal to observe empty task array:

docker exec connect bash -c "/scripts/show-status.sh"

{
  "name": "ca-source",
  "connector": {
    "state": "RUNNING",
    "worker_id": "172.18.0.5:8083"
  },
  "tasks": [],
  "type": "source"
}

[slominskir]

Must add max.request.size=5242880 to connect-distributed.properties and max.message.bytes=5242880 to topic configuration of topic connect-configs.

See: Kafka Users Discussion