DP ES crash on debug log level

[traumschule]

reporting response times to ES on log levels debug or silly values have wrong type, the request fails and DP crashes:
mapper [fields.responseTimes] cannot be changed from type [text] to [long]

Error should be handled and not crash the node.

STR

1. set up DP provider
2. enable elastic reporting with level debug
   (silly is another option)

  elastic:
    level: debug
    endpoint: https://elastic.joystreamstats.live/

3. restart the node

Errors

Jun 07 23:13:08 us.joystreamstats.live yarn[979590]: elasticsearch indexing error {
Jun 07 23:13:08 us.joystreamstats.live yarn[979590]: type: 'illegal_argument_exception',
Jun 07 23:13:08 us.joystreamstats.live yarn[979590]: reason: 'mapper [fields.responseTimes] cannot be changed from type [text] to [long]'
Jun 07 23:13:08 us.joystreamstats.live yarn[979590]: } {
Jun 07 23:13:08 us.joystreamstats.live yarn[979590]: '@timestamp': '2022-06-07T23:13:01.580Z',
Jun 07 23:13:08 us.joystreamstats.live yarn[979590]: message: 'Mean response times updated',
Jun 07 23:13:08 us.joystreamstats.live yarn[979590]: severity: 'verbose',
Jun 07 23:13:08 us.joystreamstats.live yarn[979590]: fields: {
Jun 07 23:13:08 us.joystreamstats.live yarn[979590]: label: 'NetworkingManager',
Jun 07 23:13:08 us.joystreamstats.live yarn[979590]: responseTimes: [ [Array], [Array], [Array], [Array], [Array], [Array], [Array], [Array] ],
Jun 07 23:13:08 us.joystreamstats.live yarn[979590]: [Symbol(level)]: 'verbose',
Jun 07 23:13:08 us.joystreamstats.live yarn[979590]: [Symbol(splat)]: [ [Object] ],
Jun 07 23:13:08 us.joystreamstats.live yarn[979590]: [Symbol(message)]: '{"@timestamp":"2022-06-07T23:13:01.580Z","log.level":"verbose","message":"Mean response times updated","ecs":{"version":"1.6.0"},"label":"NetworkingManager","responseTimes":[["https://new.xjames.xyz/storage/api/v1%22,371],[%22https://jssp.cryptostarter.info/storage/api/v1%22,361],[%22https://sp1.nodex.club/storage/api/v1%22,360],[%22https://joystream2.yyagi.cloud/storage/api/v1%22,367],[%22https://joystream.godshunter.su/storage/api/v1",368],["https://ipfs.joystreamstats.live/storage/api/v1%22,412],[%22https://whitesocks.ru/storage/api/v1%22,415],[%22https://ru.joystreamstats.live/storage/api/v1%22,491]]%7D'

On another occasion error wasn't handled either:

Jun 07 14:34:56 ipfs.joystreamstats.live yarn[316636]: (node:316636) UnhandledPromiseRejectionWarning: ResponseError: security_exception: [security_exception] Reason: missing authentication credentials for REST request [/_bulk?wait_for_active_shards=1&timeout=5000ms]
Jun 07 14:34:56 ipfs.joystreamstats.live yarn[316636]: at onBody (/home/joystream-storage/joystream/node_modules/@elastic/elasticsearch/lib/Transport.js:355:23)
Jun 07 14:34:56 ipfs.joystreamstats.live yarn[316636]: at IncomingMessage.onEnd (/home/joystream-storage/joystream/node_modules/@elastic/elasticsearch/lib/Transport.js:279:11)
Jun 07 14:34:56 ipfs.joystreamstats.live yarn[316636]: (node:316636) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch().

[yasiryagi]

Similar issues when level is debug

Jun 09 00:32:45 joystream3 yarn[321044]: elasticsearch indexing error {
Jun 09 00:32:45 joystream3 yarn[321044]:   type: 'illegal_argument_exception',
Jun 09 00:32:45 joystream3 yarn[321044]:   reason: 'mapper [fields.responseTimes] cannot be changed from type [text] to [float]'
Jun 09 00:32:45 joystream3 yarn[321044]: } {
Jun 09 00:32:45 joystream3 yarn[321044]:   '@timestamp': '2022-06-09T00:31:44.818Z',
Jun 09 00:32:45 joystream3 yarn[321044]:   message: 'Mean response times updated',
Jun 09 00:32:45 joystream3 yarn[321044]:   severity: 'verbose',
Jun 09 00:32:45 joystream3 yarn[321044]:   fields: {
Jun 09 00:32:45 joystream3 yarn[321044]:     label: 'NetworkingManager',
Jun 09 00:32:45 joystream3 yarn[321044]:     responseTimes: [
Jun 09 00:32:45 joystream3 yarn[321044]:       [Array], [Array],
Jun 09 00:32:45 joystream3 yarn[321044]:       [Array], [Array],
Jun 09 00:32:45 joystream3 yarn[321044]:       [Array], [Array],
Jun 09 00:32:45 joystream3 yarn[321044]:       [Array], [Array],
Jun 09 00:32:45 joystream3 yarn[321044]:       [Array], [Array]
Jun 09 00:32:45 joystream3 yarn[321044]:     ],
Jun 09 00:32:45 joystream3 yarn[321044]:     [Symbol(level)]: 'verbose',
Jun 09 00:32:45 joystream3 yarn[321044]:     [Symbol(splat)]: [ [Object] ],
Jun 09 00:32:45 joystream3 yarn[321044]:     [Symbol(message)]: '{"@timestamp":"2022-06-09T00:31:44.817Z","log.level":"verbose","message":"Mean response times updated","ecs":{"version":"1.6.0"},"label":"NetworkingManager","responseTimes":[["https://joystream2.yyagi.cloud/storage/api/v1",46.6],["https://joystream.godshunter.su/storage/api/v1",28.8],["https://jssp.cryptostarter.info/storage/api/v1",45.9],["https://sp1.nodex.club/storage/api/v1",294.6],["https://new.xjames.xyz/storage/api/v1",86.3],["https://ipfs.joystreamstats.live/storage/api/v1",110.7],["https://whitesocks.ru/storage/api/v1",110.4],["https://ru.joystreamstats.live/storage/api/v1",180.2],["https://joystream.razumv.tech/storage/api/v1",74.1],["https://joystream-storage.bwarelabs.com/storage/api/v1",105.1]]}'
Jun 09 00:32:45 joystream3 yarn[321044]:   },
Jun 09 00:32:45 joystream3 yarn[321044]:   source: 'YYAGI2'
Jun 09 00:32:45 joystream3 yarn[321044]: }
Jun 09 00:32:45 joystream3 yarn[321044]: (node:321044) UnhandledPromiseRejectionWarning: Error: Elasticsearch error
Jun 09 00:32:45 joystream3 yarn[321044]:     at /root/joystream/node_modules/winston-elasticsearch/bulk_writer.js:135:21
Jun 09 00:32:45 joystream3 yarn[321044]:     at runMicrotasks (<anonymous>)
Jun 09 00:32:45 joystream3 yarn[321044]:     at processTicksAndRejections (internal/process/task_queues.js:95:5)
Jun 09 00:32:45 joystream3 yarn[321044]: (node:321044) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 2384)
Jun 09 00:32:46 joystream3 yarn[321044]: 2022-06-09 00:32:46:3246 ContentService verbose: Performing cache cleanup...
Jun 09 00:32:46 joystream3 yarn[321044]: {
Jun 09 00:32:46 joystream3 yarn[321044]:     "supportedObjects": 14082,
Jun 09 00:32:46 joystream3 yarn[321044]:     "objectsInCache": 101
Jun 09 00:32:46 joystream3 yarn[321044]: }
Jun 09 00:32:46 joystream3 yarn[321044]: 2022-06-09 00:32:46:3246 ContentService verbose: Cache cleanup finished
Jun 09 00:32:46 joystream3 yarn[321044]: {
Jun 09 00:32:46 joystream3 yarn[321044]:     "droppedObjects": 0
Jun 09 00:32:46 joystream3 yarn[321044]: }
Jun 09 00:32:47 joystream3 yarn[321044]: elasticsearch indexing error {
Jun 09 00:32:47 joystream3 yarn[321044]:   type: 'illegal_argument_exception',
Jun 09 00:32:47 joystream3 yarn[321044]:   reason: 'mapper [fields.responseTimes] cannot be changed from type [text] to [float]'
Jun 09 00:32:47 joystream3 yarn[321044]: } {
Jun 09 00:32:47 joystream3 yarn[321044]:   '@timestamp': '2022-06-09T00:31:44.818Z',
Jun 09 00:32:47 joystream3 yarn[321044]:   message: 'Mean response times updated',
Jun 09 00:32:47 joystream3 yarn[321044]:   severity: 'verbose',
Jun 09 00:32:47 joystream3 yarn[321044]:   fields: {
Jun 09 00:32:47 joystream3 yarn[321044]:     label: 'NetworkingManager',
Jun 09 00:32:47 joystream3 yarn[321044]:     responseTimes: [
Jun 09 00:32:47 joystream3 yarn[321044]:       [Array], [Array],
Jun 09 00:32:47 joystream3 yarn[321044]:       [Array], [Array],
Jun 09 00:32:47 joystream3 yarn[321044]:       [Array], [Array],
Jun 09 00:32:47 joystream3 yarn[321044]:       [Array], [Array],
Jun 09 00:32:47 joystream3 yarn[321044]:       [Array], [Array]
Jun 09 00:32:47 joystream3 yarn[321044]:     ],
Jun 09 00:32:47 joystream3 yarn[321044]:     [Symbol(level)]: 'verbose',
Jun 09 00:32:47 joystream3 yarn[321044]:     [Symbol(splat)]: [ [Object] ],
Jun 09 00:32:47 joystream3 yarn[321044]:     [Symbol(message)]: '{"@timestamp":"2022-06-09T00:31:44.817Z","log.level":"verbose","message":"Mean response times updated","ecs":{"version":"1.6.0"},"label":"NetworkingManager","responseTimes":[["https://joystream2.yyagi.cloud/storage/api/v1",46.6],["https://joystream.godshunter.su/storage/api/v1",28.8],["https://jssp.cryptostarter.info/storage/api/v1",45.9],["https://sp1.nodex.club/storage/api/v1",294.6],["https://new.xjames.xyz/storage/api/v1",86.3],["https://ipfs.joystreamstats.live/storage/api/v1",110.7],["https://whitesocks.ru/storage/api/v1",110.4],["https://ru.joystreamstats.live/storage/api/v1",180.2],["https://joystream.razumv.tech/storage/api/v1",74.1],["https://joystream-storage.bwarelabs.com/storage/api/v1",105.1]]}'
Jun 09 00:32:47 joystream3 yarn[321044]:   },
Jun 09 00:32:47 joystream3 yarn[321044]:   source: 'YYAGI2'
Jun 09 00:32:47 joystream3 yarn[321044]: }
Jun 09 00:32:47 joystream3 yarn[321044]: elasticsearch indexing error {
Jun 09 00:32:47 joystream3 yarn[321044]:   type: 'illegal_argument_exception',
Jun 09 00:32:47 joystream3 yarn[321044]:   reason: 'mapper [fields.responseTimes] cannot be changed from type [text] to [float]'
Jun 09 00:32:47 joystream3 yarn[321044]: } {
Jun 09 00:32:47 joystream3 yarn[321044]:   '@timestamp': '2022-06-09T00:32:45.202Z',
Jun 09 00:32:47 joystream3 yarn[321044]:   message: 'Mean response times updated',
Jun 09 00:32:47 joystream3 yarn[321044]:   severity: 'verbose',
Jun 09 00:32:47 joystream3 yarn[321044]:   fields: {
Jun 09 00:32:47 joystream3 yarn[321044]:     label: 'NetworkingManager',
Jun 09 00:32:47 joystream3 yarn[321044]:     responseTimes: [
Jun 09 00:32:47 joystream3 yarn[321044]:       [Array], [Array],
Jun 09 00:32:47 joystream3 yarn[321044]:       [Array], [Array],
Jun 09 00:32:47 joystream3 yarn[321044]:       [Array], [Array],
Jun 09 00:32:47 joystream3 yarn[321044]:       [Array], [Array],
Jun 09 00:32:47 joystream3 yarn[321044]:       [Array], [Array]
Jun 09 00:32:47 joystream3 yarn[321044]:     ],
Jun 09 00:32:47 joystream3 yarn[321044]:     [Symbol(level)]: 'verbose',
Jun 09 00:32:47 joystream3 yarn[321044]:     [Symbol(splat)]: [ [Object] ],
Jun 09 00:32:47 joystream3 yarn[321044]:     [Symbol(message)]: '{"@timestamp":"2022-06-09T00:32:45.202Z","log.level":"verbose","message":"Mean response times updated","ecs":{"version":"1.6.0"},"label":"NetworkingManager","responseTimes":[["https://joystream2.yyagi.cloud/storage/api/v1",50.1],["https://joystream.godshunter.su/storage/api/v1",38.1],["https://jssp.cryptostarter.info/storage/api/v1",46.1],["https://sp1.nodex.club/storage/api/v1",296.3],["https://new.xjames.xyz/storage/api/v1",86],["https://ipfs.joystreamstats.live/storage/api/v1",120],["https://whitesocks.ru/storage/api/v1",115.1],["https://ru.joystreamstats.live/storage/api/v1",185],["https://joystream.razumv.tech/storage/api/v1",73.9],["https://joystream-storage.bwarelabs.com/storage/api/v1",99.5]]}'
Jun 09 00:32:47 joystream3 yarn[321044]:   },
Jun 09 00:32:47 joystream3 yarn[321044]:   source: 'YYAGI2'
Jun 09 00:32:47 joystream3 yarn[321044]: }
Jun 09 00:32:47 joystream3 yarn[321044]: (node:321044) UnhandledPromiseRejectionWarning: Error: Elasticsearch error
Jun 09 00:32:47 joystream3 yarn[321044]:     at /root/joystream/node_modules/winston-elasticsearch/bulk_writer.js:135:21
Jun 09 00:32:47 joystream3 yarn[321044]:     at runMicrotasks (<anonymous>)
Jun 09 00:32:47 joystream3 yarn[321044]:     at processTicksAndRejections (internal/process/task_queues.js:95:5)
Jun 09 00:32:47 joystream3 yarn[321044]: (node:321044) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 2385)

[Lezek123]

From what I can see this error doesn't crash the node, it just causes UnhandledPromiseRejectionWarning
The error itself was addressed in #4721