Cubestore: Queue storage error, Websocket error: Capacity

[gojumprope]

Describe the bug
Previously using redis does not have this error, after remove redis and using cubestore then got this error
Cubestore: Websocket error: Capacity(MessageTooLong { size: 47899532, max_size: 16777216 })
ERROR: Internal: Arrow error: Compute error: Internal: Error [ERR_STREAM_DESTROYED]: Cannot call write after a stream was destroyed

To Reproduce
Steps to reproduce the behavior:

1. DBeaver connect Cube postgres SQL
2. Run a query that return many rows and columns
3. select * from testing_abc_occupancy where Time >= cast('2023-03-01 00:00:00' as timestamp) and Time < cast('2023-03-11 23:59:59' as timestamp) + interval '1 second' and TimeFilter > cast('2000-01-01 10:00:00' as timestamp) - interval '1 millisecond' and TimeFilter < cast('2000-01-01 19:00:00' as timestamp) + interval '1 millisecond' + interval '1 second' and CompanyId = 12345 and BranchId in (45678)
4. Erros:
   { "message": "Queue storage error", "queryKey": [ "SELECT\n testing_abc_occupancy.companyid testing_abc_occupancy___company_id, testing_abc_occupancy.branchid testing_abc_occupancy___branch_id, testing_abc_occupancy.areaid testing_abc_occupancy___area_id, testing_abc_occupancy.branchname testing_abc_occupancy___branch_name, testing_abc_occupancy.areaname testing_abc_occupancy___area_name, testing_abc_occupancy.areagroupid testing_abc_occupancy___area_group_id, testing_abc_occupancy.areagroupname testing_abc_occupancy___area_group_name, testing_abc_occupancy.areagrouptypename testing_abc_occupancy___area_group_type_name, testing_abc_occupancy.areatype testing_abc_occupancy___area_type, testing_abc_occupancy.occupiedspacecount testing_abc_occupancy___occupied_space_count, testing_abc_occupancy.occupiedduration testing_abc_occupancy___occupied_duration, testing_abc_occupancy.abovecapacityduration testing_abc_occupancy___above_capacity_duration, testing_abc_occupancy.fullyoccupiedduration testing_abc_occupancy___fully_occupied_duration, testing_abc_occupancy.duration testing_abc_occupancy___duration, testing_abc_occupancy.spacecount testing_abc_occupancy___space_count, testing_abc_occupancy.Latitude testing_abc_occupancy___latitude, testing_abc_occupancy.Longitude testing_abc_occupancy___longitude, testing_abc_occupancy.pso05 testing_abc_occupancy___p_s_o05, testing_abc_occupancy.pso05_t1 testing_abc_occupancy___p_s_o05__t1, testing_abc_occupancy.pso05_t2 testing_abc_occupancy___p_s_o05__t2, testing_abc_occupancy.pso05_t3 testing_abc_occupancy___p_s_o05__t3, testing_abc_occupancy.sso02_t1 testing_abc_occupancy___s_s_o02__t1, cast(concat('2000-01-01 ', DATE_FORMAT(__time,'%H:%i:%s')) as datetime) testing_abc_occupancy___time_filter, testing_abc_occupancy.__time testing_abc_occupancy___time, testing_abc_occupancy.dayofweek testing_abc_occupancy___day_of_week, testing_abc_occupancy.utilisedescription testing_abc_occupancy___utilise_description, testing_abc_occupancy.isoperating testing_abc_occupancy___is_operating, count(1) testing_abc_occupancy___count\n FROM\n (\n select\n av.companyid,\n av.branchid,\n av.areaid,\n av.branchname,\n av.areaname,\n av.areagroupid ,\n av.areagroupname ,\n av.areagrouptypename,\n COALESCE (aaodmv.areatype, 0) areatype,\n COALESCE (aaodmv.occupiedspacecount, 0) occupiedspacecount,\n COALESCE (aaodmv.occupiedduration, 0) occupiedduration,\n COALESCE (aaodmv.abovecapacityduration, 0) abovecapacityduration,\n COALESCE (aaodmv.fullyoccupiedduration, 0) fullyoccupiedduration,\n COALESCE (aaodmv.duration, 0) duration,\n COALESCE (aaodmv.spacecount, 0) spacecount,\n rf.Latitude,\n rf.Longitude,\n COALESCE (aaodmv.pso05, 0) as pso05,\n atv1.thresholdvalue as pso05_t1,\n atv2.thresholdvalue as pso05_t2,\n atv3.thresholdvalue as pso05_t3,\n atv4.thresholdvalue as sso02_t1,\n aaodmv.__time,\n WEEKDAY(aaodmv.__time) + 1 dayofweek,\n aaodmv.UtilStatus utilisedescription,\n aaodmv.isoperating\nfrom dbo.areagroup_view av\ninner join bddf.ddbranches rf on rf.id = av.branchid and rf.IsDeleted = 0\nleft join\n(\n select *\n from dbo.test_aggregate_occupancy_data_1h_view\n where companyid = ? and branchid = ?\n and __time >= TIMESTAMP(convert_tz(?, '+00:00', @@session.time_zone)) AND __time <= TIMESTAMP(convert_tz(?, '+00:00', @@session.time_zone))\n and cast(concat('2000-01-01 ', DATE_FORMAT(__time,'%H:%i:%s')) as datetime) > TIMESTAMP(convert_tz(?, '+00:00', @@session.time_zone)) AND cast(concat('2000-01-01 ', DATE_FORMAT(__time,'%H:%i:%s')) as datetime) < TIMESTAMP(convert_tz(?, '+00:00', @@session.time_zone))\n and 1 = 1\n) aaodmv on aaodmv.companyid = av.companyid and aaodmv.areaid = av.areaid \nleft join dbo.area_threshold_view atv1 on atv1.companyid = ? and atv1.areaid = av.areaid and atv1.thresholdtype = 'so05_t1'\nleft join dbo.area_threshold_view atv2 on atv2.companyid = ? and atv2.areaid = av.areaid and atv2.thresholdtype = 'so05_t2'\nleft join dbo.area_threshold_view atv3 on atv3.companyid = ? and atv3.areaid = av.areaid and atv3.thresholdtype = 'so05_t3'\nleft join dbo.area_threshold_view atv4 on atv4.companyid = ? and atv4.areaid = av.areaid and atv4.thresholdtype = 'sso02_t1'\nwhere av.companyid = ? and av.branchid = ? and \n1 = 1) AS testing_abc_occupancy WHERE (testing_abc_occupancy.__time >= TIMESTAMP(convert_tz(?, '+00:00', @@session.time_zone)) AND testing_abc_occupancy.__time <= TIMESTAMP(convert_tz(?, '+00:00', @@session.time_zone))) AND (cast(concat('2000-01-01 ', DATE_FORMAT(__time,'%H:%i:%s')) as datetime) > TIMESTAMP(convert_tz(?, '+00:00', @@session.time_zone))) AND (cast(concat('2000-01-01 ', DATE_FORMAT(__time,'%H:%i:%s')) as datetime) < TIMESTAMP(convert_tz(?, '+00:00', @@session.time_zone))) AND (testing_abc_occupancy.companyid = ?) AND (testing_abc_occupancy.branchid = ?) GROUP BY 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27 ORDER BY 28 DESC LIMIT 500000", [ "12345", "45678", "2023-03-18T00:00:00.000", "2023-03-24T23:59:59.999", "2000-01-01T09:59:59.999", "2000-01-01T19:00:01.000", "12345", "12345", "12345", "12345", "12345", "45678", "2023-03-18T00:00:00.000", "2023-03-24T23:59:59.999", "2000-01-01T09:59:59.999", "2000-01-01T19:00:01.000", "12345", "45678" ], [] ], "requestId": "ea90e5ab-8405-4c4b-9024-c3af9d11db54-span-37", "error": "Error: write EPIPE\n at WriteWrap.onWriteComplete [as oncomplete] (node:internal/stream_base_commons:94:16)", "queuePrefix": "SQL_QUERY_STANDALONE_doris" }

Expected behavior
should not have error

Screenshots

Version:
v0.32.14

[igorlukanin]

@ovr Have this been addressed? Should @gojumprope upgrade and test the new version?

[gojumprope]

@ovr @igorlukanin Hi, the issue was fixed after I upgraded to v0.32.25 and set the configuration for CUBESTORE_TRANSPORT_MAX_MESSAGE_SIZE and CUBESTORE_TRANSPORT_MAX_FRAME_SIZE to higher value. Thanks

[mstruser]

@gojumprope Hi, I am having the same issue, I have upgraded my version, What component should I set this configuration for and what value did you use? I am using docker compose env variables should it go under cubestore_router?

[gojumprope]

@mstruser Hi, yes, you should put under cubestore_router. Below is the cubestore_router settings I set in docker compose.

cubestore_router:
restart: always
image: cubejs/cubestore:v0.32.25
environment:
- CUBESTORE_WORKERS=cubestore_worker_1:10001,cubestore_worker_2:10002
- CUBESTORE_REMOTE_DIR=/cube/data
- CUBESTORE_META_PORT=9999
- CUBESTORE_SERVER_NAME=cubestore_router:9999
- CUBESTORE_TRANSPORT_MAX_MESSAGE_SIZE=268435456 # 256 << 20 bit shift because cube max limit is 256
- CUBESTORE_TRANSPORT_MAX_FRAME_SIZE=268435456 # 256 << 20 bit shift
volumes:
- .cubestore/router:/cube/data