Noobaa command to list bucket/account giving trimmed output in pipe #8111

madhuthorat · 2024-06-05T10:24:36Z

Environment info

NooBaa Version: 5.15.0/1/2/3
Platform: other: RHEL9

Actual behavior

Bucket/account details output getting trimmed when using with pipe

Expected behavior

it should give full output in pipe

Steps to reproduce

More information - Screenshots / Logs / Other output

romayalon · 2024-06-05T10:39:28Z

@madhuthorat could you attach the command you are running and the output ? thanks

PravinRanjan10 · 2024-06-06T07:50:27Z

Hi @romayalon We are trying to read all the noobaa-cli output through the PIPE. but after receiveing few bytes, the returned output is 0 and EOF. That means full output is coming through pipe.

},
{
"_id": "6656d6d3585e0a1deb367e9b",
"name": "bucket-bmw",
"owner_account": "6656d6bcf4781cd093acb9e5",
"system_owner": "account-bmw",
"bucket_owner": "account-bmw",
"versioning": "DISABLED",
"creation_date": "2024-05-29T07:18:43.986Z",
"path": "/mnt/gpfs0/account-bmw/bucket-bmw",
"should_create_underlying_storage": false,
"fs_backend": "GPFS"
},
{
"_id": "6656d6d3585e0a1deb367e9b",
"name": "bucket-bmw",
"owner_account": "6656d6bcf4781cd093acb9e5",
"system_owner": "accou
bytes recieved: 0: EOF

romayalon · 2024-06-06T08:05:40Z

@PravinRanjan10 @madhuthorat Could you add the exact command you are running?
Please try following the issue instructions, this creates redundant ping pongs.
@naveenpaul1 @guymguym this seems to me the same as #7894
don't we need to take the same changes we did for the health script -#7898 and apply them to noobaa-cli?
maybe also this is a repro of #7900

PravinRanjan10 · 2024-06-06T08:13:05Z

@romayalon Here is the command, we are running

Running command: env LC_ALL=C /usr/local/bin/noobaa-cli bucket 2>/dev/null list --wide

madhuthorat · 2024-06-06T08:27:03Z

@PravinRanjan10 @madhuthorat Could you add the exact command you are running? Please try following the issue instructions, this creates redundant ping pongs. @naveenpaul1 @guymguym this seems to me the same as #7894 don't we need to take the same changes we did for the health script -#7898 and apply them to noobaa-cli? maybe also this is a repro of #7900

@romayalon the behavior looks same as seen with health command before #7894 fix.

romayalon · 2024-06-06T17:37:33Z

@PravinRanjan10 @madhuthorat How many buckets/accounts you have in the system?
I want to try reproduce it

PravinRanjan10 · 2024-06-06T18:10:13Z

@PravinRanjan10 @madhuthorat How many buckets/accounts you have in the system? I want to try reproduce it

Approx 5k account/buckets.

madhuthorat · 2024-06-07T11:17:04Z

@romayalon 'as the fix sounds similar to #7894 can we have the fix in 5.15.4 ? currently we are using a workaround which is little cumbersome

romayalon · 2024-06-09T14:05:28Z

@madhuthorat yes, see #8120 backport PR to 5.15.4.

madhuthorat · 2024-06-10T00:57:27Z

@madhuthorat yes, see #8120 backport PR to 5.15.4.

Thank you @romayalon

PravinRanjan10 · 2024-06-12T12:57:47Z

Re-opening as it's not working as expected. Below is some output:

I have 5k buckets

noobaa-cli bucket list 2>/dev/null |jq
parse error: Unfinished JSON term at EOF at line 4594, column 0

Some other output:

  {
    "name": "bkt4110"
  },
  {
    "name": "bkt1969"
  },
  {
    "name": "bkt3442"
  },
  {

bytes recieved: 0: EOF

guymguym · 2024-06-12T19:49:16Z

@PravinRanjan10 Thanks for the quick verification!
Can you please share with us (off github) the output file of noobaa-cli bucket list &>noobaa-cli-bucket-list.log (we need to capture the output of both stdout and stderr...) Thanks

PravinRanjan10 · 2024-06-13T03:52:49Z

@PravinRanjan10 Thanks for the quick verification! Can you please share with us (off github) the output file of noobaa-cli bucket list &>noobaa-cli-bucket-list.log (we need to capture the output of both stdout and stderr...) Thanks

output of: noobaa-cli bucket list &>noobaa-cli-bucket-list.log

noobaa-cli-bucket-list.log

PravinRanjan10 · 2024-06-13T03:54:36Z

output with : noobaa-cli bucket list |jq

[root@node]# noobaa-cli bucket list |jq
load_nsfs_nc_config.setting config.NSFS_NC_CONF_DIR /mnt/cesSharedRoot/ces/s3-config
nsfs: config_dir_path=/mnt/cesSharedRoot/ces/s3-config config.json= {
  ENDPOINT_FORKS: 2,
  ENDPOINT_PORT: 6001,
  ENDPOINT_SSL_PORT: 6443,
  UV_THREADPOOL_SIZE: 16,
  GPFS_DL_PATH: '/usr/lpp/mmfs/lib/libgpfs.so',
  NOOBAA_LOG_LEVEL: 'all',
  NSFS_NC_STORAGE_BACKEND: 'GPFS',
  NSFS_NC_CONFIG_DIR_BACKEND: 'GPFS',
  NSFS_DIR_CACHE_MAX_DIR_SIZE: 536870912,
  NSFS_DIR_CACHE_MAX_TOTAL_SIZE: 1073741824
}
2024-06-13 05:53:17.527005 [PID-1794896/TID-1794896] FS::GPFS GPFS_DL_PATH=/usr/lpp/mmfs/lib/libgpfs.so
2024-06-13 05:53:17.527117 [PID-1794896/TID-1794896] FS::GPFS found GPFS lib file GPFS_DL_PATH=/usr/lpp/mmfs/lib/libgpfs.so
2024-06-13 05:53:17.530533 [PID-1794896/TID-1794896] [L1] FS::set_debug_level 5
Jun-13 5:53:18.002 [/1794896]   [LOG] CONSOLE:: detect_fips_mode: found /proc/sys/crypto/fips_enabled with value 0
2024-06-13 05:53:18.048957 [PID-1794896/TID-1794896] [L1] FS::FSWorker::Begin: Stat _path=/mnt/cesSharedRoot/ces/s3-config
2024-06-13 05:53:18.049386 [PID-1794896/TID-1794905] [L1] FS::FSWorker::Execute: Stat _path=/mnt/cesSharedRoot/ces/s3-config _uid=0 _gid=0 _backend=GPFS
(node:1794896) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023.

Please migrate your code to use AWS SDK for JavaScript (v3).
For more information, check the migration guide at https://a.co/7PzMCcy
(Use `node --trace-warnings ...` to show where the warning was created)
2024-06-13 05:53:18.050086 [PID-1794896/TID-1794905] [L1] FS::FSWorker::Execute: Stat _path=/mnt/cesSharedRoot/ces/s3-config _uid=0 _gid=0 geteuid()=0 getegid()=0Jun-13 5:53:18.050 [/1794896]   [LOG] CONSOLE:: read_rand_seed: reading 32 bytes from /dev/urandom ...
 getuid()=0 getgid()=0
2024-06-13 05:53:18.051139 [PID-1794896/TID-1794905] [L1] FS::FSWorker::Execute: Stat _path=/mnt/cesSharedRoot/ces/s3-config  took: 0.266377 ms
Jun-13 5:53:18.062 [/1794896]   [LOG] CONSOLE:: read_rand_seed: got 32 bytes from /dev/urandom, total 32 ...
Jun-13 5:53:18.063 [/1794896]   [LOG] CONSOLE:: read_rand_seed: closing fd ...
2024-06-13 05:53:18.063688 [PID-1794896/TID-1794896] [L1] FS::Stat::OnOK: _path=/mnt/cesSharedRoot/ces/s3-config _stat_res.st_ino=54534 _stat_res.st_size=8192
Jun-13 5:53:18.063 [/1794896]    [L1] core.cmd.manage_nsfs:: nsfs.check_and_create_config_dirs: config dir exists: /mnt/cesSharedRoot/ces/s3-config
2024-06-13 05:53:18.064243 [PID-1794896/TID-1794896] [L1] FS::FSWorker::Begin: Stat _path=/mnt/cesSharedRoot/ces/s3-config/buckets
2024-06-13 05:53:18.064315 [PID-1794896/TID-1794908] [L1] FS::FSWorker::Execute: Stat _path=/mnt/cesSharedRoot/ces/s3-config/buckets Jun-13 5:53:18.064 [/1794896]   [LOG] CONSOLE:: init_rand_seed: seeding with 32 bytes
_uid=0 _gid=0 _backend=GPFS
2024-06-13 05:53:18.064761 [PID-1794896/TID-1794908] [L1] FS::FSWorker::Execute: Stat _path=/mnt/cesSharedRoot/ces/s3-config/buckets _uid=0 _gid=0 geteuid()=0 getegid()=0 getuid()=0 getgid()=0
2024-06-13 05:53:18.066812 [PID-1794896/TID-1794908] [L1] FS::FSWorker::Execute: Stat _path=/mnt/cesSharedRoot/ces/s3-config/buckets  took: 0.193503 ms
2024-06-13 05:53:18.067332 [PID-1794896/TID-1794896] [L1] FS::Stat::OnOK: _path=/mnt/cesSharedRoot/ces/s3-config/buckets _stat_res.st_ino=14338 _stat_res.st_size=262144
Jun-13 5:53:18.067 [/1794896]    [L1] core.cmd.manage_nsfs:: nsfs.check_and_create_config_dirs: config dir exists: /mnt/cesSharedRoot/ces/s3-config/buckets
2024-06-13 05:53:18.067733 [PID-1794896/TID-1794896] [L1] FS::FSWorker::Begin: Stat _path=/mnt/cesSharedRoot/ces/s3-config/accounts
2024-06-13 05:53:18.067782 [PID-1794896/TID-1794909] [L1] FS::FSWorker::Execute: Stat _path=/mnt/cesSharedRoot/ces/s3-config/accounts _uid=0 _gid=0 _backend=GPFS
2024-06-13 05:53:18.067844 [PID-1794896/TID-1794909] [L1] FS::FSWorker::Execute: Stat _path=/mnt/cesSharedRoot/ces/s3-config/accounts _uid=0 _gid=0 geteuid()=0 getegid()=0 getuid()=0 getgid()=0
2024-06-13 05:53:18.067998 [PID-1794896/TID-1794909] [L1] FS::FSWorker::Execute: Stat _path=/mnt/cesSharedRoot/ces/s3-config/accounts  took: 0.089916 ms
2024-06-13 05:53:18.068046 [PID-1794896/TID-1794896] [L1] FS::Stat::OnOK: _path=/mnt/cesSharedRoot/ces/s3-config/accounts _stat_res.st_ino=14339 _stat_res.st_size=16384
Jun-13 5:53:18.068 [/1794896]    [L1] core.cmd.manage_nsfs:: nsfs.check_and_create_config_dirs: config dir exists: /mnt/cesSharedRoot/ces/s3-config/accounts
2024-06-13 05:53:18.068318 [PID-1794896/TID-1794896] [L1] FS::FSWorker::Begin: Stat _path=/mnt/cesSharedRoot/ces/s3-config/access_keys
2024-06-13 05:53:18.068362 [PID-1794896/TID-1794910] [L1] FS::FSWorker::Execute: Stat _path=/mnt/cesSharedRoot/ces/s3-config/access_keys _uid=0 _gid=0 _backend=GPFS
2024-06-13 05:53:18.068425 [PID-1794896/TID-1794910] [L1] FS::FSWorker::Execute: Stat _path=/mnt/cesSharedRoot/ces/s3-config/access_keys _uid=0 _gid=0 geteuid()=0 getegid()=0 getuid()=0 getgid()=0
2024-06-13 05:53:18.068545 [PID-1794896/TID-1794910] [L1] FS::FSWorker::Execute: Stat _path=/mnt/cesSharedRoot/ces/s3-config/access_keys  took: 0.070583 ms
2024-06-13 05:53:18.068589 [PID-1794896/TID-1794896] [L1] FS::Stat::OnOK: _path=/mnt/cesSharedRoot/ces/s3-config/access_keys _stat_res.st_ino=14340 _stat_res.st_size=65536
Jun-13 5:53:18.068 [/1794896]    [L1] core.cmd.manage_nsfs:: nsfs.check_and_create_config_dirs: config dir exists: /mnt/cesSharedRoot/ces/s3-config/access_keys
2024-06-13 05:53:18.068834 [PID-1794896/TID-1794896] [L1] FS::FSWorker::Begin: Stat _path=/var/run/noobaa-nsfs/wal
2024-06-13 05:53:18.068879 [PID-1794896/TID-1794911] [L1] FS::FSWorker::Execute: Stat _path=/var/run/noobaa-nsfs/wal _uid=0 _gid=0 _backend=GPFS
2024-06-13 05:53:18.068927 [PID-1794896/TID-1794911] [L1] FS::FSWorker::Execute: Stat _path=/var/run/noobaa-nsfs/wal _uid=0 _gid=0 geteuid()=0 getegid()=0 getuid()=0 getgid()=0
2024-06-13 05:53:18.069014 [PID-1794896/TID-1794911] [L1] FS::FSWorker::Execute: Stat _path=/var/run/noobaa-nsfs/wal  took: 0.038081 ms
2024-06-13 05:53:18.069059 [PID-1794896/TID-1794896] [L1] FS::FSWorker::OnError: Stat _path=/var/run/noobaa-nsfs/wal  error.Message()=Invalid argument
Jun-13 5:53:18.069 [/1794896]    [L1] core.cmd.manage_nsfs:: nsfs.check_and_create_config_dirs: could not create pre requisite path /var/run/noobaa-nsfs/wal
2024-06-13 05:53:18.069747 [PID-1794896/TID-1794896] [L1] FS::FSWorker::Begin: Readdir _path=/mnt/cesSharedRoot/ces/s3-config/buckets
2024-06-13 05:53:18.069859 [PID-1794896/TID-1794912] [L1] FS::FSWorker::Execute: Readdir _path=/mnt/cesSharedRoot/ces/s3-config/buckets _uid=0 _gid=0 _backend=GPFS
2024-06-13 05:53:18.070098 [PID-1794896/TID-1794912] [L1] FS::FSWorker::Execute: Readdir _path=/mnt/cesSharedRoot/ces/s3-config/buckets _uid=0 _gid=0 geteuid()=0 getegid()=0 getuid()=0 getgid()=0
2024-06-13 05:53:18.073272 [PID-1794896/TID-1794912] [L1] FS::FSWorker::Execute: Readdir _path=/mnt/cesSharedRoot/ces/s3-config/buckets  took: 2.68073 ms
2024-06-13 05:53:18.073537 [PID-1794896/TID-1794896] [L1] FS::FSWorker::OnOK: Readdir _path=/mnt/cesSharedRoot/ces/s3-config/buckets


parse error: Unfinished JSON term at EOF at line 4594, column 0

guymguym · 2024-06-13T05:59:57Z

Hi @PravinRanjan10
I checked the output file you provided and after stripping off the log prints I can parse it with jq successfully.
Which version of jq are you using?
Does this work?

noobaa-cli bucket list 2>/dev/null >bucket-list-output
jq <bucket-list-output

PravinRanjan10 · 2024-06-13T08:01:08Z

Hi @PravinRanjan10 I checked the output file you provided and after stripping off the log prints I can parse it with jq successfully. Which version of jq are you using? Does this work?
noobaa-cli bucket list 2>/dev/null >bucket-list-output
jq <bucket-list-output

@guymguym
Jq VERSION:

jq-1.6-15.el9.x86_64

Yes, if number of buckets are less (around <=2.5k), it works fine with pipe and jq.

guymguym · 2024-06-13T08:34:01Z

@PravinRanjan10 can you test without a pipe like i suggested above?

romayalon · 2024-06-13T08:37:43Z

@PravinRanjan10 @guymguym
I tried it this way - cat noobaa-cli-bucket-list.log | jq
and it worked, I don't see any issue

PravinRanjan10 · 2024-06-13T10:44:00Z

Actually, the problem is with PIPE only. In code we are trying to collect the output of noobaa-cli bucket list and parse. The problem is, if size of bucket list increases then we need to use stdout pipe to collect those output and parse.

guymguym · 2024-06-16T14:07:05Z

@romayalon I was trying to reproduce somehow with this script below but it doesn't reproduce (not even with 100,000).
So we need to recreate this 5000 buckets case on dev env and investigate further.

> node -e '
  json = (name, count) => JSON.stringify({ response: { reply: Array(count).fill().map((x,i)=>({name:name+"-"+i})) } }, null, 2);
  write = (str) => { process.stdout.write(str + "\n", () => process.exit(0)) };
  write(json("buckets", 5000));
' | jq ".response.reply[-3:][].name"

"buckets-4997"
"buckets-4998"
"buckets-4999"

romayalon · 2024-06-17T08:04:49Z

@PravinRanjan10 I created 6000 buckets and couldn't see your issue, can we have access to your machine?

romayalon · 2024-06-20T08:52:51Z

Updating that I had a call with @PravinRanjan10 and after upgrading to the latest RPM we couldn't see the issue again.
Pravin is currently running some more tests for verifying we can close this one

romayalon · 2024-06-20T14:38:58Z

Closing the issue per @PravinRanjan10 confirmation.

romayalon mentioned this issue Jun 6, 2024

NC | CLI | trimmed output fix #8117

Merged

2 tasks

romayalon closed this as completed in #8117 Jun 9, 2024

PravinRanjan10 reopened this Jun 12, 2024

romayalon closed this as completed Jun 20, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Noobaa command to list bucket/account giving trimmed output in pipe #8111

Noobaa command to list bucket/account giving trimmed output in pipe #8111

madhuthorat commented Jun 5, 2024

romayalon commented Jun 5, 2024

PravinRanjan10 commented Jun 6, 2024

romayalon commented Jun 6, 2024

PravinRanjan10 commented Jun 6, 2024

madhuthorat commented Jun 6, 2024

romayalon commented Jun 6, 2024

PravinRanjan10 commented Jun 6, 2024

madhuthorat commented Jun 7, 2024

romayalon commented Jun 9, 2024

madhuthorat commented Jun 10, 2024

PravinRanjan10 commented Jun 12, 2024 •

edited

Loading

guymguym commented Jun 12, 2024

PravinRanjan10 commented Jun 13, 2024

PravinRanjan10 commented Jun 13, 2024 •

edited

Loading

guymguym commented Jun 13, 2024

PravinRanjan10 commented Jun 13, 2024

guymguym commented Jun 13, 2024

romayalon commented Jun 13, 2024

PravinRanjan10 commented Jun 13, 2024

guymguym commented Jun 16, 2024

romayalon commented Jun 17, 2024

romayalon commented Jun 20, 2024

romayalon commented Jun 20, 2024

Noobaa command to list bucket/account giving trimmed output in pipe #8111

Noobaa command to list bucket/account giving trimmed output in pipe #8111

Comments

madhuthorat commented Jun 5, 2024

Environment info

Actual behavior

Expected behavior

Steps to reproduce

More information - Screenshots / Logs / Other output

romayalon commented Jun 5, 2024

PravinRanjan10 commented Jun 6, 2024

romayalon commented Jun 6, 2024

PravinRanjan10 commented Jun 6, 2024

madhuthorat commented Jun 6, 2024

romayalon commented Jun 6, 2024

PravinRanjan10 commented Jun 6, 2024

madhuthorat commented Jun 7, 2024

romayalon commented Jun 9, 2024

madhuthorat commented Jun 10, 2024

PravinRanjan10 commented Jun 12, 2024 • edited Loading

guymguym commented Jun 12, 2024

PravinRanjan10 commented Jun 13, 2024

PravinRanjan10 commented Jun 13, 2024 • edited Loading

guymguym commented Jun 13, 2024

PravinRanjan10 commented Jun 13, 2024

guymguym commented Jun 13, 2024

romayalon commented Jun 13, 2024

PravinRanjan10 commented Jun 13, 2024

guymguym commented Jun 16, 2024

romayalon commented Jun 17, 2024

romayalon commented Jun 20, 2024

romayalon commented Jun 20, 2024

PravinRanjan10 commented Jun 12, 2024 •

edited

Loading

PravinRanjan10 commented Jun 13, 2024 •

edited

Loading