Ls performance changes

[Stebalien]

I'd like to make the following changes to the ls command for performance reasons:

1. Add a -q flag that only returns names. Base58 encoding a bunch of CIDs is actually quite a time-consuming operation. It would be nice to have a variant that just returns names.
2. Replace --resolve-types with --type=[always|local|never]. Currently, there's no way to say "don't resolve this type, even if we have the object". Having that option would allow us to avoid hitting the datastore for every file listed. (note, --resolve-types will only be deprecated, not removed).

[kevina]

Sounds Good To me.

[mib-kd743naq]

Base58 encoding a bunch of CIDs is actually quite a time-consuming operation.

Whoa... is this really the case? Using GMP bindings from a rather slow language on a crappy laptop I get over 260,000/s conversions of a random 70 byte string ( chosen to cover potential super-long 512 bit CIDs ). I suspect results from within golang will be even better. Or perhaps requiring a C library itself is an issue...?

~$ perl -Mwarnings -Mstrict -MBenchmark=:hireswallclock -MMath::GMPz -e '

  open my $fh, "<", "/dev/urandom";

  my $gmpz = Math::GMPz::Rmpz_init2_nobless( 8 * 70 );

  sub encode_base58 {
    my $cid = shift;

    Math::GMPz::Rmpz_import(
      $gmpz,
      length($cid),
      1, 1, 0, 0,   # magic GMPz constants
      $cid,
    );
                                                                                
    my $rv = Math::GMPz::Rmpz_get_str( $gmpz, 58 );
                                                                          
    # GMPz uses a linear alphabet:                      0..9,"A".."Z","a".."v"
    # Bitcoin is more "human friendly": grep /[^0IOl]/, 0..9,"A".."Z","a".."z"
    $rv =~ tr
      {0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuv}
      {123456789ABCDEFGHJKLMNPQRSTUVWXYZabcdefghijkmnopqrstuvwxyz}
    ;

    $rv;
  }

  timethis( -3, sub {
    sysread( $fh, my $bytes_representing_huge_cid, 70 );

    encode_base58( $bytes_representing_huge_cid );
  });
'
timethis for 3: 3.0228 wallclock secs ( 2.36 usr +  0.66 sys =  3.02 CPU) @ 265435.43/s (n=801615)

[Stebalien]

Whoa... is this really the case?

When listing a directory with 100,000 files, 100 files per ms is actually kind of slow.

[mib-kd743naq]

When listing a directory with 100,000 files...

Given my figure of 265k/s the conversion of 100,000 CIDs twice the size of a typical one should take 300ms. The actual time is likely a fifth of that, as perl itself is embarrassingly slow, it was just faster to code a single command one can paste into their terminal to validate things.

All I am trying to point out is that you are almost certainly solving the wrong problem: if Base58 conversion is a notable factor in your measurements - the converter is seriously flawed.