Commits on Feb 26, 2018

1. batch updates to the temporary db when publishing …

   updates with contents generation were super syscall-heavy. for each path
   in a package (so at least 2-4, but ordinarily >4) we'd do a db.Put in
   ContentsIndex which results in one syscall.Write. so, for every package in
   a published repo we'd have to do *at least* 2 but ordinarily >4 syscalls.
   this gets abysmally slow very quickly depending on the available system
   specs.
   
   instead, start a batch inside each package and finish it when we are done
   with the package. this should keep the memory footprint negligible, but
   reduce the write() calls from N to 1.
   
   on one of KDE's servers I have seen update publishing of 7600 packages go
   from ~28s to ~9s when using batch putting on an HDD.
   on my local system the same set of packages go from ~14s to ~6s on an SSD.
   (all inodes in cache in both cases)

   

   hsitter committed Feb 26, 2018
   Configuration menu

   • View commit details
   • Copy full SHA for 9125745
   • Browse repository at this point

   Copy the full SHA

   9125745 View commit details

   Browse the repository at this point in the history