Loading large JSON data causes error because of Firefox threshold

[eharkins]

jupyterlab/jupyterlab#4015

[metasoarous]

I see a few things that might help resolve this:

• Right now the json we spit out is pretty printed into the file, which will make the string that gets loaded up in JS way bigger than it needs to be. Changing this in the build_olmsted_data.py script of cft will likely greatly improve this.
• There may also be a way to zip/compress the json data which may resolve the issue.
• Right now for seed-lineage-pruning reconstructions, we end up sending along sequence metadata for all of the sequences, not just the sequences chosen as representatives for the seed lineage. For minadcl, we do an aggregation step that leaves us with metadata only for the representative sequences, but its merely an oversight that because this step isn't necessary for the seed-lineage trees, we end up not reducing the dimensionality of the metadata. This should be a relatively easy fix in the cft pipeline.
• Ultimately, breaking up the data payloads will be the real silver bullet here. The fundamental problem is that we're sending over not just all of the clonal families, but all of their trees, sequences, and sequence metadata. If instead we only loaded this additional data lazily as folks click on specific trees, this problem would be largely mitigated. The cost of doing this is a) much more architectural complexity loading data on demand, and b) tree/alignment details won't be able to render instantly if we do this because they'll have to wait for the data to get in. My suggestion is that we push this off as long as we can, and see where we end up.

[metasoarous]

@eharkins If you are keen to get your fingers dirty with some data processing work, you could take a stab at nailing the third of the steps above (filtering seed-lineage downsampled metadata).

[eharkins]

Should we close this for now / now that matsengrp/cft#249 has been merged? Don't see an error on Firefox but I also never checked to see if I could reproduce the error in the first place. @metasoarous?

[metasoarous]

I guess for now let's close this, since the issue isn't pressing anymore now that matsengrp/cft#249 has been merged. One of these issues has already broken off into #42. The other two could potentially be helpful to consider eventually, but aren't high priority.

[metasoarous]

Since increasing the number of clonal families sampled per unseeded sample, we're now having this issue again. The best solution is probably to #42 (split up clonal family details into separate files, which only get loaded once the given clonal family is selected). This will add a bit of delay to those detail views loading up for the first time, but will also reduce the amount of time it takes for the initial dataset load, and solve the FF loading problem for large datasets.