Automatically parse CTF parameters in RELION 3.1 star files

[zhonge]

Parse the data_optics table in RELION 3.1 star files instead of requiring --kv/-w/etc. command line arguments. There might be packages that already do this, e.g. gemmi.

[Guillawme]

Hi Ellen,

If I may suggest, take a look at the starfile library, if you haven't already seen it; one of its goals is compatibility with RELION's star files: https://github.com/alisterburt/starfile
It produces a pandas dataframe for each table in an input star file, and can also write a star file back from a dataframe you generated in your program.

I am analyzing my first runs of cryoDRGN today, and am very excited about the possibilities (and the initial results I got). Thank you for making it so easy to use! 👍

[EB-kim]

Hi Ellen,
I am trying to run the cryoDRGN using my own data. Can you explain what --w and --ps mean?

[Guillawme]

You can find an explanation by running the command cryodrgn parse_ctf_star --help: -w is the amplitude contrast, --ps is the phase shift (you probably only need this if your data were collected with a phase plate; inspect your star file in a text editor to determine if the _rlnPhaseShift contains values other than 0).

[EB-kim]

I see. Thank you. Do you ever get an error for the amp in the high resolution training because amp is undefined?

[Guillawme]

I don't know, I have not installed AMP nor tried to use it.

[EB-kim]

ok. I think amp is the part of the NVIDIA architecture . Did you not use it for the high resolution training just like in the tutorial?

[Guillawme]

I simply haven't needed the speedup enough to motivate spending time installing it and making sure it works as intended. When I run trainings overnight or over a weekend, it would make no difference to me if they took half the time to run: I will only see the results on the next workday anyway (yes, I know I could queue jobs and get more of them to complete if each one ran faster; I simply haven't had the need so far, the bottleneck is more with taking a close look at all the results generated and making sense of them).

[zhonge]

See this comment for more info on apex/amp: #54 (comment)
I believe it is also documented in the tutorial and the README.

In general, can you open a new issue if you have a question that is unrelated to the original subject? Helps to stay organized! Thanks!

[EB-kim]

Oh, I apologize. I just opened a new issuee about unboundlocal error.

[michal-g]

For v3.4.0 I ended up creating an updated version of cryodrgn.starfile.Starfile that can return optics parameters from the optics table (or the primary data table if not found in optics) as a particle-wise array. This allows for logic in parse_ctf_star that will read in these fields from the given .star file, or override them if passed as arguments from command line.

As one of the testing examples, I am using a new testing .star file tests/data/relion31.6opticsgroups.star:

# Created 2024-05-20 17:39:07.660886

data_optics

loop_
_rlnOpticsGroup
_rlnOpticsGroupName
_rlnAmplitudeContrast
_rlnSphericalAberration
_rlnVoltage
_rlnImagePixelSize
_rlnImageSize
_rlnImageDimensionality
_rlnCtfDataAreCtfPremultiplied
_rlnBeamTiltX
_rlnBeamTiltY
_rlnOddZernike
_rlnEvenZernike
_rlnMagMat00
_rlnMagMat01
_rlnMagMat10
_rlnMagMat11
_rlnMicrographPixelSize
1 opticsGroup5 0.100000 2.700000 200.000000 1.250000 196 2 0 0.657334 0.884982 [1.70554063513,1.213126104 ...
2 opticsGroup3 0.100000 2.700000 200.000000 1.240000 196 2 0 0.003119 0.358234 [0.344534641293,0.21171227 ...
3 opticsGroup2 0.100000 2.700000 200.000000 1.250000 196 2 0 -0.24449 0.600143 [0.804124927631,-0.3248104 ...
4 opticsGroup1 0.100000 2.600000 200.000000 1.250000 196 2 0 0.295988 0.614989 [0.875185976399,0.64565461 ...
5 opticsGroup4 0.100000 2.700000 200.000000 1.240000 196 2 0 -0.05841 -0.14064 [-0.122672847719,-0.004938 ...
6 opticsGroup6 0.100000 2.700000 200.000000 1.250000 196 2 0 -1.28823 2.446621 [2.87241841102,-1.78863340 ...

Thus when we load in this data using Starfile we can get the particle-wise spherical aberrations, which are 2.7 except for the seven particles in opticsGroup1:

import numpy as np
from cryodrgn.starfile import Starfile

starfile = Starfile("tests/data/relion31.6opticsgroups.star")
np.unique(starfile.get_optics_values("_rlnOpticsGroup", dtype=int), return_counts=True)
    > (array([1, 2, 3, 4, 5, 6]), array([48, 29,  5,  7, 26, 24]))
np.unique(starfile.get_optics_values("_rlnSphericalAberration", dtype=float), return_counts=True)
    > (array([2.6, 2.7]), array([  7, 132]))

When we use parse_ctf_star on this file without specifying --cs, we likewise get a CTF matrix with this set of aberration values:

cryodrgn parse_ctf_star relion31.6opticsgroups.star -o new_ctf.pkl

from cryodrgn.utils import load_pkl

ctf_params = load_pkl("new_ctf.pkl")
np.unique(ctf_params[:, 6], return_counts=True)
    > (array([2.6, 2.7], dtype=float32), array([  7, 132]))

But when we do specify --cs, all values get changed to the given value:

cryodrgn parse_ctf_star tests/data/relion31.6opticsgroups.star -o new_ctf2.pkl --cs 2.8

ctf_params = load_pkl("new_ctf2.pkl")
np.unique(ctf_params[:, 6], return_counts=True)
    > (array([2.8], dtype=float32), array([139]))

See TestParseCTFStar in tests/test_relion31.py for comprehensive unit testing of this new functionality!