Espresso Bands job and flow

[Nekkrad]

Summary of Changes

• Added job and flow to do a band calculation using quantum espresso
• Updated the documentation

Checklist

• I have read the "Guidelines" section of the contributing guide. Don't lie! 😉
• My PR is on a custom branch and is not named main.
• I have used black, isort, and ruff as described in the style guide.
• I have added relevant, comprehensive unit tests.

Summary by CodeRabbit

• New Features
  • Introduced new Espresso bands-related jobs and a Fermi surface job to enhance material calculations.
  • Updated Quantum ESPRESSO functionality for better handling of unsupported binaries.
  • Expanded the quacc library with recipes for conducting bands and Fermi surface calculations.
• Documentation
  • Updated the list of available quacc recipes to include new Espresso bands-related and Fermi surface jobs.
• Refactor
  • Modified Espresso-related modules for improved clarity and efficiency in calculations.
• Tests
  • Added test cases for validating the new bands and Fermi surface calculations.

[buildbot-princeton]

Can one of the admins verify this patch?

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 99.41%. Comparing base (842e942) to head (7ab47d7).

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #1701   +/-   ##
=======================================
  Coverage   99.41%   99.41%           
=======================================
  Files          79       80    +1     
  Lines        3055     3103   +48     
=======================================
+ Hits         3037     3085   +48     
  Misses         18       18           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[Nekkrad]

Do not merge this yet. I have to tweak few things

[Nekkrad]

@Andrew-S-Rosen This is ready to be merged from my side, but I think we should discuss if there is an easy way to generate a good k-path from the system space group. This PR so far does not specify any path so the results are crap unless the user specify a proper k-path

[Andrew-S-Rosen]

@Andrew-S-Rosen This is ready to be merged from my side, but I think we should discuss if there is an easy way to generate a good k-path from the system space group. This PR so far does not specify any path so the results are crap unless the user specify a proper k-path

@Nekkrad: Great! I will review this shortly. I'll be traveling today and tomorrow so will get to it by the 15th.

In the meantime, your question about k-path is a good one. There is a general-purpose quacc.utils.kpts module containing various utilities for k-point/k-path handling. It builds off of Pymatgen's utilities for this. Refer to https://pymatgen.org/pymatgen.symmetry.html#module-pymatgen.symmetry.bandstructure, specifically the HighSymmKpath function. I would recommend using that function with the latminer_munro method described in this article. You can just call the pymatgen function directly or, if you need to do more stuff with it, you can put something together in kpts.py that you call.

[Andrew-S-Rosen]

Thank you very much, @Nekkrad!!! Overall, this is in good shape. I just have a few logistical questions for you that might help with cleaning things up. My major comment is that I'm not sure we really need a separate bands_pp_job --- it seems like we can pack that step at the end of bands_job so users can just always have post-processed data, especially because the post-processing step is trivial in terms of compute cost.

I also think it would be good to provide a default k-path from the aforementioned Pymatgen module so this is usable out of the box.

[Nekkrad]

@Andrew-S-Rosen Thanks!

Regarding your questions:

1. The only reason behind having a separate post-processing job is because I was thinking of implementing a fermi surface job (fs.x in QE) which requires a bands pw.x calculation, but not the post-processing. For all practical applications that I can come up with, one could merge bands_pp and the future fermi_job inside the bands_job. These are both very computationally cheap post-processing jobs. Another option is to merge only bands_pp and have a different fermi_surface job

2. If we merge bands_pp job inside bands_job then there is no real need for the flow. You can directly to a static/relaxation --> bands_job.

I am willing to merge the bands_pp_job into bands_job, add k-path defaults with pymatgen. I can also remove the flow.

[Andrew-S-Rosen]

The only reason behind having a separate post-processing job is because I was thinking of implementing a fermi surface job (fs.x in QE) which requires a bands pw.x calculation, but not the post-processing. For all practical applications that I can come up with, one could merge bands_pp and the future fermi_job inside the bands_job. These are both very computationally cheap post-processing jobs. Another option is to merge only bands_pp and have a different fermi_surface job

@Nekkrad: Thanks for the info! Very helpful. If the fermi surface task is very cheap and always after a bandstructure calculation (which I assume it must be), then I would lean towards also having it be within the bands_job, perhaps triggered via an optional keyword argument like make_fermi_surface: bool = False.

If we merge bands_pp job inside bands_job then there is no real need for the flow. You can directly to a static/relaxation --> bands_job.

Agreed.

I am willing to merge the bands_pp_job into bands_job, add k-path defaults with pymatgen. I can also remove the flow.

Thanks! I think this would make the most sense. Of course, you are the most immediate end-user, so don't hesitate to let me know if the suggestion hinders your productivity in any way.

Thank you for the contribution and discussion!

[tomdemeyere]

@Andrew-S-Rosen

QE has a lot of binaries... and I think you are right here to say that they probably do not all deserve a job. Initially I wanted each binary to have its own job because I believe it is very nice in a user experience point of view. But, considering other addons such as https://anharmonic.github.io/thermal2/ that contains more than 10 binaries alone... It might be well advised to do a "all_kind" job which would be able to run any kind of software using Espresso style input.

Few constraints that might be annoying:

1. This "all_kind" job will have to take a "binary" argument that would give the full path to the binary since we cannot list them in Quacc.
2. Similarly, it should have a "read_output_function" parameter that takes a function. The function will be written by the user to read the output(s)

Also it might be fair enough to say "let the user create their own job to run these binaries".

[Andrew-S-Rosen]

Thanks for the input! I'm not opposed to having a bunch of individual jobs if you feel like people will use them frequently enough. In this case, my comment was mostly about the post-processing step, which I felt like could safely be done as part of the parent DOS job.

Re: the "all" style of job, it's an interesting idea. Certainly, I can see value for it. One challenge that comes to mind that wasn't mentioned is that different binaries might have different sensible defaults as parameters, in which case one might as well separate them into individual jobs. Also, requiring users to provide a parsing function is quite messy. Indeed, under those scenarios, we might as well just assume that quacc is providing the infrastructure for people to do things on their own if they so choose.

@Nekkrad: This is all separate from the current PR, which I think looks great and just requires some minor adjustments so that users aren't running a dedicated post-processing step unnecessarily (e.g. potentially as a separate Slurm job) and a sensible default for the k-path. Let me know if you need any help on either. 👍

[Nekkrad]

@Andrew-S-Rosen Sorry been pretty busy.

Everything is good but I have a couple of questions

1. The k-path with pymatgen is done on the primitive cell, while this is generally a good choice some users may want to have a different k-path for whatever reason (e.g. conventional cell calculation). If this is the case then we may want to specify this in the documentation/function. Or we can come up with a smart way to address this, e.g. matrix algebra to map primitive path into the conventional path (this raises some issues tho, and it is not very extendable to other possible situations)

2. Looking at the slab flow you pointed me to, it still requires to define the jobs somewhere which I thought it is something we want to avoid. Another solution is to define a job, call multiple base_jobs : one bands, one pp and one optional fermi_surface and return a list of RunSchemas. Have you got in mind any other cleaner solution?

[Andrew-S-Rosen]

The k-path with pymatgen is done on the primitive cell, while this is generally a good choice some users may want to have a different k-path for whatever reason (e.g. conventional cell calculation). If this is the case then we may want to specify this in the documentation/function. Or we can come up with a smart way to address this, e.g. matrix algebra to map primitive path into the conventional path (this raises some issues tho, and it is not very extendable to other possible situations)

@Nekkrad: This is a fair point. Perhaps it would make sense to have the default route be to: 1) make a primitive cell; 2) generate the pymatgen k-path. We could mention in the docs that this is what happens. If the user does wants to explicitly provide a k-path, then no transformation would be done on the structure. I'd avoid trying to map the primitive path to the conventional path --- seems messy.

Looking at the slab flow you pointed me to, it still requires to define the jobs somewhere which I thought it is something we want to avoid. Another solution is to define a job, call multiple base_jobs : one bands, one pp and one optional fermi_surface and return a list of RunSchemas. Have you got in mind any other cleaner solution?

Apologies --- I'm trying to find my comment about the slab flow you mention but am having trouble seeing it. In any case, there are basically two options:

1. Define three separate jobs, one for bands, one for pp, one for fermi surface. In the bands job, you could have keyword options to call the pp and/or fermi surface, removing the @job decorator from the fast post-processing jobs via the strip_decorator function. See a similar example here.

2. Simply define one job for bands that does as you suggested --- runs all three steps (optionally) and returns a dictionary of schemas. You would return {"bands": {blah}, "pp": {blah}, "fermi_surface": {blah}} that way if the user does not want (say) the fermi surface, that key is just not present. Lists are a bit harder to deal with because you have to remember the specific order from the source code. A similar example is here.

The first option only makes sense if there are scenarios where people would want to run a standalone fermi surface and/or post-processing job outside of the DOS calculation. If that's not the case, then option 2 makes more sense. But both are doing the same thing really.

[Nekkrad]

@Andrew-S-Rosen
This is a first commit for the new job, there are still some issues to address but let me know if it looks good.

The issues are

• There is no "fs" in the espresso_key.py in ASE. I will open a PR later on today. This prevents us from running any fermi_surface calculations
• The function "_cleanup_params" here check whether we are using both kspacing and kpts. The problem occurs if .get("kpts") is a list cause it will raise Value error.

[tomdemeyere]

@Andrew-S-Rosen

https://gitlab.com/ase/ase/-/merge_requests/3263 will address the first point mentioned by @Nekkrad without the need of adding any keys (that Ask does not really want)

[Andrew-S-Rosen]

Thanks, @Nekkrad! This looks very nice. I have left some comments.

There is no "fs" in the espresso_key.py in ASE. I will open a PR later on today. This prevents us from running any fermi_surface calculations

I merged the MR in ASE mentioned by @tomdemeyere. Next time you make a commit to this PR, the updated ASE will be installed.

The function "_cleanup_params" here check whether we are using both kspacing and kpts. The problem occurs if .get("kpts") is a list cause it will raise Value error.

Apologies. The issue here isn't immediately clear to me yet. Feel free to make an edit or perhaps if you wouldn't mind explaining a bit more, I can help.

[Andrew-S-Rosen]

Seems like this would be more accurate to say make_bandpath: bool = True and then mention it will convert to primitive in the docstring.

[Andrew-S-Rosen]

"generate" --> "generates"

[Andrew-S-Rosen]

"using Latminer-Munro" --> "using the Latmer-Munro"

[Andrew-S-Rosen]

Use [pymatgen.symmetry.bandstructure.HighSymmKpath][] since it will auto hyperlink in the docs.

[Andrew-S-Rosen]

You can remove this and simply leave calc_defaults as the default in base_fn. Or just put {} in the base_fn call directly.

[Andrew-S-Rosen]

Move this into the corresponding if statement that way we don't have the entry if the job isn't run.

[Andrew-S-Rosen]

Do BandsSchema(TypedDict, total=False) to indicate that not all three entries may necessarily be present.

[Andrew-S-Rosen]

I think this can be removed now.

[Andrew-S-Rosen]

For this and the other tasks, perhaps what would be best would be to make them all @jobs (and remove the _ in front). Then, when they are called within bands_job, you can do

from quacc.wflow_tools.customizers import strip_decorator

strip_decorator(fermi_surface)

Then one could run the task as a standalone job if they wish. If you do this, feel free to add the jobs to the recipe list too.

[Andrew-S-Rosen]

The Fermi surface task isn't covered, so we'll want to address this.

[Nekkrad]

@Andrew-S-Rosen

Name changed, tell me if you like them. Not very inspired but probably ok ahah

[coderabbitai]

Review Status

Actionable comments generated: 0

Configuration used: CodeRabbit UI

Commits

Files that changed from the base of the PR and between 69a0b82 and 94963d4.

Files selected for processing (8)

• docs/user/recipes/recipes_list.md (1 hunks)
• src/quacc/calculators/espresso/espresso.py (4 hunks)
• src/quacc/recipes/espresso/_base.py (2 hunks)
• src/quacc/recipes/espresso/bands.py (1 hunks)
• src/quacc/recipes/espresso/dos.py (2 hunks)
• src/quacc/settings.py (1 hunks)
• tests/core/recipes/espresso_recipes/test_bands.py (1 hunks)
• tests/core/recipes/espresso_recipes/test_dos.py (1 hunks)

Files skipped from review due to trivial changes (1)

• tests/core/recipes/espresso_recipes/test_dos.py

Additional comments: 14

tests/core/recipes/espresso_recipes/test_bands.py (2)

• 18-43: The test test_bands_job correctly sets up and executes a bands job, asserting the expected outcomes. However, it's important to ensure that the test covers all relevant aspects of the bands job functionality, including error handling and edge cases.
• 52-86: The test test_bands_job_with_fermi extends the testing to include Fermi surface calculations. It's well-structured and asserts the key outcomes. To further improve test coverage, consider adding tests for scenarios where the Fermi surface calculation might fail or produce unexpected results.

src/quacc/recipes/espresso/_base.py (1)

• 202-204: The conditional check if binary in ALL_KEYS: before converting input_data to nested dictionaries is a good practice. It ensures that the conversion only happens for supported binaries, which can prevent potential errors with unsupported binaries. This change enhances the robustness of the code.

src/quacc/recipes/espresso/dos.py (4)

• 2-2: The module documentation clearly states the purpose of the dos.py file, focusing on performing DOS calculations using Quantum ESPRESSO. This clarity helps users understand the scope and functionality of the module.
• 1-10: > 📝 NOTE

This review was outside the diff hunks, and no overlapping diff hunk was found. Original lines [52-86]

The test_bands_job_with_fermi function is well-implemented, covering the setup and execution of a bands job with Fermi surface calculations. It's important to ensure comprehensive test coverage, including scenarios where the Fermi surface calculation might fail or produce unexpected results.

• 1-10: > 📝 NOTE

This review was outside the diff hunks, and no overlapping diff hunk was found. Original lines [52-86]

The test_bands_job_with_fermi function is well-implemented, covering the setup and execution of a bands job with Fermi surface calculations. It's important to ensure comprehensive test coverage, including scenarios where the Fermi surface calculation might fail or produce unexpected results.

• 1-10: > 📝 NOTE

This review was outside the diff hunks, and no overlapping diff hunk was found. Original lines [52-86]

The test_bands_job_with_fermi function is well-implemented, covering the setup and execution of a bands job with Fermi surface calculations. It's important to ensure comprehensive test coverage, including scenarios where the Fermi surface calculation might fail or produce unexpected results.

src/quacc/recipes/espresso/bands.py (4)

• 35-143: The bands_job function provides a comprehensive approach to performing bands and Fermi surface calculations, with flexibility to include post-processing and Fermi surface calculations based on user input. It's important to ensure that the function handles all possible error scenarios gracefully, especially when dealing with external binaries like pw.x, bands.x, and fs.x.
• 146-209: The bands_pw_job function is well-implemented, providing a mechanism to perform a basic bands calculation using pw.x. The optional generation of a high symmetry k-path using the Latmer-Munro approach is a valuable feature. Ensure that the function properly handles cases where the generation of the k-path might fail or produce unexpected results.
• 212-258: The bands_pp_job function correctly sets up and executes a post-processing calculation using bands.x. This function is crucial for re-ordering bands and computing band-related properties. It's important to validate that the function correctly handles the input and output directories, especially in cases where the previous calculation's output might not be as expected.
• 261-308: The fermi_surface_job function is designed to retrieve the Fermi surface using fs.x, requiring a uniform unshifted k-point grid from a previous bands calculation. This function adds significant value to the module by enabling Fermi surface analysis. Ensure that the function includes checks for the prerequisites of the Fermi surface calculation to prevent user errors.

docs/user/recipes/recipes_list.md (1)

• 204-208: The additions of Espresso-related jobs and a Fermi surface job to the recipes list are well-structured and align with the PR's objectives to expand computational capabilities. Ensure that the placeholders for documentation links [quacc.recipes.espresso.bands.bands_job][] are intended to be filled in later or are in the process of being completed to maintain the usefulness and navigability of the documentation.

src/quacc/calculators/espresso/espresso.py (1)

• 356-368: The handling of unsupported binaries with a warning and adjustments to input_data is a thoughtful approach to maintain flexibility while informing the user about potential limitations. Ensure that this behavior is thoroughly tested with various unsupported binaries to confirm that the adjustments and warning mechanism work as intended without introducing unexpected behavior.

src/quacc/settings.py (1)

• 179-179: The addition of "fs": "fs.x" to the ESPRESSO_BINARIES dictionary is a straightforward and necessary update to support Fermi surface calculations as mentioned in the PR objectives. This change is consistent with the naming conventions and structure of the existing entries in the dictionary. However, it's important to ensure that any related documentation, especially user guides or API documentation, is updated to include this new binary and its intended use case. This will help users understand how to leverage the new Fermi surface calculation capabilities introduced in this PR.

[Andrew-S-Rosen]

@Nekkrad Thanks for your edits! The names and such look good to me.

I think we can consider this one ready. Thank you for your modifications and input. I will proceed with the merge after I merge in #1759 shortly.

Also, don't mind me as I play around with this coderabbitai 😅

[Andrew-S-Rosen]

@Nekkrad --- Thank you again for your contribution, and apologies for how long this has taken to get merged! I refactored this a bit to reduce some unnecessary code. I also changed the bands_job to a bands_flow. I know I was the one saying we should pack it all in a @job, but I thought about it some more and perhaps it is more intuitive to have it as a @flow. In any case, you can certainly feel free to open a follow-up PR and change it however you like (if you desire). I think you'll find the current code to be nice and clean :)

I am done with the cleanup here. The only challenge is that the tests run for a very long time now. Is there anything you can do in the parameters to make things less expensive?

[Nekkrad]

@Andrew-S-Rosen No worries! I will take a look at it tomorrow. I can probably make them run very quickly, will let you know once it's done

[Andrew-S-Rosen]

@Andrew-S-Rosen No worries! I will take a look at it tomorrow. I can probably make them run very quickly, will let you know once it's done

Thanks!! That would be a huge help :) Will merge ASAP afterwards!

[Nekkrad]

@Andrew-S-Rosen I added few keywords to the bands_pw_job and flow. These allow the users to have a little more flexibility if they decide to use make_bandpath.

I also changed the parameters for the tests, hopefully they should run quicker.

Let me know if they are still very slow and/or if there are other ONETEP or QE recipes that run too slowly

[Andrew-S-Rosen]

@Nekkrad: Thanks a ton! The time was dropped for the most expensive run by nearly 2x, which was a big help.

Here is the current timing:

============================= slowest 10 durations =============================
185.70s call     tests/core/recipes/espresso_recipes/test_bands.py::test_bands_flow
72.76s call     tests/core/recipes/espresso_recipes/test_dos.py::test_dos_flow
68.89s call     tests/core/recipes/espresso_recipes/test_dos.py::test_projwfc_flow
31.81s call     tests/core/recipes/espresso_recipes/test_phonons.py::test_phonon_job_list_to_do
14.21s call     tests/core/recipes/espresso_recipes/test_core.py::test_ase_relax_cell_job
11.72s call     tests/core/recipes/espresso_recipes/test_phonons.py::test_phonon_job
11.52s call     tests/core/recipes/espresso_recipes/test_core.py::test_relax_job_cell
6.47s call     tests/core/recipes/espresso_recipes/test_core.py::test_ase_relax_job
6.11s call     tests/core/recipes/espresso_recipes/test_core.py::test_relax_job
5.02s call     tests/core/recipes/espresso_recipes/test_core.py::test_static_job_v2
========== 49 passed, 8 skipped, 215 deselected in 457.10s (0:07:[37](https://github.com/Quantum-Accelerators/quacc/actions/runs/8110364645/job/22167485028#step:7:38)) ===========

I'm going to go ahead and merge this, but certainly if you can think of other words to reduce the timing, I would be greatly appreciative. At some point though, I will probably need to just split the Espresso tests into two separate parts either way. As for ONETEP, right now the test suite is only running the mocked tests because the "real" ONETEP execution would happen on the Princeton HPC machines, and I haven't yet successfully compiled ONETEP. Eventually...

Setting this to auto-merge!