Density Screening Refactor Part 1: test_erisieve.py Rework #2547

davpoolechem · 2022-04-14T19:06:17Z

Description

This PR is the first in a series of planned PRs designed to remove density screening from the TwoBodyAOInt object and into the JK object. Having density screening available in TwoBodyAOInt runs the risk of applying density screening to algorithms where density screening doesn't make sense. Thus, it would be a good idea to move the logic of density screening to where it is more correctly applied, i.e., the JK object.

This PR solves two issues simultaneously:

The primary purpose of this PR is to change the test_erisieve.py tests to work with the planned future density screening refactor. One issue that moving density screening from TwoBodyAOInt to JK currently brings up, is that it causes the tests on density screening within the pytest test_erisieve.py to fail. These failures occur because test_erisieve.py performs its screening tests directly using an ERI object generated by IntegralFactory. With density screening being removed from the TwoBodyAOInt object, this method of density screening testing can no longer be done. The current PR is designed to address this issue for when the density screening refactor happens. The aforementioned issue is addressed by implementing a new variable to the HF wavefunction, computed_shells_per_iter_, which keeps track of the number of shell quartets computed per SCF iteration. The computed_shells_per_iter_ variable is accessible to the user via Python, and thus can be used to conduct screening tests. In this way, density screening tests can be performed without the need for an ERI object.
As a bonus from the changes introduced by this PR, the DirectJK algorithm no longer has a need to print computed shell quartet counts to bench.dat. Bench.dat is used exclusively by the DirectJK object to dump the number of shell quartets computed per SCF iteration somewhere. That data is now accessible to the user in a cleaner fashion - it can be accessed through Python, in a manipulatable format.

Notes

Note that the changes in this PR have not been applied to the LinK portion of the DirectJK code. This is intentional, as Andy is planning on moving LinK out of DirectJK entirely, and editing the LinK code within DirectJK would interfere with that. Thus, the changes in this branch will be applied to LinK in a later update.

Todos

[ X ] Addition of computed_shells_ member to JK object, which keeps track of number of shells computed during the JK build process.
[ X ] Addition of computed_shells_per_iter_ member to HF wavefunction objects, which keep track of number of shells computed during each SCF iteration. This information can be accessed by the user via Python.
[ X ] Modification of density screening tests in test_erisieve.py using the above class changes to allow the tests to run without construction of an ERI object.

Questions

Currently, only the density screening tests in test_erisieve.py use the new computed_shells_per_iter_ framework to test screening. Other tests in test_erisieve.py perform their tests using a generated ERI object. Should use of computed_shells_per_iter_ comparisons be applied to other tests in test_erisieve.py, as well?

Checklist

[ X ] Tests added for any new features
[ X ] Docs added for any new features
[ X ] All or relevant fraction of full tests run

Status

[ X ] Ready for review
Ready for merge

JonathonMisiewicz · 2022-04-28T12:29:04Z

Details about how the integrals were computed should be the province of the JK object, not the HF wavefunction, so I disagree with creating this new variable as described.

Can we instead have computed_shells_per_iter_ on the JK object and query the JK object, after the HF, for test purposes?

davpoolechem · 2022-04-28T13:34:49Z

Details about how the integrals were computed should be the province of the JK object, not the HF wavefunction, so I disagree with creating this new variable as described.

Can we instead have computed_shells_per_iter_ on the JK object and query the JK object, after the HF, for test purposes?

That should definitely be doable! Give me a bit, and that change can be made.

davpoolechem · 2022-04-28T17:39:13Z

Done and done! computed_shells_per_iter_ is now in the JK object.

JonathonMisiewicz

I have two minor code cleanup requests, but LGTM otherwise.

Short PRs are appreciated!

psi4/src/export_fock.cc

psi4/src/psi4/libfock/jk.h

jeffschriber

LGTM, no comments from me

zachglick

Looks good

zachglick · 2022-04-15T11:46:06Z

psi4/src/export_wavefunction.cc

@@ -394,7 +394,9 @@ void export_wavefunction(py::module& m) {
                      "Are we to do excited-state MOM?")
        .def_property("MOM_performed_", &scf::HF::MOM_performed, &scf::HF::set_MOM_performed,
                      "MOM performed current iteration?")
-        .def_property("attempt_number_", &scf::HF::attempt_number, &scf::HF::set_attempt_number,
+        .def_property("computed_shells_per_iter_", &scf::HF::computed_shells_per_iter, &scf::HF::set_computed_shells_per_iter,
+	              "Array containing the number of shells computed (not screrened out) during each SCF iteration.")


Suggested change

"Array containing the number of shells computed (not screrened out) during each SCF iteration.")

"Array containing the number of shell quartets computed (not screened out) during each SCF iteration.")

zachglick · 2022-06-10T16:34:19Z

tests/pytests/test_erisieve.py

+    schwarz_computed_shells_expected = [20290, 20290, 20290, 20290, 20290, 20290, 20290, 20290, 20290]
+    density_computed_shells_expected = [13171, 19618, 19665, 19657, 19661, 19661, 19663, 19663, 19663]
+
+    for iter_ in range(0,9):


It would be good to explicitly check that the lists of expected and computed shell quartets are the same length (the number of SCF iterations, which has been hardcoded as 9 here).

Better yet, I think there's a compare_arrays function that you could call instead of repeated calls to compare_integers.

good point. compare_arrays is legacy name -- you can use compare_values directly. It handles float-like, while compare handles int-like.

This is indeed a very good point. I will make these changes!

All right, this change should be made now!

zachglick · 2022-06-10T16:34:46Z

tests/pytests/test_erisieve.py

-    assert compare_integers(screen_count_uhf, 19440, 'UHF Density Screened Ints Count, Cutoff 1.0e-12')
+    computed_shells_expected = [13171, 19618, 19665, 19657, 19661, 19661, 19663, 19663, 19663]
+
+    for iter_ in range(0,9):


Same as above comment

psi4/src/export_fock.cc

psi4/src/psi4/libfock/jk.h

…F wave functions

…in the future

psi4/src/psi4/libfock/DirectJK.cc

tests/pytests/test_erisieve.py

Co-authored-by: Zach Glick <glickzachary@gmail.com>

davpoolechem · 2022-06-10T20:07:44Z

So I now realize something - we may want to apply some of the benchmarking changes made in this PR to DFJCOSK, as well. It will increase the size of the PR, but the benchmarking changes in this PR currently only extend to DirectJK at the moment. Since DFJCOSK has two methods that it separately benchmarks, it will require a bit of retooling regarding some of the internals of the benchmarking framework. It should not have a significant impact on test_erisieve, however.

Thoughts?

loriab · 2022-06-10T20:11:41Z

So I now realize something - we may want to apply some of the benchmarking changes made in this PR to DFJCOSK, as well. It will increase the size of the PR, but the benchmarking changes in this PR currently only extend to DirectJK at the moment. Since DFJCOSK has two methods that it separately benchmarks, it will require a bit of retooling regarding some of the internals of the benchmarking framework. It should not have a significant impact on test_erisieve, however.

Thoughts?

Unless the DFJCOSK changes would undo much of this PR, I think a follow-up PR would be best.

davpoolechem · 2022-06-10T20:16:36Z

So I now realize something - we may want to apply some of the benchmarking changes made in this PR to DFJCOSK, as well. It will increase the size of the PR, but the benchmarking changes in this PR currently only extend to DirectJK at the moment. Since DFJCOSK has two methods that it separately benchmarks, it will require a bit of retooling regarding some of the internals of the benchmarking framework. It should not have a significant impact on test_erisieve, however.

Thoughts?

Unless the DFJCOSK changes would undo much of this PR, I think a follow-up PR would be best.

DFJCOSK won't explicitly undo most of this PR, nicely enough, though it will require some changes to how the computed_shells member functions/variables are handled. Regardless, it won't lead to significant changes in test_erisieve, so a separate PR should work fine. And ultimately, the big point of this PR is to allow testing of density screening in test_erisieve without needing to directly construct and use separate TwoBodyAOInt objects, since the plan is to remove density screening from TwoBodyAOInt entirely.

Thank you for your feedback!

davpoolechem force-pushed the dpoole34/jk-bench-rework branch from 3a87bee to 0058940 Compare April 28, 2022 13:43

JonathonMisiewicz reviewed May 4, 2022

View reviewed changes

psi4/src/export_fock.cc Outdated Show resolved Hide resolved

psi4/src/psi4/libfock/jk.h Outdated Show resolved Hide resolved

davpoolechem force-pushed the dpoole34/jk-bench-rework branch from b0f0263 to be00cd0 Compare May 5, 2022 12:09

JonathonMisiewicz approved these changes May 5, 2022

View reviewed changes

JonathonMisiewicz added this to the Psi4 1.7 milestone May 21, 2022

jeffschriber approved these changes Jun 9, 2022

View reviewed changes

zachglick reviewed Jun 10, 2022

View reviewed changes

David Poole added 16 commits June 10, 2022 13:15

Add new computed_shells_per_iter_ variable, accesible in Python, to H…

aa4cda8

…F wave functions

Rework density screening tests in test_erisieve.py

40a5109

Remove writing to bench.dat from DirectJK; will be removed from LinK …

c60062d

…in the future

Call JK::compute_shells() when LinK is used in DirectJK

b13de9b

Clean up JK::computed_shells() warning output

b50e709

Clean up documentation some

7519ab2

Clean up docs some more

381193c

Move computed_shells_per_iter_ from HF to JK

df959ff

Remove computed_shells_per_iter_ from HF

48cda5c

Update test_erisieve to work with JK.computed_shells_per_iter

580734d

Slight cleanup of docs

7bb1d31

Fix typo in export_fock.cc

fcb3868

Make JK::computed_shells() protected member function

8ba4417

Get rid of extraneous spaces

366dd5a

Fix a couple of whitespace issues

495af44

Rename computed_shells_ to num_computed_shells_

d7bfd8c

loriab approved these changes Jun 10, 2022

View reviewed changes

psi4/src/psi4/libfock/DirectJK.cc Show resolved Hide resolved

tests/pytests/test_erisieve.py Show resolved Hide resolved

davpoolechem force-pushed the dpoole34/jk-bench-rework branch from 0b27386 to d7bfd8c Compare June 10, 2022 17:23

David Poole and others added 2 commits June 10, 2022 12:26

Apply Zach's documentation suggestions from code review

11b371c

Co-authored-by: Zach Glick <glickzachary@gmail.com>

Update test_erisieve with code review suggestions

1edf615

Replace compare_values with compare in density erisieve tests

59d59f6

loriab merged commit f7f9352 into psi4:master Jun 10, 2022

loriab mentioned this pull request Jul 6, 2022

Intel compiler vs DirectJK #2625

Merged

3 tasks

davpoolechem mentioned this pull request Aug 19, 2022

Density Screening Refactor Part 2: Implementation of shell_significant() #2695

Closed

10 tasks

davpoolechem mentioned this pull request Dec 8, 2023

Reboot - Density Screening Refactor Part 2: Implementation of shell_significant() #3098

Open

9 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Density Screening Refactor Part 1: test_erisieve.py Rework #2547

Density Screening Refactor Part 1: test_erisieve.py Rework #2547

davpoolechem commented Apr 14, 2022 •

edited

Loading

JonathonMisiewicz commented Apr 28, 2022

davpoolechem commented Apr 28, 2022

davpoolechem commented Apr 28, 2022

JonathonMisiewicz left a comment

jeffschriber left a comment

zachglick left a comment

zachglick Apr 15, 2022

zachglick Jun 10, 2022

loriab Jun 10, 2022

davpoolechem Jun 10, 2022

davpoolechem Jun 10, 2022

zachglick Jun 10, 2022

davpoolechem commented Jun 10, 2022

loriab commented Jun 10, 2022

davpoolechem commented Jun 10, 2022 •

edited

Loading

	"Array containing the number of shells computed (not screrened out) during each SCF iteration.")
	"Array containing the number of shell quartets computed (not screened out) during each SCF iteration.")

Density Screening Refactor Part 1: test_erisieve.py Rework #2547

Density Screening Refactor Part 1: test_erisieve.py Rework #2547

Conversation

davpoolechem commented Apr 14, 2022 • edited Loading

Description

Notes

Todos

Questions

Checklist

Status

JonathonMisiewicz commented Apr 28, 2022

davpoolechem commented Apr 28, 2022

davpoolechem commented Apr 28, 2022

JonathonMisiewicz left a comment

Choose a reason for hiding this comment

jeffschriber left a comment

Choose a reason for hiding this comment

zachglick left a comment

Choose a reason for hiding this comment

zachglick Apr 15, 2022

Choose a reason for hiding this comment

zachglick Jun 10, 2022

Choose a reason for hiding this comment

loriab Jun 10, 2022

Choose a reason for hiding this comment

davpoolechem Jun 10, 2022

Choose a reason for hiding this comment

davpoolechem Jun 10, 2022

Choose a reason for hiding this comment

zachglick Jun 10, 2022

Choose a reason for hiding this comment

davpoolechem commented Jun 10, 2022

loriab commented Jun 10, 2022

davpoolechem commented Jun 10, 2022 • edited Loading

davpoolechem commented Apr 14, 2022 •

edited

Loading

davpoolechem commented Jun 10, 2022 •

edited

Loading