Energies Lower than FCI When Adding Determinants in Iterative Davidson Diagonalization

[JorgeIvanHdz]

Hello,

I am relatively new to using QuantumPackage, and I am encountering an issue during iterative calculations aimed at approximating the Full Configuration Interaction (FCI) energy. Below, I describe my process and the problem I observe:

1. I first perform a self-consistent field calculation using the command:
   qp run scf

2. Then, I perform a CISD calculation with:
   qp run cisd
   This provides me with the ground-state energy and the files psi_det.gz and psi_coef.gz, which contain the determinants and their respective coefficients after diagonalization.

3. I modify the psi_det.gz file by adding new determinants, update the psi_coef.gz file by initializing the coefficients of the new determinants to zero, and adjust the n_det file to reflect the new number of determinants.

4. I run the command:
   qp run diagonalize_h
   to perform a new diagonalization with the updated set of determinants, aiming to obtain the new ground-state energy.

5. After diagonalization, I remove determinants whose coefficients are below a small threshold (e.g., 1e-12 or 1e-10), add new determinants, and iterate the process.

Observed Problem

Initially, as I increase the number of determinants, the ground-state energy progressively approaches the FCI value, which is expected. However, after several iterations, I notice that the energy obtained during diagonalization becomes lower than the known FCI value, sometimes significantly so.

This seems contradictory since, according to the variational principle, the obtained energies should always be greater than or equal to the FCI value.

Additional Details

• I am working with simple molecules, such as H2O in the 6-31G basis, where the exact FCI value is known.
• I use Davidson diagonalization with different threshold values in the range of 1e-8 to 1e-12, but the issue persists.
• I understand that Davidson is an approximate diagonalization method, but the energy seems to decrease continuously as I add more determinants, and it does not converge as expected.

My Question

Am I missing an important step in the process? Could this behavior be due to accumulated errors in the Davidson diagonalization? Or could it be that the parameters I am using are not appropriate?

I am attaching examples of the determinant and coefficient files where I obtain energies lower than the FCI value. I would greatly appreciate any guidance or suggestions regarding this issue.

Additionally, I am providing the terminal output from a diagonalization step, where the calculated energy is significantly lower than the FCI value for an H2O molecule in the 6-31G basis. For reference, the known FCI energy for this molecule is -76.1223, but the result from this run is -76.7147.

Thank you in advance for your time and help!
ezfio_file.tar.gz

Error Details

|h2o_631g.ezfio>
(base) ivan@ivan-B460MAORUSELITE:~/Descargas/QP_examples/h2o$ qp run diagonalize_h 
Date: 28/11/2024 22:15:13
===============
Quantum Package
===============

Git Commit: Dev stable
Git Date  : Wed Nov 8 17:28:59 2023 +0100
Git SHA1  : 0f320db735bfdbdf9861c9cad9f3f64175cc8c3c
EZFIO Dir : h2o_631g.ezfio


Task server running : tcp://127.0.1.1:45029
 * mo_label          Canonical
* Number of determinants                                     74478
* Dimension of the psi arrays                                74478
 Read psi_coef       74478           1
* mo_num                                                        13
* N_int                                                          1
 Read  mo_coef
 Read psi_det
 MO map initialized:                  4186
.. >>>>> [ IO READ: mo_class ] <<<<< ..
 AO map initialized :                  4186
 Providing the AO integrals

.. >>>>> [ RES  MEM :          0.017 GB ] [ VIRT MEM :          1.694 GB ] <<<<< ..
.. >>>>> [ WALL TIME:           0.23  s ] [ CPU  TIME:           2.24  s ] <<<<< ..


Nuclear Coordinates (Angstroms)
===============================

================ ============ ============ ============ ============
      Atom          Charge          X            Y            Z 
================ ============ ============ ============ ============
O                    8.000000     0.000000     0.000000     0.117790
H                    1.000000     0.000000     0.755450    -0.471160
H                    1.000000     0.000000    -0.755450    -0.471160
================ ============ ============ ============ ============

Minimal interatomic distance found:       0.9579 Angstrom
 Sorting the map                                                           
 AO integrals provided:
  Size of AO map :            2.2930145263671875E-002 MB
  Number of AO integrals :                 2260
  cpu  time :  0.15765399999999996      s
  wall time :   2.1363938999911625E-002 s  ( x    7.3794443993053962       )
 
 AO -> MO integrals transformation
 ---------------------------------
 
* Total available memory (GB)                                   31
* Target maximum memory (GB)                                    31
 Getting AOs
 1st transformation
 2nd transformation
 3rd transformation
 4th transformation
 Molecular integrals provided:
  Size of MO map              3.4364700317382812E-002 MB
  Number of MO integrals:                  1411
  cpu  time :   7.5336000000000070E-002 s
  wall time :   7.3358860004191229E-003 s  ( x    10.269516183279823      )
 Providing the one-electron integrals
* Number of unique alpha determinants                         1078
* Number of unique beta determinants                          1197

.. >>>>> [ RES  MEM :          0.027 GB ] [ VIRT MEM :          3.591 GB ] <<<<< ..
.. >>>>> [ WALL TIME:           0.27  s ] [ CPU  TIME:           2.71  s ] <<<<< ..

* Nuclear repulsion energy                           9.189222121051648    

.. >>>>> [ RES  MEM :          0.030 GB ] [ VIRT MEM :          3.593 GB ] <<<<< ..
.. >>>>> [ WALL TIME:           0.30  s ] [ CPU  TIME:           2.89  s ] <<<<< ..


Davidson Diagonalization
------------------------

* Number of states                                               1
* Number of states in diagonalization                            4
* Number of determinants                                     74478
* Number of threads for diagonalization                         12
* Memory(Gb)                                        0.1049467846751213    

===== ================ =========== =========== 
Iter       Energy          S^2       Residual  
===== ================ =========== =========== 
   0    -76.7147286413    0.072741   4.866E-11
   1    -76.7147286415    0.072742   1.756E-11
====== ================ =========== ===========


.. >>>>> [ RES  MEM :          0.087 GB ] [ VIRT MEM :          3.693 GB ] <<<<< ..
.. >>>>> [ WALL TIME:           1.35  s ] [ CPU  TIME:          13.94  s ] <<<<< ..


.. >>>>> [ RES  MEM :          0.042 GB ] [ VIRT MEM :          3.604 GB ] <<<<< ..
.. >>>>> [ WALL TIME:           1.35  s ] [ CPU  TIME:          13.98  s ] <<<<< ..

* Energy of state    1                              -76.71472864148107    
* S^2 of state    1                                 0.7274212734529782E-01
 N_det =        74478
* Saved determinants                                         74478
Wall time: 0:00:04

[scemama]

Hello,
we assume that the determinant basis is orthonormal. What your see is typical from a situation where this condition is violated. Are you sure that you don't have repetitions in your determinant set?