forked from deepmodeling/deepmd-kit
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
* refactor docs * Update type-embedding.md * fix typos * Update lammps.md * Update model-deviation.md * fix typos * fix links in readme; rewrite training * refactor model part * fix several typos; move sections * Update doc/third-party/lammps-command.md Co-authored-by: tuoping <80671886+tuoping@users.noreply.github.com> * create markdown files * revert api_cc * update developer toxtree * remove unexpected api_cc.rst Co-authored-by: Han Wang <amcadmus@gmail.com> Co-authored-by: tuoping <80671886+tuoping@users.noreply.github.com>
- Loading branch information
3 people
authored
Aug 16, 2021
1 parent
6d8c31c
commit 820b3ed
Showing
70 changed files
with
1,383 additions
and
1,242 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,52 @@ | ||
| # Data conversion | ||
|
|
||
| One needs to provide the following information to train a model: the atom type, the simulation box, the atom coordinate, the atom force, system energy and virial. A snapshot of a system that contains these information is called a **frame**. We use the following convention of units: | ||
|
|
||
|
|
||
| Property | Unit | ||
| ---|--- | ||
| Time | ps | ||
| Length | Å | ||
| Energy | eV | ||
| Force | eV/Å | ||
| Virial | eV | ||
| Pressure | Bar | ||
|
|
||
|
|
||
| The frames of the system are stored in two formats. A raw file is a plain text file with each information item written in one file and one frame written on one line. The default files that provide box, coordinate, force, energy and virial are `box.raw`, `coord.raw`, `force.raw`, `energy.raw` and `virial.raw`, respectively. *We recommend you use these file names*. Here is an example of force.raw: | ||
| ```bash | ||
| $ cat force.raw | ||
| -0.724 2.039 -0.951 0.841 -0.464 0.363 | ||
| 6.737 1.554 -5.587 -2.803 0.062 2.222 | ||
| -1.968 -0.163 1.020 -0.225 -0.789 0.343 | ||
| ``` | ||
| This `force.raw` contains 3 frames with each frame having the forces of 2 atoms, thus it has 3 lines and 6 columns. Each line provides all the 3 force components of 2 atoms in 1 frame. The first three numbers are the 3 force components of the first atom, while the second three numbers are the 3 force components of the second atom. The coordinate file `coord.raw` is organized similarly. In `box.raw`, the 9 components of the box vectors should be provided on each line. In `virial.raw`, the 9 components of the virial tensor should be provided on each line in the order `XX XY XZ YX YY YZ ZX ZY ZZ`. The number of lines of all raw files should be identical. | ||
|
|
||
| We assume that the atom types do not change in all frames. It is provided by `type.raw`, which has one line with the types of atoms written one by one. The atom types should be integers. For example the `type.raw` of a system that has 2 atoms with 0 and 1: | ||
| ```bash | ||
| $ cat type.raw | ||
| 0 1 | ||
| ``` | ||
|
|
||
| Sometimes one needs to map the integer types to atom name. The mapping can be given by the file `type_map.raw`. For example | ||
| ```bash | ||
| $ cat type_map.raw | ||
| O H | ||
| ``` | ||
| The type `0` is named by `"O"` and the type `1` is named by `"H"`. | ||
|
|
||
| The second format is the data sets of `numpy` binary data that are directly used by the training program. User can use the script `$deepmd_source_dir/data/raw/raw_to_set.sh` to convert the prepared raw files to data sets. For example, if we have a raw file that contains 6000 frames, | ||
| ```bash | ||
| $ ls | ||
| box.raw coord.raw energy.raw force.raw type.raw virial.raw | ||
| $ $deepmd_source_dir/data/raw/raw_to_set.sh 2000 | ||
| nframe is 6000 | ||
| nline per set is 2000 | ||
| will make 3 sets | ||
| making set 0 ... | ||
| making set 1 ... | ||
| making set 2 ... | ||
| $ ls | ||
| box.raw coord.raw energy.raw force.raw set.000 set.001 set.002 type.raw virial.raw | ||
| ``` | ||
| It generates three sets `set.000`, `set.001` and `set.002`, with each set contains 2000 frames. One do not need to take care of the binary data files in each of the `set.*` directories. The path containing `set.*` and `type.raw` is called a *system*. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,8 @@ | ||
| # Data | ||
|
|
||
| In this section, we will introduce how to convert the DFT labeled data into the data format used by DeePMD-kit. | ||
|
|
||
| The DeePMD-kit organize data in `systems`. Each `system` is composed by a number of `frames`. One may roughly view a `frame` as a snap short on an MD trajectory, but it does not necessary come from an MD simulation. A `frame` records the coordinates and types of atoms, cell vectors if the periodic boundary condition is assumed, energy, atomic forces and virial. It is noted that the `frames` in one `system` share the same number of atoms with the same type. | ||
|
|
||
| - [Data conversion](data-conv.md) | ||
| - [Prepare data with dpdata](dpdata.md) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,11 @@ | ||
| Data | ||
| ==== | ||
| In this section, we will introduce how to convert the DFT labeled data into the data format used by DeePMD-kit. | ||
|
|
||
| The DeePMD-kit organize data in :code:`systems`. Each :code:`system` is composed by a number of :code:`frames`. One may roughly view a :code:`frame` as a snap short on an MD trajectory, but it does not necessary come from an MD simulation. A :code:`frame` records the coordinates and types of atoms, cell vectors if the periodic boundary condition is assumed, energy, atomic forces and virial. It is noted that the :code:`frames` in one :code:`system` share the same number of atoms with the same type. | ||
|
|
||
| .. toctree:: | ||
| :maxdepth: 1 | ||
|
|
||
| data-conv | ||
| dpdata |
File renamed without changes.
This file was deleted.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,81 @@ | ||
| # Compress a model | ||
|
|
||
| Once the frozen model is obtained from deepmd-kit, we can get the neural network structure and its parameters (weights, biases, etc.) from the trained model, and compress it in the following way: | ||
| ```bash | ||
| dp compress -i graph.pb -o graph-compress.pb | ||
| ``` | ||
| where `-i` gives the original frozen model, `-o` gives the compressed model. Several other command line options can be passed to `dp compress`, which can be checked with | ||
| ```bash | ||
| $ dp compress --help | ||
| ``` | ||
| An explanation will be provided | ||
| ``` | ||
| usage: dp compress [-h] [-v {DEBUG,3,INFO,2,WARNING,1,ERROR,0}] [-l LOG_PATH] | ||
| [-m {master,collect,workers}] [-i INPUT] [-o OUTPUT] | ||
| [-s STEP] [-e EXTRAPOLATE] [-f FREQUENCY] | ||
| [-c CHECKPOINT_FOLDER] | ||
| optional arguments: | ||
| -h, --help show this help message and exit | ||
| -v {DEBUG,3,INFO,2,WARNING,1,ERROR,0}, --log-level {DEBUG,3,INFO,2,WARNING,1,ERROR,0} | ||
| set verbosity level by string or number, 0=ERROR, | ||
| 1=WARNING, 2=INFO and 3=DEBUG (default: INFO) | ||
| -l LOG_PATH, --log-path LOG_PATH | ||
| set log file to log messages to disk, if not | ||
| specified, the logs will only be output to console | ||
| (default: None) | ||
| -m {master,collect,workers}, --mpi-log {master,collect,workers} | ||
| Set the manner of logging when running with MPI. | ||
| 'master' logs only on main process, 'collect' | ||
| broadcasts logs from workers to master and 'workers' | ||
| means each process will output its own log (default: | ||
| master) | ||
| -i INPUT, --input INPUT | ||
| The original frozen model, which will be compressed by | ||
| the code (default: frozen_model.pb) | ||
| -o OUTPUT, --output OUTPUT | ||
| The compressed model (default: | ||
| frozen_model_compressed.pb) | ||
| -s STEP, --step STEP Model compression uses fifth-order polynomials to | ||
| interpolate the embedding-net. It introduces two | ||
| tables with different step size to store the | ||
| parameters of the polynomials. The first table covers | ||
| the range of the training data, while the second table | ||
| is an extrapolation of the training data. The domain | ||
| of each table is uniformly divided by a given step | ||
| size. And the step(parameter) denotes the step size of | ||
| the first table and the second table will use 10 * | ||
| step as it's step size to save the memory. Usually the | ||
| value ranges from 0.1 to 0.001. Smaller step means | ||
| higher accuracy and bigger model size (default: 0.01) | ||
| -e EXTRAPOLATE, --extrapolate EXTRAPOLATE | ||
| The domain range of the first table is automatically | ||
| detected by the code: [d_low, d_up]. While the second | ||
| table ranges from the first table's upper | ||
| boundary(d_up) to the extrapolate(parameter) * d_up: | ||
| [d_up, extrapolate * d_up] (default: 5) | ||
| -f FREQUENCY, --frequency FREQUENCY | ||
| The frequency of tabulation overflow check(Whether the | ||
| input environment matrix overflow the first or second | ||
| table range). By default do not check the overflow | ||
| (default: -1) | ||
| -c CHECKPOINT_FOLDER, --checkpoint-folder CHECKPOINT_FOLDER | ||
| path to checkpoint folder (default: .) | ||
| -t TRAINING_SCRIPT, --training-script TRAINING_SCRIPT | ||
| The training script of the input frozen model | ||
| (default: None) | ||
| ``` | ||
| **Parameter explanation** | ||
|
|
||
| Model compression, which including tabulating the embedding-net. | ||
| The table is composed of fifth-order polynomial coefficients and is assembled from two sub-tables. The first sub-table takes the stride(parameter) as it's uniform stride, while the second sub-table takes 10 * stride as it's uniform stride. | ||
| The range of the first table is automatically detected by deepmd-kit, while the second table ranges from the first table's upper boundary(upper) to the extrapolate(parameter) * upper. | ||
| Finally, we added a check frequency parameter. It indicates how often the program checks for overflow(if the input environment matrix overflow the first or second table range) during the MD inference. | ||
|
|
||
| **Justification of model compression** | ||
|
|
||
| Model compression, with little loss of accuracy, can greatly speed up MD inference time. According to different simulation systems and training parameters, the speedup can reach more than 10 times at both CPU and GPU devices. At the same time, model compression can greatly change the memory usage, reducing as much as 20 times under the same hardware conditions. | ||
|
|
||
| **Acceptable original model version** | ||
|
|
||
| The model compression method requires that the version of DeePMD-kit used in original model generation should be 1.3 or above. If one has a frozen 1.2 model, one can first use the convenient conversion interface of DeePMD-kit-v1.2.4 to get a 1.3 executable model.(eg: ```dp convert-to-1.3 -i frozen_1.2.pb -o frozen_1.3.pb```) |
Oops, something went wrong.