CHESS: GE Detector
After you have completed the setup of the rietveld environment, setup the files used for this example using:
milk-examples -e 4
This will copy the files for the detector calibration and integration example into the current working directory. Note that due to the way MILK is installed, scripts inside the MILK/bin folder does not need ".py" in the call to be recognized. But in instances where a local python script is needed, we will see the use of the full command python script.py.
The goal of this example is to provide 3 ways of representing Debye-Sherrer ring data via 1D integration, 2D integration, and calibration of detector parameters.
This exercise utilizes the Python Fast Azimuthal Integration (pyFAI) package that is already installed in the rietveld conda environment. From the pyFAI documentation:
"PyFAI uses a 6-parameter geometry definition similar, while not rigorously identical to SPD: One distance, 2 coordinates to define the point of normal incidence and 3 rotations around the main axis; these parameters are saved in text files usually with the .poni extension. In addition, the poni-file may contain the wavelength and the detector definition."
The scripts created for this example will utilize the .poni file and other Maud and detector parameters to perform refinement directly on the Debye-Sherrer ring data and produce fitted diffraction patterns.
Inside the folder "calibration_files" contains:
- A tiff image of the measured rings, "CeO2_126.tif"
- A mask file, "mask.edf", that provides a mask for pyFAI to identify the valid pixels on the image. The valid pixels corresponds to 0, and is a uniquely generated file corresponding to the specific experiment and instrument. For more details, users are referred to the documentation: pyFAI mask convention, pyFAI mask generation
- A points file, "points.npt", that contains the set of control point used by pyFAI to calibrate the geometry of a scattering experiment.
- A provided poni file that we will now provide a walk-through to generate using pyFAI, assuming that we do not have detector information.
With the rietveld conda environment active, navigate to "calibration_files" subfolder and launch pyFAI using:
pyFAI-calib2 CeO2_126.tif -w 0.1839 -m mask.edf -n points.npt -c CeO2 --fix-rot3
In this command, we are specifying the diffraction image, "CeO2_126.tif", as a required argument, and we are also providing additional information such as the wavelength1, 0.1839 angstrom, the mask file, "mask.edf", an initial set of points identifying the rings, "points.npt", the calibrant, "CeO2", and rot3 parameter is fixed. These information can be left blank and entered later in the pyFAI GUI interface.
1Wavelength of the beam is often measured as part of the experiment process, but in case that it is not known, it is possible to refine multiple detector positions with the intensities to obtain the wavelength after removing the correlation.
Use the colormap tool 
to adjust the color range to better visualize the rings:
Next, enter detector description manually by clicking on the three dots in the detector box -> Manual definition -> enter the pixel and detector sizes -> click "OK":
Click "Next >" onto "Mask". Since a mask is provided in this tutorial, nothing needs to be done here. But you can manually draw the mask using the tools in this section and then save the mask for other datasets from the same detector.
Click "Next >" onto "Peak picking". With the provided points.npt file, we can see that 17 rings are already identified. Just to exemplify the functionality, click the "X" button to the right of the ring lists for rings 6 and above. Enter "12" in "Amount of rings to extract", and "3" in "number of peaks per degree". The number of peaks per degree is essentially a peak density. Then, click "Extract more rings" and we will see the original 17 rings are now identified and peaks populated. If you over-count the rings and some partial rings are selected, you can simply delete them.
When clicking "Next >", pyFAI automatically begins fitting the model to the peaks previously identified. The parameters can be adjusted and re-fit if needed. Then, pyFAI will perform integration in the next section. The top plot shows a 2D visualization of each peak on the rings, and the bottom plot shows the summed results. You can click and drag on the bottom intensity plot to zoom in on the pattern. The data is not dark-field corrected, therefore we see the vertical lines behind the pattern.
The final output from this process is the .poni file. There is already the corresponding saved .poni file in the folder, so we can skip this step and close pyFAI without saving the file for now.
Additional thoughts:
The pyFAI package has a pyFAI-average -h tool that can be used to average out a set of dark current images using mean or median filter (along the image stack). One can also reject outliers be specifying a cutoff (remove cosmic rays / zingers from dark).
pyFAI can also be used to merge many images from the same sample when using a small beam and reduce the spotty-ness
of Debye-Sherrer rings. In this case the "max-filter" is usually recommended: pyFAI-average CeO2.ge2 -d dark.ge2 -F tif
After setting up the calibration files, with the other provided files in the example, we can simply call ./tutorial.sh and run everything. The following section will go into more detail of the setup of the bash script and other parameter files so that you can modify them for your data.
The "tutorial.sh" bash script ("tutorial.bat" batch script if on Windows) ultimately calls "refine_no_ang_cal.py" and "refine_detector_ang_cal.py" that begins the Maud refinement steps for calibration without detector and with detector, respectively. The ".par" files are similar to what we have seen in previous examples that contain Maud parameters:
echo "starting MAUD calibration refinements"
echo "for No_ang_cal_1d.par"
python refine_no_ang_cal.py "No_ang_cal_1d.par"
echo "for No_ang_cal_2d.par"
python refine_no_ang_cal.py "No_ang_cal_2d.par"
echo "for Detector_ang_cal_2d.par"
python refine_detector_ang_cal.py "Detector_ang_cal_2d.par"To generate the ".par" files, "tutorial.sh" calls "milk-egg-loader.py":
## Load data into MAUD parameter files
echo ""Loading data into MAUD parameter files
milk-esg-loader \
--interface command \
--maud-input-par "templates/detector_ang_cal.par" \
--maud-detectors "det0" \
--esg-files "results/CeO2_126_det0_2d.esg" \
--poni-files "calibration_files/det0.poni" \
--maud-output-par "Detector_ang_cal_2d.par" \
--maud-run-import
milk-esg-loader \
--interface command \
--maud-input-par "templates/no_ang_cal.par" \
--maud-detectors "det0" \
--esg-files "results/CeO2_126_2d.esg" \
--maud-output-par "No_ang_cal_2d.par" \
--maud-run-import
milk-esg-loader \
--interface command \
--maud-input-par "templates/no_ang_cal.par" \
--maud-detectors "det0" \
--esg-files "results/CeO2_126.esg" \
--maud-output-par "No_ang_cal_1d.par" \
--maud-run-importThe esg-loader script reads the input files "no_ang_cal.par" and "detector_ang_cal.par" that serve as templates to be updated by inputs from the generated detector file "det0.poni" and esg-files. Finally, the script prints out the maud_output_par files.
The esg files are created by milk-integrate.py that is called by "tutorial.sh" as follows:
## 1D and 2D (caking) integration using MILKs multigeometry pyFAI tool
echo "Doing the integration"
milk-integrate \
"calibration_files/CeO2_126.tif" \
--json "1d.azimint.json" \
--output "results" \
--overwrite \
--poolsize 1 \
--format "esg" \
--histogram_plot
milk-integrate \
"calibration_files/CeO2_126.tif" \
--json "2d.azimint.json" \
--output "results" \
--overwrite \
--poolsize 1 \
--format "esg1" "esg_detector"\
--histogram_plotThe "1d.azimint.json" and "2d.azimint.json" files are identical apart from the option "npt_azimuth" where for 1d integration it is "1" and for 2d integration it is "72". "milk-integrate.py" can be called to generate the json files with the included "write_json" function, but at the moment, the user would need to modify the function directly. As shown right now, the function is setup for writing a 2d integration json:
def write_json():
"""Write a template json file for integration."""
with open('template.azimint.json', 'w') as f:
json_object = json.dump({
"poni_file": [""],
"do_mask": True,
"mask_file": [""],
"do_dark": False,
"dark_file": [""],
"dark_norm": 1.0,
"do_bright": False,
"bright_file": [""],
"bright_norm": 1.0,
"data_ops": [""],
"do_polarization": False,
"polarization_factor": 0.99,
"do_2D": False,
"unit": "2th_deg",
"npt_radial": 2000,
"do_radial_range": True,
"radial_range": [8, 70],
"npt_azimuth": 72,
"do_azimuthal_range": False,
"azimuth_range": [0, 360],
"chi_discontinuity_at_0": True,
"do_solid_angle": True,
"do_remove_nan": True,
"error_model": "Poisson",
"method": [
"full",
"histogram",
"cython"
],
"opencl_device": "cpu"
}, f)The output "esg" format will print one esg file per loop. In the case of 1D integration, there is only one loop. But in the case of 2D integration, it will generate a series of files with one loop in each. To condense the files, use "esg1" format and it will simply append the integration data of each loop after each other into one single file. Additionally, the option "esg_detector" will also output the data formatted for inclined detector geometry. To do so, the script checks a pickled "binned_detector_coord0" file that contains coordinate data that allows interpolation between the binned detector pixels for integration.
Finally, in the simple_results.txt file, we can see the refinement process for the 3 runs. They are in order of being processed, i.e. first 6 are the 1D integration refinement, middle 6 are the 2D integration refinement, and the last 7 are the 2D detector refinement.
The ".par" files are also updated during the refinement process. We can see the improvement in the "rwp%" column which is essentially a best-of-fit indicator. From this point on, using the .poni file and the refined .par files, we would be ready to proceed to process other measurements now that we have calibrated for the detector's position and geometry.
From this point, we can run milk-poni-export to take the refined detector parameters and generate an adjusted .poni file:
milk-poni-export Detector_ang_cal_2d.par -d det0 -o results
The final fit looks like:
