-
Notifications
You must be signed in to change notification settings - Fork 18
Using DataMap Files In IDL
Original Author: R. J. Barnes (JHU/APL)
This tutorial shows you how to read and write DataMap files using the DataMap IDL library (datamap.pro). The DataMap file format is the underlying format used to store the fitacf and rawacf SuperDARN, new style, data files.
The datamap library makes extensive use of IDL pointers and users should first familiarize themselves with the concepts involved (please refer to the appropriate section of the IDL documentation). Simply put, a pointer is a short way to refer to a data structure that allows code to access and modify the contents of a data structure without having to make a copy of it. In traditional programming languages pointers are usually the memory address of a data structure. To access the contents of a data structure referenced by a pointer, you must use the derefence operator, which is * (the asterisk):
a=ptr_new(10.2)
print, 'value at pointer',*a
IDL uses heap variables to store the actual variable that the pointer points to. Heap variables are claimed from a special area of memory and to prevent memory leakage is should be released when the pointer is no longer needed. In the example above, a floating point heap variable is created to store the value 10.2 and the pointer is set to reference this heap variable. Whe n the pointer is no longer required it should be destroyed by calling PTR_FREE.
The DataMap format is described in detail here. There are two types of variables in a DataMap file, scalars and arrays and they correspond to the same concepts in IDL; a scalar has a single value, and an array has multiple can have multiple values and dimensions but they must all be of the same type. The DataMap format supports a limited sub-set of IDL data types:
IDL Type Content Code Number
Byte 8-bit unsigned integer 1
Integer 16-bit signed integer 2
Long 32-bit signed integer 3
Floating-point Single precision floating point number 4
Double-precision Double precision floating point number 8
String Text String 9
Each variable in a DataMap file has an associated name that uniquely identifies it. Usually the variable name in the DataMap file will correspond to an IDL variable name.
The datamap library uses two 1-dimensional vector arrays to store information about the scalars and arrays in the DataMap file. Each element of the vector array is a structure containing the name, type and pointer to a variable. The number of elements in each vector corresponds to the number of scalars and vectors in a record of a DataMap file.
Lets create a simple DataMap file using the library:
pro dmapwrite1
; define some scalars
year=2004;
month='August'
day=30
noise=3.45
power=1.045D
; define some arrays
pulse=[0,3,6,10,15]
location=dindgen(3,3,3)
matrix=strarr(2,2)
matrix[0,0]='parameter A'
matrix[0,1]='parameter B'
matrix[1,0]='parameter C'
matrix[1,1]='parameter D'
; build the scalar vector
s=DataMapMakeScalar('year',year,sclvec,/new)
s=DataMapMakeScalar('month',month,sclvec);
s=DataMapMakeScalar('day',day,sclvec);
s=DataMapMakeScalar('noise',noise,sclvec);
s=DataMapMakeScalar('power',power,sclvec);
; build the array vector
s=DataMapMakeArray('pulse',pulse,arrvec,/new)
s=DataMapMakeArray('location',location,arrvec)
s=DataMapMakeArray('matrix',matrix,arrvec)
openw,unit,'test.dat',/get_lun
s=DataMapWrite(unit,sclvec,arrvec)
if (s eq -1) then begin
print, 'Error writing file'
stop
endif
s=DataMapFreeScalar(sclvec)
s=DataMapFreeArray(arrvec)
free_lun,unit
end
The first few lines of the program define a few simple variables that we want to store in the DataMap file and represent a mixture of scalars and arrays. The first step in writing the DataMap file is to create the first entry in the scalar vector:
s=DataMapMakeScalar('year',year,sclvec,/new)
The function DataMapMakeScalar makes an entry in the scalar vector and creates a pointer to a heap variable that stores a copy of the supplied IDL variable, in this case year. The first argument to the function is the variable name to be recorded in the DataMap file. This does not necessarily have to be the same as the IDL variable name. The second argument is the IDL scalar varible to store and the third variable is the name of the scalar vector. In this case the keyword new has been used to indicate that sclvec is undefined and should be initialized. In the subsequent calls to sclvec this keyword is omitted. The function returns the index of the new entry within sclvec
A similar call is made to create the array vector:
s=DataMapMakeArray('pulse',pulse,arrvec,/new)
The function DataMapMakeArray makes an entry in the array vector and returns the index of the entry within arrvec. For the first call to this function the new keyword should also be used.
The next step is to write out the DataMap record to the open file:
openw,unit,'test.dat',/get_lun
s=DataMapWrite(unit,sclvec,arrvec)
The DataMapWrite function writes out the DataMap record specified by sclvec and arrvec to the logical unit number unit. It returns the size of the record in bytes if successful, or -1 if an error occurred
The final step is to tidy up and release the memory used by the heap variables:
s=DataMapFreeScalar(sclvec)
s=DataMapFreeArray(arrvec)
The functions DataMapFreeScalar and DataMapFreeArray free any pointers in the scalar and array vectors.
Having written a DataMap file, we now would like to read it back in:
pro dmapread1
; Open input file for reading
openr,unit,'test.dat',/get_lun
status=DataMapRead(unit,sclvec,arrvec)
if (status eq -1) then begin
print, 'file I/O error'
stop
endif
; locate the scalars of interest
yearid=DataMapFindScalar('year',2,sclvec)
monthid=DataMapFindScalar('month',9,sclvec)
dayid=DataMapFindScalar('day',2,sclvec)
noiseid=DataMapFindScalar('noise',4,sclvec)
powerid=DataMapFindScalar('power',8,sclvec)
; locate the arrays of interest
pulseid=DataMapFindArray('pulse',2,arrvec)
locationid=DataMapFindArray('location',8,arrvec)
matrixid=DataMapFindArray('matrix',9,arrvec)
; retreive the scalar values
year=*(sclvec[yearid].ptr)
month=*(sclvec[monthid].ptr)
day=*(sclvec[monthid].ptr)
noise=*(sclvec[noiseid].ptr)
power=*(sclvec[yearid].ptr)
pulse=*(arrvec[pulseid].ptr)
location=*(arrvec[locationid].ptr)
matrix=*(arrvec[matrixid].ptr)
s=DataMapFreeScalar(sclvec)
s=DataMapFreeArray(arrvec)
print, year,month,day,noise,power
print, pulse
print, location
print, matrix
free_lun,unit
end
The program opens the DataMap file just written and prints the contents on the terminal. The first step is to open the file and read the record:
status=DataMapRead(unit,sclvec,arrvec)
The function DataMapRead reads a single record from the DataMap file specified by the logical unit number given by unit. The routine stores the scalar and array vectors in the variables sclvec and arrvec. The function returns the number of bytes read from the file.
Having read a record from the file, we now need to locate the variables of interest:
yearid=DataMapFindScalar('year',2,sclvec)
monthid=DataMapFindScalar('month',9,sclvec)
dayid=DataMapFindScalar('day',2,sclvec)
noiseid=DataMapFindScalar('noise',4,sclvec)
powerid=DataMapFindScalar('power',8,sclvec)
The function DataMapFindScalar searches through sclvec looking for variables whose name match the text string given as the first argument and whose type code matches the number given as the second argument. The function returns the index into sclvec if a match is found or -1 if no match is found. The function DataMapFindArray performs the same search but on the array vector arrvec.
Having located our variables we now copy them into IDL variables by using the dereference operator:
year=*(sclvec[yearid].ptr)
month=*(sclvec[monthid].ptr)
day=*(sclvec[monthid].ptr)
noise=*(sclvec[noiseid].ptr)
power=*(sclvec[yearid].ptr)
Finally when we are done with this record we release the heap variable memory.
s=DataMapFreeScalar(sclvec)
s=DataMapFreeArray(arrvec)
A well written program should check the values returned by DataMapFindScalar and DataMapFindArray to make sure that the variables are actually stored in the DataMap file.
Similarly the program should also check the sclvec and arrvec are both defined. It is quite possible for a DataMap file to contain either only scalars or only arrays and if this is the case DataMapRead will return only initialize sclvec or arrvec. The program should use N_ELEMENT to check.
Of course if you want to ignore error checking you can simplify the code considerably:
year=*(sclvec[DataMapFindScalar('year',2,sclvec)].ptr)
You can speed up the writing of a multi-record file by pre-creating sclvec and arrvec before you start writing:
pro dmapwrite2
; build an empty scalar vector
yearid=DataMapMakeBlankScalar('year',2,sclvec,/new)
monthid=DataMapMakeBlankScalar('month',2,sclvec)
dayid=DataMapMakeBlankScalar('day',2,sclvec)
noiseid=DataMapMakeBlankScalar('noise',4,sclvec)
; build the array vector
posid=DataMapMakeBlankArray('pos',4,2,[2,2],arrvec,/new)
openw,unit,'test.dat',/get_lun
for day=0,30 do begin
*sclvec[yearid].ptr=2004
*sclvec[monthid].ptr=10
*sclvec[dayid].ptr=day
*sclvec[noiseid].ptr=0.1*day
pos=day*findgen(2,2)
*arrvec[posid].ptr=pos
s=DataMapWrite(unit,sclvec,arrvec)
if (s eq -1) then begin
print, 'Error writing file'
stop
endif
endfor
s=DataMapFreeScalar(sclvec)
s=DataMapFreeArray(arrvec)
free_lun,unit
end
This program uses the functions DataMapMakeBlankScalar and DataMapMakeBlankArray to create empty heap variables to store the data:
yearid=DataMapMakeBlankScalar('year',2,sclvec,/new)
The function is similar to DataMapMakeScalar but instead of passing in the scalar variable to store as an argument, you pass the data type code. The routine will then add a new element to sclvec and return its index. As before, the keyword new indicates the sclvec is undefined and should be created.
To create empty arrays useDataMapMakeArray:
posid=DataMapMakeBlankArray('pos',4,2,[2,2],arrvec,/new)
In addition to the data type code, this function takes the number of dimensions of the array as the third argument and as the fourth argument a vector that lists the range of each dimension. The function adds a new element to arrvec and returns its index.
Now when we need to write out a record we can use pointer dereferencing to directly set the values of the heap variables:
*sclvec[yearid].ptr=2004
*sclvec[monthid].ptr=10
*sclvec[dayid].ptr=day
*sclvec[noiseid].ptr=0.1*day
In this case we do not free up the heap variables until we are done writing the file.
In some cases you will only need to store either scalar or array variables. In this case you supply only vector argument to DataMapWrite and use either the keyword scalar or array to indicate the type of variable in the vector:
s=DataMapWrite(unit,sclvec,/scalar)
s=DataMapWrite(unit,arrvec,/array)
If you don't know the names and types of the variables in a record, you can find out this information by inspecting the sclvec and arrvec vectors directly:
print,'Scalar names and types:'
print,sclvec[*].name,sclvec[*].type
print,'Array names and types:'
print, arrvec[*].name,arrvec[*].type