-
Notifications
You must be signed in to change notification settings - Fork 0
/
README.tex
246 lines (184 loc) · 10.5 KB
/
README.tex
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
\subsection{Installation and Setup}\label{installation-and-setup}
It is strongly advised to install scipy, numpy and shapely (globaly) as
binary\\packages and not through virtualenv, because of their large
number of non-python\\dependencies. E.g. on Debian/Ubuntu systems use\\
\texttt{sudo\ apt-get\ install\ python3-numpy\ python3-scipy\ python3-shapely}\\to
install. Make sure access to the global packages is activated in the
virtual\\environment (e.g.~through \texttt{toggleglobalsitepackages} in
\texttt{virtualenvwrapper}).
When compiling entirely from source, You also might need to install the
following\\libraries: (for shapely): libgeos-dev; (for scipy):
libblas-dev, liblapack-dev,\\gfortran.
\section{Import OSM Data}\label{import-osm-data}
Imports the OSM data for Germany. The following tables are created:
Country, State, District and Cities. The data is\\imported from
\texttt{germany-latest.osm.pbf}, obtained from
\href{http://www.geofabrik.de}{geofabrik}. The pbf file needs to be
placed in \texttt{data/}.
The importer used is \texttt{imposm3}, which is included in the vagrant
setup (and will be executed within the VM). The import\\uses a custem
mapping, which is provided in \texttt{importer/mapping.json}. The import
is conducted in two steps: First, the\\data is imported to the tables
Osm\_Admin and Osm\_Places. Second the Country, State, District and
Cities tables are created\\from these tables.
\subsection{Usage}\label{usage}
Prerequisite: No contrib tables, delete if existing:\\ python manage.py
drop\_tables -t contrib
To import all the OSM data use the manage script. To invoke the whole
pipeline use the following command (will take several hours):
\texttt{python\ manage.py\ import\_osm\ -\/-imposm\ -\/-simplify\ -\/-load\ -\/-drop\_tables}
\begin{itemize}
\itemsep1pt\parskip0pt\parsep0pt
\item
\texttt{-\/-imposm} the first import step (see above)
\item
\texttt{-\/-simplify} simplify all map data (borders)
\item
\texttt{-\/-load} the second import step (see above)
\item
\texttt{-\/-drop-tables} deletes the Osm\_Admin and Osm\_Places tables
after a successful import
\end{itemize}
All the above steps can be invoked separately.
\subsection{Mac OS X}\label{mac-os-x}
\begin{verbatim}
python manage.py import_osm --drop_tables --imposm --load
DYLD_LIBRARY_PATH=/Applications/Postgres.app/Contents/Versions/9.3/lib python manage.py import_osm --drop_tables --load
\end{verbatim}
\section{Import DWD Data}\label{import-dwd-data}
Imports weather observation data from the DWD (Deutscher Wetterdienst).
The importer is a adopted version
from\\\href{https://github.com/cholin/fuberlin_spatial_db_project}{cholin}.
Per default, it downloads all the {[}recent daily
observations{]}\\(\url{ftp://ftp.dwd.de/pub/CDC/observations_germany/climate/daily/kl}).
Details on the importer from
\href{https://github.com/cholin/fuberlin_spatial_db_project/blob/master/scripts/dwd/README.md}{cholin}:
\begin{quote}
The importer downloads the station summary file to get a list of all
weather stations. After that it downloads for each\\station the
corresponding zip file (with measurement data), extracts it in-memory
and parses it. To get information about\\which weather station is the
nearest for a given point, it also calculates a region polygon for each
station. This is done\\by computing the voronoi diagram for all
stations. The resulting regions may be outside of the country germany.
To avoid\\this there is a polygon of the border of germany (data is from
naturalearthdata.com - country extraction and exportation\\as geojson
with qgis). For each region we calculate the intersection with this
polygon and use the result as final region\\(Multi)Polygon.
\end{quote}
\subsection{Usage}\label{usage-1}
Use the importer with the \texttt{manage.py} script:
To download all data and import all observation data:
\texttt{python\ manage.py\ import\_dwd}
Create an intermediate result in \texttt{data/weather.json}:
\texttt{python\ manage.py\ import\_dwd\ -\/-to\_json}
Import the intermediate result from \texttt{data/weather.json}:
\texttt{python\ manage.py\ import\_dwd\ -\/-from\_json}
\subsection{Mac OS X}\label{mac-os-x-1}
\begin{verbatim}
python manage.py import_dwd --to_json
DYLD_LIBRARY_PATH=/Applications/Postgres.app/Contents/Versions/9.3/lib ython manage.py import_dwd --from_json
\end{verbatim}
\section{Download and Import NOAA GFS Data
(Forecasts)}\label{download-and-import-noaa-gfs-data-forecasts}
Importing the Forecast Data is done in two steps. First you have to
download the GRIB files from the NOAA FTP servers. Then you have to
import them as Postgis Raster.
\subsection{Download}\label{download}
For downloading the GFS data a date range and a target directory has to
be specified. The format for the start and enddate is
\texttt{YYYYMMDDHH} or \texttt{latest} for the most recently available
GFS calculation.\\Optionally the forecast hours can be specified as a
range (Defaults to download from 0 to 129 in 3 hour steps).
\begin{verbatim}
usage: run_gfs.py download [-h] [--hours_start HOURS_START]
[--hours_stop HOURS_STOP] [--hours_step HOURS_STEP]
[startdate] [enddate] datadir
\end{verbatim}
For example, assuming data should be stored to \texttt{data/forecasts}:
\texttt{./run\_gfs.py\ download\ 2014121112\ 2015011306\ data/forecasts}
\subsection{Import}\label{import}
To import the downloaded data, the download directory and a data range
has to be specified:
\begin{verbatim}
usage: run_gfs.py import [-h] datadir [startdate] [enddate]
\end{verbatim}
For example, assuming the data is stored in \texttt{data/forecasts}:
\texttt{./run\_gfs.py\ import\ data/forecasts\ 2014121112\ 2015011306}
\subsection{Build Contrib Tables}\label{build-contrib-tables}
To speed up some queries, the area contribution of the region (voronoi
cell) of weather stations to states and districts is precomputed and
materialized. Run \texttt{python\ manage.py\ calculate\_contrib\_area}
to create and fill the ContribState and ContribDistrict tables.
\section{Troubleshooting}\label{troubleshooting}
\subsection{Mac OS X}\label{mac-os-x-2}
\subsubsection{UnicodeEncodeError}\label{unicodeencodeerror}
Python inherist the standard locale from the current shell environment.
If this is not set to utf8 it tries to convert\\to ASCII, which
produces.
\texttt{UnicodeEncodeError:\ \textquotesingle{}ascii\textquotesingle{}\ codec\ can\textquotesingle{}t\ encode\ character}\\Test
with \texttt{\$\ locale}, this should show utf-8. If not, fix with
\begin{verbatim}
export LANG=en_US.UTF-8
export LC_ALL=en_US.UTF-8
\end{verbatim}
\subsubsection{libssl / libcrypto Error from
psycopq}\label{libssl-libcrypto-error-from-psycopq}
The \texttt{libssl} version Mac OS X uses might be too old for
\texttt{psycopg}, resulting in an error like the following:
\begin{verbatim}
...
ImportError: dlopen(...lib/python3.4/site-packages/psycopg2/_psycopg.so, 2): Library not loaded: libssl.1.0.0.dylib
Referenced from: ...lib/python3.4/site-packages/psycopg2/_psycopg.so
Reason: image not found
\end{verbatim}
This can be solved by changing the dynamic shared library install names
in the \texttt{psycopq} binary. First, find out the version
\texttt{psycopq} is using:
\begin{verbatim}
otool -L /Users/jvf/miniconda3/envs/env-sw/lib/python3.4/site-packages/psycopg2/_psycopg.so
$ /Users/jvf/miniconda3/envs/env-sw/lib/python3.4/site-packages/psycopg2/_psycopg.so:
/usr/local/lib/libpq.5.dylib (compatibility version 5.0.0, current version 5.6.0)
libssl.1.0.0.dylib (compatibility version 1.0.0, current version 1.0.0)
libcrypto.1.0.0.dylib (compatibility version 1.0.0, current version 1.0.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1213.0.0)
/usr/lib/libgcc_s.1.dylib (compatibility version 1.0.0, current version 283.0.0)
\end{verbatim}
Now, change the the shared libraries for \texttt{libssl} and
\texttt{libcrypto} (using the libraries provided by
\href{http://postgresapp.com}{Postgres.app}):
\begin{verbatim}
install_name_tool -change libssl.1.0.0.dylib /Applications/Postgres.app/Contents/Versions/9.3/lib/libssl.1.0.0.dylib /Users/jvf/miniconda3/envs/env-sw/lib/python3.4/site-packages/psycopg2/_psycopg.so
install_name_tool -change libcrypto.1.0.0.dylib /Applications/Postgres.app/Contents/Versions/9.3/lib/libcrypto.1.0.0.dylib /Users/jvf/miniconda3/envs/env-sw/lib/python3.4/site-packages/psycopg2/_psycopg.so
\end{verbatim}
\texttt{psycopq} now uses the correct libraries:
\begin{verbatim}
otool -L /Users/jvf/miniconda3/envs/env-sw/lib/python3.4/site-packages/psycopg2/_psycopg.so
$ /Users/jvf/miniconda3/envs/env-sw/lib/python3.4/site-packages/psycopg2/_psycopg.so:
/usr/local/lib/libpq.5.dylib (compatibility version 5.0.0, current version 5.6.0)
/Applications/Postgres.app/Contents/Versions/9.3/lib/libssl.1.0.0.dylib (compatibility version 1.0.0, current version 1.0.0)
/Applications/Postgres.app/Contents/Versions/9.3/lib/libcrypto.1.0.0.dylib (compatibility version 1.0.0, current version 1.0.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1213.0.0)
/usr/lib/libgcc_s.1.dylib (compatibility version 1.0.0, current version 283.0.0)
\end{verbatim}
It is strongly recommended to do all this in an virtual environment to
not mess up your system!
Source: More Information:
\href{http://superuser.com/a/721564}{superuser.com}
Another possibilty is to prefix commands with
\texttt{DYLD\_LIBRARY\_PATH} and \texttt{DYLD\_FRAMEWORK\_PATH}, but
this works less reliable and potentially messes up the linking of other
libraries. Example:
\begin{verbatim}
DYLD_LIBRARY_PATH=$(HOME)/lib:/usr/local/lib:/lib:/usr/lib:/Applications/Postgres.app/Contents/Versions/9.3/lib,DYLD_FRAMEWORK_PATH=/Library/Frameworks:/Network/Library/Frameworks:/System/Library/Frameworks python manage.py import_dwd
\end{verbatim}
providing an alternative path for a newer version of \texttt{libssl} to
the dynamic linker (in this example the libs from
\href{http://postgresapp.com}{Postgres.app} are used, but can link
against a \texttt{homebrew} installed version as well):
\begin{verbatim}
export DYLD_LIBRARY_PATH=$(HOME)/lib:/usr/local/lib:/lib:/usr/lib:/Applications/Postgres.app/Contents/Versions/9.3/lib
export DYLD_FRAMEWORK_PATH=/Library/Frameworks:/Network/Library/Frameworks:/System/Library/Frameworks
\end{verbatim}
Source:
\href{http://stackoverflow.com/questions/11365619/psycopg2-installation-error-library-not-loaded-libssl-dylib}{stackoverflow.com}