PROJ core dumps when setting data directory on specific context and it is not set on global context #1574

snowman2 · 2019-08-31T01:28:38Z

This is a wierd edge case that has taken me some time to track down as the cause has been elusive for quite some time due to the core dumping. However, I believe the symptoms and the solution should help find a solution in PROJ I believe.

In the case where the data directory is not able to be determined by PROJ in the default locations on the global context, it will core dump when attempting to use proj_create with a perfectly valid custom context that has the proper data directory set. However, this does not happen in all instances. It only happens when using the +init= syntax.

For example:

>>> from pyproj import CRS
>>> cc = CRS("+proj=utm +zone=33 +ellps=WGS84 +datum=WGS84 +units=m +no_defs")
>>> cc = CRS(init="epsg:4326")
Segmentation fault (core dumped)

My guess is it has to do with:

PROJ/src/iso19111/c_api.cpp

Lines 437 to 440 in f059b90

    
           // Only connect to proj.db if needed 
        
           if (strstr(text, "proj=") == nullptr || strstr(text, "init=") != nullptr) { 
        
               getDBcontextNoException(ctx, __FUNCTION__); 
        
           }

The fix I applied in pyproj was to set the data directory for the global PROJ context along with the custom context.

The text was updated successfully, but these errors were encountered:

snowman2 · 2019-08-31T01:30:16Z

The tricky part is to have the data directory in a place where PROJ cannot find it and then use proj_context_set_search_paths to set the data directory on a different context.

snowman2 · 2019-08-31T01:35:59Z

Note: This issue began in PROJ 6.1 and still persists in 6.2

snowman2 · 2019-08-31T01:41:38Z

This is also problematic for pyproj wheels when the PROJ_LIB is set to an old PROJ data directory. The global context finds the old data directory without the proj.db and proceeds to core dump even though the custom context has the data directory paths set properly..

rouault · 2019-09-04T12:52:47Z

I tried to reproduce, but couldn't manage. A bit confused by your explanations. Could you give an exact way to reproduce (possibly with a standalone C file) or perhaps display the stack trace where it crashes ?

gdb python
[ set args myscript.py ]
run
[ python code and wait for it to crash ]
bt

kbevers · 2019-09-04T13:01:08Z

possibly with a standalone C file

As a side note, I generally find it quite difficult to debug problems presented in the form of pyproj code. If the problem can be reproduced using PROJ apps or a bit of C code that it is preferable.

snowman2 · 2019-09-04T13:03:17Z

I definitely see that the explanation is confusing. Code is definitely a good way to communicate. Hopefully this will help:

Step 1:

Move the PROJ data directory to a non-standard location where it cannot be auto detected and is not set by PROJ_LIB.

Step 2:

Create a new context and set the search paths without setting it in the global context. Then, attempt to create a CRS using the init= syntax:

std::string dirname = "/path/to/proj_data/share234/proj";
auto ctx = proj_context_create();

const char *path = dirname.c_str();
proj_context_set_search_paths(ctx, 1, &path);
proj_context_use_proj4_init_rules(ctx, 1);
auto P = proj_create(ctx, "+init=epsg:4326 +type=crs");

Updated to allow init rules ^^

rouault · 2019-09-04T13:14:19Z

I've tested your above (updated) script, and can't make it crash

$ PROJ_DEBUG=5 LD_LIBRARY_PATH=src/.libs ./mytest 
pj_open_lib(proj.db): call fopen(/path/to/proj_data/share234/proj/proj.db) - failed
internal_proj_create: Cannot find proj.db
pj_open_lib(proj.db): call fopen(/path/to/proj_data/share234/proj/proj.db) - failed
pj_open_lib(epsg): call fopen(/home/even/proj/install-proj-master/share/proj/epsg) - failed
internal_proj_create: no database context specified

The hardcoded data path is (/home/even/proj/install-proj-master/share/proj and I renamted it to (/home/even/proj/install-proj-master/share/proj.disabed

snowman2 · 2019-09-04T13:23:42Z

One step closer. Can you try with:

std::string dirname = "/home/even/proj/install-proj-master/share/proj.disabed";

The reason is that the new context needs the path to the PROJ data directory in its moved form while the main/global PROJ context cannot find it.

rouault · 2019-09-04T13:31:33Z

One step closer. Can you try with:

std::string dirname = "/home/even/proj/install-proj-master/share/proj.disabed";

Still no crash. New output is

PROJ_DEBUG=5 LD_LIBRARY_PATH=src/.libs ./mytest 
pj_open_lib(proj.db): call fopen(/home/even/proj/install-proj-master/share/proj.dis/proj.db) - succeeded
pj_open_lib(epsg): call fopen(/home/even/proj/install-proj-master/share/proj/epsg) - failed

snowman2 · 2019-09-05T12:31:33Z

Hmmm, strange. Wonder what the difference is on my end. That's where it core dumps in the pyproj code. I wonder if running as a C program instead of Cpp makes any difference?

rouault · 2019-09-05T12:59:47Z

I wonder if running as a C program instead of Cpp makes any difference?

I doubt so. Why not running your python code under gdb and displaying the stack trace when it crashes as I suggested above in #1574 (comment) ?

snowman2 · 2019-09-05T13:17:07Z

The output is here: pyproj4/pyproj#368 (comment)

snowman2 · 2019-09-05T13:18:17Z

Another one is here: pyproj4/pyproj#351

rouault · 2019-09-05T13:34:07Z

In pyproj4/pyproj#368, there's not the full backtrace and no line number, so hard to do something with it.
pyproj4/pyproj#351 is a bit more interesting as it shows an infinite loop.
Commit 149bd81 was supposed to fix that, or something rather similar.
I'm afraid I can't do more without an exact reproducer.

snowman2 · 2019-09-05T13:58:12Z

Makes sense. I will see if I can get better debugging information later.

snowman2 · 2019-09-05T14:08:31Z

Program received signal SIGSEGV, Segmentation fault.
0x00007ffff6a169ac in osgeo::proj::io::createFromUserInput(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::shared_ptr<osgeo::proj::io::DatabaseContext> const&, bool, projCtx_t*) ()
   from /home/snowal/miniconda3/envs/proj62/lib/libproj.so.15
(gdb) bt
#0  0x00007ffff6a169ac in osgeo::proj::io::createFromUserInput(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::shared_ptr<osgeo::proj::io::DatabaseContext> const&, bool, projCtx_t*) ()
   from /home/snowal/miniconda3/envs/proj62/lib/libproj.so.15
#1  0x00007ffff6a1a318 in osgeo::proj::io::createFromUserInput(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, projCtx_t*) () from /home/snowal/miniconda3/envs/proj62/lib/libproj.so.15
#2  0x00007ffff6a45d22 in proj_create ()
   from /home/snowal/miniconda3/envs/proj62/lib/libproj.so.15
#3  0x00007ffff6a94f0e in get_init(projCtx_t*, char const*, int) ()
   from /home/snowal/miniconda3/envs/proj62/lib/libproj.so.15
#4  0x00007ffff6a95182 in pj_expand_init_internal(projCtx_t*, ARG_list*, int)
    () from /home/snowal/miniconda3/envs/proj62/lib/libproj.so.15
#5  0x00007ffff6a0e313 in osgeo::proj::io::PROJStringParser::createFromPROJString(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) () from /home/snowal/miniconda3/envs/proj62/lib/libproj.so.15
#6  0x00007ffff6a16a9e in osgeo::proj::io::createFromUserInput(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::shared_ptr<osgeo::proj::io::DatabaseContext> const&, bool, projCtx_t*) ()
   from /home/snowal/miniconda3/envs/proj62/lib/libproj.so.15
----Type <return> to continue, or q <return> to quit---
#31 0x00007ffff6a1a318 in osgeo::proj::io::createFromUserInput(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, projCtx_t*) () from /home/snowal/miniconda3/envs/proj62/lib/libproj.so.15
#32 0x00007ffff6a45d22 in proj_create () from /home/snowal/miniconda3/envs/proj62/lib/libproj.so.15
#33 0x00007ffff6a94f0e in get_init(projCtx_t*, char const*, int) () from /home/snowal/miniconda3/envs/proj62/lib/libproj.so.15
#34 0x00007ffff6a95182 in pj_expand_init_internal(projCtx_t*, ARG_list*, int) () from /home/snowal/miniconda3/envs/proj62/lib/libproj.so.15
#35 0x00007ffff6a0e313 in osgeo::proj::io::PROJStringParser::createFromPROJString(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) () from /home/snowal/miniconda3/envs/proj62/lib/libproj.so.15
#36 0x00007ffff6a16a9e in osgeo::proj::io::createFromUserInput(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::shared_ptr<osgeo::proj::io::DatabaseContext> const&, bool, projCtx_t*) () from /home/snowal/miniconda3/envs/proj62/lib/libproj.so.15
#37 0x00007ffff6a1a318 in osgeo::proj::io::createFromUserInput(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, projCtx_t*) () from /home/snowal/miniconda3/envs/proj62/lib/libproj.so.15
#38 0x00007ffff6a45d22 in proj_create () from /home/snowal/miniconda3/envs/proj62/lib/libproj.so.15
#39 0x00007ffff6a94f0e in get_init(projCtx_t*, char const*, int) () from /home/snowal/miniconda3/envs/proj62/lib/libproj.so.15
#40 0x00007ffff6a95182 in pj_expand_init_internal(projCtx_t*, ARG_list*, int) () from /home/snowal/miniconda3/envs/proj62/lib/libproj.so.15
#41 0x00007ffff6a0e313 in osgeo::proj::io::PROJStringParser::createFromPROJString(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) () from /home/snowal/miniconda3/envs/proj62/lib/libproj.so.15
#42 0x00007ffff6a16a9e in osgeo::proj::io::createFromUserInput(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::shared_ptr<osgeo::proj::io::DatabaseContext> const&, bool, projCtx_t*) () from /home/snowal/miniconda3/envs/proj62/lib/libproj.so.15
#43 0x00007ffff6a1a318 in osgeo::proj::io::createFromUserInput(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, projCtx_t*) () from /home/snowal/miniconda3/envs/proj62/lib/libproj.so.15
#44 0x00007ffff6a45d22 in proj_create () from /home/snowal/miniconda3/envs/proj62/lib/libproj.so.15
#45 0x00007ffff6a94f0e in get_init(projCtx_t*, char const*, int) () from /home/snowal/miniconda3/envs/proj62/lib/libproj.so.15
#46 0x00007ffff6a95182 in pj_expand_init_internal(projCtx_t*, ARG_list*, int) () from /home/snowal/miniconda3/envs/proj62/lib/libproj.so.15
#47 0x00007ffff6a0e313 in osgeo::proj::io::PROJStringParser::createFromPROJString(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) () from /home/snowal/miniconda3/envs/proj62/lib/libproj.so.15
#48 0x00007ffff6a16a9e in osgeo::proj::io::createFromUserInput(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::shared_ptr<osgeo::proj::io::DatabaseContext> const&, bool, projCtx_t*) () from /home/snowal/miniconda3/envs/proj62/lib/libproj.so.15
#49 0x00007ffff6a1a318 in osgeo::proj::io::createFromUserInput(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, projCtx_t*) () from /home/snowal/miniconda3/envs/proj62/lib/libproj.so.15
#50 0x00007ffff6a45d22 in proj_create () from /home/snowal/miniconda3/envs/proj62/lib/libproj.so.15
#51 0x00007ffff6a94f0e in get_init(projCtx_t*, char const*, int) () from /home/snowal/miniconda3/envs/proj62/lib/libproj.so.15
#52 0x00007ffff6a95182 in pj_expand_init_internal(projCtx_t*, ARG_list*, int) () from /home/snowal/miniconda3/envs/proj62/lib/libproj.so.15
#53 0x00007ffff6a0e313 in osgeo::proj::io::PROJStringParser::createFromPROJString(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) () from /home/snowal/miniconda3/envs/proj62/lib/libproj.so.15
#54 0x00007ffff6a16a9e in osgeo::proj::io::createFromUserInput(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&---Type <return> to continue, or q <return> to quit---

snowman2 · 2019-09-05T14:09:44Z

Seems to be the infinite loop problem.

rouault · 2019-09-05T14:58:38Z

oh! I can finally reproduce with your procedure #1574 (comment) + #1574 (comment) . The extra needed step is to have a 'epsg' file coming from PROJ 5.2 in the default PROJ resource directory. Or equivalently have PROJ_LIB pointing to the resource directory of PROJ 5.2

PROJStringParser::createFromPROJString(): avoid potential infinite recursion (fixes #1574)

…cursion (fixes #1574) The exact circumstances are a bit difficult to explain, but they involve using a non-default context, enabling proj_context_use_proj4_init_rules() on it, using proj_create(ctxt, "+init=epsg:XXXX +type=crs"), whereas PROJ_LIB is defined to a directory that has a 'epsg' file in it.

[Backport 6.2] PROJStringParser::createFromPROJString(): avoid potential infinite recursion (fixes #1574)

snowman2 mentioned this issue Aug 31, 2019

Kernel dies when executing to_crs geopandas/geopandas#1113

Closed

rouault added the awaiting_feedback Waiting feedback from reporter label Sep 4, 2019

rouault closed this as completed in 9c38bab Sep 6, 2019

rouault added a commit that referenced this issue Sep 6, 2019

Merge pull request #1589 from rouault/fix_1574

73ccf7e

PROJStringParser::createFromPROJString(): avoid potential infinite recursion (fixes #1574)

rouault added a commit that referenced this issue Sep 6, 2019

Merge pull request #1594 from OSGeo/backport-1589-to-6.2

3eb5780

[Backport 6.2] PROJStringParser::createFromPROJString(): avoid potential infinite recursion (fixes #1574)

snowman2 mentioned this issue Oct 17, 2019

Seg fault with python 2.7 rasterio 1.1.0 wheel rasterio/rasterio#1805

Closed

snowman2 mentioned this issue Oct 20, 2020

REF: Use pyproj global context instead of the one from PROJ pyproj4/pyproj#723

Merged

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

PROJ core dumps when setting data directory on specific context and it is not set on global context #1574

PROJ core dumps when setting data directory on specific context and it is not set on global context #1574

snowman2 commented Aug 31, 2019

snowman2 commented Aug 31, 2019 •

edited

Loading

snowman2 commented Aug 31, 2019

snowman2 commented Aug 31, 2019

rouault commented Sep 4, 2019 •

edited

Loading

kbevers commented Sep 4, 2019

snowman2 commented Sep 4, 2019 •

edited

Loading

rouault commented Sep 4, 2019

snowman2 commented Sep 4, 2019

rouault commented Sep 4, 2019

snowman2 commented Sep 5, 2019

rouault commented Sep 5, 2019

snowman2 commented Sep 5, 2019

snowman2 commented Sep 5, 2019

rouault commented Sep 5, 2019

snowman2 commented Sep 5, 2019

snowman2 commented Sep 5, 2019

snowman2 commented Sep 5, 2019

rouault commented Sep 5, 2019 •

edited

Loading

PROJ core dumps when setting data directory on specific context and it is not set on global context #1574

PROJ core dumps when setting data directory on specific context and it is not set on global context #1574

Comments

snowman2 commented Aug 31, 2019

snowman2 commented Aug 31, 2019 • edited Loading

snowman2 commented Aug 31, 2019

snowman2 commented Aug 31, 2019

rouault commented Sep 4, 2019 • edited Loading

kbevers commented Sep 4, 2019

snowman2 commented Sep 4, 2019 • edited Loading

Step 1:

Step 2:

rouault commented Sep 4, 2019

snowman2 commented Sep 4, 2019

rouault commented Sep 4, 2019

snowman2 commented Sep 5, 2019

rouault commented Sep 5, 2019

snowman2 commented Sep 5, 2019

snowman2 commented Sep 5, 2019

rouault commented Sep 5, 2019

snowman2 commented Sep 5, 2019

snowman2 commented Sep 5, 2019

snowman2 commented Sep 5, 2019

rouault commented Sep 5, 2019 • edited Loading

snowman2 commented Aug 31, 2019 •

edited

Loading

rouault commented Sep 4, 2019 •

edited

Loading

snowman2 commented Sep 4, 2019 •

edited

Loading

rouault commented Sep 5, 2019 •

edited

Loading