Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Segmentation fault when importing kaldi_pybind #3836

Closed
mohan-li opened this issue Jan 10, 2020 · 5 comments
Closed

Segmentation fault when importing kaldi_pybind #3836

mohan-li opened this issue Jan 10, 2020 · 5 comments
Labels

Comments

@mohan-li
Copy link

Hello, I have just compiled kaldi with brand pybind11 by
configure —shared —mathlib=ATLAS —cudatk-dir=somewhere/cud’s-10.0,
with the gcc version as 5.3.1, and I installed pybind11 for python by pip.
Compiling the regular kaldi src seemed to be normal, but when I tried to compile pybind package, the make did succeed, as all .o files were generated normally, however when the makefile tested on kaldi_pybind in python3 there was a segmentation fault error.
I was wondering if this is because of the mathlib version or something else?
Please help, thanks a lot!

@mohan-li mohan-li added the bug label Jan 10, 2020
@danpovey
Copy link
Contributor

danpovey commented Jan 10, 2020 via email

@mohan-li
Copy link
Author

please try to get a stack trace somehow. gdb or valgrind or pdb3 might help.

On Fri, Jan 10, 2020, 9:46 PM mohan-li @.***> wrote: Hello, I have just compiled kaldi with brand pybind11 by configure —shared —mathlib=ATLAS —cudatk-dir=somewhere/cud’s-10.0, with the gcc version as 5.3.1, and I installed pybind11 for python by pip. Compiling the regular kaldi src seemed to be normal, but when I tried to compile pybind package, the make did succeed, as all .o files were generated normally, however when the makefile tested on kaldi_pybind in python3 there was a segmentation fault error. I was wondering if this is because of the mathlib version or something else? Please help, thanks a lot! — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#3836?email_source=notifications&email_token=AAZFLO4LTBPUCXGDB5GZKPDQ5B33PA5CNFSM4KFIAIW2YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4IFLFL3Q>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAZFLO4Z3E3PNVA5AISIRT3Q5B33PANCNFSM4KFIAIWQ .

Thanks for your reply, and I tried to use gdb to debug python -c 'import kaldi_pybind', the backtrace command gave as:

#0 std::string::_Rep::_M_is_leaked (this=this@entry=0x956a5a94ffe9)
at /home/conda/feedstock_root/build_artifacts/ctng-compilers_1578380353001/work/.build/x86_64-conda_cos6-linux-gnu/build/build-cc-gcc-final/x86_64-conda_cos6-linux-gnu/libstdc++-v3/include/bits/basic_string.h:3228
#1 0x00007fffe91e11b4 in std::string::_Rep::_M_grab (this=0x956a5a94ffe9, __alloc1=..., __alloc2=...)
at /home/conda/feedstock_root/build_artifacts/ctng-compilers_1578380353001/work/.build/x86_64-conda_cos6-linux-gnu/build/build-cc-gcc-final/x86_64-conda_cos6-linux-gnu/libstdc++-v3/include/bits/basic_string.h:3284
#2 0x00007fffe91e11f7 in std::basic_string<char, std::char_traits, std::allocator >::basic_string (this=0x7fffffffb9c0, __str=...)
at /home/conda/feedstock_root/build_artifacts/ctng-compilers_1578380353001/work/.build/x86_64-conda_cos6-linux-gnu/build/build-cc-gcc-final/x86_64-conda_cos6-linux-gnu/libstdc++-v3/include/ext/new_allocator.h:83
#3 0x00007fffeaf96c9f in fst::FstRegisterer<fst::ConstFst<fst::ArcTpl<fst::LogWeightTpl >, unsigned int> >::FstRegisterer() ()
from /rmt/asr2/catalin/asr/kaldi/20190521-cuda9.0-v100/kaldi-master/tools/openfst-1.6.7/lib/libfst.so.10
#4 0x00007fffeaf947e6 in __static_initialization_and_destruction_0 ()
from /rmt/asr2/catalin/asr/kaldi/20190521-cuda9.0-v100/kaldi-master/tools/openfst-1.6.7/lib/libfst.so.10
#5 0x00007fffeaf948a3 in _GLOBAL__sub_I_fst_types.cc ()
from /rmt/asr2/catalin/asr/kaldi/20190521-cuda9.0-v100/kaldi-master/tools/openfst-1.6.7/lib/libfst.so.10
#6 0x00007ffff7dea973 in _dl_init_internal () from /lib64/ld-linux-x86-64.so.2
#7 0x00007ffff7def54e in dl_open_worker () from /lib64/ld-linux-x86-64.so.2
#8 0x00007ffff7dea784 in _dl_catch_error () from /lib64/ld-linux-x86-64.so.2
#9 0x00007ffff7deeb3b in _dl_open () from /lib64/ld-linux-x86-64.so.2
#10 0x00007ffff75edeeb in dlopen_doit () from /lib64/libdl.so.2
#11 0x00007ffff7dea784 in _dl_catch_error () from /lib64/ld-linux-x86-64.so.2
#12 0x00007ffff75ee4ed in _dlerror_run () from /lib64/libdl.so.2
#13 0x00007ffff75edf81 in dlopen@@GLIBC_2.2.5 () from /lib64/libdl.so.2
#14 0x0000555555749d5a in _PyImport_FindSharedFuncptr ()
at /home/conda/feedstock_root/build_artifacts/python_1573054930886/work/Python/dynload_shlib.c:95
#15 0x000055555576e080 in _PyImport_LoadDynamicModuleWithSpec ()
at /home/conda/feedstock_root/build_artifacts/python_1573054930886/work/Python/importdl.c:129
#16 0x000055555576e2c5 in _imp_create_dynamic_impl.isra.12 (file=0x0, spec=0x7ffff7e8ca58)
at /home/conda/feedstock_root/build_artifacts/python_1573054930886/work/Python/import.c:1994
#17 _imp_create_dynamic () at /home/conda/feedstock_root/build_artifacts/python_1573054930886/work/Python/clinic/import.c.h:289
#18 0x000055555566ab91 in PyCFunction_Call () at /home/conda/feedstock_root/build_artifacts/python_1573054930886/work/Objects/methodobject.c:114
#19 0x000055555571f83a in do_call_core (kwdict=0x7ffff7f13708, callargs=0x7ffff7ea3358, func=0x7ffff7f8dee8)
at /home/conda/feedstock_root/build_artifacts/python_1573054930886/work/Python/ceval.c:5102
#20 _PyEval_EvalFrameDefault () at /home/conda/feedstock_root/build_artifacts/python_1573054930886/work/Python/ceval.c:3404
#21 0x00005555556c4cf6 in _PyEval_EvalCodeWithName () at /home/conda/feedstock_root/build_artifacts/python_1573054930886/work/Python/ceval.c:4166
#22 0x00005555556c5c91 in fast_function () at /home/conda/feedstock_root/build_artifacts/python_1573054930886/work/Python/ceval.c:4978
---Type to continue, or q to quit---
#23 0x00005555556f5635 in call_function () at /home/conda/feedstock_root/build_artifacts/python_1573054930886/work/Python/ceval.c:4858
#24 0x000055555571a7aa in _PyEval_EvalFrameDefault () at /home/conda/feedstock_root/build_artifacts/python_1573054930886/work/Python/ceval.c:3335
#25 0x00005555556c5a5b in _PyFunction_FastCall (globals=, nargs=2, args=, co=)
at /home/conda/feedstock_root/build_artifacts/python_1573054930886/work/Python/ceval.c:4919
#26 fast_function () at /home/conda/feedstock_root/build_artifacts/python_1573054930886/work/Python/ceval.c:4954
#27 0x00005555556f5635 in call_function () at /home/conda/feedstock_root/build_artifacts/python_1573054930886/work/Python/ceval.c:4858
#28 0x000055555571a7aa in _PyEval_EvalFrameDefault () at /home/conda/feedstock_root/build_artifacts/python_1573054930886/work/Python/ceval.c:3335
#29 0x00005555556c5a5b in _PyFunction_FastCall (globals=, nargs=1, args=, co=)
at /home/conda/feedstock_root/build_artifacts/python_1573054930886/work/Python/ceval.c:4919
#30 fast_function () at /home/conda/feedstock_root/build_artifacts/python_1573054930886/work/Python/ceval.c:4954
#31 0x00005555556f5635 in call_function () at /home/conda/feedstock_root/build_artifacts/python_1573054930886/work/Python/ceval.c:4858
#32 0x000055555571a7aa in _PyEval_EvalFrameDefault () at /home/conda/feedstock_root/build_artifacts/python_1573054930886/work/Python/ceval.c:3335
#33 0x00005555556c5a5b in _PyFunction_FastCall (globals=, nargs=1, args=, co=)
at /home/conda/feedstock_root/build_artifacts/python_1573054930886/work/Python/ceval.c:4919
#34 fast_function () at /home/conda/feedstock_root/build_artifacts/python_1573054930886/work/Python/ceval.c:4954
#35 0x00005555556f5635 in call_function () at /home/conda/feedstock_root/build_artifacts/python_1573054930886/work/Python/ceval.c:4858
#36 0x000055555571a7aa in _PyEval_EvalFrameDefault () at /home/conda/feedstock_root/build_artifacts/python_1573054930886/work/Python/ceval.c:3335
#37 0x00005555556c5a5b in _PyFunction_FastCall (globals=, nargs=2, args=, co=)
at /home/conda/feedstock_root/build_artifacts/python_1573054930886/work/Python/ceval.c:4919
#38 fast_function () at /home/conda/feedstock_root/build_artifacts/python_1573054930886/work/Python/ceval.c:4954
#39 0x00005555556f5635 in call_function () at /home/conda/feedstock_root/build_artifacts/python_1573054930886/work/Python/ceval.c:4858
#40 0x000055555571a7aa in _PyEval_EvalFrameDefault () at /home/conda/feedstock_root/build_artifacts/python_1573054930886/work/Python/ceval.c:3335
#41 0x00005555556c60cb in _PyFunction_FastCall (globals=, nargs=2, args=, co=)
at /home/conda/feedstock_root/build_artifacts/python_1573054930886/work/Python/ceval.c:4919
#42 _PyFunction_FastCallDict () at /home/conda/feedstock_root/build_artifacts/python_1573054930886/work/Python/ceval.c:5021
#43 0x0000555555667f0f in _PyObject_FastCallDict ()
at /home/conda/feedstock_root/build_artifacts/python_1573054930886/work/Objects/abstract.c:2310
#44 0x0000555555691be0 in _PyObject_CallMethodIdObjArgs ()
at /home/conda/feedstock_root/build_artifacts/python_1573054930886/work/Objects/abstract.c:2796
#45 0x000055555565e4b0 in PyImport_ImportModuleLevelObject ()
at /home/conda/feedstock_root/build_artifacts/python_1573054930886/work/Python/import.c:1578
#46 0x000055555571d076 in import_name (level=0x555555894980 <small_ints+160>, fromlist=0x55555584da90 <_Py_NoneStruct>, name=0x7ffff7e913b0,
f=0x5555559020f8) at /home/conda/feedstock_root/build_artifacts/python_1573054930886/work/Python/ceval.c:5231
#47 _PyEval_EvalFrameDefault () at /home/conda/feedstock_root/build_artifacts/python_1573054930886/work/Python/ceval.c:2899
#48 0x00005555556ca069 in _PyEval_EvalCodeWithName (qualname=0x0, name=, closure=0x0, kwdefs=0x0, defcount=0, defs=0x0, kwstep=2,
kwcount=, kwargs=0x0, kwnames=0x0, argcount=0, args=0x0, locals=0x7ffff7f51288, globals=0x7ffff7f51288, _co=0x7ffff7f0ec00)
---Type to continue, or q to quit---
at /home/conda/feedstock_root/build_artifacts/python_1573054930886/work/Python/ceval.c:4166
#49 PyEval_EvalCodeEx () at /home/conda/feedstock_root/build_artifacts/python_1573054930886/work/Python/ceval.c:4187
#50 0x00005555556cadfc in PyEval_EvalCode (co=co@entry=0x7ffff7f0ec00, globals=globals@entry=0x7ffff7f51288, locals=locals@entry=0x7ffff7f51288)
at /home/conda/feedstock_root/build_artifacts/python_1573054930886/work/Python/ceval.c:731
#51 0x000055555576d154 in run_mod () at /home/conda/feedstock_root/build_artifacts/python_1573054930886/work/Python/pythonrun.c:1025
#52 0x000055555576d551 in PyRun_FileExFlags () at /home/conda/feedstock_root/build_artifacts/python_1573054930886/work/Python/pythonrun.c:978
#53 0x000055555576d753 in PyRun_SimpleFileExFlags ()
at /home/conda/feedstock_root/build_artifacts/python_1573054930886/work/Python/pythonrun.c:419
#54 0x000055555576d85d in PyRun_AnyFileExFlags () at /home/conda/feedstock_root/build_artifacts/python_1573054930886/work/Python/pythonrun.c:81
#55 0x00005555557712b7 in run_file (p_cf=0x7fffffffd3fc, filename=0x5555558ab2f0 L"./test.py", fp=0x5555558e4ce0)
at /home/conda/feedstock_root/build_artifacts/python_1573054930886/work/Modules/main.c:340
#56 Py_Main () at /home/conda/feedstock_root/build_artifacts/python_1573054930886/work/Modules/main.c:810
#57 0x0000555555638cde in main () at /home/conda/feedstock_root/build_artifacts/python_1573054930886/work/Programs/python.c:69
#58 0x00007ffff7813505 in __libc_start_main () from /lib64/libc.so.6
#59 0x0000555555721242 in _start () at ../sysdeps/x86_64/elf/start.S:103

I suspect the error arised form the conda environment as the path /home/conda/feedstock_root doesn't exist at all...

@csukuangfj
Copy link
Contributor

It seems that the error is from OpenFST

#3 0x00007fffeaf96c9f in fst::FstRegisterer<fst::ConstFst<fst::ArcTpl<fst::LogWeightTpl >, unsigned int> >::FstRegisterer() ()
from /rmt/asr2/catalin/asr/kaldi/20190521-cuda9.0-v100/kaldi-master/tools/openfst-1.6.7/lib/libfst.so.10

Maybe you can exclude OpenFST from kaldi pybind and recompile it again to see if import kaldi_pybind will succeed.


/home/conda/feedstock_root

is the source path of the people who compiled Python and it is embedded into the binary.
It is not the reason for your segfault.

@mohan-li
Copy link
Author

It seems that the error is from OpenFST

#3 0x00007fffeaf96c9f in fst::FstRegisterer<fst::ConstFst<fst::ArcTpl<fst::LogWeightTpl >, unsigned int> >::FstRegisterer() ()
from /rmt/asr2/catalin/asr/kaldi/20190521-cuda9.0-v100/kaldi-master/tools/openfst-1.6.7/lib/libfst.so.10

Maybe you can exclude OpenFST from kaldi pybind and recompile it again to see if import kaldi_pybind will succeed.

/home/conda/feedstock_root

is the source path of the people who compiled Python and it is embedded into the binary.
It is not the reason for your segfault.

Yes, I have recompliled the tools in kaldi including openfst, and it worked. Possibly it would be better for the developers to clarify the required gcc version for this branch. Presumably gcc has to be newer than 4.9, as I tried that 4.8.5 didn’t work but 5.3.1 did. Thanks a lot!

@danpovey
Copy link
Contributor

danpovey commented Jan 11, 2020 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

3 participants