Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

How to compile a version static library(.a) of online2-tcp-nnet3-decode-faster? #3589

Closed
ppamorim opened this issue Sep 14, 2019 · 17 comments
Closed

How to compile a version static library(.a) of online2-tcp-nnet3-decode-faster? #3589

ppamorim opened this issue Sep 14, 2019 · 17 comments
Labels
discussion

Comments

@ppamorim
Copy link

ppamorim commented Sep 14, 2019 •
edited
Loading

Hi,

Currently I am trying to compile online2-tcp-nnet3-decode-faster.cc and create an static library. I tried following the command ./configure --static inside src folder but it didn't create any static library. I tried to do this by myself based on this script below that I got from the makefile:

(this code below build a executable)

g++   -g   online2-tcp-nnet3-decode-faster.o ../online2/kaldi-online2.a ../ivector/kaldi-ivector.a ../nnet3/kaldi-nnet3.a ../chain/kaldi-chain.a ../nnet2/kaldi-nnet2.a ../cudamatrix/kaldi-cudamatrix.a ../decoder/kaldi-decoder.a ../lat/kaldi-lat.a ../fstext/kaldi-fstext.a ../hmm/kaldi-hmm.a ../feat/kaldi-feat.a ../transform/kaldi-transform.a ../gmm/kaldi-gmm.a ../tree/kaldi-tree.a ../util/kaldi-util.a ../matrix/kaldi-matrix.a ../base/kaldi-base.a ../kaldi/tools/openfst-1.6.7/lib/libfst.a -framework Accelerate -lm -lpthread -ldl  -o online2-tcp-nnet3-decode-faster

I got all the static libraries and tried to create a new big object file, it wasn't sucessful and very boring to do.

What you guys recommend? I need something that let me only import a header and link the library easily.

Thank you.
@danpovey
Copy link
Contributor

danpovey commented Sep 15, 2019 via email

@ppamorim
Copy link
Author

@danpovey Thank you. I will try to follow the path that you guys used to create the kaldi-online2.a archive.

@ppamorim
Copy link
Author

@danpovey Do you know where I can find any example of how to link kaldi-online2.a with another C++ source code? I am trying here but I getting Undefined symbols for architecture x86_64. Thank you anyway.

@danpovey
Copy link
Contributor

danpovey commented Sep 16, 2019 via email

@nshmyrev
Copy link
Contributor

@ppamorim something like https://github.com/alphacep/kaldi-websocket-python/blob/master/Makefile should be able to help you.

@ppamorim
Copy link
Author

ppamorim commented Sep 17, 2019 •
edited
Loading

@danpovey @nshmyrev I managed to compile it and list it against all kaldi static libraries sucessfully, thank you for this.
Now I am getting random crashes of Incorrect checksum for freed object and SIGSEGV when I call a new stack instance of DecodableNnetSimpleLoopedInfo, sometimes the script works but most of the time it doesn't and crash.
But this only happens when I playing with the static library added to an external project, with the same script being called from the runnable, it does work.

Basically what I did was to get that script and move all allocation to the stac, it strange to see these errors happening since the only difference is the static artifact.

@nshmyrev
Copy link
Contributor

That's probably not Kaldi issue but the issue with your code. You can close this one, share your complete code on GitHub, I will take a look.

@danpovey
Copy link
Contributor

danpovey commented Sep 17, 2019 via email

@ppamorim
Copy link
Author

ppamorim commented Sep 17, 2019 •
edited
Loading

@danpovey

What I was able to detect are these flags.

To compile the object:

g++ -std=c++11 -I.. -I/Users/user/Repository/kaldi/tools/openfst-1.6.7/include -O1 \
 -Wall -Wno-sign-compare -Wno-unused-local-typedefs -Wno-deprecated-declarations \
 -Winit-self -DKALDI_DOUBLEPRECISION=0 -DHAVE_EXECINFO_H=1 -DHAVE_CXXABI_H \
 -DHAVE_CLAPACK -msse -msse2 -pthread -g  -Wno-mismatched-tags   -c \
 -o online2-tcp-nnet3-decode-faster.o online2-tcp-nnet3-decode-faster.cc

To build the bin:

g++   -g   online2-tcp-nnet3-decode-faster.o ../online2/kaldi-online2.a \ 
../ivector/kaldi-ivector.a ../nnet3/kaldi-nnet3.a ../chain/kaldi-chain.a \ 
../nnet2/kaldi-nnet2.a ../cudamatrix/kaldi-cudamatrix.a ../decoder/kaldi-decoder.a  \
../lat/kaldi-lat.a ../fstext/kaldi-fstext.a ../hmm/kaldi-hmm.a ../feat/kaldi-feat.a \ 
../transform/kaldi-transform.a ../gmm/kaldi-gmm.a ../tree/kaldi-tree.a \ 
../util/kaldi-util.a ../matrix/kaldi-matrix.a ../base/kaldi-base.a  \ 
/Users/user/Repository/kaldi/tools/openfst-1.6.7/lib/libfst.a -framework Accelerate \
 -lm -lpthread -ldl  -o online2-tcp-nnet3-decode-faster

I am still investigating what is missing, I am using the makefile from Kaldi to compile the library too.

Edit 1:

To compile the lib, the makefile is doing:

g++ -std=c++11 -I.. -I/Users/user/Repository/kaldi/tools/openfst-1.6.7/include -O1 \
 -Wall -Wno-sign-compare -Wno-unused-local-typedefs -Wno-deprecated-declarations \
 -Winit-self -DKALDI_DOUBLEPRECISION=0 -DHAVE_EXECINFO_H=1 -DHAVE_CXXABI_H \
 -DHAVE_CLAPACK -msse -msse2 -pthread -g  -Wno-mismatched-tags   -c \
 -o online2-tcp2-nnet3-decode-faster.o online2-tcp2-nnet3-decode-faster.cc

ar -cr kaldi-tcp2.a online2-tcp2-nnet3-decode-faster.o
ranlib kaldi-tcp2.a

To compile the library that uses the static library created above, I run:

g++ -g -std=c++11 \
 main.cpp \
-L./lib/ \
-lkaldi \
-lkaldi-online2 \
-lkaldi-ivector \
-lkaldi-nnet3 \
-lkaldi-chain \
-lkaldi-nnet2 \
-lkaldi-cudamatrix \
-lkaldi-decoder \
-lkaldi-lat \
-lkaldi-fstext \
-lkaldi-hmm \
-lkaldi-feat \
-lkaldi-transform \
-lkaldi-gmm \
-lkaldi-tree \
-lkaldi-util \
-lkaldi-matrix \
-lkaldi-base \
-lfst \
-framework Accelerate \
-lm -lpthread -ldl \
-DKALDI_DOUBLEPRECISION=0 -DHAVE_EXECINFO_H=1 -DHAVE_CXXABI_H -DHAVE_CLAPACK -msse -msse2 -pthread \
-o build/run

@ppamorim
Copy link
Author

ppamorim commented Sep 18, 2019 •
edited
Loading

The script starts crashing here:

header:

class Foo {
  private:
    //...
    TransitionModel* trans_model_;
    nnet3::AmNnetSimple* am_nnet_;
    nnet3::DecodableNnetSimpleLoopedInfo* decodable_info_;
    //...
};

cc:

this->trans_model_ = new TransitionModel();
this->am_nnet_ = new nnet3::AmNnetSimple();
{
    bool binary;
    Input ki(nnet3_rxfilename, &binary);
    trans_model_->Read(ki.Stream(), binary);
    am_nnet_->Read(ki.Stream(), binary);
    SetBatchnormTestMode(true, &(am_nnet_->GetNnet()));
    SetDropoutTestMode(true, &(am_nnet_->GetNnet()));
    nnet3::CollapseModel(nnet3::CollapseModelConfig(), &(am_nnet_->GetNnet()));
}

// this object contains precomputed stuff that is used by all decodable
// objects.  It takes a pointer to am_nnet because if it has iVectors it has
// to modify the nnet to accept iVectors at intervals.
this->decodable_info_ = new nnet3::DecodableNnetSimpleLoopedInfo(
    *decodable_opts_,
    am_nnet_); //Crash here, sometimes it works...

Full stacktrace:

Click to expand 
(lldb) bt
* thread #1, queue = 'com.apple.main-thread', stop reason = signal SIGABRT
* frame #0: 0x00007fff6695d2c6 libsystem_kernel.dylib`__pthread_kill + 10
  frame #1: 0x00007fff66a18bf1 libsystem_pthread.dylib`pthread_kill + 284
  frame #2: 0x00007fff668c76a6 libsystem_c.dylib`abort + 127
  frame #3: 0x00007fff669d6077 libsystem_malloc.dylib`malloc_vreport + 545
  frame #4: 0x00007fff669eee0d libsystem_malloc.dylib`malloc_zone_error + 183
  frame #5: 0x00007fff669cef17 libsystem_malloc.dylib`tiny_malloc_from_free_list + 1310
  frame #6: 0x00007fff669ce43b libsystem_malloc.dylib`tiny_malloc_should_clear + 314
  frame #7: 0x00007fff669ce20f libsystem_malloc.dylib`szone_malloc_should_clear + 66
  frame #8: 0x00007fff669ccc99 libsystem_malloc.dylib`malloc_zone_malloc + 103
  frame #9: 0x00007fff669ccc15 libsystem_malloc.dylib`malloc + 24
  frame #10: 0x00007fff63aa1378 libc++abi.dylib`operator new(unsigned long) + 40
  frame #11: 0x0000000100008509 run`std::__1::__libcpp_allocate(__size=<unavailable>, __align=0) at new:252:10 [opt]
  frame #12: 0x000000010000aff6 run`std::__1::allocator<int>::allocate(this=<unavailable>, __n=<unavailable>, (null)=<unavailable>) at memory:1813:37 [opt]
  frame #13: 0x000000010000af59 run`std::__1::allocator_traits<std::__1::allocator<int> >::allocate(__a=<unavailable>, __n=<unavailable>) at memory:1546:21 [opt]
  frame #14: 0x000000010000af0c run`std::__1::__split_buffer<int, std::__1::allocator<int>&>::__split_buffer(this=0x00007ffeefbfe620, __cap=1, __start=0, __a=<unavailable>) at __split_buffer:311:29 [opt]
  frame #15: 0x000000010000ae10 run`std::__1::__split_buffer<int, std::__1::allocator<int>&>::__split_buffer(this=<unavailable>, __cap=<unavailable>, __start=<unavailable>, __a=<unavailable>) [opt] [artificial]
  frame #16: 0x000000010000acc3 run`void std::__1::vector<int, std::__1::allocator<int> >::__push_back_slow_path<int const&>(this=0x0000000102bf4350, __x=0x00007ffeefbfe7fc) at vector:1611:49 [opt]
  frame #17: 0x000000010000aa70 run`std::__1::vector<int, std::__1::allocator<int> >::push_back(this=<unavailable>, __x=<unavailable>) [opt] [artificial]
  frame #18: 0x00000001000cf6a0 run`kaldi::nnet3::ComputationGraphBuilder::AddDependencies(this=0x00007ffeefbfecb0, cindex_id=<unavailable>) at nnet-computation-graph.cc:713:36 [opt]
  frame #19: 0x00000001000ce405 run`kaldi::nnet3::ComputationGraphBuilder::BuildGraphOneIter(this=0x00007ffeefbfecb0) at nnet-computation-graph.cc:906:7 [opt]
  frame #20: 0x00000001000ce1b2 run`kaldi::nnet3::ComputationGraphBuilder::Compute(this=0x00007ffeefbfecb0, request=<unavailable>) at nnet-computation-graph.cc:479:5 [opt]
  frame #21: 0x00000001001013cb run`kaldi::nnet3::EvaluateComputationRequest(nnet=0x0000000103b000d0, request=0x00007ffeefbfee90, is_computable=0x00007ffeefbfeef8) at nnet-utils.cc:77:11 [opt]
  frame #22: 0x0000000100101ab0 run`kaldi::nnet3::ComputeSimpleNnetContextForShift(nnet=0x0000000103b000d0, input_start=<unavailable>, window_size=<unavailable>, left_context=0x0000000101f088a0, right_context=0x0000000101f09090) at nnet-utils.cc:130:3 [opt]
  frame #23: 0x00000001001016ce run`kaldi::nnet3::ComputeSimpleNnetContext(nnet=0x0000000103b000d0, left_context=0x0000000103b00140, right_context=0x0000000103b00144) at nnet-utils.cc:172:12 [opt]
  frame #24: 0x00000001000e50a2 run`kaldi::nnet3::AmNnetSimple::Read(this=0x0000000103b000d0, is=0x0000000103b00168, binary=true) at am-nnet-simple.cc:53:3 [opt]
  frame #25: 0x0000000100002d7e run`kaldi::Foo::Foo(this=0x0000000101f00300, argc=14, argv=0x00007ffeefbff530) at online2-tcp2-nnet3-decode-faster.cc:176:17 [opt]
  frame #26: 0x0000000100000b3b run`main(argc=15, argv=0x00007ffeefbff530) at main.cpp:34:39 <-- Entry point of my program
  frame #27: 0x00007fff668223d5 libdyld.dylib`start +

@danpovey
Copy link
Contributor

danpovey commented Sep 18, 2019 via email

@ppamorim
Copy link
Author

@danpovey Thank you, here is the full valgrind result: https://gist.github.com/ppamorim/86cfb75f8586e10ef6e9abfbeab07354

@ppamorim
Copy link
Author

I will share a minimum project with this error with you guys.

@danpovey
Copy link
Contributor

Sorry, I'm not going to help with this! This is a very simple bug in your code. Not a Kaldi issue. Closing the issue.

@ppamorim
Copy link
Author

@danpovey @nshmyrev Here is the minimum project needed to cause the crash: https://github.com/ppamorim/kaldi-crash

@ppamorim
Copy link
Author

ppamorim commented Sep 26, 2019 •
edited
Loading

@nshmyrev Good morning, could you please have a look in the project above? I will was not able to load Kaldi properly.

@nshmyrev
Copy link
Contributor

Your issue is that you use different headers for class PcmDecoder. On is the top folder is different from the one in the sample folder. In C++ you can not do like that, the constructor crashes.

If you want to hide implementation, you can do like this:

https://stackoverflow.com/questions/2272735/private-public-header-example

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
discussion
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@nshmyrev @danpovey @ppamorim