Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Crash in IvectorExtractorInfo::Init() #4791

Open
Veango opened this issue Sep 13, 2022 · 22 comments
Open

Crash in IvectorExtractorInfo::Init() #4791

Veango opened this issue Sep 13, 2022 · 22 comments
Labels
stale Stale bot on the loose waiting-for-feedback Reporter's feedback has been requested

Comments

@Veango
Copy link

Veango commented Sep 13, 2022 •
edited by kkm000
Loading

when I init the Kaldi ,

        OnlineIvectorExtractionConfig ivector_extraction_opts;
        ivector_extraction_opts.splice_config_rxfilename = model_path_str_ + "/ivector/splice.conf";
        ivector_extraction_opts.cmvn_config_rxfilename = model_path_str_ + "/ivector/online_cmvn.conf";
        ivector_extraction_opts.lda_mat_rxfilename = model_path_str_ + "/ivector/final.mat";
        ivector_extraction_opts.global_cmvn_stats_rxfilename = model_path_str_ + "/ivector/global_cmvn.stats";
        ivector_extraction_opts.diag_ubm_rxfilename = model_path_str_ + "/ivector/final.dubm";
        ivector_extraction_opts.ivector_extractor_rxfilename = model_path_str_ + "/ivector/final.ie";
        ivector_extraction_opts.max_count = 100;
        feature_info_.use_ivectors = true;
        feature_info_.ivector_extractor_info.Init(ivector_extraction_opts);

it will crash like this:
1
2

please tell me how can I fix it
@Veango Veango added the bug label Sep 13, 2022
@Veango Veango changed the title have a crash in matrix have a crash when I call init Sep 13, 2022
@kkm000
Copy link
Contributor

kkm000 commented Sep 14, 2022

Could be a mixup of files, but it better should not just crash. Cannot decode as reported, though.

Could you post the backtrace from a debug build? Also,

  1. Pictures are much less useful that the output of the bt command. If you look carefully, backtrace lines are all cut short.
  2. What was the signal name or number that was caught? Any error messages from the BLAS library on stdout? MKL often prints one before crashing.
  3. What BLAS library do you compile against (MKL, OpenBLAS...)?

This info will be helpful.

@kkm000 kkm000 changed the title have a crash when I call init Crash in IvectorExtractorInfo::Init() Sep 14, 2022
@kkm000 kkm000 added the waiting-for-feedback Reporter's feedback has been requested label Sep 14, 2022
@jtrmal
Copy link
Contributor

jtrmal commented Sep 14, 2022 via email

@Veango
Copy link
Author

Veango commented Sep 14, 2022 •
edited
Loading

@kkm000
1
2

I used openblas，the code runs in iOS ，is Kaldi not support iOS?

@Veango
Copy link
Author

Veango commented Sep 14, 2022 •
edited
Loading

another info is when I call make -C matrix test, it got some error like:
numpy-array-test.cc:82:15: error: 'system' is unavailable: not available on iOS
rc = std::system("python3 -c "import numpy; "
...

@kkm000
Copy link
Contributor

kkm000 commented Sep 14, 2022 •
edited
Loading

'system' is unavailable: not available on iOS

Are you in fact building the whole of Kaldi on iOS, or is it a nonsensical message? You are likely much better off cross-compiling.

What is your platform? MacOS has its own linalg framework, I forgot its name (@jtrmal help me please, "accel" or something?), use it instead. It's configurable. Especially with their latest accelerator in M1/M2, it must use the accelerator. I've seen iOS used only on devices no beefier than the iPad. Does it have an accelerator for linear algebra? Does OpenBLAS support it? In any case, you will be much better off fully utilizing hardware stuff which is usually a-plenty on device SoCs, which speeds up a limited set of operation manifold, frees up CPU and reduces power consumption.

[does] Kaldi not support iOS?

A gray area. There are too many things that may be called iOS, and too many hardware variants it runs on. Since there's no one around with iOS experience, rather no, not in the full sense of "support." Do use the kaldi-help list, search its archives, post a question if nothing comes up. maybe you'll find a kindred soul.

For the crash, let's start off by removing threading from the picture: set threading to 1 globally, kaldi::g_num_threads = 1; before using any Kaldi functions. i-vector extractor initialization is multithreaded by default, and you see it in the stack.

kaldi/src/util/kaldi-thread.h

Lines 58 to 65 in 727e454

namespace kaldi {
extern int32 g_num_threads; // Maximum number of threads (for programs that
// use threads, which is not many of them, e.g. the SGMM update program does.
// This is 8 by default. You can change this on the command line, where
// used, with --num-threads. Programs that think they will use threads
// should register it with their ParseOptions, as something like:
// po.Register("num-threads", &g_num_threads, "Number of threads to use.");

The default of 8 is (statically, at compile time) set in util/kaldi-thread.cc:25 and then used to initialize sequencer options a few times through src/ivector/ivector-extractor.cc.

If this has no effect, running the same init code on an x64 Linux or Windows, to make sure we don't have a case of broken or mismatched extractor files. There are a few.

@Veango
Copy link
Author

Veango commented Sep 15, 2022

Thank you for your reply,I cross-compiled openblas clapack openfst and kaldi，i ever used the same code in android ， it hasn't any problem. But i used it in iPhone platform, it crash,so i don't know witch step i was wrong

@kkm000
Copy link
Contributor

kkm000 commented Sep 15, 2022

Looks like an OpenBLAS issue. As I mentioned, the Apple's own solution, Accelerate, includes BLAS routines. And its docs mention that it's available in iOS. By all means, use it instead of OpenBLAS. It is supposed to use all hardware acceleration there is on the platform, and in the optimal way. For a device, it is especially important, as it not only does it faster but saves battery, too (assuming there is hardware available; I know little about iPhones and friends, but mid-line to high-end modern devices just have to have it).

Try to figure out why it is not selected by configure automatically. Maybe we just don't handle iOS cross correctly?

kaldi/src/configure

Lines 1029 to 1080 in 727e454

elif [ "`uname`" == "Darwin" ]; then
# Check for Darwin first, because we later call uname -o (for Cygwin)
# which crashes on Darwin.
echo "On Darwin: Checking for Accelerate framework ..."
if [ ! -e /System/Library/Frameworks/Accelerate.framework ]; then
failure "Need the Accelerate framework to compile on Darwin."
fi
OSX_VER=$(sw_vers | grep ProductVersion | awk '{print $2}' | awk '{split($0,a,"."); print a[1] "." a[2]; }')
OSX_VER_NUM=$(echo $OSX_VER | sed 's/\./ /g' | xargs printf "%d%02d")
echo "Configuring for OS X version $OSX_VER ..."
if [ $OSX_VER_NUM -ge 1005 ]; then
if [ "$MATHLIB" == "CLAPACK" ]; then
if [ -z "$CLAPACKROOT" ]; then
failure "Must specify the location of CLAPACK with --clapack-root option (and it must exist)"
fi
if [ ! -f ../tools/CLAPACK/clapack.h ]; then
failure "could not find file ../tools/CLAPACK/clapack.h"
fi
if [ ! -d "$CLAPACKROOT" ]; then
failure "The directory $CLAPACKROOT does not exist"
fi
# Also check for cblas.h and f2c.h
echo "Using CLAPACK libs from $CLAPACKROOT as the linear algebra library."
echo "CLAPACKROOT = $CLAPACKROOT" >> kaldi.mk
if [ ! -f makefiles/darwin_clapack.mk ]; then
failure "makefiles/darwin_clapack.mk not found."
fi
cat makefiles/darwin_clapack.mk >> kaldi.mk
echo "Warning (CLAPACK): this part of the configure process is not properly tested and may not work."
echo "Successfully configured for Darwin with CLAPACK libs from $CLAPACKROOT"
else
cat makefiles/darwin.mk >> kaldi.mk
fi
else
failure "Mac OS X version '$OSX_VER' is not supported."
fi
if [ $OSX_VER_NUM == 1011 ]; then
echo "**BAD WARNING**: You are using OS X El Capitan. Some versions of this OS"
echo "**BAD WARNING**: have a bug in the BLAS implementation that affects Kaldi."
echo "**BAD WARNING**: After compiling, cd to matrix/ and type 'make test'. The"
echo "**BAD WARNING**: test will fail if the problem exists in your version. "
echo "**BAD WARNING**: Eventually this issue will be fixed by system updates from"
echo "**BAD WARNING**: Apple. Unexplained crashes with reports of NaNs will"
echo "**BAD WARNING**: be caused by this bug, but some recipes will (sometimes) work."
sleep 1; echo -n .; sleep 1; echo -n .; sleep 1; echo .
fi
echo "Successfully configured for Darwin with Accelerate framework."
$use_cuda && configure_cuda
elif [ "`uname -o`" == "Cygwin" ]; then

@kkm000
Copy link
Contributor

kkm000 commented Sep 15, 2022 •
edited
Loading

If you can find a simple way to fix configure so that it picks up the correct library, we'd certainly welcome a patch. But if you find that you need to bend all the way backwards to make configure work for you, I want to share how I productize Kaldi. For training and stuff on Linux, I of course use the standard workflow, configure + make. But for compiling the product, I select necessary files from Kaldi into a library (usually static, it allows for advanced optimizations such as PGO for the final build, but I don't know what's the best for your case), and simply build it. You need to define one preprocessor variable (otherwise defined in a generated base/version.h file), namely KALDI_VERSION (to any value, it's used only in diagnostic messages) in the command line to compiler. All compile flags and dependency libraries you can figure out from your platform SDK. You can also catch Kaldi output using the hook routine in base/kaldi-error.h, so that they are logged your way instead of being output to stderr.

@kkm000
Copy link
Contributor

kkm000 commented Sep 15, 2022

OpenFST also has a logging hook in log.h; that's separate from Kaldi, you need to call it too if you want to hook up their logging.

@Veango
Copy link
Author

Veango commented Sep 15, 2022

You mean i shouldn't use openblas and clapack,just only compile openfst and kaldi,don't set the MATHLIB，and it will use the system's accelerate if it's no problem?

@kkm000
Copy link
Contributor

kkm000 commented Sep 15, 2022

No, this is not exactly what I said.

i shouldn't use openblas and clapack

Correct, use iOS native Accelerator framework.

don't set the MATHLIB, and it will use the system's accelerate

During configure? I don't know, possibly. Try it. It may or may not work. If it doesn't. try to figure out and easy fix to configure, it's hand-coded, not generated. If it happens too tricky to get right, you may do it differently. You also may do it this way to get a greater control over build options, namely:

just only compile openfst and kaldi

As an alternative to configure, you may compile the runtime-only files your own way. I do it both to OpenFST and Kaldi, normally. This is what I do for production builds. You'll have to write whatever build files your build system/IDE requires. But it is the most flexible option. Mind the KALDI_VERSION. I can't recall any other tripping points.

Optionally, regardless of the way you build, hook logging (see kaldi-error.h). You'll probably need it for diagnostics at some point. This applies to all platforms, not only iOS. Kaldi also has SetVerbosityLevel(), beyond INFO, WARNING and ERROR, from 0 to 5, 5 is insane amount of logging. When you don't do verbose logs, set to 0 (default), it's expensive to generate, so don't simply generate and discard it. INFO and above are always logged.

@Veango
Copy link
Author

Veango commented Sep 15, 2022

Thanks a lot,i will try it,if i find a simple way to fix it,i will report it here

@kkm000
Copy link
Contributor

kkm000 commented Sep 15, 2022

Thanks, we really have no idea if we configure a cross-compile for iOS correctly (or at all)! All scripting for it, if exists at all, was contributed outside of the core team. I'll leave this issue open for a while.

@kkm000 kkm000 removed the bug label Sep 15, 2022
@jtrmal
Copy link
Contributor

jtrmal commented Sep 26, 2022 via email

@jtrmal
Copy link
Contributor

jtrmal commented Sep 26, 2022 via email

@Veango
Copy link
Author

Veango commented Oct 1, 2022

IOS has not been tried. Just MacOS... that might be the confusion. Also, it might make sense that for Ios (for whatever security reason), system() doesn't work y.
…
On Wed, Sep 14, 2022 at 10:58 PM kkm000 @.> wrote: Thanks, we really have no idea if we configure a cross-compile for iOS correctly (or at all)! All scripting for it, if exists at all, was contributed outside of the core team. I'll leave this issue open for a while. — Reply to this email directly, view it on GitHub <#4791 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACUKYX3TIECARZZ3T2KB2JTV6KGEZANCNFSM6AAAAAAQLRPLDY . You are receiving this because you were mentioned.Message ID: @.>

Yes, ios didn't support System () method any more,so kaldi can't use in ios is that reason?

@jtrmal
Copy link
Contributor

jtrmal commented Oct 3, 2022 via email

@Veango
Copy link
Author

Veango commented Oct 4, 2022

kaldi itself should be fine. It's just the setup/compile pipeline and some scripts will/would have to be altered. y.
…
On Sat, Oct 1, 2022 at 5:58 AM Veango @.> wrote: IOS has not been tried. Just MacOS... that might be the confusion. Also, it might make sense that for Ios (for whatever security reason), system() doesn't work y. … <#m_-4983130712797081710_> On Wed, Sep 14, 2022 at 10:58 PM kkm000 @.> wrote: Thanks, we really have no idea if we configure a cross-compile for iOS correctly (or at all)! All scripting for it, if exists at all, was contributed outside of the core team. I'll leave this issue open for a while. — Reply to this email directly, view it on GitHub <#4791 (comment) <#4791 (comment)>>, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACUKYX3TIECARZZ3T2KB2JTV6KGEZANCNFSM6AAAAAAQLRPLDY https://github.com/notifications/unsubscribe-auth/ACUKYX3TIECARZZ3T2KB2JTV6KGEZANCNFSM6AAAAAAQLRPLDY . You are receiving this because you were mentioned.Message ID: @.> Yes, ios didn't support System () method any more,so kaldi can't use in ios is that reason? — Reply to this email directly, view it on GitHub <#4791 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACUKYXZQNMROXF2ZDEFTRMLWBADMHANCNFSM6AAAAAAQLRPLDY . You are receiving this because you were mentioned.Message ID: @.**>

But as i know,clapack used System () method, may be i should rewrite it

@jtrmal
Copy link
Contributor

jtrmal commented Oct 4, 2022 via email

@Veango
Copy link
Author

Veango commented Oct 7, 2022

Hi, clapack shouldn't need it, unless it's some special logging function or something like that. Or it might be called as a part of build process. I.e. should be possible to replace it. I cannot help, sorry. Not enough experience. y.
…
On Tue, Oct 4, 2022 at 6:27 AM Veango @.> wrote: kaldi itself should be fine. It's just the setup/compile pipeline and some scripts will/would have to be altered. y. … <#m_-7141260418549791347_> On Sat, Oct 1, 2022 at 5:58 AM Veango @.> wrote: IOS has not been tried. Just MacOS... that might be the confusion. Also, it might make sense that for Ios (for whatever security reason), system() doesn't work y. … <#m_-4983130712797081710_> On Wed, Sep 14, 2022 at 10:58 PM kkm000 @.> wrote: Thanks, we really have no idea if we configure a cross-compile for iOS correctly (or at all)! All scripting for it, if exists at all, was contributed outside of the core team. I'll leave this issue open for a while. — Reply to this email directly, view it on GitHub <#4791 <#4791> (comment) <#4791 (comment) <#4791 (comment)>>>, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACUKYX3TIECARZZ3T2KB2JTV6KGEZANCNFSM6AAAAAAQLRPLDY https://github.com/notifications/unsubscribe-auth/ACUKYX3TIECARZZ3T2KB2JTV6KGEZANCNFSM6AAAAAAQLRPLDY https://github.com/notifications/unsubscribe-auth/ACUKYX3TIECARZZ3T2KB2JTV6KGEZANCNFSM6AAAAAAQLRPLDY https://github.com/notifications/unsubscribe-auth/ACUKYX3TIECARZZ3T2KB2JTV6KGEZANCNFSM6AAAAAAQLRPLDY . You are receiving this because you were mentioned.Message ID: @.> Yes, ios didn't support System () method any more,so kaldi can't use in ios is that reason? — Reply to this email directly, view it on GitHub <#4791 (comment) <#4791 (comment)>>, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACUKYXZQNMROXF2ZDEFTRMLWBADMHANCNFSM6AAAAAAQLRPLDY https://github.com/notifications/unsubscribe-auth/ACUKYXZQNMROXF2ZDEFTRMLWBADMHANCNFSM6AAAAAAQLRPLDY . You are receiving this because you were mentioned.Message ID: @.> But as i know,clapack used System () method, may be i should rewrite it — Reply to this email directly, view it on GitHub <#4791 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACUKYXZTV5W5KJEYINORXV3WBQBA7ANCNFSM6AAAAAAQLRPLDY . You are receiving this because you were mentioned.Message ID: @.**>

https://github.com/alphacep/clapack/tree/master/F2CLIBS/libf2c)/system_.c this file used System () method，i can't cross-compile with ios sdk

@jtrmal
Copy link
Contributor

jtrmal commented Oct 7, 2022 via email

@stale
Copy link

stale bot commented Dec 16, 2022

This issue has been automatically marked as stale by a bot solely because it has not had recent activity. Please add any comment (simply 'ping' is enough) to prevent the issue from being closed for 60 more days if you believe it should be kept open.

@stale stale bot added the stale Stale bot on the loose label Dec 16, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
stale Stale bot on the loose waiting-for-feedback Reporter's feedback has been requested
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@kkm000 @jtrmal @Veango