Java bindings for non Android

[TobiasDeBruijn]

• Have I written custom code (as opposed to running examples on an unmodified clone of the repository): No
• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Windows 10 1909
• Python version: 3.8
• GPU model and memory: RTX 2080Ti, 11GB

I've been looking at integrating DeepSpeech into a Java application I am writing, the application is not for Android. I've tried adding DeepSpeech to my build.gradle, though it is not adding it, no errors whatsoever.

Going through the build.gradle in native_client/java it appears that it uses some Android specific Gradle stuff. Could be a reason for it not working in a non-Android environment.

Is there any chance the build.gradle could be modified, or a seperate non-Android version be made, for those that don't use Android?

I'd love to do it myself, but know nothing about bindings etc.

[ftyers]

@the-nose-knows it's never to early to start. What is the "Android specific gradle stuff" that you would like it to not rely on? Perhaps start there, and feel free to join us on Matrix to discuss further.

[lissyx]

Is there any chance the build.gradle could be modified, or a seperate non-Android version be made, for those that don't use Android?

There's no non-Android Java because nobody asked or stepped to do it. If you know Java well, then you are welcome.

I'd love to do it myself, but know nothing about bindings etc.

For the binding part you can rely on the existing.

[TobiasDeBruijn]

Right okay. So I've kindof cleaned up the build.gradle files, as to not include anything Android specific. gradle wrapper now runs at least:
./native_client/java/build.gradle
./native_client/java/libdeepspeech/build.gradle

Running ./gradlew build, which I think is the correct command to use, produces the following now: https://paste.md-5.net/etigohuneg.coffeescript

I'm not at all familiar with Android flavoured Gradle, so there's a very real chance I took out the wrong bits (hence why they're commented out).

[ftyers]

The error looks like something to do with SWIG, there is documentation here: http://www.swig.org/Doc1.3/Java.html

[lissyx]

Running ./gradlew build, which I think is the correct command to use, produces the following now: https://paste.md-5.net/etigohuneg.coffeescript

Have you followed the documented steps to rebuild the android bindings?

[TobiasDeBruijn]

I have now yes, and have been able to produce a jar which can be loaded by Gradle in a non-android project.

I think the last item to fix is going to be the native .so files that are required. libdeepspeech.so is working, however I'm not quite sure where deepspeech-jni.so is supposed to come from.

[lissyx]

I think the last item to fix is going to be the native .so files that are required. libdeepspeech.so is working, however I'm not quite sure where deepspeech-jni.so is supposed to come from.

As I said on Matrix, it's being built by the bindings target in the Makefile

[lissyx]

@TheDutchMC I know nothing about Java, will this work on non Android systems?
https://github.com/mozilla/DeepSpeech/blob/master/native_client/java/libdeepspeech/src/main/java/org/deepspeech/libdeepspeech/DeepSpeechModel.java#L8-L11

[TobiasDeBruijn]

@TheDutchMC I know nothing about Java, will this work on non Android systems?
https://github.com/mozilla/DeepSpeech/blob/master/native_client/java/libdeepspeech/src/main/java/org/deepspeech/libdeepspeech/DeepSpeechModel.java#L8-L11

Yes that will definitely work! That's 'vanilla' Java

[TobiasDeBruijn]

Short update. I've been able to compile the required .so's and get it to compile and run in Java. Not sure if it actually works, because I've still got no clue how to use DeepSpeech, but at least there are no errors.

Get it up and running
./native_client/java/libdeepspeech/CMakeLists.txt:

cmake_minimum_required(VERSION 3.10)

include_directories( ${CMAKE_JAVA_} $ENV{JAVA_HOME}/include $ENV{JAVA_HOME}/include/linux )

add_library( deepspeech-jni SHARED ../jni/deepspeech_wrap.cpp )

find_library(deepspeech-lib NAMES deepspeech PATHS ${CMAKE_SOURCE_DIR}/libs/ REQUIRED)
message(STATUS ${deepspeech-lib})

add_custom_command( TARGET deepspeech-jni POST_BUILD COMMAND ${CMAKE_COMMAND} -E copy ${CMAKE_SOURCE_DIR}/libs/libdeepspeech.so ${CMAKE_LIBRARY_OUTPUT_DIRECTORY}/libdeepspeech.so )

target_link_libraries( deepspeech-jni ${deepspeech-lib} )

Create java bindings with SWIG:

1. in ./native_client/java:

swig -c++ -java -package org.deepspeech.libdeepspeech -outdir libdeepspeech/src/main/java/org/deepspeech/libdeepspeech/ -o jni/deepspeech_wrap.cpp jni/deepspeech.i

Modify ./native_client/java/libdeepspeech/src/main/java/org/deepspeech/libdeepspeech/DeepSpeechModel.java:
Replace the current static block with:

    static {
        String jniName = "libdeepspeech-jni.so";
        String libName = "libdeepspeech.so";
        URL jniUrl = DeepSpeechModel.class.getResource("/jni/x86_64/" + jniName);
        URL libUrl = DeepSpeechModel.class.getResource("/jni/x86_64/" + libName);
        File tmpDir = null;
		try {
			tmpDir = Files.createTempDirectory("libdeepspeech").toFile();
		} catch (IOException e) {
			e.printStackTrace();
		}
        tmpDir.deleteOnExit();
    	
        File jniTmpFile = new File(tmpDir, jniName);
        jniTmpFile.deleteOnExit();
        File libTmpFile = new File(tmpDir, libName);
        libTmpFile.deleteOnExit();
        
        try (
        		InputStream jniIn = jniUrl.openStream();
        		InputStream libIn = libUrl.openStream();
        ) {
            Files.copy(jniIn, jniTmpFile.toPath());
            Files.copy(libIn, libTmpFile.toPath());
        } catch (IOException e) {
			e.printStackTrace();
		}
        
        System.load(jniTmpFile.getAbsolutePath());
        System.load(libTmpFile.getAbsolutePath());
    }

Compile with Gradle:

1. in ./native_client/java:./gradlew build

Compile libdeepspeech-jni.so

1. in ./native_client/java/libdeepspeech: cmake .
2. in ./native_client/java/libdeepspeech: sudo make (I had permission issues without sudo`

Use in your own project:

1. Copy ./native_client/java/libdeepspeech/libdeepspeech-jni.so to ./{YOUR PROJECT}/src/main/resources/jni/x86_64/libdeepspeech-jni.so
2. Copy ./native_client/java/libdeepspeech/libs/libdeepspeech.so to ./{YOUR PROJECT}/src/main/resources/jni/x86_64/libdeepspeech.so
3. Copy ./native_client/java/libdeepspeech/build/libs/libdeepspeech.jar to ./{YOUR PROJECT}/libs/libdeepspeech.jar
4. Modify build.gradle

plugins {
    id 'com.github.johnrengelman.shadow' version '5.0.0'
}

repositories {
    flatDir {
        dirs 'libs'
    }
}

dependencies {
    compile ':libdeepspeech'
}

5. in ./{YOUR PROJECT} run ./gradlew shadowJar

This should be all steps necessary to include DeepSpeech in a Non-Android Java application.

Todo

• Automate all this
• This way will only support x86-64 Linux, fine for me but maybe not for others

I've not yet tested if it works, since I have not yet fully figured out how to work with DeepSpeech, but my console output is as follows:

TensorFlow: v2.3.0-6-g23ad988
DeepSpeech: v0.9.3-0-gf2e9c85
2021-01-18 12:26:49.869005: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN)to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
Extracting audio features...
Running inference...

Finished! Took 1848

It runs without errors though it does not seem to produce any useful output (hence the blank line)

Test code: https://paste.md-5.net/qujujoneha.java

[Tri0de]

@TheDutchMC Can you put your fix into a fork and then link it? Nmv I found it.

[TobiasDeBruijn]

I am working on a PR, should be already in the list of open PRs. It's not ready yet though.