Native image encoder intermittently fails to send final frame of image data causing hang #2
Comments
|
hi, found your beautiful implementation, here is RaspiMJPEG project written with C language, this utility takes snapshots as fast as possible and flush them into the disk, i think this can help you a lot |
|
Thanks! The Java implementation here is based on the C code for the RaspiStill utility in the project that you linked to. Nevertheless I will look at the particular code you linked and see if I can spot anything I did wrong. |
|
here is also parts of RaspiMJPEG: |
|
yes you're right, this utility based on RaspiStill, but author of this utility has increased speed of taking pictures |
|
That code does not really help too much, the key idea behind this Java implementation is to use the MMAL API directly and not rely on forking processes and using native commands. Some more background on this issue... I very carefully did two separate Java implementations of this, from scratch, based on the source code in the raspicam project, in particular the source code that used the native MMAL API directly. Both of those implementations I did suffered the exact same problems, so clearly I missed something non-obvious, twice. It is very much possible to invoke the native command-line tools from Java, and there are already Java projects that do that. But I do not want to do that, I want a Java API that does not execute native commands in their own processes, but instead to implement the exact same thing that those native commands do when they talk to the MMAL API. This is much more difficult, and requires me to write a lot more code, but the end result in my opinion will be much better. If it only worked 100% reliably! |
|
yes, exactly for direct calls to MMAL API from Java by using JNA i liked your project |
|
I still hope to find a resolution to this issue, so the project is still very much alive, it's just stalled right now. |
|
maybe this is a bug of MMAL API? |
|
also take me a piont where is that happening at code, i want to see how can i help you |
|
I can't really point directly to the code that is broken. The code dealing with handling picture data is in https://github.com/caprica/picam/blob/master/src/main/java/uk/co/caprica/picam/EncoderBufferCallback.java This is a Java implementation of a native callback invoked by the MMAL API when camera data is ready to be sent. This callback accepts the picture data and then either finishes if it's the last data block, or requests more picture data if it is not the last data block. The problem is that this code executes correctly, it requests the next block when needed, but sometimes the native code simply refuses to invoke this callback with the last black of data. This failure happens outside of Java code. Now, it could be that the Java code has exposed a rare bug in the native code, or it could be that there is something wrong in the Java code. Another problem is that there is a lot of stuff that has to be set up and linked together via the MMAL API, and any part of that could be slightly wrong. All of the native setup code is here https://github.com/caprica/picam/blob/master/src/main/java/uk/co/caprica/picam/Camera.java |
|
Your post inspired me to fully upgrade my Pi and after doing so I took 50 pictures without any problems - no hanging. I'll do some more testing. I can't say this issue is definitely gone, but so far it's working better than before. |
|
it's still here, i get it time to time |
|
can you say how did you test it? |
|
Oh well, my testing was just ad-hoc casual testing using my own test application. Doing that same test previously it failed about 10% of the time. Today it hasn't failed yet. I was going to write a proper stress-test later. I guess there's still an issue. |
|
i have added extra logs looks like |
|
That is indeed the same behaviour I was seeing previously. And yeah, that's a native buffer, not a Java buffer. |
|
Sadly my stress test did not show good results. Strange messages from MMAL dumped to the error console, looks like a native race to me. Still hangs intermittently. So updating everything to latest does not resolve the issue, unfortunately. I still don't know what's actually wrong. |
|
This old abandoned issue looks somewhat similar: https://www.raspberrypi.org/forums/viewtopic.php?t=108707&p=749409 |
|
With debug logging turned on on the Pi, I occasionally see these warnings in the log (they do not correspond to a hang): When I CTRL+C a hanged app, this line pops up in the log: My stress-test app ran to 90 out of 100 pictures taken. What's interesting is it runs better with more logging (so it seems) and that to me implies a race somewhere. |
|
This also looks somewhat related: |
|
with this part of code it looks like nothing happens |
|
ah just got it again, never mind :) |
|
Yeah, I've been down that path myself before. The reason it's there is because it was in the original C code. |
|
I'm afraid I don't really know the MMAL API, nor the Pi platform. I just attempted to port the code from C to Java/JNA. |
|
i am in the same way :) |
|
snapshots also possible by DispmanX it is part of Raspberry VideoCore |
|
Ideally I'd like to find the minimal possible working native code to get a snapshot/still-picture. I don't care about video. I don't care what native API I use to get it. |
|
ok, then this is what you looking for DispmanX |
|
@vrosca Your hunch is spot on, that's exactly what I did. I also saw that
once thread hangs, nothing else can access the camera without a reboot,
even raspistill.
…On Oct 17, 2017 9:59 AM, "vrosca" ***@***.***> wrote:
There are two separate problems:
1. the pictureData callback working reliably
2. a rapid succession of camera/encoder destory/creates not blocking
I haven't invested any time into #2
<#2> - seems like a fairly exotic
scenario. However, I'm starting to be reasonably confident that #1
<#1> is properly addressed now.
Lets see what the RPI2b test yields, if it handles a multicore PI properly
then it should be good to go
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#2 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AFoIk3OztSgX094lI1hMPkI0gkeQdOMjks5stMDhgaJpZM4NeUZ0>
.
|
|
Yeah, to be honest I don't really care if we never solve your scenario #2, that approach was never intended to be the end-game. |
|
That's understandable, maybe just adding a note saying not to do that would
be good.
…On Oct 17, 2017 10:46 AM, "Mark Lee" ***@***.***> wrote:
Yeah, to be honest I don't really care if we never solve your scenario #2
<#2>, that approach was never
intended to be the end-game.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#2 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AFoIk6okXCxva-EncdfDDhzsfOGSYPtWks5stMvIgaJpZM4NeUZ0>
.
|
|
Trying to digest everything that's been mentioned in this thread already, but during the initial development of MMALSharp I used to see MMAL hangs due to not clearing port pools correctly. The hang was because there was still a reference on the buffer and wasn't released by MMAL. So in MMALSharp specifically, I've managed to get a clean shutdown of resources by doing the following:
Happy to help if you need it. This problem was bugging me for months! |
|
Thanks for such a helpful comment. I may revisit this. I did try to be very careful about allocating and freeing resources, but I just couldn't see where I may have went wrong... on three separate, from scratch, implementations. |
|
@techyian one follow-up question if you don't mind, in your clean-up suggestions, are you doing the create/process-image-data/cleanup for every image capture individually, or do you keep the camera "open" and do multiple captures before cleaning up? |
|
Sure - if you take a look at the BeginProcessing method, I keep the camera component resources active, but cleanup ports and stop capture after each run. That way, people using the library can then add/remove components and create a different pipeline after each run if they wish to. |
|
Just wanted to mention that after taking the approach explained in my first post (23 Sep 2017) my surveillance camera now has 6 months of uptime without any freezes |
|
That is impressive, thanks for the update. Would be so awesome if you could isolate the essential code changes, last time you said the code was "butchered" :-D |
|
I'll try - at some point in Q2 - to get over my anti-git bias, rebase my changes on your codebase and send a pull request |
|
@vrosca just share your code as a zip file if you not good with Git |
|
I finally found some time to revisit this. I applied the changes suggested by @vrosca on 23rd September and so far I've been able to capture hundreds upon hundreds of pictures without any problems (my code uses the PNG encoder rather than JPG for what it's worth). My last test ran 1,000 straight captures perfectly. I will work on cleaning up my code and committing it soon and maybe even pushing a release to Maven Central. I still don't see anything "wrong" with the original code and I still worry about races, but so far these changes look good. Thank you all in this thread who helped, especially @vrosca for the suggested code changes. |
|
Cool, do you have statistics on how many pictures it can take in one second? |
|
with C++ implementation it can take about 30 pictures per second and save them into Ramdisk |
|
Honestly I have no idea since I have not measured it, but it seems slow-ish to me - then again, my tests so far save all images to disk rather than ramdisk so that's going to slow it down, and using the JPG encoder would likely improve things. On the other hand, raw performance is not the primary goal of this project, rather the goal is ease of use and Java is a lot easier to deal with than C++, especially for less experienced developers. |
|
@caprica try to use |
|
Sure, but like I mentioned raw performance is not the primary goal of this project, although I would be interested in the numbers :) To make any significant performance improvements likely would mean removing JNA and using JNI directly - and again, that will make things more difficult for people, not easier. |
|
I understand your position. when I found your project I thought it was a godsend for my project (controlled tank from a mobile phone) but when using it I was disappointed with extremely slow work |
|
Well, now that it is hopefully working reliably, performance concerns could be looked at. But if you want best performance, use C/C++ rather than Java I suppose. I mean, the JVM is fundamentally slow on these low-power devices isn't it. |
|
I agree, native applications will always be faster, but in our hands to bring the speed close to the native, we (community) just need to try |
|
@mark: glad the changes worked for you! I don't have the thread that fresh in my mind but seem to recall the issue was that JNA was automatically filling the structures (including buffer pointers) BEFORE you acquired the lock from the broadcom libs. When doing large JPG or H264 encoding this pretty much led to a freeze all the time (one callback per 64K-ish buffer, quickly runs into the race condition). The core of the fix was giving up on that approach and getting the buffer/callback pointers manually after the lock is acquired.
I'd also like to take this chance to apologize for being a poor sport and not backporting the changes to picam. My PI has been overseeing my orchard for the last 9 months and I didn't find the time to set up a second unit at home to try out the original picam package + changes. In my case things got more complicated (I'm using it for motion detection, video recording and still captures) hence I switched to JNI for some of the heavier operations and ended up rewriting most of the framework around it.
My motion detection needs aside, I've found the biggest drag on performance as oposed to raw raspivid / raspistill comes from the fact JNA allocates a new thread for each callback! You can confirm that easily with a logger or debugger, notice how the thread name always changes. Biggest jump in performance came from adding a Callback manager component on the Java side and the JNI to go with it to supply pointers for operation callbacks. If you ever need to increase throughput for many large images / video, let me know and I'll share some more. Find below a bit of code - the key idea is that the broadcom library usually issues callbacks from the same thread, so when making any calls with callbacks you just supply the pointer to void callback(MMAL_PORT_T *port, MMAL_BUFFER_HEADER_T *buffer) which then calls back into the java side void receiveCallback(long portAddress, long bufferAddress)
BR--Val
Java side (simplified):package org.ease.oversight.picamera.mmal.bindings;
import org.ease.oversight.picamera.mmal.bindings.internal.LibInitializer;
import com.sun.jna.Pointer;
import java.io.IOException;
/**
* @author vrosca - 26/09/2017 12:41
*/
public class LibPiCamera {
public static void receiveCallback(long portAddress, long bufferAddress) {
CallbackManager.receiveCallback(portAddress, bufferAddress);
}
public static native long getCallbackAddress();
public static native long getReleaseBufferCallbackAddress();
public static native long jmalloc(int size);
public static native long jmemcpy(long destination, long source, int size);
public static native int countDifferences(long dataPointer1, long dataPointer2, int width, int height);
public static native int countAndMarkDifferences(long target, long source, int width, int height);
public static native int averageLuma(long image, int width, int height, int stepSize);
static {
try {
LibInitializer.initialize("picamera");
} catch (IOException e) {
throw new RuntimeException(e);
}
}
public static void jmemcpy(Pointer destination, Pointer source, int size) {
jmemcpy(Pointer.nativeValue(destination), Pointer.nativeValue(source), size);
}
public static int countDifferences(Pointer dataPointer1, Pointer dataPointer2, int width, int height) {
return countDifferences(Pointer.nativeValue(dataPointer1), Pointer.nativeValue(dataPointer2), width, height);
}
public static int countAndMarkDifferences(Pointer target, Pointer source, int width, int height) {
return countAndMarkDifferences(Pointer.nativeValue(target), Pointer.nativeValue(source), width, height);
}
public static int averageLuma(Pointer image, int width, int height, int stepSize) {
return averageLuma(Pointer.nativeValue(image), width, height, stepSize);
}
}
package org.ease.oversight.picamera.mmal.bindings;
import org.ease.oversight.picamera.picam.bindings.internal.MMAL_PORT_BH_CB_T;
import org.ease.oversight.picamera.picam.bindings.internal.MMAL_PORT_T;
import com.sun.jna.Pointer;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import java.util.HashMap;
import java.util.Map;
/**
* @author vrosca - 26/09/2017 12:38
*/
public class CallbackManager {
private static final Logger logger = LoggerFactory.getLogger(CallbackManager.class);
private static final long CALLBACK_ADDRESS = LibPiCamera.getCallbackAddress();
private static final long RELEASE_BUFFER_CALLBACK_ADDRESS = LibPiCamera.getReleaseBufferCallbackAddress();
private static final Pointer CALLBACK_POINTER = new Pointer(CALLBACK_ADDRESS);
private static final Pointer RELEASE_BUFFER_CALLBACK_POINTER = new Pointer(RELEASE_BUFFER_CALLBACK_ADDRESS);
private static final Map<Long, Callback> callbackMap = new HashMap<>();
public interface Callback extends MMAL_PORT_BH_CB_T {
}
public static Pointer registerCallback(MMAL_PORT_T port, Callback callback) {
registerCallback(Pointer.nativeValue(port.getPointer()), callback);
return CALLBACK_POINTER;
}
public static void registerCallback(long address, Callback callback) {
callbackMap.put(address, callback);
}
public static void cancelCallback(MMAL_PORT_T port) {
cancelCallback(Pointer.nativeValue(port.getPointer()));
}
public static void cancelCallback(long address) {
callbackMap.remove(address);
}
public static void receiveCallback(long portAddress, long bufferAddress) {
Callback callback = callbackMap.get(portAddress);
if (callback != null) {
callback.apply(new Pointer(portAddress), new Pointer(bufferAddress));
}
}
public static Pointer getReleaseBufferCallbackPointer() {
return RELEASE_BUFFER_CALLBACK_POINTER;
}
}
C side:#include <stdio.h>
#include <jni.h>
#include <pthread.h>
#include "interface/mmal/mmal.h"
#include "interface/mmal/mmal_logging.h"
#include "interface/mmal/mmal_buffer.h"
#include "interface/mmal/util/mmal_util.h"
#include "interface/mmal/util/mmal_util_params.h"
#include "interface/mmal/util/mmal_default_components.h"
#include "interface/mmal/util/mmal_connection.h"
#include "interface/mmal/mmal_parameters_camera.h"
// cached refs for later callbacks
JavaVM * g_vm;
jclass callbackDispatcherClass;
jmethodID callbackDispatchingMethodId;
jint JNI_OnLoad(JavaVM* vm, void* reserved)
{
g_vm = vm;
JNIEnv* env;
if ( (*g_vm)->GetEnv(g_vm, &env, JNI_VERSION_1_8) != JNI_OK ) {
printf("Can't get JNIEnv, aborting\n");
return -1;
} else {
jclass clazz = (*env)->FindClass(env, "org/ease/oversight/picamera/mmal/bindings/LibPiCamera");
if (clazz == NULL) {
printf("Can't find LibPiCamera class, aborting\n");
return -1;
} else {
callbackDispatcherClass = (*env)->NewGlobalRef(env, clazz);
(*env)->DeleteLocalRef(env, clazz);
callbackDispatchingMethodId = (*env)->GetStaticMethodID(env, callbackDispatcherClass, "receiveCallback", "(JJ)V");
if (callbackDispatchingMethodId == NULL) {
printf("Can't find receiveCallback static method, aborting\n");
return -1;
} else {
return JNI_VERSION_1_8;
}
}
}
}
pthread_once_t destructorKeyOnce = PTHREAD_ONCE_INIT;
pthread_key_t destructorKey;
void callback_destructor(void* buf) {
unsigned long *_tid = buf;
printf("Destroying bound thread: %lu\n", *_tid);
free(buf);
(*g_vm)->DetachCurrentThread(g_vm);
}
void callback_destructor_creator() {
pthread_key_create(&destructorKey, callback_destructor);
}
void callback(MMAL_PORT_T *port, MMAL_BUFFER_HEADER_T *buffer)
{
JNIEnv * g_env;
int getEnvStat = (*g_vm)->GetEnv(g_vm, (void **)&g_env, JNI_VERSION_1_8);
if (getEnvStat == JNI_EDETACHED) {
// vrosca: associate destructor
pthread_once(&destructorKeyOnce, callback_destructor_creator);
if ((*g_vm)->AttachCurrentThreadAsDaemon(g_vm, (void **) &g_env, NULL) != 0) {
printf("JNI Failed to attach\n");
}
} else if (getEnvStat == JNI_OK) {
//
} else if (getEnvStat == JNI_EVERSION) {
printf("GetEnv: version not supported\n");
}
(*g_env)->CallStaticVoidMethod(g_env, callbackDispatcherClass, callbackDispatchingMethodId, (jlong)(uintptr_t) port, (jlong)(uintptr_t) buffer);
/* if ((*g_env)->ExceptionCheck(g_env)) {
(*g_env)->ExceptionDescribe(g_env);
}*/
/*
if (getEnvStat == JNI_EDETACHED) {
(*g_vm)->DetachCurrentThread(g_vm);
}*/
}
JNIEXPORT jlong JNICALL
Java_org_ease_oversight_picamera_mmal_bindings_LibPiCamera_getCallbackAddress(JNIEnv *env, jclass type) {
return (jlong)(uintptr_t) &callback;
}
static void release_buffer_callback(MMAL_PORT_T *port, MMAL_BUFFER_HEADER_T *buffer)
{
mmal_buffer_header_release(buffer);
}
JNIEXPORT jlong JNICALL
Java_org_ease_oversight_picamera_mmal_bindings_LibPiCamera_getReleaseBufferCallbackAddress(JNIEnv *env, jclass type) {
return (jlong)(uintptr_t) &release_buffer_callback;
}
JNIEXPORT jlong JNICALL
Java_org_ease_oversight_picamera_mmal_bindings_LibPiCamera_jmalloc(JNIEnv *env, jclass type, jint size) {
return (jlong)(uintptr_t) malloc(size);
}
JNIEXPORT jlong JNICALL
Java_org_ease_oversight_picamera_mmal_bindings_LibPiCamera_jmemcpy(JNIEnv *env, jclass type, jlong destination, jlong source, jint size) {
//printf("Size: %d", size);
return (jlong)(uintptr_t) memcpy((void*) destination, (void*) source, size);
}
On Wednesday, 21 November 2018, 16:10, Alexander Shniperson <notifications@github.com> wrote:
I agree, native applications will always be faster, but in our hands to bring the speed close to the native, we (community) just need to try—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.
|
|
PS: I use the Oracle JRE on the PI (it has a proper hotspot VM), last year at least OpenJDK was horrendously slow I think they actually ran in interpreted mode on the ARM. With the trick above (and some use of NIO buffers on the Java side) performance absolutely matches raspistill / raspivid
On Wednesday, 21 November 2018, 16:58, Val Rosca <val_rosca@yahoo.com> wrote:
@mark: glad the changes worked for you! I don't have the thread that fresh in my mind but seem to recall the issue was that JNA was automatically filling the structures (including buffer pointers) BEFORE you acquired the lock from the broadcom libs. When doing large JPG or H264 encoding this pretty much led to a freeze all the time (one callback per 64K-ish buffer, quickly runs into the race condition). The core of the fix was giving up on that approach and getting the buffer/callback pointers manually after the lock is acquired.
I'd also like to take this chance to apologize for being a poor sport and not backporting the changes to picam. My PI has been overseeing my orchard for the last 9 months and I didn't find the time to set up a second unit at home to try out the original picam package + changes. In my case things got more complicated (I'm using it for motion detection, video recording and still captures) hence I switched to JNI for some of the heavier operations and ended up rewriting most of the framework around it.
My motion detection needs aside, I've found the biggest drag on performance as oposed to raw raspivid / raspistill comes from the fact JNA allocates a new thread for each callback! You can confirm that easily with a logger or debugger, notice how the thread name always changes. Biggest jump in performance came from adding a Callback manager component on the Java side and the JNI to go with it to supply pointers for operation callbacks. If you ever need to increase throughput for many large images / video, let me know and I'll share some more. Find below a bit of code - the key idea is that the broadcom library usually issues callbacks from the same thread, so when making any calls with callbacks you just supply the pointer to void callback(MMAL_PORT_T *port, MMAL_BUFFER_HEADER_T *buffer) which then calls back into the java side void receiveCallback(long portAddress, long bufferAddress)
BR--Val
Java side (simplified):package org.ease.oversight.picamera.mmal.bindings;
import org.ease.oversight.picamera.mmal.bindings.internal.LibInitializer;
import com.sun.jna.Pointer;
import java.io.IOException;
/**
* @author vrosca - 26/09/2017 12:41
*/
public class LibPiCamera {
public static void receiveCallback(long portAddress, long bufferAddress) {
CallbackManager.receiveCallback(portAddress, bufferAddress);
}
public static native long getCallbackAddress();
public static native long getReleaseBufferCallbackAddress();
public static native long jmalloc(int size);
public static native long jmemcpy(long destination, long source, int size);
public static native int countDifferences(long dataPointer1, long dataPointer2, int width, int height);
public static native int countAndMarkDifferences(long target, long source, int width, int height);
public static native int averageLuma(long image, int width, int height, int stepSize);
static {
try {
LibInitializer.initialize("picamera");
} catch (IOException e) {
throw new RuntimeException(e);
}
}
public static void jmemcpy(Pointer destination, Pointer source, int size) {
jmemcpy(Pointer.nativeValue(destination), Pointer.nativeValue(source), size);
}
public static int countDifferences(Pointer dataPointer1, Pointer dataPointer2, int width, int height) {
return countDifferences(Pointer.nativeValue(dataPointer1), Pointer.nativeValue(dataPointer2), width, height);
}
public static int countAndMarkDifferences(Pointer target, Pointer source, int width, int height) {
return countAndMarkDifferences(Pointer.nativeValue(target), Pointer.nativeValue(source), width, height);
}
public static int averageLuma(Pointer image, int width, int height, int stepSize) {
return averageLuma(Pointer.nativeValue(image), width, height, stepSize);
}
}
package org.ease.oversight.picamera.mmal.bindings;
import org.ease.oversight.picamera.picam.bindings.internal.MMAL_PORT_BH_CB_T;
import org.ease.oversight.picamera.picam.bindings.internal.MMAL_PORT_T;
import com.sun.jna.Pointer;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import java.util.HashMap;
import java.util.Map;
/**
* @author vrosca - 26/09/2017 12:38
*/
public class CallbackManager {
private static final Logger logger = LoggerFactory.getLogger(CallbackManager.class);
private static final long CALLBACK_ADDRESS = LibPiCamera.getCallbackAddress();
private static final long RELEASE_BUFFER_CALLBACK_ADDRESS = LibPiCamera.getReleaseBufferCallbackAddress();
private static final Pointer CALLBACK_POINTER = new Pointer(CALLBACK_ADDRESS);
private static final Pointer RELEASE_BUFFER_CALLBACK_POINTER = new Pointer(RELEASE_BUFFER_CALLBACK_ADDRESS);
private static final Map<Long, Callback> callbackMap = new HashMap<>();
public interface Callback extends MMAL_PORT_BH_CB_T {
}
public static Pointer registerCallback(MMAL_PORT_T port, Callback callback) {
registerCallback(Pointer.nativeValue(port.getPointer()), callback);
return CALLBACK_POINTER;
}
public static void registerCallback(long address, Callback callback) {
callbackMap.put(address, callback);
}
public static void cancelCallback(MMAL_PORT_T port) {
cancelCallback(Pointer.nativeValue(port.getPointer()));
}
public static void cancelCallback(long address) {
callbackMap.remove(address);
}
public static void receiveCallback(long portAddress, long bufferAddress) {
Callback callback = callbackMap.get(portAddress);
if (callback != null) {
callback.apply(new Pointer(portAddress), new Pointer(bufferAddress));
}
}
public static Pointer getReleaseBufferCallbackPointer() {
return RELEASE_BUFFER_CALLBACK_POINTER;
}
}
C side:#include <stdio.h>
#include <jni.h>
#include <pthread.h>
#include "interface/mmal/mmal.h"
#include "interface/mmal/mmal_logging.h"
#include "interface/mmal/mmal_buffer.h"
#include "interface/mmal/util/mmal_util.h"
#include "interface/mmal/util/mmal_util_params.h"
#include "interface/mmal/util/mmal_default_components.h"
#include "interface/mmal/util/mmal_connection.h"
#include "interface/mmal/mmal_parameters_camera.h"
// cached refs for later callbacks
JavaVM * g_vm;
jclass callbackDispatcherClass;
jmethodID callbackDispatchingMethodId;
jint JNI_OnLoad(JavaVM* vm, void* reserved)
{
g_vm = vm;
JNIEnv* env;
if ( (*g_vm)->GetEnv(g_vm, &env, JNI_VERSION_1_8) != JNI_OK ) {
printf("Can't get JNIEnv, aborting\n");
return -1;
} else {
jclass clazz = (*env)->FindClass(env, "org/ease/oversight/picamera/mmal/bindings/LibPiCamera");
if (clazz == NULL) {
printf("Can't find LibPiCamera class, aborting\n");
return -1;
} else {
callbackDispatcherClass = (*env)->NewGlobalRef(env, clazz);
(*env)->DeleteLocalRef(env, clazz);
callbackDispatchingMethodId = (*env)->GetStaticMethodID(env, callbackDispatcherClass, "receiveCallback", "(JJ)V");
if (callbackDispatchingMethodId == NULL) {
printf("Can't find receiveCallback static method, aborting\n");
return -1;
} else {
return JNI_VERSION_1_8;
}
}
}
}
pthread_once_t destructorKeyOnce = PTHREAD_ONCE_INIT;
pthread_key_t destructorKey;
void callback_destructor(void* buf) {
unsigned long *_tid = buf;
printf("Destroying bound thread: %lu\n", *_tid);
free(buf);
(*g_vm)->DetachCurrentThread(g_vm);
}
void callback_destructor_creator() {
pthread_key_create(&destructorKey, callback_destructor);
}
void callback(MMAL_PORT_T *port, MMAL_BUFFER_HEADER_T *buffer)
{
JNIEnv * g_env;
int getEnvStat = (*g_vm)->GetEnv(g_vm, (void **)&g_env, JNI_VERSION_1_8);
if (getEnvStat == JNI_EDETACHED) {
// vrosca: associate destructor
pthread_once(&destructorKeyOnce, callback_destructor_creator);
if ((*g_vm)->AttachCurrentThreadAsDaemon(g_vm, (void **) &g_env, NULL) != 0) {
printf("JNI Failed to attach\n");
}
} else if (getEnvStat == JNI_OK) {
//
} else if (getEnvStat == JNI_EVERSION) {
printf("GetEnv: version not supported\n");
}
(*g_env)->CallStaticVoidMethod(g_env, callbackDispatcherClass, callbackDispatchingMethodId, (jlong)(uintptr_t) port, (jlong)(uintptr_t) buffer);
/* if ((*g_env)->ExceptionCheck(g_env)) {
(*g_env)->ExceptionDescribe(g_env);
}*/
/*
if (getEnvStat == JNI_EDETACHED) {
(*g_vm)->DetachCurrentThread(g_vm);
}*/
}
JNIEXPORT jlong JNICALL
Java_org_ease_oversight_picamera_mmal_bindings_LibPiCamera_getCallbackAddress(JNIEnv *env, jclass type) {
return (jlong)(uintptr_t) &callback;
}
static void release_buffer_callback(MMAL_PORT_T *port, MMAL_BUFFER_HEADER_T *buffer)
{
mmal_buffer_header_release(buffer);
}
JNIEXPORT jlong JNICALL
Java_org_ease_oversight_picamera_mmal_bindings_LibPiCamera_getReleaseBufferCallbackAddress(JNIEnv *env, jclass type) {
return (jlong)(uintptr_t) &release_buffer_callback;
}
JNIEXPORT jlong JNICALL
Java_org_ease_oversight_picamera_mmal_bindings_LibPiCamera_jmalloc(JNIEnv *env, jclass type, jint size) {
return (jlong)(uintptr_t) malloc(size);
}
JNIEXPORT jlong JNICALL
Java_org_ease_oversight_picamera_mmal_bindings_LibPiCamera_jmemcpy(JNIEnv *env, jclass type, jlong destination, jlong source, jint size) {
//printf("Size: %d", size);
return (jlong)(uintptr_t) memcpy((void*) destination, (void*) source, size);
}
On Wednesday, 21 November 2018, 16:10, Alexander Shniperson <notifications@github.com> wrote:
I agree, native applications will always be faster, but in our hands to bring the speed close to the native, we (community) just need to try—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.
|
|
My understanding is that JNA does indeed allocate a new Java Thread object for each callback, but no actual thread gets allocated, so that's not as bad as it might initially seem. And yes, your comment about getting the structures filled before them being locked now makes sense. Thanks a lot. I do want to reiterate that initially the main goal of this project was to make it very easy for inexperienced developers to use the Pi camera (Pi is foremost a learning platform after all), and for me that meant just using Java without asking people to compile JNI wrappers and such. If this issue is truly vanquished, I'm more motivated to advance this project now so if there are things that we can do to improve performance (or other things) I would like to hear about it. |
|
In hindsight, as is often the case, this bug is obvious and stupid. In short, by using the JNA auto-mapped object in the callback method signature, the native memory was being read by the Java code before that memory buffer was explicitly locked. Clearly this is bad, m'kay. Anyway, I have now captured literally thousands of pictures of varying sizes from small up to 10mb without a single problem during my most recent testing and I will commit the fix today. |
|
In my case it hasn't locked once in 9 months of operation (pi zero, 1 cpu) - hopefully we're not overlooking some other configuration where it might fail.
A JNI pthread attachment and custom callback logic is overkill for most applications (had to squeeze every bit of performance I could to run motion detection on the pi zero at 8 Mp). That said, once you profile the callbacks you'll probably notice the same thing I did: with the custom callback and using direct bytebuffers performance and cpu usage is identical to raspicam / raspivid. In my case, I bundle the so in the jar and have a simple class that extracts it in a temp file and loads the library from there. Since there aren't that many pi configurations out there, cross-compiling it in a makefile isn't that hard but yeah it does add to overall complexity...
Happy holidays ahead !
On Wednesday, 21 November 2018, 18:48, Mark Lee <notifications@github.com> wrote:
In hindsight, as is often the case, this bug is obvious and stupid.In short, by using the JNA auto-mapped object in the callback method signature, the native memory was being read by the Java code before that memory buffer was explicitly locked. Clearly this is bad, m'kay.Anyway, I have now captured literally thousands of pictures of varying sizes from small up to 10mb without a single problem during my most recent testing and I will commit the fix today.—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.
|
Fix for #2. Before the fix the JNA structures were present in the method signature. This meant that those structures would be automatically read (which is usually nice and convenient) but in this case the native memory was being accessed before it was locked (and therefore there was nothing preventing the contents of that memory from being changed while it was being accessed). This caused a race and would lead to an intermittent hang, usually observed when the native library seemingly stopped sending frames of picture data (always seemed to be the last frame). The fix therefore is to replace the JNA structure object in the method signature with a simple Pointer, then to use the native API to lock that pointer, and only then access the data in the object.
|
Fixed at 32deac5. |
From time to time, when requesting an image, the native image encoder simply fails to send the final frame of image data.
The image data is fetched in chunks, this is repeated 1..n times by the native code invoking a callback with the image data. The return value of that callback signifies to the native encoder that it should send the next frame. Tracing the Java code shows that the callback is correctly invoked, and that the correct return value is being sent back to native code. Nevertheless, from time to time the native encoder simply seems to stall and will stop sending image data. It always seems to be the last chunk of data.
What this means in practical terms is that the Java application hangs while waiting for a native callback that will never occur. The JVM must then be terminated.
This is a show-stopper issue that precludes the use of this library.
There is similar strangeness, it may be unrelated, during native resource clean-up whereby native warning messages are intermittently dumped to stdout or stderr (e.g. freeing a resource that has already been freed).
This may point to there being a fundamental problem with the implementation of this library.
Having said that, the code for handling capture data from the native library has been implemented and debugged very carefully from the ground up on two entirely separate occasions. So either there is something wrong in native code that the JVM is exposing, or the exact same mistake has been made on two separate Java implementations.
Until the root cause of this issue can be identified, this library can not be used reliably.
Any hints from anyone and anywhere on what could be wrong here will be most welcome.
The text was updated successfully, but these errors were encountered: