Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Exposure for indoors #86

Merged
merged 4 commits into from
Aug 15, 2017
Merged

Exposure for indoors #86

merged 4 commits into from
Aug 15, 2017

Conversation

jgoppert
Copy link
Member

@jgoppert jgoppert commented Oct 12, 2016

@LorenzMeier @bkueng @ecmnet @mhkabir please test, this is a huge improvement for indoors and I suspect outdoors as well.

Changes

  • Add a lot more control over camera gain and exposure, can set with params not to allow fast tuning
  • Tune exposure and gain settings to work well with low light and outdoor/ bright light settings using auto gain and exposure control.
  • Instead of publishing flow every 10 received frames, publishes at fixed rate set by PUB_RATE param

@jgoppert jgoppert force-pushed the exposure_for_indoors branch from 5ed0082 to 08ced78 Compare October 12, 2016 05:36
@jgoppert jgoppert force-pushed the exposure_for_indoors branch from 08ced78 to ee88862 Compare October 12, 2016 05:41
@ecmnet
Copy link

ecmnet commented Oct 14, 2016

@jgoppert I've not setup my development environment for PX4Flow. Could you provide a binary to upload?

@jgoppert
Copy link
Member Author

jgoppert commented Oct 14, 2016

@ecmnet, I've uploaded the built firmware to my google drive here: https://drive.google.com/open?id=0B3wIEs1gcVahNktlQkhCMndIeUE

@ecmnet
Copy link

ecmnet commented Oct 14, 2016

@jgoppert Thanks a lot! I'll give it a try a soon as I've mounted flow to my device again.

@ecmnet
Copy link

ecmnet commented Oct 14, 2016

@jgoppert Unfortunately, my PX4Flow does not work after upgrading (does not connect to QGC at all)

@jgoppert
Copy link
Member Author

Weird, will try uploading the image I created for you tonight on my device.

@ecmnet
Copy link

ecmnet commented Oct 15, 2016

@jgoppert I've setup my environment now and was able to compile and upload this branch. Is it correct to have some white square dots in the image now?

@jgoppert
Copy link
Member Author

Yes, the dots indicate high quality optical flow in that region of the image.

@jgoppert
Copy link
Member Author

Or more specifically, that 8x8 tile had enough difference between its pixels (f thrsh param) that it included that tile in the optical flow calculation.

@ecmnet
Copy link

ecmnet commented Oct 15, 2016

@jgoppert Thanks for clarification. I will do some testflights as soon I've mounted flow again.

@ecmnet
Copy link

ecmnet commented Nov 21, 2016

@jgoppert Flow quality seems not to be reasonable with this patch. Even, if I darken the lens, quality is reported at 220-250 in my case.

@jgoppert
Copy link
Member Author

It increases the gain and exposure when you cover the lens so you start picking up noise. Look at the image in qgc. The max gain and max exposure can be lowered to limit the effect, but if you don't cover the lens I think it gives good low and high light performance.

Copy link
Contributor

@ChristophTobler ChristophTobler left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. I think I can give this a test the next few days

@@ -699,7 +707,7 @@ int main(void)
uint16_t frame = 0;
for (frame = 0; frame < image_size_send / MAVLINK_MSG_ENCAPSULATED_DATA_FIELD_DATA_LEN + 1; frame++)
{
mavlink_msg_encapsulated_data_send(MAVLINK_COMM_2, frame, &((uint8_t *) current_image)[frame * MAVLINK_MSG_ENCAPSULATED_DATA_FIELD_DATA_LEN]);
mavlink_msg_encapsulated_data_send(MAVLINK_COMM_2, frame, &((uint8_t *) previous_image)[frame * MAVLINK_MSG_ENCAPSULATED_DATA_FIELD_DATA_LEN]);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why did you change this to previous_image?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

so that it gets overlayed with the dots for flow quality

@@ -79,7 +79,7 @@ void mt9v034_context_configuration(void)
uint16_t new_height_context_b = FULL_IMAGE_COLUMN_SIZE * 4;

/* blanking settings */
uint16_t new_hor_blanking_context_a = 709 + MINIMUM_HORIZONTAL_BLANKING;// 709 is minimum value without distortions
uint16_t new_hor_blanking_context_a = 425 + MINIMUM_HORIZONTAL_BLANKING;// 709 is minimum value without distortions
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

comment needs to be adapted

@lericson
Copy link

I've tested this and it doesn't work very well at all for me. It might be my hardware of course but I get pretty heavy banding with this.

@jgoppert
Copy link
Member Author

jgoppert commented Jun 16, 2017

@lericson Go into qgc and play with exposure_max and brightness. Lowering brightness down should fix it for you (try brightness 10, max_exposure 300). I've tested this a lot, so trust me, it is a huge difference for indoors. But watch out for specular reflections and shadows. I've seen the banding before as well, it is also there on the defualt firmware. It just can be more obvious when the auto gain is adjusting etc.

@lericson
Copy link

@jgoppert I played around a bit with the parameters. Couldn't get very good results sadly. For your perusing, here is my merged version of this PR and #90: https://github.com/KTH-CAS-UAV/Flow/tree/master-with-exposure-pr

@jgoppert
Copy link
Member Author

jgoppert commented Jun 16, 2017

Don't have time to load it myself. What does your flow quality say? It should be much higher with this patch. Also, when you evaluate the quality, don't just look at the image, enable the flag SEND_LPOS. This will give you output of a system onboard estimator of the position, assuming that the board is always aligned with north. Then I usually walk in a 2 m square.

@lericson
Copy link

lericson commented Jun 17, 2017 via email

@jgoppert
Copy link
Member Author

jgoppert commented Jun 17, 2017

When you set the parameters it doesn't stick after a reboot, if you set brightness to 8 and max exposure to 200 (in the firmware and rebulid), that is bascially the same as the default firmware. This firmware is basically just giving you more control over the settings.

Brightness is basically the desired mean of the pixel values and it plays with auto gain to and auto exposure to get that.

It also shows you the debug for the optical flow, the dots mean that the algorithm is using that cell for flow calculation. If you turn it really dark and you see no dots, there is not enough difference in intensity between the pixels in that cell for it to use it for the flow calculation.

It also stabilizes the flow rate output to the rate you set instead of depending on your exposure setting.

@@ -50,7 +50,7 @@ OBJDUMP = $(CROSSDEV)objdump

# Check if the right version of the toolchain is available
#
CROSSDEV_VER_SUPPORTED = 4.7.4 4.7.5 4.7.6 4.8.4 4.9.3
CROSSDEV_VER_SUPPORTED = 4.7.4 4.7.5 4.7.6 4.8.4 4.9.3 5.4.1
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This supports the latest gcc, so I included it here.

@@ -40,7 +40,7 @@
#define ONBOARD_PARAM_NAME_LENGTH 15
#define BOTTOM_FLOW_IMAGE_HEIGHT 64
#define BOTTOM_FLOW_IMAGE_WIDTH 64
#define BOTTOM_FLOW_SEARCH_WINDOW_SIZE 4
#define BOTTOM_FLOW_SEARCH_WINDOW_SIZE 8
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can play with this, but this makes it possible for it to see a large max velocity, but you have to watch the update rate and make sure the cpu isn't maxing out.

PARAM_HDR,
PARAM_AEC,
PARAM_AGC,
PARAM_BRIGHT,
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here I exposed settings that were hard coded in the firmware.

dma_reconfigure();
buffer_reset();
}

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is just setting the parameters that you request in QGC.


}
}
}
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This piece of code was commented out and shows you flow quality indicators for each grid cell.

if (counter % (uint32_t)global_data.param[PARAM_BOTTOM_FLOW_SERIAL_THROTTLE_FACTOR] == 0)//throttling factor
uint32_t now = get_boot_time_us();
uint32_t time_since_last_pub = now - time_last_pub;
if (time_since_last_pub > (1.0e6f/global_data.param[PARAM_BOTTOM_FLOW_PUB_RATE]))
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the logic to publish at a fixed rate instead of based on the exposure setting.

lpos.vz = 0;
lpos.vx = ground_distance*accumulated_flow_x/integration_timespan;
lpos.vy = ground_distance*accumulated_flow_y/integration_timespan;
lpos.vz = 0; // no direct measurement, just ignore
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I included the velocity here for debug.

strcpy(global_data.param_name[PARAM_AGC], "AGC");
global_data.param_access[PARAM_AGC] = READ_WRITE;

global_data.param[PARAM_BRIGHT] = 20;
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you have trouble with washout/ banding, try lowering this value. 8 is the minimum.

strcpy(global_data.param_name[PARAM_BOTTOM_FLOW_PUB_RATE], "BFLOW_RATE");
global_data.param_access[PARAM_BOTTOM_FLOW_PUB_RATE] = READ_WRITE;

global_data.param[PARAM_EXPOSURE_MAX] = 500;
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Increasing this value helps in low light, but it also makes the max velocity you can go lower. If you have trouble with velocity maxing out, lower this to around 200 or 100.

strcpy(global_data.param_name[PARAM_EXPOSURE_MAX], "EXPOSURE_MAX");
global_data.param_access[PARAM_EXPOSURE_MAX] = READ_WRITE;

global_data.param[PARAM_GAIN_MAX] = 16;
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gain can make the image pixelated. If your image has a lot of noise, try lowering this to 4 or 8.

@jgoppert
Copy link
Member Author

@lericson I tried to explain my changes in this patch, hope that helps and you find some of it useful for your work.

@ChristophTobler
Copy link
Contributor

@jgoppert I have bench tested this PR so far. Looks very good. I increased the flow feature threshold because it reported good quality even though it was just noisy and pointing to a plain white surface. I think it's safer this way.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants