Bug Fix in Image Generator - Now Allows FCN's to be used #5228

joeyearsley · 2017-01-30T17:20:20Z

I needed FCN's so have altered this preprocessing step, I can also make a gist for the tutorials if anyone else prefers to learn through a simple example.

This does not allow for no sizes to be set if max poolings are used. In that case, the sizes can be none-square but have to be multiple of 2^m - where m is the number of max pooling layers.

In the future it might be possible to auto-resize to the nearest factor, but another variable will need adding to ensure we can disentangle the model from the preprocessing.

Made minor changes to allow for FCN's:

Target Size is now a tuple of (None, None) as only this can propagate through the directory iterator
Altered batch_x to be created in the loop, as you should only use a batch size of 1 for FCN of shape (None, None, Channels)
Allowed the directory iterator to use the folder it is passed if it finds no subfolders for classes (Usual use case in FCN)

Allowed BGR to be specified (for when using caffe converted models).

Made minor changes to allow for FCN's: - Target Size is now a tuple of (None, None) as only this can propagate through the directory iterator - Altered batch_x to be created in the loop, as you should only use a batch size of 1 for FCN of shape (None, None, Channels) - Allowed the directory iterator to use the folder it is passed if it finds no subfolders for classes (Usual use case in FCN) Allowed BGR to be specified (for when using caffe converted models).

PEP8 :|

fchollet · 2017-01-31T19:58:33Z

keras/preprocessing/image.py

@@ -191,7 +191,7 @@ def flip_axis(x, axis):
    return x


-def array_to_img(x, dim_ordering='default', scale=True):
+def array_to_img(x, dim_ordering='default', scale=True, color_mode='rgb'):


If you add an argument you should document it in the docstring.

fchollet · 2017-01-31T19:58:39Z

keras/preprocessing/image.py

@@ -241,7 +243,7 @@ def array_to_img(x, dim_ordering='default', scale=True):
        raise ValueError('Unsupported channel number: ', x.shape[2])


-def img_to_array(img, dim_ordering='default'):
+def img_to_array(img, dim_ordering='default', color_mode='rgb'):


Fixed both, please tell me if the docstring doesn't suffice.

Added docstrings

ahundt · 2017-03-28T17:39:17Z

@joeyearsley I'd appreciate an example. Also, https://github.com/aurora95/Keras-FCN has some good scripts for training FCNs

fchollet

Sorry for the long delay. I feel like I do not fully understand what you are trying to do, maybe because I don't fully understand what you mean by 'FCN'.

My definition of 'FCN' is a fully-convolutional network, i.e. a network that only includes convolutional layers (in particular no final classifier). This makes not assumption about targets.

fchollet · 2017-05-05T00:41:29Z

keras/preprocessing/image.py

@@ -298,7 +308,7 @@ def load_img(path, grayscale=False, target_size=None):
        img = img.convert('L')
    else:  # Ensure 3 channel even when loaded image is grayscale
        img = img.convert('RGB')
-    if target_size:
+    if target_size[0] and target_size[1]:  # Can't pass None in from iterator, hence tuple


We can keep the default to None and convert to (None, None) here. At the very least, target_size=None should keep being supported (for backwards compatibility).

fchollet · 2017-05-05T00:42:09Z

keras/preprocessing/image.py

@@ -777,6 +787,9 @@ def __init__(self, directory, image_data_generator,
            for subdir in sorted(os.listdir(directory)):
                if os.path.isdir(os.path.join(directory, subdir)):
                    classes.append(subdir)
+        # User passed in None and no sub-folders, hence FCN. N.B.: Pass in classes=['.'] for improvements.


I cannot quite understand this comment. Can you clarify?

Also: line too long.

ahundt · 2017-05-05T17:34:06Z

@fchollet I believe by FCN he means support for dense prediction tasks like image segmentation of which the FCN paper happens to be one implementation, right or not the two terms are often used interchangeably.

This class of tasks are not yet well supported by Keras in the "best practices by default" sense in which Keras is designed. A number of implementations have popped up to handle the problem. Examples of missing keras capabilities required for best practice dense prediction include:

APIs need to apply augmentation (zoom, flip, translation) to both the input data and the labels
Class weights do not work with segmentation predictions
The keras loss functions don't support "2D" loss, such as comparing against a (height x width x num_classes) tensor
no segmentation datasets are yet supported which I'm trying to alleviate in keras-contrib with coco and pascalvoc.
support for conversion of imagenet trained weights for dense prediction tasks, typically by some combination of steps taken in the U-Net, FCN, and multi scale context aggregation by dilated convolution papers such as:
- removing some or all pooling layers and add equivalent dilation to the next convolution layer
- adding skip connections and upsampling

It would be nice if support for dense prediction could be added to the requests for contributions so dense prediction improvements would qualify for direct inclusion in keras, rather than going through keras-contrib.

fchollet · 2017-05-05T20:38:38Z

@ahundt would you like to take over that project? It would involve:

coming up with a list of features to support
agreeing on a set of API changes and getting them approved (by me)
implementing it, either by managing the work of other contributors or coding it yourself
reviewing any PRs related to this goals (such as the present PR)

Yes, we can add to the 'request for contributions' section.

ahundt

have a couple questions & a padding issue that should be handled

ahundt · 2017-05-06T00:39:30Z

keras/preprocessing/image.py

-        batch_x = np.zeros((current_batch_size,) + self.image_shape)
+        # If image_shape has None in position 1 it is intended to be used as a FCN
+        if self.image_shape[1]:
+            batch_x = np.zeros((current_batch_size,) + self.image_shape)


does this assume all the images have the same shape?
If so, varying image dimensions should be handled, and some kind of target dimensions could be utilized to apply padding to the images so they are all the same size.

reference code:
https://github.com/aurora95/Keras-FCN/blob/370eb767df86f364d93eff9068a3edb0b194a18b/utils/SegDataGenerator.py#L221
https://github.com/nicolov/segmentation_keras/blob/master/utils/image_reader.py#L79

ahundt · 2017-05-06T00:41:00Z

keras/preprocessing/image.py

+            x = img_to_array(img, dim_ordering=self.dim_ordering, color_mode=self.color_mode)
+            # Allows shape to be None, but batch size must equal one.
+            if not self.image_shape[1] and i == 0:
+                if current_batch_size != 1:


Why must batch size be one? Just looking for clarification because I'm not 100% sure what this case is checking for because dense prediction tasks including FCNs can train with batch sizes >1.

ahundt · 2017-05-06T00:46:26Z

tests/keras/preprocessing/test_image.py

@@ -160,6 +160,33 @@ def test_directory_iterator(self):
        assert(sorted(dir_iterator.filenames) == sorted(filenames))
        shutil.rmtree(tmp_folder)

+    ''' Test for the directory iterator when we wish to use it for a FCN.'''
+    def test_directory_iterator_fcn(self):


I suggest also checking if labels can be loaded from pngs without palette conversion as described in the following, perhaps an additonal color mode 'segmentation_label' would be appropriate:
nicolov/segmentation_keras#14

Though perhaps something like that is out of scope for this pull request.

ahundt · 2017-05-06T01:44:28Z

@fchollet sure I can give organizing that project a shot, though it may depend on how busy my research gets this summer. The bullets from my previous comment are a start. Where should this discussion continue so this pull request doesn't need to be hijacked any more than I've already done?

fchollet · 2017-05-06T02:53:42Z

@ahundt please open an issue with the list of changes you would like to see, and we will create a 'request for contributions' entry that will link to the issue. When you have an API proposal ready for review, please ping me.

joeyearsley added 4 commits January 30, 2017 17:15

Update image.py

31e8ce9

Update test_image.py

3d34162

Update test_image.py

86826ce

PEP8 :|

joeyearsley changed the title ~~Altered to allow for FCN's~~ Bug Fix in Image Generator - Now Allows FCN's to be used Jan 31, 2017

fchollet reviewed Jan 31, 2017

View reviewed changes

joeyearsley added 2 commits January 31, 2017 23:40

Update image.py

d598805

Added docstrings

Update image.py

a0ab1e8

fchollet reviewed May 5, 2017

View reviewed changes

ahundt suggested changes May 6, 2017

View reviewed changes

ahundt mentioned this pull request May 8, 2017

Dense Prediction API Design, Including Segmentation and Fully Convolutional Networks #6538

Closed

fchollet closed this Jun 10, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bug Fix in Image Generator - Now Allows FCN's to be used #5228

Bug Fix in Image Generator - Now Allows FCN's to be used #5228

joeyearsley commented Jan 30, 2017

fchollet Jan 31, 2017

fchollet Jan 31, 2017

joeyearsley Feb 2, 2017

ahundt commented Mar 28, 2017

fchollet left a comment

fchollet May 5, 2017

fchollet May 5, 2017

ahundt commented May 5, 2017 •

edited

Loading

fchollet commented May 5, 2017

ahundt left a comment

ahundt May 6, 2017

ahundt May 6, 2017

ahundt May 6, 2017

ahundt commented May 6, 2017

fchollet commented May 6, 2017

Bug Fix in Image Generator - Now Allows FCN's to be used #5228

Bug Fix in Image Generator - Now Allows FCN's to be used #5228

Conversation

joeyearsley commented Jan 30, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ahundt commented Mar 28, 2017

fchollet left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ahundt commented May 5, 2017 • edited Loading

fchollet commented May 5, 2017

ahundt left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ahundt commented May 6, 2017

fchollet commented May 6, 2017

ahundt commented May 5, 2017 •

edited

Loading