Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

iop-order: fix lut3d order (after colorin). #3075

Merged
merged 3 commits into from
Oct 6, 2019

Conversation

TurboGit
Copy link
Member

@TurboGit TurboGit commented Oct 4, 2019

For #3062.

Another iop-order bug!

@aurelienpierre : I'd like to review the iop-order after this patch to ensure we are ok to avoid multiplying the iop-order version (already 5 of them just for a dev version :).

  db:  1,00000000000   xmp   1,0000         rawprepare
  db:  2,00000000000   xmp   2,0000             invert
  db:  3,00000000000   xmp   3,0000        temperature
  db:  4,00000000000   xmp   4,0000         highlights
  db:  5,00000000000   xmp   5,0000          cacorrect
  db:  6,00000000000   xmp   6,0000          hotpixels
  db:  7,00000000000   xmp   7,0000         rawdenoise
  db:  8,00000000000   xmp   8,0000           demosaic
  db:  9,00000000000   xmp   9,0000     denoiseprofile
  db: 10,00000000000   xmp  10,0000          bilateral
  db: 11,00000000000   xmp  11,0000       rotatepixels
  db: 12,00000000000   xmp  12,0000        scalepixels
  db: 13,00000000000   xmp  13,0000               lens
  db: 14,00000000000   xmp  14,0000        hazeremoval
  db: 15,00000000000   xmp  15,0000             ashift
  db: 16,00000000000   xmp  16,0000               flip
  db: 17,00000000000   xmp  17,0000           clipping
  db: 18,00000000000   xmp  18,0000            liquify
  db: 19,00000000000   xmp  19,0000              spots
  db: 20,00000000000   xmp  20,0000            retouch
  db: 21,00000000000   xmp  21,0000           exposure
  db: 22,00000000000   xmp  22,0000       mask_manager
  db: 23,00000000000   xmp  23,0000            tonemap
  db: 24,00000000000   xmp  24,0000          toneequal
  db: 25,00000000000   xmp  25,0000        graduatednd
  db: 26,00000000000   xmp  26,0000      profile_gamma
  db: 27,00000000000   xmp  27,0000          equalizer
  db: 28,00000000000   xmp  28,0000            colorin
  db: 29,00000000000   xmp  29,0000              lut3d
  db: 30,00000000000   xmp  30,0000            nlmeans
  db: 31,00000000000   xmp  31,0000           defringe
  db: 32,00000000000   xmp  32,0000             atrous
  db: 33,00000000000   xmp  33,0000            lowpass
  db: 34,00000000000   xmp  34,0000           highpass
  db: 35,00000000000   xmp  35,0000            sharpen
  db: 36,00000000000   xmp  36,0000       channelmixer
  db: 37,00000000000   xmp  37,0000       colorchecker
  db: 38,00000000000   xmp  38,0000      colortransfer
  db: 39,00000000000   xmp  39,0000       colormapping
  db: 40,00000000000   xmp  40,0000       colorbalance
  db: 41,00000000000   xmp  41,0000           basicadj
  db: 42,00000000000   xmp  42,0000           rgbcurve
  db: 43,00000000000   xmp  43,0000          rgblevels
  db: 44,00000000000   xmp  44,0000              bloom
  db: 45,00000000000   xmp  45,0000          basecurve
  db: 46,00000000000   xmp  46,0000             filmic
  db: 47,00000000000   xmp  47,0000          filmicrgb
  db: 48,00000000000   xmp  48,0000             colisa
  db: 49,00000000000   xmp  49,0000          tonecurve
  db: 50,00000000000   xmp  50,0000             levels
  db: 51,00000000000   xmp  51,0000             shadhi
  db: 52,00000000000   xmp  52,0000              bilat
  db: 53,00000000000   xmp  53,0000    colorcorrection
  db: 54,00000000000   xmp  54,0000         colorzones
  db: 55,00000000000   xmp  55,0000           vibrance
  db: 56,00000000000   xmp  56,0000             velvia
  db: 57,00000000000   xmp  57,0000           colorize
  db: 58,00000000000   xmp  58,0000      colorcontrast
  db: 59,00000000000   xmp  59,0000      globaltonemap
  db: 60,00000000000   xmp  60,0000           lowlight
  db: 61,00000000000   xmp  61,0000         monochrome
  db: 62,00000000000   xmp  62,0000         zonesystem
  db: 63,00000000000   xmp  63,0000            relight
  db: 64,00000000000   xmp  64,0000              grain
  db: 65,00000000000   xmp  65,0000             soften
  db: 66,00000000000   xmp  66,0000        splittoning
  db: 67,00000000000   xmp  67,0000           vignette
  db: 68,00000000000   xmp  68,0000   colorreconstruct
  db: 69,00000000000   xmp  69,0000           colorout
  db: 70,00000000000   xmp  70,0000              clahe
  db: 71,00000000000   xmp  71,0000         finalscale
  db: 72,00000000000   xmp  72,0000        overexposed
  db: 73,00000000000   xmp  73,0000     rawoverexposed
  db: 74,00000000000   xmp  74,0000             dither
  db: 75,00000000000   xmp  75,0000            borders
  db: 76,00000000000   xmp  76,0000          watermark
  db: 77,00000000000   xmp  77,0000              gamma

@TurboGit TurboGit added the bugfix pull request fixing a bug label Oct 4, 2019
@TurboGit TurboGit added this to the 2.8 milestone Oct 4, 2019
@TurboGit TurboGit self-assigned this Oct 4, 2019
@TurboGit
Copy link
Member Author

TurboGit commented Oct 4, 2019

And I have checked that the only changes with this patch are the swap for:

from:

  db: 28,00000000000   xmp  28,0000              lut3d
  db: 29,00000000000   xmp  29,0000            colorin

to:

  db: 28,00000000000   xmp  28,0000            colorin
  db: 29,00000000000   xmp  29,0000              lut3d

@jenshannoschwalm
Copy link
Collaborator

On-the-fly v3->v(master) :-) ???

@TurboGit
Copy link
Member Author

TurboGit commented Oct 4, 2019

Yes, no way around that :)

@jenshannoschwalm
Copy link
Collaborator

jenshannoschwalm commented Oct 4, 2019

Also v4->v(master) ?

@TurboGit
Copy link
Member Author

TurboGit commented Oct 4, 2019

@aurelienpierre : new order proposed is:

dt_ioppr_set_default_iop_order 
  db:  1,00000000000   xmp   1,0000         rawprepare
  db:  2,00000000000   xmp   2,0000             invert
  db:  3,00000000000   xmp   3,0000        temperature
  db:  4,00000000000   xmp   4,0000         highlights
  db:  5,00000000000   xmp   5,0000          cacorrect
  db:  6,00000000000   xmp   6,0000          hotpixels
  db:  7,00000000000   xmp   7,0000         rawdenoise
  db:  8,00000000000   xmp   8,0000           demosaic
  db:  9,00000000000   xmp   9,0000     denoiseprofile
  db: 10,00000000000   xmp  10,0000          bilateral
  db: 11,00000000000   xmp  11,0000       rotatepixels
  db: 12,00000000000   xmp  12,0000        scalepixels
  db: 13,00000000000   xmp  13,0000               lens
  db: 14,00000000000   xmp  14,0000        hazeremoval
  db: 15,00000000000   xmp  15,0000             ashift
  db: 16,00000000000   xmp  16,0000               flip
  db: 17,00000000000   xmp  17,0000           clipping
  db: 18,00000000000   xmp  18,0000            liquify
  db: 19,00000000000   xmp  19,0000              spots
  db: 20,00000000000   xmp  20,0000            retouch
  db: 21,00000000000   xmp  21,0000           exposure
  db: 22,00000000000   xmp  22,0000       mask_manager
  db: 23,00000000000   xmp  23,0000            tonemap
  db: 24,00000000000   xmp  24,0000          toneequal
  db: 25,00000000000   xmp  25,0000        graduatednd
  db: 26,00000000000   xmp  26,0000      profile_gamma
  db: 27,00000000000   xmp  27,0000          equalizer
  db: 28,00000000000   xmp  28,0000            colorin
  db: 29,00000000000   xmp  29,0000            nlmeans
  db: 30,00000000000   xmp  30,0000           defringe
  db: 31,00000000000   xmp  31,0000             atrous
  db: 32,00000000000   xmp  32,0000            lowpass
  db: 33,00000000000   xmp  33,0000           highpass
  db: 34,00000000000   xmp  34,0000            sharpen
  db: 35,00000000000   xmp  35,0000       channelmixer
  db: 36,00000000000   xmp  36,0000       colorchecker
  db: 37,00000000000   xmp  37,0000              lut3d
  db: 38,00000000000   xmp  38,0000      colortransfer
  db: 39,00000000000   xmp  39,0000       colormapping
  db: 40,00000000000   xmp  40,0000       colorbalance
  db: 41,00000000000   xmp  41,0000           basicadj
  db: 42,00000000000   xmp  42,0000           rgbcurve
  db: 43,00000000000   xmp  43,0000          rgblevels
  db: 44,00000000000   xmp  44,0000              bloom
  db: 45,00000000000   xmp  45,0000          basecurve
  db: 46,00000000000   xmp  46,0000             filmic
  db: 47,00000000000   xmp  47,0000          filmicrgb
  db: 48,00000000000   xmp  48,0000             colisa
  db: 49,00000000000   xmp  49,0000          tonecurve
  db: 50,00000000000   xmp  50,0000             levels
  db: 51,00000000000   xmp  51,0000             shadhi
  db: 52,00000000000   xmp  52,0000              bilat
  db: 53,00000000000   xmp  53,0000    colorcorrection
  db: 54,00000000000   xmp  54,0000         colorzones
  db: 55,00000000000   xmp  55,0000           vibrance
  db: 56,00000000000   xmp  56,0000             velvia
  db: 57,00000000000   xmp  57,0000           colorize
  db: 58,00000000000   xmp  58,0000      colorcontrast
  db: 59,00000000000   xmp  59,0000      globaltonemap
  db: 60,00000000000   xmp  60,0000           lowlight
  db: 61,00000000000   xmp  61,0000         monochrome
  db: 62,00000000000   xmp  62,0000         zonesystem
  db: 63,00000000000   xmp  63,0000            relight
  db: 64,00000000000   xmp  64,0000              grain
  db: 65,00000000000   xmp  65,0000             soften
  db: 66,00000000000   xmp  66,0000        splittoning
  db: 67,00000000000   xmp  67,0000           vignette
  db: 68,00000000000   xmp  68,0000   colorreconstruct
  db: 69,00000000000   xmp  69,0000           colorout
  db: 70,00000000000   xmp  70,0000              clahe
  db: 71,00000000000   xmp  71,0000         finalscale
  db: 72,00000000000   xmp  72,0000        overexposed
  db: 73,00000000000   xmp  73,0000     rawoverexposed
  db: 74,00000000000   xmp  74,0000             dither
  db: 75,00000000000   xmp  75,0000            borders
  db: 76,00000000000   xmp  76,0000          watermark
  db: 77,00000000000   xmp  77,0000              gamma

@TurboGit
Copy link
Member Author

TurboGit commented Oct 4, 2019

Well v4 -> (master) ... not sure this is just a minor issue. No crash and how many people have been using v4 ?

@aurelienpierre
Copy link
Member

aurelienpierre commented Oct 4, 2019

It's probably safer to put tonemap after toneequalizer, although these two don't really fit in the same workflow anyway, so I'm not sure it's important to bother.

Then, from colorin, the order should be:

  • colorin
  • nlmeans // signal processing (denoising) -> needs a signal as scene-referred as possible (even if it works in Lab)
  • colorcheckr // calibration to "neutral" exchange colour space -> improve colour calibration of colorin and reproductibility of further edits (styles etc.)
  • defringe // desaturate fringes in Lab, so needs properly calibrated colours in order for chromaticity to be meaningful,
  • atrous // frequential operation, needs a signal as scene-referred as possible to avoid halos
  • lowpass // same, mostly useless anyway since it overlaps shadhi use-case
  • highpass // same, mostly useless anyway since it overlaps sharpen use-case
  • sharpen // same, worst than atrous in same use-case, less control overall
  • lut3d // apply a creative style or film emulation, possibly non-linear, so better move it after frequential ops that need L2 Hilbert spaces of square summable functions
  • colortransfer / colormapping // probably better if source and destination colours are neutralized in the same colour exchange space, hence after colorin and colorcheckr, but apply after frequential ops in case it does non-linear witchcraft, just to be safe
  • channelmixer // does exactly the same thing as colorin, aka RGB to RGB matrix conversion, but coefs are user-defined instead of calibrated and read from ICC profile. Really versatile yet under-used module, doing linear ops, very good in scene-referred workflow
  • basicadj // WTF module mixing view/model/control at once, usage should be discouraged
  • colorbalance // scene-referred color manipulation
  • rgbcurve // really versatile way to edit colour in scene-referred and display-referred workflow
  • rgblevels // same
  • basecurve // conversion from scene-referred to display referred, reverse-engineered on camera JPEG default look
  • filmic // same, but different (parametric) approach
  • filmicrgb // same, upgraded
  • colisa // edit contrast while damaging colour
  • tonecurve // same
  • levels // same
  • shadhi // same
  • zonesystem // same
  • globaltonemap // same
  • relight // flatten local contrast while pretending do add lightness -- we really have too many modules doing the exact same thing
  • bilat // improve clarity/local contrast after all the bad things we have done to it with tonemapping
  • colorcorrection // now that the colours have been fucked by contrast manipulations, try to recover them - global adjustment of white balance for shadows and highlights
  • colorcontrast // adjust chrominance globally
  • velvia // same
  • vibrance // same, but more subtle
  • colorzones // same, but locally
  • all the silly creative modules: grain, soften, lowlight, splittoning, colorize, vignette… at that point colour doesn't mean anything, so we don't care.
  • colorreconstruct // try to salvage blown areas before ICC intents in LittleCMS2 do silly things with them.
  • colorout // end of journey

@jenshannoschwalm
Copy link
Collaborator

@aurelienpierre I couldn't find such a description about the rationale of a good/suggested pipeline order (with user readable module names) in the manual yet? Something like that should be in a section about iop reordering.

Or maybe we ask some people about their personal workflow and could have that as an introduction?

@aurelienpierre
Copy link
Member

aurelienpierre commented Oct 4, 2019

Most people's workflow is just an accidental habit, better not ask them. A rational pipeline is based on the meaning of the operations we do, past the maths of it. For example, the beginning of the pipeline is pure signal processing, where we revert by calculation the flaws left by the sensor and the optical media in the opposite order they are applied in reality. See #2905 for further explainations. At that point, you have recovered the latent image, aka the theoriticaly perfect original image, exempt of any optical artifact.

Then, we aim at matching whatever sensor RGB space (that is not colour, RGB doesn't mean colour, it means some spectrum got split into some tristimulus) to sensible, human-defined colours, through the input colour profile and the colour checkr modules. After that point only, RGB means colour in our perceptual reference (but is still scene-linear encoded).

After that come the frequential operations, high-pass, low-pass and band-pass filters, which theory is grounded in Fourier's spectral analysis. They aim at affecting local or global contrast without changing the average lightness. You can see them as a modulation around a neutral value that is left unchanged.

Then come the scene-referred colour manipulations, that aim at adjusting colours as if we were physically playing with light emissions. The huge benefit of these operations is the are using physically-accurate models, so they can tolerate dramatic edits with no weird stuff happening. It's really like having a virtual camera in-computer allowing you to redo the shot a posteriori.

Then comes the camera transform, that does something similar to what film and brain do the scene-referred tristimulus : raise midtones, add contrast. Either filmic or basecurve in darktable. At that point, your signal becomes display-referred, meaning you have lost the physical relationship with light and whatever operation you might perform after might blow up in your face. Here come most of darktable 2.6 operations, using Lab colour space, and working not so well for dramatic edits (witness colour shifts and saturation issues while pushing lightness or contrast). This is a legacy workflow inherited from the 1990's where dynamic range was not an issue, because it was pretty low.

You may find that series of posts useful too: https://medium.com/the-hitchhikers-guide-to-digital-colour

I would be grateful if someone else had time to make a nice summary of all this info for dt's manual, because my nights are quite short already.

TurboGit added a commit to TurboGit/darktable that referenced this pull request Oct 4, 2019
Take the opportunity to set a clean state as discussed on darktable-org#3075
with Aurélien Pierre.

For darktable-org#3062.
@TurboGit
Copy link
Member Author

TurboGit commented Oct 4, 2019

Ok, I have update the iop-order to v5 with a clean flat list has proposed by Aurélien (@aurelienpierre thanks a lot again!).

I do hope that v5 is the definite iop-order for next release.

As hinted by @jenshannoschwalm I have also enabled v3->v5 conversion to ensure no more v4 are remaining on earth!

@aurelienpierre : can you double check I have not messed up the order ?
@jenshannoschwalm : a quick review will also make me more confident before merging.

@@ -1433,11 +1558,11 @@ int dt_ioppr_convert_onthefly(const int imgid)
// already latest
if (my_iop_order_version == DT_IOP_ORDER_VERSION) return my_iop_order_version;

// ??? we handle only iop-version 3 (which has been broken) and move
// it to new v4. this routine will be reused later when dt will
// ??? we handle only iop-version 3/4 (which has been broken) and move
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

iop-version 3/4 (which has been broken) --> (which have been broken)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed.

@jenshannoschwalm
Copy link
Collaborator

Yes go ahead. (5 times finger crossing makes ten, not many left ...)

Take the opportunity to set a clean state as discussed on darktable-org#3075
with Aurélien Pierre.

For darktable-org#3062.
@TurboGit
Copy link
Member Author

TurboGit commented Oct 4, 2019

Yes go ahead. (5 times finger crossing makes ten, not many left ...)

Thanks! I'll wait for Aurélien review to be sure all is good and avoid introducing a v6 :)

@jenshannoschwalm
Copy link
Collaborator

From #3071

it looks like we need an auto migration from v1 -> v2 as v2 is only v1 plus the new modules.

Yes.

BTW i also found 0 as the iop_version in my images database. ?? Didn't know about that.

If i develop such an image in darkroom and add a new module like filmicrgb, the iop_order version is changed in the database.
I think that is correct because the iop_orders of the original are modified but kept in the same order as the original had it.

This part of the code is heavy stuff. I have done lots of real time audio processing with live reordering of filters but this is driving mad. Keep going!

There is one more thing i am suspicious about. Copy/merge/paste of history. There might be more problems ... i will look into that tomorrow and ping you about surprises.

@TurboGit
Copy link
Member Author

TurboGit commented Oct 5, 2019

BTW i also found 0 as the iop_version in my images database. ?? Didn't know about that.

0 means not yet developed, just discard history from lighttablle and look at the database, iop_order_version should be 0. Such mage when edited will get the latest iop_order version.

@jenshannoschwalm
Copy link
Collaborator

In general, any sort of 'history insertion' from one image to another of does not take care of different iop_order versions. It could be done but would mean endless changes in all parts of the history stack prone to introducing new bugs i guess.

One suggestion, after looking at the onthefly code again, i think it is factored in a wrong way for re-usability. The test for version like
if (my_iop_order_version != 3) return my_iop_order_version;
should probably not be here. Testing should be done in dt_dev_read_history_ext

@TurboGit
Copy link
Member Author

TurboGit commented Oct 5, 2019

@jenshannoschwalm : just saw your comment, look at my new implementation it should be cleaner and handles more cases v1 -> v2 and v3-4 -> v5.

I hope we are seeing the light at the end of the tunnel :)

@jenshannoschwalm
Copy link
Collaborator

@TurboGit will do a thorough look later today - currently at work.

BTW if we are here, i played around with iop rordering to do a lot of tests. Mostly fine.
But - i am surprised you are allowed to move lens correction and clipping in the wrong order. That gives really strange results and should never happen. Any hard fence?

@aurelienpierre
Copy link
Member

I will review that later today too.

@TurboGit
Copy link
Member Author

TurboGit commented Oct 5, 2019

Any hard fence?

We probably need more. But at this stage I'd like to avoid that to not introduce more regressions. This is a delicate area (fences) and I do not master this part of code yet. So I want to be conservative.

@jenshannoschwalm
Copy link
Collaborator

About fences. Yes we need more later.

But as there are many issues and discussions about a modified iop_order.

If i try to think of module reordering in an absolutely non-technical way a user might think the layout is only a layout change and not a reordering of the pipeline which can give at least strange results.

Could we add a button to reorder the pipeline to default or did i miss that?

@TurboGit
Copy link
Member Author

TurboGit commented Oct 5, 2019

Could we add a button to reorder the pipeline to default or did i miss that?

No, you did not miss that there is none. But that was my idea when I said that later we will use the code you've implemented to migrate to last IOP order version. That means also that we will somehow revert to latest has changed the order manually. All this for after 3.0 anyway.

// returns the history version of imgid
int dt_ioppr_convert_onthefly(const int imgid)
// migrate to another iop_order version the given image
// not that this is actually a non exported routine but it will
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. migrate the given image to another iop_order version
  2. note that this is :-) First i misunderstood

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed.

// it to new v5. this routine will be reused later when dt will
// propose in GUI a possibility to migrate old edits to a new
// version of iop-order.
return _ioppr_migrate_iop_order(imgid, my_iop_order_version, DT_IOP_ORDER_VERSION);
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably DT_IOP_ORDER_VERSION is right here instead of 5. Maybe leave a comment ?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

@jenshannoschwalm
Copy link
Collaborator

That refactoring seems to be perfect.

This is needed as new modules (filmic rgb, levels rgb, ...) are only
known starting with v2 which is the first iop-order version supported.
The v1 is a snapshot of the original static order hardcoded into dt.

Fixes darktable-org#3071.
@TurboGit
Copy link
Member Author

TurboGit commented Oct 5, 2019

Thanks for the review!

@jenshannoschwalm
Copy link
Collaborator

btw, there is one superflucious undef at the end of file.

I took the last hours running master with this pr included and tested a lot of images having log on.

The migrations all went fine!

I had one dt crash, that was a v2 history with wrong iop_orders. I don't bother. Some thousands worked just fine.

There is one thing i observed here, some v2 files have these warnings.

[_ioppr_check_rules] found rule flip clipping module clipping (17,000000) is before flip (20,000000) image 0 (dt_dev_pop_history_items_ext end)
[_ioppr_check_rules] found rule flip clipping module flip (20,000000) is after clipping (17,000000) image 0 (dt_dev_pop_history_items_ext end)
[_ioppr_check_rules] found rule flip clipping module clipping (17,000000) is before flip (20,000000) image 29876 (dt_styles_apply_to_image 2)
[_ioppr_check_rules] found rule flip clipping module flip (20,000000) is after clipping (17,000000) image 29876 (dt_styles_apply_to_image 2)

  1. i know for sure all there pictures have been rotated before any other development from lightroom. Don't understand.
  2. How is image 0 creeping in here?

Copy link
Member

@aurelienpierre aurelienpierre left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IOP order is good to me. Not sure I understand the dt_ioppr_convert_onthefly, I trust you on this one.

@TurboGit
Copy link
Member Author

TurboGit commented Oct 6, 2019

Ok, let's merge this now to avoid issues propagating more. Thanks all for review!

@TurboGit TurboGit merged commit 4934bbf into darktable-org:master Oct 6, 2019
@TurboGit TurboGit deleted the lut3d-iop-order branch October 8, 2019 11:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bugfix pull request fixing a bug
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants