paradox/set6 integration

[RaphaelS1]

@berndbischl @mllg @mb706 @be-marc summarising our discussions here and including action points.

Problem: Want to avoid two R6 parameter interfaces that perform a similar task (paradox and param6). param6 created by me as paradox did not work for use-cases that involved more complex sets and had significant bottlenecks.

Solution: Change R6->S3 for Param object in paradox and internally all Params wrap set6 objects. mlr packages will all use sugar functionality to make it easier to develop internal changes without breaking.

Comments: Given the size of the to-do list below and other use-cases paradox cannot currently handle, for example extracting parameters with the same IDs but ignoring prefixes (essential to distr6), I think we should consider converging in the 'other direction'. i.e. start with param6 (if needed we can change the name to paradox) and converge towards paradox, but I currently think param6 actually has more functionality so it could be quicker to change variable names and add sugar functions to param6, than it would be to change underlying objects in paradox.

If we agree that the final package is still called paradox and has at least the same amount of functionality as paradox, then it could be much quicker to start from the param6 code.

To-do (not complete):

• Remove ParamX classes, just have a single Param class - Raphael
• Convert Param from R6 to S3 - Raphael
• Merge param6 Dictionary with mlr3misc dictionary - Raphael/Michel
• Add support_dictionary to paradox - Raphael
• Remove all fields from Param which are in properties/traits of a Set - Raphael
• Redirect active bindings/methods in Param to set properties/traits - Raphael
• Move sampling and design grids to bbotk - Marc/Michel
• Update ParamSet with new Param - Raphael
• Update ParamSetCollection with new Param - Raphael
• Consider converting Condition from R6 to S3, this can copy cnd from param6 -- Note there are a few conditions in param6 not handled by paradox that are essential in other use-cases, for example checking a parameter value relative to another
• In all mlr packages, update paradox in examples, docs, etc. to sugar functions. Big job that may take a while, the more help the better. Listed package below to help keep track and suggested people to do updates (basically just maintainers so maybe outdated), feel free to re-assign:
  • mlr3proba - Raphael
  • mlr3extralearners - Raphael
  • bbotk - Marc
  • mlr3 - Michel
  • mlr3cluster - Damir
  • mlr3filters - Patrick
  • mlr3fselect - Marc
  • mlr3hyperband - Marc
  • mlr3learners - Michel
  • mlr3pipelines - Martin
  • mlr3spatiotempcv - Patrick
  • mlr3verse - Michel
  • mlrintermbo - Martin
  • DoubleML - ?
  • mlr3misc - Michel
  • mlr3viz - Michel

[mb706]

Things that seem to be missing from set6 objects that we use in paradox:

• $convert() function
• qunif() function (possibly not needed)
• tolerance for ParamDbl

also, how sure are we that param6's cnd mechanism are not constraints instead of dependencies?

[RaphaelS1]

What does convert do? If qunif is needed it can be added easily but not sure if it's appropriate in the package ? And how does tol work in ParamDbl?

Edit: Looked at code for convert and tolerance. It seems this is another point why set6 is useful. ParamDbl with tol=0 is equivalent to a set6 interval with closed bounds, whereas with tol>0 this is equivalently to an interval with open bounds (can also be half-open). So all we need is to make the tolerance amount controllable as either a global option (currently) or local parameter in the Interval object. About convert, I don't fully understand why this is needed as either an element should or shouldn't be in a set. Also I think that conversion, if it is needed for any reason, should really take place when the value is set immediately and not as a method. If needed we can add it easily.

And about cnd it's actually both. I realised that many checks in distr6 could be handled much mor easily and cleanly by using the same mechanism. So my cnd does the same as paradox it you only use value-based and not variable-based dependencies, and has same set of conditions, as well as more functionality for comparing variables. It makes it super useful for checks like "is Param A lower than Param B"

[mllg]

AFAICT qunif() is only needed for sampling and can go to bbotk as (internal) S3.

[mb706]

tol>0 this is equivalently to an interval with open bounds

the way I understand 'open bounds' in set6 is that it makes the set smaller by some small number (if you're lucky enough and your bounds are not so large that machine precision eats that number). What tol does in paradox is that it makes the acceptable range larger by some small number, so that numbers that are only off by machine imprecision are still accepted by the ParamSet. What convert then does is that it puts numbers that are outside of bounds by that small amount to exactly that bound. It furthermore converts numeric values to integer for ParamInt, as some methods (python or C based) make a distinction between these types when R otherwise generally doesn't.

[mb706]

Does anything make any use of the constraints in cnd beyond give a yes/no answer about the feasibility of a configuration point as a whole? Otherwise it could be replaced by a more general mechanism, like a function that maps the configuration space to logical(1). What dependencies are used for in paradox is for (1) removing components that are not 'active' as per the dependencies and for (2) allowing hierarchical sampling of parameter values, depending on whether their dependencies are fulfilled. AFAIUI this would not be possible with more general conditions, where you would just do rejection sampling, which a more general function would also work for?

[berndbischl]

AFAICT qunif() is only needed for sampling and can go to bbotk as (internal) S3.

even there we don't need it that much

[RaphaelS1]

the way I understand 'open bounds' in set6 is that it makes the set smaller by some small number (if you're lucky enough and your bounds are not so large that machine precision eats that number). What tol does in paradox is that it makes the acceptable range larger by some small number, so that numbers that are only off by machine imprecision are still accepted by the ParamSet. What convert then does is that it puts numbers that are outside of bounds by that small amount to exactly that bound. It furthermore converts numeric values to integer for ParamInt, as some methods (python or C based) make a distinction between these types when R otherwise generally doesn't.

Got it, we could easily add this to set6.

Does anything make any use of the constraints in cnd beyond give a yes/no answer about the feasibility of a configuration point as a whole?

I guess it asks the paradox question: can this parameter be set, as well as: if it can be set then how can it be set. And the check is only triggered if the value is non-NULL, so it has the same use-case as paradox, i.e. "let this value be set if a condition on the dependency is satisfied". This is still much faster than a general check over the full parameter set

[mb706]

much faster than a general check over the full parameter set

So speed is the only reason here? Or is cnd used somewhere in your code for something beyond making a yes / no decision about feasibility?