operator T() considered harmful

[theodelrieu]

operator T() considered harmful

There's two subjects of importance that I'd like to tackle.
This is the first one, I'll open another issue quickly (hopefully).

The problem

There's been several issues in the past related to the implicit conversion operator that the library provides.
The current Readme demonstrates the following use-case:

// conversion: json -> person
ns::person p2 = j;

This will call json::operator T() with T = ns::person, so no problems here.

You'd expect the following code to always work too:

ns::person p2;
p2 = j;

And it works! Sometimes.

If ns::person has a user-defined operator=, it will break with the following message:

t.cpp: In function ‘int main(int, const char**)’:
t.cpp:14:7: error: ambiguous overload for ‘operator=’ (operand types are ‘ns::person’ and ‘nlohmann::json {aka nlohmann::basic_json<>}’)
   p2 = j;
       ^
t.cpp:6:6: note: candidate: ns::person& ns::person::operator=(int)
   ns::person& operator=(int) { return *this; }
      ^~~~~~~~
t.cpp:3:8: note: candidate: constexpr ns::person& ns::person::operator=(const ns::person&)
 struct ns::person
        ^
t.cpp:3:8: note: candidate: constexpr ns::person& ns::person::operator=(ns::person&&)

Hard to understand that error, and it's not something that can be fixed by the library.
It's triggered by the compiler before any template machinery on our side.

Now with such code:

void need_precise_measurement(Milliseconds);

// later on
json j = json::parse(config);
need_precise_measurement(j);

It can fail in two cases:

1. Someone adds an overload for need_precise_measurement. Compiler error.
2. Someone changes the type of the argument to Nanoseconds. Runtime error at best.

Implicit conversions

Implicit conversions make the library pleasant to use. Especially when implicitely converting to JSON.
That's because we have a templated non-explicit constructor with lots of constraints to correctly handle user types.
There's also a single type to convert to: nlohmann::json, so everything works.

However, when converting implicitely from json, it's the complete opposite. If the compiler cannot decide which type it should implicit convert the nlohmann::json value to, you get a compiler error.

And it could happen in a codebase months after writing the code, (e.g. when adding an operator= to a type).

In the end the only consistent way to convert from a json value is to call json.get<T>.

Proposed change

I propose to simply and completely remove operator T().

It will obviously break code, so it can only be done in a major version.

Of course this change cannot be adopted right away, so I propose to add a deprecation warning inside operator T(), as we did for basic_json::basic_json(std::istream&) overloads.

Then it would be reasonable to remove it in a future major version.

[nlohmann]

Thanks for opening a separate issue and separating this from #951.

I think I understand the problem, and this really may be something to tackle with 4.0.0. The question is whether there could be some middle ground, for instance a "whitelist" of supported types for which we would still have an implicit conversion. Or an extension point to let users add their desired implicit conversions. Or ...

[theodelrieu]

First thing I don't like about having a whitelist is that it prevents consistency. Everyone has to remember which types are supported and nobody will.

Secondly, if types that we support get a new operator=: Ka-Boom.

Thirdly, it doesn't solve the second example I've shown with Milliseconds and Nanoseconds.

Now for the extension point, the best way to do that is to define another constructor or operator= to user types.

But if users want to have an extension point for std::string, that's not possible. As I said, compilation fails before we can do anything. It will fail before the extension point mechanism gets called as well.

I don't think we can have a middle-ground, but I might be wrong.

[theodelrieu]

I think the customization point way can work, I'll try to experiment this weekend.

[theodelrieu]

I had some time to experiment before lunch:

#include <iostream>
#include <string>
#include <type_traits>

template <typename T, typename = void> struct custom;

class json;

template <> struct custom<std::string> {
  static std::string convert(json const &j) { return "works"; }
};

/* Uncommenting this will break with: use of overloaded operator '=' is ambiguous
template <> struct custom<const char*> {
  static const char* convert(json const &j) { return "works"; }
};
*/


class json {
public:
  template <typename T, std::enable_if_t<sizeof(custom<T>), int> = 0>
  operator T() {
    return custom<T>::convert(*this);
  }
};

int main(int argc, char const *argv[]) {
  json j;
  std::string s;

  s = j;
  std::cout << s << std::endl;
}

But the problem remains, customization point or not, it does not solve anything...

As for the whitelist, we already have a blacklist inside operator T(), where we explicitely disable const char*, std::initializer_list<...> etc... just to make std::string work.

But it does not make std::vector, nor any user type with custom operator= work.
About a macro that would conditionally disable operator T(), I find that very dangerous. Just imagine a LibA which uses a LibB, both have json as a dependency, but not with the same #define...

If LibB implementation relies on -DJSON_DISABLE_OPERATOR_T=ON, and LibA does not, everything will break.

My conclusion: I fear that the operator T() feature is broken by design.

[match41]

I vote for the removal of the implicit converter.

This is the same reason that C++ std::string does not convert to const char* implicitly.

[scinart]

I have a case that supports such removal. Having operator ValueType() const makes the following code very confusing.

template <typename T>
void f(T x)
{
    nlohmann::json j = nlohmann::json::array();
    j.push_back(x);
    std::tuple<T> t = j;
}

int main()
{
    f<std::string>("123"); // OK
    f<int>(123);           // throws an exception
    return 0;
}

There is nothing wrong with from_json_tuple, but rather, tuple has this strange behavior:

  template<typename... _UElements, typename
       enable_if<
	  _TMC<_UElements...>::template
                _MoveConstructibleTuple<_UElements...>()
              && _TMC<_UElements...>::template
                _ImplicitlyMoveConvertibleTuple<_UElements...>()
              && (sizeof...(_Elements) >= 1),
    bool>::type=true>
    constexpr tuple(_UElements&&... __elements)
    : _Inherited(std::forward<_UElements>(__elements)...) { }

  template<typename... _UElements, typename
    enable_if<
	  _TMC<_UElements...>::template
                _MoveConstructibleTuple<_UElements...>()
              && !_TMC<_UElements...>::template
                _ImplicitlyMoveConvertibleTuple<_UElements...>()
              && (sizeof...(_Elements) >= 1),
    bool>::type=false>
    explicit constexpr tuple(_UElements&&... __elements)
: _Inherited(std::forward<_UElements>(__elements)...) {	}

So when it is int, it requires an json->int conversion by using the implicit tuple constructor.

Perhaps it would be something like json_cast<T>(...) that does the explicit conversion, and throws exception on cast error?

[maddanio]

if anyone cares: i am all for removing the implicit conversion. implicit conversions are a very common root of evil and should be avoided as much as possible.

[nlohmann]

I still think that implicit conversion is one of the central features of the library that allows to write code in a simple fashion. I do not like the way other libraries force the user to always spell out which type goes in and out. I do understand the problems that arise with implicit conversions (and we do have them as well!), but I am not yet convinced that the dangers outweigh the benefits. But since the removal would be a breaking change, we have time for a longer discussion.

[maddanio]

Hmm. Not convinced. You can always save mentioning the type by means. For example:

auto a = mytype{j}.
template<typename T> from_json(const json& k, T&& t);

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[maddanio]

yikes. this just bit me in the behind. having operator T like that really opens very...interesting cans of worms.
For my approach to error reporting i extended the constructors of basic_json with an additional source_location element. but now they become ambiguous once on the (size_t, json) constructor, because json converts to anything...including source location...even though i never implemented such a conversion...

[chemphys]

I don't know if this is gonna be useful or has been mentioned anywhere else, but there is a bypass for this issue (not elegant, but fixes temporary the issue)

Let's say we have something like this:

// Try to get box
    // Default: no box (empty vector)
    std::vector<double> box;
    try {
        box= j["MBX"]["box"];
        throw 1;
    } catch(int i) {
        box.clear();
    }
    box_ = box;

As mentioned in the issue description, it won't work due to the T() problem. However, if it is needed to keep the structure, an ugly fix is to do:

// Try to get box
    // Default: no box (empty vector)
    std::vector<double> box;
    try {
        std::vector<double> box2 = j["MBX"]["box"];
        box = box2;
        throw 1;
    } catch(int i) {
        box.clear();
    }
    box_ = box;

I am not sure if this will be helpful to anybody, but this makes it work keeping the scopes in the right place. If is irrelevant, please feel free to delete the message.

[abrownsword]

I just got bitten by this -- a code change caused a json variable to be implicitly converted to a bool, and it would throw an exception (which was caught and handled a bit too silently). I didn't actually expect json -> bool to be implicit and had assumed the compiler would show me all the places where I had to update the code when making the code change. This is bad, and I vote for operatorT() to go the way of the dinosaur.

[Lord-Kamina]

I came for the conversions; I stayed for the drama.

(Just began using nlohmann-json on a project; subscribing to thread)

[mdealer]

Why would anyone be surprised from getting bit by C++ these days?

I'm against deprecation, but not against opt-in. I completely support the idea behind this library being clean and intuitive to use.

[AlexandreBossard]

There is nothing clean nor intuitive when you get hundreds of lines of compilation errors from failed template instantiation just because you dare to use a json library that overstep its fanciness.

Again, the problem is not this library by itself but its the interaction with a complex code base. Your day can go down the drain just because something, anything, little or less, changed and you happen to anger the laws of template resolution.

Not everyone has time nor wants to know every intricate implementation details for every dependencies they use in their decade old project and every (bad) interactions they have.

Deprecation with an opt-in or opt-out switch is fine, I don't care, but as it is, this library is a liability to my code base.

[maddanio]

but do you update regularly? if you keep json at a fixed version it should be ok. but yes, having a macro to opt-in to, or more conservatively opt out from implicit templated conversion operators would of course be fine, at least by me. and @mdealer i think its a very bad idea to just promote the perceived status quo that c++ is inherently unstable. it is actually the most stable platform in our codebase, much more controllable imho than go, python, js, php, rust (the other languages in our codebase).

[AlexandreBossard]

but do you update regularly? if you keep json at a fixed version it should be ok.

I do, be it for feature or bug fix. And again, even if I keep json to a "stable" release, there is still a possibility a code-base change will trigger an unwanted operator T shenanigan.

[nlohmann]

I understand your concerns, and I think the first step should definitely be a preprocessor switch to switch off implicit conversions. I can make a PR - is anything in addition to operator T() to be taken care of?

[theodelrieu]

@nlohmann The PR is already there: #1559, it needs a rebase though

[nlohmann]

@theodelrieu Oh, sorry, totally forgot about #1559. Can you rebase?

[maddanio]

I would also like the list initalization to be switchable. It has similat issues.

[nlohmann]

I would also like the list initalization to be switchable. It has similat issues.

What do you mean? What do you want to switch on/off?

[theodelrieu]

@nlohmann I just rebased