[Python] Support for untyped Json

[andrueastman]

Similar to #4095 we need to add support for untyped content in request/response payloads.

In summary, we need to

• Add UntypedNode base type in the abstractions library (see Create types for untyped nodes kiota-dotnet#177)
• Add derived types of UntypedNode to represent object, array and primitive types in the abstractions (see Create types for untyped nodes kiota-dotnet#177)
• Update the getObjectValue method in the json serialization library to handle scenarios of untyped content (see Support untyped nodes kiota-serialization-json-dotnet#198)
• Remove the call to RemoveUntypedNodePropertyValues in the refiner so that untyped properties are not stripped out.
• Add tests in the generator.
• Ensure the "apisguru::apis.guru" integration test is passing.

[rohitkrsoni]

Hi @andrueastman, I would like to work on this issue, could you please let me know starting point so that I can start my investigation, thanks

[andrueastman]

Thanks for reaching out @rohitkrsoni

To start with you'd need to create the types that have a GetValue method in the abstraction library at https://github.com/microsoft/kiota-abstractions-python.
(This is the first two points in the issue description with link to C# implementation in C#)
The types would be

• UntypedNode (base type that implements IParsable and has a GetValue method)
• UntypedObject - for object types
• UnytypedArray - for collection types
• UntypedBoolean - for bool
• UntypedNull - for null
• UntypedString - for string
• UntypedLong/UntypedDouble/UntypedFloat/UntypedInteger - for number types. If python supports a single type for numbers this can just be one type for UntypedNumber.

The next thing would be to update the serialization library to support serializing/deserializing these types in the getObjectValue and writeObjectValue methods. https://github.com/microsoft/kiota-serialization-json-python (third task in description)

Once these two are done we just need to remove this line in the python refiner so that the types show up in generation. (task 4)

kiota/src/Kiota.Builder/Refiners/PythonRefiner.cs

Line 29 in 101d528

RemoveUntypedNodePropertyValues(generatedCode);

We can then add tests and validate that the sdk generated for the endpoint "apisguru::apis.guru" works without issue and we can read the data.

Let us know if you need any other clarifications.

[rohitkrsoni]

Thanks for the above explanation @andrueastman, I think I have got an idea about what I need to do. But I am looking in why we need to do. As far as I have understood we are trying to handle the untyped JSON node. More specifically, those parameter which are not defined in the schema but they are in the response when API is hit. So these nodes should be handled and parsed during the runtime. I can see in the generated class that we do have additional_data Object and this should be handling those "extra" parameters/properties. I see that we are using AdditionalDataHolder, Parsable, ParseNode, SerializationWriter from kiota_abstractions.serialization to parse the objects.
So, my question is if we are handling these nodes, why do we need this change? Is there any repo which have got API Client generated for such APIs which sends additional data and we are handling the types at the runtime. I just want to see the end to end process how things are happening currently and how we will be handling after this change. Hope I am able to explain my question well, thanks!

[rohitkrsoni]

Hi @andrueastman, could you please provide your insights here, thanks

[andrueastman]

The existence of the additionaldata is only limited to object types that can hold the additionaldata as a property inside it. This would then hold the property name as the key and the property value as the value.

However, in the event an Api response is not described (response schema is undefined), this mean that the API would not return an object (property names and their values), so the return type can't be an object if it's a primitive or a collection as a dictionary couldn't be used to represent the values and we would need to generate a return type value.

The scenario also applies to scenarios where a property could return multi-dimensional arrays which would complicate the serializer/deserialization logic as well as the number of dimensions would need to be known beforehand for the serializer serialize effectively. See example at #4549

Let me know if this helps clarifies this abit more.

[rohitkrsoni]

Thanks for the additional info @andrueastman,
Now, things are a bit more clear.

I have created a basic SampleAPI which returns id and name, with swagger.json
Repo Link

I have also created a SampleAPIClient to get the response after generating the client in the generated folder
Repo Link

I am trying to log the response to the console with the below code:

var authProvider = new AnonymousAuthenticationProvider();
var requestAdapter = new HttpClientRequestAdapter(authProvider);

var client = new GraphClient(requestAdapter);
var x = await client.GetAsGetResponseAsync();
Console.WriteLine($"Id: {x.Id}");
Console.WriteLine($"Name: {x.Name}");

I am getting the value of Id correct, but value of Name getting printed is Microsoft.Kiota.Abstractions.Serialization.UntypedString

And while debugging I can see the correct value of in non Non-Public Members of Name i.e. _value = John Doe

Am I doing something wrong here? Meanwhile I will continue to look into it. Once it is figured out and I get the value of the Untyped Node for C#, I will start implementation of Support of Python.

Edit 1:

When I use x.Name.GetValue(), I am getting System.NotImplementedException: 'The method or operation is not implemented.' Error during runtime.

And the method GetValue() is referring to GetValue() method of namespace Microsoft.Kiota.Abstractions.Serialization, class UntypedNode, which is the base class and the method is not implemented. Ideally it should refer to UntypedString.

Edit 2:

Okay, so I am getting the value of name with this piece of extra code:

var serializedValue = await KiotaJsonSerializer.SerializeAsStringAsync(x.Name);
var nameObjectElement = JsonDocument.Parse(serializedValue);
var y = nameObjectElement.Deserialize<String>();
Console.WriteLine($"Name: {y}");

My understanding is, we need to serialize the UntypedNode as String and then parse it to JsonDocumentObjectElement and finally Deserialize the result to our desired output?

But, while reserializing it, we are also passing the type which we want to deserialize to String :
var y = nameObjectElement.Deserialize<String>();

But how will the project implementing the generated Client will know the type of the UntypedNode. Here I know that Name is of type String and I have deserialized it to correct form but let's say API returns Integer instead of String during runtime. How we are handling this? Am I missing something in the implementation?

[andrueastman]

You can checkout a sample at here

Essentially, since you don't know the type, you'lll need to call the GetValue method once the type is matched.

[rohitkrsoni]

Here, is the PR of added untyped nodes. microsoft/kiota-python#286, please review, thanks

[rohitkrsoni]

Serialization Support for UntypedNodes: microsoft/kiota-python#356

[kaakaa]

I appreciate that you are doing towards supportingUntypedNode, but are there any workarounds to avoid NameError: name 'UntypedNode' is not defined other than editing OAS file?

[baywet]

Unfortunately, not as far as I know. We could imagine creating this class as a placeholder to unblock people while waiting on the implementation work to be completed if that helps.

[kaakaa]

Thanks @baywet . Currently, the generated source does not even have an import statement for UntypedNode, so I feel that the existence of a UntypedNode class, even as a placeholder, is easier to deal with this issue.

[baywet]

we could add the imports once we have a placeholder as well.
is this something you'd like to submit a pull request for provided some guidance?

[kaakaa]

Thanks @baywet . I'm newbie at Python, so I'll take a look into this at first.