Incorrect parsing of complex JS nested objects in POST request

[vrfdivino]

I created a simple Express server with one POST endpoint. The POST endpoint just return the request body sent from the client. The code for this server looks like this:

const express = require('express')

const app = express()
app.use(express.urlencoded({ extended: true}))
app.use(express.json())

app.post("/", (req,res) => {
        res.status(200).json({
                status: 'ok',
                data: req.body
        })
})

app.listen(8081, () => console.log('up and running on port 8081')

I also added a script file to test the said POST endpoint using needle. This is the code for the script file:

const needle = require('needle')

const postSomething = async () => {
        const res = await needle('post', 'http://localhost:8081', {
                field1: 'hello',
                field2: [
                        'line1',
                        'line2',
                        'line3',
                ],
                field3: [
                {
                        subfield1: 'this is a string',
                        subfield2: ['this is array'],
                        subfield3: ['another array']
                },
                {
                        subfield1: 'this is a string',
                        subfield2: ['this is array'],
                        subfield3: ['another array']
                },
                {
                        subfield1: 'this is a string',
                        subfield2: ['this is array'],
                        subfield3: ['another array']
                },
                ]
        })
        console.log(res.body.data) // Note this line of code
}

postSomething()

Now, the output of console.log(res.body.data) does not match with the request body that was sent originally to the server. field3 becomes an array with single object instead of three and the subfield1 becomes an array. The output looks like this:

{
  field1: 'hello',
  field2: [ 'line1', 'line2', 'line3' ],
  field3: [ { subfield1: [Array], subfield2: [Array], subfield3: [Array] } ]
}

Now, to verify that the needle output is incorrect, I tested it in Postman. The correct output should look like this:

{
        "field1": "hello",
        "field2": [
            "line1",
            "line2",
            "line3"
        ],
        "field3": [
            {
                "subfield1": "this is a string",
                "subfield2": [
                    "this is array"
                ],
                "subfield3": [
                    "another array"
                ]
            },
            {
                "subfield1": "this is a string",
                "subfield2": [
                    "this is array"
                ],
                "subfield3": [
                    "another array"
                ]
            },
            {
                "subfield1": "this is a string",
                "subfield2": [
                    "this is array"
                ],
                "subfield3": [
                    "another array"
                ]
            }
        ]
    }

In summary, upon sending a complex nested objects in needle via POST request, it changes the structure of the original request body. To replicate the issue, this is my environment:
node --version v16.14.2
npm --version 8.5.0

{
  "name": "needle-json-post-error",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "keywords": [],
  "author": "",
  "license": "ISC",
  "dependencies": {
    "express": "^4.17.3",
    "needle": "^3.0.0"
  }
}

[tomas]

Are you sure the server is returning the JSON data in the correct format? Needle simply does a JSON.parse on the response data so I'm not sure if Needle's the culprit in this one.

[yetzt]

looks to me like this happens when the post data is sent in querystring format (default). use { json: true } in your needle options to send the data as json.

i would refrain from using querystring format to transfer complex objects, because to my knowledge there is no consistant standard about how to encode and decode complex objects. see for yourself:

const qs = require("querystring");

const data = {
	field1: 'hello',
	field2: [
		'line1',
		'line2',
		'line3',
	],
	field3: [
	{
		subfield1: 'this is a string',
		subfield2: ['this is array'],
		subfield3: ['another array']
	},
	{
		subfield1: 'this is a string',
		subfield2: ['this is array'],
		subfield3: ['another array']
	},
	{
		subfield1: 'this is a string',
		subfield2: ['this is array'],
		subfield3: ['another array']
	},
	]
};

console.log(qs.decode(qs.encode(data)));

needle uses this method to encode the data, and the express.urlencoded middleware that ultimately parses the data comes from here.

[tomas]

Oh I see. I misunderstood then; I thought the problem was related the the decoding of JSON, not the encoding part. Ok I'll take a look in a while.

[tomas]

Ok, we can try and see whether adding the array item index this line makes it works like expected. The resulting encoded string would look like this (using your example data):

[yetzt]

maybe writing a test utilizing the body-parser middleware from express would be in order. keep in mind that this would fix it for one kind of backend, other backends like php might end up not parsing it in the same way.