Unexpected, intermittent (!) deserialization error

[peterldowns]

I'm seeing an intermittent failure right now where sometimes (inconsistently!) a Github Graphql API response fails to decode, resulting in an error like this:

Message: slice doesn't exist in any of 1 places to unmarshal, Locations: []

This error is thrown in the deserialization code whensomeSliceExist == false. I'm curious if it's related to the changes in #102 — a PR that has no description, no tests, and changes something suspicious. If you look at that PR, it makes a change so that when there is a slice, but it's of zero length, someSliceExist == false, whereas before this change, someSliceExist would be set to true. I'm trying to understand why I'm seeing this error intermittently, which is always a red flag that something is horribly wrong, especially considering that the API response is valid JSON and seems to match what I am deserializing into (because it succeeds intermittently!)

An example of the failure happening is when the API responds with:

{
  "data": {
    "search": {
      "pageInfo": {
        "endCursor": "Y3Vyc29yOjE=",
        "hasNextPage": false
      },
      "nodes": [
        {
          "number": 32,
          "id": "I_kwDOJBmYg86J6maD",
          "createdAt": "2024-05-23T21:03:09Z",
          "updatedAt": "2024-05-28T15:42:02Z",
          "state": "OPEN",
          "stateReason": null,
          "closedAt": null,
          "author": {
            "login": "drewAdorno"
          },
          "authorAssociation": "NONE",
          "title": "subdomains not redirecting",
          "body": "Hi,\r\nI was able to configure and successfully use localias in was wsl env (ubuntu 22.04). Everything works fine when i use my SLD (xxxx.test) but if I add my subdomain to my yaml file (admin.xxxx.test), it doesn't redirect properly\r\n\r\nIf you have any recommendations to get subdomains working, I'd greatly appreciate it\r\n\r\nDrew"
        }
      ]
    }
  }
}

and I'm trying to deserialize into a struct like this:

type queryIssues struct {
	// arguments
	repoSlug string               `json:"-"`
	ctx      context.Context      `json:"-"`
	exec     boil.ContextExecutor `json:"-"`
	// results
	Search struct {
		PageInfo PageInfo
		Nodes    []issue
	}
}

type issue struct {
	Number            int64
	ID                string
	CreatedAt         time.Time
	UpdatedAt         time.Time
	State             string
	StateReason       *string
	ClosedAt          *time.Time
	Author            struct {
		Login string
	}
	AuthorAssociation string
	Title             string
	Body              string
}

The actual call to graphql looks like this:

query := `query($cursor0: String, $searchQuery: String!) {
	search(type: ISSUE, query: $searchQuery, first: 100, after: $cursor0) {
		pageInfo {
			endCursor
			hasNextPage
		}
		nodes {
			... on Issue {
				number
				id
				createdAt
				updatedAt
				state
				stateReason
				closedAt
				author {
					login
				}
				authorAssociation
				title
				body
			}
		}
	}
}`
result := queryIssues{}
variables := map[string]any{"searchQuery": "...", "cursor0": nil}
githubClient := graphql.NewClient("/graphql", authClient).WithDebug(true)
data, err := githubClient.Exec(ctx, query, &result, variables)

Any ideas what's going on? I've worked around it for now by literally using the exact same query and struct, but using ExecRaw and decoding the JSON response myself. Doing this succeeds every time.

data, err := githubClient.ExecRaw(ctx, query, variables)
if err != nil {
    return err
}
if err := json.Unmarshal(data, &result); err != nil {
    return err
}

Finally, I'll add that I was unable to reproduce the issue ever in a testcase calling into jsonutil.UnmarshalGraphQL. Here's my testcase that always passes (which is weird, because I'd expect this to fail):

func TestExpectedToFail(t *testing.T) {
	t.Parallel()
	rawData := `{"search":{"pageInfo":{"endCursor":"Y3Vyc29yOjE=","hasNextPage":false},"nodes":[{"number":32,"id":"I_kwDOJBmYg86J6maD","createdAt":"2024-05-23T21:03:09Z","updatedAt":"2024-05-28T15:42:02Z","state":"OPEN","stateReason":null,"closedAt":null,"author":{"login":"drewAdorno"},"authorAssociation":"NONE","title":"subdomains not redirecting","body":"Hi,\r\nI was able to configure and successfully use localias in was wsl env (ubuntu 22.04). Everything works fine when i use my SLD (xxxx.test) but if I add my subdomain to my yaml file (admin.xxxx.test), it doesn't redirect properly\r\n\r\nIf you have any recommendations to get subdomains working, I'd greatly appreciate it\r\n\r\nDrew"}]}}`
	x := queryIssues{}
	err := jsonutil.UnmarshalGraphQL([]byte(rawData), &x)
	assert.Nil(t, err)
}

[hgiasac]

So the check to ensure that the slice is not empty. If not the library will panic at this line.

copied, err := copyTemplate(v.Index(0))

But you're right, someSliceExist should be outside the not-empty check.

[peterldowns]

@hgiasac ok, that makes sense, and it clarifies that my problem is probably due to something else. Thanks.

Do you have any idea why I would be getting this failure intermittently? Could it be because I'm re-using the result object? I'm iterating through responses that have nested cursors, and I've been just re-using the same result instance each time. I didn't think that it would matter, since deserialization usually totally overwrites the response struct.

[hgiasac]

Do you have any idea why I would be getting this failure intermittently? Could it be because I'm re-using the result object? I'm iterating through responses that have nested cursors, and I've been just re-using the same result instance each time. I didn't think that it would matter, since deserialization usually totally overwrites the response struct.

We may need the graphql schema on the server to know.

I reproduced your use case and didn't see anything wrong with the response example that you sent. However, can you try this commit to see if it works #141