Add support for fetching Audit Records

[theckman]

https://developer.pagerduty.com/api-reference/b3A6Mjc0ODExNA-list-audit-records

[t-junjie]

Hi there, can I work on this issue?

[theckman]

@t-junjie that would be great, I appreciate it! I do have a few suggestions which may be helpful for when you do get to implementing it.

The first is that for new methods we're adding to the project, we're not creating new WithContext() variations. Instead, the new base method is always accepting a ctx context.Context as its first argument. So I think the method definition would be something like ListAuditRecords(ctx context.Context, o ListAuditRecordsOptions) (ListAuditRecordsResponse, error).

The second is that for these types of paginated API calls, for the future we're looking to have two separate methods. The first, ListAuditRecords, which makes a single API call and returns the response back (including pagination inputs for the next call). The other is one which handles pagination, and just returns the full set of entities back to the caller. So that would be ListAuditRecordsPaginated, and I talk about that change in this issue: #295.

I also think we should be moving to a mode where we allow consumers to provide an include function in the Paginated variants, so that they can filter out any records they don't want before the full slice is returned to them. I discuss that in this issue: #296.

So the result is I think we'd end up with two new methods:

func (c *Client) ListAuditRecords(ctx context.Context, o ListAuditRecordsOptions) (ListAuditRecordsResponse, error) {}

func (c *Client) ListAuditRecordsPaginated(ctx context.Context, o ListAuditRecordOptions, include func(AuditRecord) bool) ([]AuditRecord, error) {}

I thought this information might be more helpful to share now instead of during a PR review. Please let me know if you have any questions.

[theckman]

I forgot to include this above, here is an example of another Paginated method in the codebase. One note, it was built before I'd considered adding the include function (#296) so you'll need to add that:

go-pagerduty/service.go

Lines 156 to 185 in 2f646c3

	// ListServicesPaginated lists existing services processing paginated responses
	func (c *Client) ListServicesPaginated(ctx context.Context, o ListServiceOptions) ([]Service, error) {
	v, err := query.Values(o)
	if err != nil {
	return nil, err
	}
	var services []Service
	responseHandler := func(response *http.Response) (APIListObject, error) {
	var result ListServiceResponse
	if err := c.decodeJSON(response, &result); err != nil {
	return APIListObject{}, err
	}
	services = append(services, result.Services...)
	return APIListObject{
	More: result.More,
	Offset: result.Offset,
	Limit: result.Limit,
	}, nil
	}
	if err := c.pagedGet(ctx, "/services?"+v.Encode(), responseHandler); err != nil {
	return nil, err
	}
	return services, nil
	}

[t-junjie]

Thanks for the suggestions and example!
I might be a little slow, but I will get started on this and follow up with you if/when I have any questions.

[t-junjie]

I took a look at the example in #296, as well as the above example.
I am confused as to whether I should be using pagedGet or get to retrieve the audit records.

With reference to the following code, would option 2 be more of what you were thinking for the ListAuditRecordsPaginated?

Or would it be similar to option 1, but we would have to convert include to a responseHandler such that we can append the result from the responseHandler to auditRecords? (similar to above example)

func (c *Client) ListAuditRecordsPaginated(ctx context.Context, o ListAuditRecordsOptions, include func(AuditRecord) bool) ([]AuditRecord, error) {
	v, err := query.Values(o)
	if err != nil {
		return nil, err
	}

	if include == nil {
		include = func(AuditRecord) bool { return true }
	}

	var auditRecords []AuditRecord

	// Option 1
	// if err := c.pagedGet(ctx,"/audit/records?"+v.Encode(),include); err != nil {
	// 	// this does not work because include is not a responseHandler
	// }

	// Option 2
	// resp, err := c.get(ctx, "/audit/records?"+v.Encode())
	// if err != nil {
	// 	return nil, err
	// }

	// var auditResponse ListAuditRecordsResponse
	// if err = c.decodeJSON(resp, &auditResponse); err != nil {
	// 	return nil, err
	// }

	// records := auditResponse.Records
	// for _, r := range records {
	// 	if include(r) {
	// 		auditRecords = append(auditRecords, r)
	// 	}
	// }

	return auditRecords, nil

}

[theckman]

This is mostly pseudocode, so the .Records field I reference may not make sense. But I was thinking something like...

func (c *Client) ListAuditRecordsPaginated(ctx context.Context, o ListAuditRecordsOptions, include func(AuditRecord) bool) ([]AuditRecord, error) {
	if include == nil {
		include = func(AuditRecord) bool { return true }
	}

	v, err := query.Values(o)
	if err != nil {
		return nil, err
	}

	var records []AuditRecord

	responseHandler := func(response *http.Response) (APIListObject, error) {
		var result ListAuditRecordsResponse
		if err := c.decodeJSON(response, &result); err != nil {
			return APIListObject{}, err
		}

		for _, r := range result.Records {
			if include(r) {
				records = append(records, r)
			}
		}

		return APIListObject{
			More:   result.More,
			Offset: result.Offset,
			Limit:  result.Limit,
		}, nil
	}

	if err := c.pagedGet(ctx, "/THEACTUALPATH?"+v.Encode(), responseHandler); err != nil {
		return nil, err
	}

	return records, nil
}

[t-junjie]

I just tried implementing this and realised that the style of pagination supported has changed. (Classic pagination->Cursor-based pagination).
The difference is mentioned here.
Note: See response body fields, which includes only limit and next_cursor

The current implementation of APIListObject contains the field Limit,More,Offset,Total.

Should we change the implementation of APIListObject to use the new cursor-based fields, or create a separate struct e.g. APIListObjectCursorBased for cursor-based pagination?

[theckman]

When creating this issue I was unaware of this endpoint using the cursor pagination, so I was under the impression it would mostly be a copy/paste of other paginated calls. I was wrong. Sorry about that. 😅

Let me take a look at this a bit to see how to best handle this type of pagination. I assume it would be to create something similar to APIListObject, but I need to verify that based on something else I recently discovered about the APIListObject type.

[theckman]

So I think we need to create something similar to pagedGet for cursor-based endpoints in the API. I played around with it a little bit, and think maybe this should be added to the bottom of client.go:

type cursor struct {
	Limit      uint
	NextCursor string
}

type cursorHandler func(r *http.Response) (cursor, error)

func (c *Client) cursorGet(ctx context.Context, basePath string, handler cursorHandler) error {
	var next string

	basePrefix := getBasePrefix(basePath)

	for {
		var cs string
		if len(next) > 0 {
			cs = fmt.Sprintf("cursor=%s", next)
		}

		resp, err := c.do(ctx, http.MethodGet, fmt.Sprintf("%s%s", basePrefix, cs), nil, nil)
		if err != nil {
			return err
		}

		c, err := handler(resp)
		if err != nil {
			return err
		}

		if len(c.NextCursor) == 0 {
			break
		}

		next = c.NextCursor
	}

	return nil
}

Then I think the actual paginated method call, based roughly on the one I linked above, you'd do something like:

handler := func(r *http.Response) (cursor, error) {
	var res ListAuditRecordsResponse
	if err := c.decodeJSON(r.Body, &res); err != nil {
		return cursor{}, err
	}

	for _, ar := range res.Records {
		if include(ar) {
			auditRecords = append(auditRecords, ar)
		}
	}

	return cursor{
		Limit: res.Limit,
		NextCursor: res.NextCursor,
	}, nil
}

if err := c.cursorGet(ctx, "/audit/records?"+queryParams.Encode(), handler); err != nil {
	return nil, err
}

return auditRecords, nil

We'd also need to add a Limit and Cursor field directly to both the ListAuditRecordsOptions and ListAuditRecordsResponse structs.

Does that make sense?

[t-junjie]

Yes, that does make sense.
I have implemented these 2 methods here.

I am trying to write tests for the cursorGet method (in client.go) taking reference from pagedGet, but it's not immediately obvious to me where these tests are written (in client_test.go), as there is a TestClient_Do method but no TestClient_PagedGet or something similar.

Strangely enough, my editor shows that pagedGet has test coverage so I am confused as to whether the tests are actually there. Could you point me to where the tests are written, if they have been written?

I will look into writing tests for ListAuditRecords and ListAuditRecordsPaginated for now.

[t-junjie]

Also, TestClient_Do seems to be failing. Is this a cause for concern?

go test -v -run TestClient_Do
=== RUN   TestClient_Do
--- FAIL: TestClient_Do (0.00s)
panic: runtime error: invalid memory address or nil pointer dereference [recovered]

[theckman]

#408 was merged. Thank you for helping out with this!