x-pack/filebeat/input/httpjson: make sure interval data clearing happens before processing

[efd6]

What does this PR do?

This change ensures that request interval data is not cleared after a request has begun.

Why is it important?

Although extremely unlikely, there is otherwise no happens before relationship between the clearing of request interval data and the processing of the obtained data. If the interval data has been cleared during a request, spurious default first and last event, and last response values in the transform context can be set in the resulting published event.

The conditions for this to happen are that the request manages to progress at least one page of the pagination iterator before the call to trCtx.clearIntervalData() can complete. The chances of this happening are vanishingly small. However...

[2022-02-27T03:07:14.640Z] === Failed
[2022-02-27T03:07:14.640Z] === FAIL: x-pack/filebeat/input/httpjson TestInput/Test_first_event (2.02s)
[2022-02-27T03:07:14.640Z]     input_test.go:441: 
[2022-02-27T03:07:14.640Z]         	Error Trace:	input_test.go:441
[2022-02-27T03:07:14.640Z]         	Error:      	Not equal: 
[2022-02-27T03:07:14.640Z]         	            	expected: map[string]interface {}{"first":"a", "foo":"b"}
[2022-02-27T03:07:14.640Z]         	            	actual  : map[string]interface {}{"first":"none", "foo":"b"}
[2022-02-27T03:07:14.640Z]         	            	
[2022-02-27T03:07:14.640Z]         	            	Diff:
[2022-02-27T03:07:14.640Z]         	            	--- Expected
[2022-02-27T03:07:14.640Z]         	            	+++ Actual
[2022-02-27T03:07:14.640Z]         	            	@@ -1,3 +1,3 @@
[2022-02-27T03:07:14.640Z]         	            	 (map[string]interface {}) (len=2) {
[2022-02-27T03:07:14.640Z]         	            	- (string) (len=5) "first": (string) (len=1) "a",
[2022-02-27T03:07:14.640Z]         	            	+ (string) (len=5) "first": (string) (len=4) "none",
[2022-02-27T03:07:14.640Z]         	            	  (string) (len=3) "foo": (string) (len=1) "b"
[2022-02-27T03:07:14.640Z]         	Test:       	TestInput/Test_first_event
[2022-02-27T03:07:14.640Z]     --- FAIL: TestInput/Test_first_event (2.02s)
[2022-02-27T03:07:14.640Z] 
[2022-02-27T03:07:14.640Z] === FAIL: x-pack/filebeat/input/httpjson TestInput (7.20s)
[2022-02-27T03:07:14.640Z] 
[2022-02-27T03:07:14.640Z] DONE 801 tests, 7 skipped, 2 failures in 189.135s
[2022-02-27T03:07:14.640Z] Error: go test returned a non-zero value: exit status 1

Placing the call within (*responseProcessor).startProcessing provides an HB relationship between the interval clearing and the initiation of the pagination iteration and so precludes clearing the data before the events have been published.

I have not been able to replicate the failure over ~1000 test iterations in the original state, which is consistent with the analysis, though still unsatisfying.

Checklist

• My code follows the style guidelines of this project
  - [ ] I have commented my code, particularly in hard-to-understand areas
  - [ ] I have made corresponding changes to the documentation
  - [ ] I have made corresponding change to the default configuration files
  - [ ] I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

• Backport prior to 7.17?

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2022-03-29T01:50:26.102+0000

• Duration: 23 min 54 sec

❕ Flaky test report

No test was executed to be analysed.

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

• /package : Generate the packages and run the E2E tests.

• /beats-tester : Run the installation tests with beats-tester.

• run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b happensbeforerequest upstream/happensbeforerequest
git merge upstream/main
git push upstream happensbeforerequest

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b happensbeforerequest upstream/happensbeforerequest
git merge upstream/main
git push upstream happensbeforerequest

[andrewkroh]

LGTM

[efd6]

This will need to be rebased onto the delint change after it is merged.

[efd6]

/package

[efd6]

/test