Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

nightwatch test-observability integration phase-1 #5

Merged
merged 42 commits into from
May 25, 2023

Conversation

grkpranaykumar
Copy link

No description provided.

nightwatch/globals.js Outdated Show resolved Hide resolved
nightwatch/globals.js Outdated Show resolved Hide resolved
src/utils/requestQueueHandler.js Outdated Show resolved Hide resolved
src/utils/helper.js Show resolved Hide resolved
src/utils/helper.js Outdated Show resolved Hide resolved
nightwatch/globals.js Outdated Show resolved Hide resolved
console.log(`Getting ${module} from ${GLOBAL_MODULE_PATH}`);

const global_path = path.join(GLOBAL_MODULE_PATH, module);
if (!fs.existsSync(global_path)) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No sure if looking for nightwatch in the global path is a good idea.

@gravityvi thoughts?

src/utils/helper.js Outdated Show resolved Hide resolved
src/utils/requestQueueHandler.js Outdated Show resolved Hide resolved
src/utils/requestQueueHandler.js Outdated Show resolved Hide resolved
src/utils/requestQueueHandler.js Show resolved Hide resolved
}
}
} catch (error) {
console.log(`nightwatch-browserstack-plugin: Something went wrong in processing report file for test observability - ${error}`);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

create a helper function for logs.

src/testObservability.js Outdated Show resolved Hide resolved
nightwatch/globals.js Outdated Show resolved Hide resolved
src/testObservability.js Outdated Show resolved Hide resolved
src/testObservability.js Outdated Show resolved Hide resolved
src/testObservability.js Outdated Show resolved Hide resolved
src/testObservability.js Outdated Show resolved Hide resolved
src/utils/helper.js Show resolved Hide resolved
src/testObservability.js Outdated Show resolved Hide resolved
src/testObservability.js Show resolved Hide resolved
src/testObservability.js Show resolved Hide resolved
src/testObservability.js Outdated Show resolved Hide resolved
src/testObservability.js Outdated Show resolved Hide resolved
src/testObservability.js Outdated Show resolved Hide resolved
Comment on lines 169 to 175
await this.sendTestRunEvent(eventData, testFileReport, 'HookRunStarted', globalBeforeEachHookId, 'GLOBAL_BEFORE_EACH', sectionName);
if (eventData.httpOutput && eventData.httpOutput.length > 0) {
for (let i=0; i<eventData.httpOutput.length; i+=2) {
await this.createHttpLogEvent(eventData.httpOutput[i], eventData.httpOutput[i+1], globalBeforeEachHookId);
}
}
await this.sendTestRunEvent(eventData, testFileReport, 'HookRunFinished', globalBeforeEachHookId, 'GLOBAL_BEFORE_EACH', sectionName);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this section is repeated again and again, extract is out into a function

nightwatch/globals.js Outdated Show resolved Hide resolved
src/utils/helper.js Outdated Show resolved Hide resolved
src/utils/helper.js Outdated Show resolved Hide resolved
let data = eventData;
let event_api_url = 'api/v1/event';

exports.requestQueueHandler.start();
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is does not make sense.

requestQueueHandler seems to be a singleton, this state management should be in requestQueueHandler.js itself.

src/utils/helper.js Show resolved Hide resolved
src/testObservability.js Outdated Show resolved Hide resolved
src/utils/requestQueueHandler.js Outdated Show resolved Hide resolved
Comment on lines 23 to 32
static filterPII(settings) {
const configWithoutPII = JSON.parse(JSON.stringify(settings));
if (configWithoutPII['@nightwatch/browserstack'] && configWithoutPII['@nightwatch/browserstack'].test_observability) {
this.deletePIIKeysFromObject(configWithoutPII['@nightwatch/browserstack'].test_observability);
}
if (configWithoutPII.desiredCapabilities && configWithoutPII.desiredCapabilities['bstack:options']) {
this.deletePIIKeysFromObject(configWithoutPII.desiredCapabilities['bstack:options']);
}
return configWithoutPII;
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

deletePIIKeysFromObject is defined as static and only being used in filterPII, so we can include the delete functionality in filterPII itself.

Suggested change
static filterPII(settings) {
const configWithoutPII = JSON.parse(JSON.stringify(settings));
if (configWithoutPII['@nightwatch/browserstack'] && configWithoutPII['@nightwatch/browserstack'].test_observability) {
this.deletePIIKeysFromObject(configWithoutPII['@nightwatch/browserstack'].test_observability);
}
if (configWithoutPII.desiredCapabilities && configWithoutPII.desiredCapabilities['bstack:options']) {
this.deletePIIKeysFromObject(configWithoutPII.desiredCapabilities['bstack:options']);
}
return configWithoutPII;
}
static filterPII(settings) {
const keysToDelete = ['user', 'username', 'userName', 'key', 'accessKey'];
const configWithoutPII = JSON.parse(JSON.stringify(settings));
const deleteKeys = (obj) => {
if (!obj) {
return;
}
keysToDelete.forEach(key => delete obj[key]);
}
if (configWithoutPII['@nightwatch/browserstack'] && configWithoutPII['@nightwatch/browserstack'].test_observability) {
deleteKeys(configWithoutPII['@nightwatch/browserstack'].test_observability);
}
if (configWithoutPII.desiredCapabilities && configWithoutPII.desiredCapabilities['bstack:options']) {
deleteKeys(configWithoutPII.desiredCapabilities['bstack:options']);
}
return configWithoutPII;
}

Comment on lines 14 to 40
const httpKeepAliveAgent = new http.Agent({
keepAlive: true,
timeout: 60000,
maxSockets: 2,
maxTotalSockets: 2
});

const httpsKeepAliveAgent = new https.Agent({
keepAlive: true,
timeout: 60000,
maxSockets: 2,
maxTotalSockets: 2
});

const httpScreenshotsKeepAliveAgent = new http.Agent({
keepAlive: true,
timeout: 60000,
maxSockets: 2,
maxTotalSockets: 2
});

const httpsScreenshotsKeepAliveAgent = new https.Agent({
keepAlive: true,
timeout: 60000,
maxSockets: 2,
maxTotalSockets: 2
});
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can be consolidated into a function, which will be much cleaner

function createKeepAliveAgent(protocol) {
  return new protocol.Agent({
    keepAlive: true,
    timeout: 60000,
    maxSockets: 2,
    maxTotalSockets: 2
  });
}

const httpKeepAliveAgent = createKeepAliveAgent(http);
const httpsKeepAliveAgent = createKeepAliveAgent(https);
const httpScreenshotsKeepAliveAgent = createKeepAliveAgent(http);
const httpsScreenshotsKeepAliveAgent = createKeepAliveAgent(https);

delete modulesWithEnv[testSetting][testFile].completed[completedSection].testcases;
}
}
promises.push(testObservability.processTestReportFile(JSON.parse(JSON.stringify(modulesWithEnv[testSetting][testFile]))));
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

JSON.parse and JSON.stringify is unnecessary

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Deep copy is necessary because the git modules are changing the values in the object while parsing.

};

exports.shutDownRequestHandler = async () => {
await requestQueueHandler.shutdown();
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

methods related to requesQueuetHandler should be moved to requestQueueHandler.js

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added this method to remove circular dependency and made requesQueuetHandler as singleton

Comment on lines +400 to +405
requestQueueHandler.start();
const {
shouldProceed,
proceedWithData,
proceedWithUrl
} = requestQueueHandler.add(eventData);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

methods related to requesQueuetHandler should be moved to requestQueueHandler.js

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@swrdfish these are just the method calls, which can be called anywhere in the code. Can you suggest how can this be moved inside the class itself.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The entire uploadEventData should be moved to requestQueueHandler.

agent: API_URL.includes('https') ? httpsKeepAliveAgent : httpKeepAliveAgent
}};

if (url === requestQueueHandler.screenshotEventUrl) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

methods related to requesQueuetHandler should be moved to requestQueueHandler.js

const httpScreenshotsKeepAliveAgent = createKeepAliveAgent(http);
const httpsScreenshotsKeepAliveAgent = createKeepAliveAgent(https);

const requestQueueHandler = require('./requestQueueHandler');
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

methods related to requesQueuetHandler should be moved to requestQueueHandler.js

['HookRunFinished']: 'Hook_End_Upload'
}[eventData.event_type];

if (process.env.BS_TESTOPS_JWT !== 'null') {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What happens if process.env.BS_TESTOPS_JWT is undefined?

@grkpranaykumar grkpranaykumar changed the title added build start event nightwatch test-observability integration phase-1 May 22, 2023
Comment on lines 16 to 48
reporter: function(results, done) {
if (helper.isTestObservabilitySession()) {
const promises = [];
try {
const modulesWithEnv = results['modulesWithEnv'];
for (const testSetting in modulesWithEnv) {
for (const testFile in modulesWithEnv[testSetting]) {
for (const completedSection in modulesWithEnv[testSetting][testFile].completed) {
if (modulesWithEnv[testSetting][testFile].completed[completedSection]) {
delete modulesWithEnv[testSetting][testFile].completed[completedSection].steps;
delete modulesWithEnv[testSetting][testFile].completed[completedSection].testcases;
}
}
promises.push(testObservability.processTestReportFile(JSON.parse(JSON.stringify(modulesWithEnv[testSetting][testFile]))));
}
}

Promise.all(promises).then(() => {
done();
}).catch((err) =>{
Logger.error(`Something went wrong in processing report file for test observability - ${err.message} with stacktrace ${err.stack}`);
CrashReporter.uploadCrashReport(err.message, err.stack);
done();
});

return;
} catch (error) {
CrashReporter.uploadCrashReport(error.message, error.stack);
Logger.error(`Something went wrong in processing report file for test observability - ${error.message} with stacktrace ${error.stack}`);
}
}
done(results);
},
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this whole code can be refactored like this

reporter: async function(results, done) {
  if (!helper.isTestObservabilitySession()) {
    done(results);
    return;
  }

  try {
    const modulesWithEnv = results['modulesWithEnv'];
    const promises = [];

    for (const testSetting in modulesWithEnv) {
      for (const testFile in modulesWithEnv[testSetting]) {
      const completedSections = modulesWithEnv[testSetting][testFile].completed;

        for (const completedSection in completedSections) {
          if (completedSections[completedSection]) {
            delete completedSections[completedSection].steps;
            delete completedSections[completedSection].testcases;
          }
        }

        // Maybe create a helper method to do `.parse` and `.stringify`
        promises.push(testObservability.processTestReportFile(JSON.parse(JSON.stringify(modulesWithEnv[testSetting][testFile]))));
      }
    }

    await Promise.all(promises);
    done();
  } catch (error) {
    Logger.error(`Something went wrong in processing report file for test observability - ${error.message} with stacktrace ${error.stack}`);
    CrashReporter.uploadCrashReport(error.message, error.stack);
  }
}

name: helper.getObservabilityBuild(this._settings, this._bstackOptions),
build_identifier: options.buildIdentifier,
description: options.buildDescription || '',
start_time: (new Date()).toISOString(),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
start_time: (new Date()).toISOString(),
start_time: new Date().toISOString(),

Comment on lines 82 to 90
if (response.data && response.data.jwt) {
process.env.BS_TESTOPS_JWT = response.data.jwt;
}
if (response.data && response.data.build_hashed_id) {
process.env.BS_TESTOPS_BUILD_HASHED_ID = response.data.build_hashed_id;
}
if (response.data && response.data.allow_screenshots) {
process.env.BS_TESTOPS_ALLOW_SCREENSHOTS = response.data.allow_screenshots.toString();
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
if (response.data && response.data.jwt) {
process.env.BS_TESTOPS_JWT = response.data.jwt;
}
if (response.data && response.data.build_hashed_id) {
process.env.BS_TESTOPS_BUILD_HASHED_ID = response.data.build_hashed_id;
}
if (response.data && response.data.allow_screenshots) {
process.env.BS_TESTOPS_ALLOW_SCREENSHOTS = response.data.allow_screenshots.toString();
}
const responseData = response.data || {};
if (responseData.jwt) {
process.env.BS_TESTOPS_JWT = responseData.jwt;
}
if (responseData.build_hashed_id) {
process.env.BS_TESTOPS_BUILD_HASHED_ID = responseData.build_hashed_id;
}
if (responseData.allow_screenshots) {
process.env.BS_TESTOPS_ALLOW_SCREENSHOTS = responseData.allow_screenshots.toString();
}

await helper.shutDownRequestHandler();
try {
const response = await helper.makeRequest('PUT', `api/v1/builds/${process.env.BS_TESTOPS_BUILD_HASHED_ID}/stop`, data, config);
if (response.data && response.data.error) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
if (response.data && response.data.error) {
if (response.data?.error) {

};
}
const data = {
'stop_time': (new Date()).toISOString()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
'stop_time': (new Date()).toISOString()
'stop_time': new Date().toISOString()

try {
const response = await helper.makeRequest('PUT', `api/v1/builds/${process.env.BS_TESTOPS_BUILD_HASHED_ID}/stop`, data, config);
if (response.data && response.data.error) {
throw ({message: response.data.error});
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
throw ({message: response.data.error});
throw {message: response.data.error};

break;
}
default: {
if (eventData.retryTestData && eventData.retryTestData.length>0) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
if (eventData.retryTestData && eventData.retryTestData.length>0) {
if (eventData.retryTestData?.length) {

}
}
}
if (skippedTests && skippedTests.length > 0) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
if (skippedTests && skippedTests.length > 0) {
if (skippedTests?.length > 0) {


async processTestRunData (eventData, sectionName, testFileReport, hookIds) {
const testUuid = uuidv4();
const errorData = eventData.commands.find(command => command.result && command.result.stack);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
const errorData = eventData.commands.find(command => command.result && command.result.stack);
const errorData = eventData.commands.find(command => command.result?.stack);

Comment on lines 218 to 222
if (eventData.httpOutput && eventData.httpOutput.length > 0) {
for (let i=0; i<eventData.httpOutput.length; i+=2) {
await this.createHttpLogEvent(eventData.httpOutput[i], eventData.httpOutput[i+1], testUuid);
}
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
if (eventData.httpOutput && eventData.httpOutput.length > 0) {
for (let i=0; i<eventData.httpOutput.length; i+=2) {
await this.createHttpLogEvent(eventData.httpOutput[i], eventData.httpOutput[i+1], testUuid);
}
}
for (const [index, output] of eventData.httpOutput.entries()) {
if (index % 2 === 0) {
await this.createHttpLogEvent(output, eventData.httpOutput[index + 1], testUuid);
}
}

Copy link
Member

@vaibhavsingh97 vaibhavsingh97 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I made a small refactor suggestion, but overall I think this PR is looking great!

Comment on lines 20 to 56
exports.makeRequest = (type, url, data, config) => {

return new Promise((resolve, reject) => {
const options = {...config, ...{
method: type,
url: `${API_URL}/${url}`,
body: data,
json: config.headers['Content-Type'] === 'application/json',
agent: API_URL.includes('https') ? httpsKeepAliveAgent : httpKeepAliveAgent
}};

if (url === SCREENSHOT_EVENT_URL) {
options.agent = API_URL.includes('https') ? httpsScreenshotsKeepAliveAgent : httpScreenshotsKeepAliveAgent;
}

request(options, function callback(error, response, body) {
if (error) {
reject(error);
} else if (response.statusCode !== 200) {
if (response.statusCode === 401) {
reject(response && response.body ? response.body : `Received response from BrowserStack Server with status : ${response.statusCode}`);
} else {
reject(`Received response from BrowserStack Server with status : ${response.statusCode}`);
}
} else {
try {
if (body && typeof(body) !== 'object') {body = JSON.parse(body)}
} catch (e) {
reject('Not a JSON response from BrowserStack Server');
}
resolve({
data: body
});
}
});
});
};
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nitpick: I had suggested small refactor

Suggested change
exports.makeRequest = (type, url, data, config) => {
return new Promise((resolve, reject) => {
const options = {...config, ...{
method: type,
url: `${API_URL}/${url}`,
body: data,
json: config.headers['Content-Type'] === 'application/json',
agent: API_URL.includes('https') ? httpsKeepAliveAgent : httpKeepAliveAgent
}};
if (url === SCREENSHOT_EVENT_URL) {
options.agent = API_URL.includes('https') ? httpsScreenshotsKeepAliveAgent : httpScreenshotsKeepAliveAgent;
}
request(options, function callback(error, response, body) {
if (error) {
reject(error);
} else if (response.statusCode !== 200) {
if (response.statusCode === 401) {
reject(response && response.body ? response.body : `Received response from BrowserStack Server with status : ${response.statusCode}`);
} else {
reject(`Received response from BrowserStack Server with status : ${response.statusCode}`);
}
} else {
try {
if (body && typeof(body) !== 'object') {body = JSON.parse(body)}
} catch (e) {
reject('Not a JSON response from BrowserStack Server');
}
resolve({
data: body
});
}
});
});
};
exports.makeRequest = (type, url, data, config) => {
const isHttps = API_URL.includes('https');
if (url === SCREENSHOT_EVENT_URL) {
agent = isHttps ? httpsScreenshotsKeepAliveAgent : httpScreenshotsKeepAliveAgent;
} else {
agent = isHttps ? httpsKeepAliveAgent : httpKeepAliveAgent;
}
const options = {
...config,
method: type,
url: `${API_URL}/${url}`,
body: data,
json: config.headers['Content-Type'] === 'application/json',
agent
};
return new Promise((resolve, reject) => {
request(options, function callback(error, response, body) {
if (error) {
reject(error);
} else if (response.statusCode !== 200) {
if (response.statusCode === 401) {
reject(response && response.body ? response.body : `Received response from BrowserStack Server with status : ${response.statusCode}`);
} else {
reject(`Received response from BrowserStack Server with status : ${response.statusCode}`);
}
} else {
try {
if (body && typeof(body) !== 'object') {body = JSON.parse(body)}
} catch (e) {
reject('Not a JSON response from BrowserStack Server');
}
resolve({
data: body
});
}
});
});
};

};

exports.getObservabilityProject = (options, bstackOptions={}) => {
if (options.test_observability && options.test_observability.projectName) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Have you considered using optional chaining? I've noticed it being used throughout the files.

@swrdfish swrdfish merged commit 006ac1f into nightwatchjs:main May 25, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants