Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Reporting/Discover] include the document's entire set of fields #92730

Merged
merged 8 commits into from
Feb 25, 2021
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,10 @@ const getSharingDataFields = async (
timeFieldName: string,
hideTimeColumn: boolean
) => {
if (selectedFields.length === 1 && selectedFields[0] === '_source') {
if (
selectedFields.length === 0 ||
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This fixes the bug that was immediately noticed: only the date field is populated when exporting a new search (if no other fields are selected as columns)

(selectedFields.length === 1 && selectedFields[0] === '_source')
) {
const fieldCounts = await getFieldCounts();
return {
searchFields: undefined,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -35,13 +35,19 @@ export const runTaskFnFactory: RunTaskFnFactory<ImmediateExecuteFn> = function e

return async function runTask(jobId, jobPayload, context, req) {
const generateCsv = createGenerateCsv(logger);
const { panel, visType } = jobPayload;
const { panel } = jobPayload;

logger.debug(`Execute job generating [${visType}] csv`);
logger.debug(`Execute job generating saved search CSV`);
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

logging cleanup: the visType variable was undefined


const savedObjectsClient = context.core.savedObjects.client;
const uiSettingsClient = await reporting.getUiSettingsServiceFactory(savedObjectsClient);
const job = await getGenerateCsvParams(jobPayload, panel, savedObjectsClient, uiSettingsClient);
const job = await getGenerateCsvParams(
jobPayload,
panel,
savedObjectsClient,
uiSettingsClient,
logger
);

const elasticsearch = reporting.getElasticsearchService();
const { callAsCurrentUser } = elasticsearch.legacy.client.asScoped(req);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@
import { IUiSettingsClient, SavedObjectsClientContract } from 'kibana/server';
import { EsQueryConfig } from 'src/plugins/data/server';
import { esQuery, Filter, Query } from '../../../../../../../src/plugins/data/server';
import { LevelLogger } from '../../../lib';
import { TimeRangeParams } from '../../common';
import { GenerateCsvParams } from '../../csv/generate_csv';
import {
Expand Down Expand Up @@ -44,7 +45,8 @@ export const getGenerateCsvParams = async (
jobParams: JobParamsPanelCsv,
panel: SearchPanel,
savedObjectsClient: SavedObjectsClientContract,
uiConfig: IUiSettingsClient
uiConfig: IUiSettingsClient,
logger: LevelLogger
): Promise<GenerateCsvParams> => {
let timerange: TimeRangeParams | null;
if (jobParams.post?.timerange) {
Expand Down Expand Up @@ -75,6 +77,14 @@ export const getGenerateCsvParams = async (
fields: indexPatternFields,
} = indexPatternSavedObject;

if (!indexPatternFields || indexPatternFields.length === 0) {
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

logging addition for the open issues with Download CSV from a dashboard panel (to be fixed in #88303)

logger.error(
new Error(
`No fields are selected in the saved search! Please select fields as columns in the saved search and try again.`
)
);
}

let payloadQuery: QueryFilter | undefined;
let payloadSort: any[] = [];
let docValueFields: DocValueFields[] | undefined;
Expand Down
40 changes: 40 additions & 0 deletions x-pack/test/functional/apps/discover/__snapshots__/reporting.snap

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

77 changes: 73 additions & 4 deletions x-pack/test/functional/apps/discover/reporting.ts
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ export default function ({ getService, getPageObjects }: FtrProviderContext) {
const es = getService('es');
const esArchiver = getService('esArchiver');
const browser = getService('browser');
const PageObjects = getPageObjects(['reporting', 'common', 'discover']);
const PageObjects = getPageObjects(['reporting', 'common', 'discover', 'timePicker']);
const filterBar = getService('filterBar');

describe('Discover', () => {
Expand All @@ -31,7 +31,7 @@ export default function ({ getService, getPageObjects }: FtrProviderContext) {
});
});

describe('Generate CSV button', () => {
describe('Generate CSV: new search', () => {
beforeEach(() => PageObjects.common.navigateToApp('discover'));

it('is not available if new', async () => {
Expand Down Expand Up @@ -69,14 +69,83 @@ export default function ({ getService, getPageObjects }: FtrProviderContext) {
await PageObjects.reporting.setTimepickerInDataRange();
await PageObjects.discover.saveSearch('my search - with data - expectReportCanBeCreated');
await PageObjects.reporting.openCsvReportingPanel();
expect(await PageObjects.reporting.canReportBeCreated()).to.be(true);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Got it, so we were previously testing that reporting is available, but not the contents of the reports.

await PageObjects.reporting.clickGenerateReportButton();

const url = await PageObjects.reporting.getReportURL(60000);
const res = await PageObjects.reporting.getResponse(url);

expect(res.status).to.equal(200);
expect(res.get('content-type')).to.equal('text/csv; charset=utf-8');
expectSnapshot(res.text).toMatch();
});

it('generates a report with no data', async () => {
await PageObjects.reporting.setTimepickerInNoDataRange();
await PageObjects.discover.saveSearch('my search - no data - expectReportCanBeCreated');
await PageObjects.reporting.openCsvReportingPanel();
expect(await PageObjects.reporting.canReportBeCreated()).to.be(true);
await PageObjects.reporting.clickGenerateReportButton();

const url = await PageObjects.reporting.getReportURL(60000);
const res = await PageObjects.reporting.getResponse(url);

expect(res.status).to.equal(200);
expect(res.get('content-type')).to.equal('text/csv; charset=utf-8');
expectSnapshot(res.text).toMatchInline(`
"
"
`);
});
});

describe('Generate CSV: archived search', () => {
before(async () => {
await esArchiver.load('reporting/ecommerce');
await esArchiver.load('reporting/ecommerce_kibana');
});

after(async () => {
await esArchiver.unload('reporting/ecommerce');
await esArchiver.unload('reporting/ecommerce_kibana');
});

beforeEach(() => PageObjects.common.navigateToApp('discover'));

it('generates a report with data', async () => {
await PageObjects.discover.loadSavedSearch('Ecommerce Data');
const fromTime = 'Apr 27, 2019 @ 23:56:51.374';
const toTime = 'Aug 23, 2019 @ 16:18:51.821';
await PageObjects.timePicker.setAbsoluteRange(fromTime, toTime);

await PageObjects.reporting.openCsvReportingPanel();
await PageObjects.reporting.clickGenerateReportButton();

const url = await PageObjects.reporting.getReportURL(60000);
const res = await PageObjects.reporting.getResponse(url);

expect(res.status).to.equal(200);
expect(res.get('content-type')).to.equal('text/csv; charset=utf-8');
expectSnapshot(res.text).toMatch();
});

it('generates a report with filtered data', async () => {
await PageObjects.discover.loadSavedSearch('Ecommerce Data');
const fromTime = 'Apr 27, 2019 @ 23:56:51.374';
const toTime = 'Aug 23, 2019 @ 16:18:51.821';
await PageObjects.timePicker.setAbsoluteRange(fromTime, toTime);

// filter and re-save
await filterBar.addFilter('currency', 'is', 'EUR');
await PageObjects.discover.saveSearch(`Ecommerce Data: EUR Filtered`);

await PageObjects.reporting.openCsvReportingPanel();
await PageObjects.reporting.clickGenerateReportButton();

const url = await PageObjects.reporting.getReportURL(60000);
const res = await PageObjects.reporting.getResponse(url);

expect(res.status).to.equal(200);
expect(res.get('content-type')).to.equal('text/csv; charset=utf-8');
expectSnapshot(res.text).toMatch();
});
});
});
Expand Down
4 changes: 2 additions & 2 deletions x-pack/test/functional/page_objects/reporting_page.ts
Original file line number Diff line number Diff line change
Expand Up @@ -140,8 +140,8 @@ export function ReportingPageProvider({ getService, getPageObjects }: FtrProvide

async setTimepickerInDataRange() {
log.debug('Reporting:setTimepickerInDataRange');
const fromTime = 'Sep 19, 2015 @ 06:31:44.000';
const toTime = 'Sep 19, 2015 @ 18:01:44.000';
const fromTime = 'Apr 27, 2019 @ 23:56:51.374';
const toTime = 'Aug 23, 2019 @ 16:18:51.821';
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These values have been wrong for a long time, but in this PR it matters because the content of the CSV is actually checked against a snapshot.

await PageObjects.timePicker.setAbsoluteRange(fromTime, toTime);
}

Expand Down