Skip to content

Commit

Permalink
rename advanced setting ml:fileDataVisualizerMaxFileSize to fileUploa…
Browse files Browse the repository at this point in the history
…d:maxFileSize and increase max geojson upload size to 1GB (elastic#92620)

* rename ml:fileDataVisualizerMaxFileSize to fileUppload:maxFileSize

* add saved object migration

* file preview

* importing status

* remove console statement

* import complete view

* fix geojson_importer test

* tslint

* i18n fixes

* cleanup

* update documenation for advanced setting rename

* advanced settings usage_collection

* remove ml:fileDataVisualizerMaxFileSize from schemas and types

* add copy buttons for import response and fix geojson upload functional tests

* tslint

* remove clipboard-read check

* return early if env does not support reading from clipboard

* fix reporting tests

* review feedback

* update GeoJsonFileSource to support showing results trimmed icon and tooltip

* add fileUpload to useMlKibana context and replace dependencyCache with useMlKibana

* tslint

* review feedback

* lower case file name

* default to selecting geo_shape when file contains both points and shapes

* fix wizard onError callback to not advance to next step

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
  • Loading branch information
nreese and kibanamachine committed Mar 3, 2021
1 parent 1def580 commit cbf5d02
Show file tree
Hide file tree
Showing 47 changed files with 1,428 additions and 940 deletions.
10 changes: 5 additions & 5 deletions docs/management/advanced-options.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -83,6 +83,10 @@ specific dashboard, application, or saved object as they enter each space.
[[fields-popularlimit]]`fields:popularLimit`::
The top N most popular fields to show.

[[fileupload-maxfilesize]]`fileUpload:maxFileSize`::
Sets the file size limit when importing files. The default
value is `100MB`. The highest supported value for this setting is `1GB`.

[[filtereditor-suggestvalues]]`filterEditor:suggestValues`::
Set this property to `false` to prevent the filter editor from suggesting values
for fields.
Expand Down Expand Up @@ -258,7 +262,7 @@ Hides the "Time" column in *Discover* and in all saved searches on dashboards.
Highlights results in *Discover* and saved searches on dashboards. Highlighting
slows requests when working on big documents.

[[doctable-legacy]]`doc_table:legacy`::
[[doctable-legacy]]`doc_table:legacy`::
Controls the way the document table looks and works. Set this property to `true` to revert to the legacy implementation.

[[discover-searchFieldsFromSource]]`discover:searchFieldsFromSource`::
Expand All @@ -282,10 +286,6 @@ must contain `from` and `to` values (see
{ref}/common-options.html#date-math[accepted formats]). It is ignored unless
`ml:anomalyDetection:results:enableTimeDefaults` is enabled.

[[ml-filedatavisualizermaxfilesize]]`ml:fileDataVisualizerMaxFileSize`::
Sets the file size limit when importing data in the {data-viz}. The default
value is `100MB`. The highest supported value for this setting is `1GB`.


[float]
[[kibana-notification-settings]]
Expand Down
28 changes: 14 additions & 14 deletions docs/user/ml/index.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -20,10 +20,10 @@ image::user/ml/images/ml-data-visualizer-sample.jpg[{data-viz} for sample flight
experimental[] You can also upload a CSV, NDJSON, or log file. The *{data-viz}*
identifies the file format and field mappings. You can then optionally import
that data into an {es} index. To change the default file size limit, see
<<kibana-ml-settings,Machine learning advanced settings>>.
<<kibana-general-settings, fileUpload:maxFileSize advanced settings>>.

If {stack-security-features} are enabled, users must have the necessary
privileges to use {ml-features}. Refer to
privileges to use {ml-features}. Refer to
{ml-docs}/setup.html#setup-privileges[Set up {ml-features}].

NOTE: There are limitations in {ml-features} that affect {kib}. For more information, refer to {ml-docs}/ml-limitations.html[Machine learning].
Expand All @@ -40,15 +40,15 @@ false positives. {anomaly-detect-cap} runs in and scales with {es}, and
includes an intuitive UI on the {kib} *Machine Learning* page for creating
{anomaly-jobs} and understanding results.

If you have a license that includes the {ml-features}, you can
If you have a license that includes the {ml-features}, you can
create {anomaly-jobs} and manage jobs and {dfeeds} from the *Job Management*
pane:
pane:

[role="screenshot"]
image::user/ml/images/ml-job-management.png[Job Management]

You can use the *Settings* pane to create and edit
{ml-docs}/ml-calendars.html[calendars] and the filters that are used in
You can use the *Settings* pane to create and edit
{ml-docs}/ml-calendars.html[calendars] and the filters that are used in
{ml-docs}/ml-rules.html[custom rules]:

[role="screenshot"]
Expand All @@ -69,13 +69,13 @@ occurring in your operational environment at that time:
image::user/ml/images/ml-annotations-list.png[Single Metric Viewer with annotations]

In some circumstances, annotations are also added automatically. For example, if
the {anomaly-job} detects that there is missing data, it annotates the affected
time period. For more information, see
{ml-docs}/ml-delayed-data-detection.html[Handling delayed data]. The
the {anomaly-job} detects that there is missing data, it annotates the affected
time period. For more information, see
{ml-docs}/ml-delayed-data-detection.html[Handling delayed data]. The
*Job Management* pane shows the full list of annotations for each job.

NOTE: The {kib} {ml-features} use pop-ups. You must configure your web
browser so that it does not block pop-up windows or create an exception for your
NOTE: The {kib} {ml-features} use pop-ups. You must configure your web
browser so that it does not block pop-up windows or create an exception for your
{kib} URL.

For more information about the {anomaly-detect} feature, see
Expand All @@ -89,7 +89,7 @@ experimental[]

The Elastic {ml} {dfanalytics} feature enables you to analyze your data using
{classification}, {oldetection}, and {regression} algorithms and generate new
indices that contain the results alongside your source data.
indices that contain the results alongside your source data.

If you have a license that includes the {ml-features}, you can create
{dfanalytics-jobs} and view their results on the *Data Frame Analytics* page in
Expand All @@ -98,5 +98,5 @@ If you have a license that includes the {ml-features}, you can create
[role="screenshot"]
image::user/ml/images/outliers.png[{oldetection-cap} results in {kib}]

For more information about the {dfanalytics} feature, see
{ml-docs}/ml-dfanalytics.html[{ml-cap} {dfanalytics}].
For more information about the {dfanalytics} feature, see
{ml-docs}/ml-dfanalytics.html[{ml-cap} {dfanalytics}].
34 changes: 34 additions & 0 deletions src/core/server/ui_settings/saved_objects/migrations.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -44,3 +44,37 @@ describe('ui_settings 7.9.0 migrations', () => {
});
});
});

describe('ui_settings 7.13.0 migrations', () => {
const migration = migrations['7.13.0'];

test('returns doc on empty object', () => {
expect(migration({} as SavedObjectUnsanitizedDoc)).toEqual({
references: [],
});
});
test('properly renames ml:fileDataVisualizerMaxFileSize to fileUpload:maxFileSize', () => {
const doc = {
type: 'config',
id: '8.0.0',
attributes: {
buildNum: 9007199254740991,
'ml:fileDataVisualizerMaxFileSize': '250MB',
},
references: [],
updated_at: '2020-06-09T20:18:20.349Z',
migrationVersion: {},
};
expect(migration(doc)).toEqual({
type: 'config',
id: '8.0.0',
attributes: {
buildNum: 9007199254740991,
'fileUpload:maxFileSize': '250MB',
},
references: [],
updated_at: '2020-06-09T20:18:20.349Z',
migrationVersion: {},
});
});
});
19 changes: 19 additions & 0 deletions src/core/server/ui_settings/saved_objects/migrations.ts
Original file line number Diff line number Diff line change
Expand Up @@ -28,4 +28,23 @@ export const migrations = {
}),
references: doc.references || [],
}),
'7.13.0': (doc: SavedObjectUnsanitizedDoc<any>): SavedObjectSanitizedDoc<any> => ({
...doc,
...(doc.attributes && {
attributes: Object.keys(doc.attributes).reduce(
(acc, key) =>
key === 'ml:fileDataVisualizerMaxFileSize'
? {
...acc,
['fileUpload:maxFileSize']: doc.attributes[key],
}
: {
...acc,
[key]: doc.attributes[key],
},
{}
),
}),
references: doc.references || [],
}),
};
Original file line number Diff line number Diff line change
Expand Up @@ -220,7 +220,7 @@ export const stackManagementSchema: MakeSchemaFrom<UsageStats> = {
type: 'boolean',
_meta: { description: 'Non-default value of setting.' },
},
'ml:fileDataVisualizerMaxFileSize': {
'fileUpload:maxFileSize': {
type: 'keyword',
_meta: { description: 'Non-default value of setting.' },
},
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -74,7 +74,7 @@ export interface UsageStats {
'discover:sort:defaultOrder': string;
'context:step': number;
'accessibility:disableAnimations': boolean;
'ml:fileDataVisualizerMaxFileSize': string;
'fileUpload:maxFileSize': string;
'ml:anomalyDetection:results:enableTimeDefaults': boolean;
'ml:anomalyDetection:results:timeDefaults': string;
'truncate:maxHeight': number;
Expand Down
2 changes: 1 addition & 1 deletion src/plugins/telemetry/schema/oss_plugins.json
Original file line number Diff line number Diff line change
Expand Up @@ -7742,7 +7742,7 @@
"description": "Non-default value of setting."
}
},
"ml:fileDataVisualizerMaxFileSize": {
"fileUpload:maxFileSize": {
"type": "keyword",
"_meta": {
"description": "Non-default value of setting."
Expand Down
2 changes: 2 additions & 0 deletions x-pack/plugins/file_upload/common/constants.ts
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,8 @@
* 2.0.
*/

export const UI_SETTING_MAX_FILE_SIZE = 'fileUpload:maxFileSize';

export const MB = Math.pow(2, 20);
export const MAX_FILE_SIZE = '100MB';
export const MAX_FILE_SIZE_BYTES = 104857600; // 100MB
Expand Down
3 changes: 2 additions & 1 deletion x-pack/plugins/file_upload/kibana.json
Original file line number Diff line number Diff line change
Expand Up @@ -4,5 +4,6 @@
"kibanaVersion": "kibana",
"server": true,
"ui": true,
"requiredPlugins": ["data", "usageCollection"]
"requiredPlugins": ["data", "usageCollection"],
"requiredBundles": ["kibanaReact"]
}
2 changes: 2 additions & 0 deletions x-pack/plugins/file_upload/public/api/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,8 @@ import type { IImporter, ImportFactoryOptions } from '../importer';
export interface FileUploadStartApi {
getFileUploadComponent(): Promise<React.ComponentType<FileUploadComponentProps>>;
importerFactory(format: string, options: ImportFactoryOptions): Promise<IImporter | undefined>;
getMaxBytes(): number;
getMaxBytesFormatted(): string;
}

export async function getFileUploadComponent(): Promise<
Expand Down
155 changes: 155 additions & 0 deletions x-pack/plugins/file_upload/public/components/geojson_file_picker.tsx
Original file line number Diff line number Diff line change
@@ -0,0 +1,155 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/

import React, { Component } from 'react';
import { EuiFilePicker, EuiFormRow } from '@elastic/eui';
import { i18n } from '@kbn/i18n';
import { MB } from '../../common';
import { getMaxBytesFormatted } from '../get_max_bytes';
import { validateFile } from '../importer';
import { GeoJsonImporter, GeoJsonPreview, GEOJSON_FILE_TYPES } from '../importer/geojson_importer';

interface Props {
onSelect: ({
features,
hasPoints,
hasShapes,
importer,
indexName,
previewCoverage,
}: GeoJsonPreview & {
indexName: string;
importer: GeoJsonImporter;
}) => void;
onClear: () => void;
}

interface State {
error: string | null;
isLoadingPreview: boolean;
previewSummary: string | null;
}

export class GeoJsonFilePicker extends Component<Props, State> {
private _isMounted = false;

state: State = {
error: null,
isLoadingPreview: false,
previewSummary: null,
};

async componentDidMount() {
this._isMounted = true;
}

componentWillUnmount() {
this._isMounted = false;
}

_onFileSelect = (files: FileList | null) => {
this.props.onClear();

this.setState({
error: null,
isLoadingPreview: false,
previewSummary: null,
});

if (files && files.length) {
this._loadFilePreview(files[0]);
}
};

async _loadFilePreview(file: File) {
this.setState({ isLoadingPreview: true });

let importer: GeoJsonImporter | null = null;
let previewError: string | null = null;
let preview: GeoJsonPreview | null = null;
try {
validateFile(file, GEOJSON_FILE_TYPES);
importer = new GeoJsonImporter(file);
preview = await importer.previewFile(10000, MB * 3);
if (preview.features.length === 0) {
previewError = i18n.translate('xpack.fileUpload.geojsonFilePicker.noFeaturesDetected', {
defaultMessage: 'No GeoJson features found in selected file.',
});
}
} catch (error) {
previewError = error.message;
}

if (!this._isMounted) {
return;
}

this.setState({
error: previewError,
isLoadingPreview: false,
previewSummary:
!previewError && preview
? i18n.translate('xpack.fileUpload.geojsonFilePicker.previewSummary', {
defaultMessage: 'Previewing {numFeatures} features, {previewCoverage}% of file.',
values: {
numFeatures: preview.features.length,
previewCoverage: preview.previewCoverage,
},
})
: null,
});

if (importer && preview) {
this.props.onSelect({
...preview,
importer,
indexName: file.name.split('.')[0].toLowerCase(),
});
}
}

_renderHelpText() {
return this.state.previewSummary !== null ? (
this.state.previewSummary
) : (
<span>
{i18n.translate('xpack.fileUpload.geojsonFilePicker.acceptedFormats', {
defaultMessage: 'Formats accepted: {fileTypes}',
values: { fileTypes: GEOJSON_FILE_TYPES.join(', ') },
})}
<br />
{i18n.translate('xpack.fileUpload.geojsonFilePicker.maxSize', {
defaultMessage: 'Max size: {maxFileSize}',
values: { maxFileSize: getMaxBytesFormatted() },
})}
<br />
{i18n.translate('xpack.fileUpload.geojsonFilePicker.acceptedCoordinateSystem', {
defaultMessage: 'Coordinates must be in EPSG:4326 coordinate reference system.',
})}
</span>
);
}

render() {
return (
<EuiFormRow
isInvalid={!!this.state.error}
error={!!this.state.error ? [this.state.error] : []}
helpText={this._renderHelpText()}
>
<EuiFilePicker
initialPromptText={i18n.translate('xpack.fileUpload.geojsonFilePicker.filePicker', {
defaultMessage: 'Select or drag and drop a file',
})}
onChange={this._onFileSelect}
accept={GEOJSON_FILE_TYPES.join(',')}
isLoading={this.state.isLoadingPreview}
/>
</EuiFormRow>
);
}
}
Loading

0 comments on commit cbf5d02

Please sign in to comment.