Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

rename advanced setting ml:fileDataVisualizerMaxFileSize to fileUpload:maxFileSize and increase max geojson upload size to 1GB #92620

Merged
merged 32 commits into from
Mar 3, 2021
Merged
Show file tree
Hide file tree
Changes from 13 commits
Commits
Show all changes
32 commits
Select commit Hold shift + click to select a range
a102bab
rename ml:fileDataVisualizerMaxFileSize to fileUppload:maxFileSize
nreese Feb 22, 2021
bf4bf6a
add saved object migration
nreese Feb 22, 2021
1db54f7
file preview
nreese Feb 23, 2021
eb63959
importing status
nreese Feb 23, 2021
1665c45
remove console statement
nreese Feb 23, 2021
aca5463
import complete view
nreese Feb 24, 2021
1889840
fix geojson_importer test
nreese Feb 24, 2021
21c38f2
tslint
nreese Feb 24, 2021
e7f6866
i18n fixes
nreese Feb 24, 2021
07ff940
cleanup
nreese Feb 24, 2021
1818acf
Merge branch 'master' into file_upload_max_size
kibanamachine Feb 24, 2021
ed7a0fa
update documenation for advanced setting rename
nreese Feb 24, 2021
295ef6c
advanced settings usage_collection
nreese Feb 24, 2021
494f5ac
Merge branch 'master' into file_upload_max_size
kibanamachine Feb 25, 2021
5035cc1
remove ml:fileDataVisualizerMaxFileSize from schemas and types
nreese Feb 25, 2021
c18846c
Merge branch 'file_upload_max_size' of github.com:nreese/kibana into …
nreese Feb 25, 2021
a15ca72
add copy buttons for import response and fix geojson upload functiona…
nreese Feb 27, 2021
61b9203
Merge branch 'master' into file_upload_max_size
kibanamachine Feb 27, 2021
b2e03fb
tslint
nreese Feb 27, 2021
f48dc87
remove clipboard-read check
nreese Feb 27, 2021
be10ff7
return early if env does not support reading from clipboard
nreese Feb 28, 2021
88c3a4a
Merge branch 'master' into file_upload_max_size
kibanamachine Mar 1, 2021
cd56eda
fix reporting tests
nreese Mar 1, 2021
600bea2
review feedback
nreese Mar 1, 2021
56957f4
update GeoJsonFileSource to support showing results trimmed icon and …
nreese Mar 1, 2021
a1cbbfb
add fileUpload to useMlKibana context and replace dependencyCache wit…
nreese Mar 2, 2021
7f4ce1c
tslint
nreese Mar 2, 2021
ef47a78
merge with master
nreese Mar 2, 2021
812b511
review feedback
nreese Mar 2, 2021
e80285a
lower case file name
nreese Mar 2, 2021
eedb74e
default to selecting geo_shape when file contains both points and shapes
nreese Mar 2, 2021
5968394
fix wizard onError callback to not advance to next step
nreese Mar 2, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 5 additions & 5 deletions docs/management/advanced-options.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -83,6 +83,10 @@ specific dashboard, application, or saved object as they enter each space.
[[fields-popularlimit]]`fields:popularLimit`::
The top N most popular fields to show.

[[fileupload-maxfilesize]]`fileUpload:maxFileSize`::
Sets the file size limit when importing files. The default
value is `100MB`. The highest supported value for this setting is `1GB`.
nreese marked this conversation as resolved.
Show resolved Hide resolved

[[filtereditor-suggestvalues]]`filterEditor:suggestValues`::
Set this property to `false` to prevent the filter editor from suggesting values
for fields.
Expand Down Expand Up @@ -257,7 +261,7 @@ Hides the "Time" column in *Discover* and in all saved searches on dashboards.
Highlights results in *Discover* and saved searches on dashboards. Highlighting
slows requests when working on big documents.

[[doctable-legacy]]`doc_table:legacy`::
[[doctable-legacy]]`doc_table:legacy`::
Controls the way the document table looks and works. Set this property to `true` to revert to the legacy implementation.

[[discover-searchFieldsFromSource]]`discover:searchFieldsFromSource`::
Expand All @@ -281,10 +285,6 @@ must contain `from` and `to` values (see
{ref}/common-options.html#date-math[accepted formats]). It is ignored unless
`ml:anomalyDetection:results:enableTimeDefaults` is enabled.

[[ml-filedatavisualizermaxfilesize]]`ml:fileDataVisualizerMaxFileSize`::
Sets the file size limit when importing data in the {data-viz}. The default
value is `100MB`. The highest supported value for this setting is `1GB`.


[float]
[[kibana-notification-settings]]
Expand Down
28 changes: 14 additions & 14 deletions docs/user/ml/index.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -20,10 +20,10 @@ image::user/ml/images/ml-data-visualizer-sample.jpg[{data-viz} for sample flight
experimental[] You can also upload a CSV, NDJSON, or log file. The *{data-viz}*
identifies the file format and field mappings. You can then optionally import
that data into an {es} index. To change the default file size limit, see
<<kibana-ml-settings,Machine learning advanced settings>>.
<<kibana-general-settings, fileUpload:maxFileSize advanced settings>>.

If {stack-security-features} are enabled, users must have the necessary
privileges to use {ml-features}. Refer to
privileges to use {ml-features}. Refer to
{ml-docs}/setup.html#setup-privileges[Set up {ml-features}].

NOTE: There are limitations in {ml-features} that affect {kib}. For more information, refer to {ml-docs}/ml-limitations.html[Machine learning].
Expand All @@ -40,15 +40,15 @@ false positives. {anomaly-detect-cap} runs in and scales with {es}, and
includes an intuitive UI on the {kib} *Machine Learning* page for creating
{anomaly-jobs} and understanding results.

If you have a license that includes the {ml-features}, you can
If you have a license that includes the {ml-features}, you can
create {anomaly-jobs} and manage jobs and {dfeeds} from the *Job Management*
pane:
pane:

[role="screenshot"]
image::user/ml/images/ml-job-management.png[Job Management]

You can use the *Settings* pane to create and edit
{ml-docs}/ml-calendars.html[calendars] and the filters that are used in
You can use the *Settings* pane to create and edit
{ml-docs}/ml-calendars.html[calendars] and the filters that are used in
{ml-docs}/ml-rules.html[custom rules]:

[role="screenshot"]
Expand All @@ -69,13 +69,13 @@ occurring in your operational environment at that time:
image::user/ml/images/ml-annotations-list.png[Single Metric Viewer with annotations]

In some circumstances, annotations are also added automatically. For example, if
the {anomaly-job} detects that there is missing data, it annotates the affected
time period. For more information, see
{ml-docs}/ml-delayed-data-detection.html[Handling delayed data]. The
the {anomaly-job} detects that there is missing data, it annotates the affected
time period. For more information, see
{ml-docs}/ml-delayed-data-detection.html[Handling delayed data]. The
*Job Management* pane shows the full list of annotations for each job.

NOTE: The {kib} {ml-features} use pop-ups. You must configure your web
browser so that it does not block pop-up windows or create an exception for your
NOTE: The {kib} {ml-features} use pop-ups. You must configure your web
browser so that it does not block pop-up windows or create an exception for your
{kib} URL.

For more information about the {anomaly-detect} feature, see
Expand All @@ -89,7 +89,7 @@ experimental[]

The Elastic {ml} {dfanalytics} feature enables you to analyze your data using
{classification}, {oldetection}, and {regression} algorithms and generate new
indices that contain the results alongside your source data.
indices that contain the results alongside your source data.

If you have a license that includes the {ml-features}, you can create
{dfanalytics-jobs} and view their results on the *Data Frame Analytics* page in
Expand All @@ -98,5 +98,5 @@ If you have a license that includes the {ml-features}, you can create
[role="screenshot"]
image::user/ml/images/outliers.png[{oldetection-cap} results in {kib}]

For more information about the {dfanalytics} feature, see
{ml-docs}/ml-dfanalytics.html[{ml-cap} {dfanalytics}].
For more information about the {dfanalytics} feature, see
{ml-docs}/ml-dfanalytics.html[{ml-cap} {dfanalytics}].
34 changes: 34 additions & 0 deletions src/core/server/ui_settings/saved_objects/migrations.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -44,3 +44,37 @@ describe('ui_settings 7.9.0 migrations', () => {
});
});
});

describe('ui_settings 7.13.0 migrations', () => {
const migration = migrations['7.13.0'];

test('returns doc on empty object', () => {
expect(migration({} as SavedObjectUnsanitizedDoc)).toEqual({
references: [],
});
});
test('properly renames ml:fileDataVisualizerMaxFileSize to fileUpload:maxFileSize', () => {
const doc = {
type: 'config',
id: '8.0.0',
attributes: {
buildNum: 9007199254740991,
'ml:fileDataVisualizerMaxFileSize': '250MB',
},
references: [],
updated_at: '2020-06-09T20:18:20.349Z',
migrationVersion: {},
};
expect(migration(doc)).toEqual({
type: 'config',
id: '8.0.0',
attributes: {
buildNum: 9007199254740991,
'fileUpload:maxFileSize': '250MB',
},
references: [],
updated_at: '2020-06-09T20:18:20.349Z',
migrationVersion: {},
});
});
});
19 changes: 19 additions & 0 deletions src/core/server/ui_settings/saved_objects/migrations.ts
Original file line number Diff line number Diff line change
Expand Up @@ -28,4 +28,23 @@ export const migrations = {
}),
references: doc.references || [],
}),
'7.13.0': (doc: SavedObjectUnsanitizedDoc<any>): SavedObjectSanitizedDoc<any> => ({
...doc,
...(doc.attributes && {
attributes: Object.keys(doc.attributes).reduce(
(acc, key) =>
key === 'ml:fileDataVisualizerMaxFileSize'
? {
...acc,
nreese marked this conversation as resolved.
Show resolved Hide resolved
['fileUpload:maxFileSize']: doc.attributes[key],
}
: {
...acc,
[key]: doc.attributes[key],
},
{}
),
}),
references: doc.references || [],
}),
};
Original file line number Diff line number Diff line change
Expand Up @@ -73,6 +73,7 @@ export const stackManagementSchema: MakeSchemaFrom<UsageStats> = {
'discover:sort:defaultOrder': { type: 'keyword' },
'context:step': { type: 'long' },
'accessibility:disableAnimations': { type: 'boolean' },
'fileUpload:maxFileSize': { type: 'keyword' },
nreese marked this conversation as resolved.
Show resolved Hide resolved
'ml:fileDataVisualizerMaxFileSize': { type: 'keyword' },
'ml:anomalyDetection:results:enableTimeDefaults': { type: 'boolean' },
'ml:anomalyDetection:results:timeDefaults': { type: 'keyword' },
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -73,6 +73,7 @@ export interface UsageStats {
'discover:sort:defaultOrder': string;
'context:step': number;
'accessibility:disableAnimations': boolean;
'fileUpload:maxFileSize': string;
'ml:fileDataVisualizerMaxFileSize': string;
'ml:anomalyDetection:results:enableTimeDefaults': boolean;
'ml:anomalyDetection:results:timeDefaults': string;
Expand Down
3 changes: 3 additions & 0 deletions src/plugins/telemetry/schema/oss_plugins.json
Original file line number Diff line number Diff line change
Expand Up @@ -4235,6 +4235,9 @@
"accessibility:disableAnimations": {
"type": "boolean"
},
"fileUpload:maxFileSize": {
"type": "keyword"
},
"ml:fileDataVisualizerMaxFileSize": {
"type": "keyword"
},
Expand Down
2 changes: 2 additions & 0 deletions x-pack/plugins/file_upload/common/constants.ts
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,8 @@
* 2.0.
*/

export const UI_SETTING_MAX_FILE_SIZE = 'fileUpload:maxFileSize';

export const MB = Math.pow(2, 20);
export const MAX_FILE_SIZE = '100MB';
export const MAX_FILE_SIZE_BYTES = 104857600; // 100MB
Expand Down
3 changes: 2 additions & 1 deletion x-pack/plugins/file_upload/kibana.json
Original file line number Diff line number Diff line change
Expand Up @@ -4,5 +4,6 @@
"kibanaVersion": "kibana",
"server": true,
"ui": true,
"requiredPlugins": ["data", "usageCollection"]
"requiredPlugins": ["data", "usageCollection"],
"requiredBundles": ["kibanaReact"]
}
2 changes: 2 additions & 0 deletions x-pack/plugins/file_upload/public/api/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,8 @@ import type { IImporter, ImportFactoryOptions } from '../importer';
export interface FileUploadStartApi {
getFileUploadComponent(): Promise<React.ComponentType<FileUploadComponentProps>>;
importerFactory(format: string, options: ImportFactoryOptions): Promise<IImporter | undefined>;
getMaxBytes(): number;
getMaxBytesFormatted(): string;
}

export async function getFileUploadComponent(): Promise<
Expand Down
160 changes: 160 additions & 0 deletions x-pack/plugins/file_upload/public/components/geojson_file_picker.tsx
Original file line number Diff line number Diff line change
@@ -0,0 +1,160 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/

import React, { Component } from 'react';
import { Feature } from 'geojson';
import { EuiFilePicker, EuiFormRow } from '@elastic/eui';
import { i18n } from '@kbn/i18n';
import { MB } from '../../common';
import { getMaxBytesFormatted } from '../get_max_bytes';
import { validateFile } from '../importer';
import { GeoJsonImporter, GEOJSON_FILE_TYPES } from '../importer/geojson_importer';

interface Props {
onSelect: ({
features,
geoFieldTypes,
importer,
indexName,
}: {
features: Feature[];
indexName: string;
importer: GeoJsonImporter;
geoFieldTypes: string[];
}) => void;
onClear: () => void;
}

interface State {
error: string | null;
isLoadingPreview: boolean;
previewSummary: string | null;
}

export class GeoJsonFilePicker extends Component<Props, State> {
private _isMounted = false;

state: State = {
error: null,
isLoadingPreview: false,
previewSummary: null,
};

async componentDidMount() {
this._isMounted = true;
}

componentWillUnmount() {
this._isMounted = false;
}

_onFileSelect = (files: FileList | null) => {
this.props.onClear();

this.setState({
error: null,
isLoadingPreview: false,
previewSummary: null,
});

if (files && files.length) {
this._loadFilePreview(files[0]);
}
};

async _loadFilePreview(file: File) {
this.setState({ isLoadingPreview: true });

let importer: GeoJsonImporter | null = null;
let previewError: string | null = null;
let preview: {
features: Feature[];
geoFieldTypes: string[];
previewCoverage: number;
} | null = null;
try {
validateFile(file, GEOJSON_FILE_TYPES);
importer = new GeoJsonImporter(file);
preview = await importer.previewFile(10000, MB * 3);
if (preview.features.length === 0) {
previewError = i18n.translate('xpack.fileUpload.geojsonFilePicker.noFeaturesDetected', {
defaultMessage: 'No GeoJson features found in selected file.',
});
}
} catch (error) {
previewError = error.message;
}

if (!this._isMounted) {
return;
}

this.setState({
error: previewError,
isLoadingPreview: false,
previewSummary:
!previewError && preview
? i18n.translate('xpack.fileUpload.geojsonFilePicker.previewSummary', {
defaultMessage: 'Previewing {numFeatures} features, {previewCoverage}% of file.',
values: {
numFeatures: preview.features.length,
previewCoverage: preview.previewCoverage,
},
})
: null,
});

if (importer && preview) {
this.props.onSelect({
...preview,
importer,
indexName: file.name.split('.')[0],
});
}
}

_renderHelpText() {
return this.state.previewSummary !== null ? (
this.state.previewSummary
) : (
<span>
{i18n.translate('xpack.fileUpload.geojsonFilePicker.acceptedFormats', {
defaultMessage: 'Formats accepted: {fileTypes}',
values: { fileTypes: GEOJSON_FILE_TYPES.join(', ') },
})}
<br />
{i18n.translate('xpack.fileUpload.geojsonFilePicker.maxSize', {
defaultMessage: 'Max size: {maxFileSize}',
values: { maxFileSize: getMaxBytesFormatted() },
})}
<br />
{i18n.translate('xpack.fileUpload.geojsonFilePicker.acceptedCoordinateSystem', {
defaultMessage: 'Coordinates must be in EPSG:4326 coordinate reference system.',
})}
</span>
);
}

render() {
return (
<EuiFormRow
isInvalid={!!this.state.error}
error={!!this.state.error ? [this.state.error] : []}
helpText={this._renderHelpText()}
>
<EuiFilePicker
initialPromptText={i18n.translate('xpack.fileUpload.geojsonFilePicker.filePicker', {
defaultMessage: 'Select or drag and drop a file',
})}
onChange={this._onFileSelect}
accept={GEOJSON_FILE_TYPES.join(',')}
isLoading={this.state.isLoadingPreview}
/>
</EuiFormRow>
);
}
}
Loading