Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add errors array to hub api metadata and status to polling event metadata #425

Merged
merged 2 commits into from
Nov 16, 2020
Merged

Add errors array to hub api metadata and status to polling event metadata #425

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
91db059
fix(@esri/hub-downloads): add errors array to hub api download metada…
rgwozdz Nov 13, 2020
a2128c1
refactor(@esri/hub-downloads): use type/enum for status. Replace err…
rgwozdz Nov 16, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions packages/downloads/src/download-status.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
export type DownloadStatus =
| "ready"
| "ready_unknown"
| "stale"
| "not_ready"
| "creating"
| "updating"
| "error_creating"
| "error_updating"
| "error";
7 changes: 5 additions & 2 deletions packages/downloads/src/hub/hub-poll-download-metadata.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,15 +35,18 @@ class HubPoller implements IPoller {
}

if (exportDatasetFailed(metadata)) {
const {
errors: [error]
} = metadata;
eventEmitter.emit(`${downloadId}ExportError`, {
detail: { metadata }
detail: { error, metadata }
});
return this.disablePoll();
}
})
.catch(error => {
eventEmitter.emit(`${downloadId}PollingError`, {
detail: { error }
detail: { error, metadata: { status: "error" } }
});
return this.disablePoll();
});
Expand Down
2 changes: 2 additions & 0 deletions packages/downloads/src/hub/hub-request-download-metadata.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -82,6 +82,7 @@ function formatApiResponse(json: any, downloadId: string) {
contentLastModified,
lastModified,
status,
errors,
contentLength,
cacheTime,
source: { lastEditDate }
Expand All @@ -95,6 +96,7 @@ function formatApiResponse(json: any, downloadId: string) {
lastEditDate,
lastModified,
status,
errors: errors || [],
downloadUrl,
contentLength,
cacheTime
Expand Down
16 changes: 13 additions & 3 deletions packages/downloads/src/portal/portal-poll-export-job-status.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,11 @@ import { urlBuilder } from "../utils";
import { UserSession } from "@esri/arcgis-rest-auth";
import { IPoller } from "../poller";

enum DownloadStatus {
READY = "ready",
ERROR = "error"
}

class ExportCompletionError extends Error {
constructor(message: string) {
/* istanbul ignore next */
Expand Down Expand Up @@ -87,7 +92,9 @@ class PortalPoller implements IPoller {
if (metadata.status === "failed") {
eventEmitter.emit(`${downloadId}ExportError`, {
detail: {
error: new Error(metadata.statusMessage),
metadata: {
status: DownloadStatus.ERROR,
errors: [new Error(metadata.statusMessage)]
}
}
Expand All @@ -98,11 +105,14 @@ class PortalPoller implements IPoller {
.catch((error: any) => {
if (error instanceof ExportCompletionError) {
eventEmitter.emit(`${downloadId}ExportError`, {
detail: { metadata: { errors: [error] } }
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this makes it look like a breaking change. in order to avoid that we should probably keep metadata.errors (in addition to detail.error) until the next major release. You can add it to this list: #305

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've made the change to keep metadata.errors. As of this writing, the code snip above isn't updating to show it, but you can view it here.

detail: {
error,
metadata: { status: DownloadStatus.ERROR, errors: [error] }
}
});
} else {
eventEmitter.emit(`${downloadId}PollingError`, {
detail: { error }
detail: { error, metadata: { status: DownloadStatus.ERROR } }
});
}
return this.disablePoll();
Expand Down Expand Up @@ -173,7 +183,7 @@ function completedHandler(params: any): Promise<any> {
detail: {
metadata: {
downloadId,
status: "ready",
status: DownloadStatus.READY,
lastModified: new Date().toISOString(),
downloadUrl: urlBuilder({
host: authentication.portal,
Expand Down
28 changes: 16 additions & 12 deletions packages/downloads/src/request-download-metadata.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,8 @@
import { portalRequestDownloadMetadata } from "./portal/portal-request-download-metadata";
import { hubRequestDownloadMetadata } from "./hub/hub-request-download-metadata"
import { hubRequestDownloadMetadata } from "./hub/hub-request-download-metadata";
import { DownloadFormat } from "./download-format";
import { UserSession } from '@esri/arcgis-rest-auth';
import { DownloadStatus } from "./download-status";
import { UserSession } from "@esri/arcgis-rest-auth";

export interface IDownloadMetadataRequestParams {
/* API target for downloads: 'hub' (default) or 'portal' */
Expand All @@ -24,28 +25,31 @@ export interface IDownloadMetadataRequestParams {

export interface IDownloadMetadataResults {
/* Identifier for the download */
downloadId: string,
downloadId: string;

/* ready, not_ready, creating, updating, failed*/
status: string,
/* ready, not_ready, creating, updating, failed */
status: DownloadStatus;

/* array of any errors related to exporting*/
errors?: Error[];

/* ISO date of the service's last edit date */
lastEditDate?:string,
lastEditDate?: string;

/* ISO date of the download file's data - the last edit date of the service when the download export started */
contentLastModified?: string,
contentLastModified?: string;

/* File timestamp */
lastModified?: string,
lastModified?: string;

/* URL for downloading the file */
downloadUrl?: string,
downloadUrl?: string;

/* File size */
contentLength?: number,
contentLength?: number;

/* Time (milliseconds) it took to export and cache the download file */
cacheTime?: number
cacheTime?: number;
}

/**
Expand Down Expand Up @@ -89,7 +93,7 @@ export function requestDownloadMetadata(
authentication
} = params;

if (target === 'portal') {
if (target === "portal") {
return portalRequestDownloadMetadata({
datasetId,
format,
Expand Down
59 changes: 34 additions & 25 deletions packages/downloads/test/hub/hub-poll-download-metadata.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ const fixtures = {
format: "CSV",
status: "error_creating",
featureSet: "full",
errors: [{ message: "Some error" }],
source: {
type: "Feature Service",
url:
Expand Down Expand Up @@ -77,20 +78,26 @@ describe("hubPollDownloadMetadata", () => {
}
);
const mockEventEmitter = new EventEmitter();
spyOn(mockEventEmitter, 'emit');
spyOn(mockEventEmitter, "emit");
const poller = hubPollDownloadMetadata({
host: 'http://hub.com',
datasetId: 'abcdef0123456789abcdef0123456789_0',
downloadId: 'test-id',
spatialRefId: '4326',
format: 'CSV',
host: "http://hub.com",
datasetId: "abcdef0123456789abcdef0123456789_0",
downloadId: "test-id",
spatialRefId: "4326",
format: "CSV",
eventEmitter: mockEventEmitter,
pollingInterval: 10
});
await delay(100);
expect(mockEventEmitter.emit as any).toHaveBeenCalledTimes(1);
expect((mockEventEmitter.emit as any).calls.first().args).toEqual([
'test-idPollingError', { detail: { error: new Error('Bad Gateway') } }
"test-idPollingError",
{
detail: {
error: new Error("Bad Gateway"),
metadata: { status: "error" }
}
}
]);
expect(poller.pollTimer).toEqual(null);
} catch (err) {
Expand All @@ -110,13 +117,13 @@ describe("hubPollDownloadMetadata", () => {
}
);
const mockEventEmitter = new EventEmitter();
spyOn(mockEventEmitter, 'emit');
spyOn(mockEventEmitter, "emit");
const poller = hubPollDownloadMetadata({
host: 'http://hub.com',
datasetId: 'abcdef0123456789abcdef0123456789_0',
downloadId: 'test-id',
spatialRefId: '4326',
format: 'CSV',
host: "http://hub.com",
datasetId: "abcdef0123456789abcdef0123456789_0",
downloadId: "test-id",
spatialRefId: "4326",
format: "CSV",
eventEmitter: mockEventEmitter,
pollingInterval: 10
});
Expand All @@ -133,6 +140,7 @@ describe("hubPollDownloadMetadata", () => {
contentLastModified: undefined,
lastEditDate: "2020-06-18T01:17:31.492Z",
lastModified: undefined,
errors: [{ message: "Some error" }],
status: "error_creating",
downloadUrl:
"https://dev-hub-indexer.s3.amazonaws.com/files/dd4580c810204019a7b8eb3e0b329dd6/0/full/4326/dd4580c810204019a7b8eb3e0b329dd6_0_full_4326.csv",
Expand All @@ -157,13 +165,13 @@ describe("hubPollDownloadMetadata", () => {
}
);
const mockEventEmitter = new EventEmitter();
spyOn(mockEventEmitter, 'emit');
spyOn(mockEventEmitter, "emit");
const poller = hubPollDownloadMetadata({
host: 'http://hub.com',
datasetId: 'abcdef0123456789abcdef0123456789_0',
downloadId: 'test-id',
spatialRefId: '4326',
format: 'CSV',
host: "http://hub.com",
datasetId: "abcdef0123456789abcdef0123456789_0",
downloadId: "test-id",
spatialRefId: "4326",
format: "CSV",
eventEmitter: mockEventEmitter,
pollingInterval: 10
});
Expand All @@ -181,6 +189,7 @@ describe("hubPollDownloadMetadata", () => {
lastEditDate: "2020-06-18T01:15:31.492Z",
lastModified: "2020-06-17T13:04:28.000Z",
status: "ready",
errors: [],
downloadUrl:
"https://dev-hub-indexer.s3.amazonaws.com/files/dd4580c810204019a7b8eb3e0b329dd6/0/full/4326/dd4580c810204019a7b8eb3e0b329dd6_0_full_4326.csv",
contentLength: 1391454,
Expand Down Expand Up @@ -233,13 +242,13 @@ describe("hubPollDownloadMetadata", () => {
}
);
const mockEventEmitter = new EventEmitter();
spyOn(mockEventEmitter, 'emit');
spyOn(mockEventEmitter, "emit");
const poller = hubPollDownloadMetadata({
host: 'http://hub.com',
datasetId: 'abcdef0123456789abcdef0123456789_0',
downloadId: 'test-id',
spatialRefId: '4326',
format: 'CSV',
host: "http://hub.com",
datasetId: "abcdef0123456789abcdef0123456789_0",
downloadId: "test-id",
spatialRefId: "4326",
format: "CSV",
eventEmitter: mockEventEmitter,
pollingInterval: 10
});
Expand Down
3 changes: 3 additions & 0 deletions packages/downloads/test/hub/hub-request-download-metadata.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -195,6 +195,7 @@ describe("hubRequestDownloadMetadata", () => {
lastEditDate: "2020-06-18T01:17:31.492Z",
lastModified: "2020-06-17T13:04:28.000Z",
status: "stale",
errors: [],
downloadUrl:
"https://dev-hub-indexer.s3.amazonaws.com/files/dd4580c810204019a7b8eb3e0b329dd6/0/full/4326/dd4580c810204019a7b8eb3e0b329dd6_0_full_4326.csv",
contentLength: 1391454,
Expand Down Expand Up @@ -231,6 +232,7 @@ describe("hubRequestDownloadMetadata", () => {
lastEditDate: "2020-06-18T01:17:31.492Z",
lastModified: "2020-06-17T13:04:28.000Z",
status: "stale",
errors: [],
downloadUrl:
"https://dev-hub-indexer.s3.amazonaws.com/files/dd4580c810204019a7b8eb3e0b329dd6/0/full/4326/dd4580c810204019a7b8eb3e0b329dd6_0_full_4326.csv",
contentLength: 1391454,
Expand Down Expand Up @@ -276,6 +278,7 @@ describe("hubRequestDownloadMetadata", () => {
lastEditDate: "2020-06-18T01:17:31.492Z",
lastModified: "2020-06-17T13:04:28.000Z",
status: "stale",
errors: [],
downloadUrl:
"https://dev-hub-indexer.s3.amazonaws.com/files/dd4580c810204019a7b8eb3e0b329dd6/0/full/4326/dd4580c810204019a7b8eb3e0b329dd6_0_full_4326.csv",
contentLength: 1391454,
Expand Down
Loading