Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add integration for offline support #2778

Merged
merged 15 commits into from
Aug 25, 2020
Merged
Show file tree
Hide file tree
Changes from 9 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions packages/integrations/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@
"dependencies": {
"@sentry/types": "5.20.1",
"@sentry/utils": "5.20.1",
davidmyersdev marked this conversation as resolved.
Show resolved Hide resolved
"localforage": "^1.8.1",
"tslib": "^1.9.3"
},
"devDependencies": {
Expand Down
1 change: 1 addition & 0 deletions packages/integrations/src/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ export { Debug } from './debug';
export { Dedupe } from './dedupe';
export { Ember } from './ember';
export { ExtraErrorData } from './extraerrordata';
export { Offline } from './offline';
export { ReportingObserver } from './reportingobserver';
export { RewriteFrames } from './rewriteframes';
export { SessionTiming } from './sessiontiming';
Expand Down
107 changes: 107 additions & 0 deletions packages/integrations/src/offline.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,107 @@
import { Event, EventProcessor, Hub, Integration } from '@sentry/types';
import { getGlobalObject, logger, uuid4 } from '@sentry/utils';
// @ts-ignore
import localforage = require('localforage');
davidmyersdev marked this conversation as resolved.
Show resolved Hide resolved

/**
* cache offline errors and send when connected
*/
export class Offline implements Integration {
/**
* @inheritDoc
*/
public static id: string = 'Offline';

/**
* @inheritDoc
*/
public readonly name: string = Offline.id;

/**
* the current hub instance
*/
public hub?: Hub;

/**
* event cache
*/
public offlineEventStore: LocalForage; // type imported from localforage

/**
* @inheritDoc
*/
public constructor() {
this.offlineEventStore = localforage.createInstance({
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could createInstance fail? What happens to this.offlineEventStore then?

We can import localforage typings if they exist as well.

Also, what browsers does localforage support? It's fine if it doesn't match Sentry's (https://docs.sentry.io/platforms/javascript/#browser-table) - we just have to document it then.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, good catch. I meant to wrap this in a try/catch. I will look into importing the type file too.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it meets the compatibility requirements. It is a wrapper around multiple storage mechanisms. It attempts to use IndexedDB, then falls back to WebSQL, and if all else fails, it falls back to localStorage.
https://github.com/localForage/localForage/wiki/Supported-Browsers-Platforms

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice ok, makes sense. Just one last thing, what happens when local storage is full? Is it a queue, or do new events just get dropped?

@bruno-garcia, how do we handle offline queue in mobile right now?

Copy link
Member

@bruno-garcia bruno-garcia Jul 30, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Only envelopes are cached (no event json alone are cached anymore is what I mean). The order they were cached is kept when attempting to send things to Sentry (i.e device is back online). If maxItems is reached (by default 30 IIRC), the oldest file is deleted to make space to the newest one.

We don't use dependencies or a DB of any sort. It's just files written to a directory.

If session data is stored (release health), the init=true flag must be preserved. So when deleting a file to make room for a new one (max items reached), the envelope would need to be unwrapped, checked for sessions and if init=true exists, that must be moved to the next session update queued up for submission (from oldest to newest).
Events capture when session tracking is on must be in an envelope together with a session update.

Note that the order being kept means that if you get a call to captureException, you won't be sending that to Sentry if there's stuff cached. You must prioritize what's on the storage first.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks Bruno for coming through 🚀. We can ignore all the sessions stuff, and envelope details - although important for me, probably not as for you @voraciousdev

Basically there are two things we need to keep in mind.

  1. Enforce some kind of maxItems in local storage. This can be changed through an option.

  2. Make sure that we send all events from local storage first before we continue on with other events. I know we are using an event listener right now to listen to online, do you think that is good enough? Maybe we need another check in the globalEventProcessor to be sure.

As for how this will work with sessions, we can get to that when release health comes to JS 😄

Copy link
Contributor Author

@davidmyersdev davidmyersdev Jul 30, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@AbhiPrasad the LocalForage.setItem promise will be rejected if there is not enough space on the device. With the way it's currently written in this integration, that would result in logger.warn('could not cache event while offline') being called.

Copy link
Member

@AbhiPrasad AbhiPrasad Jul 30, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok perfect we should be good then.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Make sure that we send all events from local storage first before we continue on with other events. I know we are using an event listener right now to listen to online, do you think that is good enough? Maybe we need another check in the globalEventProcessor to be sure.

@AbhiPrasad I'm not sure if this is directed at me, but there is definitely a possible race condition here regarding the online event. Given the nature of this integration (to support offline-capable apps), I think it's a real scenario that we could have real error events triggered as soon as the online event is fired. For offline apps, it's common practice to listen to the online event and run some code (which could then lead to an error being thrown/recorded).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just added localforage typings as well. Good call on that. 👍

name: 'sentry/offlineEventStore',
});

const global = getGlobalObject<Window>();

if ('addEventListener' in global) {
global.addEventListener('online', (): void => {
this._sendEvents().catch(() => {
logger.warn('could not send cached events');
});
});
}
}

/**
* @inheritDoc
*/
public setupOnce(addGlobalEventProcessor: (callback: EventProcessor) => void, getCurrentHub: () => Hub): void {
this.hub = getCurrentHub();

addGlobalEventProcessor((event: Event) => {
if (this.hub && this.hub.getIntegration(Offline)) {
const global = getGlobalObject<Window>();

// cache if we are positively offline
if ('navigator' in global && 'onLine' in global.navigator && !global.navigator.onLine) {
this._cacheEvent(event).catch((_error: any) => {
logger.warn('could not cache event while offline');
});

// return null on success or failure, because being offline will still result in an error
return null;
}
}

return event;
});
}

/**
* cache an event to send later
* @param event an event
*/
private async _cacheEvent(event: Event): Promise<Event> {
return this.offlineEventStore.setItem<Event>(uuid4(), event);
}

/**
* purge event from cache
*/
private async _purgeEvent(cacheKey: string): Promise<void> {
return this.offlineEventStore.removeItem(cacheKey);
}

/**
* send all events
*/
private async _sendEvents(): Promise<void> {
return this.offlineEventStore.iterate<Event, void>((event: Event, cacheKey: string, _index: number): void => {
if (this.hub) {
const newEventId = this.hub.captureEvent(event);

if (newEventId) {
this._purgeEvent(cacheKey).catch((_error: any): void => {
logger.warn('could not purge event from cache');
});
}
} else {
logger.warn('no hub found - could not send cached event');
}
});
}
}
133 changes: 133 additions & 0 deletions packages/integrations/test/offline.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,133 @@
import { Event, EventProcessor, Hub, Integration } from '@sentry/types';
import * as utils from '@sentry/utils';

import { Offline } from '../src/offline';

// mock localforage methods
jest.mock('localforage', () => ({
createInstance(_options: object): any {
let items: object[] = [];

return {
async getItem(key: string): object {
return items.find(item => item.key === key);
},
async iterate(callback: () => void): void {
items.forEach((item, index) => {
callback(item.value, item.key, index);
});
},
async length(): number {
return items.length;
},
async removeItem(key: string): void {
items = items.filter(item => item.key !== key);
},
async setItem(key: string, value: Event): void {
items.push({
key,
value,
});
},
};
},
}));

// mock sentry utils
jest.mock('@sentry/utils');

let integration: Integration;
let online: boolean;

describe('Offline', () => {
describe('when app is online', () => {
beforeEach(() => {
online = true;

setupIntegration();
processEvents();
});

it('does not store events in offline store', async () => {
expect(await integration.offlineEventStore.length()).toEqual(0);
});
});

describe('when app is offline', () => {
beforeEach(() => {
online = false;

setupIntegration();
processEvents();
});

it('stores events in offline store', async () => {
expect(await integration.offlineEventStore.length()).toEqual(1);
});

describe('when connectivity is restored', () => {
it('sends stored events', async done => {
processEventListeners();

expect(await integration.offlineEventStore.length()).toEqual(0);

setImmediate(done);
});
});
});
});

let eventListeners: any[];
let eventProcessors: EventProcessor[];

/** JSDoc */
function addGlobalEventProcessor(callback: () => void): void {
eventProcessors.push(callback);
}

/** JSDoc */
function getCurrentHub(): Hub {
return {
captureEvent(_event: Event): string {
return 'an-event-id';
},
getIntegration(_integration: Integration): any {
// pretend integration is enabled
return true;
},
};
}

/** JSDoc */
function processEventListeners(): void {
eventListeners.forEach(listener => {
listener();
});
}

/** JSDoc */
function processEvents(): void {
eventProcessors.forEach(processor => {
processor({
message: 'There was an error!',
});
});
}

/** JSDoc */
function setupIntegration(): void {
eventListeners = [];
eventProcessors = [];

utils.getGlobalObject.mockImplementation(() => ({
addEventListener: (_windowEvent, callback) => {
eventListeners.push(callback);
},
navigator: {
onLine: online,
},
}));

integration = new Offline();
integration.setupOnce(addGlobalEventProcessor, getCurrentHub);
}
54 changes: 53 additions & 1 deletion yarn.lock
Original file line number Diff line number Diff line change
Expand Up @@ -1939,11 +1939,32 @@ after@0.8.2:
resolved "https://registry.yarnpkg.com/after/-/after-0.8.2.tgz#fedb394f9f0e02aa9768e702bda23b505fae7e1f"
integrity sha1-/ts5T58OAqqXaOcCvaI7UF+ufh8=

agent-base@4, agent-base@5, agent-base@6, agent-base@^4.3.0, agent-base@~4.2.1:
agent-base@4, agent-base@^4.3.0:
version "4.3.0"
resolved "https://registry.yarnpkg.com/agent-base/-/agent-base-4.3.0.tgz#8165f01c436009bccad0b1d122f05ed770efc6ee"
integrity sha512-salcGninV0nPrwpGNn4VTXBb1SOuXQBiqbrNXoeizJsHrsL6ERFM2Ne3JUSBWRE6aeNJI2ROP/WEEIDUiDe3cg==
dependencies:
es6-promisify "^5.0.0"

agent-base@5:
version "5.1.1"
resolved "https://registry.yarnpkg.com/agent-base/-/agent-base-5.1.1.tgz#e8fb3f242959db44d63be665db7a8e739537a32c"
integrity sha512-TMeqbNl2fMW0nMjTEPOwe3J/PRFP4vqeoNuQMG0HlMrtm5QxKqdvAkZ1pRBQ/ulIyDD5Yq0nJ7YbdD8ey0TO3g==

agent-base@6:
version "6.0.1"
resolved "https://registry.yarnpkg.com/agent-base/-/agent-base-6.0.1.tgz#808007e4e5867decb0ab6ab2f928fbdb5a596db4"
integrity sha512-01q25QQDwLSsyfhrKbn8yuur+JNw0H+0Y4JiGIKd3z9aYk/w/2kxD/Upc+t2ZBBSUNff50VjPsSW2YxM8QYKVg==
dependencies:
debug "4"

agent-base@~4.2.1:
version "4.2.1"
resolved "https://registry.yarnpkg.com/agent-base/-/agent-base-4.2.1.tgz#d89e5999f797875674c07d87f260fc41e83e8ca9"
integrity sha512-JVwXMr9nHYTUXsBFKUqhJwvlcYU/blreOEUkhNR2eXZIvwd+c+o5V4MgDPKWnMS/56awN3TRzIP+KoPn+roQtg==
dependencies:
es6-promisify "^5.0.0"

agentkeepalive@^3.4.1:
version "3.5.2"
resolved "https://registry.yarnpkg.com/agentkeepalive/-/agentkeepalive-3.5.2.tgz#a113924dd3fa24a0bc3b78108c450c2abee00f67"
Expand Down Expand Up @@ -5114,6 +5135,18 @@ es6-object-assign@^1.1.0:
resolved "https://registry.yarnpkg.com/es6-object-assign/-/es6-object-assign-1.1.0.tgz#c2c3582656247c39ea107cb1e6652b6f9f24523c"
integrity sha1-wsNYJlYkfDnqEHyx5mUrb58kUjw=

es6-promise@^4.0.3:
version "4.2.8"
resolved "https://registry.yarnpkg.com/es6-promise/-/es6-promise-4.2.8.tgz#4eb21594c972bc40553d276e510539143db53e0a"
integrity sha512-HJDGx5daxeIvxdBxvG2cb9g4tEvwIk3i8+nhX0yGrYmZUzbkdg8QbDevheDB8gd0//uPj4c1EQua8Q+MViT0/w==

es6-promisify@^5.0.0:
version "5.0.0"
resolved "https://registry.yarnpkg.com/es6-promisify/-/es6-promisify-5.0.0.tgz#5109d62f3e56ea967c4b63505aef08291c8a5203"
integrity sha1-UQnWLz5W6pZ8S2NQWu8IKRyKUgM=
dependencies:
es6-promise "^4.0.3"

escalade@^3.0.1:
version "3.0.2"
resolved "https://registry.yarnpkg.com/escalade/-/escalade-3.0.2.tgz#6a580d70edb87880f22b4c91d0d56078df6962c4"
Expand Down Expand Up @@ -6528,6 +6561,11 @@ ignore@^5.1.4:
resolved "https://registry.yarnpkg.com/ignore/-/ignore-5.1.8.tgz#f150a8b50a34289b33e22f5889abd4d8016f0e57"
integrity sha512-BMpfD7PpiETpBl/A6S498BaIJ6Y/ABT93ETbby2fP00v4EbvPBXWEoaR1UBPKs3iR53pJY7EtZk5KACI57i1Uw==

immediate@~3.0.5:
version "3.0.6"
resolved "https://registry.yarnpkg.com/immediate/-/immediate-3.0.6.tgz#9db1dbd0faf8de6fbe0f5dd5e56bb606280de69b"
integrity sha1-nbHb0Pr43m++D13V5Wu2BigN5ps=

import-fresh@^2.0.0:
version "2.0.0"
resolved "https://registry.yarnpkg.com/import-fresh/-/import-fresh-2.0.0.tgz#d81355c15612d386c61f9ddd3922d4304822a546"
Expand Down Expand Up @@ -8106,6 +8144,13 @@ libnpmpublish@^1.1.1:
semver "^5.5.1"
ssri "^6.0.1"

lie@3.1.1:
version "3.1.1"
resolved "https://registry.yarnpkg.com/lie/-/lie-3.1.1.tgz#9a436b2cc7746ca59de7a41fa469b3efb76bd87e"
integrity sha1-mkNrLMd0bKWd56QfpGmz77dr2H4=
dependencies:
immediate "~3.0.5"

lines-and-columns@^1.1.6:
version "1.1.6"
resolved "https://registry.yarnpkg.com/lines-and-columns/-/lines-and-columns-1.1.6.tgz#1c00c743b433cd0a4e80758f7b64a57440d9ff00"
Expand Down Expand Up @@ -8155,6 +8200,13 @@ loader-utils@^2.0.0:
emojis-list "^3.0.0"
json5 "^2.1.2"

localforage@^1.8.1:
version "1.8.1"
resolved "https://registry.yarnpkg.com/localforage/-/localforage-1.8.1.tgz#f6c0a24b41ab33b10e4dc84342dd696f6f3e3433"
integrity sha512-azSSJJfc7h4bVpi0PGi+SmLQKJl2/8NErI+LhJsrORNikMZnhaQ7rv9fHj+ofwgSHrKRlsDCL/639a6nECIKuQ==
dependencies:
lie "3.1.1"

locate-path@^2.0.0:
version "2.0.0"
resolved "https://registry.yarnpkg.com/locate-path/-/locate-path-2.0.0.tgz#2b568b265eec944c6d9c0de9c3dbbbca0354cd8e"
Expand Down