Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add SES experiment toggle (iOS) #8373

Merged
merged 36 commits into from
Jan 30, 2024
Merged

Conversation

leotm
Copy link
Member

@leotm leotm commented Jan 23, 2024

Description

A feature toggle for SES, within our settings menu under experiments, enabled by default

TODO

  • add react-native-mmkv
  • MVP
  • ensure only visible on iOS not Android
    • show only on iOS
    • or show if storage key exists
  • check next to Blockaid experiment enabled
  • fix test, update snapshot
  • add descriptive content
  • translate text
  • update styling
  • replace react-native-mmkv with react-native-default-preference
  • fix Android debug bundling on Hermes
  • consider content revision to include Hardened JS and SES (Secure EcmaScript) terms
  • update to newly discussed text with link
  • fix styling, move switch above next to title
  • display both SES and Blockaid under same Security heading
  • preserve Android UI
  • update SES to be enabled by default

Related issues

Fixes: partially mitigates rollout risk

Follow-up to

Manual testing steps

iOS

  • boot app
  • login / create wallet
  • go to bottom tab Settings menu
  • open Experimental menu
  • SES enabled by default, like currently in (main) branch
  • disable SES ft toggle
  • reboot app
  • SES now disabled
  • verify in Experimental menu
  • repeat steps above to re-enable

Android

  • boot app
  • login / create wallet
  • go to bottom tab Settings menu
  • open Experimental menu
  • UI remains the same
    • Blockaid shown if feature is enabled

Screenshots/Recordings

Previous MVP video (old UI) demo showing the behaviour, a simple settings menu ft toggle that persists, enabling/disabling lockdown upon app reboot, on iOS only

Android remains unchanged (Blockaid disabled, Blockaid enabled)

Before

Blockaid ft disabled

Screenshot 2024-01-26 at 7 20 46 pm

Blockaid ft enabled

Screenshot 2024-01-26 at 7 25 05 pm

After

Blockaid ft disabled

Screenshot 2024-01-30 at 1 07 07 pm

Blockaid ft enabled

Screenshot 2024-01-30 at 1 07 48 pm

Navigating to Learn more

Screenshot 2024-01-26 at 6 52 17 pm

Pre-merge author checklist

  • I’ve followed MetaMask Coding Standards.
  • I've clearly explained what problem this PR is solving and how it is solved.
  • I've linked related issues
  • I've included manual testing steps
  • I've included screenshots/recordings if applicable
  • I’ve included tests if applicable
  • I’ve documented my code using JSDoc format if applicable
  • I’ve applied the right labels on the PR (see labeling guidelines). Not required for external contributors.
  • I’ve properly set the pull request status:
    • In case it's not yet "ready for review", I've set it to "draft".
    • In case it's "ready for review", I've changed it from "draft" to "non-draft".

Pre-merge reviewer checklist

  • I've manually tested the PR (e.g. pull and build branch, run the app, test code being changed).
  • I confirm that this PR addresses all acceptance criteria described in the ticket it closes and includes the necessary testing evidence such as recordings and or screenshots.

- add NPM module
- update Yarn lockfile
- update CocoaPods lockfile
Copy link
Contributor

CLA Signature Action: All authors have signed the CLA. You may need to manually re-run the blocking PR check if it doesn't pass in a few minutes.

@codecov-commenter
Copy link

codecov-commenter commented Jan 23, 2024

Codecov Report

Attention: 3 lines in your changes are missing coverage. Please review.

Comparison is base (a3ea469) 40.60% compared to head (c1f0d24) 40.62%.

Files Patch % Lines
...ents/Views/Settings/ExperimentalSettings/index.tsx 70.00% 2 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #8373      +/-   ##
==========================================
+ Coverage   40.60%   40.62%   +0.01%     
==========================================
  Files        1239     1239              
  Lines       29978    29989      +11     
  Branches     2868     2870       +2     
==========================================
+ Hits        12174    12182       +8     
- Misses      17107    17109       +2     
- Partials      697      698       +1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@sethkfman
Copy link
Contributor

@gauthierpetetin @Cal-L Can you take a look at this contribution to make sure it aligns with our platform for a product perspective?

@leotm Is there a more descriptive Header and content we can add so users and understand? Maybe something like Lockdown - This feature prevents dynamic modification of JavaScript running in the application.

@gauthierpetetin
Copy link
Contributor

Hi @sethkfman , sorry I only see this now. I think end users won't understand the feature in its current form. I'll setup a meeting with @yanrong-chen @coreyjanssen @leotm and @hesterbruikman to discuss how we could present it.

@leotm leotm force-pushed the feat/ses-experimental-setting branch from 017066e to 75d8c02 Compare January 24, 2024 13:33
- refactor to callbacks
- refactor booleans to strings
- refactor conditions
- DefaultPreference working in settings menu only

Blocker: DefaultPreference.get value remains undefined called at Initializecore
@leotm
Copy link
Member Author

leotm commented Jan 24, 2024

@sethkfman i refactored to replace mmkv with default-preference ea5674c
but had to revert to mmkv, since default-preference only partially works here as a toggle
(fine in settings menu, but in InitializeCore .get is unable to return the storage value)

@leotm leotm marked this pull request as ready for review January 24, 2024 16:56
@leotm leotm requested a review from a team as a code owner January 24, 2024 16:56
@leotm leotm added the needs-dev-review PR needs reviews from other engineers (in order to receive required approvals) label Jan 24, 2024
@github-actions github-actions bot added the Run Smoke E2E Triggers smoke e2e on Bitrise label Jan 24, 2024
Copy link
Contributor

github-actions bot commented Jan 24, 2024

E2E test started on Bitrise: https://app.bitrise.io/app/be69d4368ee7e86d/pipelines/7161d54a-73e6-4e27-aa77-f1e084500449
You can also kick off another Bitrise E2E smoke test by removing and re-applying the (Run Smoke E2E) label

ios_e2e_build ✅
android_e2e_build ❌

Emitted 4 errors. exiting.
> Task :app:createBundleProdDebugJsAndAssets FAILED
> Task :react-native:ReactAndroid:hermes-engine:buildCMakeRelease[x86_64][libhermes]
C/C++: Hard link from '/root/.gradle/caches/transforms-3/67c92f014f4b560624ef31d7bf26af2b/transformed/fbjni-0.3.0/prefab/modules/fbjni/libs/android.x86_64/libfbjni.so' to '/bitrise/src/node_modules/react-native/ReactAndroid/hermes-engine/build/intermediates/cxx/Release/5v3r1w6e/obj/x86_64/libfbjni.so' failed. Doing a slower copy instead.
C/C++: Hard link from '/root/.gradle/caches/transforms-3/67c92f014f4b560624ef31d7bf26af2b/transformed/fbjni-0.3.0/prefab/modules/fbjni/libs/android.x86_64/libfbjni.so' to '/bitrise/src/node_modules/react-native/ReactAndroid/hermes-engine/build/intermediates/cxx/Release/5v3r1w6e/obj/x86_64/libfbjni.so' failed. Doing a slower copy instead.
C/C++: Hard link from '/opt/android-sdk-linux/ndk/23.1.7779620/toolchains/llvm/prebuilt/linux-x86_64/sysroot/usr/lib/x86_64-linux-android/libc++_shared.so' to '/bitrise/src/node_modules/react-native/ReactAndroid/hermes-engine/build/intermediates/cxx/Release/5v3r1w6e/obj/x86_64/libc++_shared.so' failed. Doing a slower copy instead.
C/C++: Hard link from '/opt/android-sdk-linux/ndk/23.1.7779620/toolchains/llvm/prebuilt/linux-x86_64/sysroot/usr/lib/x86_64-linux-android/libc++_shared.so' to '/bitrise/src/node_modules/react-native/ReactAndroid/hermes-engine/build/intermediates/cxx/Release/5v3r1w6e/obj/x86_64/libc++_shared.so' failed. Doing a slower copy instead.
FAILURE: Build completed with 2 failures.
1: Task failed with an exception.
-----------
* What went wrong:
java.lang.StackOverflowError (no error message)

:suspect: test locally and re-run

nvm use && export METAMASK_BUILD_TYPE='main' && export METAMASK_ENVIRONMENT='local' && IGNORE_BOXLOGS_DEVELOPMENT="true" FORCE_BUNDLING=true yarn test:e2e:android:bitrise:run --testNamePattern="Smoke"

@leotm leotm marked this pull request as draft January 24, 2024 18:01
Copy link
Contributor

github-actions bot commented Jan 26, 2024

E2E test started on Bitrise: https://app.bitrise.io/app/be69d4368ee7e86d/pipelines/6cae41a9-ae95-49d0-8e58-8fec25000b9a
You can also kick off another Bitrise E2E smoke test by removing and re-applying the (Run Smoke E2E) label

ios_e2e_build ✅
android_e2e_build ✅

run_smoke_e2e_ios_android_stage ❌

TEST_SUITE_FOLDER value is: ./e2e/specs/confirmations/*
TEST_SUITE_TAG value is: .*SmokeConfirmations.*
v18.18.2
yarn run v1.22.19
$ IS_TEST='[REDACTED]' detox test -c ios.sim.debug './e2e/specs/confirmations/*' '--testNamePattern=.*SmokeConfirmations.*'
/bin/sh: emulator: command not found

:suspect:


https://app.bitrise.io/app/be69d4368ee7e86d/pipelines/2c926904-9ef9-4e82-b7cd-5f3652ce10b6

ios_e2e_build ✅
android_e2e_build ✅

run_smoke_e2e_ios_android_stage ✅

@leotm leotm mentioned this pull request Jan 26, 2024
13 tasks
@leotm leotm marked this pull request as draft January 30, 2024 10:11
@naugtur
Copy link

naugtur commented Jan 30, 2024

@leotm SES must be on by default. the toggle is a safety valve.

@sethkfman SES is on by default in the RC. You wanted the toggle. There's not a lot of review activity here and RC is cut. Is this not going in then?

@hesterbruikman
Copy link
Contributor

@naugtur IMO we should not release SES without a toggle to disable. My understanding is that conventional dapp use will not break, but we can't possibly test for all dapp use cases and so users need to be able to turn this feature off.

Is there a reason this PR is in draft?

If SES is already in 7.16.0 RC I imagine we want to add merge and add this PR asap.

  • Has any Learn more copy been prepared? @naugtur please let me know so we can involve the documentation team to add knowledge base content to link out to.
  • Believe this needs a translation PR for the copy

@leotm
Copy link
Member Author

leotm commented Jan 30, 2024

@naugtur IMO we should not release SES without a toggle to disable. My understanding is that conventional dapp use will not break, but we can't possibly test for all dapp use cases and so users need to be able to turn this feature off.

Is there a reason this PR is in draft?

If SES is already in 7.16.0 RC I imagine we want to add merge and add this PR asap.

  • Has any Learn more copy been prepared? @naugtur please let me know so we can involve the documentation team to add knowledge base content to link out to.
  • Believe this needs a translation PR for the copy

agreed would be ideal, been pushing to get this merged last week prior our Thurs RC cut (extended to Fri)
so presuming this needs cherry-pick into 7.16.0, preferably also #8413

i converted this PR to draft only now, to make the change of SES enabled by default (instead of disabled)
then updating the screenshots and re-readying up
(i think this is more convenient than merging this, then 2 PRs to be cherry-picked into 7.16.0)

Suggestion agreed with @coreyjanssen @leotm and @hesterbruikman :

  • Title: Code lockdown
  • Description: This is a test feature that blocks any changes to the app's javascript code without permission. Learn more.

last Fri we agreed the Learn more link to be this above (see screenshot), who else should i speak to (on docs team) to clarify this is ok?

RE translations i checked with @tommasini last week, i've understood we usually do translations in a separate PR
but since already included them here last week, it's not worth removing them now to create a separate translations PR

@leotm leotm marked this pull request as ready for review January 30, 2024 12:33
Copy link
Contributor

E2E test started on Bitrise: https://app.bitrise.io/app/be69d4368ee7e86d/pipelines/626c71dd-b9ce-44b0-b622-4a1d214e733e
You can also kick off another Bitrise E2E smoke test by removing and re-applying the (Run Smoke E2E) label

@hesterbruikman
Copy link
Contributor

Thanks @leotm ! And apologies, hadn't noticed the link for Learn more. Can you please explain the trade-off in enabling this feature?

I'd like to understand if we can treat the toggle as a temporary Experimental fallback, in case of unforeseen issues. Or if long-term there might always be end-user use cases to disable the feature under Advanced. In case of the latter we can reach out to the documentation team to create Knowledge Base content.

@leotm
Copy link
Member Author

leotm commented Jan 30, 2024

Thanks @leotm ! And apologies, hadn't noticed the link for Learn more. Can you please explain the trade-off in enabling this feature?

np! if by feature we mean this toggle for SES, the trade-off: our users might disable SES (reduce security), in the hopes of fixing an edge case of functionality (not yet covered by our tests)

or if by feature we mean SES, the trade-off: better security, at the compromise of possibly encountering an edge case of functionality not working as expected (i.e. discovery of a library we're using that calls non-standard JS)

I'd like to understand if we can treat the toggle as a temporary Experimental fallback, in case of unforeseen issues. Or if long-term there might always be end-user use cases to disable the feature under Advanced. In case of the latter we can reach out to the documentation team to create Knowledge Base content.

exactly the toggle is a temp Experimental fallback, long-term we want SES enabled indefinitely (like in mm-extension) for better security (permanently prevent prototype pollution - now that's a tongue twister), then ofc run LavaMoat (which runs on SES)

sethkfman
sethkfman previously approved these changes Jan 30, 2024
Copy link
Contributor

@sethkfman sethkfman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

sethkfman pushed a commit that referenced this pull request Jan 30, 2024
## **Description**

Update SES lockdown options to improve error stacks

Set error taming to unsafe
- make error stacks also available by the error instance stack
- preserve error stack filtered content
- like we
[do](https://github.com/MetaMask/metamask-extension/blob/develop/app/scripts/lockdown-run.js#L6)
in metamask-extension

Set stack filtering to verbose
- show full raw error info for each deep stack level
- preserve _noise_ that the (default) `concise` option was removing

Follow-up to
- #8033

Ref:
https://github.com/endojs/endo/blob/master/packages/ses/docs/reference.md#options-quick-reference

Nb: we're looking into a lockdown/repairIntrinsics option to disable
touching errors entirely (for cases like ours involving React Native
surfacing JS/Android/iOS errors, then later newer engine Hermes)

## **Related issues**

- note: #8352

## **Manual testing steps**

Local testing in debug-mode:
- Update
[InitializeCore](https://github.com/MetaMask/metamask-mobile/blob/main/patches/react-native%2B0.71.15.patch#L13)
to enable SES in debug/dev mode
- Trigger an error somewhere in the app
  - `new Error('test')`
  - `Sentry.captureException(new Error('test'))`
- Check simulator
  - ensure original error preserved in call stack
  - ensure tapping error redirects to source code
- Check Sentry event
  - ensure original error preserved in call stack

But note above we're looking into a better lockdown option for React
Native debug-mode,
since we disabled SES in debug-mode
[earlier](#7924) from
React Devtools interfering

Production testing:
After we merge #8373,
we'll be able to ft toggle lockdown via our in-app settings menu, which
will persist the choice and apply after the app has been rebooted

## **Screenshots/Recordings**

<!-- If applicable, add screenshots and/or recordings to visualize the
before and after of your change. -->

### **Before**

<!-- [screenshots/recordings] -->

### **After**

<!-- [screenshots/recordings] -->

## **Pre-merge author checklist**

- [x] I’ve followed [MetaMask Coding
Standards](https://github.com/MetaMask/metamask-mobile/blob/main/.github/guidelines/CODING_GUIDELINES.md).
- [x] I've clearly explained what problem this PR is solving and how it
is solved.
- [x] I've linked related issues
- [x] I've included manual testing steps
- [x] I've included screenshots/recordings if applicable
- [x] I’ve included tests if applicable
- [x] I’ve documented my code using [JSDoc](https://jsdoc.app/) format
if applicable
- [x] I’ve applied the right labels on the PR (see [labeling
guidelines](https://github.com/MetaMask/metamask-mobile/blob/main/.github/guidelines/LABELING_GUIDELINES.md)).
Not required for external contributors.
- [x] I’ve properly set the pull request status:
  - [x] In case it's not yet "ready for review", I've set it to "draft".
- [x] In case it's "ready for review", I've changed it from "draft" to
"non-draft".

## **Pre-merge reviewer checklist**

- [ ] I've manually tested the PR (e.g. pull and build branch, run the
app, test code being changed).
- [ ] I confirm that this PR addresses all acceptance criteria described
in the ticket it closes and includes the necessary testing evidence such
as recordings and or screenshots.
@sethkfman sethkfman added Run Smoke E2E Triggers smoke e2e on Bitrise and removed Run Smoke E2E Triggers smoke e2e on Bitrise labels Jan 30, 2024
Copy link
Contributor

E2E test started on Bitrise: https://app.bitrise.io/app/be69d4368ee7e86d/pipelines/977b8808-bfd5-4a59-ab1c-fc31e0bc5c57
You can also kick off another Bitrise E2E smoke test by removing and re-applying the (Run Smoke E2E) label

metamaskbot pushed a commit that referenced this pull request Jan 30, 2024
## **Description**

Update SES lockdown options to improve error stacks

Set error taming to unsafe
- make error stacks also available by the error instance stack
- preserve error stack filtered content
- like we
[do](https://github.com/MetaMask/metamask-extension/blob/develop/app/scripts/lockdown-run.js#L6)
in metamask-extension

Set stack filtering to verbose
- show full raw error info for each deep stack level
- preserve _noise_ that the (default) `concise` option was removing

Follow-up to
- #8033

Ref:
https://github.com/endojs/endo/blob/master/packages/ses/docs/reference.md#options-quick-reference

Nb: we're looking into a lockdown/repairIntrinsics option to disable
touching errors entirely (for cases like ours involving React Native
surfacing JS/Android/iOS errors, then later newer engine Hermes)

## **Related issues**

- note: #8352

## **Manual testing steps**

Local testing in debug-mode:
- Update
[InitializeCore](https://github.com/MetaMask/metamask-mobile/blob/main/patches/react-native%2B0.71.15.patch#L13)
to enable SES in debug/dev mode
- Trigger an error somewhere in the app
  - `new Error('test')`
  - `Sentry.captureException(new Error('test'))`
- Check simulator
  - ensure original error preserved in call stack
  - ensure tapping error redirects to source code
- Check Sentry event
  - ensure original error preserved in call stack

But note above we're looking into a better lockdown option for React
Native debug-mode,
since we disabled SES in debug-mode
[earlier](#7924) from
React Devtools interfering

Production testing:
After we merge #8373,
we'll be able to ft toggle lockdown via our in-app settings menu, which
will persist the choice and apply after the app has been rebooted

## **Screenshots/Recordings**

<!-- If applicable, add screenshots and/or recordings to visualize the
before and after of your change. -->

### **Before**

<!-- [screenshots/recordings] -->

### **After**

<!-- [screenshots/recordings] -->

## **Pre-merge author checklist**

- [x] I’ve followed [MetaMask Coding
Standards](https://github.com/MetaMask/metamask-mobile/blob/main/.github/guidelines/CODING_GUIDELINES.md).
- [x] I've clearly explained what problem this PR is solving and how it
is solved.
- [x] I've linked related issues
- [x] I've included manual testing steps
- [x] I've included screenshots/recordings if applicable
- [x] I’ve included tests if applicable
- [x] I’ve documented my code using [JSDoc](https://jsdoc.app/) format
if applicable
- [x] I’ve applied the right labels on the PR (see [labeling
guidelines](https://github.com/MetaMask/metamask-mobile/blob/main/.github/guidelines/LABELING_GUIDELINES.md)).
Not required for external contributors.
- [x] I’ve properly set the pull request status:
  - [x] In case it's not yet "ready for review", I've set it to "draft".
- [x] In case it's "ready for review", I've changed it from "draft" to
"non-draft".

## **Pre-merge reviewer checklist**

- [ ] I've manually tested the PR (e.g. pull and build branch, run the
app, test code being changed).
- [ ] I confirm that this PR addresses all acceptance criteria described
in the ticket it closes and includes the necessary testing evidence such
as recordings and or screenshots.
@hesterbruikman
Copy link
Contributor

Thanks @leotm ! And apologies, hadn't noticed the link for Learn more. Can you please explain the trade-off in enabling this feature?

np! if by feature we mean this toggle for SES, the trade-off: our users might disable SES (reduce security), in the hopes of fixing an edge case of functionality (not yet covered by our tests)

or if by feature we mean SES, the trade-off: better security, at the compromise of possibly encountering an edge case of functionality not working as expected (i.e. discovery of a library we're using that calls non-standard JS)

I'd like to understand if we can treat the toggle as a temporary Experimental fallback, in case of unforeseen issues. Or if long-term there might always be end-user use cases to disable the feature under Advanced. In case of the latter we can reach out to the documentation team to create Knowledge Base content.

exactly the toggle is a temp Experimental fallback, long-term we want SES enabled indefinitely (like in mm-extension) for better security (permanently prevent prototype pollution - now that's a tongue twister), then ofc run LavaMoat (which runs on SES)

Just to conclude on this: I don't think we need further Learn more end-user documentation.

Copy link

sonarcloud bot commented Jan 30, 2024

Quality Gate Passed Quality Gate passed

The SonarCloud Quality Gate passed, but some issues were introduced.

1 New issue
0 Security Hotspots
80.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

@sethkfman sethkfman merged commit 6f84f18 into main Jan 30, 2024
26 checks passed
@sethkfman sethkfman deleted the feat/ses-experimental-setting branch January 30, 2024 20:21
@github-actions github-actions bot locked and limited conversation to collaborators Jan 30, 2024
@github-actions github-actions bot removed the needs-dev-review PR needs reviews from other engineers (in order to receive required approvals) label Jan 30, 2024
@metamaskbot metamaskbot added the release-7.17.0 Issue or pull request that will be included in release 7.17.0 label Jan 30, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
release-7.17.0 Issue or pull request that will be included in release 7.17.0 Run Smoke E2E Triggers smoke e2e on Bitrise team-lavamoat
Projects
None yet
Development

Successfully merging this pull request may close these issues.

8 participants