Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use a node-gyp version which supports Python3 #4784

Merged
merged 1 commit into from
Apr 28, 2021
Merged

Use a node-gyp version which supports Python3 #4784

merged 1 commit into from
Apr 28, 2021

Conversation

WhyNotHugo
Copy link
Contributor

@WhyNotHugo WhyNotHugo commented Jan 11, 2021

The currently pinned version of node-gyp requires Python 2. Python 2
has reached its EOL a long time ago and is unsupported by upstream.

The requirement on Python 2 is also an unnecessary barrier for new
developers to contribute to Signal.

This changeset updates the required version of node-gyp to one
compatible with Python 3.

CONTRIBUTING.md remains unchanged, since this dependency was not
previously mentioned anyway.

Supersedes #4407
Fixes #4783

First time contributor checklist:

Contributor checklist:

@WhyNotHugo
Copy link
Contributor Author

Linting is failing, not entirely sure why:

[lint-deps    ] 29491 files scanned. 1 questionable lines, 1 unused exceptions, 1785 total exceptions.
[lint-deps    ]
[lint-deps    ] Questionable lines:
[lint-deps    ] [
[lint-deps    ]   {
[lint-deps    ]     "rule": "jQuery-$(",
[lint-deps    ]     "path": "node_modules/node-gyp/lib/configure.js",
[lint-deps    ]     "line": "        !gyp.opts.nodedir ? '<(target_arch)' : '$(Configuration)',",
[lint-deps    ]     "lineNumber": 300,
[lint-deps    ]     "reasonCategory": "falseMatch|testCode|exampleCode|otherUtilityCode|regexMatchedSafeCode|notExercisedByOurApp|ruleNeeded|usageTrusted",
[lint-deps    ]     "updated": "2021-01-11T10:04:26.665Z",
[lint-deps    ]     "reasonDetail": "<optional>"
[lint-deps    ]   }
[lint-deps    ] ]
[lint-deps    ]
[lint-deps    ] Unused exceptions!
[lint-deps    ] [
[lint-deps    ]   {
[lint-deps    ]     "rule": "jQuery-$(",
[lint-deps    ]     "path": "node_modules/node-gyp/lib/configure.js",
[lint-deps    ]     "line": "        !gyp.opts.nodedir ? '<(target_arch)' : '$(Configuration)',",
[lint-deps    ]     "lineNumber": 299,
[lint-deps    ]     "reasonCategory": "falseMatch",
[lint-deps    ]     "updated": "2019-07-19T17:16:02.404Z"
[lint-deps    ]   }
[lint-deps    ] ]
[lint-deps    ] error Command failed with exit code 1.
[lint-deps    ] info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

Have to jump to work now, but will try to debug this futher at the end of the day.

@Xhale1
Copy link

Xhale1 commented Jan 26, 2021

Linting is failing, not entirely sure why:

Did you figure out why lint-deps is failing and how to fix it? I've been trying to figure out what exactly this command does but I'm coming up blank. Maybe a Signal team member could shed some light here?

@WhyNotHugo
Copy link
Contributor Author

It runs ts/util/lint/linter.ts, but I'm not fully certain what these checks do.

I think we're going to need the input of one of the devs here.

@WhyNotHugo
Copy link
Contributor Author

Note that, you can create a development environment with this branch, and the do you work based off master without any issues.

@Xhale1
Copy link

Xhale1 commented Jan 27, 2021

I think we're going to need the input of one of the devs here.

I agree, I can't wrap my head around that massive exceptions.json file. I opened an issue #4952 to hopefully consolidate communication and help with SEO.

Seems a few other pull requests ran into the same problem without ever getting it resolved.

@WhyNotHugo
Copy link
Contributor Author

@scottnonnenberg-signal: it seems you did some previous work on this area, do you have any insight?

@hiqua
Copy link
Contributor

hiqua commented Feb 12, 2021

@WhyNotHugo could you rebase your changes?

@WhyNotHugo
Copy link
Contributor Author

I did a manual rebase, but it won't install for me. I tried re-doing the work in this commit, but it does not work either.

node-gyp versions 6.x and 7.x all yield the same error:

gyp info using node-gyp@7.1.2
gyp info using node@12.18.3 | linux | x64
gyp info spawn make
gyp info spawn args [ 'BUILDTYPE=Release', '-C', 'build' ]
make: Entering directory '/home/hugo/workspace/Forks/Signal-Desktop/node_modules/@journeyapps/sqlcipher/build'
  CC(target) Release/obj.target/nothing/../../node-addon-api/nothing.o
  AR(target) Release/obj.target/../../node-addon-api/nothing.a
  COPY Release/nothing.a
  ACTION deps_sqlite3_gyp_action_before_build_target_unpack_sqlite_dep Release/obj/gen/sqlcipher-amalgamation-3033000/sqlite3.c
Traceback (most recent call last):
  File "/usr/lib/python3.9/tarfile.py", line 1682, in gzopen
    t = cls.taropen(name, mode, fileobj, **kwargs)
  File "/usr/lib/python3.9/tarfile.py", line 1659, in taropen
    return cls(name, mode, fileobj, **kwargs)
  File "/usr/lib/python3.9/tarfile.py", line 1522, in __init__
    self.firstmember = self.next()
  File "/usr/lib/python3.9/tarfile.py", line 2327, in next
    tarinfo = self.tarinfo.fromtarfile(self)
  File "/usr/lib/python3.9/tarfile.py", line 1112, in fromtarfile
    buf = tarfile.fileobj.read(BLOCKSIZE)
  File "/usr/lib/python3.9/gzip.py", line 300, in read
    return self._buffer.read(size)
  File "/usr/lib/python3.9/_compression.py", line 68, in readinto
    data = self.read(len(byte_view))
  File "/usr/lib/python3.9/gzip.py", line 487, in read
    if not self._read_gzip_header():
  File "/usr/lib/python3.9/gzip.py", line 435, in _read_gzip_header
    raise BadGzipFile('Not a gzipped file (%r)' % magic)
gzip.BadGzipFile: Not a gzipped file (b've')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/hugo/workspace/Forks/Signal-Desktop/node_modules/@journeyapps/sqlcipher/deps/./extract.py", line 7, in <module>
    tfile = tarfile.open(tarball,'r:gz');
  File "/usr/lib/python3.9/tarfile.py", line 1629, in open
    return func(name, filemode, fileobj, **kwargs)
  File "/usr/lib/python3.9/tarfile.py", line 1686, in gzopen
    raise ReadError("not a gzip file")
tarfile.ReadError: not a gzip file
make: *** [deps/action_before_build.target.mk:13: Release/obj/gen/sqlcipher-amalgamation-3033000/sqlite3.c] Error 1
make: Leaving directory '/home/hugo/workspace/Forks/Signal-Desktop/node_modules/@journeyapps/sqlcipher/build'
gyp ERR! build error
gyp ERR! stack Error: `make` failed with exit code: 2
gyp ERR! stack     at ChildProcess.onExit (/home/hugo/workspace/Forks/Signal-Desktop/node_modules/node-gyp/lib/build.js:194:23)
gyp ERR! stack     at ChildProcess.emit (events.js:315:20)
gyp ERR! stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:275:12)
gyp ERR! System Linux 5.10.16-arch1-1
gyp ERR! command "/home/hugo/.nvm/versions/node/v12.18.3/bin/node" "/home/hugo/workspace/Forks/Signal-Desktop/node_modules/node-gyp/bin/node-gyp.js" "build" "--build-from-source" "--module=/home/hugo/workspace/Forks/Signal-Desktop/node_modules/@journeyapps/sqlcipher/lib/binding/napi-v6-linux-x64/node_sqlite3.node" "--module_name=node_sqlite3" "--module_path=/home/hugo/workspace/Forks/Signal-Desktop/node_modules/@journeyapps/sqlcipher/lib/binding/napi-v6-linux-x64" "--napi_version=6" "--node_abi_napi=napi" "--napi_build_version=6" "--node_napi_label=napi-v6"
gyp ERR! cwd /home/hugo/workspace/Forks/Signal-Desktop/node_modules/@journeyapps/sqlcipher
gyp ERR! node -v v12.18.3
gyp ERR! node-gyp -v v7.1.2
gyp ERR! not ok
node-pre-gyp ERR! build error
node-pre-gyp ERR! stack Error: Failed to execute '/home/hugo/.nvm/versions/node/v12.18.3/bin/node /home/hugo/workspace/Forks/Signal-Desktop/node_modules/node-gyp/bin/node-gyp.js build --build-from-source --module=/home/hugo/workspace/Forks/Signal-Desktop/node_modules/@journeyapps/sqlcipher/lib/binding/napi-v6-linux-x64/node_sqlite3.node --module_name=node_sqlite3 --module_path=/home/hugo/workspace/Forks/Signal-Desktop/node_modules/@journeyapps/sqlcipher/lib/binding/napi-v6-linux-x64 --napi_version=6 --node_abi_napi=napi --napi_build_version=6 --node_napi_label=napi-v6' (1)
node-pre-gyp ERR! stack     at ChildProcess.<anonymous> (/home/hugo/workspace/Forks/Signal-Desktop/node_modules/node-pre-gyp/lib/util/compile.js:83:29)
node-pre-gyp ERR! stack     at ChildProcess.emit (events.js:315:20)
node-pre-gyp ERR! stack     at maybeClose (internal/child_process.js:1021:16)
node-pre-gyp ERR! stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:286:5)
node-pre-gyp ERR! System Linux 5.10.16-arch1-1
node-pre-gyp ERR! command "/home/hugo/.nvm/versions/node/v12.18.3/bin/node" "/home/hugo/workspace/Forks/Signal-Desktop/node_modules/@journeyapps/sqlcipher/node_modules/.bin/node-pre-gyp" "install" "--build-from-source"
node-pre-gyp ERR! cwd /home/hugo/workspace/Forks/Signal-Desktop/node_modules/@journeyapps/sqlcipher
node-pre-gyp ERR! node -v v12.18.3
node-pre-gyp ERR! node-pre-gyp -v v0.15.0
node-pre-gyp ERR! not ok

Not entirely sure what changed in the development branch or why I'm seeing this error.

@hiqua
Copy link
Contributor

hiqua commented Feb 15, 2021

@WhyNotHugo see #4995 (comment)

You need to have git-lfs installed, and possibly run git-lfs install from the Signal-Desktop folder before you install the npm dependencies.

You might have to clean the yarn cache before you try again: yarn cache clean and remove node_modules.

@WhyNotHugo
Copy link
Contributor Author

Oh, nice, thanks!

Slipped my mind to re-read CONTRIBUTING.md, and it's been added there as a dependency very clearly 🤦

Rebased properly this time! :)

@WhyNotHugo
Copy link
Contributor Author

lint-deps still fails with the same error. Still need some feedback on what ts/util/lint/linter.ts means/does, since new entries need to be listed there somehow.

@hiqua
Copy link
Contributor

hiqua commented Feb 16, 2021

AFAICT it checks for bad patterns in the dependencies, just like the lint step checks for them in the main code.

A solution could be to whitelist node-gyp in ts/util/lint/linter.js, but I don't know if that's acceptable for the Signal team. I think node-gyp is maintained by the nodejs team, so it seems reasonable.

@WhyNotHugo
Copy link
Contributor Author

I've added it as an exception there.

I guess whenever it's reviewed by a Signal developer we'll know if it's the right place.

Not sure how to actually get some feedback here though. It's been over a month.

@WhyNotHugo
Copy link
Contributor Author

Figured it out. There's some linting rules, and there a list of files that are exempt (I guess they've been determined safe).

However, these exemptions are listed with their filename and line number. The upgrade in node-gyp shifted this exception from line 299 to line 300, so there were two errors:

  • Superfluous exemption for line 299.
  • Missing exemption for line 300.

@WhyNotHugo WhyNotHugo changed the title WIP: Use a node-gyp version which supports Python3 Use a node-gyp version which supports Python3 Feb 16, 2021
@WhyNotHugo
Copy link
Contributor Author

@scottnonnenberg-signal Any feedback for this?

@diegoe
Copy link

diegoe commented Mar 21, 2021

FWIW I run into the other side of the coin of this issue: journeyapps/sqlcipher has an outdated deps/sqlite3.gyp that explicitly calls on python (note the lack of version number) which on modern Unix/Linux systems doesn't exist... Or might point to python3 like in @WhyNotHugo use case.

Ironically the whole point of PEP394, and removing python, is to avoid this kind of "unexplained" issues, where one user has python=python3 and another has python=python2, and maybe a third one has no python at all.

I sent a PR for the above to journeyapps: https://github.com/journeyapps/node-sqlcipher/pull/80/files

Update Also sent a PR to Signal's fork, in case upstream never gets to that: EvanHahn-Signal/node-sqlcipher#2

But in the meantime, if you run into this, you can hack your way around it by telling yarn to pull from my repo (see below).

OBVIOUSLY I'm not proposing you change the upstream, but since this PR is the only one tending to python issues, this might be useful to anyone else losing hours into making sense of this.

diff --git a/package.json b/package.json
index 6ace400a..dffd8eee 100644
--- a/package.json
+++ b/package.json
@@ -63,7 +63,7 @@
     "fs-xattr": "0.3.0"
   },
   "dependencies": {
-    "@journeyapps/sqlcipher": "https://github.com/EvanHahn-signal/node-sqlcipher.git#16916949f0c010f6e6d3d5869b10a0ab813eae75",
+    "@journeyapps/sqlcipher": "https://github.com/diegoe/node-sqlcipher.git#c30f38722dcc97a497835c234bdc913a38710160",
     "@sindresorhus/is": "0.8.0",
     "@types/pino": "6.3.6",
     "@types/pino-multi-stream": "5.1.0",
diff --git a/yarn.lock b/yarn.lock
index 02670e08..0fd91e61 100644
--- a/yarn.lock
+++ b/yarn.lock
@@ -1336,9 +1336,9 @@
   resolved "https://registry.yarnpkg.com/@icons/material/-/material-0.2.4.tgz#e90c9f71768b3736e76d7dd6783fc6c2afa88bc8"
   integrity sha512-QPcGmICAPbGLGb6F/yNf/KzKqvFx8z5qx3D1yFqVAjoFmXK35EgyW+cJ57Te3CNsmzblwtzakLGFqHPqrfb4Tw==
 
-"@journeyapps/sqlcipher@https://github.com/EvanHahn-signal/node-sqlcipher.git#16916949f0c010f6e6d3d5869b10a0ab813eae75":
+"@journeyapps/sqlcipher@https://github.com/diegoe/node-sqlcipher.git#c30f38722dcc97a497835c234bdc913a38710160":
   version "5.0.0"
-  resolved "https://github.com/EvanHahn-signal/node-sqlcipher.git#16916949f0c010f6e6d3d5869b10a0ab813eae75"
+  resolved "https://github.com/diegoe/node-sqlcipher.git#c30f38722dcc97a497835c234bdc913a38710160"
   dependencies:
     node-addon-api "^3.0.0"
     node-pre-gyp "^0.15.0"

@cryptomilk
Copy link
Contributor

Looks like this needs rebasing!

@WhyNotHugo
Copy link
Contributor Author

WhyNotHugo commented Apr 9, 2021

There's other dependencies that don't build for me:

error /home/hugo/workspace/Forks/Signal-Desktop/node_modules/@journeyapps/sqlcipher: Command failed.
Exit code: 1
Command: node-pre-gyp install --build-from-source
Arguments:
Directory: /home/hugo/workspace/Forks/Signal-Desktop/node_modules/@journeyapps/sqlcipher
Output:
internal/fs/utils.js:269
    throw err;
    ^

Error: ENOENT: no such file or directory, open '/home/hugo/workspace/Forks/Signal-Desktop/node_modules/@journeyapps/sqlcipher/package.json'
    at Object.openSync (fs.js:462:3)
    at Object.readFileSync (fs.js:364:35)
    at Run.parseOpts [as parseArgv] (/home/hugo/workspace/Forks/Signal-Desktop/node_modules/node-pre-gyp/lib/node-pre-gyp.js:136:36)
    at Object.<anonymous> (/home/hugo/workspace/Forks/Signal-Desktop/node_modules/node-pre-gyp/bin/node-pre-gyp:24:6)
    at Module._compile (internal/modules/cjs/loader.js:1137:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1157:10)
    at Module.load (internal/modules/cjs/loader.js:985:32)
    at Function.Module._load (internal/modules/cjs/loader.js:878:14)
    at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:71:12)
    at internal/main/run_main_module.js:17:47 {
  errno: -2,
  syscall: 'open',

I've wasted too many hours on this project, and maintainers clearly don't care about contributions. There's plenty of references to this PR by many contributors hitting this issue, yet, much like on many other PRs, maintainers just don't care. Even though this PR has been in a "ready" state for months.

Apparently, expectation is that people to build and install ancient, unsupported software in order to develop on Signal (just to later ignore contributions anyway).

@hiqua
Copy link
Contributor

hiqua commented Apr 9, 2021

I don't think it makes sense to rebase until the devs have approved this change at least in spirit.

Once (if) they have, I'm willing to do the rebase.

Copy link
Contributor

@EvanHahn-Signal EvanHahn-Signal left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a good change that we'd like to accept. Could you rebase and we can get it merged?

(BTW, we plan to do one more release with node-gyp@5, so we may not merge this until v5.0.0 goes to production.)

The currently pinned version of `node-gyp` requires Python 2.  Python 2
has reached its EOL a long time ago and is unsupported by upstream.

The requirement on Python 2 is also an unnecessary barrier for new
developers to contribute to Signal.

This changeset updates the required version of node-gyp to one
compatible with Python 3.

`CONTRIBUTING.md` remains unchanged, since this dependency was not
previously mentioned anyway.

Supersedes #4407
Fixes #4783
@WhyNotHugo
Copy link
Contributor Author

I've rebased, but can't fully tests locally. yarn fails for me with:

error /home/hugo/workspace/Forks/Signal-Desktop/node_modules/@journeyapps/sqlcipher: Command failed.
Exit code: 1
Command: node-pre-gyp install --build-from-source
Arguments:
Directory: /home/hugo/workspace/Forks/Signal-Desktop/node_modules/@journeyapps/sqlcipher
Output:
internal/fs/utils.js:269
    throw err;
    ^

Error: ENOENT: no such file or directory, open '/home/hugo/workspace/Forks/Signal-Desktop/node_modules/@journeyapps/sqlcipher/package.json'
    at Object.openSync (fs.js:462:3)
    at Object.readFileSync (fs.js:364:35)
    at Run.parseOpts [as parseArgv] (/home/hugo/workspace/Forks/Signal-Desktop/node_modules/node-pre-gyp/lib/node-pre-gyp.js:136:36)
    at Object.<anonymous> (/home/hugo/workspace/Forks/Signal-Desktop/node_modules/node-pre-gyp/bin/node-pre-gyp:24:6)
    at Module._compile (internal/modules/cjs/loader.js:1137:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1157:10)
    at Module.load (internal/modules/cjs/loader.js:985:32)
    at Function.Module._load (internal/modules/cjs/loader.js:878:14)
    at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:71:12)
    at internal/main/run_main_module.js:17:47 {
  errno: -2,
  syscall: 'open',

However, I also have this problem on development, so it's unrelated to this PR.

@hiqua
Copy link
Contributor

hiqua commented Apr 9, 2021

I've rebased, but can't fully tests locally. yarn fails for me with:

Awesome, thanks!

As long as it succeeds on GH it should be ok, otherwise I can help out. It might be about deleting the node_modules and starting from scratch...

@EvanHahn-Signal
Copy link
Contributor

This is likely an issue with git lfs. You can try this in the root of the repo:

# make sure you have git lfs installed
git-lfs install
yarn cache clean
rm -rf node_modules
yarn install --frozen-lockfile
yarn grunt

@WhyNotHugo
Copy link
Contributor Author

I'd cleaned via git clean -xdf. Running git-lfs install fixed the issue.
It's odd though, since I'd certainly run that in the past.

yarn ready passes now.

@EvanHahn-Signal
Copy link
Contributor

Thanks! We'll plan to merge this after v5.0.0 is released to production.

@scottnonnenberg-signal scottnonnenberg-signal merged commit 31a777a into signalapp:development Apr 28, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

Support modern Python versions
7 participants