Could not run as a plugin, but works perfectly as a standalone service #116

sethforprivacy · 2022-05-19T20:05:21Z

Hi there, just finished building out a Core LN + c-lightning-REST + RTL setup, and ran into this error when trying to use c-lightning-REST as a plugin for lightningd:

plugin-plugin.js: Killing plugin: exited before replying to getmanifest

The plugin works perfectly when run as it's own service (systemd script here, config file here), but no matter what I tried lightningd would crash instantly on trying to load the plugin.

Here are the commands I ran before adding it to the lightningd config file:

    5  wget https://github.com/Ride-The-Lightning/c-lightning-REST/releases/download/v0.7.2/v0.7.2.tar.gz.asc
    6  tar -xvf v0.7.2.tar.gz
    7  cd c-lightning-REST-0.7.2
    8  npm install --only=prod

Then I added via full path as specified here: https://github.com/Ride-The-Lightning/c-lightning-REST#option-2-run-as-c-lightning-plugin

Not sure how to get better logs, but happy to collect what I can for you!

The text was updated successfully, but these errors were encountered:

saubyk · 2022-05-20T15:53:48Z

Hi @sethforprivacy is it possible for you to share your core lightning config settings? It should be in the .lightning folder.

sethforprivacy · 2022-05-20T16:04:45Z

Hi @sethforprivacy is it possible for you to share your core lightning config settings? It should be in the .lightning folder.

Here it is:

https://paste.sethforprivacy.com/?3e2080ae7cbf5ad0#BMcHpNDXn4rtv8SjQ8sd2HNTxxjMbqkYyLisHGdYgEU

When I was trying c-lightning-rest as a plugin I called it via full-path, i.e.:

important-plugin=/home/lightningd/c-lightning-REST-0.7.2/plugin.js
rest-port=3001
rest-docport=4001
rest-protocol=https

My best guess would be it being related to lightningd not properly recognizing the location of the node binary, but nothing I could find to resolve that worked.

saubyk · 2022-05-20T22:08:10Z

what all do you have in the lightningd folder?
The only difference that I could find in my settings is that I am keeping the c-lightning-REST folder separate from all of core lightning's and plugin folders.

Keep in mind that C-Lightning-REST is not an "official" plugin, so not sure if there is manifest where it needs to be registered. But that's not required for it to run as a plugin.

May be, I need to get it registered as an official plugin (not sure if that's a thing either) 😄

devastgh · 2022-05-21T09:33:38Z

Adding my 2c, maybe it helps:
C-ln tries to load every executable it finds in ~/.lightning/plugins/* .
So having plugin-plugin.js: Killing plugin: exited before replying to getmanifest is expected, if you have any executable in that path, that is not a c-ln plugin. I think loading the plugin might fail, cause it's not an executable.
Could you check what permissions you have on plugin.js ? If it does not have executable flag on the file, you need to chmod +x plugin.js

litch · 2022-05-21T19:48:40Z

I just got bit by the same exact issue - where the plugin config in the .lightning/config seems incorrect.

May 21 19:45:22  lightningd[871350]: 2022-05-21T19:45:22.631Z INFO    plugin-plugin.js: Killing plugin: exited before replying to getmanifest
May 21 19:45:22  lightningd[871350]: lightningd: /home/litch/.lightning/config line 21: rest-port=3002: unrecognized option

saubyk · 2022-05-22T13:55:02Z

I just got bit by the same exact issue - where the plugin config in the .lightning/config seems incorrect.

May 21 19:45:22  lightningd[871350]: 2022-05-21T19:45:22.631Z INFO    plugin-plugin.js: Killing plugin: exited before replying to getmanifest
May 21 19:45:22  lightningd[871350]: lightningd: /home/litch/.lightning/config line 21: rest-port=3002: unrecognized option

Hi @litch can you share your config file settings?

litch · 2022-05-22T15:21:04Z

The config file sections were exactly as in the readme:

mainnet
bitcoin-rpcuser=<>
bitcoin-rpcpassword=<>
bitcoin-rpcconnect=<>
bitcoin-rpcport=8332

proxy=127.0.0.1:9050
announce-addr=<>:9735

large-channels
experimental-dual-fund
funder-policy=match
funder-policy-mod=100
lease-fee-base-msat=500sat
lease-fee-basis=52
channel-fee-max-base-msat=5sat
channel-fee-max-proportional-thousandths=1


clboss-auto-close=true
clboss-min-onchain=10000000

plugin=/home/<>/c-lightning-REST/plugin.js
rest-port=3002
rest-docport=4001
rest-protocol=https

Similarly to @sethforprivacy - it works swimmingly if I run it otherwise.

sethforprivacy · 2022-05-23T12:26:45Z

@saubyk - I don't have any plugins in there, just this structure:

tiny in tiny in ~ on  master [!?] took 5s ❯ sudo tree /etc/lightningd
/etc/lightningd
├── lightningd.conf
├── lightningd.log
└── lightning-rpc

0 directories, 3 files
tiny in tiny in ~ on  master [!?] ❯ sudo tree /home/lightningd/.lightning
/home/lightningd/.lightning
└── bitcoin
    ├── backup.lock
    ├── crash.log.20220517195817
    ├── crash.log.20220522165328
    ├── data.clboss
    ├── gossip_store
    ├── hsm_secret
    ├── keys.clboss
    ├── lightningd.sqlite3
    └── lightning-rpc

1 directory, 9 files

I was loading the c-lightning-REST plugin from a directory in /home/lightningd (where it's running from now as a standalone app without issues under the same user as lightningd).

@devastgh - plugin.js is properly permissioned for the user, and has the execute bit set:

-rwxrwxr-x   1 lightningd lightningd   1698 May  9 22:31 plugin.js

Happy to help however I can!

sethforprivacy · 2022-08-15T12:39:57Z

Still the same issue with v0.8.0:

2022-08-15T12:34:26.952Z DEBUG   lightningd: Opened log file /etc/lightningd/lightningd.log
2022-08-15T12:34:27.319Z INFO    plugin-plugin.js: Killing plugin: exited before replying to getmanifest
2022-08-15T12:34:27.319Z **BROKEN** plugin-plugin.js: Plugin marked as important, shutting down lightningd!

Config tested:

# c-lightning-REST settings
important-plugin=/home/lightningd/c-lightning-REST-0.8.0/plugin.js
rest-port=3001
rest-docport=4001
rest-protocol=http
rest-domain=cln.sethforprivacy.com

Folder structure for cln-REST:

sethforprivacy in apps in ~ on  master [!?] via  v16.16.0 ❯ sudo tree -L 1 /home/lightningd/c-lightning-REST-0.8.0/
/home/lightningd/c-lightning-REST-0.8.0/
├── amd64.Dockerfile
├── app.js
├── arm32v7.Dockerfile
├── arm64v8.Dockerfile
├── certs
├── cl-rest-config.json
├── cl-rest.js
├── controllers
├── docapp.js
├── docker-entrypoint.sh
├── docs
├── LICENSE
├── lightning-client-js.js
├── methods.json
├── node_modules
├── package.json
├── package-lock.json
├── plugin.js
├── README.md
├── routes
├── sample-cl-rest-config.json
├── screenshots
├── test
└── utils

devastgh · 2022-08-15T12:47:34Z

Could you just check it without any options specified in c-lightning config?
If it works, set the options one by one, to check which one doesn't work.
I assure you it loads fine as a plugin for me without any options specified.

sethforprivacy · 2022-08-15T12:57:46Z

Could you just check it without any options specified in c-lightning config? If it works, set the options one by one, to check which one doesn't work. I assure you it loads fine as a plugin for me without any options specified.

Same issue, doesn't help:

# c-lightning-REST settings
important-plugin=/home/lightningd/c-lightning-REST-0.8.0/plugin.js
#rest-port=3001
#rest-docport=4001
#rest-protocol=http
#rest-domain=cln.sethforprivacy.com

sethforprivacy in apps in ~ on  master [!?] via  v16.16.0 ❯ tail -f /etc/lightningd/lightningd.log
2022-08-15T12:56:55.282Z DEBUG   lightningd: Opened log file /etc/lightningd/lightningd.log
2022-08-15T12:56:55.427Z INFO    plugin-plugin.js: Killing plugin: exited before replying to getmanifest
2022-08-15T12:56:55.427Z **BROKEN** plugin-plugin.js: Plugin marked as important, shutting down lightningd!




2022-08-15T12:56:55.747Z DEBUG   lightningd: Opened log file /etc/lightningd/lightningd.log
2022-08-15T12:56:55.893Z INFO    plugin-plugin.js: Killing plugin: exited before replying to getmanifest
2022-08-15T12:56:55.893Z **BROKEN** plugin-plugin.js: Plugin marked as important, shutting down lightningd!

devastgh · 2022-08-16T09:17:23Z

Maybe try it with just plugin= not important.
Also, try to load it while the file cl-rest-config.json is not present in c-lightning-REST directory.
If it still fails to load, then i'm out of ideas.

jochemin · 2022-11-02T08:01:52Z

Same error on fresh install, it works when run as it's own service but fails trying to run it as plugin.

ShahanaFarooqui · 2022-11-04T16:07:11Z

Could you run the plugin after adding rest-execmode=test in the config. It might provide us better insights on the failure reason.

I tried all above mentioned scenarios on my local Ubuntu 22 and they all are working successfully. It seems like some local setting issue.

sethforprivacy · 2022-11-04T16:08:19Z

I'll test again ASAP.

sethforprivacy · 2022-11-04T16:16:11Z

@ShahanaFarooqui:

Nov 04 16:15:22 lightningd[2517231]: # Fatal error in , line 0
Nov 04 16:15:22 lightningd[2517231]: # Check failed: reservation_.SetPermissions(protect_start, protect_size, permission).

sethforprivacy · 2022-11-04T16:16:37Z

Still fails to start the plugin, above seems the error returned with execmode=test set.

sethforprivacy · 2022-11-04T16:17:56Z

The only thing I can think of that may be causing this is the recommended hardening in the lightningd systemd script:

# Hardening measures
####################

# Provide a private /tmp and /var/tmp.
PrivateTmp=true

# Disallow the process and all of its children to gain
# new privileges through execve().
NoNewPrivileges=true

# Use a new /dev namespace only populated with API pseudo devices
# such as /dev/null, /dev/zero and /dev/random.
PrivateDevices=true

# Deny the creation of writable and executable memory mappings.
MemoryDenyWriteExecute=true

Testing without it now.

sethforprivacy · 2022-11-04T16:19:46Z

Disabling those protections resolved the issue, will nail down which one is to blame shortly.

sethforprivacy · 2022-11-04T16:23:20Z

Narrowed down to this:

# Deny the creation of writable and executable memory mappings.
MemoryDenyWriteExecute=true

If the user has this hardening measure enabled in their systemd script, c-lightning-REST cannot function as a plugin.

Disabling the hardening has everything working perfectly!

Ideally there is some way you can have both this hardening and the plugin, but that's far above my pay grade.

ShahanaFarooqui · 2022-11-04T16:30:07Z

@sethforprivacy Thank you for the quick testing and solution.

@jochemin & @litch Could you also verify your permissions or add rest-execmode=test to get some failure insights. I am keeping this issue open till then.

litch · 2022-11-04T16:34:00Z

I was also running the "hardening" systemd script, and that makes perfect sense as to the cause.

I echo the, "I would prefer to run this and have the thing hardened" but likewise don't have enough insight into the changes required to have a strong opinion about whether that would be better than, perhaps a log message that detects that flag and tells the user that that is the cause of the error.

jochemin · 2022-11-04T18:24:21Z

No luck for me. This is my systemd service:

`# /etc/systemd/system/lightningd.service

[Unit]
Description=Core Lightning daemon
Requires=bitcoind.service
After=bitcoind.service
Wants=network.target
After=network.target

[Service]
ExecStart=/home/jochemin/clightning/usr/bin/lightningd --daemon
ExecStop=/bin/sh -c '/home/jochemin/clightning/usr/bin/lightning-cli stop'

`# Process management
####################
Type=forking
Restart=on-failure
TimeoutSec=300
RestartSec=30

`# Directory creation and permissions
####################################
User=jochemin
Group=adm
Environment=PATH=/home/jochemin/clightning/usr/bin:/home/jochemin/bitcoin/bin

# Hardening measures #################### # Provide a private /tmp and /var/tmp.
PrivateTmp=true

`# Mount /usr, /boot/ and /etc read-only for the process.
ProtectSystem=full

# Disallow the process and all of its children to gain # new privileges through execve().
NoNewPrivileges=true

# Use a new /dev namespace only populated with API pseudo devices # such as /dev/null, /dev/zero and /dev/random.
PrivateDevices=true

# Deny the creation of writable and executable memory mappings. #MemoryDenyWriteExecute=true

[Install]
WantedBy=multi-user.target`

I've commented MemoryDenyWriteExecute but same error:

2022-11-04T18:21:08.430Z INFO plugin-plugin.js: Killing plugin: exited before replying to getmanifest

sethforprivacy · 2022-11-04T18:27:16Z

Try also commenting out this line, I realized I had it commented out already for other issues:

ProtectSystem=full

ShahanaFarooqui · 2022-11-04T18:29:46Z

@jochemin Also add rest-execmode=test in your lightning config file to see the failure log from this plugin.

jochemin · 2022-11-04T18:33:42Z

nov 04 19:32:02 jbtc lightningd[537230]: /usr/bin/env: 'node': No such file or directory
nov 04 19:32:02 jbtc lightningd[537213]: lightningd: /home/jochemin/.lightning/config line 16: rest-execmode=test: unrecognized option

It seems a problem with node executable.

I've checked and I have node executable in /user/bin

Out of ideas

jochemin · 2022-11-04T18:38:52Z

Try also commenting out this line, I realized I had it commented out already for other issues:

ProtectSystem=full

Nope, same error.

ShahanaFarooqui · 2022-11-04T19:45:20Z

@jochemin It either looks like an incorrect naming issue which can be resolved via symlink OR nodejs-legacy installation might work.

Below are some links for your reference related to these solutions:
nodejs/node-v0.x-archive#3911 (comment)
nodejs/node-v0.x-archive#3911 (comment)
https://askubuntu.com/questions/1044692/usr-bin-env-node-no-such-file-or-directory-even-though-it-is-in-path
https://stackoverflow.com/questions/30281057/node-forever-usr-bin-env-node-no-such-file-or-directory

jochemin · 2022-11-07T07:00:37Z

/usr/bin/env: 'node': No such file or directory

Thanks @ShahanaFarooqui , no luck for me. My executable name is node so no naming isuue, I installed nodejs from Ubuntu repositories with sudo apt-install nodejs.

I've tried to install last LTS version but the error continues.

Mi SO is Ubuntu Server 22.04 and node version is 18.12.1

I'll keep trying.

ShahanaFarooqui · 2022-11-07T15:43:11Z

@jochemin Thank you for the update. Please post your solution, if you find any, as it might help other users.

I am closing this issue now as there is no action required from us anymore.

jochemin · 2022-11-09T07:45:28Z

I've solved it. The problem was I had this line in the systemd service:

Environment=PATH=/home/jochemin/clightning/usr/bin:/home/jochemin/bitcoin/bin

But node executable is in /usr/bin so it was not accesible. I've added that route and now it works perfect!

…de.js plugins See-also: Ride-The-Lightning/c-lightning-REST#116 Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

saubyk added the bug Something isn't working label May 20, 2022

ShahanaFarooqui closed this as completed Nov 7, 2022

rustyrussell mentioned this issue Nov 11, 2022

lightningd.service: note that the hardening setting seems to break no… ElementsProject/lightning#5706

Merged

cdecker pushed a commit to ElementsProject/lightning that referenced this issue Nov 11, 2022

lightningd.service: note that the hardening setting seems to break no…

d45b13d

…de.js plugins See-also: Ride-The-Lightning/c-lightning-REST#116 Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

ddustin pushed a commit to ddustin/lightning that referenced this issue Apr 11, 2023

lightningd.service: note that the hardening setting seems to break no…

dc465f3

…de.js plugins See-also: Ride-The-Lightning/c-lightning-REST#116 Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

ddustin pushed a commit to ddustin/lightning that referenced this issue May 12, 2023

lightningd.service: note that the hardening setting seems to break no…

1536166

…de.js plugins See-also: Ride-The-Lightning/c-lightning-REST#116 Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Could not run as a plugin, but works perfectly as a standalone service #116

Could not run as a plugin, but works perfectly as a standalone service #116

sethforprivacy commented May 19, 2022

saubyk commented May 20, 2022

sethforprivacy commented May 20, 2022 •

edited

Loading

saubyk commented May 20, 2022

devastgh commented May 21, 2022

litch commented May 21, 2022 •

edited

Loading

saubyk commented May 22, 2022

litch commented May 22, 2022

sethforprivacy commented May 23, 2022

sethforprivacy commented Aug 15, 2022

devastgh commented Aug 15, 2022 •

edited

Loading

sethforprivacy commented Aug 15, 2022

devastgh commented Aug 16, 2022 •

edited

Loading

jochemin commented Nov 2, 2022

ShahanaFarooqui commented Nov 4, 2022

sethforprivacy commented Nov 4, 2022

sethforprivacy commented Nov 4, 2022

sethforprivacy commented Nov 4, 2022

sethforprivacy commented Nov 4, 2022

sethforprivacy commented Nov 4, 2022

sethforprivacy commented Nov 4, 2022

ShahanaFarooqui commented Nov 4, 2022

litch commented Nov 4, 2022

jochemin commented Nov 4, 2022 •

edited

Loading

sethforprivacy commented Nov 4, 2022

ShahanaFarooqui commented Nov 4, 2022

jochemin commented Nov 4, 2022 •

edited

Loading

jochemin commented Nov 4, 2022

ShahanaFarooqui commented Nov 4, 2022

jochemin commented Nov 7, 2022

ShahanaFarooqui commented Nov 7, 2022

jochemin commented Nov 9, 2022

Could not run as a plugin, but works perfectly as a standalone service #116

Could not run as a plugin, but works perfectly as a standalone service #116

Comments

sethforprivacy commented May 19, 2022

saubyk commented May 20, 2022

sethforprivacy commented May 20, 2022 • edited Loading

saubyk commented May 20, 2022

devastgh commented May 21, 2022

litch commented May 21, 2022 • edited Loading

saubyk commented May 22, 2022

litch commented May 22, 2022

sethforprivacy commented May 23, 2022

sethforprivacy commented Aug 15, 2022

devastgh commented Aug 15, 2022 • edited Loading

sethforprivacy commented Aug 15, 2022

devastgh commented Aug 16, 2022 • edited Loading

jochemin commented Nov 2, 2022

ShahanaFarooqui commented Nov 4, 2022

sethforprivacy commented Nov 4, 2022

sethforprivacy commented Nov 4, 2022

sethforprivacy commented Nov 4, 2022

sethforprivacy commented Nov 4, 2022

sethforprivacy commented Nov 4, 2022

sethforprivacy commented Nov 4, 2022

ShahanaFarooqui commented Nov 4, 2022

litch commented Nov 4, 2022

jochemin commented Nov 4, 2022 • edited Loading

sethforprivacy commented Nov 4, 2022

ShahanaFarooqui commented Nov 4, 2022

jochemin commented Nov 4, 2022 • edited Loading

jochemin commented Nov 4, 2022

ShahanaFarooqui commented Nov 4, 2022

jochemin commented Nov 7, 2022

ShahanaFarooqui commented Nov 7, 2022

jochemin commented Nov 9, 2022

sethforprivacy commented May 20, 2022 •

edited

Loading

litch commented May 21, 2022 •

edited

Loading

devastgh commented Aug 15, 2022 •

edited

Loading

devastgh commented Aug 16, 2022 •

edited

Loading

jochemin commented Nov 4, 2022 •

edited

Loading

jochemin commented Nov 4, 2022 •

edited

Loading