clef: failing to invoke the UI API after reading the docs

[attila-lendvai]

System information

On Guix (Linux):

Geth
Version: 1.10.18-stable
Git Commit: de23cf910b814867d5c5d1ad6164835d79069638
Git Commit Date: 20220525
Architecture: amd64
Go Version: go1.18.1
Operating System: linux
GOPATH=
GOROOT=go

Expected behaviour

clef is started like this (as captured from ps afxu, see below):

/path/geth-binary-1.10.18-clef/bin/clef --stdio-ui --keystore /var/lib/swarm/mainnet/clef/keystore
 --configdir /var/lib/swarm/mainnet/clef --chainid 100 --rules /path/rules.js --nousb --lightkdf
 --4bytedb-custom /path/4byte.json --pcscdpath '' --auditlog '' --loglevel 3
 --ipcpath /var/lib/swarm/mainnet/clef

then:

echo '{"id": 0, "jsonrpc": "2.0", "method": "clef_listAccounts", "params": []}' >>/var/lib/swarm/mainnet/clef/stdin

to which clef should respond with the account details in a json to its stdout.

Actual behaviour

in the terminal, where clef is running, i see the following:

WARN [06-10|13:13:15.423] Served clef_listAccounts                 conn=/dev/stdin reqid=4 duration="27.262µs" err="the method clef_listAccounts does not exist/is not available"
{"jsonrpc":"2.0","id":4,"error":{"code":-32601,"message":"the method clef_listAccounts does not exist/is not available"}}

same happens when using ui_listAccounts (the doc is not consistent about the names).

My ultimate goal

is to be able to create new accounts as needed (for running a dynamic number of Bee nodes), while clef is already started and running (i.e. i cannot just spawn a new clef process and request the new account creation from the command line... or can i run multiple clef instances using the same --configdir and --keystore? i guess i shouldn't, even if it works.)

i also cannot add a rule for new account creation (because it needs a password argument), which leads to a convoluted, multi-step interaction when using the RPC API.

Sidenote on starting clef

the actual starting of the clef process is more convoluted. this may interfere, but i don't suspect it:

mkfifo ${DATA_DIR}/stdin ${DATA_DIR}/stdout
(
exec 3>${DATA_DIR}/stdin
while read < ${DATA_DIR}/stdout; do
  if [[ "$REPLY" =~ "enter the password" ]]; then
    echo '{ "jsonrpc": "2.0", "id":1, "result": { "text":"'"$CLEF_PASSWORD"'" } }' > ${DATA_DIR}/stdin
    # We don't break the piping, so that it remains possible to echo 'foo' >stdin.
    # break
  fi
done
) &

exec ${CLEF} --stdio-ui --keystore "${KEYSTORE}" --configdir "${DATA_DIR}" --chainid "${CHAINID}" --rules "${RULES}" --nousb --lightkdf --4bytedb-custom "${FOURBYTE}" --pcscdpath "" --auditlog "" --loglevel 3 --ipcpath "${DATA_DIR}" < ${DATA_DIR}/stdin | tee ${DATA_DIR}/stdout

[holiman]

The ui_ namespace and the clef namespace are two different things. We have to different APIs,

1. External API.
2. Internal/privileged API.

In the internal API, there is a bidirectionality. The clef binary can ask for things from the ui using the ui_ namespace. For example when it needs a password. And the ui, or "wrapper", can ask for things using the clef_ api.
So I don't think there's any inconsistency in the docs).

---

I can repro your case though.

[user@work clique_clef]$ clef --keystore /tmp/foo  --chainid 15  --suppress-bootwarn --stdio-ui
INFO [07-08|14:56:21.883] Using stdin/stdout as UI-channel 
INFO [07-08|14:56:22.172] Loaded 4byte database                    embeds=268,621 locals=0 local=./4byte-custom.json
{"jsonrpc":"2.0","id":1,"method":"ui_onInputRequired","params":[{"title":"Master Password","prompt":"Please enter the password to decrypt the master seed","isPassword":true}]}
{  "id": 13,  "jsonrpc": "2.0",  "method": "clef_listWallets"}
WARN [07-08|14:56:31.544] Served clef_listWallets                  conn=/dev/stdin reqid=13 duration="31.665µs" err="the method clef_listWallets does not exist/is not available"
{"jsonrpc":"2.0","id":13,"error":{"code":-32601,"message":"the method clef_listWallets does not exist/is not available"}}

But if I tell it to use a signer-secret, instead of asking the ui for it, things work:

[user@work clique_clef]$ clef --keystore /tmp/foo  --chainid 15  --suppress-bootwarn --stdio-ui --signersecret /tmp/foo
INFO [07-08|14:56:49.968] Using stdin/stdout as UI-channel 
INFO [07-08|14:56:50.272] Loaded 4byte database                    embeds=268,621 locals=0 local=./4byte-custom.json
WARN [07-08|14:56:50.272] Failed to open master, rules disabled    err="failed stat on /tmp/foo: stat /tmp/foo: no such file or directory"
INFO [07-08|14:56:50.272] Starting signer                          chainid=15 keystore=/tmp/foo light-kdf=false advanced=false
INFO [07-08|14:56:50.278] IPC endpoint opened                      url=/home/user/.clef/clef.ipc
{"jsonrpc":"2.0","method":"ui_onSignerStartup","params":[{"info":{"extapi_http":"n/a","extapi_ipc":"/home/user/.clef/clef.ipc","extapi_version":"6.1.0","intapi_version":"7.0.1"}}]}
...
DEBUG[07-08|14:56:53.894] Served clef_listWallets                  conn=/dev/stdin reqid=13 duration="155.286µs"
{"jsonrpc":"2.0","id":13,"result":[]}

When clef needs input for something, you need to provide it, I guess

[holiman]

Yeah, it tries to read the master key early on, before registering the RPC/IPC hooks

	configDir := c.GlobalString(configdirFlag.Name)
	if stretchedKey, err := readMasterKey(c, ui); err != nil {
		log.Warn("Failed to open master, rules disabled", "err", err)
...

	// Establish the bidirectional communication, by creating a new UI backend and registering
	// it with the UI.
	ui.RegisterUIServer(core.NewUIServerAPI(apiImpl))

So if you don't provide a master key, it will simply be stuck on that.

[attila-lendvai]

@holiman there are some issues with your test above, but it's not relevant anymore. i have found the reason it doesn't work for me:

i'm providing a --rules file for clef, and in that case it doesn't respond to the clef_ requests.

i'm not sure i understand clef's design well enough to judge whether that is intended behavior, or not. is it?

this is my rules.js, although it's probably not relevant:

function OnSignerStartup() {
    return "Approve"
}
function OnApprovedTx() {
    return "Approve"
}
function ApproveListing() {
    return "Approve"
}
function ApproveTx() {
    return "Approve"
}
function ApproveSignData() {
    return "Approve"
}

note that, IIUC, i cannot add a new rule for this, because clef needs a password for the new account, and it cannot be provided through the IPC API call.

so, i'm back to my original high-level goal: how can i request a running clef to create a new account in a single step (i.e. without getting into a convoluted multi-step interaction that is communicated on two channels, namely 1) on the IPC to request the new account, and 2) the stdio ui for providing the password for the new account)?

with my limited understanding, i see the following options to fix/implement this:

1. extend the account_new IPC call so that a password can be provided
2. keep listening for the clef_ API calls on the stdio UI even when a --rules files is used
3. maybe i could implement what's currently in the rules.js file through the stdio ui? i'd like to avoid that, because currently i'm using the vanilla rules.js file from upstream, and i'd like to avoid diverging if possible.

any insights on which route i should take? maybe 3) is not as bad as it seems at first sight?

with some guidance/help, i'm willing to do 1), too. but grepping the source for account_new yields no insights.

random tangential: it took me some time to understand that these are the same thing: the file is called masterseed.json, the function opening it is called readMasterKey, and the CLI flag is called --signersecret.

[attila-lendvai]

FTR, i have reported the above as a new issue at: #25298