Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: improve general performance by reducing the total numbers of call to garbage collector #969

Merged
merged 4 commits into from
Nov 20, 2024
Merged

feat: improve general performance by reducing the total numbers of call to garbage collector #969

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
0f795a5
refactor: try to improve performance
matteo-cristino Nov 19, 2024
588ab41
feat: on zencode run perform garbage collection only if memory being …
matteo-cristino Nov 19, 2024
5d0e2a8
fix: typo in default value of maxmem
matteo-cristino Nov 19, 2024
138dcea
docs: update config page
matteo-cristino Nov 20, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
164 changes: 117 additions & 47 deletions docs/pages/zenroom-config.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,87 +1,157 @@
## Usage

Zenroom configuration is used to set some application wide parameters at initialization time. The configuration is passed as a parameter (not as file!) using the "-c" option, or as a parameter if Zenroom is used as lib. You can pass Zenroom several attributes, wrapped in quotes and separated by a comma, as in the example below:
Zenroom configuration is used to set some application wide parameters at initialization time.
The configuration is passed as a parameter (not as file!) using the "-c" option, or as a parameter
if Zenroom is used as lib. You can pass Zenroom several attributes, wrapped in quotes and separated
by a comma, as in the example below:

```shell
zenroom -z keypair.zen -c "debug=3, rngseed=hex:74eeeab870a394175fae808dd5dd3b047f3ee2d6a8d01e14bff94271565625e98a63babe8dd6cbea6fedf3e19de4bc80314b861599522e44409fdd20f7cd6cfc"
zenroom -z keyring.zen -c "debug=3, rngseed=hex:74eeeab870a394175fae808dd5dd3b047f3ee2d6a8d01e14bff94271565625e98a63babe8dd6cbea6fedf3e19de4bc80314b861599522e44409fdd20f7cd6cfc"
```

Below a list of the config parameters with a description and usage examples.

## Debug verbosity

### Syntax and values: **debug=1, 2, 3**
Syntax and values: **debug=1|2|3** or **verbose=1|2|3**

Define the verbosity of Zenroom's output, bug reports need to be sent with "debug=3"
*debug* and *verbose* are synonyms. They define the verbosity of Zenroom's output.
Moreove if this value is grater than 1 the zenroom watchdog is activated and at each step a check on all internal
data is performed to assert all values in memory are converted to zenroom types.

*Default*: 2

## Scope

## Color

### Syntax and values: **color=1, 0**

Defines color from printout log, "color=0" removes colors.


## Seccomp (CLI isolation)

### Syntax and values: **seccomp=0, 1**

Secure execution isolation for the CLI: separates the CLI to the OS kernel, file system and network in a provable way. Works only in Linux.
Syntax and values: **scope=given|full**

Scope represent wich part of the zenroom contract should be executed. When it is set to *full*, that is the default value,
all the contract is run, on the other hand when it is *given* only the given part is run and the result is the CODEC
status after the given phase, it can be used to know what data a user needs to pass to the contract.

*Default*: full.

## Random seed

### Syntax and values: **rngseed=hex:[64 bytes in hex notation]**
Syntax and values: **rngseed=hex:[64 bytes in hex notation]**

Loads a random seed at start, that will be used through the Zenroom execution whenever a random seed is requested. A fixed random can be used to test determinism. For example, when generating an ECDH keypair, using:
Loads a random seed at start, that will be used through the Zenroom execution whenever a random seed is requested.
A fixed random can be used to test determinism. For example, when generating an ECDH key, using:


```shell
rngseed=hex:74eeeab870a394175fae808dd5dd3b047f3ee2d6a8d01e14bff94271565625e98a63babe8dd6cbea6fedf3e19de4bc80314b861599522e44409fdd20f7cd6cfc
```

Should always generate the keypair:
Should always generate the keyring:

```json
{
"Alice": {
"keypair": {
"private_key": "Aku7vkJ7K01gQehKELav3qaQfTeTMZKgK+5VhaR3Ui0=",
"public_key": "BBCQg21VcjsmfTmNsg+I+8m1Cm0neaYONTqRnXUjsJLPa8075IYH+a9w2wRO7rFM1cKmv19Igd7ntDZcUvLq3xI="
}
{
"keyring": {
"private_key": "Aku7vkJ7K01gQehKELav3qaQfTeTMZKgK+5VhaR3Ui0="
}

}
```

## Limit iterations
```

### Syntax and values: **maxiter=dec:[at most 10 decimal digits]**
## Log format

Define the maximum number of iterations the contract is allowed to do.
Syntax and values: **logfmt=text|json**

When using *text* as log format, all the log is printed as a text, while using the *json* it is an array,
where, when an error happens, the trace and heap are base64 encoded and can be found respectively in the
values starting with *J64 TRACE:* and *J64 HEAP:*.

Example of log in text:
```
Release version: v4.36.0
Build commit hash: 760ca7bb
Memory manager: libc
ECDH curve is SECP256K1
ECP curve is BLS381
[W] Zencode is missing version check, please add: rule check version N.N.N
[W] {
KEYRING = {
bbs = int[32] ,
bitcoin = octet[32] ,
dilithium = octet[2528] ,
ecdh = octet[32] ,
eddsa = octet[32] ,
es256 = octet[32] ,
ethereum = octet[32] ,
pvss = int[32] ,
reflow = int[32] ,
schnorr = octet[32]
}
}
[W] {
a_GIVEN_in = {},
c_CACHE_ack = {},
c_CODEC_ack = {
keyring = {
encoding = "def",
name = "keyring",
schema = "keyring",
zentype = "e"
}
},
c_WHEN_ack = {
keyring = "(hidden)"
},
d_THEN_out = {}
}
+18 Given nothing
+23 When I create the ecdh key
+24 When I create the es256 key
+25 When I create the ethereum key
+26 When I create the reflow key
+27 When I create the schnorr key
+28 When I create the bitcoin key
+29 When I create the eddsa key
+30 When I create the bbs key
+31 When I create the pvss key
+32 When I create the dilithium key
+34 Then print the 'keyrig'
[!] Error at Zencode line 34
[!] /zencode_then.lua:158: Cannot find object: keyrig
[!] Zencode runtime error
[!] /zencode.lua:706: Zencode line 34: Then print the 'keyrig'
[!] Execution aborted with errors.
[*] Zenroom teardown.
Memory used: 606 KB
```
the same log in json
```json
[ "ZENROOM JSON LOG START",
" Release version: v4.36.0",
" Build commit hash: 760ca7bb",
" Memory manager: libc",
" ECDH curve is SECP256K1",
" ECP curve is BLS381",
"[W] Zencode is missing version check, please add: rule check version N.N.N",
"J64 HEAP: eyJDQUNIRSI6W10sIkNPREVDIjp7ImtleXJpbmciOnsiZW5jb2RpbmciOiJkZWYiLCJuYW1lIjoia2V5cmluZyIsInNjaGVtYSI6ImtleXJpbmciLCJ6ZW50eXBlIjoiZSJ9fSwiR0lWRU5fZGF0YSI6W10sIlRIRU4iOltdLCJXSEVOIjp7ImtleXJpbmciOiIoaGlkZGVuKSJ9fQ==",
"J64 TRACE: WyIrMTggIEdpdmVuIG5vdGhpbmciLCIrMjMgIFdoZW4gSSBjcmVhdGUgdGhlIGVjZGgga2V5IiwiKzI0ICBXaGVuIEkgY3JlYXRlIHRoZSBlczI1NiBrZXkiLCIrMjUgIFdoZW4gSSBjcmVhdGUgdGhlIGV0aGVyZXVtIGtleSIsIisyNiAgV2hlbiBJIGNyZWF0ZSB0aGUgcmVmbG93IGtleSIsIisyNyAgV2hlbiBJIGNyZWF0ZSB0aGUgc2Nobm9yciBrZXkiLCIrMjggIFdoZW4gSSBjcmVhdGUgdGhlIGJpdGNvaW4ga2V5IiwiKzI5ICBXaGVuIEkgY3JlYXRlIHRoZSBlZGRzYSBrZXkiLCIrMzAgIFdoZW4gSSBjcmVhdGUgdGhlIGJicyBrZXkiLCIrMzEgIFdoZW4gSSBjcmVhdGUgdGhlIHB2c3Mga2V5IiwiKzMyICBXaGVuIEkgY3JlYXRlIHRoZSBkaWxpdGhpdW0ga2V5IiwiKzM0ICBUaGVuIHByaW50IHRoZSAna2V5cmlnJyIsIlshXSBFcnJvciBhdCBaZW5jb2RlIGxpbmUgMzQiLCJbIV0gL3plbmNvZGVfdGhlbi5sdWE6MTU4OiBDYW5ub3QgZmluZCBvYmplY3Q6IGtleXJpZyJd",
"[!] Zencode runtime error",
"[!] /zencode.lua:706: Zencode line 34: Then print the 'keyrig'",
"[!] Execution aborted with errors.",
"[*] Zenroom teardown.",
" Memory used: 663 KB",
"ZENROOM JSON LOG END" ]
```
*Default*: *json* in javascript bindings and *text* otherwise.

## Memory manager
### Syntax and values: **memmanager=sys, lw, je**
## Limit iterations

Switch the use of a different memory manager, between:
Syntax and values: **maxiter=dec:[at most 10 decimal digits]**

- **sys**: system memory manager
- **lw**: "lightweight" memory manager, internal to Zenroom, can be more performant on some situation and required when running Zenroom on embedded systems with RTOS, baremetal or situations where there is no memory manager offered by the OS
- **je**: experimental memory manager based on "jemalloc", safe and fast alternative supported by some OS.
Define the maximum number of iterations the contract is allowed to do.

## Print output
### Syntax and values: **print=sys, stb**
*Default*: 1000.

Defines the function used to print the output.
## Limit of memory

- **sys** = uses system defined print, typically "sprintf" or "vsprintf"
- **stb** = internal print function based on [stb](https://github.com/nothings/stb) to be used with on embedded systems with RTOS, baremetal
Syntax and values: **maxmem=dec:[at most 10 decimal digits]**

## Log format
Since Zenroom 3, the default error log is printed out in base64. To print out the errors in text, use
```shell
logfmt=text
```
Define the maximum memory in MB that lua can occupy before calling the garbage collector
during the run phase.

*Default*: 1024 (1GB)
2 changes: 1 addition & 1 deletion src/lua/init.lua
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,7 @@ MACHINE = require('statemachine')
SEMVER = require('semver')
_G['ZENROOM_VERSION'] = SEMVER(VERSION)
_G['MAXITER'] = tonumber(STR_MAXITER)
_G['MAXMEM'] = tonumber(STR_MAXMEM) * 1024

OCTET = require('zenroom_octet')
BIG = require('zenroom_big')
Expand Down Expand Up @@ -203,4 +204,3 @@ function ZKP_challenge(list)
end

collectgarbage 'collect'

76 changes: 51 additions & 25 deletions src/lua/zencode.lua
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,11 @@ ZEN = {
checks = {version = false}, -- version, scenario checked, etc.
OK = true, -- set false by asserts
jump = nil,
ITER = {}, -- foreach infos
ITER_present = false,
BRANCH_present = false,
ITER = {}, -- foreach infos,
ITER_parse = {},
ITER_head = nil,
traceback = {}, -- transferred into HEAP by zencode_begin
linenum = 0,
last_valid_statement = false,
Expand Down Expand Up @@ -194,9 +198,22 @@ function ZEN:begin(new_heap)
from = from,
to = to,
hook = func,
linenum = linenum
linenum = linenum,
f = index == 'foreach' or nil,
ef = index == 'endforeach' or nil,
i = index == 'if' or nil,
ei = index == 'endif' or nil
}
) -- function
if index == 'foreach' then
ctx.Z.ITER_present = true
ctx.Z.ITER[ctx.Z.id] = { jump = ctx.Z.id, pos = 1 }
table.insert(ctx.Z.ITER_parse, ctx.Z.id)
elseif index == 'endforeach' then
local id = table.remove(ctx.Z.ITER_parse)
ctx.Z.ITER[id].end_id = ctx.Z.id
end
ctx.Z.BRANCH_present = ctx.Z.BRANCH_present or index == 'if'
ctx.Z.OK = true
end
if not ctx.Z.OK and CONF.parser.strict_match then
Expand Down Expand Up @@ -609,15 +626,15 @@ local function manage_branching(stack, x)
local v = stack.branch_valid
local s = x.section

if s:sub(1, 2) == 'if' then
if x.i then
stack.branch_condition = true
stack.branch = b+1
if v == b then
stack.branch_valid = v+1
return false
end
return true
elseif s:sub(1, 5) == 'endif' then
elseif x.ei then
if v == b then
stack.branch_valid = v-1
end
Expand All @@ -630,38 +647,35 @@ end
-- return true: caller skip execution and go to ::continue::
-- return false: execute statement
local function manage_foreach(stack, x, ci)
local last_iter = stack.ITER[#stack.ITER]
if string.match(x.section, '^foreach') then
if not last_iter then
table.insert(stack.ITER, {jump = ci, pos = 1 })
return false
elseif last_iter.jump ~= ci then
local not_valid_parent_loop = last_iter.pos == 0
table.insert(stack.ITER, {jump = ci, pos = fif(not_valid_parent_loop, 0, 1)})
return not_valid_parent_loop
else
return false
end
end
if string.match(x.section, '^endforeach') then
if not last_iter.end_id then last_iter.end_id = x.id end
if last_iter.pos == 0 and last_iter.end_id ~= x.id then
local last_iter = stack.ITER_head
-- if no foreach is defined skip all
if not last_iter and not x.f then return false end
if x.f then
if last_iter and last_iter.pos == 0 then
stack.jump = last_iter.end_id
return true
end
stack.ITER_head = stack.ITER[x.id]
if last_iter and stack.ITER_head.pos == 1 then
stack.ITER_head.parent = last_iter
end
return false
elseif x.ef then
if last_iter.pos > 0 then
last_iter.pos = last_iter.pos + 1
stack.jump = last_iter.jump
return true
else
table.remove(stack.ITER)
last_iter = stack.ITER[#stack.ITER]
last_iter.pos = 1
last_iter = stack.ITER_head.parent
stack.ITER_head = last_iter
end
end
if last_iter then
if last_iter.pos > MAXITER then
error("Limit of iterations exceeded: " .. MAXITER)
-- skip all statements on last (not valid) loop
elseif last_iter.pos == 0 and last_iter.end_id then
elseif last_iter.pos == 0 then
stack.jump = last_iter.end_id
return true
end
Expand All @@ -672,14 +686,24 @@ end
local function AST_iterator()
local i = 0
local AST_size <const> = table_size(AST)
local manage
if ZEN.ITER_present and ZEN.BRANCH_present then
manage = function(z, v, j) return manage_branching(z, v) or manage_foreach(z, v, j) end
elseif ZEN.ITER_present and not ZEN.BRANCH_present then
manage = function(z, v, j) return manage_foreach(z, v, j) end
elseif not ZEN.ITER_present and ZEN.BRANCH_present then
manage = function(z, v, j) return manage_branching(z, v) end
else
manage = function(z, v, j) return false end
end
return function()
i = i+1
-- End of iteration (i exceeds the AST length)
if i > AST_size then
return nil
end
local value = AST[i]
while i < AST_size and (manage_branching(ZEN, value) or manage_foreach(ZEN, value, i)) do
while i < AST_size and manage(ZEN, value, i) do
if ZEN.jump then
i = ZEN.jump - 1
ZEN.jump = nil
Expand Down Expand Up @@ -769,7 +793,9 @@ function ZEN:run()
-- give a notice about the CACHE being used
-- TODO: print it in debug
if #CACHE > 0 then xxx('Contract CACHE is in use') end
collectgarbage 'collect'
if collectgarbage('count') > MAXMEM then
collectgarbage('collect')
end
end
-- PRINT output
self:ftrace('--- Zencode execution completed')
Expand Down
Loading