Skip to content

Commit

Permalink
feat(transpiler): transpile gno standard libraries (#1695)
Browse files Browse the repository at this point in the history
Merge order:

1. #1700 
2. #1702
3. #1695 (this one!) -- review earlier ones first, if they're still
open!

This PR modifies the Gno transpiler (fka precompiler) to use Gno's
standard libraries rather than Go's when performing transpilation. This
creates the necessity to transpile Gno standard libraries, and as such
support their native bindings. And it removes the necessity for a
package like `stdshim`, and a mechanism like `stdlibWhitelist`.

- Fixes #668. Fixes #1865.
- Resolves #892.
- Part of #814. 
- Makes #1475 / #1576 possible without using hacks like `stdshim`.

cc/ @leohhhn @tbruyelle, as this relates to your work

## Why?

- This PR enables us to perform Go type-checking across the board, and
not use Go's standard libraries in transpiled code. This enables us to
_properly support our own standard libraries_, such as `std` but any
others we might want or need.
- It also paves the way further to go full circle, and have Gno code be
transpiled to Go, and then have "compilable" gno code

## Summary of changes

- The transpiler has been thoroughly refactored.
- The biggest change is described above: instead of maintaing the import
paths like `"strconv"` and `"math"` the same (so using Gno's stdlibs in
Gno, and Go's in Go), the import paths for standard libraries is now
also updated to point to the Gno standard libraries.
- Native functions are handled by removing their definitions when
transpiling, and changing their call expressions where appropriate. This
links the transpiled code directly to their native counterparts.
  - This removes the necessity for `stdlibWhitelist`. 
- As a consequence, `stdshim` is no longer needed and has been removed.
- Test files are still not "strictly checked": they may reference
stdlibs with no matching source, and will not be tested when running
with `--gobuild`. This is because packages like `fmt` have no
representation in Gno code; they only exist as injections in
`tests/imports.go`. I'll fix this eventually :)
- The CLI (`gno transpile`) has been changed to reflect the above
changes.
- Flag `--skip-fmt` has been removed (the result of transpile is always
formatted, anyway), and `--gofmt-binary` too, obviously. `gno transpile`
does not perform validation, but will gladly provide helpful validation
with the `--gobuild` flag.
- There is another PR that adds type checking in `gno lint`, without
needing to run through the transpilation step first:
#1730
- It now works by default by looking at "packages" rather than
individual files. This is necessary so that when performing `transpile`
on the `examples` directory, we can skip those where the gno.mod marks
the module as draft. These modules make use of packages like "fmt",
which because they don't have an underlying gno/go source, cannot be
transpiled.
- Running with `-gobuild` now handles more errors correctly; ie., all
errors not previously captured by the `errorRe` which only matches those
pertaining to a specific file/line.
  - `gnoFilesFromArgs` was unused and as such deleted
- `gnomod`'s behaviour was slightly changed.
- I am of the opinion that `gno mod download` should not precompile what
it downloads; _especially_ to gather the dependencies it has. I've
changed it so that it does a `OnlyImports` parse of the file it
downloads to fetch additional dependencies

Misc:

- `Makefile` now contains a recipe to calculate the coverage for
`gnovm/cmd/gno`, and also view it via the HTML interface. This is needed
as it has a few extra steps (which @gfanton already previously added in
the CI).
- Realms r/demo/art/gnoface and r/x/manfred_outfmt have been marked as
draft, as they depend on packages which are not actually present in the
Gno standard libraries.
  - The transpiler now ignores draft packages by default.
- `ReadMemPackage` now also considers Go files. This is meant to have
on-chain the code for standard libraries like `std` which have native
bindings. We still exclude Go code if it's not in a standard library.
- `//go:build` constraints have been removed from standard libraries, as
go files can only have one and we already add our own when transpiling

## Further improvements

after this PR

- Scope understanding in `transpiler` (so call expressions are not
incorrectly rewritten)
- Correctly transpile gno.mod

---------

Co-authored-by: Antonio Navarro Perez <antnavper@gmail.com>
Co-authored-by: Miloš Živković <milos.zivkovic@tendermint.com>
  • Loading branch information
3 people authored Jun 19, 2024
1 parent feb3051 commit c4664ed
Show file tree
Hide file tree
Showing 49 changed files with 1,327 additions and 971 deletions.
2 changes: 2 additions & 0 deletions examples/gno.land/r/demo/art/gnoface/gno.mod
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
// Draft

module gno.land/r/demo/art/gnoface

require (
Expand Down
2 changes: 2 additions & 0 deletions examples/gno.land/r/x/manfred_outfmt/gno.mod
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
// Draft

module gno.land/r/x/manfred_outfmt

require (
Expand Down
23 changes: 22 additions & 1 deletion gnovm/Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,9 @@ GNOROOT_DIR ?= $(abspath $(lastword $(MAKEFILE_LIST))/../../)
# We can't use '-trimpath' yet as amino use absolute path from call stack
# to find some directory: see #1236
GOBUILD_FLAGS ?= -ldflags "-X github.com/gnolang/gno/gnovm/pkg/gnoenv._GNOROOT=$(GNOROOT_DIR)"
# file where to place cover profile; used for coverage commands which are
# more complex than adding -coverprofile, like test.cmd.coverage.
GOTEST_COVER_PROFILE ?= cmd-profile.out

########################################
# Dev tools
Expand Down Expand Up @@ -66,6 +69,24 @@ test: _test.cmd _test.pkg _test.gnolang
_test.cmd:
go test ./cmd/... $(GOTEST_FLAGS)

# Run tests on ./cmd/, saving the result of the coverage in
# GOTEST_COVER_PROFILE.
.PHONY: test.cmd.coverage
test.cmd.coverage:
$(eval export TXTARCOVERDIR := $(shell mktemp -d --tmpdir gnovm-make.XXXXXXX))
go test ./cmd/... -covermode atomic -test.gocoverdir='$(TXTARCOVERDIR)' $(GOTEST_FLAGS)
@echo "coverage results:"
go tool covdata percent -i="$(TXTARCOVERDIR)"
go tool covdata textfmt -v 1 -i="$(TXTARCOVERDIR)" -o '$(GOTEST_COVER_PROFILE)'
rm -rf "$(TXTARCOVERDIR)"

# Run test.cmd.coverage, then view the result in the HTML browser render
# and delete the original file.
.PHONY: test.cmd.coverage_view
test.cmd.coverage_view: test.cmd.coverage
go tool cover -html='$(GOTEST_COVER_PROFILE)'
rm '$(GOTEST_COVER_PROFILE)'

.PHONY: _test.pkg
_test.pkg:
go test ./pkg/... $(GOTEST_FLAGS)
Expand All @@ -74,7 +95,7 @@ _test.pkg:
_test.gnolang: _test.gnolang.native _test.gnolang.stdlibs _test.gnolang.realm _test.gnolang.pkg0 _test.gnolang.pkg1 _test.gnolang.pkg2 _test.gnolang.other
_test.gnolang.other:; go test tests/*.go -run "(TestFileStr|TestSelectors)" $(GOTEST_FLAGS)
_test.gnolang.realm:; go test tests/*.go -run "TestFiles/^zrealm" $(GOTEST_FLAGS)
_test.gnolang.pkg0:; go test tests/*.go -run "TestStdlibs/(bufio|crypto|encoding|errors|internal|io|math|sort|std|stdshim|strconv|strings|testing|unicode)" $(GOTEST_FLAGS)
_test.gnolang.pkg0:; go test tests/*.go -run "TestStdlibs/(bufio|crypto|encoding|errors|internal|io|math|sort|std|strconv|strings|testing|unicode)" $(GOTEST_FLAGS)
_test.gnolang.pkg1:; go test tests/*.go -run "TestStdlibs/regexp" $(GOTEST_FLAGS)
_test.gnolang.pkg2:; go test tests/*.go -run "TestStdlibs/bytes" $(GOTEST_FLAGS)
_test.gnolang.native:; go test tests/*.go -test.short -run "TestFilesNative/" $(GOTEST_FLAGS)
Expand Down
3 changes: 1 addition & 2 deletions gnovm/cmd/gno/test.go
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,6 @@ import (
"github.com/gnolang/gno/gnovm/pkg/gnoenv"
gno "github.com/gnolang/gno/gnovm/pkg/gnolang"
"github.com/gnolang/gno/gnovm/pkg/gnomod"
"github.com/gnolang/gno/gnovm/pkg/transpiler"
"github.com/gnolang/gno/gnovm/tests"
"github.com/gnolang/gno/tm2/pkg/commands"
"github.com/gnolang/gno/tm2/pkg/errors"
Expand Down Expand Up @@ -259,7 +258,7 @@ func gnoTestPkg(
if gnoPkgPath == "" {
// unable to read pkgPath from gno.mod, generate a random realm path
io.ErrPrintfln("--- WARNING: unable to read package path from gno.mod or gno root directory; try creating a gno.mod file")
gnoPkgPath = transpiler.GnoRealmPkgsPrefixBefore + random.RandStr(8)
gnoPkgPath = gno.RealmPathPrefix + random.RandStr(8)
}
}
memPkg := gno.ReadMemPackage(pkgPath, gnoPkgPath)
Expand Down
36 changes: 0 additions & 36 deletions gnovm/cmd/gno/testdata/gno_transpile/05_skip_fmt_flag.txtar

This file was deleted.

38 changes: 0 additions & 38 deletions gnovm/cmd/gno/testdata/gno_transpile/06_build_flag.txtar

This file was deleted.

Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
# Run gno transpile with gno files with whitelist errors
# Run gno transpile with gno files with an invalid import path

! gno transpile .

! stdout .+
stderr '^main.gno:5:2: import "xxx" is not in the whitelist$'
stderr '^sub/sub.gno:3:8: import "xxx" is not in the whitelist$'
stderr '^main.gno:5:2: import "xxx" does not exist$'
stderr '^sub/sub.gno:3:8: import "xxx" does not exist$'
stderr '^2 transpile error\(s\)$'

# no *.gen.go files are created
Expand Down
31 changes: 31 additions & 0 deletions gnovm/cmd/gno/testdata/gno_transpile/valid_gobuild_file.txtar
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
# Run gno transpile with -gobuild flag on an individual file

gno transpile -gobuild -v main.gno

! stdout .+
cmp stderr stderr.golden

cmp main.gno.gen.go main.gno.gen.go.golden

-- stderr.golden --
main.gno
main.gno [build]
-- main.gno --
package main

func main() {
var x = 1
_=x
}
-- main.gno.gen.go.golden --
// Code generated by github.com/gnolang/gno. DO NOT EDIT.

//go:build gno

//line main.gno:1:1
package main

func main() {
var x = 1
_ = x
}
72 changes: 72 additions & 0 deletions gnovm/cmd/gno/testdata/gno_transpile/valid_gobuild_flag.txtar
Original file line number Diff line number Diff line change
@@ -0,0 +1,72 @@
# Run gno transpile with -gobuild flag

gno transpile -gobuild -v .

! stdout .+
cmp stderr stderr.golden

# The test file will be excluded from transpilation unless we pass it explicitly.
cmp main.gno.gen.go main.gno.gen.go.golden
! exists .main_test.gno.gen_test.go
cmp sub/sub.gno.gen.go sub/sub.gno.gen.go.golden
rm mai.gno.gen.gosub/sub.gno.gen.go

# Re-try, but use an absolute path.
gno transpile -gobuild -v $WORK

! stdout .+
cmpenv stderr stderr2.golden

cmp main.gno.gen.go main.gno.gen.go.golden
! exists .main_test.gno.gen_test.go
cmp sub/sub.gno.gen.go sub/sub.gno.gen.go.golden

-- stderr.golden --
.
sub
. [build]
sub [build]
-- stderr2.golden --
$WORK
$WORK/sub
$WORK [build]
$WORK/sub [build]
-- main.gno --
package main

func main() {
var x = 1
_=x
}
-- main.gno.gen.go.golden --
// Code generated by github.com/gnolang/gno. DO NOT EDIT.

//go:build gno

//line main.gno:1:1
package main

func main() {
var x = 1
_ = x
}
-- main_test.gno --
package main

import (
"testing"
"badimport"
)

func TestMain(t *testing.T) {
badimport.DoesNotExist()
}
-- sub/sub.gno --
package sub
-- sub/sub.gno.gen.go.golden --
// Code generated by github.com/gnolang/gno. DO NOT EDIT.

//go:build gno

//line sub.gno:1:1
package sub
41 changes: 41 additions & 0 deletions gnovm/cmd/gno/testdata/gno_transpile/valid_output_flag.txtar
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
# Run gno transpile with valid gno files, using the -output flag.

gno transpile -v -output directory/hello/ .
! stdout .+
cmp stderr stderr1.golden

exists directory/hello/main.gno.gen.go
! exists main.gno.gen.go
rm directory

# Try running using the absolute path to the directory.
gno transpile -v -output directory/hello $WORK
! stdout .+
cmpenv stderr stderr2.golden

exists directory/hello$WORK/main.gno.gen.go
! exists directory/hello/main.gno.gen.go
rm directory

# Try running in subdirectory, using a "relative non-local path." (ie. has "../")
mkdir subdir
cd subdir
gno transpile -v -output hello ..
! stdout .+
cmpenv stderr ../stderr3.golden

exists hello$WORK/main.gno.gen.go
! exists main.gno.gen.go

-- stderr1.golden --
.
-- stderr2.golden --
$WORK
-- stderr3.golden --
..
-- main.gno --
package main

func main() {
println("hello")
}
44 changes: 44 additions & 0 deletions gnovm/cmd/gno/testdata/gno_transpile/valid_output_gobuild.txtar
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
# Run gno transpile with valid gno files, using the -output and -gobuild flags together.

gno transpile -v -output directory/hello/ -gobuild .
! stdout .+
cmp stderr stderr1.golden

exists directory/hello/main.gno.gen.go
! exists main.gno.gen.go
rm directory

# Try running using the absolute path to the directory.
gno transpile -v -output directory/hello -gobuild $WORK
! stdout .+
cmpenv stderr stderr2.golden

exists directory/hello$WORK/main.gno.gen.go
! exists directory/hello/main.gno.gen.go
rm directory

# Try running in subdirectory, using a "relative non-local path." (ie. has "../")
mkdir subdir
cd subdir
gno transpile -v -output hello -gobuild ..
! stdout .+
cmpenv stderr ../stderr3.golden

exists hello$WORK/main.gno.gen.go
! exists main.gno.gen.go

-- stderr1.golden --
.
directory/hello [build]
-- stderr2.golden --
$WORK
directory/hello$WORK [build]
-- stderr3.golden --
..
hello$WORK [build]
-- main.gno --
package main

func main() {
println("hello")
}
Loading

0 comments on commit c4664ed

Please sign in to comment.