Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added support for win_flex_bison on Windows #771

Merged
merged 4 commits into from
Apr 20, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 35 additions & 0 deletions .github/workflows/windows-compile.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
# Platform specific Windows test. See verible-ci for main continuous integration.
name: windows-compile

on:
push:
branches:
- master
- main
pull_request:
branches:
- master
- main

jobs:
WindowsBuild:
runs-on: windows-latest
steps:
# Need winflexbison for Kythe, bazel fails if environment
# FLEX/BISON is not set to a valid program
- name: Install deps
run: |
choco install winflexbison
- name: Checkout code
uses: actions/checkout@v2
with:
fetch-depth: 0
# Only check flex/bison targets at the moment, since Verible cannot be
# fully compiled on Windows (yet!)
- name: Build flex/bison files
run: |
$env:FLEX = ($env:ChocolateyInstall + '/lib/winflexbison/tools/win_flex.exe') -replace '\\', '/'
$env:BISON = ($env:ChocolateyInstall + '/lib/winflexbison/tools/win_bison.exe') -replace '\\', '/'
bazel build //common/analysis:command_file_lex
corco marked this conversation as resolved.
Show resolved Hide resolved
bazel build //verilog/parser:verilog_y
bazel build //verilog/parser:verilog_lex
10 changes: 9 additions & 1 deletion WORKSPACE
Original file line number Diff line number Diff line change
Expand Up @@ -78,7 +78,15 @@ load("@com_github_google_rules_install//:setup.bzl", "install_rules_setup")

install_rules_setup()

all_content = """filegroup(name = "all", srcs = glob(["**"]), visibility = ["//visibility:public"])"""
# Need to load before rules_flex/rules_bison to make sure
# win_flex_bison is the chosen toolchain on Windows
load("//bazel:win_flex_bison.bzl", "win_flex_configure")

win_flex_configure(
name = "win_flex_bison",
sha256 = "39c6086ce211d5415500acc5ed2d8939861ca1696aee48909c7f6daf5122b505",
url = "https://github.com/lexxmark/winflexbison/releases/download/v2.5.24/win_flex_bison-2.5.24.zip",
hzeller marked this conversation as resolved.
Show resolved Hide resolved
)

http_archive(
name = "rules_m4",
Expand Down
18 changes: 12 additions & 6 deletions bazel/bison.bzl
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
# -*- Python -*-
# Copyright 2017-2020 The Verible Authors.
# Copyright 2017-2021 The Verible Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
Expand Down Expand Up @@ -30,9 +30,15 @@ def genyacc(
name = name,
srcs = [src],
outs = [header_out, source_out] + extra_outs,
cmd = "M4=$(M4) $(BISON) --defines=$(location " + header_out + ") --output-file=$(location " + source_out + ") " + " ".join(extra_options) + " $<",
toolchains = [
"@rules_bison//bison:current_bison_toolchain",
"@rules_m4//m4:current_m4_toolchain",
],
cmd = select({
hzeller marked this conversation as resolved.
Show resolved Hide resolved
"@platforms//os:windows": "$(BISON) --defines=$(location " + header_out + ") --output-file=$(location " + source_out + ") " + " ".join(extra_options) + " $<",
"//conditions:default": "M4=$(M4) $(BISON) --defines=$(location " + header_out + ") --output-file=$(location " + source_out + ") " + " ".join(extra_options) + " $<",
}),
toolchains = select({
"@platforms//os:windows": ["@rules_bison//bison:current_bison_toolchain"],
"//conditions:default": [
"@rules_bison//bison:current_bison_toolchain",
"@rules_m4//m4:current_m4_toolchain",
],
}),
)
18 changes: 12 additions & 6 deletions bazel/flex.bzl
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
# -*- Python -*-
# Copyright 2017-2020 The Verible Authors.
# Copyright 2017-2021 The Verible Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
Expand All @@ -24,9 +24,15 @@ def genlex(name, src, out):
name = name,
srcs = [src],
outs = [out],
cmd = "M4=$(M4) $(FLEX) --outfile=$@ $<",
toolchains = [
"@rules_flex//flex:current_flex_toolchain",
"@rules_m4//m4:current_m4_toolchain",
],
cmd = select({
"@platforms//os:windows": "$(FLEX) --outfile=$@ $<",
"//conditions:default": "M4=$(M4) $(FLEX) --outfile=$@ $<",
}),
toolchains = select({
"@platforms//os:windows": ["@rules_flex//flex:current_flex_toolchain"],
"//conditions:default": [
"@rules_flex//flex:current_flex_toolchain",
"@rules_m4//m4:current_m4_toolchain",
],
}),
)
48 changes: 48 additions & 0 deletions bazel/win_flex_bison.BUILD
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
load("@com_google_verible//bazel:win_flex_bison.bzl", "win_bison_toolchain", "win_flex_toolchain")

filegroup(
name = "win_flex",
srcs = ["win_flex.exe"],
visibility = ["//visibility:public"],
)

filegroup(
name = "flex_lexer_h",
srcs = ["FlexLexer.h"],
visibility = ["//visibility:public"],
)

filegroup(
name = "win_bison",
srcs = ["win_bison.exe"],
visibility = ["//visibility:public"],
)

win_flex_toolchain(
name = "flex",
flex_lexer_h = ":flex_lexer_h",
flex_tool = ":win_flex",
visibility = ["//visibility:public"],
)

toolchain(
name = "flex_toolchain",
exec_compatible_with = ["@platforms//os:windows"],
target_compatible_with = ["@platforms//os:windows"],
toolchain = ":flex",
toolchain_type = "@rules_flex//flex:toolchain_type",
)

win_bison_toolchain(
name = "bison",
bison_tool = ":win_bison",
visibility = ["//visibility:public"],
)

toolchain(
name = "bison_toolchain",
exec_compatible_with = ["@platforms//os:windows"],
target_compatible_with = ["@platforms//os:windows"],
toolchain = ":bison",
toolchain_type = "@rules_bison//bison:toolchain_type",
)
119 changes: 119 additions & 0 deletions bazel/win_flex_bison.bzl
Original file line number Diff line number Diff line change
@@ -0,0 +1,119 @@
# -*- Python -*-
# Copyright 2021 The Verible Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

"""Bazel rule to run flex/bison on Windows
"""

# buildifier: disable=provider-params
FlexToolchainInfo = provider(
fields = [
"all_files",
"flex_tool",
"flex_lexer_h",
],
)

def _win_flex_toolchain_impl(ctx):
flex_runfiles = ctx.attr.flex_tool[DefaultInfo].default_runfiles.files

return platform_common.ToolchainInfo(
flex_toolchain = FlexToolchainInfo(
all_files = depset(
direct = [ctx.executable.flex_tool],
transitive = [flex_runfiles],
),
flex_tool = ctx.attr.flex_tool.files_to_run,
flex_lexer_h = ctx.file.flex_lexer_h,
),
)

win_flex_toolchain = rule(
_win_flex_toolchain_impl,
attrs = {
"flex_tool": attr.label(
mandatory = True,
executable = True,
cfg = "host",
),
"flex_lexer_h": attr.label(
mandatory = True,
allow_single_file = [".h"],
),
},
)

# buildifier: disable=provider-params
BisonToolchainInfo = provider(
fields = [
"all_files",
"bison_tool",
],
)

def _win_bison_toolchain_impl(ctx):
bison_runfiles = ctx.attr.bison_tool[DefaultInfo].default_runfiles.files

return platform_common.ToolchainInfo(
bison_toolchain = BisonToolchainInfo(
all_files = depset(
direct = [ctx.executable.bison_tool],
transitive = [bison_runfiles],
),
bison_tool = ctx.attr.bison_tool.files_to_run,
),
)

win_bison_toolchain = rule(
_win_bison_toolchain_impl,
attrs = {
"bison_tool": attr.label(
mandatory = True,
executable = True,
cfg = "host",
),
},
)

def _remote_win_flex_bison(repository_ctx):
repository_ctx.download_and_extract(
url = repository_ctx.attr.url,
sha256 = repository_ctx.attr.sha256,
)

repository_ctx.symlink(
Label("@com_google_verible//bazel:win_flex_bison.BUILD"),
"BUILD.bazel",
)

remote_win_flex_bison = repository_rule(
implementation = _remote_win_flex_bison,
local = True,
attrs = {
"url": attr.string(mandatory = True),
"sha256": attr.string(default = ""),
},
)

def win_flex_configure(name, url, sha256 = ""):
remote_win_flex_bison(
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could this win_flex_configure() be made to return early if we're not on Windows ?

There seems to be an issue with pre-fetching all the needed dependencies vs. what is then actually needed in download later in environments that separate out the fetching and building like on nixos; this is why it needs this patch:
https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/tools/language-servers/verible/remove-unused-deps.patch

I suspect what is happening: the win_flex_configure() called from the WORKSPACE unconditionally downloads the dependencies , but then on a non-Windows system, the select() in the {bison,flex}.bzl realizes that it doesn't actually needs it, so it then leaves it out of the prepeared download blob.
Later, after disconnect from the network, on build, the unconditional download attempt will fail. At least this is what I think is happening.

Anyway, if here, we could just skip calling the remote_win_flex_bison() and register toolchain part if we're not on windows, that should probably help. Do you think you can add that @corco ?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll will look at it at the end of the week

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I took a look. It seems that this issue is part of the problem. There may be a way to fix it in either nix or bazel, but I am not comfortable with either...

I would suggest to either wait until bazel fix the issue, or simply remove downloading win_flex_bison and require the user to install it prior with chocolatey, like the use_local_flex_bison flag. I would move toward that second solution if @hzeller agrees.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@hzeller Have you look at my previous comment now that Christmas vacations are over?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, sorry, indeed I didn't see your previous comment.
I think removing the download and let the user use_local_flex_bison sounds good to me.

name = name,
url = url,
sha256 = sha256,
)

native.register_toolchains(
"@" + name + "//:flex_toolchain",
"@" + name + "//:bison_toolchain",
)