Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rewrite CROSSTOOL in Starlark #5380

Closed
scentini opened this issue Jun 12, 2018 · 0 comments
Closed

Rewrite CROSSTOOL in Starlark #5380

scentini opened this issue Jun 12, 2018 · 0 comments
Assignees
Labels
P1 I'll work on this now. (Assignee required) team-Rules-CPP Issues for C++ rules type: feature request

Comments

@scentini
Copy link
Contributor

scentini commented Jun 12, 2018

Description of the problem / feature request:

Change the file format used to configure C++ toolchain from text proto to Skylark

Feature requests: what underlying problem are you trying to solve with this feature?

  • It will make maintenance of the C++ toolchain easier, it will allow getting rid of the duplication and boilerplate using standard programming practices, e.g. we will be able to provide a library of common features.
  • We will benefit from the tooling being developed for Skylark (e.g. testing, debugger).
  • It will make future C++ rules consistent with all the other Skylark rules. It will unblock rewriting of cc_toolchain rule in Skylark.

More info on this effort can be found in @hlopko 's doc: https://docs.google.com/document/d/1Nqf16jqDGWSrPp4VuRxh0iNnVBoAXsO0meDH69J9xoc/edit#heading=h.r30au8wdo4dh

@scentini scentini added type: feature request P1 I'll work on this now. (Assignee required) category: rules > C++ labels Jun 12, 2018
@scentini scentini self-assigned this Jun 12, 2018
bazel-io pushed a commit that referenced this issue Jun 13, 2018
Currently the selection of C++ toolchain configuration depends on 2 different things:
1. CROSSTOOL file: a proto text file that contains multiple CrosstoolConfig.CToolchain messages, out of which we want to select the appropriate CToolchain;
2. cc_toolchain_suite rule, specified by --crosstool_top option: this rule contains "toolchains" map attribute. The map's keys are of type <cpu>|<compiler>, or just <cpu>, and the values are labels of cc_toolchain rules (which contain additional C++ information).

If there is an entry in cc_toolchain_suite.toolchains that corresponds to the --cpu and --compiler options, we use it, otherwise we loop through all the CToolchains in the CROSSTOOL file, select the one that corresponds to the --cpu and --compiler options, and then get the cc_toolchain label from cc_toolchain_suite.toolchains[toolchain.targetCpu|toolchain.compiler].
In both cases we read the CROSSTOOL file and pass all its information forward, to be used in creation of CppConfiguration and CcToolchainProvider creation.

As part of the efforts of rewriting CROSSTOOL in Skylark, we need to make obtaining the cc_toolchain label independent of the CROSSTOOL file and toolchain selection. As a step towards that goal, we add a new "toolchain_identifier" attribute to cc_toolchain, which uniquely identifies a CToolchain in the CROSSTOOL file.

Now the process of getting the CToolchain goes as follows:
Check for existence of cc_toolchain_suite.toolchains[<cpu>|<compiler>], if --compiler is specified, otherwise check for cc_toolchain_suite.toolchains[<cpu>].

1. if a value is found, load the cc_toolchain rule and look for the toolchain_identifier attribute.
    1.a if the attribute exists, loop through all the CToolchains in CROSSTOOL and select the one with the matching toolchain identifier.
    1.b otherwise fall back to selecting the CToolchain from CROSSTOOL by matching the --cpu and --compiler values.

2. If a value is not found, select the CToolchain from CROSSTOOL by matching the --cpu and --compiler values, and construct the key as follows: <toolchain.cpu>|<toolchain.compiler>.

In the future we will get rid of 2. by making sure that cc_toolchain_suite.toolchains[<cpu>|<compiler>] and cc_toolchain_suite.toolchains[<cpu>] are always defined.
1.b will be removed by making the cc_toolchain.toolchain_identifier attribute mandatory
After this deriving the cc_toolchain label will be independent of the CROSSTOOL file

1.a will ultimately be replaced by an attribute that points to a skylark_crosstool rule, that will contain all the information that CROSSTOOL contains, allowing us to remove CROSSTOOL altogether.

Work towards issue #5380
RELNOTES: None.
PiperOrigin-RevId: 200388550
bazel-io pushed a commit that referenced this issue Jun 19, 2018
…ionConfig that are needed for FeatureConfiguration creation

In the future CcToolchainFeatures will not be created from a CToolchain but from a CrosstoolInfo provider that will encapsule all relevant information from the CROSSTOOL. Feature and ActionConfig classes need to carry all the information that is currently obtained from CToolchain.

Work towards issue #5380

RELNOTES: None.
PiperOrigin-RevId: 201147336
ArielleA pushed a commit to ArielleA/bazel that referenced this issue Jun 19, 2018
Currently the selection of C++ toolchain configuration depends on 2 different things:
1. CROSSTOOL file: a proto text file that contains multiple CrosstoolConfig.CToolchain messages, out of which we want to select the appropriate CToolchain;
2. cc_toolchain_suite rule, specified by --crosstool_top option: this rule contains "toolchains" map attribute. The map's keys are of type <cpu>|<compiler>, or just <cpu>, and the values are labels of cc_toolchain rules (which contain additional C++ information).

If there is an entry in cc_toolchain_suite.toolchains that corresponds to the --cpu and --compiler options, we use it, otherwise we loop through all the CToolchains in the CROSSTOOL file, select the one that corresponds to the --cpu and --compiler options, and then get the cc_toolchain label from cc_toolchain_suite.toolchains[toolchain.targetCpu|toolchain.compiler].
In both cases we read the CROSSTOOL file and pass all its information forward, to be used in creation of CppConfiguration and CcToolchainProvider creation.

As part of the efforts of rewriting CROSSTOOL in Skylark, we need to make obtaining the cc_toolchain label independent of the CROSSTOOL file and toolchain selection. As a step towards that goal, we add a new "toolchain_identifier" attribute to cc_toolchain, which uniquely identifies a CToolchain in the CROSSTOOL file.

Now the process of getting the CToolchain goes as follows:
Check for existence of cc_toolchain_suite.toolchains[<cpu>|<compiler>], if --compiler is specified, otherwise check for cc_toolchain_suite.toolchains[<cpu>].

1. if a value is found, load the cc_toolchain rule and look for the toolchain_identifier attribute.
    1.a if the attribute exists, loop through all the CToolchains in CROSSTOOL and select the one with the matching toolchain identifier.
    1.b otherwise fall back to selecting the CToolchain from CROSSTOOL by matching the --cpu and --compiler values.

2. If a value is not found, select the CToolchain from CROSSTOOL by matching the --cpu and --compiler values, and construct the key as follows: <toolchain.cpu>|<toolchain.compiler>.

In the future we will get rid of 2. by making sure that cc_toolchain_suite.toolchains[<cpu>|<compiler>] and cc_toolchain_suite.toolchains[<cpu>] are always defined.
1.b will be removed by making the cc_toolchain.toolchain_identifier attribute mandatory
After this deriving the cc_toolchain label will be independent of the CROSSTOOL file

1.a will ultimately be replaced by an attribute that points to a skylark_crosstool rule, that will contain all the information that CROSSTOOL contains, allowing us to remove CROSSTOOL altogether.

Work towards issue bazelbuild#5380
RELNOTES: None.
PiperOrigin-RevId: 200388550
ArielleA pushed a commit to ArielleA/bazel that referenced this issue Jun 19, 2018
…ionConfig that are needed for FeatureConfiguration creation

In the future CcToolchainFeatures will not be created from a CToolchain but from a CrosstoolInfo provider that will encapsule all relevant information from the CROSSTOOL. Feature and ActionConfig classes need to carry all the information that is currently obtained from CToolchain.

Work towards issue bazelbuild#5380

RELNOTES: None.
PiperOrigin-RevId: 201147336
bazel-io pushed a commit that referenced this issue Jun 21, 2018
CrosstoolInfo carries the necessary information from the CROSSTOOL text proto. Later on, instead from the CROSSTOOL file and the corresponding CToolchain, CrosstoolInfo will be derived from a skylark_crosstool rule implemented in Skylark. Therefore CToolchain involvement in CppConfiguration and CcToolchain creation needs to be eliminated.

Work towards issue #5380

RELNOTES: None.
PiperOrigin-RevId: 201491207
werkt pushed a commit to werkt/bazel that referenced this issue Aug 2, 2018
Currently the selection of C++ toolchain configuration depends on 2 different things:
1. CROSSTOOL file: a proto text file that contains multiple CrosstoolConfig.CToolchain messages, out of which we want to select the appropriate CToolchain;
2. cc_toolchain_suite rule, specified by --crosstool_top option: this rule contains "toolchains" map attribute. The map's keys are of type <cpu>|<compiler>, or just <cpu>, and the values are labels of cc_toolchain rules (which contain additional C++ information).

If there is an entry in cc_toolchain_suite.toolchains that corresponds to the --cpu and --compiler options, we use it, otherwise we loop through all the CToolchains in the CROSSTOOL file, select the one that corresponds to the --cpu and --compiler options, and then get the cc_toolchain label from cc_toolchain_suite.toolchains[toolchain.targetCpu|toolchain.compiler].
In both cases we read the CROSSTOOL file and pass all its information forward, to be used in creation of CppConfiguration and CcToolchainProvider creation.

As part of the efforts of rewriting CROSSTOOL in Skylark, we need to make obtaining the cc_toolchain label independent of the CROSSTOOL file and toolchain selection. As a step towards that goal, we add a new "toolchain_identifier" attribute to cc_toolchain, which uniquely identifies a CToolchain in the CROSSTOOL file.

Now the process of getting the CToolchain goes as follows:
Check for existence of cc_toolchain_suite.toolchains[<cpu>|<compiler>], if --compiler is specified, otherwise check for cc_toolchain_suite.toolchains[<cpu>].

1. if a value is found, load the cc_toolchain rule and look for the toolchain_identifier attribute.
    1.a if the attribute exists, loop through all the CToolchains in CROSSTOOL and select the one with the matching toolchain identifier.
    1.b otherwise fall back to selecting the CToolchain from CROSSTOOL by matching the --cpu and --compiler values.

2. If a value is not found, select the CToolchain from CROSSTOOL by matching the --cpu and --compiler values, and construct the key as follows: <toolchain.cpu>|<toolchain.compiler>.

In the future we will get rid of 2. by making sure that cc_toolchain_suite.toolchains[<cpu>|<compiler>] and cc_toolchain_suite.toolchains[<cpu>] are always defined.
1.b will be removed by making the cc_toolchain.toolchain_identifier attribute mandatory
After this deriving the cc_toolchain label will be independent of the CROSSTOOL file

1.a will ultimately be replaced by an attribute that points to a skylark_crosstool rule, that will contain all the information that CROSSTOOL contains, allowing us to remove CROSSTOOL altogether.

Work towards issue bazelbuild#5380
RELNOTES: None.
PiperOrigin-RevId: 200388550
werkt pushed a commit to werkt/bazel that referenced this issue Aug 2, 2018
…ionConfig that are needed for FeatureConfiguration creation

In the future CcToolchainFeatures will not be created from a CToolchain but from a CrosstoolInfo provider that will encapsule all relevant information from the CROSSTOOL. Feature and ActionConfig classes need to carry all the information that is currently obtained from CToolchain.

Work towards issue bazelbuild#5380

RELNOTES: None.
PiperOrigin-RevId: 201147336
werkt pushed a commit to werkt/bazel that referenced this issue Aug 2, 2018
CrosstoolInfo carries the necessary information from the CROSSTOOL text proto. Later on, instead from the CROSSTOOL file and the corresponding CToolchain, CrosstoolInfo will be derived from a skylark_crosstool rule implemented in Skylark. Therefore CToolchain involvement in CppConfiguration and CcToolchain creation needs to be eliminated.

Work towards issue bazelbuild#5380

RELNOTES: None.
PiperOrigin-RevId: 201491207
bazel-io pushed a commit that referenced this issue Aug 21, 2018
Work towards issue #5380.

//tools/cpp/cc_toolchain_config_lib.bzl contains all the necessary functionality for
one to be able to create a CcToolchainConfigInfo provider from Skylark. CcToolchainConfigInfo currently encapsulates the info read from the CROSSTOOL file. We are moving away from using the CROSSTOOL files, in the future we will reroute the logic
through an attribute of cc_toolchain that will point to a Skylark rule.
A CcToolchainConfigInfo provider can be created with the cc_common.create_cc_toolchain_config_info()
method.

e.g:

crosstool_rule = rule{
    implementation = _impl,
    provides = [CcToolchainConfigInfo],
)

def _impl(ctx):
    feature1 = feature(
        name = "feature_one",
        flag_sets = [
            flag_set(actions = ["action1"]),
            flag_set(
                actions = ["action1"],
                flag_groups = [
                    flag_group(
                        flags = ["flag1"],
                    ),
                    flag_group(
                        flag_groups = [
                            flag_group(
                                flags = ["flag1"],
                            ),
                        ],
                        expand_if_equal = variable_with_value(
                            name = "variable1",
                            value = "value",
                        ),
                    ),
                ],
                expand_if_all_available = ["sysroot"],
            ),
        ],
    )
    action_config1 = action_config(action_name = "action_one", enabled = True)
    action_config2 = action_config(
        action_name = "action_two",
        enabled = True,
        tools = [tool(path = "/a/b/c")],
    )
    artifact_name_pattern1 = artifact_name_pattern(
        category_name = "static_library",
        prefix = "prefix",
        extension = ".a",
    )
    cc_target_os = "os"

    return  cc_common.create_cc_toolchain_config_info(
        features = [feature1],
        action_configs = [action_config1, action_config2],
        artifact_name_patterns = [artifact_name_pattern1],
        cc_target_os = cc_target_os,
        make_variables = [make_variable(name = "v1", value = "val")],
    )

RELNOTES: None.
PiperOrigin-RevId: 209648136
bazel-io pushed a commit that referenced this issue Sep 6, 2018
…esn't depend on the CROSSTOOL file.

Work towards issue #5380

RELNOTES: None.
PiperOrigin-RevId: 211771190
bazel-io pushed a commit that referenced this issue Sep 6, 2018
…CROSSTOOL with a skylark rule

Work towards issue #5380.

RELNOTES: None.
PiperOrigin-RevId: 211793570
@hlopko hlopko added team-Rules-CPP Issues for C++ rules and removed category: rules > C++ labels Oct 11, 2018
bazel-io pushed a commit that referenced this issue Oct 24, 2018
…hainConfigInfo

Work towards #5380
RELNOTES: None.
PiperOrigin-RevId: 218484449
bazel-io pushed a commit that referenced this issue Nov 15, 2018
…ture behavior

After selecting the CToolchain, Bazel tempers with it by adding and moving features, action configs, and artifact name patterns (see CppToolchainInfo#addLegacyFeatures() ).

What it does is :

For every ArtifactCategory defined in, if the CToolchain does not contain an equivalent ArtifactNamePattern, we add it.

if CToolchain does not contain "no_legacy_features" feature:
    - if it contains "legacy_compile_flags" feature, move it to the very top of the list of features (above both CToolchain features and legacy ones).
    - prepend a bunch of action_configs in front of the ones that CToolchain defines
    - prepend a bunch of features in front of the ones that CToolchain defines, IF they are not already present.
    - append a bunch of features to the list that CToolchain defines ,IF they are not already present.

This cl introduces the above described behavior to cc_common.create_cc_toolchain_config_info().

Work towards #5380

RELNOTES: None.
PiperOrigin-RevId: 221595847
bazel-io pushed a commit that referenced this issue Nov 26, 2018
…on between data obtained from starlark rule and a CToolchain from CROSSTOOL file.

Work towards issue #5380

RELNOTES: None.
PiperOrigin-RevId: 222815904
@hlopko hlopko changed the title Rewrite CROSSTOOL in Skylark Rewrite CROSSTOOL in Starlark Nov 29, 2018
bazel-io pushed a commit that referenced this issue Dec 4, 2018
Work towards issue #5380
RELNOTES: None.
PiperOrigin-RevId: 223936920
bazel-io pushed a commit that referenced this issue Dec 4, 2018
Work towards issue #5380
RELNOTES: None.
PiperOrigin-RevId: 223946963
bazel-io pushed a commit that referenced this issue Dec 5, 2018
…host configuration

Work towards issue #5380.
RELNOTES: None.
PiperOrigin-RevId: 224125325
bazel-io pushed a commit that referenced this issue Dec 5, 2018
… to match the one of creating 'flag_set's from CToolchain.

A flag_set that is meant for an action_config should not have elements in it's actions parameter. Instead, we put the action's name in the flag_set.actions list.

Work towards issue #5380

RELNOTES: None.
PiperOrigin-RevId: 224203125
bazel-io pushed a commit that referenced this issue Mar 18, 2019
Issue #5380
RELNOTES: None.
PiperOrigin-RevId: 238980912
bazel-io pushed a commit that referenced this issue Mar 18, 2019
Issue #5380
RELNOTES: None.
PiperOrigin-RevId: 238994787
bazel-io pushed a commit that referenced this issue Mar 18, 2019
Issue #5380

RELNOTES: None
PiperOrigin-RevId: 239020370
bazel-io pushed a commit that referenced this issue Mar 18, 2019
Issue #5380
RELNOTES: None.
PiperOrigin-RevId: 239053687
bazel-io pushed a commit that referenced this issue Mar 19, 2019
Issue #5380
RELNOTES: None.
PiperOrigin-RevId: 239168544
pull bot pushed a commit to ishikhman/bazel that referenced this issue Mar 19, 2019
bazel-io pushed a commit that referenced this issue Mar 19, 2019
Issue #5380
RELNOTES: None.
PiperOrigin-RevId: 239212945
bazel-io pushed a commit that referenced this issue Mar 20, 2019
Issue #5380

RELNOTES: None.
PiperOrigin-RevId: 239352742
bazel-io pushed a commit that referenced this issue Mar 21, 2019
Issue #5380

RELNOTES: None.
PiperOrigin-RevId: 239568494
bazel-io pushed a commit that referenced this issue Mar 21, 2019
Issue #5380

RELNOTES: None.
PiperOrigin-RevId: 239580767
bazel-io pushed a commit that referenced this issue Mar 21, 2019
Issue #5380

RELNOTES: None.
PiperOrigin-RevId: 239627148
bazel-io pushed a commit that referenced this issue Mar 26, 2019
…STOOL file

Issue #5380

RELNOTES: None.
PiperOrigin-RevId: 240352653
bazel-io pushed a commit that referenced this issue Mar 27, 2019
…OL file

Issue #5380

RELNOTES: None.
PiperOrigin-RevId: 240614078
bazel-io pushed a commit that referenced this issue Mar 28, 2019
…ool_file flag

Issue #5380
Issue #7320
RELNOTES: None.
PiperOrigin-RevId: 240741379
bazel-io pushed a commit that referenced this issue Apr 2, 2019
Issue #5380

RELNOTES: None.
PiperOrigin-RevId: 241518537
@scentini scentini closed this as completed May 3, 2019
bazel-io pushed a commit that referenced this issue May 3, 2019
CROSSTOOL files are permanently disallowed in Bazel. This flag was introduced to follow the incompatible changes policy (https://docs.bazel.build/versions/master/backward-compatibility.html), and has served its purpose.

See issue #5380

RELNOTES: None.
PiperOrigin-RevId: 246522041
swiple-rules-gardener pushed a commit to bazelbuild/rules_apple that referenced this issue Aug 12, 2019
swiple-rules-gardener pushed a commit to bazelbuild/rules_apple that referenced this issue Aug 12, 2019
swiple-rules-gardener pushed a commit to bazelbuild/rules_apple that referenced this issue Aug 12, 2019
swiple-rules-gardener pushed a commit to bazelbuild/rules_apple that referenced this issue Aug 12, 2019
swiple-rules-gardener pushed a commit to bazelbuild/rules_apple that referenced this issue Aug 13, 2019
swiple-rules-gardener pushed a commit to bazelbuild/rules_apple that referenced this issue Aug 13, 2019
…ol and rules_swift... but not objC yet.

bazelbuild/bazel#5380

bazelbuild/bazel#4984 is the general bug for asan/tsan/ubsan.

WIP NOTES: Currently Under PY3 I'm seeing "ModuleNotFoundError: No module named 'macholib'". https://storage.googleapis.com/bazel-untrusted-buildkite-artifacts/e203259d-8963-405b-b285-c8353a5115ac/test/ios_application_clang_rt_test.simulator/attempt_1.log

RELNOTES: None
PiperOrigin-RevId: 262913436
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
P1 I'll work on this now. (Assignee required) team-Rules-CPP Issues for C++ rules type: feature request
Projects
None yet
Development

No branches or pull requests

2 participants