Create an internal prelude #4161

tgross35 · 2024-11-27T18:13:42Z

When building with rustc-dep-of-std, we don't get the core types imported by default (Clone, Copy, Option). In order to avoid needing to import these individually, introduce a prelude that includes them.

This also includes the C numeric types that are frequently used.

rustbot · 2024-11-27T18:13:47Z

r? @JohnTitor

rustbot has assigned @JohnTitor.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

rustbot · 2024-11-27T18:15:32Z

Some changes occurred in OpenBSD module

cc @semarie

Some changes occurred in solarish module

cc @jclulow, @pfmooney

Some changes occurred in OpenBSD module

cc @semarie

tgross35 · 2024-11-27T20:52:13Z

@eduardosm I still have a bit to tweak here but would you mind giving this a second set of eyes?

Current script

#!/bin/bash

set -eux

files=()

# Types either defined in this crate or in `core`
prelude_types=(
    c_char
    c_double
    c_float
    c_int
    c_longlong
    c_long
    c_short
    c_uchar
    c_uint
    c_ulonglong
    c_ulong
    c_ushort
    c_void
    intptr_t
    size_t
    ssize_t
    Clone
    Copy
    Option
    Send
    Sync
)

# Reexports from core
prelude_modules=(
    fmt
    hash
    iter
    mem
)

# Everything in the prelude
prelude=( "${prelude_types[@]}" "${prelude_modules[@]}" )

# Generate a list of all files excluding `lib.rs` (since the prelude being
# defined there makes string matching weird).
while IFS= read -r -d '' file; do
    files+=("$file")
done < <(find src -name '*.rs' -not -name '*lib.rs' -not -name '*macros.rs' -print0)

for file in "${files[@]}"; do
    needs_prelude=0

    # If the file already has some sort of glob import, skip it
    if rg --pcre2 -q 'use (crate|super)::(?!prelude).*\*' "$file"; then
        continue
    fi

    # Core types always require the prelude to handle rustc-dep-of-std
    if rg --pcre2 -q "\b(?<!\.)(Option|Clone|Copy|Send|Sync)\b" "$file"; then
        needs_prelude=1
    fi

    # If we use any types that are specified in the prelude then we will import it
    for ty in "${prelude[@]}"; do
        # If the type is defined in the current module, we don't need it from the prelude
        if rg -q "type $ty =" "$file"; then
            continue
        fi

        if rg -qU '((crate|super)::'"$ty"'|use (crate|super)::(\{\n){0,2}.*'"$ty)" "$file"; then
            echo NEEDS PRELUDE WITH TYPE $ty
            needs_prelude=1
        fi
    done

    # Check if the prelude is needed and does not already exist; if so, add it
    if [ "$needs_prelude" = "1" ] && ! rg -q "use crate::prelude::\*" "$file"; then
        perl -pi -0777 -e 's/(use (crate|super)::)/use crate::prelude::*;\n$1/' "$file"
    fi

    for ty in "${prelude[@]}"; do
        export TY="$ty" # env for perl to use

        # Remove simple imports `use crate::ty;`
        perl -pi -0777 -e 's/use (crate|super)::($ENV{TY});//g' "$file"

        # Remove the type if it is part of a group import
        perl -pi -0777 -e 's/(use (crate|super)::\{?(.*|(\n.*){0,2}))\b$ENV{TY}\b,? ?/$1/g' "$file"

        # Replace pathed `crate::ty`
        perl -pi -0777 -e 's/(crate|super)::($ENV{TY})\b/$2/g' "$file"
    done

    # For some reason, rustfmt doesn't trim leading newlines. Do so manually here.
    perl -pi -0777 -e 's/\A\n+//' "$file"

    rustfmt "$file"
done

./ci/style.sh

Edit: should be all done now.

eduardosm · 2024-11-27T22:01:19Z

Script looks good to me and produces identical results in my machine.

However I'm seeing some new compilation failures on some targets (Tier 3, I'm not sure how important it is):

aarch64-unknown-trusty:

error[E0432]: unresolved import `crate::intptr_t`
   --> src/macros.rs:83:57
    |
83  |                 c_ulong, c_ulonglong, c_ushort, c_void, intptr_t, size_t, ssize_t,
    |                                                         ^^^^^^^^ no `intptr_t` in the root
    |
   ::: src/lib.rs:112:9
    |
112 |         prelude!();
    |         ---------- in this macro invocation
    |
    = note: this error originates in the macro `prelude` (in Nightly builds, run with -Z macro-backtrace for more info)

Maybe intptr_t (and uintptr_t) could be added to that target?

I believe they should be the same as ssize_t / size_t:

https://cs.android.com/android/platform/superproject/main/+/main:external/trusty/musl/include/alltypes.h.in;l=1-5

riscv32imac-unknown-xous-elf:

error: no rules expected `full`
   --> src/lib.rs:136:18
    |
136 |         prelude!(full);
    |                  ^^^^ no rules expected this token in macro call
    |
   ::: src/macros.rs:65:1
    |
65  | macro_rules! prelude {
    | -------------------- when calling this macro
    |
    = note: while trying to match end of macro

Changing prelude!(full); to prelude!(); fixes it.

eduardosm · 2024-11-27T22:13:56Z

Another failure on powerpc64-unknown-openbsd:

error[E0433]: failed to resolve: use of undeclared crate or module `mem`
 --> src/unix/bsd/netbsdlike/openbsd/powerpc64.rs:5:39
  |
5 | pub(crate) const _ALIGNBYTES: usize = mem::size_of::<c_long>() - 1;
  |                                       ^^^ use of undeclared crate or module `mem`
  |
help: consider importing one of these modules
  |
1 + use crate::prelude::mem;
  |
1 + use core::mem;

The script changed crate::mem::size_of to mem::size_of but did not import the prelude.

Probably because this line will not insert the prelude import unless the file has already any other import:

perl -pi -0777 -e 's/(use (crate|super)::)/use crate::prelude::*;\n$1/' "$file"

Other platforms export these types, so update Trusty to do so as well.

When building with `rustc-dep-of-std`, we don't get the core types imported by default (`Clone`, `Copy`, `Option`). In order to avoid needing to import these individually, introduce a prelude that includes them, along with commonly used C numeric types. This allows cleaning up some of the `use` statements.

Automatically apply changes with the following: #!/bin/bash set -eux files=() # Types either defined in this crate or in `core` prelude_types=( c_char c_double c_float c_int c_longlong c_long c_short c_uchar c_uint c_ulonglong c_ulong c_ushort c_void intptr_t size_t ssize_t Clone Copy Option Send Sync ) # Reexports from core prelude_modules=( fmt hash iter mem ) # Everything in the prelude prelude=( "${prelude_types[@]}" "${prelude_modules[@]}" ) # Generate a list of all files excluding `lib.rs` (since the prelude being # defined there makes string matching weird). while IFS= read -r -d '' file; do files+=("$file") done < <(find src -name '*.rs' -not -name '*lib.rs' -not -name '*macros.rs' -not -name 'fixed_width_ints.rs' -print0) for file in "${files[@]}"; do needs_prelude=0 # If the file already has some sort of glob import, skip it if rg --pcre2 -q 'use (crate|super)::(?!prelude).*\*' "$file"; then continue fi # Core types always require the prelude to handle rustc-dep-of-std if rg --pcre2 -q '\b(?<!\.)(Option|Clone|Copy|Send|Sync|fmt|hash|iter|mem)\b' "$file"; then needs_prelude=1 fi # If we use any types that are specified in the prelude then we will import it for ty in "${prelude[@]}"; do # If the type is defined in the current module, we don't need it from the prelude if rg -q "type $ty =" "$file"; then continue fi if rg -qU '((crate|super)::'"$ty"'|use (crate|super)::(\{\n){0,2}.*'"$ty)" "$file"; then needs_prelude=1 fi done # Check if the prelude is needed and does not already exist; if so, add it if [ "$needs_prelude" = "1" ] && ! rg -q 'use crate::prelude::\*' "$file"; then # Split the file into two parts: module-level attributes and rest # Imports will be added after module-level attributes attrs=$(awk '/^#!|^\/\/!/ {found=NR} {lines[NR]=$0} END {for (i=1; i<=found; i++) print lines[i]}' "$file") rest=$(awk '/^#!|^\/\/!/ {found=NR} END {if (found) {for (i=found+1; i<=NR; i++) print lines[i]} else {for (i=1; i<=NR; i++) print lines[i]}} {lines[NR]=$0}' "$file") printf "%s\n" "$attrs" > "$file" printf "\n%s\n\n" "use crate::prelude::*;" >> "$file" printf "%s" "$rest" >> "$file" fi for ty in "${prelude[@]}"; do export TY="$ty" # env for perl to use # Remove simple imports `use crate::ty;` perl -pi -0777 -e 's/use ((crate|super)::)?($ENV{TY});//g' "$file" # Remove the type if it is part of a group import perl -pi -0777 -e 's/(use (crate|super)::\{?(.*|(\n.*){0,2}))\b$ENV{TY}\b,? ?/$1/g' "$file" # Replace pathed `crate::ty` perl -pi -0777 -e 's/(crate|super)::($ENV{TY})\b/$2/g' "$file" done # For some reason, rustfmt doesn't trim leading newlines. Do so manually here. perl -pi -0777 -e 's/\A\n+//' "$file" rustfmt "$file" done ./ci/style.sh

tgross35 · 2024-11-27T22:56:26Z

Great catches, thank you for taking a look! Updated to add intptr_t and uintptr_t to Trusty (cc @randomPoison), fixed the prelude!(full), and adjusted the script to use your awk commands from #4132 rather than adding the import via substitution (next script will be in python or rustscript...).

I'll make sure to verify builds for tier3 once this is closer to ready for a release which will probably be tomorrow, since I need to make some other adjustments to fix rustc-dep-of-std (#4158).

Current script:

Current script

#!/bin/bash

set -eux

files=()

# Types either defined in this crate or in `core`
prelude_types=(
    c_char
    c_double
    c_float
    c_int
    c_longlong
    c_long
    c_short
    c_uchar
    c_uint
    c_ulonglong
    c_ulong
    c_ushort
    c_void
    intptr_t
    size_t
    ssize_t
    Clone
    Copy
    Option
    Send
    Sync
)

# Reexports from core
prelude_modules=(
    fmt
    hash
    iter
    mem
)

# Everything in the prelude
prelude=( "${prelude_types[@]}" "${prelude_modules[@]}" )

# Generate a list of all files excluding `lib.rs` (since the prelude being
# defined there makes string matching weird).
while IFS= read -r -d '' file; do
    files+=("$file")
done < <(find src -name '*.rs' -not -name '*lib.rs' -not -name '*macros.rs' -not -name 'fixed_width_ints.rs' -print0)

for file in "${files[@]}"; do
    needs_prelude=0

    # If the file already has some sort of glob import, skip it
    if rg --pcre2 -q 'use (crate|super)::(?!prelude).*\*' "$file"; then
        continue
    fi

    # Core types always require the prelude to handle rustc-dep-of-std
    if rg --pcre2 -q '\b(?<!\.)(Option|Clone|Copy|Send|Sync|fmt|hash|iter|mem)\b' "$file"; then
        needs_prelude=1
    fi

    # If we use any types that are specified in the prelude then we will import it
    for ty in "${prelude[@]}"; do
        # If the type is defined in the current module, we don't need it from the prelude
        if rg -q "type $ty =" "$file"; then
            continue
        fi

        if rg -qU '((crate|super)::'"$ty"'|use (crate|super)::(\{\n){0,2}.*'"$ty)" "$file"; then
            needs_prelude=1
        fi
    done

    # Check if the prelude is needed and does not already exist; if so, add it
    if [ "$needs_prelude" = "1" ] && ! rg -q 'use crate::prelude::\*' "$file"; then
        # Split the file into two parts: module-level attributes and rest
        # Imports will be added after module-level attributes

        attrs=$(awk '/^#!|^\/\/!/ {found=NR} {lines[NR]=$0} END {for (i=1; i<=found; i++) print lines[i]}' "$file")
        rest=$(awk '/^#!|^\/\/!/ {found=NR} END {if (found) {for (i=found+1; i<=NR; i++) print lines[i]} else {for (i=1; i<=NR; i++) print lines[i]}} {lines[NR]=$0}' "$file")


        printf "%s\n" "$attrs" > "$file"
        printf "\n%s\n\n" "use crate::prelude::*;" >> "$file"
        printf "%s" "$rest" >> "$file"
    fi

    for ty in "${prelude[@]}"; do
        export TY="$ty" # env for perl to use

        # Remove simple imports `use crate::ty;`
        perl -pi -0777 -e 's/use ((crate|super)::)?($ENV{TY});//g' "$file"

        # Remove the type if it is part of a group import
        perl -pi -0777 -e 's/(use (crate|super)::\{?(.*|(\n.*){0,2}))\b$ENV{TY}\b,? ?/$1/g' "$file"

        # Replace pathed `crate::ty`
        perl -pi -0777 -e 's/(crate|super)::($ENV{TY})\b/$2/g' "$file"
    done

    # For some reason, rustfmt doesn't trim leading newlines. Do so manually here.
    perl -pi -0777 -e 's/\A\n+//' "$file"

    rustfmt "$file"
done

./ci/style.sh

randomPoison · 2024-11-27T23:02:37Z

The Trusty change looks good 👍 Thanks for adding those.

Other platforms export these types, so update Trusty to do so as well. (backport <rust-lang#4161>) (cherry picked from commit 6bee30e)

When building with `rustc-dep-of-std`, we don't get the core types imported by default (`Clone`, `Copy`, `Option`). In order to avoid needing to import these individually, introduce a prelude that includes them, along with commonly used C numeric types. This allows cleaning up some of the `use` statements. (backport <rust-lang#4161>) (cherry picked from commit 30bc78b)

Automatically apply changes with the following: #!/bin/bash set -eux files=() # Types either defined in this crate or in `core` prelude_types=( c_char c_double c_float c_int c_longlong c_long c_short c_uchar c_uint c_ulonglong c_ulong c_ushort c_void intptr_t size_t ssize_t Clone Copy Option Send Sync ) # Reexports from core prelude_modules=( fmt hash iter mem ) # Everything in the prelude prelude=( "${prelude_types[@]}" "${prelude_modules[@]}" ) # Generate a list of all files excluding `lib.rs` (since the prelude being # defined there makes string matching weird). while IFS= read -r -d '' file; do files+=("$file") done < <(find src -name '*.rs' -not -name '*lib.rs' -not -name '*macros.rs' -not -name 'fixed_width_ints.rs' -print0) for file in "${files[@]}"; do needs_prelude=0 # If the file already has some sort of glob import, skip it if rg --pcre2 -q 'use (crate|super)::(?!prelude).*\*' "$file"; then continue fi # Core types always require the prelude to handle rustc-dep-of-std if rg --pcre2 -q '\b(?<!\.)(Option|Clone|Copy|Send|Sync|fmt|hash|iter|mem)\b' "$file"; then needs_prelude=1 fi # If we use any types that are specified in the prelude then we will import it for ty in "${prelude[@]}"; do # If the type is defined in the current module, we don't need it from the prelude if rg -q "type $ty =" "$file"; then continue fi if rg -qU '((crate|super)::'"$ty"'|use (crate|super)::(\{\n){0,2}.*'"$ty)" "$file"; then needs_prelude=1 fi done # Check if the prelude is needed and does not already exist; if so, add it if [ "$needs_prelude" = "1" ] && ! rg -q 'use crate::prelude::\*' "$file"; then # Split the file into two parts: module-level attributes and rest # Imports will be added after module-level attributes attrs=$(awk '/^#!|^\/\/!/ {found=NR} {lines[NR]=$0} END {for (i=1; i<=found; i++) print lines[i]}' "$file") rest=$(awk '/^#!|^\/\/!/ {found=NR} END {if (found) {for (i=found+1; i<=NR; i++) print lines[i]} else {for (i=1; i<=NR; i++) print lines[i]}} {lines[NR]=$0}' "$file") printf "%s\n" "$attrs" > "$file" printf "\n%s\n\n" "use crate::prelude::*;" >> "$file" printf "%s" "$rest" >> "$file" fi for ty in "${prelude[@]}"; do export TY="$ty" # env for perl to use # Remove simple imports `use crate::ty;` perl -pi -0777 -e 's/use ((crate|super)::)?($ENV{TY});//g' "$file" # Remove the type if it is part of a group import perl -pi -0777 -e 's/(use (crate|super)::\{?(.*|(\n.*){0,2}))\b$ENV{TY}\b,? ?/$1/g' "$file" # Replace pathed `crate::ty` perl -pi -0777 -e 's/(crate|super)::($ENV{TY})\b/$2/g' "$file" done # For some reason, rustfmt doesn't trim leading newlines. Do so manually here. perl -pi -0777 -e 's/\A\n+//' "$file" rustfmt "$file" done ./ci/style.sh (backport <rust-lang#4161>) (cherry picked from commit f8a018a) Applied by rerunning the script rather than resolving conflicts manually.

rustbot assigned JohnTitor Nov 27, 2024

rustbot added O-android O-arm O-bsd O-dragonfly O-gnu O-illumos O-linux O-linux-like O-macos O-mips O-musl O-newlib O-redox O-riscv O-solarish O-sparc O-unix O-wasi O-windows O-x86 S-waiting-on-review labels Nov 27, 2024

tgross35 force-pushed the crate-prelude branch from 9fc7bb2 to 79ee819 Compare November 27, 2024 18:15

tgross35 force-pushed the crate-prelude branch 5 times, most recently from 7102017 to 91bb8d4 Compare November 27, 2024 18:57

tgross35 force-pushed the crate-prelude branch from b1f9221 to 00521c0 Compare November 27, 2024 20:48

tgross35 force-pushed the crate-prelude branch from 00521c0 to 628c89d Compare November 27, 2024 21:02

tgross35 added the stable-nominated This PR should be considered for cherry-pick to libc's stable release branch label Nov 27, 2024

tgross35 mentioned this pull request Nov 27, 2024

Fix usage with rustc-dep-of-std #4158

Merged

tgross35 force-pushed the crate-prelude branch 2 times, most recently from 3e4cf1b to 4d1c04c Compare November 27, 2024 22:11

tgross35 force-pushed the crate-prelude branch 4 times, most recently from bbda648 to 1259e85 Compare November 27, 2024 22:49

tgross35 added 3 commits November 27, 2024 17:54

trusty: Add intptr_t and uintptr_t

6bee30e

Other platforms export these types, so update Trusty to do so as well.

tgross35 force-pushed the crate-prelude branch from 1259e85 to f8a018a Compare November 27, 2024 22:54

tgross35 added this pull request to the merge queue Nov 28, 2024

Merged via the queue into rust-lang:main with commit 018948e Nov 28, 2024
45 checks passed

tgross35 deleted the crate-prelude branch November 28, 2024 01:16

tgross35 added a commit to tgross35/rust-libc that referenced this pull request Nov 28, 2024

trusty: Add intptr_t and uintptr_t

1abd74b

Other platforms export these types, so update Trusty to do so as well. (backport <rust-lang#4161>) (cherry picked from commit 6bee30e)

tgross35 mentioned this pull request Nov 28, 2024

[0.2] Backports #4166

Merged

github-actions bot mentioned this pull request Nov 28, 2024

chore: release v0.2.167 #4156

Merged

tgross35 added stable-applied This PR has been cherry-picked to libc's stable release branch and removed stable-nominated This PR should be considered for cherry-pick to libc's stable release branch labels Nov 28, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Create an internal prelude #4161

Create an internal prelude #4161

tgross35 commented Nov 27, 2024

rustbot commented Nov 27, 2024

rustbot commented Nov 27, 2024

tgross35 commented Nov 27, 2024 •

edited

Loading

eduardosm commented Nov 27, 2024

eduardosm commented Nov 27, 2024

tgross35 commented Nov 27, 2024 •

edited

Loading

randomPoison commented Nov 27, 2024

Create an internal prelude #4161

Create an internal prelude #4161

Conversation

tgross35 commented Nov 27, 2024

rustbot commented Nov 27, 2024

rustbot commented Nov 27, 2024

tgross35 commented Nov 27, 2024 • edited Loading

eduardosm commented Nov 27, 2024

eduardosm commented Nov 27, 2024

tgross35 commented Nov 27, 2024 • edited Loading

randomPoison commented Nov 27, 2024

tgross35 commented Nov 27, 2024 •

edited

Loading

tgross35 commented Nov 27, 2024 •

edited

Loading