Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tracking Issue for RFC2509: concat_bytes!() #87555

Open
2 of 4 tasks
m-ou-se opened this issue Jul 28, 2021 · 14 comments
Open
2 of 4 tasks

Tracking Issue for RFC2509: concat_bytes!() #87555

m-ou-se opened this issue Jul 28, 2021 · 14 comments
Labels
C-tracking-issue Category: An issue tracking the progress of sth. like the implementation of an RFC T-libs-api Relevant to the library API team, which will review and decide on the PR/issue.

Comments

@m-ou-se
Copy link
Member

m-ou-se commented Jul 28, 2021

Feature gate: #![feature(concat_bytes)]

This is a tracking issue for rust-lang/rfcs#2509

Public API

#[macro_export]
macro_rules! concat_bytes { .. }

Steps / History

Unresolved Questions

  • Should additional literal types be supported? Byte string literals are basically the same thing as byte slice references, so it might make sense to support those as well (support &[0, 1, 2] in addition to [0, 1, 2]).
  • What to do with string and character literals? They could either be supported with their underlying UTF-8 representation being concatenated, or rejected.
@m-ou-se m-ou-se added T-libs-api Relevant to the library API team, which will review and decide on the PR/issue. C-tracking-issue Category: An issue tracking the progress of sth. like the implementation of an RFC labels Jul 28, 2021
@syvb
Copy link
Contributor

syvb commented Jul 29, 2021

I'm working on an implementation at Smittyvb:concat_bytes.

@syvb
Copy link
Contributor

syvb commented Jul 29, 2021

Implementation PR: #87599

@Fishrock123
Copy link
Contributor

Regarding the unresolved questions - can we reject regular string slices if we accept &[] since str::as_bytes() is const?

@syvb
Copy link
Contributor

syvb commented Jul 30, 2021

@Fishrock123 Macro arguments are not a constant evaluation context, so calls to const fns in macros result in the tokens representing the function call getting passed to the macro, not the result of evaluating the function call.

For example,

const fn add_one(x: u8) -> u8 { x + 1 }
macro_rules! demo {
    ($x:expr) => { let x = $x; }
}
trace_macros!(true);
demo!(add_one(2));

gives the output

note: trace_macro
 --> src/main.rs:8:1
  |
8 | demo!(add_one(2));
  | ^^^^^^^^^^^^^^^^^^
  |
  = note: expanding `demo! { add_one(2) }`
  = note: to `let x = add_one(2) ;`

The add_one(2) call never gets expanded to 3 (although LLVM can optimise that) , since it's not used in a constant evaluation context. It is treated no different then any other function.

Mark-Simulacrum pushed a commit to syvb/rust that referenced this issue Dec 7, 2021
The tracking issue for this is rust-lang#87555.
matthiaskrgr added a commit to matthiaskrgr/rust that referenced this issue Dec 9, 2021
…acrum

Implement concat_bytes!

This implements the unstable `concat_bytes!` macro, which has tracking issue rust-lang#87555. It can be used like:
```rust
#![feature(concat_bytes)]

fn main() {
    assert_eq!(concat_bytes!(), &[]);
    assert_eq!(concat_bytes!(b'A', b"BC", [68, b'E', 70]), b"ABCDEF");
}
```
If strings or characters are used where byte strings or byte characters are required, it suggests adding a `b` prefix. If a number is used outside of an array it suggests arrayifying it. If a boolean is used it suggests replacing it with the numeric value of that number. Doubly nested arrays of bytes are disallowed.
syvb added a commit to syvb/rust that referenced this issue Dec 18, 2021
matthiaskrgr added a commit to matthiaskrgr/rust that referenced this issue Dec 28, 2021
Support [x; n] expressions in concat_bytes!

Currently trying to use `concat_bytes!` with a repeating array value like `[42; 5]` results in an error:
```
error: expected a byte literal
 --> src/main.rs:3:27
  |
3 |     let x = concat_bytes!([3; 4]);
  |                           ^^^^^^
  |
  = note: only byte literals (like `b"foo"`, `b's'`, and `[3, 4, 5]`) can be passed to `concat_bytes!()`
```

This makes it so repeating array syntax can be used the same way normal arrays can be. The RFC doesn't explicitly mention repeat expressions, but it seems reasonable to allow them as well, since normal arrays are allowed.

It is possible to make the compiler get stuck compiling forever with `concat_bytes!([3; 999999999])`, but I don't think that's much of an issue since you can do that already with `const X: [u8; 999999999] = [3; 999999999];`.

Contributes to rust-lang#87555.
bors added a commit to rust-lang-ci/rust that referenced this issue Jan 2, 2022
Support [x; n] expressions in concat_bytes!

Currently trying to use `concat_bytes!` with a repeating array value like `[42; 5]` results in an error:
```
error: expected a byte literal
 --> src/main.rs:3:27
  |
3 |     let x = concat_bytes!([3; 4]);
  |                           ^^^^^^
  |
  = note: only byte literals (like `b"foo"`, `b's'`, and `[3, 4, 5]`) can be passed to `concat_bytes!()`
```

This makes it so repeating array syntax can be used the same way normal arrays can be. The RFC doesn't explicitly mention repeat expressions, but it seems reasonable to allow them as well, since normal arrays are allowed.

It is possible to make the compiler get stuck compiling forever with `concat_bytes!([3; 999999999])`, but I don't think that's much of an issue since you can do that already with `const X: [u8; 999999999] = [3; 999999999];`.

Contributes to rust-lang#87555.
@octylFractal
Copy link

@m-ou-se should the implementation box be checked off now? It seems the PR to implement it was merged.

@mkpankov
Copy link
Contributor

Was FCP started?

@Mythra
Copy link

Mythra commented Oct 23, 2022

👋 , just ran into this trying to use it on a stable-codebase I'm working on. I'm curious is there any blockers/open-questions with this issue that aren't reflected here. Could we start stabilizing this?

@Nugine
Copy link
Contributor

Nugine commented Oct 26, 2022

Hi. I have just implemented this in my library: const_str::concat_bytes. It supports const-evaluation, more powerful than std::concat_bytes.

@ratankaliani
Copy link

Is this feature still unstable?

@faern
Copy link
Contributor

faern commented Aug 26, 2023

I just ended up needing this macro for the first time. I needed to include a certificate in my binary. And my dependency's X509 type requires the byte slice to end with null (they later pass it on to FFI, and re-allocating just to add a null at runtime is a waste, especially in embedded).

static CERTIFICATE: X509 = X509::pem_until_nul(concat_bytes!(
    include_bytes!("../certs/server_certificate.pem"),
    b"\0"
));

I switched over to const_str::concat_bytes for now, since it's stable. But I personally think the std macro should adopt any advantages that it has and stabilize it. This type of macro belongs in core IMHO. Given how it's related to concat! and given how it's related to pretty fundamental include-style stuff that I would expect the language to have built in.

@syvb
Copy link
Contributor

syvb commented Aug 30, 2023

I think the only issues that need to be resolved before stabilization are:

  • deciding if we want to allow unnested u8s: currently concat_bytes!(b'A', b'B') and concat_bytes!([65,66]) are ok but not concat_bytes!(65,66)
  • deciding if we want to allow chars and &strs to be concatenated (currently they can't be)

I think the status quo is fine. Does anyone think concat_bytes! should behave differently?

@faern
Copy link
Contributor

faern commented Aug 30, 2023

Not that I currently need it, but supporting const evaluation of non-literals seem very handy. Not being able to currently do concat_bytes!(b'a', "foo".as_bytes()) like const_str supports is limiting.

I have two questions:

  1. Would it be hard to just make std::concat_bytes! as powerful as const_str::concat_bytes!? If not, let's just do it. I don't see any downsides.
  2. Is allowing strictly more input to the macro really breaking? Otherwise we can stabilize it as is and improve it later.

@jmillikin
Copy link
Contributor

jmillikin commented Oct 22, 2023

Are the remaining questions irreversible decisions? For example, if concat_bytes! were stabilized without support for non-nested u8 or &str, could that be changed in a backwards-compatible way later?

If the answer is "yes" then I think it clearly should be stabilized. Combining &[u8] literals is important for no_std programs that need to embed complex data.


Also, I've been using this implementation of concat_bytes! as a placeholder -- it can be dropped in as-is and doesn't have any dependencies.

macro_rules! concat_bytes {
	($( $chunk:expr ),+ $( , )?) => {{
		struct Chunk<T>(T);
		#[allow(dead_code)]
		impl<const N: usize> Chunk<[u8; N]> {
			const fn get(&self) -> &[u8] { &self.0 }
		}
		#[allow(dead_code)]
		impl<const N: usize> Chunk<&[u8; N]> {
			const fn get(&self) -> &[u8] { self.0 }
		}
		#[allow(dead_code)]
		impl Chunk<&[u8]> {
			const fn get(&self) -> &[u8] { self.0 }
		}
		const fn bytes_len(chunks: &[&[u8]]) -> usize {
			let mut len = 0;
			let mut ii = 0;
			while ii < chunks.len() {
				len += chunks[ii].len();
				ii += 1;
			}
			len
		}
		const fn chunks_concat<const N: usize>(chunks: &[&[u8]]) -> [u8; N] {
			let mut buf = [0u8; N];
			let mut ii = 0;
			let mut buf_idx = 0;
			while ii < chunks.len() {
				let mut jj = 0;
				while jj < chunks[ii].len() {
					buf[buf_idx] = chunks[ii][jj];
					jj += 1;
					buf_idx += 1;
				}
				ii += 1;
			}
			buf
		}
		const CHUNKS: &[&[u8]] = &[$( Chunk($chunk).get() ),+];
		const BYTES_LEN: usize = bytes_len(CHUNKS);
		const BYTES: [u8; BYTES_LEN] = chunks_concat(CHUNKS);
		BYTES
	}};
}

@dead-claudia
Copy link

dead-claudia commented Oct 30, 2023

I just ended up needing this macro for the first time. I needed to include a certificate in my binary. And my dependency's X509 type requires the byte slice to end with null (they later pass it on to FFI, and re-allocating just to add a null at runtime is a waste, especially in embedded).

static CERTIFICATE: X509 = X509::pem_until_nul(concat_bytes!(
    include_bytes!("../certs/server_certificate.pem"),
    b"\0"
));

I switched over to const_str::concat_bytes for now, since it's stable. But I personally think the std macro should adopt any advantages that it has and stabilize it. This type of macro belongs in core IMHO. Given how it's related to concat! and given how it's related to pretty fundamental include-style stuff that I would expect the language to have built in.

That feels like something better for an include_c_str!(...) than this. You can always c_str.to_bytes_with_null() if you need to include the null. (Very much poor API design for them not to just expose a X509::pem_c_str(&CStr) here, but that's just my personal opinion.)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
C-tracking-issue Category: An issue tracking the progress of sth. like the implementation of an RFC T-libs-api Relevant to the library API team, which will review and decide on the PR/issue.
Projects
None yet
Development

No branches or pull requests