Expose size_hint() for TokenStream's iterator

[dtolnay]

The iterator for proc_macro::TokenStream is a wrapper around a Vec iterator:

rust/library/proc_macro/src/lib.rs

Lines 363 to 371 in babff22

	pub struct IntoIter(
	std::vec::IntoIter<
	bridge::TokenTree<
	bridge::client::TokenStream,
	bridge::client::Span,
	bridge::client::Symbol,
	>,
	>,
	);

so it can cheaply provide a perfectly precise size hint, with just a pointer subtraction:

rust/library/alloc/src/vec/into_iter.rs

Lines 170 to 177 in babff22

	fn size_hint(&self) -> (usize, Option<usize>) {
	let exact = if mem::size_of::<T>() == 0 {
	self.end.addr().wrapping_sub(self.ptr.addr())
	} else {
	unsafe { self.end.sub_ptr(self.ptr) }
	};
	(exact, Some(exact))
	}

I need the size hint in syn (https://github.com/dtolnay/syn/blob/1.0.98/src/buffer.rs) to reduce allocations when converting TokenStream into syn's internal TokenBuffer representation.

Aside from size_hint, the other non-default methods in std::vec::IntoIter's Iterator impl are advance_by, count, and __iterator_get_unchecked. I've included count in this PR since it is trivial. I did not include __iterator_get_unchecked because it is spoopy and I did not feel like dealing with that. Lastly, I did not include advance_by because that requires feature(iter_advance_by) (#77404) and I noticed this comment at the top of libproc_macro:

rust/library/proc_macro/src/lib.rs

Lines 20 to 22 in babff22

	// This library is copied into rust-analyzer to allow loading rustc compiled proc macros.
	// Please avoid unstable features where possible to minimize the amount of changes necessary
	// to make it compile with rust-analyzer on stable.

[rustbot]

Hey! It looks like you've submitted a new PR for the library teams!

If this PR contains changes to any rust-lang/rust public library APIs then please comment with @rustbot label +T-libs-api -T-libs to tag it appropriately. If this PR contains changes to any unstable APIs please edit the PR description to add a link to the relevant API Change Proposal or create one if you haven't already. If you're unsure where your change falls no worries, just leave it as is and the reviewer will take a look and make a decision to forward on if necessary.

Examples of T-libs-api changes:

• Stabilizing library features
• Introducing insta-stable changes such as new implementations of existing stable traits on existing stable types
• Introducing new or changing existing unstable library APIs (excluding permanently unstable features / features without a tracking issue)
• Changing public documentation in ways that create new stability guarantees
• Changing observable runtime behavior of library APIs

[rust-highfive]

r? @petrochenkov

(rust-highfive has picked a reviewer for you, use r? to override)

[petrochenkov]

@bors r+ rollup

[bors]

📌 Commit 63e74ab has been approved by petrochenkov

It is now in the queue for this repository.

[mystor]

I'm definitely considering changing the iterator to not be a thin wrapper around Vec's IntoIter in the future, but providing size_hint() should be something we're generally able to do. Main worry I have going forward is if I end up moving tokens into the client more, and we need to split a TokenTree in two due to it being a Literal(-1) or similar, but that's probably not going to be an issue.