Remove some debug asserts #12153

Job	Run time
Format + check + test	5m 41s
Check wasm32 + wasm-bindgen	2m 56s
cargo-deny aarch64-apple-darwin	51s
android	37s
cargo-deny aarch64-linux-android	53s
Check Windows	3m 30s
cargo-deny i686-pc-windows-gnu	49s
cargo-deny i686-pc-windows-msvc	58s
cargo-deny i686-unknown-linux-gnu	52s
cargo-deny wasm32-unknown-unknown	48s
cargo-deny x86_64-apple-darwin	51s
cargo-deny x86_64-pc-windows-gnu	51s
cargo-deny x86_64-pc-windows-msvc	49s
cargo-deny x86_64-unknown-linux-gnu	53s
cargo-deny x86_64-unknown-linux-musl	55s
cargo-deny x86_64-unknown-redox	58s
	23m 12s

Provide feedback