More and better benchmarks are needed

[emberian]

We have few good, solid benchmarks that stress a single part of the codegen or libraries. These are necessary to be able to be able to easily spot perfomance regressions, as well as have a good quantitative data about Rust's performance.

Benchmarks should cover both space and time characteristics, and should use the built-in bench runner where possible. Some ideas:

• Allocating an object with a destructor
• Allocating a @-ptr
• Destructuring an enum/struct/tuple/tuple struct/vector
• Pattern matching
• Numeric manipulation (should be covered fairly well by the current shootout stuff, but should be extracted)
• Data structure manipulation (vec benchmarks etc, similar to what core-map and friends do)
• String manipulation
• Method call
• Trait object method call
• Task communication (like pingpong does) with the various primitives (one-shots, streams, protocols)
• Logging

An important note: these benchmarks aren't intended for comparison to other languages (although they could be useful for that in the future), but for tracking the performance of Rust.

[emberian]

Nominating for well-covered

[emberian]

Benchmark #7543 (newtype vs original type)

[emberian]

Benchmark FFI calls

[emberian]

Benchmark task switch time

[emberian]

Benchmark all forms of allocation

[emberian]

Benchmark parallel allocation/ffi (for the hit contention would give)

[emberian]

Benchmark the impact of segmented stacks in various situations

[pnathan]

cmr:

What do you think about taking some code from here: https://github.com/pnathan/flaky-data-structures, marking a particular changeset and using that to build benchmarks on? The BST and linked list should handle these tasks, today; I'd just need to cook up a pile of data and a benchmarking driver to use against them:

Allocating a @-ptr
Destructuring an enum//tuple
Pattern matching
Method call
Trait object method call

[emberian]

@pnathan: ideally the benchmarks would be included in the rust source, I'm not sure the license is suitable (@graydon?). While it'd be fine as a datapoint, and I'd love to include it, it does "too much" in that its performance doesn't really reflect how well Rust is doing for any single feature. But, it'd be a good start. (Aside: you call it "Mozilla Rust" in your readme. it's just plain "Rust")

[pnathan]

@cmr: I don't mind relicencing the code for the Rust project if you want to include it as a datapoint. I see what you mean about the single-feature focus you want. I can cook something up this weekend for a couple of those benchmarks.

[emberian]

That'd be great. I'd love to include it and it'd certainly be useful as
part of the whole perf picture

[abhijeetgaiha]

@pnathan Are you working on individual benchmarks as well?

[graydon]

Note: rust has a unit benchmark driver in the unit test driver. These are really small things to write. I'll land a bunch of them today.

[emberian]

@graydon could you help land #7687 first? I'd like to keep the benchmarks in a separate crate from libstd etc, to make them standaloneish

[huonw]

(For anyone wondering how to use it, see the benchmark section of the unit testing wiki page.)

[emberian]

See: https://gist.github.com/sanxiyn/6040905

[emberian]

As rustdoc_ng calms down, I'm going to be adding benchmarks all over the place for things that you wouldn't even normally benchmark. Mooooore data yay!

[catamorphism]

Anyone want to suggest a criterion for determining when this task will be complete?

[emberian]

When every performance characteristic of the language and standard library is repeatably measurable.

[catamorphism]

Too broad to be a milestone blocker. De-nominating

[satwiki]

Hi I want to contribute on this project. Would anybody please assist me?

[catamorphism]

@satwiki Great! The best thing to do is to go on #rust on IRC and ask for suggestions for getting started. You can either just ask on the channel, or specifically ask one of the people with an @ in front of their nickname (those are the core team members), or ask cmr, who left several comments above. I'm probably not going to be on IRC much today, but you can ask me as well, tomorrow (my IRC nick is tjc).

[catamorphism]

(See the "Communication" section at https://github.com/mozilla/rust/wiki/Note-development-policy for how to get on IRC.)

[ghost]

Hey, I want to contribute to this project. How can I??

[emberian]

@umasharma see @catamorphism's comments above

[emberian]

Another very interesting idea would be maintaining the benchmarks in multiple languages (C, C++) and comparing to them.

[steveklabnik]

I'm pulling a massive triage effort to get us ready for 1.0. As part of this, I'm moving stuff that's wishlist-like to the RFCs repo, as that's where major new things should get discussed/prioritized.

This issue has been moved to the RFCs repo: rust-lang/rfcs#627