rewrite the entire crate

This commit introduces a ground-up rewrite of the entire crate. Most or
all use cases served by `aho-corasick 0.6` should be served by this
rewrite as well. Pretty much everything has been improved. The API is
simpler, and much more flexible with many new configuration knobs for
controlling the space-vs-time tradeoffs of Aho-Corasick automatons. In
particular, there are several tunable optimizations for controlling
space usage such as state ID representation and byte classes.

The API is simpler in that there is now just one type that encapsulates
everything: `AhoCorasick`.

Support for streams has been improved quite a bit, with new APIs for
stream search & replace.

Test and benchmark coverage has increased quite a bit.

This also fixes a subtle but important bug: empty patterns are now
handled correctly. Previously, they could never match, but now they can
match at any position.

Finally, I believe this is now the only Aho-Corasick implementation to
support leftmost-first and leftmost-longest semantics by using what I
think is a novel alteration to the Aho-Corasick construction algorithm.
I surveyed some other implementations, and there are a few Java
libraries that support leftmost-longest match semantics, but they
implement it by adding a sliding queue at search time. I also looked
into Perl's regex implementation which has an Aho-Corasick optimization
for `foo|bar|baz|...|quux` style regexes, and therefore must somehow
implement leftmost-first semantics. The code is a bit hard to grok, but
it looks like this is being handled at search time as opposed to baking
it into the automaton.

Fixes #18, Fixes #19, Fixes #26, Closes #34

.travis.yml

@@ -1,6 +1,6 @@
 language: rust
 rust:
-  - 1.13.0
+  - 1.24.1
   - stable
   - beta
   - nightly

Cargo.toml

@@ -2,50 +2,35 @@
 name = "aho-corasick"
 version = "0.6.10"  #:version
 authors = ["Andrew Gallant <jamslam@gmail.com>"]
-description = "Fast multiple substring searching with finite state machines."
+description = "Fast multiple substring searching."
 homepage = "https://github.com/BurntSushi/aho-corasick"
 repository = "https://github.com/BurntSushi/aho-corasick"
 readme = "README.md"
-keywords = ["string", "search", "text", "aho", "corasick"]
+keywords = ["string", "search", "text", "aho", "corasick", "multi", "pattern"]
 license = "Unlicense/MIT"
-exclude = ["/benches/sherlock.txt", "/ci/*", "/.travis.yml", "/Makefile", "/ctags.rust", "/session.vim"]
+categories = ["text-processing"]
+autotests = false
+exclude = ["/aho-corasick-debug", "/bench", "/ci/*", "/.travis.yml"]
 
 [badges]
 travis-ci = { repository = "BurntSushi/aho-corasick" }
+appveyor = { repository = "BurntSushi/regex-automata" }
+
+[workspace]
+members = ["bench", "aho-corasick-debug"]
 
 [lib]
 name = "aho_corasick"
 
-[[bin]]
-name = "aho-corasick-dot"
-path = "src/main.rs"
-test = false
-doc = false
-bench = false
+[features]
+default = ["std"]
+std = ["memchr/use_std"]
 
 [dependencies]
-memchr = "2"
-
-[dev-dependencies]
-csv = "1"
-docopt = "1"
-memmap = "0.6"
-quickcheck = { version = "0.7", default-features = false }
-rand = "0.5"
-serde = "1"
-serde_derive = "1"
+memchr = { version = "2.2.0", default-features = false }
 
-[[bench]]
-name = "bench"
-path = "benches/bench.rs"
-test = false
-bench = true
-
-[profile.test]
+[profile.release]
 debug = true
 
 [profile.bench]
 debug = true
-
-[profile.release]
-debug = true

README.md

@@ -1,55 +1,188 @@
-This crate provides an implementation of the
-[Aho-Corasick](http://en.wikipedia.org/wiki/Aho%E2%80%93Corasick_string_matching_algorithm)
-algorithm. Its intended use case is for fast substring matching, particularly
-when matching multiple substrings in a search text. This is achieved by
-compiling the substrings into a finite state machine.
-
-This implementation provides optimal algorithmic time complexity. Construction
-of the finite state machine is `O(p)` where `p` is the length of the substrings
-concatenated. Matching against search text is `O(n + p + m)`, where `n` is
-the length of the search text and `m` is the number of matches.
-
-[![Build status](https://api.travis-ci.org/BurntSushi/aho-corasick.png)](https://travis-ci.org/BurntSushi/aho-corasick)
+aho-corasick
+============
+A library for finding occurrences of many patterns at once. This library
+provides multiple pattern search principally through an implementation of the
+[Aho-Corasick algorithm](https://en.wikipedia.org/wiki/Aho%E2%80%93Corasick_algorithm),
+which builds a fast finite state machine for executing searches in linear time.
+Features include case insensitive matching, overlapping matches, search &
+replace in streams.
+
+[![Linux build status](https://api.travis-ci.org/BurntSushi/aho-corasick.svg)](https://travis-ci.org/BurntSushi/aho-corasick)
+[![Windows build status](https://ci.appveyor.com/api/projects/status/github/BurntSushi/aho-corasick?svg=true)](https://ci.appveyor.com/project/BurntSushi/aho-corasick)
 [![](http://meritbadge.herokuapp.com/aho-corasick)](https://crates.io/crates/aho-corasick)
 
 Dual-licensed under MIT or the [UNLICENSE](http://unlicense.org).
 
 
 ### Documentation
 
-[https://docs.rs/aho-corasick/](https://docs.rs/aho-corasick/).
+https://docs.rs/aho-corasick
+
+
+### Usage
+
+Add this to your `Cargo.toml`:
+
+```toml
+[dependencies]
+aho-corasick = "0.7"
+```
+
+and this to your crate root (if you're using Rust 2015):
+
+```rust
+extern crate aho_corasick;
+```
+
+
+# Example: basic searching
+
+This example shows how to search for occurrences of multiple patterns
+simultaneously. Each match includes the pattern that matched along with the
+byte offsets of the match.
+
+```rust
+use aho_corasick::AhoCorasick;
+
+let patterns = &["apple", "maple", "Snapple"];
+let haystack = "Nobody likes maple in their apple flavored Snapple.";
+
+let ac = AhoCorasick::new(patterns);
+let mut matches = vec![];
+for mat in ac.find_iter(haystack) {
+    matches.push((mat.pattern(), mat.start(), mat.end()));
+}
+assert_eq!(matches, vec![
+    (1, 13, 18),
+    (0, 28, 33),
+    (2, 43, 50),
+]);
+```
+
+
+# Example: case insensitivity
+
+This is like the previous example, but matches `Snapple` case insensitively
+using `AhoCorasickBuilder`:
+
+```rust
+use aho_corasick::AhoCorasickBuilder;
+
+let patterns = &["apple", "maple", "snapple"];
+let haystack = "Nobody likes maple in their apple flavored Snapple.";
+
+let ac = AhoCorasickBuilder::new()
+    .ascii_case_insensitive(true)
+    .build(patterns);
+let mut matches = vec![];
+for mat in ac.find_iter(haystack) {
+    matches.push((mat.pattern(), mat.start(), mat.end()));
+}
+assert_eq!(matches, vec![
+    (1, 13, 18),
+    (0, 28, 33),
+    (2, 43, 50),
+]);
+```
+
 
+# Example: replacing matches in a stream
 
-### Example
+This example shows how to execute a search and replace on a stream without
+loading the entire stream into memory first.
+
+```rust
+use aho_corasick::AhoCorasick;
 
-The documentation contains several examples, and there is a more complete
-example as a full program in `examples/dict-search.rs`.
+# fn example() -> Result<(), ::std::io::Error> {
+let patterns = &["fox", "brown", "quick"];
+let replace_with = &["sloth", "grey", "slow"];
+
+// In a real example, these might be `std::fs::File`s instead. All you need to
+// do is supply a pair of `std::io::Read` and `std::io::Write` implementations.
+let rdr = "The quick brown fox.";
+let mut wtr = vec![];
+
+let ac = AhoCorasick::new(patterns);
+ac.stream_replace_all(rdr.as_bytes(), &mut wtr, replace_with)?;
+assert_eq!(b"The slow grey sloth.".to_vec(), wtr);
+# Ok(()) }; example().unwrap()
+```
 
-Here is a quick example showing simple substring matching:
+
+# Example: finding the leftmost first match
+
+In the textbook description of Aho-Corasick, its formulation is typically
+structured such that it reports all possible matches, even when they overlap
+with another. In many cases, overlapping matches may not be desired, such as
+the case of finding all successive non-overlapping matches like you might with
+a standard regular expression.
+
+Unfortunately the "obvious" way to modify the Aho-Corasick algorithm to do
+this doesn't always work in the expected way, since it will report matches as
+soon as they are seen. For example, consider matching the regex `Samwise|Sam`
+against the text `Samwise`. Most regex engines (that are Perl-like, or
+non-POSIX) will report `Samwise` as a match, but the standard Aho-Corasick
+algorithm modified for reporting non-overlapping matches will report `Sam`.
+
+A novel contribution of this library is the ability to change the match
+semantics of Aho-Corasick (without additional search time overhead) such that
+`Samwise` is reported instead. For example, here's the standard approach:
 
 ```rust
-use aho_corasick::{Automaton, AcAutomaton, Match};
-
-let aut = AcAutomaton::new(vec!["apple", "maple"]);
-let mut it = aut.find("I like maple apples.");
-assert_eq!(it.next(), Some(Match {
-    pati: 1,
-    start: 7,
-    end: 12,
-}));
-assert_eq!(it.next(), Some(Match {
-    pati: 0,
-    start: 13,
-    end: 18,
-}));
-assert_eq!(it.next(), None);
+use aho_corasick::AhoCorasick;
+
+let patterns = &["Samwise", "Sam"];
+let haystack = "Samwise";
+
+let ac = AhoCorasick::new(patterns);
+let mat = ac.find(haystack).expect("should have a match");
+assert_eq!("Sam", &haystack[mat.start()..mat.end()]);
 ```
 
+And now here's the leftmost-first version, which matches how a Perl-like
+regex will work:
+
+```rust
+use aho_corasick::{AhoCorasickBuilder, MatchKind};
+
+let patterns = &["Samwise", "Sam"];
+let haystack = "Samwise";
+
+let ac = AhoCorasickBuilder::new()
+    .match_kind(MatchKind::LeftmostFirst)
+    .build(patterns);
+let mat = ac.find(haystack).expect("should have a match");
+assert_eq!("Samwise", &haystack[mat.start()..mat.end()]);
+```
+
+In addition to leftmost-first semantics, this library also supports
+leftmost-longest semantics, which match the POSIX behavior of a regular
+expression alternation. See `MatchKind` in the docs for more details.
+
+
+### Minimum Rust version policy
+
+This crate's minimum supported `rustc` version is `1.24.1`.
+
+In general, this crate will be conservative with respect to the minimum
+supported version of Rust. In general, it will follow the `regex` crate's
+policy, since `regex` is an important dependent.
+
+
+# Future work
 
-### Alternatives
+Here are some plans for the future:
 
-Aho-Corasick is useful for matching multiple substrings against many long
-strings. If your long string is fixed, then you might consider building a
-[suffix array](https://github.com/BurntSushi/suffix)
-of the search text (which takes `O(n)` time). Matches can then be found in
-`O(p log(n))` time.
+* Despite the crate's name, it seems prudent to consolidate all
+  multi-pattern search optimizations into this crate so that they get the
+  widest possible use. A good place to start will be to move the regex
+  crate's Teddy algorithm into this one. (This is more than just a move. It
+  will require fleshing out the somewhat simplistic `Prefilter` design that
+  exists internally currently.) In the future, it would be good to loot
+  Hyperscan for some of its pertinent algorithms, such as FDR.
+* Support stream searching with leftmost match semantics. Currently, only
+  standard match semantics are supported. Getting this right seems possible,
+  but is tricky since the match state needs to be propagated through multiple
+  searches. (With standard semantics, as soon as a match is seen the search
+  ends.)

aho-corasick-debug/Cargo.toml

@@ -0,0 +1,22 @@
+[package]
+publish = false
+name = "aho-corasick-debug"
+version = "0.0.1"
+authors = ["Andrew Gallant <jamslam@gmail.com>"]
+description = "A simple command line tool for playing with Aho-Corasick."
+license = "Unlicense/MIT"
+categories = ["text-processing"]
+autotests = false
+edition = "2018"
+
+[[bin]]
+name = "aho-corasick-debug"
+path = "main.rs"
+
+[dependencies]
+aho-corasick = { version = "*", path = ".." }
+memmap = "0.7.0"
+
+[dependencies.clap]
+version = "2.32.0"
+default-features = false