Add support for INSERT INTO table (...) SELECT ...

This feature has been in the works for a very long time, and has a lot
of context... I've added headers so if you already know about the
iteration of the API and the evolution of `InsertStatement` internally,
skip to the third section.

Getting to this API
===

I'd like to give a bit of context on the APIs that have been considered,
and how I landed on this one.

To preface, all of the iterations around this have been trying to
carefully balance three things:

- Easy to discover in the API
- Being syntactically close to the generated SQL
- Avoiding rightward drift

For most of Diesel's life, our API was `insert(values).into(table)`.
That API was originally introduced in 0.2 "to mirror `update` and
`delete` (it didn't mirror them. It was backwards. It's always been
backwards).

My main concern with the old API actually was related to this feature.
I couldn't come up with a decent API that had you specify the column
list (you basically always need to specify the column list for this
feature).

So in 0.99 we changed it to what we have now, and I had toyed around
with `insert_into(table).columns(columns).values(select)`, as well as
`insert_into(table).from_select(columns, select)`. I was leaning towards
the second one for a while (I didn't realize at the time that it was
exactly SQLAlchemy's API). I hated the `columns` method because it was
unclear what it was doing until you saw you were inserting from a select
statement. It also implied an interaction with tuples that didn't exist.

However, another thing that happened in 0.99 was that we deprecated our
old upsert API. The `from_select` form reminded me far too much of the
old `on_conflict` API, which we had just removed. In practice what that
API would give you was something like this:

```rust
insert_into(posts::table)
    .from_select(
        (posts::user_id, posts::title),
        users::table
            .select((
                users::id,
                users::name.concat("'s First Post"),
            )),
    )
```

That's just far too much rightward drift for me. Yes, you can assign the
args to local variables, but they're awkward to name and now your code
reads weird. I thought moving the columns list to another method call
would help, but it doesn't.

```rust
insert_into(posts::table)
    .from_select(
        users::table
            .select((
                users::id,
                users::name.concat("'s First Post"),
            )),
    )
    .into_columns((posts::user_id, posts::title))
```

Eventually a member of the Diesel team had the idea of making this an
`insert_into` method on select statements. This solves all of my
concerns around rightward drift, though at the cost of the other two
priorities. The API now looked like this:

```
users::table
    .select((
        users::id,
        users::name.concat("'s First Post"),
    ))
    .insert_into(posts::table)
    .into_columns((posts::user_id, posts::title))
```

I liked the way the code flowed, but I had concerns around
discoverability, and didn't like how backwards it was from SQL. But I
could live with it and started implementing.

Up until this point I had been assuming that we would have an
`InsertFromSelectStatement` struct, which was distinct from
`InsertStatement` and necessitated the differently named methods. I
realized when I started digging into it though, that we really just want
to re-use `InsertStatement` for this. It seems obvious in hindsight.

And if we were going to use that structure, that meant that it wouldn't
be much harder to just make passing `SelectStatement` to `values` work.
This automatically solves most of my concerns around discoverability,
since it now just works exactly like every other form of insert.

That said, I really don't like the rightward drift. I liked the
`.insert_into` form for being able to avoid that. But for the final
implementation, I just generalized that feature. *Anything* that can be
written as `insert_into(table).values(values)` can now be written as
`values.insert_into(table)`.

Context around InsertStatement
===

This file has churned more than just about any other part of Diesel. I
feel like I've re-written it nearly every release at this point. I think
the reason it's churned so much is for two reasons. The first is that
it's kept a fundamental design flaw through 1.1 (which I'll get to), and
we've been constantly working around it. The second is that `INSERT` is
actually one of the most complex queries in SQL. It has less variations
than `SELECT`, but far more of its variations are backend specific, or
have different syntaxes between backends.

`InsertStatement` was originally added for 0.2 in c9894b3 which has a
very helpful commit message "WIP" (good job past Sean). At the time we
only supported PG, so we actually had two versions -- `InsertStatement`
and `InsertQuery`, the latter having a returning clause. I'm honestly
not sure why I didn't do the `ReturningClause<T>` `NoReturningClause`
dance we do now and did back then in `SelectStatement`. Maybe I thought
it'd be easier?

Anyway this file had to go through some pretty major changes in 0.5 when
we added SQLite support. We needed to disallow batch insert and
returning clauses on that backend. Additionally, the way we handle
default values had to work differently on SQLite since it doesn't
support the `DEFAULT` keyword.

At this point though, it still a few things. The query starts with
`INSERT INTO`, had a columns list, then the word `VALUES` and then some
values. It also managed all parenthesis. (Yes it was trivial to make it
generate invalid SQL at that point).

Fast forward to 0.9, I decided to support batch insert on SQLite. This
needs to do one query per row, and I felt that meant we needed a
separate struct. I didn't want to have `BatchInsertStatement` and
`BatchInsertQuery` and `InsertStatement` and `InsertQuery`, so we got
the `NoReturningClause` struct, and got a little closer to how literally
every other part of Diesel works.

In 0.14 we added `insert_default_values` which left us with
`InsertStatement`, `BatchInsertStatement`, and `DefaultInsertStatement`,
which eventually went back down to the two.

The last time the file went through a big rewrite was in 0.99 when I
finally unified it down to the one struct we have today.

However, it still had the fatal flaw I mentioned earlier. It was trying
to handle too much logic, and was too disjoint from the rest of the
query builder. It assumed that the two forms were `DEFAULT VALUES` or
`(columns) VALUES ...`, and also handled parens.

We've gone through a lot of refactoring to get rid of that. I finally
think this struct is at a point where it will stop churning, mostly
because it looks like the rest of Diesel now. It doesn't do anything at
all, and the only thing it assumes is that the word `INTO` appears in
the query (which I'm pretty sure actually is true).

The actual implementation of this commit
====

With all that said, this commit is relatively simple. The main addition
is the `InsertFromSelect` struct. It's definitely a Diesel struct, in
that it basically does nothing, and all the logic is in its `where`
clauses.

I tried to make `Insertable` work roughly everywhere that methods like
`filter` work. I couldn't actually do a blanket impl for tables in Rust
itself, because it made rustc vomit on recursion with our impls on
`Option`. That shouldn't be happening, but I suspect it's a lot of work
to fix that in the language.

I've also implemented it for references, since most of the time that you
pass values to `.values`, you pass a reference. That should "just work"
here unless we have a good reason not to.

The majority of the additions here are tests. This is a feature that
fundamentally interacts with many of our most complex features, and I've
tried to be exhaustive. Theres ~3 lines of tests for every line of code
added. I've done at least a minimal test of this feature's interaction
with every other feature on inserts that I could think of. I am not so
much worried about whether they work, but I was worried about if there
was a syntactic edge case I didn't know about. There weren't. Same with
the compile tests, I've tried to think of every way the feature could be
accidentally misused (bad arguments to `into_columns` basically), as
well as all the nonsense things we should make sure don't work (putting
it in a tuple or vec).

I didn't add any compile tests on making sure that the select statement
itself is valid. The fact that the `Query` bound only matches valid
complete select statements is already very well tested, and we don't
need to re-test that here. I also did not explicitly disallow selecting
from the same table as the insert, as this appears to be one of the few
places where the table can appear twice with no ambiguity.

Fixes #1106

CHANGELOG.md

@@ -8,6 +8,15 @@ for Rust libraries in [RFC #1105](https://github.com/rust-lang/rfcs/blob/master/
 
 ### Added
 
+* Added support for `INSERT INTO table (...) SELECT ...` queries. Tables, select
+  select statements, and boxed select statements can now be used just like any
+  other `Insertable` value.
+
+* Any insert query written as `insert_into(table).values(values)` can now be
+  written as `values.insert_into(table)`. This is particularly useful when
+  inserting from a select statement, as select statements tend to span multiple
+  lines.
+
 * Added support for specifying `ISOLATION LEVEL`, `DEFERRABLE`, and `READ ONLY`
   on PG transactions. See [`PgConnection::build_transaction`] for details.
 

diesel/src/insertable.rs

@@ -3,7 +3,7 @@ use std::marker::PhantomData;
 use backend::{Backend, SupportsDefaultKeyword};
 use expression::{AppearsOnTable, Expression};
 use result::QueryResult;
-use query_builder::{AstPass, QueryFragment, UndecoratedInsertRecord, ValuesClause};
+use query_builder::{AstPass, InsertStatement, QueryFragment, UndecoratedInsertRecord, ValuesClause};
 use query_source::{Column, Table};
 #[cfg(feature = "sqlite")]
 use sqlite::Sqlite;
@@ -20,21 +20,79 @@ use sqlite::Sqlite;
 /// struct differs from the name of the column, you can annotate the field
 /// with `#[column_name = "some_column_name"]`.
 pub trait Insertable<T> {
+    /// The `VALUES` clause to insert these records
+    ///
+    /// The types used here are generally internal to Diesel.
+    /// Implementations of this trait should use the `Values`
+    /// type of other `Insertable` types.
+    /// For example `<diesel::dsl::Eq<column, &str> as Insertable<table>>::Values`.
     type Values;
 
+    /// Construct `Self::Values`
+    ///
+    /// Implementations of this trait typically call `.values`
+    /// on other `Insertable` types.
     fn values(self) -> Self::Values;
+
+    /// Insert `self` into a given table.
+    ///
+    /// `foo.insert_into(table)` is identical to `insert_into(table).values(foo)`.
+    /// However, when inserting from a select statement,
+    /// this form is generally preferred.
+    ///
+    /// # Example
+    ///
+    /// ```rust
+    /// # #[macro_use] extern crate diesel;
+    /// # include!("doctest_setup.rs");
+    /// #
+    /// # fn main() {
+    /// #     run_test().unwrap();
+    /// # }
+    /// #
+    /// # fn run_test() -> QueryResult<()> {
+    /// #     use schema::{posts, users};
+    /// #     let conn = establish_connection();
+    /// #     diesel::delete(posts::table).execute(&conn)?;
+    /// users::table
+    ///     .select((
+    ///         users::name.concat("'s First Post"),
+    ///         users::id,
+    ///     ))
+    ///     .insert_into(posts::table)
+    ///     .into_columns((posts::title, posts::user_id))
+    ///     .execute(&conn)?;
+    ///
+    /// let inserted_posts = posts::table
+    ///     .select(posts::title)
+    ///     .load::<String>(&conn)?;
+    /// let expected = vec!["Sean's First Post", "Tess's First Post"];
+    /// assert_eq!(expected, inserted_posts);
+    /// #     Ok(())
+    /// # }
+    /// ```
+    fn insert_into(self, table: T) -> InsertStatement<T, Self::Values>
+    where
+        Self: Sized,
+    {
+        ::insert_into(table).values(self)
+    }
 }
 
 pub trait CanInsertInSingleQuery<DB: Backend> {
-    fn rows_to_insert(&self) -> usize;
+    /// How many rows will this query insert?
+    ///
+    /// This function should only return `None` when the query is valid on all
+    /// backends, regardless of how many rows get inserted.
+    fn rows_to_insert(&self) -> Option<usize>;
 }
 
 impl<'a, T, DB> CanInsertInSingleQuery<DB> for &'a T
 where
     T: ?Sized + CanInsertInSingleQuery<DB>,
     DB: Backend,
 {
-    fn rows_to_insert(&self) -> usize {
+    fn rows_to_insert(&self) -> Option<usize> {
         (*self).rows_to_insert()
     }
 }
@@ -43,17 +101,17 @@ impl<'a, T, Tab, DB> CanInsertInSingleQuery<DB> for BatchInsert<'a, T, Tab>
 where
     DB: Backend + SupportsDefaultKeyword,
 {
-    fn rows_to_insert(&self) -> usize {
-        self.records.len()
+    fn rows_to_insert(&self) -> Option<usize> {
+        Some(self.records.len())
     }
 }
 
 impl<T, U, DB> CanInsertInSingleQuery<DB> for ColumnInsertValue<T, U>
 where
     DB: Backend,
 {
-    fn rows_to_insert(&self) -> usize {
-        1
+    fn rows_to_insert(&self) -> Option<usize> {
+        Some(1)
     }
 }
 

diesel/src/macros/mod.rs

@@ -707,6 +707,7 @@ macro_rules! table_body {
                 JoinTo,
             };
             use $crate::associations::HasTable;
+            use $crate::insertable::Insertable;
             use $crate::query_builder::*;
             use $crate::query_builder::nodes::Identifier;
             use $crate::query_source::{AppearsInFromClause, Once, Never};
@@ -861,6 +862,30 @@ macro_rules! table_body {
                 }
             }
 
+            // This impl should be able to live in Diesel,
+            // but Rust tries to recurse for no reason
+            impl<T> Insertable<T> for table
+            where
+                <table as AsQuery>::Query: Insertable<T>,
+            {
+                type Values = <<table as AsQuery>::Query as Insertable<T>>::Values;
+
+                fn values(self) -> Self::Values {
+                    self.as_query().values()
+                }
+            }
+
+            impl<'a, T> Insertable<T> for &'a table
+            where
+                table: Insertable<T>,
+            {
+                type Values = <table as Insertable<T>>::Values;
+
+                fn values(self) -> Self::Values {
+                    (*self).values()
+                }
+            }
+
             /// Contains all of the columns of this table
             pub mod columns {
                 use super::table;

diesel/src/pg/upsert/on_conflict_clause.rs

@@ -33,7 +33,7 @@ impl<Values, Target, Action> CanInsertInSingleQuery<Pg> for OnConflictValues<Val
 where
     Values: CanInsertInSingleQuery<Pg>,
 {
-    fn rows_to_insert(&self) -> usize {
+    fn rows_to_insert(&self) -> Option<usize> {
         self.values.rows_to_insert()
     }
 }

diesel/src/query_builder/functions.rs

@@ -249,6 +249,47 @@ pub fn delete<T: IntoUpdateTarget>(source: T) -> DeleteStatement<T::Table, T::Wh
 /// # }
 /// ```
 ///
+/// ### Insert from select
+///
+/// When inserting from a select statement,
+/// the column list can be specified with [`.into_columns`].
+/// (See also [`SelectStatement::insert_into`], which generally
+/// reads better for select statements)
+///
+/// [`SelectStatement::insert_into`]: prelude/trait.Insertable.html#method.insert_into
+/// [`.into_columns`]: query_builder/struct.InsertStatement.html#method.into_columns
+///
+/// ```rust
+/// # #[macro_use] extern crate diesel;
+/// # include!("../doctest_setup.rs");
+/// #
+/// # fn main() {
+/// #     run_test().unwrap();
+/// # }
+/// #
+/// # fn run_test() -> QueryResult<()> {
+/// #     use schema::{posts, users};
+/// #     let conn = establish_connection();
+/// #     diesel::delete(posts::table).execute(&conn)?;
+/// let new_posts = users::table
+///     .select((
+///         users::name.concat("'s First Post"),
+///         users::id,
+///     ));
+/// diesel::insert_into(posts::table)
+///     .values(new_posts)
+///     .into_columns((posts::title, posts::user_id))
+///     .execute(&conn)?;
+///
+/// let inserted_posts = posts::table
+///     .select(posts::title)
+///     .load::<String>(&conn)?;
+/// let expected = vec!["Sean's First Post", "Tess's First Post"];
+/// assert_eq!(expected, inserted_posts);
+/// #     Ok(())
+/// # }
+/// ```
+///
 /// ### With return value
 ///
 /// ```rust

diesel/src/query_builder/insert_statement/column_list.rs

@@ -0,0 +1,29 @@
+use backend::Backend;
+use query_builder::*;
+use query_source::Column;
+use result::QueryResult;
+
+/// Represents the column list for use in an insert statement.
+///
+/// This trait is implemented by columns and tuples of columns.
+pub trait ColumnList {
+    /// The table these columns belong to
+    type Table;
+
+    /// Generate the SQL for this column list.
+    ///
+    /// Column names must *not* be qualified.
+    fn walk_ast<DB: Backend>(&self, out: AstPass<DB>) -> QueryResult<()>;
+}
+
+impl<C> ColumnList for C
+where
+    C: Column,
+{
+    type Table = <C as Column>::Table;
+
+    fn walk_ast<DB: Backend>(&self, mut out: AstPass<DB>) -> QueryResult<()> {
+        out.push_identifier(C::NAME)?;
+        Ok(())
+    }
+}

diesel/src/query_builder/insert_statement/insert_from_select.rs

@@ -0,0 +1,66 @@
+use backend::Backend;
+use expression::{Expression, NonAggregate, SelectableExpression};
+use insertable::*;
+use query_builder::*;
+use query_source::Table;
+
+/// Represents `(Columns) SELECT FROM ...` for use in an `INSERT` statement
+#[derive(Debug, Clone, Copy)]
+pub struct InsertFromSelect<Select, Columns> {
+    query: Select,
+    columns: Columns,
+}
+
+impl<Select, Columns> InsertFromSelect<Select, Columns> {
+    /// Construct a new `InsertFromSelect` where the target column list is
+    /// `T::AllColumns`.
+    pub fn new<T>(query: Select) -> Self
+    where
+        T: Table<AllColumns = Columns>,
+        Columns: SelectableExpression<T> + NonAggregate,
+    {
+        Self {
+            query,
+            columns: T::all_columns(),
+        }
+    }
+
+    /// Replace the target column list
+    pub fn with_columns<C>(self, columns: C) -> InsertFromSelect<Select, C> {
+        InsertFromSelect {
+            query: self.query,
+            columns,
+        }
+    }
+}
+
+impl<DB, Select, Columns> CanInsertInSingleQuery<DB> for InsertFromSelect<Select, Columns>
+where
+    DB: Backend,
+{
+    fn rows_to_insert(&self) -> Option<usize> {
+        None
+    }
+}
+
+impl<DB, Select, Columns> QueryFragment<DB> for InsertFromSelect<Select, Columns>
+where
+    DB: Backend,
+    Columns: ColumnList + Expression<SqlType = Select::SqlType>,
+    Select: Query + QueryFragment<DB>,
+{
+    fn walk_ast(&self, mut out: AstPass<DB>) -> QueryResult<()> {
+        out.push_sql("(");
+        self.columns.walk_ast(out.reborrow())?;
+        out.push_sql(") ");
+        self.query.walk_ast(out.reborrow())?;
+        Ok(())
+    }
+}
+
+impl<Select, Columns> UndecoratedInsertRecord<Columns::Table> for InsertFromSelect<Select, Columns>
+where
+    Columns: ColumnList + Expression<SqlType = Select::SqlType>,
+    Select: Query,
+{
+}