Skip to content

Commit

Permalink
Feature/hlist store type (#695)
Browse files Browse the repository at this point in the history
* Using HLists instead of tuples for inference.

* Upgrading method signatures

* Trying to force cast

* Duck typing generics to collide HList types if enclosing type is found.

* Removing useless printlns

* Adding more documentation and dealing with warnings.

* Adding more documentation.

* Adding shapeless implementation.

* Removing unused imports.

* Removing println statements, tweaking echo and info usage and fixing SingleGenerics

* A bit more work and tests

* Cheating

* Fixing logging for batch staments.

* Upgrading batch query mechanism.

* Adding macro paradise plugin dependency

* Adding more docs.

* Fixing recurvise call

* Renaming batch method [version skip]
  • Loading branch information
alexflav23 authored Jun 11, 2017
1 parent 26a1b88 commit a565a15
Show file tree
Hide file tree
Showing 23 changed files with 502 additions and 119 deletions.
6 changes: 6 additions & 0 deletions build.sbt
Original file line number Diff line number Diff line change
Expand Up @@ -186,6 +186,9 @@ lazy val phantomJdk8 = (project in file("phantom-jdk8"))
crossScalaVersions := Seq("2.10.6", "2.11.8", "2.12.1"),
concurrentRestrictions in Test := Seq(
Tags.limit(Tags.ForkedTestGroup, defaultConcurrency)
),
libraryDependencies ++= Seq(
compilerPlugin("org.scalamacros" % "paradise" % Versions.macroParadise cross CrossVersion.full)
)
).settings(
sharedSettings: _*
Expand Down Expand Up @@ -213,6 +216,7 @@ lazy val phantomFinagle = (project in file("phantom-finagle"))
crossScalaVersions := Seq("2.10.6", "2.11.8", "2.12.1"),
testFrameworks in Test ++= Seq(new TestFramework("org.scalameter.ScalaMeterFramework")),
libraryDependencies ++= Seq(
compilerPlugin("org.scalamacros" % "paradise" % Versions.macroParadise cross CrossVersion.full),
"com.twitter" %% "util-core" % Versions.twitterUtil(scalaVersion.value),
"com.outworkers" %% "util-testing" % Versions.util % Test,
"com.outworkers" %% "util-testing-twitter" % Versions.util % Test,
Expand Down Expand Up @@ -269,6 +273,7 @@ lazy val phantomStreams = (project in file("phantom-streams"))
testFrameworks in Test ++= Seq(new TestFramework("org.scalameter.ScalaMeterFramework")),
crossScalaVersions := Seq("2.10.6", "2.11.8", "2.12.1"),
libraryDependencies ++= Seq(
compilerPlugin("org.scalamacros" % "paradise" % Versions.macroParadise cross CrossVersion.full),
"com.typesafe" % "config" % Versions.typesafeConfig force(),
"com.typesafe.play" %% "play-iteratees" % Versions.play(scalaVersion.value) exclude ("com.typesafe", "config"),
"org.reactivestreams" % "reactive-streams" % Versions.reactivestreams,
Expand All @@ -289,6 +294,7 @@ lazy val phantomExample = (project in file("phantom-example"))
crossScalaVersions := Seq("2.10.6", "2.11.8", "2.12.1"),
moduleName := "phantom-example",
libraryDependencies ++= Seq(
compilerPlugin("org.scalamacros" % "paradise" % Versions.macroParadise cross CrossVersion.full),
"org.json4s" %% "json4s-native" % Versions.json4s % Test,
"com.outworkers" %% "util-samplers" % Versions.util % Test
),
Expand Down
31 changes: 27 additions & 4 deletions docs/quickstart.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,9 +8,9 @@
There are no additional resolvers required for any version of phantom newer than 2.0.0. All Outworkers libraries are open source, licensed via Apache V2. As of version 2.2.1, phantom has no external transitive dependencies other than shapeless
and the Java driver.

#### For must things, all you need is a dependency on the phantom-dsl module.
#### For most things, all you need is a dependency on the phantom-dsl module.

For most things, all you need is the main `phantom-ds` module. This will bring in the default module with all the query generation ability, as well as `phantom-connectors` and database objects that help you manage your entire database layer on the fly. All other modules implement enhanced integration with other tools, but you don't need them to get started.
For most things, all you need is the main `phantom-dsl` module. This will bring in the default module with all the query generation ability, as well as `phantom-connectors` and database objects that help you manage your entire database layer on the fly. All other modules implement enhanced integration with other tools, but you don't need them to get started.
This module only depends on the `datastax-java-driver` and the `shapeless` library.

```scala
Expand All @@ -19,6 +19,16 @@ libraryDependencies ++= Seq(
)
```

If you are still using Scala 2.10, you must also add a compile time dependency on the macro paradise plugin. This
is because phantom uses `shapeless.Generic` under the hood and that needs the `paradise` plugin to work on
Scala 2.10.

```scala
libraryDependencies ++= Seq(
compilerPlugin("org.scalamacros" % "paradise" % Versions.macroParadise cross CrossVersion.full)
)
```

#### The full list of available modules

The full list of available modules is:
Expand All @@ -43,7 +53,21 @@ resolvers += "twitter-repo" at "http://maven.twttr.com"

#### Using phantom-sbt to test requires custom resolvers

In your `plugins.sbt`, you will also need this if you plan to use `phantom-sbt`:
In your `plugins.sbt`, you will also need this if you plan to use `phantom-sbt`. Phantom SBT is no longer
actively maintained as we are looking to produce a replacement for it. There are many many problems with
running embedded Cassandra in a forked VM that are very difficult to address and not worthwhile, as even a perfect
implementation is severely lacking in features.

The replacement module will not be available open source, but it will allow you to:

- Configure the Cassandra setup from within SBT or tests.
- Uses more powerful mechanisms under the hood to allow testing against an entire cluster instead of just one node.
- Allow you to configure which version of Cassandra to target.
- Has in depth options for configuring logging, health metrics, VM performance and much more.
- It will also have the capacity to auto-generate the entire schema for a project using an SBT task
in a way compatible with `phantom-migrations`.

This will be available via a phantom-pro paid subscription only.

```scala

Expand All @@ -56,7 +80,6 @@ resolvers ++= Seq(
)

addSbtPlugin("com.outworkers" %% "phantom-sbt" % phantomVersion)

```


Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -22,11 +22,12 @@ import com.outworkers.phantom.builder.query.{RootCreateQuery, _}
import com.outworkers.phantom.builder.syntax.CQLSyntax
import com.outworkers.phantom.column.{AbstractColumn, CollectionColumn}
import com.outworkers.phantom.connectors.KeySpace
import com.outworkers.phantom.macros.TableHelper
import com.outworkers.phantom.macros.{==:==, SingleGeneric, TableHelper}
import org.slf4j.{Logger, LoggerFactory}
import shapeless.{Generic, HList}

import scala.concurrent.duration._
import scala.concurrent.{Await, ExecutionContextExecutor}
import scala.concurrent.{Await, ExecutionContextExecutor, Future}

/**
* Main representation of a Cassandra table.
Expand All @@ -37,7 +38,6 @@ abstract class CassandraTable[T <: CassandraTable[T, R], R](
implicit val helper: TableHelper[T, R]
) extends SelectTable[T, R] { self =>


@deprecated("Use Table instead of CassandraTable, and skip passing in the 'this' argument", "2.9.1")
class ListColumn[RR](t: CassandraTable[T, R])(
implicit ev: Primitive[RR],
Expand Down Expand Up @@ -146,9 +146,23 @@ abstract class CassandraTable[T <: CassandraTable[T, R], R](
* @tparam V1 The type of the input.
* @return A default input query.
*/
def store[V1](input: V1)(
implicit keySpace: KeySpace
): InsertQuery.Default[T, R] = helper.store(instance, input.asInstanceOf[helper.Repr])
def store[V1, Repr <: HList, HL, Out <: HList](input: V1)(
implicit keySpace: KeySpace,
thl: TableHelper.Aux[T, R, Repr],
gen: Generic.Aux[V1, HL],
sg: SingleGeneric.Aux[V1, Repr, HL, Out],
ev: Out ==:== Repr
): InsertQuery.Default[T, R] = thl.store(instance, (sg to input).asInstanceOf[Repr])

def storeRecord[V1, Repr <: HList, HL <: HList, Out <: HList](input: V1)(
implicit keySpace: KeySpace,
session: Session,
thl: TableHelper.Aux[T, R, Repr],
ex: ExecutionContextExecutor,
gen: Generic.Aux[V1, HL],
sg: SingleGeneric.Aux[V1, Repr, HL, Out],
ev: Out ==:== Repr
): Future[ResultSet] = store(input).future()

final def delete()(implicit keySpace: KeySpace): DeleteQuery.Default[T, R] = DeleteQuery[T, R](instance)

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -25,17 +25,24 @@ import com.outworkers.phantom.builder.{ConsistencyBound, QueryBuilder, Specified
import scala.annotation.implicitNotFound
import scala.concurrent.{ExecutionContextExecutor, Future => ScalaFuture}

case class BatchWithQuery(
statement: Statement,
queries: String,
batchType: BatchType
) {
val debugString = s"BEGIN BATCH ${batchType.batch} ($queries) APPLY BATCH;"
}

class BatchType(val batch: String)

object BatchType {
case object Logged extends BatchType(CQLSyntax.Batch.Logged)
case object Unlogged extends BatchType(CQLSyntax.Batch.Unlogged)
case object Counter extends BatchType(CQLSyntax.Batch.Counter)

}

sealed class BatchQuery[Status <: ConsistencyBound](
val iterator: Iterator[_ <: Statement],
val iterator: Iterator[Batchable with ExecutableStatement],
batchType: BatchType,
usingPart: UsingPart = UsingPart.empty,
added: Boolean = false,
Expand All @@ -45,31 +52,37 @@ sealed class BatchQuery[Status <: ConsistencyBound](
override def future()(
implicit session: Session,
ec: ExecutionContextExecutor
): ScalaFuture[ResultSet] = scalaQueryStringExecuteToFuture(makeBatch())
): ScalaFuture[ResultSet] = {
batchToPromise(makeBatch()).future
}

def initBatch(): BatchStatement = batchType match {
case BatchType.Logged => new BatchStatement()
case BatchType.Unlogged => new BatchStatement(BatchStatement.Type.UNLOGGED)
case BatchType.Counter => new BatchStatement(BatchStatement.Type.COUNTER)
}

def makeBatch()(implicit session: Session): Statement = {
def makeBatch()(implicit session: Session): BatchWithQuery = {
val batch = initBatch()

for (st <- iterator) {
batch.add(st)
}
val builder = List.newBuilder[String]

options.consistencyLevel match {
case Some(level) => batch.setConsistencyLevel(level)
case None => batch
for (st <- iterator) {
builder += st.queryString
batch.add(st.statement())
}

val statement = options.consistencyLevel.fold[Statement](batch)(l => batch.setConsistencyLevel(l))
val strings = builder.result()
BatchWithQuery(statement, strings.mkString("\n"), batchType)
}


@implicitNotFound("A ConsistencyLevel was already specified for this query.")
final def consistencyLevel_=(level: ConsistencyLevel)(implicit ev: Status =:= Unspecified, session: Session): BatchQuery[Specified] = {
final def consistencyLevel_=(level: ConsistencyLevel)(
implicit ev: Status =:= Unspecified,
session: Session
): BatchQuery[Specified] = {
if (session.protocolConsistency) {
new BatchQuery[Specified](
iterator,
Expand All @@ -91,7 +104,7 @@ sealed class BatchQuery[Status <: ConsistencyBound](

def add(query: Batchable with ExecutableStatement)(implicit session: Session): BatchQuery[Status] = {
new BatchQuery(
iterator ++ Iterator(query.statement()),
iterator ++ Iterator(query),
batchType,
usingPart,
added,
Expand All @@ -101,7 +114,7 @@ sealed class BatchQuery[Status <: ConsistencyBound](

def add(queries: Batchable with ExecutableStatement*)(implicit session: Session): BatchQuery[Status] = {
new BatchQuery(
iterator ++ queries.map(_.statement()).iterator,
iterator ++ queries.iterator,
batchType,
usingPart,
added,
Expand All @@ -111,7 +124,7 @@ sealed class BatchQuery[Status <: ConsistencyBound](

def add(queries: Iterator[Batchable with ExecutableStatement])(implicit session: Session): BatchQuery[Status] = {
new BatchQuery(
iterator ++ queries.map(_.statement()),
iterator ++ queries,
batchType,
usingPart,
added,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ package com.outworkers.phantom.builder.query

import com.datastax.driver.core.{PreparedStatement, Session, Statement, ResultSet => DatastaxResultSet}
import com.google.common.util.concurrent.{FutureCallback, Futures, ListenableFuture}
import com.outworkers.phantom.batch.BatchWithQuery
import com.outworkers.phantom.{Manager, ResultSet}
import com.outworkers.phantom.connectors.SessionAugmenterImplicits

Expand All @@ -28,10 +29,18 @@ private[phantom] trait CassandraOperations extends SessionAugmenterImplicits {
implicit session: Session,
executor: ExecutionContextExecutor
): ScalaFuture[ResultSet] = {
scalaQueryStringToPromise(st).future
statementToPromise(st).future
}

protected[this] def scalaQueryStringToPromise(st: Statement)(
protected[this] def batchToPromise(batch: BatchWithQuery)(
implicit session: Session,
executor: ExecutionContextExecutor
): ScalaPromise[ResultSet] = {
Manager.logger.debug(s"Executing query: ${batch.debugString}")
statementToPromise(batch.statement)
}

protected[this] def statementToPromise(st: Statement)(
implicit session: Session,
executor: ExecutionContextExecutor
): ScalaPromise[ResultSet] = {
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,78 @@
/*
* Copyright 2013 - 2017 Outworkers Ltd.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.outworkers.phantom.macros

import shapeless.{HList, HNil}

import scala.annotation.tailrec
import scala.reflect.macros.whitebox

/**
* A special implicitly materialized typeclass that compares HLists for equality.
* This also prints more useful information to the end user if the HLists don't match.
* However, it is required because the standard [[=:=]] type class in Scala
* is not able to see through an HList generated as an abstract type member of another
* implicitly materialized type class.
* @tparam LL The left [[shapeless.HList]] type.
* @tparam RR The right [[shapeless.HList]] type.
*/
trait ==:==[LL <: HList, RR <: HList]

object ==:== {

def apply[LL <: HList, RR <: HList](implicit ev: LL ==:== RR): ==:==[LL, RR] = ev

implicit def materialize[
LL <: HList,
RR <: HList
]: ==:==[LL, RR] = macro EqsMacro.materialize[LL, RR]
}


@macrocompat.bundle
class EqsMacro(val c: whitebox.Context) extends WhiteboxToolbelt with HListHelpers {

import c.universe._

protected[this] val macroPkg = q"_root_.com.outworkers.phantom.macros"


def mkEqs(left: Type, right: Type): Tree = {
if (left =:= right) {
val tree = q"""new $macroPkg.==:==[$left, $right] {}"""
if (showTrees) {
echo(showCode(tree))
}

tree
} else {
val debugString = s"Types ${showHList(left)} did not equal ${showHList(right)}"
error(debugString)
abort( debugString)
}
}

def materialize[LL <: HList : c.WeakTypeTag, RR <: HList : c.WeakTypeTag]: Tree = {
val left = weakTypeOf[LL]
val right = weakTypeOf[RR]

memoize[(Type, Type), Tree](
WhiteboxToolbelt.specialEqsCache
)(Tuple2(left, right), { case (t, s) => mkEqs(t, s)})
}


}
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ object BindHelper {
}

@macrocompat.bundle
class BindMacros(override val c: whitebox.Context) extends WhiteboxToolbelt(c) with RootMacro {
class BindMacros(override val c: whitebox.Context) extends WhiteboxToolbelt with RootMacro {

import c.universe._

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ object DatabaseHelper {
}

@macrocompat.bundle
class DatabaseHelperMacro(override val c: whitebox.Context) extends WhiteboxToolbelt(c) with RootMacro {
class DatabaseHelperMacro(override val c: whitebox.Context) extends WhiteboxToolbelt with RootMacro {
import c.universe._

private[this] val seqTpe: Tree => Tree = { tpe =>
Expand Down
Loading

0 comments on commit a565a15

Please sign in to comment.