Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

SPARK-1637: Clean up examples for 1.0 #571

Closed
wants to merge 4 commits into from
Closed

SPARK-1637: Clean up examples for 1.0 #571

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
0a8dd77
Move all Scala Examples to org.apache.spark.examples (some are in org…
techaddict Apr 27, 2014
5f96121
Move Python examples into examples/src/main/python
techaddict Apr 27, 2014
8ed2d3f
Docs Updated for changes, Change for java examples
techaddict Apr 27, 2014
47ef86c
Changes based on Discussions on PR, removing use of RawTextHelper fro…
techaddict Apr 30, 2014
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions examples/src/main/java/org/apache/spark/streaming/examples/JavaFlumeEventCount.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@
package org.apache.spark.streaming.examples;

import org.apache.spark.api.java.function.Function;
import org.apache.spark.examples.streaming.StreamingExamples;
import org.apache.spark.streaming.*;
import org.apache.spark.streaming.api.java.*;
import org.apache.spark.streaming.flume.FlumeUtils;
Expand Down
1 change: 1 addition & 0 deletions examples/src/main/java/org/apache/spark/streaming/examples/JavaKafkaWordCount.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@
import org.apache.spark.api.java.function.Function;
import org.apache.spark.api.java.function.Function2;
import org.apache.spark.api.java.function.PairFunction;
import org.apache.spark.examples.streaming.StreamingExamples;
import org.apache.spark.streaming.Duration;
import org.apache.spark.streaming.api.java.JavaDStream;
import org.apache.spark.streaming.api.java.JavaPairDStream;
Expand Down
1 change: 1 addition & 0 deletions examples/src/main/java/org/apache/spark/streaming/examples/JavaNetworkWordCount.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@
import org.apache.spark.api.java.function.FlatMapFunction;
import org.apache.spark.api.java.function.Function2;
import org.apache.spark.api.java.function.PairFunction;
import org.apache.spark.examples.streaming.StreamingExamples;
import org.apache.spark.streaming.Duration;
import org.apache.spark.streaming.api.java.JavaDStream;
import org.apache.spark.streaming.api.java.JavaPairDStream;
Expand Down
1 change: 1 addition & 0 deletions examples/src/main/java/org/apache/spark/streaming/examples/JavaQueueStream.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@
import org.apache.spark.api.java.JavaRDD;
import org.apache.spark.api.java.function.Function2;
import org.apache.spark.api.java.function.PairFunction;
import org.apache.spark.examples.streaming.StreamingExamples;
import org.apache.spark.streaming.Duration;
import org.apache.spark.streaming.api.java.JavaDStream;
import org.apache.spark.streaming.api.java.JavaPairDStream;
Expand Down
2 changes: 1 addition & 1 deletion ...ache/spark/sql/examples/RDDRelation.scala → ...ache/spark/examples/sql/RDDRelation.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
* limitations under the License.
*/

package org.apache.spark.sql.examples
package org.apache.spark.examples.sql

import org.apache.spark.SparkContext
import org.apache.spark.sql.SQLContext
Expand Down
2 changes: 1 addition & 1 deletion ...he/spark/sql/examples/HiveFromSpark.scala → ...ark/examples/sql/hive/HiveFromSpark.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
* limitations under the License.
*/

package org.apache.spark.sql.hive.examples
package org.apache.spark.examples.sql.hive

import org.apache.spark.SparkContext
import org.apache.spark.sql._
Expand Down
2 changes: 1 addition & 1 deletion ...k/streaming/examples/ActorWordCount.scala → ...k/examples/streaming/ActorWordCount.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
* limitations under the License.
*/

package org.apache.spark.streaming.examples
package org.apache.spark.examples.streaming

import scala.collection.mutable.LinkedList
import scala.reflect.ClassTag
Expand Down
2 changes: 1 addition & 1 deletion .../streaming/examples/FlumeEventCount.scala → .../examples/streaming/FlumeEventCount.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
* limitations under the License.
*/

package org.apache.spark.streaming.examples
package org.apache.spark.examples.streaming

import org.apache.spark.storage.StorageLevel
import org.apache.spark.streaming._
Expand Down
2 changes: 1 addition & 1 deletion ...rk/streaming/examples/HdfsWordCount.scala → ...rk/examples/streaming/HdfsWordCount.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
* limitations under the License.
*/

package org.apache.spark.streaming.examples
package org.apache.spark.examples.streaming

import org.apache.spark.streaming.{Seconds, StreamingContext}
import org.apache.spark.streaming.StreamingContext._
Expand Down
2 changes: 1 addition & 1 deletion ...k/streaming/examples/KafkaWordCount.scala → ...k/examples/streaming/KafkaWordCount.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
* limitations under the License.
*/

package org.apache.spark.streaming.examples
package org.apache.spark.examples.streaming

import java.util.Properties

Expand Down
2 changes: 1 addition & 1 deletion ...rk/streaming/examples/MQTTWordCount.scala → ...rk/examples/streaming/MQTTWordCount.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
* limitations under the License.
*/

package org.apache.spark.streaming.examples
package org.apache.spark.examples.streaming

import org.eclipse.paho.client.mqttv3.{MqttClient, MqttClientPersistence, MqttException, MqttMessage, MqttTopic}
import org.eclipse.paho.client.mqttv3.persist.MqttDefaultFilePersistence
Expand Down
2 changes: 1 addition & 1 deletion ...streaming/examples/NetworkWordCount.scala → ...examples/streaming/NetworkWordCount.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
* limitations under the License.
*/

package org.apache.spark.streaming.examples
package org.apache.spark.examples.streaming

import org.apache.spark.streaming.{Seconds, StreamingContext}
import org.apache.spark.streaming.StreamingContext._
Expand Down
2 changes: 1 addition & 1 deletion ...park/streaming/examples/QueueStream.scala → ...park/examples/streaming/QueueStream.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
* limitations under the License.
*/

package org.apache.spark.streaming.examples
package org.apache.spark.examples.streaming

import scala.collection.mutable.SynchronizedQueue

Expand Down
2 changes: 1 addition & 1 deletion ...k/streaming/examples/RawNetworkGrep.scala → ...k/examples/streaming/RawNetworkGrep.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
* limitations under the License.
*/

package org.apache.spark.streaming.examples
package org.apache.spark.examples.streaming

import org.apache.spark.storage.StorageLevel
import org.apache.spark.streaming._
Expand Down
2 changes: 1 addition & 1 deletion ...xamples/RecoverableNetworkWordCount.scala → ...reaming/RecoverableNetworkWordCount.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
* limitations under the License.
*/

package org.apache.spark.streaming.examples
package org.apache.spark.examples.streaming

import org.apache.spark.streaming.{Time, Seconds, StreamingContext}
import org.apache.spark.streaming.StreamingContext._
Expand Down
2 changes: 1 addition & 1 deletion ...g/examples/StatefulNetworkWordCount.scala → .../streaming/StatefulNetworkWordCount.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
* limitations under the License.
*/

package org.apache.spark.streaming.examples
package org.apache.spark.examples.streaming

import org.apache.spark.streaming._
import org.apache.spark.streaming.StreamingContext._
Expand Down
2 changes: 1 addition & 1 deletion ...treaming/examples/StreamingExamples.scala → ...xamples/streaming/StreamingExamples.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
* limitations under the License.
*/

package org.apache.spark.streaming.examples
package org.apache.spark.examples.streaming

import org.apache.spark.Logging

Expand Down
2 changes: 1 addition & 1 deletion ...reaming/examples/TwitterAlgebirdCMS.scala → ...amples/streaming/TwitterAlgebirdCMS.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
* limitations under the License.
*/

package org.apache.spark.streaming.examples
package org.apache.spark.examples.streaming

import com.twitter.algebird._

Expand Down
2 changes: 1 addition & 1 deletion ...reaming/examples/TwitterAlgebirdHLL.scala → ...amples/streaming/TwitterAlgebirdHLL.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
* limitations under the License.
*/

package org.apache.spark.streaming.examples
package org.apache.spark.examples.streaming

import com.twitter.algebird.HyperLogLogMonoid
import com.twitter.algebird.HyperLogLog._
Expand Down
2 changes: 1 addition & 1 deletion ...reaming/examples/TwitterPopularTags.scala → ...amples/streaming/TwitterPopularTags.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
* limitations under the License.
*/

package org.apache.spark.streaming.examples
package org.apache.spark.examples.streaming

import org.apache.spark.streaming.{Seconds, StreamingContext}
import StreamingContext._
Expand Down
2 changes: 1 addition & 1 deletion .../streaming/examples/ZeroMQWordCount.scala → .../examples/streaming/ZeroMQWordCount.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
* limitations under the License.
*/

package org.apache.spark.streaming.examples
package org.apache.spark.examples.streaming

import akka.actor.ActorSystem
import akka.actor.actorRef2Scala
Expand Down
2 changes: 1 addition & 1 deletion ...mples/clickstream/PageViewGenerator.scala → ...aming/clickstream/PageViewGenerator.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
* limitations under the License.
*/

package org.apache.spark.streaming.examples.clickstream
package org.apache.spark.examples.streaming.clickstream

import java.net.ServerSocket
import java.io.PrintWriter
Expand Down
4 changes: 2 additions & 2 deletions ...examples/clickstream/PageViewStream.scala → ...treaming/clickstream/PageViewStream.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,12 +15,12 @@
* limitations under the License.
*/

package org.apache.spark.streaming.examples.clickstream
package org.apache.spark.examples.streaming.clickstream

import org.apache.spark.SparkContext._
import org.apache.spark.streaming.{Seconds, StreamingContext}
import org.apache.spark.streaming.StreamingContext._
import org.apache.spark.streaming.examples.StreamingExamples
import org.apache.spark.examples.streaming.StreamingExamples
// scalastyle:off
/** Analyses a streaming dataset of web page views. This class demonstrates several types of
* operators available in Spark streaming.
Expand Down
5 changes: 2 additions & 3 deletions streaming/src/main/scala/org/apache/spark/streaming/util/RawTextHelper.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,10 +22,10 @@ import org.apache.spark.SparkContext._
import org.apache.spark.util.collection.OpenHashMap
import scala.collection.JavaConversions.mapAsScalaMap

private[streaming]
private[spark]
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure about this, needed by RawNetworkGrep and KafkaWordCount. There has to be better way.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We shouldn't use any private methods in examples. The usage of RawTextHelper in KafkaWordCount is very simple. warmUp is used in RawNetworkGrep, is it necessary? @tdas

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well, the RawTextHelper was essentially the way to provide data for running benchmarks. However, I havent quite maintained it, so that example code has been languishing in unknown state. I am inclined to keep it there as it is, as long as it hidden behind private[*]. Though I dont see the logic behind changing it from private[streaming] to private[spark]

object RawTextHelper {

/**
/**
* Splits lines and counts the words.
*/
def splitAndCountPartitions(iter: Iterator[String]): Iterator[(String, Long)] = {
Expand Down Expand Up @@ -114,4 +114,3 @@ object RawTextHelper {

def max(v1: Long, v2: Long) = math.max(v1, v2)
}