-
Notifications
You must be signed in to change notification settings - Fork 15
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
Showing
1 changed file
with
112 additions
and
2 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,3 +1,113 @@ | ||
Use https://github.com/sakserv/hadoop-mini-clusters | ||
#Usage | ||
##Integration testing (will start each component present into classpath) | ||
With maven, add dependencies of components which are needed | ||
|
||
Need : https://github.com/jetoile/hadoop-bootstrap-test-utils | ||
Sample: | ||
```xml | ||
<dependency> | ||
<groupId>fr.jetoile.hadoop</groupId> | ||
<artifactId>hadoop-unit-hdfs</artifactId> | ||
<version>1.1-SNAPSHOT</version> | ||
<scope>test</scope> | ||
</dependency> | ||
``` | ||
|
||
In test do: | ||
```java | ||
@BeforeClass | ||
public static void setup() { | ||
HadoopBootstrap.INSTANCE.startAll(); | ||
} | ||
|
||
@AfterClass | ||
public static void tearDown() { | ||
HadoopBootstrap.INSTANCE.stopAll(); | ||
} | ||
``` | ||
|
||
##WIntegration testing v2 (with specific component) | ||
With maven, add dependencies of components which are needed | ||
|
||
Sample: | ||
```xml | ||
<dependency> | ||
<groupId>fr.jetoile.hadoop</groupId> | ||
<artifactId>hadoop-unit-hdfs</artifactId> | ||
<version>1.1-SNAPSHOT</version> | ||
<scope>test</scope> | ||
</dependency> | ||
``` | ||
|
||
In test do: | ||
```java | ||
@BeforeClass | ||
public static void setup() throws NotFoundServiceException { | ||
HadoopBootstrap.INSTANCE | ||
.start(Component.ZOOKEEPER) | ||
.start(Component.HDFS) | ||
.start(Component.HIVEMETA) | ||
.start(Component.HIVESERVER2); | ||
} | ||
|
||
@AfterClass | ||
public static void tearDown() throws NotFoundServiceException { | ||
HadoopBootstrap.INSTANCE | ||
.stop(Component.HIVESERVER2) | ||
.stop(Component.HIVEMETA) | ||
.stop(Component.HDFS) | ||
.stop(Component.ZOOKEEPER); | ||
} | ||
``` | ||
|
||
##Standalone mode | ||
Unzip hadoop-unit-standalone-<version>.tar.gz | ||
Change `conf/default.properties` | ||
Change `conf/hadoop.properties` | ||
|
||
Start in fg with: | ||
```bash | ||
./bin/hadoop-unit-standalone console | ||
``` | ||
|
||
Start in bg with: | ||
```bash | ||
./bin/hadoop-unit-standalone start | ||
``` | ||
|
||
Stop with: | ||
```bash | ||
./bin/hadoop-unit-standalone stop | ||
``` | ||
|
||
#Sample | ||
See hadoop-unit-standalone/src/test/java/fr/jetoile/hadoopunit/integrationtest | ||
|
||
#Component available | ||
|
||
* SolrCloud 5.4.1 | ||
* Kafka 0.9.0 | ||
* Hive (metastore and server2) | ||
* Hdfs | ||
* Zookeeper | ||
* Oozie (WIP) | ||
* HBase | ||
|
||
Built on: | ||
|
||
* hadoop-mini-cluster-0.1.3 (0.1.4 not released on central maven repo) (https://github.com/sakserv/hadoop-mini-clusters) | ||
|
||
Use: | ||
* download and unzip hadoop | ||
* download and unzip oozie (http://s3.amazonaws.com/public-repo-1.hortonworks.com/HDP/centos6/2.x/updates/2.3.4.0/tars/oozie-4.2.0.2.3.4.0-3485-distro.tar.gz) | ||
* edit default.properties and indicate HADOOP_HOME | ||
* edit default.properties and indicate oozie.sharelib.path | ||
|
||
Todo: | ||
* male client utils for kafka produce/consume | ||
* make sample with spark streaming and kafka | ||
|
||
Issues: | ||
* oozie does not work on windows 7 (see http://stackoverflow.com/questions/25790319/getting-access-denied-error-while-running-hadoop-2-3-mapreduce-jobs-in-windows-7) | ||
* integrate phoenix | ||
* can only manage one solr collection | ||
* better docs ;) |