SPARK-938 - Openstack Swift object storage support #1010
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This is initial documentation describing how to integrate Spark with Swift. This commit contains documentation for stand alone cluster. Next patches will contain details how to integrate Swift in other deployment of Spark.
|
Can one of the admins verify this patch? |
|
|
||
|
|
||
| <h2>Configuring Spark - stand alone cluster</h2> | ||
| You need to configure the compute-classpath.sh and add Hadoop classpath for |
There was a problem hiding this comment.
Is the swift jar not included in hadoop-client? Is there a way to specify this through Maven dependencies rather than manually including the path?
Documentation how to integrate Spark with Openstack Swift.
Conflicts: docs/openstack-integration.md
Conflicts: docs/openstack-integration.md
| To configure Hadoop to work with Swift one need to modify core-sites.xml of Hadoop and | ||
| setup Swift FS. | ||
|
|
||
| <configuration> |
There was a problem hiding this comment.
Is this needed? Can we just put this in core-site.xml under conf? (Basically removing the configuring Hadoop section)
|
Removed all openstack dependencies from pom.xml |
SPARK-938 - Openstack Swift object storage support
| @@ -132,7 +132,7 @@ | |||
| <codahale.metrics.version>3.0.0</codahale.metrics.version> | |||
| <avro.version>1.7.6</avro.version> | |||
| <jets3t.version>0.7.1</jets3t.version> | |||
|
|
|||
Bring the branch up to date and fixed some documentation typos.
|
Can one of the admins verify this patch? |
rxin
added a commit
to rxin/spark
that referenced
this pull request
Sep 6, 2014
SPARK-938 - Openstack Swift object storage support
|
This is subsumed by #2298 |
asfgit
pushed a commit
that referenced
this pull request
Sep 8, 2014
See compiled doc at http://people.apache.org/~rxin/tmp/openstack-swift/_site/storage-openstack-swift.html This is based on #1010. Closes #1010. Author: Reynold Xin <rxin@apache.org> Author: Gil Vernik <gilv@il.ibm.com> Closes #2298 from rxin/openstack-swift and squashes the following commits: ff4e394 [Reynold Xin] Two minor comments from Patrick. 279f6de [Reynold Xin] core-sites -> core-site dfb8fea [Reynold Xin] Updated based on Gil's suggestion. 846f5cb [Reynold Xin] Added a link from overview page. 0447c9f [Reynold Xin] Removed sample code. e9c3761 [Reynold Xin] Merge pull request #1010 from gilv/master 9233fef [Gil Vernik] Fixed typos 6994827 [Gil Vernik] Merge pull request #1 from rxin/openstack ac0679e [Reynold Xin] Fixed an unclosed tr. 47ce99d [Reynold Xin] Merge branch 'master' into openstack cca7192 [Gil Vernik] Removed white spases from pom.xml 99f095d [Reynold Xin] Pending openstack changes. eb22295 [Reynold Xin] Merge pull request #1010 from gilv/master 39a9737 [Gil Vernik] Spark integration with Openstack Swift c977658 [Gil Vernik] Merge branch 'master' of https://github.com/gilv/spark 2aba763 [Gil Vernik] Fix to docs/openstack-integration.md 9b625b5 [Gil Vernik] Merge branch 'master' of https://github.com/gilv/spark eff538d [Gil Vernik] SPARK-938 - Openstack Swift object storage support ce483d7 [Gil Vernik] SPARK-938 - Openstack Swift object storage support b6c37ef [Gil Vernik] Openstack Swift support (cherry picked from commit eddfedd) Signed-off-by: Patrick Wendell <pwendell@gmail.com>
wangyum
pushed a commit
that referenced
this pull request
May 26, 2023
…ectory (#1010) * [CARMEL-6076][Followup] Fix UT failure caused by compacting empty directory dddd tttt * show path
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is very initial commit to initiate discussion about Spark and OpenStack Swift integration.
At this point I provided information how to setup standalone Spark cluster and connect it to Swift using Hadoop 2.3.0. Current patch can be used by all to experiment integration between Swift and Spark.
I am working to extend this patch and future patches will contain information how to configure Spark with previous Hadoop versions and how to configure Swift for other cluster deployment of Spark.
Will glad to hear all remarks / suggestion / comments from the community.