Skip to content

DFS-Lib is a scala flavoured api to the Hadoop java filesystem api

Notifications You must be signed in to change notification settings

fbraza/scala-dfs-lib

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

26 Commits
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

DFS-Lib

logo

❗ work in progress ❗

Functionality

DFS-Lib provides a simple Scala interface for the HDFS filesystem API.

Setup

Usage

Right now and because work is in progress DFS-Lib does not aim to be a complete replacement for the org.apache.hadoop.fs. That being said, DFS-Lib should provide all necessary functions to perform basic and advanced file operations.

touch

It creates a file to the specified path with default permissions. Keep in mind, that any missing parent directory found in the path is also created (default behavior brought by the java API, it might change in the future). You can decide to (i) overwrite the file, set (ii) the hadoop replication factor, (iii) the hadoop block size, (iv) the writing buffer size. It returns true when operation is successful:

val fs = yourHadoopClusterInstance.getFileSystem()
val pathFile = "parent/directory/test_file01.txt"
dfs.touch(fs = fs, path = pathFile)

mkdir

mv

mv.into

mv.over

cp

rm

For developers

Author

Acknowledgement

I would like to thank @lihaoyi for his fabulous scala libraries and notably the OS-Lib tool he developed. I got heavely inspired by the way code is written there.

About

DFS-Lib is a scala flavoured api to the Hadoop java filesystem api

Topics

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages