Accessing Big Data Files

Accessing data files:

1. in code to determine the root directory based on OS (like here), which I can turn into an easy to use function that would supply the prefix to any paths subsequently set in the code.

• TODO: come up with analog to cross-platform setup.sh of Darren's flow script.

1. on network

• at UCSB

  • on Mac in Finder menu Go > Connect to Server... smb://neptune/data_edit. Then it's available locally at /Volumes/data_edit.

  • on Windows in Windows Explorer, right click on Computer > Map network drive... \\neptune\data_edit to drive N:. Then it's available locally at N:\.

• outside UCSB see details at Mapping network drive via SSH on Linux, Windows and Mac

  • on Mac

    1. open Terminal and run the following 2 commands substituting bbest with your username

       sudo ifconfig lo0 127.0.0.2 alias up
       sudo ssh -NL 127.0.0.2:139:localhost:139 -NL 127.0.0.2:445:localhost:445 bbest@neptune.nceas.ucsb.edu

    2. in Finder menu, Go > Connect to Server... smb://bbest@127.0.0.2/data_edit, substituting bbest with your username. Then it's available locally at /Volumes/data_edit.

1. offline via:

• Cyberduck For the case of having an internet connection but outside UCSB, we previously suggested using local copies of files copied over with Cyberduck. This can make it tricky to determine which files need to be copied back to the server, although there is a Cyberduck synchronization option.
• Dropbox analogue For larger data files, there's automated sync'ing options like Dropbox and Google Drive. Nick has been exploring open-source options like OwnCloud, which now may be mature enough to host on neptune. I like seeing OwnCloud's Add Folder option for selectively being able to choose which folders to sync akin to Dropbox's Selective Sync. This is critical for being able to locally store only the currently needed folders from neptune and not the entire server.