README.Rmd

---
output: github_document
---

<!-- README.md is generated from README.Rmd. Please edit that file -->

```{r, echo = FALSE}
knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>",
  fig.path = "README-",
  eval = FALSE
)
```

# condor

The goal of condor is to run R batch commands in `HTCondor` on a remote cluster directly from a local terminal or `RStudio` work environment. 

## Installation

You can install condor from github with:

```{r gh-installation}
# install.packages("devtools")
devtools::install_github("yuliasidi/condor")
```

## SSH Key Setup

  - Installation of the R package [ssh](https://www.github.com/ropensci/ssh)
  - setting up an ssh key
    - create a key pairing on the local machine
    - on the remote machine 
      - In the user directory (`~`) create the subdirectory `~/.ssh` if it is not already there
        - `mkdir ~/.ssh`
      - Create a file in `~/.ssh` called `authorized_keys`
        - `touch ~/.ssh/authorized_keys`
    - copy the public key contents to the remote server into .ssh/authorized_keys
      - `cat ~/.ssh/id_rsa.pub | pbcopy` (mac)
    - paste the contents of the clipboard on the remote into `~/.ssh/authorized_keys`
      - `echo '[PASTE CONTENTS OF CLIPBOARD]' > ~/.ssh/authorized_keys`

## Workflow

### Load Libraries

```{r remedy001}
library(condor)
library(ssh)

```

### Preprocessing

Populate package template and create `Rcalcpi.condor` in `example` subdir. 

```{r remedy002}
condor::build_template(
  file = 'calcpi.R',
  args = c('$(Process)'),
  tag = 'pi',
  jobs = 5,
  init_dir = 'jobs/run',
  template_file = 'example/Rcalcpi.condor',
  job_type = 'test')


```

Lines in file that will be run on the cluster

```{r,eval = TRUE}
readLines(system.file('example/calcpi.R',package='condor'))
```

Lines in the populated condor file

```{r,eval = TRUE}
readLines(system.file('example/Rcalcpi.condor',package='condor'))
```

Connect to ssh 

```{r remedy003}
session <- ssh::ssh_connect(Sys.getenv('UCONN_USER'))

```

Upload files needed for the job to the cluster

```{r remedy004}
ssh::scp_upload(session,
                files = c('example/calcpi.R',
                          'example/Rcalcpi.condor', 
                          'example/emailMyself.txt'),
                to = '~'
)

```

Create directories needed for job outputs

```{r remedy005}
condor::create_dirs(session, file = 'example/Rcalcpi.condor')

```

Submit the jobs

```{r remedy006}
condor::condor_submit(session,'Rcalcpi.condor')

```

### During the job

```{r remedy007}
condor::condor_q(session)

```

```{r}
condor::condor_rm(session,'5000.1')

```

### Post Processing

Retrieve the files

```{r remedy008}
condor::pull(session,
             from = c('jobs/run/log',
                      'jobs/run/out',
                      'jobs/run/err',
                      'jobs/run/*.rds'),
             to = c('output',
                    'output',
                    'output',
                    'output/data'))

```

Remove files from the cluster

```{r remedy009}
condor::cleanup_remote(session)

```

Collect output from the local into a single object

```{r remedy010}
RETURN <- purrr::map(
  list.files(path = "example/output/data",full.names = TRUE),
  readRDS
  )

```

Remove files from the local machine

```{r remedy011}
condor::cleanup_local(dir = 'example/output',tag = 'pi')

```

Close ssh connection

```{r remedy012}
ssh::ssh_disconnect(session)

```