Skip to content

Latest commit

 

History

History
535 lines (408 loc) · 20.9 KB

README.MD

File metadata and controls

535 lines (408 loc) · 20.9 KB
Raw

Udacimak - A Udacity Nanodegree Downloader

npm version Downloads Github stars License Commits to be deployed PRs Welcome

Opened issues Closed issues Opened pull requests Closed pull requests Github last commit

Table of Contents

Description

Udacimak is a command-line interface tool to download Udacity Nanodegree contents and keep them locally on your computer.

Udacimak downloads all videos, images, text contents and present them as local web pages in a similar manner to Udacity classroom website.

Udacimak was inspired after Udacity announced in October 2018 that students who graduate will no longer have life-time access to the course content. Instead, all access will be removed 12 months after graduation date. There are also cases where access to content will be removed immediately after graduation (eg. if you're an Enterprise or Scholarship student, etc.)

If you want to keep the contents to yourself, Udacimak is built for you; or if you would like to have a local copy to quickly revise the lessons while doing the course, Udacimak is also a good option.

We hope you enjoy this downloader.

Stay Udacious!

If you find this CLI helpful, please support the developers by starring this repository.

Resources:

Installation

  1. Make sure you have node and npm installed. Install via nvm is highly recommended.
  2. Install git
  3. To install Udacimak, run:
npm install -g udacimak

Update

To update to the latest version, run the same install command:

npm install -g udacimak

Change Log

We do not generate change log files for this project.

But if desired, you can check changes in a new release by looking at the Git commit messages. The commit messages prior to a version number commit inform the changes that has been made to that version release.

For example, from the commit messages below, we know what changes have been made in version 1.1.1, 1.1.2 and 1.1.3

# version 1.1.3 example
afb3ade (tag: v1.1.3) 1.1.3
c0f7b4f Add user agent for requests
ad93fb3 Update README
fdb9b96 Add user agent for requests
129a9cd Add test setup

# version 1.1.2 example
597e727 (tag: v1.1.2) 1.1.2
1dc07e6 Update README
b9c4f2a Add case handle for wrong url in addHttp

# version 1.1.1 example
01f7308 (tag: v1.1.1) 1.1.1
03ab9ad Add handle case for wrong url in addHttp
19c4ae0 Add handle case for wrong url in addHttp
da55eef Update README
6346a68 Fix bug makeDir in CheckboxQuizAtom

Use Udacimak via Docker container

There is a Udacimak image that you can use. Download the image by running:

docker pull udacimak/udacimak

Udacimak is the entrypoint of the image so you can directly use it. Examples:

# show help
docker container run -it -v "$(pwd)/:/downloads" udacimak/udacimak --help

Remember to use volumes (-v) so that downloaded files are saved onto host machine.

You can also ssh into the container to directly work in their instead by changing entrypoint to /bin/bash:

docker container run --rm -it -v "$(pwd)/:/downloads" --entrypoint /bin/bash udacimak/udacimak

Remember to set the --targetdir option to the shared volume (eg. "downloads" in the examples above) so that downloaded or rendered contents are saved directly into the host machine, eg:

docker container run -it -v "$(pwd)/:/downloads" udacimak/udacimak render "downloads/_json/Front-End Web Developer Nanodegree nd001 v7.0.0" --targetdir "downloads"

How to use

Udacimak allows you to keep Udacity content locally in 2 main steps:

  1. Download: Download means fetching all JSON data of a course or Nanodegree from Udacity API and store them locally as .json files.
  2. Render: Render means converting the downloaded course JSON data into "readable" files. It will download all videos, images and render all contents into HTML files. You will use these HTML files to view the materials on your computer.

Please read further instruction below for detailed usage.

Contents

settoken: Set Udacity Authentication Token

First, your Udacity authentication token (officially called JSON Web Tokens (JWT)) should be configured so that the cli can download course contents from Udacity API:

udacimak settoken YOUR_UDACITY_AUTH_TOKEN

Udacity authentication token can be found in Udacity website browser cookies. To get your Udacity authentication, follow these steps:

  1. Visit classroom.udacity.com and log in
  2. Open the Developer Tools (press Cmd + Opt + J on MacOS or F12 on Windows)
  3. Go to Application tab, then Storage > Cookies > https://classroom.udacity.com.
  4. Find _jwt key, its Value is the Udacity authentication token.

Alternatively, copy the Javascript code below in the Console in Developer Tools, it will get the token and copy it to clipboard for you:

/**
 * Retrieve cookie by name
 **/
function getCookie(name) {
  var value = "; " + document.cookie;
  var parts = value.split("; " + name + "=");
  if (parts.length == 2) return parts.pop().split(";").shift();
}

// get jwt and copy to clipboard
copy(getCookie('_jwt'));

Security Note

Udacity authentication token is a secret key to is unique to you and you should not share it publicly.

The secret token also will expire automatically after a period of time. If you find Udacimak's listnd and download commands unusable due to unauthorized error, try to get the token again from the browser and settoken with the new token.

listnd: List your Nanodegree keys

To download the Nanodegrees, you need to know the Nanodegree key to pass to the command.

Nanodegree key is an id that is assigned to a Nanodegree in the format of ndXXX, ndXXXt, etc. where XXX is the number assigned to that Nanodegree. You can see it in the Udacity classroom or catalog url. The two examples below shows urls of Nanodegree key nd1337:

https://classroom.udacity.com/nanodegrees/nd1337/syllabus/core-curriculum

or

https://www.udacity.com/course/cybersecurity-nanodegree--nd1337

You can list the Nanodegree keys of the ones you have graduated and currently enrolled in as such:

udacimak listnd

Additional Information

Note that you for each Nanodegree you will have an associated version and language (locale) that you can download. Udacity occasionally updates their Nanodegree content, hence the Nanodegree version. You will only have access to the latest version at the time you purchased the Nanodegree.

Course key

Udacity also provides lots of free courses. If you're interested in downloading these courses, head to https://www.udacity.com/courses/all, click on a course that you're interested in and get its key in the url. For example, the course key in the url below is ud281:

https://www.udacity.com/course/high-performance-computing--ud281

download: Download course content as JSON data from Udacity API

download command will fetch all necessary JSON data from Udacity API to download them and save them locally:

udacimak download nd001

# or download multiple Nanodegree and courses:
udacimak download nd001 nd002 st101

download Example GIF

You can have a peek at these files to see what they contain. The JSON data files are not "readable". But note that they're the core contents that you need to save the Nanodegree/course locally to your computer.

To view the content in a readable manner, one could build a web app that read the JSON files and render it in HTML format. This can be a tedious process since you'll need to study the JSON structure built by Udacity to process them. That is why Udacimak provide render functionality to render all the course contents into static web pages.

Important downloaded directory note

You should not modify the structure of the downloaded directory:

  • Do not create new directory inside the downloaded directory.
  • Do not render into the downloaded directory.
  • Do not rename any .json files.
  • Not critical, but avoid renaming directory names inside the downloaded directory.

Note regarding workspaces

Udacity has a number of types of workspaces for projects and quizzes/exercises, including Jupyter Notebook workspaces, IDE workspaces. The files of these workspaces are not downloadable, so all your workspaces will not be accessible from Udacimak.

It is recommended that you download them manually. For Jupyter Notebook workspaces, please refer to this StackOverflow answer on how to download all files in one go.

Please also note that some of the workspace materials are available on Udaciy Github page, so have a peek there first to see if you can clone them.

render: Render downloaded JSON course contents

After you have downloaded all course content as JSON data from Udacity API, it is now time to convert them into readable format. render command is provided to do this:

# --targetdir is optional
udacimak render PATH_TO_DOWNLOADED_COURSE_JSON_DIRECTORY --targetdir OPTIONAL_TARGET_DIRECTORY

Parameters:

-t, --targetdir <targetdir>  (Optional) Target directory to save rendered course contents
-v, --verbose                (Optional) Force youtube-dl to log debugging information
--userquizanswer             (Optional) Force rendering user's Programming Question code answer

Render Example GIF

render command will:

  • ✅ Create summary page for Nanodegree and all lessons to easily navigate the course content and find all lessons
  • ✅ For each lesson:
    • Download all videos (including subtitles), images and create lesson contents as HTML files to view on your favorite browser.
    • Create HTML files for labs, project descriptions and project rubrics pages.

It is also ❗ important to know what render is not capable of:

  • Quiz: for all quizzes:
    • ❌ It is not possible to "submit" the quizzes to check for result from Udacity API. We simply can't have access to Udacity API to check the quizzes.
    • ❌ For a few quiz types, (Programming Quiz and Image Format Quiz) there are no solutions provided.
  • ❌ Workspaces: As noted above, Udacimak can't download the workspace files for you. So the lesson pages that contain workspaces will not display any workspace for you.

The process can take a few hours due to large amount of videos to download. Once it is done, you can enjoy the lessons on your computer:

HTML Lesson Example 01

HTML Lesson Example 02

Important render directory note

Do not render into any target locations that are sub-diretories of a downloaded directory. If you try to do so, the renderer will throw error. In other words, separate the directories for downloaded and render directories, do not render into target directories inside downloaded directories.

For example, this is an example of folder structure that will work:

├── Udacity Nanodegrees
│   ├── # ✔ This directory can be target directory for render command
│   ├── _json/
│   │   ├── # ✔ This directory contains downloaded JSON data using download command
│   │   ├── # ❌ Any render target that points into directories inside _json/ will fail!!!
│   ├── _renders/
│   │   ├── # ✔ Alternatively, this can also be target directory for render command

Re-rendering a Course

If you cancel the command or the CLI is cut off for some reason (eg. the CLI stops because internet is disconnected); or when the CLI is updated with new rendering features (better formatting, fixing missing content, etc.) you can re-run the render process later. Just use the same command, the CLI will:

  • NOT re-download video/image files that are already downloaded. Only a few image files may be re-downloaded because the original image url to download doesn't contain image file name, which means Udacimak won't be able to tell if those images have been downloaded or not.
  • Re-create all HTML files

rerender Example GIF

As long as Udacimak sees that the folder name in the --targetdir dir (or the current terminal working directory if --targetdir is not provided) is the same as the source folder that contains course JSON data, it will think that this is a re-render, and will avoid re-downloading any online media files that already exist in the course folder.

Watching Video with Subtitles

Since v1.1.x, Udacimak supports showing subtitles on the videos on the web video player. On Firefox, the subtitles should show up normally. If you use Google Chrome or Microsoft Edge, a web server needs to be set up for the subtitles to show up.

To set up a web server, you can use npm serve package, or Python 3's http.server module, etc.

It is important that you set up a server from the Nanodegree or course's root directory. Otherwise, when going to the lesson pages, they won't be able to find the assets folder to load the required libraries for the web page to run properly.

Known Issue: On Google Chrome, when viewing the pages with a web server, you will not be able to "jump" the video. See here for why:

renderdir: Iteratively render multiple JSON course contents

This is a convenient method to render all the courses that you have downloaded. If you have downloaded Nanodegree/course contents into a single directory, you can use renderdir to render all of them:

udacimak renderdir PATH_TO_DIRECTORY

Parameters:

-t, --targetdir <targetdir>  (Optional) Target directory to save rendered course contents
-v, --verbose                (Optional) Force youtube-dl to log debugging information
--userquizanswer             (Optional) Force rendering user's Programming Question code answer

Log Files

Please note that the CLI will produce log files in the working directory of your terminal, including:

  • udacimak.error.log
  • udacimak.info.log

The log files will contain the latest CLI output of the commands. They are limited to 10MB per file

If you face any issue, you may want to share the section of the log file content that contains error log to the developer when reporting the issue.

Note to Windows OS users:

The CLI is prone to some known issues when using on Windows:

  • 260-character file path limit: When facing this issue, you may have trouble viewing video file, moving files, etc. Windows will never fix this file path limitation, so...
  • Chinese character in file name: Node application seems to have issue working with file names that contain Chinese, Portuguese character on Windows. So, when rendering the course JSON, it may fails to name the Youtube subtitle files correctly.

It is recommended that you use a Linux system to download and render the course to make sure all files are processed safely.

If you use Windows, you will need to accept that:

  • You will have trouble moving/copying the downloaded/rendered course folder. Though you can find softwares that can get around this issue for you.
  • Some subtitles won't be named according to video files (so that video players such as VLC can automatically detect them)

Breaking Changes from Udacity API

Because of the complexity of the course structure, the CLI is error prone to changes being made in Udacity API, especially in a newly released course. If the JSON structures from the API have changed, we will have to reverse-engineer again and make update to Udacimak to adapt to the new structure.

Please open an issue in Issues tab to help us keep the CLI up-to-date if you face any problems using the CLI.

CLI Update Note

When Udacimak is updated to a newer version, it may contain new rendering features. You can check the changes of each version on the Commits page

In this case, you may want to run the render process again. It should take a few seconds per Nanodegree/course because the CLI doesn't re-download online media files when re-render a course. Also see note above

Reporting Issues

Before reporting any issues, please make sure you're using the latest version of Udacimak first:

# show current version
npm list udacimak
# or
udacimak --version

# update udacimak
npm install -g udacimak

# check current version
udacimak --version

If the issue persists, please open issue in Issues tab. Please give us as much information as you can. However, note that you should not provide your Udacity authentication token to anyone - it's a secret token of your own account.

User Privacy

Since you will have to provide the secret Udacity authentication token for the CLI to work, the CLI does have access to your personal information via Udacity API, including name, email address, graduated/enrolled nanodegrees, etc. The token is saved in the CLI config, located at:

# for Linux
~/.udacimak

# for Windows
%USERPROFILE%/.udacimak

However, these details are only used internally in the app to serve its functionalities. None of your personal details, as well as the secret authentication token, will be sent to anybody else.

Disclaimer

This CLI is provided to help you download Udacity Nanodegrees and courses for personal use only. Sharing the content of your subscribed Nanodegrees and courses is strictly prohibited under Udacity's Terms of Use.

By using this CLI, the developers of this CLI are not responsible for any law infringement caused by the users of this CLI.

Resources: