Skip to content

Git and Github Guide

Jump to bottom
Sean Kelly edited this page Aug 15, 2024 · 43 revisions

Getting Started with Github

Instead of reinventing the wheel, there are countless trainings online. Here are few good ones we have found:

GitHub Workflow for Developers

The PDS generally follows the GitHub flow, which is a lightweight, branch-based workflow that supports teams and projects where deployments are made regularly. The guide here explains how and why GitHub flow works. Depending upon the application, we may skip the official "deploy" step, but there is usually some level of deployment, at least locally, in order to properly test the application.

Creating a New Repo

Creating a new repo under NASA-PDS org is restricted to the @NASA-PDS/pds-software-pmc team.

New repositories can be of 3 types:

  • library: a re-usable piece of code. The CICD must publish stable versions on artifactories such as maven, pypi, npm...
  • micro-service: an elementary component which will run as a service. The CICD must publish stable version on a docker image repository (e.g. docker hub, AWS/ECR)
  • application: a user facing application. The repository contains, user's manual, integration tests and deployment scripts configuration (e.g. docker-compose, terraform...). Applications can either refer to libraries and microservices but can also contain their own code.

When creating a new repo, here are some things we should set by default for each repo:

  • Use template repo whenever possible:
  • Update CODEOWNERS with the applicable team that will manage the code, if different than @NASA-PDS/pds-software-committers
  • Assign teams:
    • Always assign the @NASA-PDS/pds-software-committers team and give it write access
    • Always assign the @NASA-PDS/pds-software-pmc team and give it admin access
    • Always assign the @NASA-PDS/pdsen-operations team and give it admin access
    • Always assign the CODEOWNERS team(s) and give it admin access
  • Enable GitHub pages:
    • From Settings → Pages → Branch gh-pages, 📁 / Root, then click Save
  • Setup main branch protections:
    • Go to Settings → Branches → Branch Protections → Branch Protections
    • Enter Branch name pattern = main
    • Select:
      • Check Require a pull request before merging, then check Require approvals and set Number required approvals before merging: 1; then also check Require review from Code Owners
      • Check Require status checks to pass before merging, and in the "search for status checks" box, find and enable secret-detection
      • Check Require branches to be up to date before merging
      • Restrict who can push to matching branches
      • Allow force pushes
      • Allow deletions
    • Click Create to submit
  • Add the DOI assignment webhook (only for repositories that should have digital object identifiers assigned to releases):
    • Go to Zenodo and click Login
    • Log in with GitHub; use the PDS Engineering Node Continuous Integration account, pdsen-ci. You can use your own account, however, Zenodo makes frequent use of the GitHub API and those calls will be charged to your own account. If you are concerned about API limits, use the pdsen-ci account instead.
    • From the profile menu (upper right), click the ▼ and choose GitHub.
    • Find the new repository under "Repositories" and slide the switch from Off to On.
    • On GitHub or in a clone of the new repository, create a new file .zenodo.json with this single line—{"license": "Apache-2.0", "communities": [{"identifier": "nasa-pds"}]}—and commit this change. `
  • Update the README.md and remove/replace any placeholder information.

Python Repositories

After you create the repository, create the corresponding projects on the Python Package Index ("cheese shop"). Note that there are two instances of the cheese shop, pypi.org and test.pypi.org. You'll need to update both instances.

First, create the projects on pypi.org and test.pypi.org. The easiest way to do this is to first build your distributable artifacts; from your local repository directory, run:

python3 -m build .

to build the source and wheel distributable artifacts, which will go into a newly created dist subdirectory. Then, upload the artifacts to both pypi.org and test.pypi.org:

twine upload dist/*
twine upload --repository-url https://test.pypi.org/legacy/ dist/*

👉 Note: Setting up a ~/.pypirc file can simplify things. Such a file might look like:

[pypi]
repository = https://upload.pypi.org/legacy/
username = __token__
password = pypi-TOKEN-HERE

[testpypi]
repository = https://test.pypi.org/legacy/
username = __token__
password = pypi-TEST-TOKEN-HERE

Then you can use twine upload dist/* and twine upload --repository testpypi dist/* instead of the --repository-url option—and avoid password prompts.

Second, you'll then need to visit https://pypi.org/ and https://test.pypi.org/, find your newly created project, click "Manage", and add the following people (at a minimum) to the project with the Role of Owner:

  • alexdunnjpl
  • collinss-jpl
  • jpadams
  • nutjob
  • pdsen-ci
  • tloubrieu
  • viviant (note: on pypi.org there are multiple users that have the prefix viviant; ensure you select only viviant)

DOI Assignment

If the DOI assignment webhook was set up above (and the .zenodo.json file created), then each time a release is made (whether automatically or by hand), a new digital object identifier will be assigned to the release (and one assigned one time only to the entire repository, called a "concept DOI"). Once a DOI is assigned, you can add a badge to the repository's README by adding (using Markdown syntax) the following:

[![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.XXXXXXX.svg)](https://doi.org/10.5281/zenodo.XXXXXXX)

Replace XXXXXXX with the concept DOI that got assigned.

Github Notifications

Github email notifications can become very difficult to parse through without proper email management and setup. The following section helps you pull high priority emails out to ensure you do not miss out on various activities that require your more immediate attention.

Setup Notifications

To ensure you are up-to-date with all things happening in repositories, check your notification settings send you emails mobile notifications when needed. For PDS development, at minimum, please check the following:

  • Participating:
    ✅ Email ✅ Web and Mobile

  • Watching:
    ✅ Email ✅ Web and Mobile

Managing Repository Notifications

You can manage the notifications you receive for specific repositories from here: https://github.com/watching . From this page, you can determine what types of notifications you would like to receive from repositories you are watching.

For more information on managing notifications, see the Github documentation

@mentions Email Filter

Think of a Github @mention as a direct message, versus other issue and commit notifications being group messages. It is highly recommended you setup an email filter to pull out these @mentions (aka direct messages) so you are able to respond in a timely manner.

Here is an example email filter from Microsoft Exchange:

When a new message arrives that meets all these conditions:
  Body     Contains    @my_username
  Sent To  Contains    mention@noreply.github.com
  From     Contains    github.com

Do the following:
  Move to Folder      Github-@mentions (you will need to create this folder)

[checked] Do not apply other rules to message that meet the same conditions

[checked] Enabled

Pull Requests Review Requested Self Email Filter

In order to prioritize pull request reviews where you are explicitly called out above other Github email notifications, this email filter will help you pull out emails where your review is requested.

Note: This applies to both personal reviews and reviews requested of a team you belong to.

Here is an example email filter from Microsoft Exchange:

When a new message arrives that meets all these conditions:
  Body     Contains    requested your review
  Sent To  Contains    review_requested@noreply.github.com
  From     Is          notifications@github.com

Do the following:
  Move to Folder      Github-@reviews (you will need to create this folder)

[checked] Do not apply other rules to message that meet the same conditions

[checked] Enabled

Pull Requests Review Requested Team Email Filter

In order to prioritize pull request reviews for Github teams that you are apart of above other Github email notifications, this email filter will help you pull out emails where your review is requested.

Note: This applies to both personal reviews and reviews requested of a team you belong to.

Here is an example email filter from Microsoft Exchange:

When a new message arrives that meets all these conditions:
  Sent To  Contains    review_requested@noreply.github.com
  From     Is          notifications@github.com

Do the following:
  Move to Folder      Github-@reviews-team (you will need to create this folder)

[checked] Do not apply other rules to message that meet the same conditions

[checked] Enabled

Github Discussions Email Filter

This email filter will pull all discussion-related notifications into a folder.

NOTE: Make sure the @mentions filter comes before this one.

When a new message arrives that meets all these conditions:
  Subject     Contains    "(Discussion #"
  From     Contains    notifications@github.com

Do the following:
  Move to Folder      Github-@discussions (you will need to create this folder)

[checked] Do not apply other rules to message that meet the same conditions

[checked] Enabled

Github Notifications Email Filter

This email filter is intended to be a catchall for remaining github notifications for a particular organization. This can be adapted to handle individual repositories as you see fit.

NOTE: Make sure this is the last email filter in your list, otherwise the filters created above will be ignored.

Here is an example email filter for the NASA-PDS Github Org through Microsoft Exchange:

When a new message arrives that meets all these conditions:
  Subject  Contains    NASA-PDS/
  From     Contains    "noreply@github.com" or "notifications@github.com"

Do the following:
  Move to Folder      PDS-Github (you will need to create this folder)

[checked] Do not apply other rules to message that meet the same conditions

[checked] Enabled

Here is an example email filter for the PDS-Data-Dictionaries Github Org through Microsoft Exchange:

When a new message arrives that meets all these conditions:
  Subject  Contains    pds-data-dictionaries/
  From     Contains    "noreply@github.com" or "notifications@github.com"

Do the following:
  Move to Folder      pds-ldd-github (you will need to create this folder)

[checked] Do not apply other rules to message that meet the same conditions

[checked] Enabled

Git Secrets

As of late 2023, Git Secrets is no longer being used by the Planetary Data System's Engineering node. We have transitioned to Detect Secrets, which is described below. This section, however, remains for historical use.

Accidentally committing a change that contains a credential—such as a password or API key—is regrettable but avoidable thanks to Git Secrets. The Engineering Node's template repositories include support for Git Secrets in their pre-commit hooks.

However, one extra step must be done one time only: setting up your Git global configuration to understand the patterns of secrets to avoid and those to allow. Doing so is easy:

  1. Install Git Secrets
  2. Run git secrets --register-aws --global

This adds patterns to your ~/.gitconfig so that commits and commit messages that contain AWS secrets will be aborted. You can edit your ~/.gitconfig (or use git secrets --add) to include more patterns to watch out for.

Detect Secrets

Intro

The Planetary Data System's Engineering Node uses detect-secrets, specifically a version forked by the SLIM Team, called slim-detect-secrets. This tool replaces Git Secrets but serves the same purpose: it helps prevent committing information to a repository that should remain secret. Unlike Git Secrets, though, the kinds of secrets detect-secrets finds includes not just passwords or API keys, but also hostnames and email addresses. It also uses entropy analysis to detect randomized strings that could be passwords.

Detect Secrets is already integrated into the Python template repository and the Java template repository, so creating new repositories from those templates will give you the support for Detect Secrets.

However, you must manually install the detect-secrets command-line tool and make it available on your shell's execution PATH in order to support pre-commit and in the event you need to create or update the .secrets.baseline file associated with your repository.

Installation

The tool is written in Python, so you'll also need Python. Many systems come with Python already, but if it's not on your system, you can install it from the Microsoft Store (on Windows) or from popular package managers on other systems—or directly from the Python website.

We recommend creating a virtual environment for Detect Secrets. For example, as a Unix (such as Linux or macOS) user, you might do:

python3 -m venv ~/Tools/detect-secrets
cd ~/Tools/detect-secrets
bin/pip install git+https://github.com/NASA-AMMOS/slim-detect-secrets.git@exp

Then add ~/Tools/detect-secrets/bin to your PATH.

Create or Update Secrets Baseline

Each repo should have a .secrets.baseline file at the top level of the repo in order to avoid false results in the execution of the Secrets Detection scan in GitHub Actions. The baseline tells what "secrets" are actually okay, things like email addresses we want in the repository, or which example API keys are just example API keys and not actual ones, etc. It's a catalog of false positives.

To establish or update a secrets baseline in your repository depends on the kind of repository.

For Python:

detect-secrets scan --disable-plugin AbsolutePathDetectorExperimental \
    --exclude-files '\.secrets..*' \
    --exclude-files '\.git.*' \
    --exclude-files '\.pre-commit-config\.yaml' \
    --exclude-files '\.mypy_cache' \
    --exclude-files '\.pytest_cache' \
    --exclude-files '\.tox' \
    --exclude-files '\.venv' \
    --exclude-files 'venv' \
    --exclude-files 'dist' \
    --exclude-files 'build' \
    --exclude-files '.*\.egg-info' \
    > .secrets.baseline

For Java:

detect-secrets scan --disable-plugin AbsolutePathDetectorExperimental \
    --exclude-files '\.git.*' \
    --exclude-files '\.pre-commit-config\.yaml' \
    --exclude-files '\.secrets..*' \
    --exclude-files 'target' \
    > .secrets.baseline

And for Node.js:

detect-secrets scan --disable-plugin AbsolutePathDetectorExperimental \
    --exclude-files '\.git.*' \
    --exclude-files '\.pre-commit-config\.yaml' \
    --exclude-files '\.secrets..*' \
    --exclude-files 'node_modules'\
    > .secrets.baseline

Review the secrets to determine which should be allowed and which are false positives:

detect-secrets audit .secrets.baseline

Please remove any secrets that should not be seen by the public. You can then add the baseline file to the commit:

git add .secrets.baseline

Commit Message Best Practices

For commit message best practices, the PDS tries to follow these Github Commit Messages Guidelines where possible.

Once you have read through the guidelines, here are some highlights to touch on and some additional notes:

A properly formed git commit subject line should always be able to complete the following sentence

If applied, this commit will <your subject line here>
  • Rules for a great commit message style
    • Separate subject from body with a blank line
    • Do not end the subject line with a period
    • Capitalize the subject line and each paragraph
    • Use the imperative mood in the subject line
    • Wrap lines at 72 characters
    • Use the body to explain what and why you have done something. In most cases, you can leave out details about how a change has been made.
  • References in commit messages
    • If the commit refers to an issue, add this information to the commit message header or body.
    • See how to autolink references and urls for more information on linking to other issues, repos, commits, etc.
    • Do this for ticket references, people, etc.
    • Be sure use the correct referencing method when going across repositories.
Short (72 chars or less) summary

More detailed explanatory text. Wrap it to 72 characters. The blank
line separating the summary from the body is critical (unless you omit
the body entirely).

Write your commit message in the imperative: "Fix bug" and not "Fixed
bug" or "Fixes bug." This convention matches up with commit messages
generated by commands like git merge and git revert.

Further paragraphs come after blank lines.

- Bullet points are okay, too.
- Typically a hyphen or asterisk is used for the bullet, followed by a
  single space. Use a hanging indent.

For example:

$ git add my_file.txt

# Try to avoid using `git commit -m` unless it is a very trivial update/change
$ git commit

# Vim or Emacs window will open. To change editor that opens, see https://help.github.com/en/github/using-git/associating-text-editors-with-git .
Created new my_file with text in it

This new file includes:
* Part 1
* Part 2
* Part 3

Issue Tracking

See more information at PDS Issue Tracking Guide

Advanced Help

Help With Rebasing

Rebasing can get messy and be painful. We highly recommend reading this article on The Golden Rule of Rebasing.

References

Copyright © 2021-2024 California Institute of Technology.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Clone this wiki locally