zindexer

Fetches data from GitHub, Jira, CircleCI (and more to come) and pushes it to an Elasticsearch instance.

_This project aims at replacing ZenCrepes's github-indexer, adding abilities to fetch data from Jira (thus the need for a rename). _

• Introduction
• Quick start with Docker
• Configuration
• Usage
• Commands

Introduction

This script has been created to easily export Data from GitHub, Jira, CircleCI & more and import it into an Elasticsearch instance to be later used for data analytics (via ZenCrepes, Kibana or other)

Whenever possible (i.e. issues, milestones, projects), it loads data sorted by the updated date in descending order (most recent first) and will stop as soon as it find the same node already in Elasticsearch. This way, first load takes some time, then you can just cron it to keep your Elasticsearch instance up to date.

The overall logic is articulated around 3 stages:

• Identify sources (GitHub repositories and/or Jira Projects) to load data from
• [OPTIONAL] Select which sources to load data from by editing ~/.config/zindexer/sources.yml
• Load data from the selected repositories (for example zindxer gIssues to load GitHub issues)

You can then re-run the scripts at regular interval to fetch the updated nodes.

Note: GitHub doesn't provide a mechanism to fetch new or updated labels so the script will (flush the index and) load all labels every time gLabels is executed.

Documentation

You can find ZenCrepes documentation on docs.zencrepes.io, issues should be created here.

This readme only contains developer-focused details.

About Bit

Bit components are exported from Zindexer, those are used to share logic between the various ZenCrepes services.

• https://bit.dev/
• https://docs.bit.dev/docs/quick-start
• https://medium.com/javascript-in-plain-english/how-i-share-react-components-between-projects-3896d853cbee

bit login
bit status
bit tag config 0.0.11 / bit tag --all 0.0.11
bit build
bit export zencrepes.zindexer

Add new component:

bit add src/components/testingPerfs

Usage

$ npm install -g zindexer
$ zindexer COMMAND
running command...
$ zindexer (-v|--version|version)
zindexer/0.0.1 darwin-x64 node-v12.13.0
$ zindexer --help [COMMAND]
USAGE
  $ zindexer COMMAND
...

Commands

• zindexer gIssues
• zindexer gLabels [FILE]
• zindexer gMilestones
• zindexer gProjects
• zindexer gPullrequests
• zindexer gReleases
• zindexer gRepos
• zindexer help [COMMAND]
• zindexer jIssues
• zindexer jProjects
• zindexer sources [FILE]

zindexer gIssues

Github: Fetches issues data from configured sources

USAGE
  $ zindexer gIssues

OPTIONS
  -h, --help  show CLI help

See code: src/commands/gIssues.ts

zindexer gLabels [FILE]

describe the command here

USAGE
  $ zindexer gLabels [FILE]

OPTIONS
  -f, --force
  -h, --help       show CLI help
  -n, --name=name  name to print

See code: src/commands/gLabels.ts

zindexer gMilestones

Github: Fetches milestones data from configured sources

USAGE
  $ zindexer gMilestones

OPTIONS
  -h, --help  show CLI help

See code: src/commands/gMilestones.ts

zindexer gProjects

Github: Fetches projects data from configured sources

USAGE
  $ zindexer gProjects

OPTIONS
  -h, --help  show CLI help

See code: src/commands/gProjects.ts

zindexer gPullrequests

Github: Fetches Pullrequests data from configured sources

USAGE
  $ zindexer gPullrequests

OPTIONS
  -h, --help  show CLI help

See code: src/commands/gPullrequests.ts

zindexer gReleases

Github: Fetches releases data from configured sources

USAGE
  $ zindexer gReleases

OPTIONS
  -h, --help  show CLI help

See code: src/commands/gReleases.ts

zindexer gRepos

Github: Fetches repos data from configured sources

USAGE
  $ zindexer gRepos

OPTIONS
  -h, --help  show CLI help

See code: src/commands/gRepos.ts

zindexer help [COMMAND]

display help for zindexer

USAGE
  $ zindexer help [COMMAND]

ARGUMENTS
  COMMAND  command to show help for

OPTIONS
  --all  see all commands in CLI

See code: @oclif/plugin-help

zindexer jIssues

Jira: Fetches issues data from configured sources

USAGE
  $ zindexer jIssues

OPTIONS
  -h, --help  show CLI help

See code: src/commands/jIssues.ts

zindexer jProjects

Jira: Fetches project data from configured sources

USAGE
  $ zindexer jProjects

OPTIONS
  -h, --help  show CLI help

See code: src/commands/jProjects.ts

zindexer sources [FILE]

Manage sources of data

USAGE
  $ zindexer sources [FILE]

OPTIONS
  -a, --active                     Automatically make the new sources active by default
  -g, --ggrab=affiliated|org|repo  [default: affiliated] If Github, Select how to fetch repositories
  -h, --help                       show CLI help
  -l, --load                       Load active status from status file: CONFIG_DIR/sources.yml
  -o, --gorg=gorg                  If Github, organization login
  -r, --grepo=grepo                If Github, repository name
  -t, --type=JIRA|GITHUB           [default: GITHUB] Type of source (JIRA or GitHUB)

See code: src/commands/sources.ts