zindexer

Fetches data from GitHub, Jira, CircleCI (and more to come) and pushes it to an Elasticsearch instance.

_This project aims at replacing ZenCrepes's github-indexer, adding abilities to fetch data from Jira (thus the need for a rename). _

zindexer
Introduction
Documentation
About Bit
Usage
Commands

Introduction

This script has been created to easily export Data from GitHub, Jira, CircleCI & more and import it into an Elasticsearch instance to be later used for data analytics (via ZenCrepes, Kibana or other)

Whenever possible (i.e. issues, milestones, projects), it loads data sorted by the updated date in descending order (most recent first) and will stop as soon as it find the same node already in Elasticsearch. This way, first load takes some time, then you can just cron it to keep your Elasticsearch instance up to date.

The overall logic is articulated around 3 stages:

Identify sources (GitHub repositories and/or Jira Projects) to load data from
[OPTIONAL] Select which sources to load data from by editing ~/.config/zindexer/sources.yml
Load data from the selected repositories (for example zindxer gIssues to load GitHub issues)

You can then re-run the scripts at regular interval to fetch the updated nodes.

Note: GitHub doesn't provide a mechanism to fetch new or updated labels so the script will (flush the index and) load all labels every time gLabels is executed.

Documentation

You can find ZenCrepes documentation on docs.zencrepes.io, issues should be created here.

This readme only contains developer-focused details.

About Bit

Bit components are exported from Zindexer, those are used to share logic between the various ZenCrepes services.

bit login
bit status
bit tag config 0.0.11 / bit tag --all 0.0.11
bit build
bit export zencrepes.zindexer

Add new component:

bit add src/components/testingPerfs

Usage

$ npm install -g zindexer
$ zindexer COMMAND
running command...
$ zindexer (-v|--version|version)
zindexer/1.1.16 linux-x64 node-v16.16.0
$ zindexer --help [COMMAND]
USAGE
  $ zindexer COMMAND
...

Commands

zindexer bamboo:runs
zindexer circleci:envvars
zindexer circleci:insights
zindexer circleci:pipelines
zindexer github:import
zindexer github:issues
zindexer github:labels
zindexer github:mavenpoms
zindexer github:milestones
zindexer github:projects
zindexer github:pullrequests
zindexer github:releases
zindexer github:repos
zindexer github:stargazers
zindexer github:vulnerabilities
zindexer github:watchers
zindexer help [COMMAND]
zindexer jira:ghexport
zindexer jira:issues
zindexer jira:projects
zindexer jira:remotelinks
zindexer sources [FILE]
zindexer startup
zindexer testing:perfs
zindexer testing:runs
zindexer testing:states
zindexer utils:updateLinks
zindexer zencrepes [FILE]

`zindexer bamboo:runs`

States: Initialize ES indeices for Testing States

States: Initialize ES indeices for Testing States

USAGE
  $ zindexer bamboo:runs

OPTIONS
  -a, --all                  Delete all issues from Elasticsearch and fetch all again
  -c, --config               Only update ZenCrepes configuration
  -h, --help                 show CLI help
  -p, --plan=plan            Fetch issues for a particular plan key
  -r, --reset                Reset ZenCrepes configuration to default
  --envUserConf=envUserConf  User Configuration passed as an environment variable, takes precedence over config file

See code: src/commands/bamboo/runs.ts

`zindexer circleci:envvars`

Fetches Environment variables from configured sources

Fetches Environment variables from configured sources

USAGE
  $ zindexer circleci:envvars

OPTIONS
  -c, --config               Only update ZenCrepes configuration
  -h, --help                 show CLI help
  -r, --reset                Reset ZenCrepes configuration to default
  --envUserConf=envUserConf  User Configuration passed as an environment variable, takes precedence over config file

See code: src/commands/circleci/envvars.ts

`zindexer circleci:insights`

Fetches insights data from configured sources

Fetches insights data from configured sources

USAGE
  $ zindexer circleci:insights

OPTIONS
  -c, --config               Only update ZenCrepes configuration
  -h, --help                 show CLI help
  -r, --reset                Reset ZenCrepes configuration to default
  --envUserConf=envUserConf  User Configuration passed as an environment variable, takes precedence over config file

See code: src/commands/circleci/insights.ts

`zindexer circleci:pipelines`

Fetches pipelines data from configured sources

Fetches pipelines data from configured sources

USAGE
  $ zindexer circleci:pipelines

OPTIONS
  -c, --config               Only update ZenCrepes configuration
  -h, --help                 show CLI help
  -r, --reset                Reset ZenCrepes configuration to default
  --envUserConf=envUserConf  User Configuration passed as an environment variable, takes precedence over config file

See code: src/commands/circleci/pipelines.ts

`zindexer github:import`

Github: Imports data to GitHub

Github: Imports data to GitHub

USAGE
  $ zindexer github:import

OPTIONS
  -a, --action=submit|check|resubmit|crosscheck  [default: submit] Import action to be performed
  -h, --help                                     show CLI help

  --envUserConf=envUserConf                      User Configuration passed as an environment variable, takes precedence
                                                 over config file

See code: src/commands/github/import.ts

`zindexer github:issues`

Github: Fetches issues data from configured sources

Github: Fetches issues data from configured sources

USAGE
  $ zindexer github:issues

OPTIONS
  -a, --all                  Clear nodes before fetching all of them again
  -c, --config               Only update ZenCrepes configuration
  -h, --help                 show CLI help
  -r, --reset                Reset ZenCrepes configuration to default
  --envUserConf=envUserConf  User Configuration passed as an environment variable, takes precedence over config file

See code: src/commands/github/issues.ts

`zindexer github:labels`

Github: Fetches labels attached to configured sources

Github: Fetches labels attached to configured sources

USAGE
  $ zindexer github:labels

OPTIONS
  -c, --config               Only update ZenCrepes configuration
  -h, --help                 show CLI help
  -r, --reset                Reset ZenCrepes configuration to default
  --envUserConf=envUserConf  User Configuration passed as an environment variable, takes precedence over config file

See code: src/commands/github/labels.ts

`zindexer github:mavenpoms`

Github: Fetches maven data from repositories default branch

Github: Fetches maven data from repositories default branch

USAGE
  $ zindexer github:mavenpoms

OPTIONS
  -c, --config               Only update ZenCrepes configuration
  -h, --help                 show CLI help
  -r, --reset                Reset ZenCrepes configuration to default
  --envUserConf=envUserConf  User Configuration passed as an environment variable, takes precedence over config file

See code: src/commands/github/mavenpoms.ts

`zindexer github:milestones`

Github: Fetches milestones data from configured sources

Github: Fetches milestones data from configured sources

USAGE
  $ zindexer github:milestones

OPTIONS
  -a, --all                  Clear nodes before fetching all of them again
  -c, --config               Only update ZenCrepes configuration
  -h, --help                 show CLI help
  -r, --reset                Reset ZenCrepes configuration to default
  --envUserConf=envUserConf  User Configuration passed as an environment variable, takes precedence over config file

See code: src/commands/github/milestones.ts

`zindexer github:projects`

Github: Fetches projects data from configured sources

Github: Fetches projects data from configured sources

USAGE
  $ zindexer github:projects

OPTIONS
  -a, --all                  Clear nodes before fetching all of them again
  -c, --config               Only update ZenCrepes configuration
  -h, --help                 show CLI help
  -r, --reset                Reset ZenCrepes configuration to default
  --envUserConf=envUserConf  User Configuration passed as an environment variable, takes precedence over config file

See code: src/commands/github/projects.ts

`zindexer github:pullrequests`

Github: Fetches Pullrequests data from configured sources

Github: Fetches Pullrequests data from configured sources

USAGE
  $ zindexer github:pullrequests

OPTIONS
  -a, --all                  Clear nodes before fetching all of them again
  -c, --config               Only update ZenCrepes configuration
  -h, --help                 show CLI help
  -r, --reset                Reset ZenCrepes configuration to default
  --envUserConf=envUserConf  User Configuration passed as an environment variable, takes precedence over config file

See code: src/commands/github/pullrequests.ts

`zindexer github:releases`

Github: Fetches releases data from configured sources

Github: Fetches releases data from configured sources

USAGE
  $ zindexer github:releases

OPTIONS
  -a, --all                  Clear nodes before fetching all of them again
  -c, --config               Only update ZenCrepes configuration
  -h, --help                 show CLI help
  -r, --reset                Reset ZenCrepes configuration to default
  --envUserConf=envUserConf  User Configuration passed as an environment variable, takes precedence over config file

See code: src/commands/github/releases.ts

`zindexer github:repos`

Github: Fetches repos data from configured sources

Github: Fetches repos data from configured sources

USAGE
  $ zindexer github:repos

OPTIONS
  -c, --config               Only update ZenCrepes configuration
  -h, --help                 show CLI help
  -r, --reset                Reset ZenCrepes configuration to default
  --envUserConf=envUserConf  User Configuration passed as an environment variable, takes precedence over config file

See code: src/commands/github/repos.ts

`zindexer github:stargazers`

Github: Fetches Stargazers data from configured sources

Github: Fetches Stargazers data from configured sources

USAGE
  $ zindexer github:stargazers

OPTIONS
  -c, --config               Only update ZenCrepes configuration
  -h, --help                 show CLI help
  -r, --reset                Reset ZenCrepes configuration to default
  --envUserConf=envUserConf  User Configuration passed as an environment variable, takes precedence over config file

See code: src/commands/github/stargazers.ts

`zindexer github:vulnerabilities`

Github: Fetches Vulnerabilities data from configured sources

Github: Fetches Vulnerabilities data from configured sources

USAGE
  $ zindexer github:vulnerabilities

OPTIONS
  -c, --config               Only update ZenCrepes configuration
  -h, --help                 show CLI help
  -r, --reset                Reset ZenCrepes configuration to default
  --envUserConf=envUserConf  User Configuration passed as an environment variable, takes precedence over config file

See code: src/commands/github/vulnerabilities.ts

`zindexer github:watchers`

Github: Fetches Watchers data from configured sources

Github: Fetches Watchers data from configured sources

USAGE
  $ zindexer github:watchers

OPTIONS
  -c, --config               Only update ZenCrepes configuration
  -h, --help                 show CLI help
  -r, --reset                Reset ZenCrepes configuration to default
  --envUserConf=envUserConf  User Configuration passed as an environment variable, takes precedence over config file

See code: src/commands/github/watchers.ts

`zindexer help [COMMAND]`

display help for zindexer

display help for <%= config.bin %>

USAGE
  $ zindexer help [COMMAND]

ARGUMENTS
  COMMAND  command to show help for

OPTIONS
  --all  see all commands in CLI

See code: @oclif/plugin-help

`zindexer jira:ghexport`

Jira: Prepare Jira issues for export to GitHub

Jira: Prepare Jira issues for export to GitHub

USAGE
  $ zindexer jira:ghexport

OPTIONS
  -h, --help                 show CLI help
  --envUserConf=envUserConf  User Configuration passed as an environment variable, takes precedence over config file

See code: src/commands/jira/ghexport.ts

`zindexer jira:issues`

Jira: Fetches issues data from configured sources

Jira: Fetches issues data from configured sources

USAGE
  $ zindexer jira:issues

OPTIONS
  -a, --all                  Delete all issues from Elasticsearch and fetch all again
  -c, --config               Only update ZenCrepes configuration
  -h, --help                 show CLI help
  -p, --project=project      Fetch issues for a particular source ID
  -r, --reset                Reset ZenCrepes configuration to default
  --envUserConf=envUserConf  User Configuration passed as an environment variable, takes precedence over config file

See code: src/commands/jira/issues.ts

`zindexer jira:projects`

Jira: Fetches project data from configured sources

Jira: Fetches project data from configured sources

USAGE
  $ zindexer jira:projects

OPTIONS
  -h, --help                 show CLI help
  --envUserConf=envUserConf  User Configuration passed as an environment variable, takes precedence over config file

See code: src/commands/jira/projects.ts

`zindexer jira:remotelinks`

Jira: Fetches remote links from configured projects

Jira: Fetches remote links from configured projects

USAGE
  $ zindexer jira:remotelinks

OPTIONS
  -a, --action=LOAD|SAVE|FETCH  [default: FETCH] Action to be performed. (SAVE locally, LOAD into es, FETCH from Jira)
  -h, --help                    show CLI help
  --envUserConf=envUserConf     User Configuration passed as an environment variable, takes precedence over config file

See code: src/commands/jira/remotelinks.ts

`zindexer sources [FILE]`

Manage data sources (GitHub or Jira)

Manage data sources (GitHub or Jira)

USAGE
  $ zindexer sources [FILE]

OPTIONS
  -a, --active                     Automatically make the new sources active by default
  -g, --ggrab=affiliated|org|repo  [default: affiliated] If Github, Select how to fetch repositories
  -h, --help                       show CLI help
  -l, --load                       Load active status from file: CONFIG_DIR/sources.yml
  -o, --gorg=gorg                  If Github, organization login
  -r, --grepo=grepo                If Github, repository name
  -t, --type=JIRA|GITHUB|BAMBOO    [default: GITHUB] Type of source (JIRA, GitHUB or BAMBOO)

  --envUserConf=envUserConf        User Configuration passed as an environment variable, takes precedence over config
                                   file

See code: src/commands/sources.ts

`zindexer startup`

Generates a config file is none is present

Generates a config file is none is present

USAGE
  $ zindexer startup

OPTIONS
  -h, --help  show CLI help

See code: src/commands/startup.ts

`zindexer testing:perfs`

Perfs: Initialize ES indices for Testing Perfs

Perfs: Initialize ES indices for Testing Perfs

USAGE
  $ zindexer testing:perfs

OPTIONS
  -c, --config               Only update ZenCrepes configuration
  -d, --delete               Delete the corresponding ES index and re-create (WARNING, DANGEROUS)
  -h, --help                 show CLI help
  -r, --reset                Reset ZenCrepes configuration to default
  --envUserConf=envUserConf  User Configuration passed as an environment variable, takes precedence over config file

See code: src/commands/testing/perfs.ts

`zindexer testing:runs`

Runs: Initialize ES indices for Testing Runs

Runs: Initialize ES indices for Testing Runs

USAGE
  $ zindexer testing:runs

OPTIONS
  -c, --config               Only update ZenCrepes configuration
  -h, --help                 show CLI help
  -r, --reset                Reset ZenCrepes configuration to default
  --envUserConf=envUserConf  User Configuration passed as an environment variable, takes precedence over config file

See code: src/commands/testing/runs.ts

`zindexer testing:states`

States: Initialize ES indices for Testing States

States: Initialize ES indices for Testing States

USAGE
  $ zindexer testing:states

OPTIONS
  -c, --config               Only update ZenCrepes configuration
  -h, --help                 show CLI help
  -r, --reset                Reset ZenCrepes configuration to default
  --envUserConf=envUserConf  User Configuration passed as an environment variable, takes precedence over config file

See code: src/commands/testing/states.ts

`zindexer utils:updateLinks`

Scan all GitHub issues for Jira links and replaces with corresponding GitHub link

Scan all GitHub issues for Jira links and replaces with corresponding GitHub link

USAGE
  $ zindexer utils:updateLinks

OPTIONS
  -h, --help                 show CLI help
  --envUserConf=envUserConf  User Configuration passed as an environment variable, takes precedence over config file

See code: src/commands/utils/updateLinks.ts

`zindexer zencrepes [FILE]`

Manage Zencrepes (UI) configuration

Manage Zencrepes (UI) configuration

USAGE
  $ zindexer zencrepes [FILE]

OPTIONS
  -h, --help                 show CLI help
  -l, --load                 Load configuration from file: CONFIG_DIR/zencrepes.yml
  -s, --save                 Save configuration to file: CONFIG_DIR/zencrepes.yml
  --envUserConf=envUserConf  User Configuration passed as an environment variable, takes precedence over config file

See code: src/commands/zencrepes.ts

zindexer

zindexer

Introduction

Documentation

About Bit

Usage

Commands

zindexer bamboo:runs

zindexer circleci:envvars

zindexer circleci:insights

zindexer circleci:pipelines

zindexer github:import

zindexer github:issues

zindexer github:labels

zindexer github:mavenpoms

zindexer github:milestones

zindexer github:projects

zindexer github:pullrequests

zindexer github:releases

zindexer github:repos

zindexer github:stargazers

zindexer github:vulnerabilities

zindexer github:watchers

zindexer help [COMMAND]

zindexer jira:ghexport

zindexer jira:issues

zindexer jira:projects

zindexer jira:remotelinks

zindexer sources [FILE]

zindexer startup

zindexer testing:perfs

zindexer testing:runs

zindexer testing:states

zindexer utils:updateLinks

zindexer zencrepes [FILE]

Readme

Keywords

Package Sidebar

Install

Repository

Homepage

DownloadsWeekly Downloads

Version

License

Unpacked Size

Total Files

Last publish

Collaborators

`zindexer bamboo:runs`

`zindexer circleci:envvars`

`zindexer circleci:insights`

`zindexer circleci:pipelines`

`zindexer github:import`

`zindexer github:issues`

`zindexer github:labels`

`zindexer github:mavenpoms`

`zindexer github:milestones`

`zindexer github:projects`

`zindexer github:pullrequests`

`zindexer github:releases`

`zindexer github:repos`

`zindexer github:stargazers`

`zindexer github:vulnerabilities`

`zindexer github:watchers`

`zindexer help [COMMAND]`

`zindexer jira:ghexport`

`zindexer jira:issues`

`zindexer jira:projects`

`zindexer jira:remotelinks`

`zindexer sources [FILE]`

`zindexer startup`

`zindexer testing:perfs`

`zindexer testing:runs`

`zindexer testing:states`

`zindexer utils:updateLinks`

`zindexer zencrepes [FILE]`

Weekly Downloads