Skip to content

A database iterator that includes a set of tools to help manage documents in a CouchDB database.

License

Notifications You must be signed in to change notification settings

dsquier/couchit

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

5 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Couchit

Couchit is a database iterator with tools to validate and manage documents in a CouchDB database.

Couchit runs a set of user-defined JavaScript functions against all documents in a CouchDB database or view, or only some of them by specifying a start and/or an end key(s). Couchit comes with a built-in set of Utility Functions to perform helpful operations such as generating hashes, performing json-schema validation, and storing objects from iteration for post-processing.

Installation

Command Line Interface (CLI)

Couchit can run as a stand-alone application. Install it via npm:

npm install -g couchit

Then, to run:

couchit ./config.js

As a module

Couchit can also run as a module in another program. First, add it to your projects package.json

npm install --save couchit

Then, include it in your program:

const Couchit = require('couchit');
const config = require('./config.js');
new Couchit().iterate(config, console.log);

Configuration

Couchit can be controlled by environment variables or a config file.

The preferred way is through environment variables. This allows Couchit to run without storing sensititve information in code. It will be useful when setting Couchit to run as an AWS Lambda, for example.

Configuration Settings

The following list all configuration settings. If no environment variables are set, the Default Value will be used.

Setting Description Default Value
COUCHDB_ENDPOINT URI and port of CouchDB server, (does not include http://) 'localhost:5984'
COUCHDB_DATABASE CouchDB database 'db'
COUCHDB_USERNAME CouchDB user 'couchdb'
COUCHDB_PASSWORD CouchDB password 'couchdb'
OPTS_INTERVAL Number of ms to wait between page requests 100
OPTS_START_KEY View start key null
OPTS_END_KEY View end key null
OPTS_PAGE_SIZE Number of documents to retrieve per batch 1000
OPTS_NUM_PAGES Number of pages to retrieve undefined
OPTS_BATCH_SIZE Batch update size 100
OPTS_QUIET Suppress report at end of run false
OPTS_TASKS JavaScript functions with tasks to run { "count-docs": (util, doc) => { util.count('total-docs') } }

You can use a combination of environment variables and config.js Default Values to run Couchit. Just remember that environment variables always override defaults.

Setting an environment variable in Windows Powershell

$env:COUCHDB_PASSWORD="couchdb"

Setting an environment variable in macOS/Linux bash

export COUCHDB_PASSWORD="couchdb"

Tasks

Tasks are how you validate and manage documents as they are iterated over. There are a number of built-in Utility Functions that can be used by calling their util method, for example, to get a document hash: util.hash(doc). Additional document functionality is provided via util.nano, which exposes nano document functions. See the dependent-updates task below for an example of how nano document functions can be used.

Utility Functions

On each document iteration, the following functions are available via the Util() object:

Function Description
audit Add an object to the audit array, which is returned in the callback
count Increment a counter associated with a particular key
dereference Reference external json-schema file definitions from other files
log Alias for console.log
hash Generate a SHA256 hash for a given document, object, or string
nano Exposes nano document functions
remove Delete the document from the database
save Save the document back to the database
validate Validate a document using json-schema
incrementWaits Set a wait. Used to ensure asyncronous process can complete for callback response
decrementWaits Remove a wait

Example Tasks

You can define any number of named tasks to run, which will be run once per document. The following are examples of common tasks that may be useful to adopt for your needs.

Count all documents:

This is a trivial example where a counter (total-docs) is incremented for each document retrieved.

{
    "count-docs": (util, doc) => util.count('total-docs'),
}

Audit data for later user:

By using util.audit(), you can store any object for use after processing has completed. This is useful for initiating a post-processing step that is based on the output of Couchit run.

{
    "audit-bad-docs": (util, doc) => {
        if (doc.status && doc.status === 'bad') {
            const object = { bad_doc_id: doc._id, status: doc.status }
            util.audit(object);
        }
    }
}

Calculate the hash of part of a document:

By using util.audit(), you can store any object for use after processing has completed. This is useful for initiating a post-processing step that is based on the output of Couchit run.

{
    "hash-doc-contents": (util, doc) => {
        const hash = util.hash(doc);
        util.log('SHA256 hash of doc: ' + hash);
    }
}

Validate a schema:

Determine if a document is valid based on a json-schema specification. This task uses ajv for validation.

{
    "validate-schema": (util, doc) => {
        const schema = require('./test/schema.json');
        const data = require('./test/data.json');
        const valid = util.validate(schema, data);
        util.log(doc._id + ' is valid? ' + valid);
    }
}

Update a parent document by checking for existence of child docs:

Note that nano functions run asyncrounously while documents are interated over. To ensure complete stats and (optional) audit object tracking, use the util.incrementWaits() to set a wait prior calling the async function and util.decrementWaits() to remove the wait upon completion.

{
    "dependent-updates": (util, doc) => {
      if (doc.childKeys) {
        const keys = doc.childKeys;

        // Set a wait
        util.incrementWaits();

        // Do a bulk get for all childKeys
        util.nano.fetch(keys, (err, result) => {
          if (err) {
            console.log(err);
          } else {
            const newChildKeys = [];
            const rows = result.rows;
            const hasMissingChildKeys = false;

            // Only add docs to the newChildKeys that were found in the db
            rows.forEach(row => {
              if (row.doc) {
                newChildKeys.push(row.id);
              } else {
                hasOrphans = true;
                util.log('Missing child doc: ' + row.id);
              }
            });

            if (hasMissingChildKeys) {
              doc.childKeys = newChildKeys;
              util.nano.insert(doc, (err, result) => {
                console.log(result);

                // Remove a wait
                util.decrementWaits();
              });
            }
          }
        });
      }
    }
}

Credits

  • Couchit is based on Couchtato; thanks to Cliffano Subagio for his work.

About

A database iterator that includes a set of tools to help manage documents in a CouchDB database.

Resources

License

Stars

Watchers

Forks

Packages

No packages published