tap-kustomer

This is a Singer tap that produces JSON-formatted data following the Singer spec.

This tap:

• Pulls raw data from the https://api.kustomerapp.com
• Extracts the following resources:
  • Conversations, Customers, Kobjects, Messages, Notes, Teams, Tags, Shortcuts, Users
• Pagination for Conversations, Custoemrs, Kobjects, Messages, Note
  • Next request based on last updatedAt in respose.
  • See, https://dev.kustomer.com/v1/customers/customer-search#pagination
  • pageSize set to 300 to circumvent large numbers of records with same updatedAt
• Outputs the schema for each resource
• Incrementally pulls data based on the input state

Streams

Conversations

• Endpoint: POST https://api.kustomerapp.com/v1/customers/search
• Body

    {
    "and": [
        {
        "conversation_updated_at": {
            "gte": "2017-01-01"
            }
        }
    ],
    "sort": [
        {
        "conversation_updated_at": "asc"
        }
    ],
    "queryContext": "conversation"
    }

• Primary key fields: id
• Replication strategy: INCREMENTAL (query filtered)
  • Bookmark query fields: gte
  • Bookmark: date-time
• Transformations: Nodes attributes and relationships denested.
  • attributes.sla renamed to sla_data
  • Snake to camel case

Customers

• Endpoint: POST https://api.kustomerapp.com/v1/customers/search
• Body

    {
    "and": [
        {
        "customer_updated_at": {
            "gte": "2017-01-01"
            }
        }
    ],
    "sort": [
        {
        "customer_updated_at": "asc"
        }
    ],
    "queryContext": "customer"
    }

• Primary key fields: id
• Replication strategy: INCREMENTAL (query filtered)
  • Bookmark query fields: gte
  • Bookmark: date-time
• Transformations:
  • Nodes attributes and relationships denested.
  • Snake to camel case

Kobjects

• Endpoint: POST https://api.kustomerapp.com/v1/customers/search
• Body

    {
    "and": [
        {
        "kobject_updated_at": {
            "gte": "2017-01-01"
            }
        }
    ],
    "sort": [
        {
        "kobject_updated_at": "asc"
        }
    ],
    "queryContext": "kobject"
    }

• Primary key fields: id
• Replication strategy: INCREMENTAL (query filtered)
  • Bookmark query fields: gte
  • Bookmark: date-time
• Transformations
  • Nodes attributes and relationships denested.
  • Snake to camel case

Messages

• Endpoint: POST https://api.kustomerapp.com/v1/customers/search
• Body

    {
    "and": [
        {
        "message_updated_at": {
            "gte": "2017-01-01"
            }
        }
    ],
    "sort": [
        {
        "message_updated_at": "asc"
        }
    ],
    "queryContext": "message"
    }

• Primary key fields: id
• Replication strategy: INCREMENTAL (query filtered)
  • Bookmark query fields: gte
  • Bookmark: date-time
• Transformations
  • Nodes attributes and relationships denested.
  • Snake to camel case

Notes

• Endpoint: POST https://api.kustomerapp.com/v1/customers/search
• Body

    {
    "and": [
        {
        "note_updated_at": {
            "gte": "2017-01-01"
            }
        }
    ],
    "sort": [
        {
        "note_updated_at": "asc"
        }
    ],
    "queryContext": "note"
    }

• Primary key fields: id
• Replication strategy: INCREMENTAL (query filtered)
  • Bookmark query fields: gte
  • Bookmark: date-time
• Transformations
  • Nodes attributes and relationships denested.
  • Snake to camel case

Shortcuts

• Endpoint: GET https://api.kustomerapp.com/v1/shortcuts?page=1
• Primary key fields: id
• Replication strategy: INCREMENTAL (query filtered)
  • Bookmark query fields: updated_at
  • Bookmark: date-time
• Transformations
  • Nodes attributes and relationships denested
  • Snake to camel case

Tags

• Endpoint: GET https://api.kustomerapp.com/v1/tag?page=1
• Primary key fields: id
• Replication strategy: INCREMENTAL (query filtered)
  • Bookmark query fields: updated_at
  • Bookmark: date-time
• Transformations
  • Nodes attributes and relationships denested
  • Snake to camel case

Teams

• Endpoint: GET https://api.kustomerapp.com/v1/teams?page=1
• Primary key fields: id
• Replication strategy: INCREMENTAL (query filtered)
  • Bookmark query fields: updated_at
  • Bookmark: date-time
• Transformations
  • Nodes attributes and relationships denested
  • Snake to camel case

Users

• Endpoint: GET https://api.kustomerapp.com/v1/users?page=1
• Primary key fields: id
• Replication strategy: INCREMENTAL (query filtered)
  • Bookmark query fields: updated_at
  • Bookmark: date-time
• Transformations
  • Nodes attributes and relationships denested
  • Snake to camel case

Quick Start

1. Install

   Clone this repository, and then install using setup.py. We recommend using a virtualenv:

   > virtualenv -p python3 venv
   > source venv/bin/activate
   > python setup.py install
   OR
   > cd .../tap-kustomer
   > pip install .

2. Dependent libraries The following dependent libraries were installed.

   > pip install singer-python
   > pip install singer-tools
   > pip install target-stitch
   > pip install target-json
   

   • singer-tools
   • target-stitch

3. Create your tap's config.json file. The token is the credential supplied from the Kustomer integration. The date_window_size is the integer number of days (between the from and to dates) for date-windowing through the date-filtered endpoints (default = 60). The page_size_limit is the integer number of records to return per API request.

   {
       "token": "YOUR_API_TOKEN",
       "start_date": "2019-01-01T00:00:00Z",
       "user_agent": "tap-kustomer <api_user_email@your_company.com>",
       "date_window_size": "60",
       "page_size_limit": "100"
   }

   Optionally, also create a state.json file. currently_syncing is an optional attribute used for identifying the last object to be synced in case the job is interrupted mid-stream. The next run would begin where the last job left off.

     {
       "bookmarks": {
         "conversations": "2019-12-26T18:35:41.583Z",
         "customers": "2019-12-26T18:39:38.186Z",
         "kobjects": "2019-12-26T18:24:18.283Z",
         "messages": "2019-12-26T18:39:45.018Z",
         "shortcuts": "2019-12-25T18:00:00Z",
         "tags": "2019-12-25T18:00:00Z",
         "users": "2019-12-25T18:00:00Z"
       },
       "currently_syncing": "teams"
     }

4. Run the Tap in Discovery Mode This creates a catalog.json for selecting objects/fields to integrate:

   tap-kustomer --config config.json --discover > catalog.json

   See the Singer docs on discovery mode here.

5. Run the Tap in Sync Mode (with catalog) and write out to state file

   For Sync mode:

   > tap-kustomer --config tap_config.json --catalog catalog.json > state.json
   > tail -1 state.json > state.json.tmp && mv state.json.tmp state.json

   To load to json files to verify outputs:

   > tap-kustomer --config tap_config.json --catalog catalog.json | target-json > state.json
   > tail -1 state.json > state.json.tmp && mv state.json.tmp state.json

   To pseudo-load to Stitch Import API with dry run:

   > tap-kustomer --config tap_config.json --catalog catalog.json | target-stitch --config target_config.json --dry-run > state.json
   > tail -1 state.json > state.json.tmp && mv state.json.tmp state.json

6. Test the Tap

   While developing the KUSTOMER tap, the following utilities were run in accordance with Singer.io best practices: Pylint to improve code quality:

   > pylint tap_kustomer -d missing-docstring -d logging-format-interpolation -d too-many-locals -d too-many-arguments

   Pylint test resulted in the following score:

   Your code has been rated at 9.83/10

   To check the tap and verify working:

   > tap-kustomer --config tap_config.json --catalog catalog.json | singer-check-tap > state.json
   > tail -1 state.json > state.json.tmp && mv state.json.tmp state.json

   Check tap resulted in the following:

   The output is valid.
   It contained 35416 messages for 16 streams.
   
   16 schema messages
   35416 record messages
   116 state messages
   
   Details by stream:
   +---------------+---------+---------+
   | stream        | records | schemas |
   +---------------+---------+---------+
   | conversations | 1274    | 1       |
   | kobjects      | 258     | 1       |
   | tags          | 0       | 1       |
   | notes         | 0       | 1       |
   | users         | 0       | 1       |
   | customers     | 32300   | 1       |
   | shortcuts     | 0       | 1       |
   | teams         | 0       | 1       |
   | messages      | 1584    | 1       |
   +---------------+---------+---------+

---

Copyright © 2019 Stitch