Shopify connector should convert prices into numbers #4841

rohansingh · 2021-07-19T22:12:30Z

Tell us about the problem you're trying to solve

Most price-related fields created by the Shopify connector are strings. For example, total_price on orders is a string. This makes it difficult to do simple math or create sales reports.

Describe the solution you’d like

I'd like the Shopify connector to transform price fields into numbers.

Describe the alternative you’ve considered or used

I could write a custom transformation with dbt, but it seems like the Shopify connector should natively do this.

Additional context

Discussion here:
https://airbytehq.slack.com/archives/C01MFR03D5W/p1626731077032400

Shopify's API returns prices as strings, often with a separate field specifying the currency. See fields like total_price and total_tax here:
https://shopify.dev/api/admin/rest/reference/orders/order#properties-2021-07

So it makes sense that the connector treats these as strings. However, it'd be really nice if it could handle the transformation.

Are you willing to submit a PR?

This is a bit beyond me at the moment.

The text was updated successfully, but these errors were encountered:

Zirochkaa · 2021-07-26T08:44:24Z

Scoping:

Look through each stream and change the type of field from "string" to "number" where field is actually a number (4-5 h because, at least for me, Shopify docs are very laggy and slow)
Create transform() function and apply it on each record in ShopifyStream.parse_response() and IncrementalShopifyStream.filter_records_newer_than_state() functions (~2h)
Check that transform() works for all streams (~1h)

Link to zira.

vitaliizazmic · 2021-08-03T15:26:52Z

I detected next price fields, which should be changed to number:

Abandoned checkouts
- tax_lines
  - price
- total_line_items_price
- total_tax
- subtotal_price
- line_items
  - compare_at_price
  - line_price
  - discount_allocations
    - amount
  - tax_lines
    - price
  - price
- total_discounts
- shipping_lines
  - price
  - tax_lines
  - price
  - original_shop_price
  - original_shop_markup
- customer
  - total_spent
- total_price
Collects
Custom Collections
Customers
- total_spent
Discount Codes
Draft Orders
- customer
  - total_spent
- line_items
  - price
  - price_set
    - shop_money
      - amount
    - presentment_money
      - amount
  - total_discount
  - total_discount_set
    - shop_money
      - amount
    - presentment_money
      - amount
  - tax_lines
    - price
    - price_set
  - shop_money
    - amount
    - presentment_money
      - amount
- shipping_line
  - tax_lines
    - price
- price_set
  - shop_money
  - amount
  - presentment_money
  - amount
  - price
- tax_lines
  - price
  - price_set
    - shop_money
      - amount
    - presentment_money
      - amount
- total_price
Metafields
Order Refunds
- refund_line_items
  - subtotal_set
    - shop_money
      - amount
    - presentment_money
      - amount
- total_tax_set
  - shop_money
    - amount
  - presentment_money
    - amount
- total_tax
- line_item
  - price
  - tax_lines
    - price_set
      - shop_money
        
        amount
      - presentment_money
        
        amount
    - price
  - price_set
    - shop_money
      - amount
    - presentment_money
      - amount
  - pre_tax_price
  - total_discount_set
    - shop_money
      - amount
    - presentment_money
      - amount
  - discount_allocations
    - amount
    - amount_set
      - shop_money
        
        amount
      - presentment_money
        
        amount
  - pre_tax_price_set
    - shop_money
      - amount
    - presentment_money
      - amount
  - total_discount
Order Risks
- score - this field has type error
Orders
- current_subtotal_price
- current_subtotal_price_set
  - shop_money
    - amount
  - presentment_money
    - amount
- current_total_discounts
- current_total_discounts_set
  - shop_money
    - amount
  - presentment_money
    - amount
- current_total_duties_set - should be checked type
- current_total_price
- current_total_price_set
  - shop_money
    - amount
  - presentment_money
    - amount
- current_total_tax
- current_total_tax_set
  - shop_money
    - amount
  - presentment_money
    - amount
- original_total_duties_set - should be checked type
- subtotal_price
- subtotal_price_set
  - shop_money
    - amount
  - presentment_money
    - amount
- tax_lines
  - price
  - price_set
    - shop_money
      - amount
    - presentment_money
      - amount
- total_discounts
- total_discounts_set
  - shop_money
    - amount
  - presentment_money
    - amount
- total_line_items_price
- total_line_items_price_set
  - shop_money
    - amount
  - presentment_money
    - amount
- total_outstanding
- total_price
- total_price_set
  - shop_money
    - amount
  - presentment_money
    - amount
- total_price_usd
- total_shipping_price_set
  - shop_money
    - amount
  - presentment_money
    - amount
- total_tax
- total_tax_set
  - shop_money
    - amount
  - presentment_money
    - amount
- total_tip_received
- customer
  - total_spent
- fulfillments
  - line_items
    - price
    - price_set
      - shop_money
        
        amount
      - presentment_money
        
        amount
    - total_discount
    - total_discount_set
      - shop_money
        
        amount
      - presentment_money
        
        amount
    - tax_lines
      - price
      - price_set
        
        shop_money
        
        amount
        
        presentment_money
        
        amount
- line_items
  - price
  - price_set
    - shop_money
      - amount
    - presentment_money
      - amount
  - total_discount
  - total_discount_set
    - shop_money
      - amount
    - presentment_money
      - amount
  - tax_lines
    - price
    - price_set
      - shop_money
        
        amount
      - presentment_money
        
        amount
- shipping_lines
  - discounted_price
  - discounted_price_set
    - shop_money
      - amount
    - presentment_money
      - amount
  - price
  - price_set
    - shop_money
      - amount
    - presentment_money
      - amount
Pages
Price Rules
Products
- variants
  - price
  - compare_at_price
  - presentment_prices
    - price
      - amount
    - compare_at_price
Transactions
- amount

* Source Shopify #4841 - update schemas, transform root properties * Source Shopify #4841 - transform nested properties * Source Shopify #4841 - ignore transforming for null values * Source Shopify #4841 - improving according to PR review * Source Shopify #4841 - create transformer * Source Shopify #4841 - reformat * Source Shopify #4841 - reformat(fix) * Source Shopify #4841 - improving extracting schemas type * Source Shopify #4841 - unit tests for transform solution * Source Shopify #4841 - reformat * Source Shopify - fix according to review * Source Shopify - fix DRY * Source Shopify - fix orders schema (add type) * Source Shopify - fixing according to PR review * Source Shopify - bump version * Source Shopify - update changelogs

rohansingh added the type/enhancement New feature or request label Jul 19, 2021

sherifnada added area/connectors Connector related issues lang/python labels Jul 19, 2021

sherifnada modified the milestones: Connectors July 23, 2021, Connectors August 6th Jul 23, 2021

Zirochkaa self-assigned this Jul 25, 2021

Zirochkaa assigned Zirochkaa and unassigned Zirochkaa Jul 26, 2021

sherifnada mentioned this issue Jul 26, 2021

Handle APIs which emit numbers as strings #4984

Closed

vitaliizazmic assigned vitaliizazmic and unassigned Zirochkaa Jul 26, 2021

Zirochkaa mentioned this issue Aug 5, 2021

CDK: Create a generic method for transforming records fields in accordance with streams schemas #5224

Closed

vitaliizazmic added a commit that referenced this issue Aug 5, 2021

Source Shopify #4841 - update schemas, transform root properties

bfb8a80

vitaliizazmic added a commit that referenced this issue Aug 6, 2021

Source Shopify #4841 - transform nested properties

a37c12b

vitaliizazmic mentioned this issue Aug 6, 2021

🎉 Source Shopify: convert prices into numbers #5244

Merged

9 tasks

sherifnada modified the milestones: Connectors August 6th, Connectors August 20th Aug 6, 2021

vitaliizazmic added a commit that referenced this issue Aug 9, 2021

Source Shopify #4841 - ignore transforming for null values

4409209

vitaliizazmic added a commit that referenced this issue Aug 17, 2021

Source Shopify #4841 - improving according to PR review

5aae64b

vitaliizazmic added a commit that referenced this issue Aug 18, 2021

Source Shopify #4841 - create transformer

fe8e8b3

vitaliizazmic added a commit that referenced this issue Aug 18, 2021

Source Shopify #4841 - reformat

45c10a7

vitaliizazmic added a commit that referenced this issue Aug 18, 2021

Source Shopify #4841 - reformat(fix)

8adaeb0

vitaliizazmic added a commit that referenced this issue Aug 18, 2021

Source Shopify #4841 - improving extracting schemas type

7c18eb9

sherifnada modified the milestones: Connectors August 20th, Connectors, September 3rd Aug 25, 2021

sherifnada modified the milestones: Connectors, September 3rd, Connectors, September 17th 2021 Sep 3, 2021

vitaliizazmic added a commit that referenced this issue Sep 7, 2021

Source Shopify #4841 - unit tests for transform solution

2a95129

vitaliizazmic added a commit that referenced this issue Sep 7, 2021

Source Shopify #4841 - reformat

26524c2

vitaliizazmic mentioned this issue Sep 16, 2021

Make Shopify enforced data solution a general utility in the CDK #6152

Closed

sherifnada modified the milestones: Connectors, September 17th 2021, Connectors, October 1st 2021 Sep 17, 2021

vitaliizazmic closed this as completed in #5244 Sep 17, 2021

inesmarcal mentioned this issue Mar 30, 2024

[Snyk] Upgrade webpack-dev-server from 4.9.2 to 4.15.1 inesmarcal/airbyte#2

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Shopify connector should convert prices into numbers #4841

Shopify connector should convert prices into numbers #4841

rohansingh commented Jul 19, 2021

Zirochkaa commented Jul 26, 2021 •

edited

Loading

vitaliizazmic commented Aug 3, 2021 •

edited

Loading

Shopify connector should convert prices into numbers #4841

Shopify connector should convert prices into numbers #4841

Comments

rohansingh commented Jul 19, 2021

Tell us about the problem you're trying to solve

Describe the solution you’d like

Describe the alternative you’ve considered or used

Additional context

Are you willing to submit a PR?

Zirochkaa commented Jul 26, 2021 • edited Loading

vitaliizazmic commented Aug 3, 2021 • edited Loading

Zirochkaa commented Jul 26, 2021 •

edited

Loading

vitaliizazmic commented Aug 3, 2021 •

edited

Loading