Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature/deprecating messy tables #79

Merged
merged 34 commits into from
Aug 29, 2024
Merged

Conversation

s7clarke10
Copy link
Owner

@s7clarke10 s7clarke10 commented Aug 19, 2024

Problem

  1. When doing discovery, tap-s3-csv uses a sample of the data from csv files to guess the most suitable type of each column in the files. This sometimes leads to type mismatch issues when an unexpected value, of different type e.g float number in a column that was interpreted as integer, appears somewhere in the file and breaks the pipeline.
  2. Currently, development and automated testing on this tap requires an AWS S3 bucket, this is a bottleck to running CI check for PRs from forks, as well as general dev/testing experience

Proposed changes

  1. All columns in the csv header would be interpreted as string types and drop the usage of messytables to guess the column types
  2. Use a local Minio server as an S3 server for dev and testing purpose.

Pros

  • No more runtime errors related to conflict between data and expected type.
  • Getting rid of old and seemingly unmaintained library messytables, which would help in future migrations to recent Python versions the library doesn't support messytables
  • Integration tests will be automatically tested.

Cons

  • This is a breaking change.

Types of changes

What types of changes does your code introduce to PipelineWise?
Put an x in the boxes that apply

  • Bugfix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation Update (if none of the other choices apply)

Checklist

  • Description above provides context of the change
  • I have added tests that prove my fix is effective or that my feature works
  • Unit tests for changes (not needed for documentation changes)
  • CI checks pass with my changes
  • Bumping version in setup.py is an individual PR and not mixed with feature or bugfix PRs
  • Commit message/PR title starts with [AP-NNNN] (if applicable. AP-NNNN = JIRA ID)
  • Branch name starts with AP-NNN (if applicable. AP-NNN = JIRA ID)
  • Commits follow "How to write a good git commit message"
  • Relevant documentation is updated including usage instructions

@s7clarke10 s7clarke10 merged commit 8ddb803 into master Aug 29, 2024
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant