Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

adapters: postgres & redshift #259

Merged
merged 49 commits into from
Jan 20, 2017
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
49 commits
Select commit Hold shift + click to select a range
3145139
initial commit, acquire_connection working for postgres with unit test
Dec 28, 2016
13fe3e5
cleanup strings, add docker unit test harness
Dec 28, 2016
973c4ad
move execute_without_auto_commit to postgres adapter
Dec 28, 2016
f4f0b41
wired in w/ strict mode
Jan 6, 2017
d0a6f17
turn strict mode on for all tests, not just default ones
Jan 6, 2017
32e20a3
add schema to unit test harness
Jan 6, 2017
9a46a2e
merged development
Jan 6, 2017
c612333
back to pep8 compliance
Jan 6, 2017
7023921
again, improve test harness
Jan 6, 2017
e848e59
remove now-unused test scripts
Jan 6, 2017
7076a7d
implement connection sharing w/ test, reset cache on dbt invocation i…
Jan 6, 2017
b2e31c9
move drop into adapter
Jan 7, 2017
0886bc4
add truncate, rename functions & add master exception handler
Jan 9, 2017
31502f8
back to pep-8 compliance
Jan 9, 2017
f6e0813
test harness++
Jan 9, 2017
4a6750e
snowflake integration test (not really working yet)
Jan 9, 2017
5676c34
added snowflake adapter
Jan 10, 2017
dcb8278
tearing out most of schema helper
Jan 10, 2017
7a52b80
schema helper is dead (long live schema helper)
Jan 10, 2017
6251c07
possibly snowflake-ready?
Jan 10, 2017
64ee3eb
snowflake workin
Jan 10, 2017
d2ea805
merged development
Jan 11, 2017
62f2f68
snowflake / circle test harness
Jan 11, 2017
6023586
merged development
Jan 12, 2017
ec50446
two more tests
Jan 12, 2017
a5e71f0
get back into pep8 compliance
Jan 12, 2017
9c95a92
debug setup_db.sh
Jan 12, 2017
3a157fd
display a reasonable status message
Jan 12, 2017
114fb91
add date functions all around
Jan 12, 2017
b959fb8
use absolute import to resolve ns conflict
Jan 13, 2017
f5d7be8
group tests by warehouse type
Jan 13, 2017
2913e81
Merge branch 'development' of github.com:analyst-collective/dbt into …
Jan 14, 2017
736dcf9
remove targets from seeder, and mark it as not implemented for snowflake
Jan 14, 2017
cd1fe4f
pep8
Jan 14, 2017
7c0f26b
rip snowflake support out for windows
Jan 15, 2017
6cf9684
fix appveyor
Jan 15, 2017
887fe85
only do snowflake imports in snowflake adapter module
Jan 15, 2017
2c0e5ec
fix SnowflakeAdapter NameError
Jan 15, 2017
6be3d44
improved error handling
Jan 16, 2017
6e14eb8
pep8
Jan 16, 2017
95e4a75
Merge branch 'development' of github.com:analyst-collective/dbt into …
Jan 16, 2017
4970d6d
update required version of snowflake connector
Jan 16, 2017
15be495
run tests on python 3.6 instead of 3.5
Jan 20, 2017
64e4b67
add python 3.6 to global pyenv
Jan 20, 2017
e30931a
also isntall python 3.6
Jan 20, 2017
527eaa8
install!
Jan 20, 2017
67cc3b4
sudo !!
Jan 20, 2017
73cde44
try python 2.7.12
Jan 20, 2017
2b3e8dd
sneak in a logger name change, and CHANGELOG
Jan 20, 2017
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,7 @@ nosetests.xml
coverage.xml
*,cover
.hypothesis/
test.env

# Translations
*.mo
Expand Down
6 changes: 6 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,9 @@
## dbt 0.7.0 (unreleased)

#### New Features

- dbt now supports [Snowflake](https://www.snowflake.net/) as a warehouse ([#259](https://github.com/analyst-collective/dbt/pull/259))

## dbt 0.6.2 (January 16, 2017)

#### Changes
Expand Down
8 changes: 2 additions & 6 deletions Dockerfile
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
FROM python
FROM python:3.5

RUN apt-get update

Expand All @@ -8,11 +8,7 @@ RUN apt-get install -y python-dev python3-dev
RUN pip install pip --upgrade
RUN pip install virtualenv
RUN pip install virtualenvwrapper

COPY . /usr/src/app
RUN pip install tox

WORKDIR /usr/src/app
RUN cd /usr/src/app
RUN ./test/setup.sh


8 changes: 5 additions & 3 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -2,15 +2,17 @@

changed_tests := `git status --porcelain | grep '^\(M\| M\|A\| A\)' | awk '{ print $$2 }' | grep '\/test_[a-zA-Z_\-\.]\+.py'`

test: test-unit test-integration
test:
@echo "Full test run starting..."
@time docker-compose run test tox

test-unit:
@echo "Unit test run starting..."
tox -e unit-py27,unit-py35
@time docker-compose run test tox -e unit-py27,unit-py35,pep8
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've been meaning to talk to you about this. The reason these tests didn't previously use Docker is because we wanted to run CI tests on Windows (via Appveyor). My understanding is that when these tests now run on Appveyor, we're not actually testing if the code works on Windows. Rather, we're testing if it runs in Docker on Windows. Is that what's happening here?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's not exactly right, since circle & appveyor don't use the Makefile to run tests -- they just call tox directly. The reason to do it this way is so that the tests always run in the same environment during development.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok makes sense, thanks


test-integration:
@echo "Integration test run starting..."
@docker-compose run test /usr/src/app/test/integration.sh
@time docker-compose run test tox -e integration-postgres-py27,integration-postgres-py35,integration-snowflake-py27,integration-snowflake-py35

test-new:
@echo "Test run starting..."
Expand Down
6 changes: 4 additions & 2 deletions circle.yml
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
machine:
python:
version: 3.6.0
post:
- pyenv global 2.7.9 3.5.0
- pyenv global 2.7.12 3.6.0
hosts:
database: 127.0.0.1

Expand All @@ -13,7 +15,7 @@ dependencies:
- pip install --upgrade pip setuptools || true
- pip install --upgrade tox tox-pyenv
override:
- pyenv local 2.7.9 3.5.0
- pyenv local 2.7.12 3.6.0

test:
override:
Expand Down
Empty file added dbt/adapters/__init__.py
Empty file.
5 changes: 5 additions & 0 deletions dbt/adapters/cache.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
import dbt.adapters.postgres as postgres


def reset():
postgres.connection_cache = {}
35 changes: 35 additions & 0 deletions dbt/adapters/factory.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
import platform

import dbt.exceptions

from dbt.adapters.postgres import PostgresAdapter
from dbt.adapters.redshift import RedshiftAdapter

if platform.system() != 'Windows':
from dbt.adapters.snowflake import SnowflakeAdapter
else:
SnowflakeAdapter = None


def get_adapter(profile):
adapter_type = profile.get('type', None)

if platform.system() == 'Windows' and \
adapter_type == 'snowflake':
raise dbt.exceptions.NotImplementedException(
"ERROR: 'snowflake' is not supported on Windows.")

adapters = {
'postgres': PostgresAdapter,
'redshift': RedshiftAdapter,
'snowflake': SnowflakeAdapter,
}

adapter = adapters.get(adapter_type, None)

if adapter is None:
raise RuntimeError(
"Invalid adapter type {}!"
.format(adapter_type))

return adapter
Loading