Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

added chicken param to remove methods, consistent apis #628

Closed
wants to merge 169 commits into from
Closed
Changes from 1 commit
Commits
Show all changes
169 commits
Select commit Hold shift + click to select a range
2e21663
Times out long-running workers
daveFNbuck Dec 11, 2014
9f76b31
[scalding] allow single jobs in requires
Dec 19, 2014
9330bde
Limit number of shown tasks in visualiser
daveFNbuck Dec 26, 2014
fff5a48
Remove deprecated apply() builtin.
berkerpeksag Dec 29, 2014
c3b9daf
Document Task History
jcrobak Dec 31, 2014
6641136
Make task_id parsing support additional ids.
jcrobak Dec 31, 2014
f70ea9b
Initial implementation of the SQLAlchemy target
gouthambs Jan 3, 2015
af08487
Fixed the broken tests
gouthambs Jan 3, 2015
22fb2a2
Minor modifications
gouthambs Jan 4, 2015
aae9cb1
Incorporated chunk size in the sqla implementation
gouthambs Jan 6, 2015
1d7b426
Added test for chunk size. Rearranged tests.
gouthambs Jan 6, 2015
582d14e
Fixed the test issues.
gouthambs Jan 8, 2015
5762945
Minor docs and copyright info
gouthambs Jan 8, 2015
2b8db9e
Add docs, incorporarte PR sugggestions, and improve test coverage
gouthambs Jan 9, 2015
da2d40d
Keeps track of tasks in scheduler grouped by status
daveFNbuck Dec 23, 2014
7b729a5
Minor fixes and cleanups
gouthambs Jan 9, 2015
43595c3
Added support for specifying the effective user when using the Snakeb…
RobCherry Jan 12, 2015
6674800
Add sqla to docs
gouthambs Jan 12, 2015
0e75c71
fixed formatting for example code in luigi.contrib.sqla
Jan 12, 2015
ac4b7d1
Fixed another documentation issue
Jan 12, 2015
6fee969
removed some empty lines to see if sphinx formatting works
Jan 12, 2015
9bafe0a
Fix broken docs
gouthambs Jan 12, 2015
15879b2
add explicit bulk_complete API
ulzha Jan 13, 2015
7ebfe16
factored out minicluster functionality
Jan 14, 2015
6c099d2
hadoop_test.py now uses MiniCluster functionality
Jan 14, 2015
b1f8b24
Remove hadoop_test.py from gitignore
Tarrasch Jan 15, 2015
f5050d5
Use MiniClusterTestCase in snakebite tests
Tarrasch Jan 15, 2015
0946358
fixed some small things with luigi.scalding
Jan 16, 2015
e276c24
Moved luigi.scalding to luigi.contrib.scalding
Jan 16, 2015
5a34ef0
Adding catch for old imports
Jan 16, 2015
9e11108
add no-lock to all parameter unit tests
Jan 18, 2015
56b6668
Make it possible to override global parameters
Jan 18, 2015
d8fc682
removed deprecated default_from_config argument
Jan 18, 2015
6e09e5e
doc - remove colons from sample decorator usage
hblanks Jan 18, 2015
248d0d4
fixed documentation for scalding
Jan 18, 2015
aee679d
additional documentation fix for scalding
Jan 18, 2015
1ec517f
Improved Spark error formatting
Jan 16, 2015
72ecfc8
Updated tests for improved spark error formatting
Jan 20, 2015
660c28a
Fix test @attr-ibutes in hadoop_test.py
Tarrasch Jan 19, 2015
8708ff5
Move some argparse/optparse stuff to luigi.Parameter
Jan 20, 2015
99a1b47
Cleaned up code for dynamic dependencies and added another test
Jan 20, 2015
fbfdc82
some docs on dynamic dependencies
Jan 21, 2015
9e884b4
Remove whoops from dependencies
Tarrasch Jan 21, 2015
89c908f
more doc about requiring a task
Jan 21, 2015
6b453c6
fix for broken formatting in doc
Jan 21, 2015
de84458
remove lock parameter
Jan 20, 2015
9c2841f
Don't have contrib-dependencies
Tarrasch Jan 21, 2015
21f40ab
Propagate tox's args to nosetest
Tarrasch Jan 15, 2015
2fa4741
less strict timeout
Jan 21, 2015
13e2b76
Changed links to documentation slightly
Jan 21, 2015
62d8569
moved authors to the bottom of README
Jan 21, 2015
5533e45
Added some external presentations and blog posts
Jan 21, 2015
6638401
fix
Jan 21, 2015
96c30d9
Fix for new deps
Jan 21, 2015
87fc6db
Set default temp directory in a Windows friendly way.
themalkolm Jan 21, 2015
28bdf24
Explicitly note that examples work with local_scheduler only.
themalkolm Jan 21, 2015
f0aac8e
Default values should not be mutable.
themalkolm Jan 21, 2015
01cf177
Add date_format to DateHourParameter.
themalkolm Jan 22, 2015
fa1de30
Add DateMinuteParameter.
themalkolm Jan 21, 2015
05c68ec
removed an is_global check, not sure why it wasn't caught by unit tests
Jan 21, 2015
1507e45
Removed anything dealing with global parameters from task.py
Jan 21, 2015
f608d02
removed from_str_params and simplified interface.py a bit
Jan 21, 2015
3ab0d58
simplified add_global_parameters
Jan 21, 2015
54ae1c0
Moar tests.
themalkolm Jan 22, 2015
555a4e4
Add DateMinuteParameter support to previous()
themalkolm Jan 22, 2015
591952b
Preserve the overrided method's signature.
themalkolm Jan 22, 2015
97be711
Use iteritems(), it is faster as does not create a list.
themalkolm Jan 22, 2015
dc7c04d
removed some deprecated stuff in luigi.util
Jan 22, 2015
634cc53
Include host in error notifications
Jan 22, 2015
24a5f3a
Fixed bug introduced in #634
Jan 23, 2015
c97f20f
Fixed another issue introduced in #634
Jan 23, 2015
3328ffd
Added SeatGeek to the list of Luigi users
Jan 23, 2015
a8ebe77
Do not initialize _task_history twice
themalkolm Jan 23, 2015
02404d3
Trying coveralls
Jan 23, 2015
3b9d40a
Do not open atomic_file for makedirs result.
themalkolm Jan 22, 2015
c0e399a
Add missing call the the parent.
themalkolm Jan 23, 2015
4da5852
s/pig/hive in readme :)
Jan 25, 2015
8a0e394
Add coveralls badge to README
themalkolm Jan 25, 2015
8849d21
Accidentally was running all tests in Travis twice. Removed one of th…
Jan 25, 2015
5c7dce1
Added landscape.io badge and moved badges to the bottom
Jan 25, 2015
316b18a
Moar badges, back on top
Jan 25, 2015
9c37ae6
Fix worker-keep-alive documentation
themalkolm Jan 25, 2015
0364dc1
Add pep8 and autopep8 targets to tox
themalkolm Jan 25, 2015
ac5e834
Fix all default pep8 issues.
themalkolm Jan 25, 2015
7aedd2a
Enable pep8 check for travis builds.
themalkolm Jan 25, 2015
d9c61a4
Remove unused import
themalkolm Jan 25, 2015
f01a771
Do not import postgres twice
themalkolm Jan 25, 2015
42eb10e
Remove unnecessary pass statement
themalkolm Jan 25, 2015
605966c
Remove unnecessary pass statement
themalkolm Jan 25, 2015
9842511
Added a license badge and added copyright headers to some files
Jan 25, 2015
7deb94d
Add tests for most_common.
themalkolm Jan 22, 2015
2228680
Improve most_common.
themalkolm Jan 22, 2015
83d4052
Removed workaround for deprecated get_work RPC call
Jan 25, 2015
cb38ecc
Restructure tox.ini and test environments
Tarrasch Jan 26, 2015
29aac75
Now is_global is only used by Parameter
Jan 21, 2015
114ecbd
Make it possible to override *any* parameter from the cmd line
Jan 22, 2015
346ad73
Added more tests for new-style global parameters
Jan 22, 2015
de71487
added documentation for new-style global parameters
Jan 22, 2015
528d507
Return 0 even when pep8 fails
Tarrasch Jan 26, 2015
6ae07cc
removed broken file accidentally added
Jan 26, 2015
e615bb2
Rename BooleanParameter to BoolParameter.
themalkolm Jan 22, 2015
a6774f8
Proper backward compatibility.
themalkolm Jan 22, 2015
bf8cbaf
Extract deprecate_kwarg to a separate file.
themalkolm Jan 22, 2015
58f242c
Deprecate is_boolean kwarg in luigi.Parameter.
themalkolm Jan 22, 2015
e40c9e1
Warn if BooleanParameter is instantiated.
themalkolm Jan 22, 2015
74a4969
Rename member variable is_boolean => is_bool.
themalkolm Jan 22, 2015
41b89cf
Moved luigi.id_to_task_name_and_params to luigi.util
Jan 26, 2015
ef0b975
pep8 fixes
Jan 26, 2015
99e09a5
Enable HdfsClientTest
Tarrasch Jan 26, 2015
64fde9d
pep8 fixes
Jan 27, 2015
e72a54e
Removed the RemoteSchedulerResponder class
Jan 26, 2015
5dc5fb2
Add support for rename_dont_move in snakebite
Tarrasch Jan 15, 2015
f8574d1
Make rename_dont_move part of HdfsClient interface
Tarrasch Jan 15, 2015
041063a
Use rename_dont_move in HdfsClient.move_dir()
Tarrasch Jan 15, 2015
dbb5d54
Revert "Manually revert parts of spotify/luigi#522"
Tarrasch Jan 15, 2015
5014725
Usually first parameter of @classmethods is named cls.
themalkolm Jan 27, 2015
ba72cc1
Do not use octal numbers.
themalkolm Jan 27, 2015
32cccbd
Run autopep8 in more aggressive mode.
themalkolm Jan 27, 2015
1b4b8e6
Run autopep8 on repo.
themalkolm Jan 27, 2015
7049608
Do not use old-style classes.
themalkolm Jan 27, 2015
4181aa6
Remove unused import.
themalkolm Jan 27, 2015
753a1ab
Rename local variable in pythonic way.
themalkolm Jan 27, 2015
64be810
removed some deprecated methods and simplified the code
Jan 27, 2015
161a199
Fixed various things that Landscape was complaining about
Jan 27, 2015
183cb9a
Only upload coverage stats for non-hdfs tests
Jan 29, 2015
b2b22b3
Reverting previous commit
Jan 29, 2015
0b6ed69
added python classifiers.
Jan 30, 2015
3317195
refactor out RangeBase (in preparation to reuse for RangeDaily)
ulzha Jan 30, 2015
6ac1e43
add RangeDaily
ulzha Jan 30, 2015
f6e11c3
tweak formatting so log output looks consistent with parameter serial…
ulzha Jan 30, 2015
2fd1d73
fix pep8
ulzha Jan 30, 2015
dc87cb0
Use default values pulled from configuration for all parameters
Jan 28, 2015
a19be9c
More tests
Jan 31, 2015
39a6747
Check if glob exists before attempting to listdir.
Jan 26, 2015
762b62e
Performance fix in argparse interface
Feb 2, 2015
8c206a4
Improve performance of slow test
Feb 2, 2015
d68c2e1
Reset global variables between runs
Feb 2, 2015
fb37d9d
Trying to remove default=None for parameters
Feb 2, 2015
6ab0fae
Fixed a bug in argparse for Python 2.7.9 + added some new tests
Feb 3, 2015
9830c4a
extend ftp to support 1) port, 2) modified time, 3) tls
Jan 28, 2015
e20bcfb
docstring standardization.
Feb 2, 2015
704a645
fixed PEP8 error.
Feb 3, 2015
728b593
tox ini settings now consistent.
Feb 3, 2015
a1cfcbc
ignore autopep8 E309 since if class has docstring, it will be separat…
Feb 3, 2015
87cbda1
autopep8 changes.
Feb 3, 2015
3945c63
Use sphinx-build as it is more Windows friendly.
themalkolm Feb 3, 2015
7fe72a6
Add docsapi target to tox.
themalkolm Feb 3, 2015
4be16e8
Update doc/api
themalkolm Feb 3, 2015
0a6cccf
fix for access to self.path is done before definition.
Feb 3, 2015
a80bac5
New Config class
Jan 27, 2015
a4481da
Added deprecation warning for is_global
Jan 27, 2015
bdc078b
Renamed is_config to is_without_section
Feb 1, 2015
39414ce
More tests
Feb 1, 2015
2dc0273
Fix bug where Redshift began throwing a different exception when tabl…
Feb 3, 2015
109ecf8
Add support for table truncation
Feb 3, 2015
7bb45d6
Add support for passing Redshift table attributes to create table sta…
Feb 4, 2015
a3ce173
hdfs.py ported to use new-style parameters
Feb 3, 2015
372357d
refactored email tests to get test coverage a bit higher
Feb 4, 2015
4930397
Pig Task
Feb 3, 2015
45af94a
fix sphinx build errors and enhanced .gitignore.
Feb 3, 2015
8fe967f
Add task to kill open redshift sessions
Feb 4, 2015
249568c
pep8: remove trailing whitespace
Feb 4, 2015
ea2dd52
moving pig test to test.contrib
Feb 4, 2015
147f271
usage of string format as logging function parameters.
Feb 4, 2015
6777b89
Use @ trick to avoid escaping every / in sed script.
themalkolm Feb 4, 2015
6ad0ec3
Remove the temporary README.rst
themalkolm Feb 4, 2015
7c68e36
Add isort target.
themalkolm Feb 4, 2015
e32aa51
Sort all imports. Because we can.
themalkolm Feb 4, 2015
644e642
added chicken param to remove methods, consistent apis
coderfi Feb 5, 2015
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Add support for table truncation
  • Loading branch information
Doug Daniels authored and coderfi committed Feb 5, 2015
commit 109ecf809f49baacb3019a80e51a5295af8c0664
17 changes: 17 additions & 0 deletions luigi/contrib/redshift.py
Original file line number Diff line number Diff line change
@@ -84,6 +84,20 @@ def copy_options(self):
"""
return ''

def do_truncate_table(self):
"""
Return True if table should be truncated before copying new data in.
"""
return False

def truncate_table(self, connection):
query = "truncate %s" % self.table
cursor = connection.cursor()
try:
cursor.execute(query)
finally:
cursor.close()

def run(self):
"""
If the target table doesn't exist, self.create_table will be called to attempt to create the table.
@@ -98,6 +112,9 @@ def run(self):
logger.info("Creating table %s", self.table)
connection.reset()
self.create_table(connection)
elif self.do_truncate_table():
logger.info("Truncating table %s", self.table)
self.truncate_table(connection)

logger.info("Inserting file: %s", path)
cursor = connection.cursor()