Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

sync with carto repo #1

Merged
merged 443 commits into from
Dec 7, 2016
Merged
Changes from 4 commits
Commits
Show all changes
443 commits
Select commit Hold shift + click to select a range
3cc942b
Regenerate npm-shrinkwrap.json
rochoa Sep 26, 2016
89f9a99
Delete table so suite ends clear
rochoa Sep 26, 2016
0e03bcd
Delete table so suite ends clear
rochoa Sep 26, 2016
aa0ce62
Implement batch logger to log query times when queries are defined wi…
Sep 29, 2016
ed27b67
Simplified batch logger construction
Sep 29, 2016
2cce19c
Regenerated npm shrinkwrap
Sep 29, 2016
59e5c10
Set batch queries log path
Sep 29, 2016
60546de
Move log logic to each job
rochoa Sep 30, 2016
9555a2c
Bring back log logic for FallbackJob
rochoa Sep 30, 2016
ba34412
Return on boolean on log
rochoa Sep 30, 2016
20573a7
Always log queries from fallback jobs
rochoa Sep 30, 2016
72fb851
Fix typo
rochoa Sep 30, 2016
f8a8d33
Merge pull request #368 from CartoDB/367-batch-logger
Sep 30, 2016
b0dfe1d
Only reload log files if logger exists
rochoa Sep 30, 2016
1321eea
Bump version and update news
rochoa Sep 30, 2016
d685585
Release 1.36.0
rochoa Sep 30, 2016
d7762c9
Stubs next version
rochoa Sep 30, 2016
bf1a677
Merge branch 'master' into express-4.x
rochoa Sep 30, 2016
b269418
Tag logs
rochoa Sep 30, 2016
857ba74
Rename
rochoa Sep 30, 2016
9099b6a
Update news
rochoa Sep 30, 2016
eef302a
Release 1.36.1
rochoa Sep 30, 2016
ccbadf4
Stubs next version
rochoa Sep 30, 2016
a093737
Merge branch 'master' into express-4.x
rochoa Sep 30, 2016
6c2db43
Batch Queries: use path instead of stream to be able to reopen FD
rochoa Oct 3, 2016
51fedbb
Release 1.36.2
rochoa Oct 3, 2016
4f83da3
Stubs next version
rochoa Oct 3, 2016
0e43f7c
Merge branch 'master' into express-4.x
rochoa Oct 3, 2016
41f11e0
Merge pull request #366 from CartoDB/express-4.x
Oct 4, 2016
fcc2931
Merge pull request #178 from strk/master-tolerant-configure
Oct 4, 2016
fa96ba6
Bump version and update news
rochoa Oct 4, 2016
11a9d59
Release 1.37.0
rochoa Oct 4, 2016
0cc6066
Stubs next version
rochoa Oct 4, 2016
b07e2d3
Merge pull request #346 from CartoDB/fix-batch-api-test
Oct 4, 2016
3b6bc14
Increment errors on err
rochoa Oct 4, 2016
cb9db0c
Make request.profiler always available
rochoa Oct 4, 2016
2edc750
Do not condition req.profiler
rochoa Oct 4, 2016
b139b9a
Add context object in all requests
rochoa Oct 4, 2016
20f50d9
Use user middleware in job controller
rochoa Oct 4, 2016
7b7d651
DRY while authenticating requests
rochoa Oct 4, 2016
c32a219
Use request bootstrapper to add host header
rochoa Oct 4, 2016
05cbd55
Use db host from request's context
rochoa Oct 4, 2016
19c9bec
Callback with job
rochoa Oct 4, 2016
2a2a54a
DRY in job response handler
rochoa Oct 4, 2016
86b2b2b
Merge pull request #369 from CartoDB/batch-queries-limits
Oct 4, 2016
025b3f3
Increase body limit to 20mb
rochoa Oct 5, 2016
6309318
Use body-parser from old connect module
rochoa Oct 5, 2016
42a1f3a
Accept multipart requests
rochoa Oct 5, 2016
14a8a99
Update news
rochoa Oct 5, 2016
2014036
Release 1.37.1
rochoa Oct 5, 2016
2bba193
Stubs next version
rochoa Oct 5, 2016
eb2768c
Add dbhost attribute to batch queries logs
rochoa Oct 5, 2016
2ac39b5
Move all batch tests to their own directory
rochoa Oct 6, 2016
26fe6a1
Add readme for batch queries feature
rochoa Oct 6, 2016
5d3f4f0
Merge pull request #370 from CartoDB/batch-queries-limits
Oct 6, 2016
7c73200
Merge branch 'master' into limit-batch-queries
rochoa Oct 6, 2016
aa19120
Merge pull request #357 from CartoDB/limit-batch-queries
Oct 6, 2016
e108d0d
Debug query to run
rochoa Oct 6, 2016
578c43b
Multiple queries jobs pushed as first job between queries
rochoa Oct 6, 2016
1d20f11
Remove unused var
rochoa Oct 6, 2016
1004131
Merge pull request #371 from CartoDB/query-order
Oct 6, 2016
595d4f0
Fix sample config for batch timeout
rochoa Oct 10, 2016
e23edad
Fix and use a batch timeout good for tests
rochoa Oct 10, 2016
51d4ff0
Differentiate between statement timeout and user cancelled query
rochoa Oct 10, 2016
5401a7e
Timeout is passed into query runner
rochoa Oct 10, 2016
8a4f54b
Allow users to set max statement_timeout for their queries
rochoa Oct 10, 2016
deb1ccf
DRY job final statuses
rochoa Oct 10, 2016
d8c3181
Update news
rochoa Oct 10, 2016
fd6eb49
Merge pull request #372 from CartoDB/batch-queries-limits
Oct 10, 2016
faa02a7
Test for 578c43b1a8cf
rochoa Oct 10, 2016
8ac6564
Move to support directory
rochoa Oct 10, 2016
4657909
Fix test scenario description
rochoa Oct 10, 2016
ece95af
Add test client to run plain queries
rochoa Oct 10, 2016
f4950f7
Add test to validate multiple consumers query order scenario
rochoa Oct 10, 2016
66820a6
Make possible to specify a name for batch
rochoa Oct 10, 2016
a0ace8d
Use name in test
rochoa Oct 10, 2016
41d7823
Drop table if exists
rochoa Oct 10, 2016
f036e29
Make test stable
rochoa Oct 10, 2016
56a6323
Inject publisher
rochoa Oct 10, 2016
90c4891
Add distributed lock implementation with redis distlock
rochoa Oct 10, 2016
0de5d94
Use debug with same params, no considering job status
rochoa Oct 10, 2016
2c06404
Add dist lock to run all jobs by host in order
rochoa Oct 10, 2016
b6c2d60
Merge pull request #373 from CartoDB/batch-queue-leader
Oct 10, 2016
d50fd89
Bump version
rochoa Oct 11, 2016
e7068b8
Release 1.38.0
rochoa Oct 11, 2016
6f30ff9
Stubs next version
rochoa Oct 11, 2016
e4b1711
pub/sub package
rochoa Oct 11, 2016
d15c7ab
Always return client to pool
rochoa Oct 11, 2016
e7c4ee3
Share redis channel config
rochoa Oct 11, 2016
611508c
Hide queue seeker behind job subscriber
rochoa Oct 11, 2016
ecc6bf0
Use real on message handler
rochoa Oct 11, 2016
01cf6f2
Share redis pool for pubsub
rochoa Oct 11, 2016
2822b68
onJobHandler receives host with job
rochoa Oct 11, 2016
dc1a23e
Add error handler for channel subscriber
rochoa Oct 11, 2016
8bc52b0
Remove console call
rochoa Oct 11, 2016
8139319
Add callback to jobseeker result from initial load
rochoa Oct 11, 2016
22d8e48
Only lock on dequeue
rochoa Oct 11, 2016
e401c01
Only log on non-test environments
rochoa Oct 11, 2016
e1d0ffc
Logger set to fatal on test environment
rochoa Oct 11, 2016
167ecc6
Set NODE_ENV environment var to test
rochoa Oct 11, 2016
98185e5
Remove Job Queue Pool and use internal structure
rochoa Oct 12, 2016
c74f9bc
More aggressive on seek interval
rochoa Oct 12, 2016
67566c1
Callback in subscriber unsubscribe errors
rochoa Oct 12, 2016
3f1b679
Locker keep refreshing lock by itself
rochoa Oct 12, 2016
88f6d46
Reuse existing redlock
rochoa Oct 12, 2016
75fc212
Locker TTL is configured
rochoa Oct 12, 2016
c21f373
Tests for locker
rochoa Oct 12, 2016
782caaf
Run acceptance tests at the end
rochoa Oct 12, 2016
b86f82d
Batch.stop removes all listeners
rochoa Oct 12, 2016
6bb2abd
Only start lock renewal on lock acquisition
rochoa Oct 12, 2016
189aff2
Only log message on empty queue
rochoa Oct 12, 2016
f7d1f94
Use constants for queues
rochoa Oct 12, 2016
aa68dff
Use constants for tests
rochoa Oct 12, 2016
1f038ac
Moves from host queues to user queues
rochoa Oct 12, 2016
1e442b3
Allow to set a max number of queued jobs per user
rochoa Oct 12, 2016
32b4d66
Merge pull request #375 from CartoDB/batch-subscriber-improvements
Oct 13, 2016
734a147
Update news
rochoa Oct 13, 2016
24eb1ad
Release 1.38.1
rochoa Oct 13, 2016
df32c1a
Stubs next version
rochoa Oct 13, 2016
05eda29
Create one client for queue-seeker and share per seek cycle
rochoa Oct 13, 2016
a8802d1
redis-distlock acquires and releases redis clients by operation
rochoa Oct 13, 2016
a6cc444
Update news
rochoa Oct 13, 2016
39b4bae
Release 1.38.2
rochoa Oct 13, 2016
9d952a8
Stubs next version
rochoa Oct 13, 2016
10776a0
Merge tag '1.38.2' into cdb
rochoa Oct 13, 2016
b510a36
Add option to skip script downloading
rochoa Oct 14, 2016
5bb7d8f
Merge branch 'master' into batch-user-queues
rochoa Oct 14, 2016
b8c63f5
Rename
rochoa Oct 14, 2016
6179327
Rename
rochoa Oct 14, 2016
c62fe29
Load config on object creation
rochoa Oct 17, 2016
cb23b7f
Make test finish quicker
rochoa Oct 17, 2016
f6dc991
Allow to override configuration per API call
rochoa Oct 17, 2016
8b9a30e
Queue seeker was not _finding_ queues when only one present
rochoa Oct 17, 2016
b8a5746
Handle stop gracefully
rochoa Oct 17, 2016
4dad54d
Add test to validate query order by user
rochoa Oct 17, 2016
39bb7e6
Lock resources by host+user
rochoa Oct 17, 2016
180ba19
Fix host queue seeking
rochoa Oct 17, 2016
431f728
250 queued jobs as default limit
rochoa Oct 17, 2016
4da7363
Merge pull request #374 from CartoDB/batch-user-queues
Oct 17, 2016
cdde1be
Re-use redis pool as much as possible
rochoa Oct 17, 2016
4203696
Bump version and update news
rochoa Oct 17, 2016
66d1c18
Default to 64 queued jobs as max
rochoa Oct 17, 2016
42ca313
Release 1.39.0
rochoa Oct 17, 2016
9635ed6
Stubs next version
rochoa Oct 17, 2016
803a4b5
Add some notes about redis data structures for batch queries
rochoa Oct 17, 2016
3772b1c
Log created at time and waiting time for fallback jobs
rochoa Oct 17, 2016
179fb4e
Update news
rochoa Oct 17, 2016
cb9aaef
Release 1.39.1
rochoa Oct 17, 2016
e51a4fc
Stubs next version
rochoa Oct 17, 2016
c6e906d
Use same debug group
rochoa Oct 17, 2016
a8e03f0
Add debug information in Jobs Queue
rochoa Oct 17, 2016
761fbe5
Separate job draining from processing
rochoa Oct 17, 2016
ac7bad4
Lock by host instead of host + user
rochoa Oct 17, 2016
ef6cd24
Correct debug
rochoa Oct 18, 2016
a1400e9
Don't rely on batch.on(job:status)
rochoa Oct 18, 2016
a29f847
Don't rely on batch.on(job:status)
rochoa Oct 18, 2016
1e490be
Don't rely on batch.on(job:status)
rochoa Oct 18, 2016
d1e3be2
Do not emit job:status from batch
rochoa Oct 18, 2016
dce051d
Make leader locker to emit on renewal errors
rochoa Oct 18, 2016
3a57331
Delegate job scheduling
rochoa Oct 18, 2016
ac65c1c
Rename
rochoa Oct 19, 2016
2853c7b
Fix status check
rochoa Oct 19, 2016
51ac1a3
Remove TODO as it is already done
rochoa Oct 19, 2016
6c232a1
Discard numeric status
rochoa Oct 19, 2016
ca3d71e
Tasks with their own entity
rochoa Oct 19, 2016
e26bed2
Move status close to entity
rochoa Oct 19, 2016
4daa39b
Start scheduler from host-scheduler
rochoa Oct 19, 2016
71d32e0
Better debug
rochoa Oct 19, 2016
1ee0878
Scheduler uses a red–black tree to decide on next job candidate
rochoa Oct 19, 2016
5030fdd
Allow to override test client
rochoa Oct 19, 2016
372c9f5
Basic test to test scheduler happy case
rochoa Oct 19, 2016
b164ec8
Better debugging
rochoa Oct 19, 2016
95b3a8a
Be explicit about queue status
rochoa Oct 19, 2016
604e285
Fix and improve test
rochoa Oct 19, 2016
9596ac4
Scheduler handles new tasks when there is free slots
rochoa Oct 19, 2016
0af5cf7
Allow to configure capacity strategy
rochoa Oct 19, 2016
80d2e19
Fix test, use fixed to replace one and infinity
rochoa Oct 19, 2016
a327b30
fixed broken docs hyperlink
Oct 19, 2016
232e69d
Merge pull request #377 from CartoDB/docs-hyperlink-fixes
csobier Oct 19, 2016
4a57d64
Update news and bump version
rochoa Oct 20, 2016
66cc137
Split http capacity between simple and load
rochoa Oct 20, 2016
19def2f
Default to 2 jobs in fixed capacity.
rochoa Oct 20, 2016
75f1ddb
Timeout for http capacity requests
rochoa Oct 20, 2016
d3f3d5c
Call parent with params
rochoa Oct 20, 2016
5185c1e
Cache valid responses for 500 ms
rochoa Oct 20, 2016
4e3bff9
Simplify scheduler to only consider task creation and number of queries
rochoa Oct 20, 2016
5ed8e65
Merge pull request #376 from CartoDB/queue-work-scheduling
Oct 20, 2016
4bfbf04
Release 1.40.0
rochoa Oct 20, 2016
30e353d
Stubs next version
rochoa Oct 20, 2016
aa69bcf
Increase to 4 the default value for fixed capacity
rochoa Oct 20, 2016
e4d54e9
Fix condition to pick next candidate
rochoa Oct 20, 2016
58deb49
Remove runAt property as it is not used
rochoa Oct 20, 2016
66a1c33
Simplify listener subscription logic
rochoa Oct 20, 2016
62cfe97
Use constants for users
rochoa Oct 21, 2016
7563868
Re-insert into the tree if there was a user in done state that gets a…
rochoa Oct 21, 2016
26e4cb3
Get timeout from async function
rochoa Oct 21, 2016
16e9e70
Cancel with user statement_timeout limit from redis
rochoa Oct 21, 2016
506581a
Update news
rochoa Oct 21, 2016
594aba6
Stop migrating old queues by default
rochoa Oct 21, 2016
17ab40f
Release 1.41.0
rochoa Oct 21, 2016
861b9bb
Stubs next version
rochoa Oct 21, 2016
ad3428e
Merge tag '1.41.0' into cdb
rochoa Oct 21, 2016
f6dffb8
Use default name
Oct 27, 2016
8691392
Implement function to save work-in-progress jobs
Oct 27, 2016
ed5b2fb
Implement proxy function to save work-in-progress jobs
Oct 27, 2016
c1f2f93
Change signature
Oct 27, 2016
5b8108d
Use job service to add jobs to work-in-progress list
Oct 27, 2016
8f65e6b
Add test
Oct 27, 2016
f65208b
Add listWorkInProgressJobByUser function
Oct 27, 2016
0085b8e
List users with work in progress jobs
Oct 27, 2016
7241907
Improve var definitions
Oct 27, 2016
97836e6
Add debug message
Oct 27, 2016
1f657a4
Ensure test is isolated
Oct 28, 2016
224a4c9
Rename
Oct 28, 2016
7b48e43
Rename
Oct 28, 2016
f0de347
Rename
Oct 28, 2016
c35b214
Call list of work in progress job
Oct 28, 2016
af75bbd
Rename
Oct 28, 2016
64d0dc9
Removed unnecessary debug
Oct 28, 2016
4a64d37
Add work in progrees list endpoint
Oct 28, 2016
ea06581
Add clear work in progress job
Oct 28, 2016
eaa3315
Add clear work in progress job to service
Oct 28, 2016
aa1527b
Use clear work-in-progress after finishing jobs
Oct 28, 2016
e007b1c
Fix jshint typo
Oct 28, 2016
92b01f1
Hide add/clear job from work in progress list
Oct 28, 2016
cfeabf9
Fix jshint typo
Nov 2, 2016
e459494
Be more accurate on query sleep times
Nov 2, 2016
0da96e5
Set DEBUG env variable to show what queries are being performed
Nov 2, 2016
f282993
Set DEBUG env variable to show what queries are being performed
Nov 2, 2016
d3f4298
Merge branch '378-endpoint-user-batch-queries' of github.com:CartoDB/…
Nov 2, 2016
041b69a
Isolate failing test in CI
Nov 2, 2016
bd71686
Revert last env test changes
Nov 2, 2016
802c93d
Moved work in progress test
Nov 2, 2016
d65be7d
Update NEWS
Nov 2, 2016
5130ea0
Release 1.42.0
Nov 2, 2016
b4d70ba
Stubs next version
Nov 2, 2016
78f5706
Avoid to use scan command to search work in progress queues
Nov 3, 2016
502b63f
Merge pull request #379 from CartoDB/378-endpoint-user-batch-queries
dgaubert Nov 3, 2016
c8b1498
Release 1.42.1
Nov 3, 2016
4a2b725
Do not return error if job could not added to work-in-progress list
Nov 7, 2016
ff463cc
Regenerate npm-shrinkwrap
Nov 7, 2016
8d1db4f
Specify minor version of node in travis config
Nov 7, 2016
3f2a887
Merge pull request #381 from CartoDB/fix-broken-test-in-travis
dgaubert Nov 7, 2016
eed5f5b
Release 1.42.2
Nov 7, 2016
33e6cc0
Stubs next version
Nov 7, 2016
8f668d4
Revert to use a specific minor version in travis config
Nov 7, 2016
99c7a6e
Raise job query size to 16kb
Nov 7, 2016
6d421e0
Update doc
Nov 7, 2016
7d0c69e
Release 1.42.3
Nov 7, 2016
a70ff81
Merge tag '1.42.3' into cdb
Nov 8, 2016
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 8 additions & 1 deletion NEWS.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,14 @@
1.36.2 - 2016-mm-dd
1.36.3 - 2016-mm-dd
-------------------


1.36.2 - 2016-10-03
-------------------

Bug fixes:
- Batch Queries logs: use path instead of stream to be able to reopen FD.


1.36.1 - 2016-09-30
-------------------

14 changes: 9 additions & 5 deletions batch/batch-logger.js
Original file line number Diff line number Diff line change
@@ -1,16 +1,20 @@
'use strict';

var bunyan = require('bunyan');
var fs = require('fs');

function BatchLogger (path) {
var stream = {
level: 'info'
};
if (path) {
stream.path = path;
} else {
stream.stream = process.stdout;
}
this.path = path;
this.logger = bunyan.createLogger({
name: 'batch-queries',
streams: [{
level: 'info',
stream: path ? fs.createWriteStream(path, { flags: 'a', encoding: 'utf8' }) : process.stdout
}]
streams: [stream]
});
}

2 changes: 1 addition & 1 deletion npm-shrinkwrap.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion package.json
Original file line number Diff line number Diff line change
@@ -5,7 +5,7 @@
"keywords": [
"cartodb"
],
"version": "1.36.2",
"version": "1.36.3",
"repository": {
"type": "git",
"url": "git://github.com/CartoDB/CartoDB-SQL-API.git"