-
Notifications
You must be signed in to change notification settings - Fork 4.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[feature] Analytics plugin #272
Conversation
b227424
to
4ee2f95
Compare
I recommend flushing the buffer every 1 second as well. So flush to the server when it gets to 100 ALFs and every time it hasn't been flushed in 1000 ms. Otherwise you could end up with data that gets stuck in the buffer for too long when the traffic is low and API Analytics wouldn't be real-time anymore. |
Yes that's a good idea. However it doesn't make sense to periodically send stuff.
I would rather implement:
|
4ee2f95
to
2354950
Compare
Implemented in 2354950 New config value: |
2354950
to
e23706c
Compare
Yes, that's what I meant, perfect! |
Should I rename it to "mashape analytics" or "analytics"? @sinzone |
e23706c
to
a474015
Compare
a474015
to
6cf77f6
Compare
@thibaultcha call it "Analytics" ( |
0e9751f
to
3fa5826
Compare
86e67de
to
82af65a
Compare
- almost complete ALF serialization - basic data push to apianalytics - entries can be queued in the serializer, and flushed anytime - serializer can output itself in JSON, being a valid ALF object - more tests, with fixtures - introduce serializers for logging plugins Only a temporary solution before each serializer to be a module of its own (maybe?) and each logging plugin to be independent from Kong, and require serializers as dependencies. Following the discussion in #86
- fix: don't mix up different APIs. Using a shared memory zone for having per-API ALFs instead of one ALF having all the APIs entries, which was a mistake.
If the API has low traffic, this timer will flush the data to API Analytics after some delay, otherwise the batch size is never reached, and API Analytics is not real-time anymore.
82af65a
to
5deb878
Compare
5deb878
to
9de27b6
Compare
9de27b6
to
692d8a6
Compare
[feature] Analytics plugin Former-commit-id: 5f9fc30ac9670dc02177b330f2916d5fec56eb1e
#### 1.7.0 (2019-10-14) ##### New features - utils.quote_arg will now optionally take an array of arguments and escape them all into a single string. - app.parse_args now accepts a 3rd parameter with a list of valid flags and aliases - app.script_name returns the name of the current script (previously a private function) ##### Changes - Documentation updates - utils.quit: exit message is no longer required, and closes the Lua state (on 5.2+). - utils.assert_arg and utils.assert_string: now return the validated value - pl.compat: now exports the jit and jit52 flags - pretty.write: now sorts the output for easier diffs #293 ##### Fixes - utils.raise changed the global on_error-level when passing in bad arguments - utils.writefile now checks and returns errors when writing - compat.execute now handles the Windows exitcode -1 properly - types.is_empty would return true on spaces always, indepedent of the parameter - types.to_bool will now compare case-insensitive for the extra passed strings - app.require_here will now properly handle an absolute base path - stringx.split will no longer append an empty match if the number of requested elements has already been reached #295 - path.common_prefix and path.relpath return the result in the original casing (only impacted Windows) #297 - dir.copyfile, dir.movefile, and dir.makepath create the new file/path with the requested casing, and no longer force lowercase (only impacted Windows) #297 - added a missing assertion on path.getmtime #291 - stringx.rpartition returned bad results on a not-found #299 #### 1.6.0 (2018-11-23) ##### New features - pl.compat now provides unpack as table.unpack on Lua 5.1 ##### Changes - utils.unpack is now documented and respects .n field of its argument. - tablex.deepcopy and tablex.deepcompare are now cycle aware (#262) - Installing through LuaRocks will now include the full rendered documentation ##### Fixes - Fixed seq.last returning nil instead of an empty list when given an empty iterator (#253). - pl.template now applies tostring when substituting values in templates, avoiding errors when they are not strings or numbers (#256). - Fixed pl.import_into not importing some Penlight modules (#268). - Fixed version number stuck at 1.5.2 (#260). - Fixed types.is_empty returning true on tables containing false key (#267). - Fixed test.assertraise throwing an error when passed an array with a function to call plus its arguments (#272). - Fixed test.assertraise not throwing an error when given function does not error but instead returns a string matching given error pattern. - Fixed placeholder expressions being evaluated with wrong precedence of binary and unary negation. - Fixed placeholder expressions being evaluated assuming wrong binary operator associativity (e.g. _1-(_2+_3) was evaluated as (_1-_2)+_3. - Fixed placeholder expressions being evaluated as if unary operators take precedence over power operator (e.g. (-_1)^_2) was evaluated as -(_1^2)). - Fixed vulnerable backtracking pattern in pl.stringx.strip (#275)
#### 1.7.0 (2019-10-14) ##### New features - utils.quote_arg will now optionally take an array of arguments and escape them all into a single string. - app.parse_args now accepts a 3rd parameter with a list of valid flags and aliases - app.script_name returns the name of the current script (previously a private function) ##### Changes - Documentation updates - utils.quit: exit message is no longer required, and closes the Lua state (on 5.2+). - utils.assert_arg and utils.assert_string: now return the validated value - pl.compat: now exports the jit and jit52 flags - pretty.write: now sorts the output for easier diffs #293 ##### Fixes - utils.raise changed the global on_error-level when passing in bad arguments - utils.writefile now checks and returns errors when writing - compat.execute now handles the Windows exitcode -1 properly - types.is_empty would return true on spaces always, indepedent of the parameter - types.to_bool will now compare case-insensitive for the extra passed strings - app.require_here will now properly handle an absolute base path - stringx.split will no longer append an empty match if the number of requested elements has already been reached #295 - path.common_prefix and path.relpath return the result in the original casing (only impacted Windows) #297 - dir.copyfile, dir.movefile, and dir.makepath create the new file/path with the requested casing, and no longer force lowercase (only impacted Windows) #297 - added a missing assertion on path.getmtime #291 - stringx.rpartition returned bad results on a not-found #299 #### 1.6.0 (2018-11-23) ##### New features - pl.compat now provides unpack as table.unpack on Lua 5.1 ##### Changes - utils.unpack is now documented and respects .n field of its argument. - tablex.deepcopy and tablex.deepcompare are now cycle aware (#262) - Installing through LuaRocks will now include the full rendered documentation ##### Fixes - Fixed seq.last returning nil instead of an empty list when given an empty iterator (#253). - pl.template now applies tostring when substituting values in templates, avoiding errors when they are not strings or numbers (#256). - Fixed pl.import_into not importing some Penlight modules (#268). - Fixed version number stuck at 1.5.2 (#260). - Fixed types.is_empty returning true on tables containing false key (#267). - Fixed test.assertraise throwing an error when passed an array with a function to call plus its arguments (#272). - Fixed test.assertraise not throwing an error when given function does not error but instead returns a string matching given error pattern. - Fixed placeholder expressions being evaluated with wrong precedence of binary and unary negation. - Fixed placeholder expressions being evaluated assuming wrong binary operator associativity (e.g. _1-(_2+_3) was evaluated as (_1-_2)+_3. - Fixed placeholder expressions being evaluated as if unary operators take precedence over power operator (e.g. (-_1)^_2) was evaluated as -(_1^2)). - Fixed vulnerable backtracking pattern in pl.stringx.strip (#275)
#### 1.7.0 (2019-10-14) ##### New features - utils.quote_arg will now optionally take an array of arguments and escape them all into a single string. - app.parse_args now accepts a 3rd parameter with a list of valid flags and aliases - app.script_name returns the name of the current script (previously a private function) ##### Changes - Documentation updates - utils.quit: exit message is no longer required, and closes the Lua state (on 5.2+). - utils.assert_arg and utils.assert_string: now return the validated value - pl.compat: now exports the jit and jit52 flags - pretty.write: now sorts the output for easier diffs #293 ##### Fixes - utils.raise changed the global on_error-level when passing in bad arguments - utils.writefile now checks and returns errors when writing - compat.execute now handles the Windows exitcode -1 properly - types.is_empty would return true on spaces always, indepedent of the parameter - types.to_bool will now compare case-insensitive for the extra passed strings - app.require_here will now properly handle an absolute base path - stringx.split will no longer append an empty match if the number of requested elements has already been reached #295 - path.common_prefix and path.relpath return the result in the original casing (only impacted Windows) #297 - dir.copyfile, dir.movefile, and dir.makepath create the new file/path with the requested casing, and no longer force lowercase (only impacted Windows) #297 - added a missing assertion on path.getmtime #291 - stringx.rpartition returned bad results on a not-found #299 #### 1.6.0 (2018-11-23) ##### New features - pl.compat now provides unpack as table.unpack on Lua 5.1 ##### Changes - utils.unpack is now documented and respects .n field of its argument. - tablex.deepcopy and tablex.deepcompare are now cycle aware (#262) - Installing through LuaRocks will now include the full rendered documentation ##### Fixes - Fixed seq.last returning nil instead of an empty list when given an empty iterator (#253). - pl.template now applies tostring when substituting values in templates, avoiding errors when they are not strings or numbers (#256). - Fixed pl.import_into not importing some Penlight modules (#268). - Fixed version number stuck at 1.5.2 (#260). - Fixed types.is_empty returning true on tables containing false key (#267). - Fixed test.assertraise throwing an error when passed an array with a function to call plus its arguments (#272). - Fixed test.assertraise not throwing an error when given function does not error but instead returns a string matching given error pattern. - Fixed placeholder expressions being evaluated with wrong precedence of binary and unary negation. - Fixed placeholder expressions being evaluated assuming wrong binary operator associativity (e.g. _1-(_2+_3) was evaluated as (_1-_2)+_3. - Fixed placeholder expressions being evaluated as if unary operators take precedence over power operator (e.g. (-_1)^_2) was evaluated as -(_1^2)). - Fixed vulnerable backtracking pattern in pl.stringx.strip (#275)
#### 1.7.0 (2019-10-14) ##### New features - utils.quote_arg will now optionally take an array of arguments and escape them all into a single string. - app.parse_args now accepts a 3rd parameter with a list of valid flags and aliases - app.script_name returns the name of the current script (previously a private function) ##### Changes - Documentation updates - utils.quit: exit message is no longer required, and closes the Lua state (on 5.2+). - utils.assert_arg and utils.assert_string: now return the validated value - pl.compat: now exports the jit and jit52 flags - pretty.write: now sorts the output for easier diffs #293 ##### Fixes - utils.raise changed the global on_error-level when passing in bad arguments - utils.writefile now checks and returns errors when writing - compat.execute now handles the Windows exitcode -1 properly - types.is_empty would return true on spaces always, indepedent of the parameter - types.to_bool will now compare case-insensitive for the extra passed strings - app.require_here will now properly handle an absolute base path - stringx.split will no longer append an empty match if the number of requested elements has already been reached #295 - path.common_prefix and path.relpath return the result in the original casing (only impacted Windows) #297 - dir.copyfile, dir.movefile, and dir.makepath create the new file/path with the requested casing, and no longer force lowercase (only impacted Windows) #297 - added a missing assertion on path.getmtime #291 - stringx.rpartition returned bad results on a not-found #299 #### 1.6.0 (2018-11-23) ##### New features - pl.compat now provides unpack as table.unpack on Lua 5.1 ##### Changes - utils.unpack is now documented and respects .n field of its argument. - tablex.deepcopy and tablex.deepcompare are now cycle aware (#262) - Installing through LuaRocks will now include the full rendered documentation ##### Fixes - Fixed seq.last returning nil instead of an empty list when given an empty iterator (#253). - pl.template now applies tostring when substituting values in templates, avoiding errors when they are not strings or numbers (#256). - Fixed pl.import_into not importing some Penlight modules (#268). - Fixed version number stuck at 1.5.2 (#260). - Fixed types.is_empty returning true on tables containing false key (#267). - Fixed test.assertraise throwing an error when passed an array with a function to call plus its arguments (#272). - Fixed test.assertraise not throwing an error when given function does not error but instead returns a string matching given error pattern. - Fixed placeholder expressions being evaluated with wrong precedence of binary and unary negation. - Fixed placeholder expressions being evaluated assuming wrong binary operator associativity (e.g. _1-(_2+_3) was evaluated as (_1-_2)+_3. - Fixed placeholder expressions being evaluated as if unary operators take precedence over power operator (e.g. (-_1)^_2) was evaluated as -(_1^2)). - Fixed vulnerable backtracking pattern in pl.stringx.strip (#275)
#### 1.7.0 (2019-10-14) ##### New features - utils.quote_arg will now optionally take an array of arguments and escape them all into a single string. - app.parse_args now accepts a 3rd parameter with a list of valid flags and aliases - app.script_name returns the name of the current script (previously a private function) ##### Changes - Documentation updates - utils.quit: exit message is no longer required, and closes the Lua state (on 5.2+). - utils.assert_arg and utils.assert_string: now return the validated value - pl.compat: now exports the jit and jit52 flags - pretty.write: now sorts the output for easier diffs #293 ##### Fixes - utils.raise changed the global on_error-level when passing in bad arguments - utils.writefile now checks and returns errors when writing - compat.execute now handles the Windows exitcode -1 properly - types.is_empty would return true on spaces always, indepedent of the parameter - types.to_bool will now compare case-insensitive for the extra passed strings - app.require_here will now properly handle an absolute base path - stringx.split will no longer append an empty match if the number of requested elements has already been reached #295 - path.common_prefix and path.relpath return the result in the original casing (only impacted Windows) #297 - dir.copyfile, dir.movefile, and dir.makepath create the new file/path with the requested casing, and no longer force lowercase (only impacted Windows) #297 - added a missing assertion on path.getmtime #291 - stringx.rpartition returned bad results on a not-found #299 #### 1.6.0 (2018-11-23) ##### New features - pl.compat now provides unpack as table.unpack on Lua 5.1 ##### Changes - utils.unpack is now documented and respects .n field of its argument. - tablex.deepcopy and tablex.deepcompare are now cycle aware (#262) - Installing through LuaRocks will now include the full rendered documentation ##### Fixes - Fixed seq.last returning nil instead of an empty list when given an empty iterator (#253). - pl.template now applies tostring when substituting values in templates, avoiding errors when they are not strings or numbers (#256). - Fixed pl.import_into not importing some Penlight modules (#268). - Fixed version number stuck at 1.5.2 (#260). - Fixed types.is_empty returning true on tables containing false key (#267). - Fixed test.assertraise throwing an error when passed an array with a function to call plus its arguments (#272). - Fixed test.assertraise not throwing an error when given function does not error but instead returns a string matching given error pattern. - Fixed placeholder expressions being evaluated with wrong precedence of binary and unary negation. - Fixed placeholder expressions being evaluated assuming wrong binary operator associativity (e.g. _1-(_2+_3) was evaluated as (_1-_2)+_3. - Fixed placeholder expressions being evaluated as if unary operators take precedence over power operator (e.g. (-_1)^_2) was evaluated as -(_1^2)). - Fixed vulnerable backtracking pattern in pl.stringx.strip (#275)
#### 1.7.0 (2019-10-14) ##### New features - utils.quote_arg will now optionally take an array of arguments and escape them all into a single string. - app.parse_args now accepts a 3rd parameter with a list of valid flags and aliases - app.script_name returns the name of the current script (previously a private function) ##### Changes - Documentation updates - utils.quit: exit message is no longer required, and closes the Lua state (on 5.2+). - utils.assert_arg and utils.assert_string: now return the validated value - pl.compat: now exports the jit and jit52 flags - pretty.write: now sorts the output for easier diffs #293 ##### Fixes - utils.raise changed the global on_error-level when passing in bad arguments - utils.writefile now checks and returns errors when writing - compat.execute now handles the Windows exitcode -1 properly - types.is_empty would return true on spaces always, indepedent of the parameter - types.to_bool will now compare case-insensitive for the extra passed strings - app.require_here will now properly handle an absolute base path - stringx.split will no longer append an empty match if the number of requested elements has already been reached #295 - path.common_prefix and path.relpath return the result in the original casing (only impacted Windows) #297 - dir.copyfile, dir.movefile, and dir.makepath create the new file/path with the requested casing, and no longer force lowercase (only impacted Windows) #297 - added a missing assertion on path.getmtime #291 - stringx.rpartition returned bad results on a not-found #299 #### 1.6.0 (2018-11-23) ##### New features - pl.compat now provides unpack as table.unpack on Lua 5.1 ##### Changes - utils.unpack is now documented and respects .n field of its argument. - tablex.deepcopy and tablex.deepcompare are now cycle aware (#262) - Installing through LuaRocks will now include the full rendered documentation ##### Fixes - Fixed seq.last returning nil instead of an empty list when given an empty iterator (#253). - pl.template now applies tostring when substituting values in templates, avoiding errors when they are not strings or numbers (#256). - Fixed pl.import_into not importing some Penlight modules (#268). - Fixed version number stuck at 1.5.2 (#260). - Fixed types.is_empty returning true on tables containing false key (#267). - Fixed test.assertraise throwing an error when passed an array with a function to call plus its arguments (#272). - Fixed test.assertraise not throwing an error when given function does not error but instead returns a string matching given error pattern. - Fixed placeholder expressions being evaluated with wrong precedence of binary and unary negation. - Fixed placeholder expressions being evaluated assuming wrong binary operator associativity (e.g. _1-(_2+_3) was evaluated as (_1-_2)+_3. - Fixed placeholder expressions being evaluated as if unary operators take precedence over power operator (e.g. (-_1)^_2) was evaluated as -(_1^2)). - Fixed vulnerable backtracking pattern in pl.stringx.strip (#275)
- utils.quote_arg will now optionally take an array of arguments and escape them all into a single string. - app.parse_args now accepts a 3rd parameter with a list of valid flags and aliases - app.script_name returns the name of the current script (previously a private function) - Documentation updates - utils.quit: exit message is no longer required, and closes the Lua state (on 5.2+). - utils.assert_arg and utils.assert_string: now return the validated value - pl.compat: now exports the jit and jit52 flags - pretty.write: now sorts the output for easier diffs #293 - utils.raise changed the global on_error-level when passing in bad arguments - utils.writefile now checks and returns errors when writing - compat.execute now handles the Windows exitcode -1 properly - types.is_empty would return true on spaces always, indepedent of the parameter - types.to_bool will now compare case-insensitive for the extra passed strings - app.require_here will now properly handle an absolute base path - stringx.split will no longer append an empty match if the number of requested elements has already been reached #295 - path.common_prefix and path.relpath return the result in the original casing (only impacted Windows) #297 - dir.copyfile, dir.movefile, and dir.makepath create the new file/path with the requested casing, and no longer force lowercase (only impacted Windows) #297 - added a missing assertion on path.getmtime #291 - stringx.rpartition returned bad results on a not-found #299 - pl.compat now provides unpack as table.unpack on Lua 5.1 - utils.unpack is now documented and respects .n field of its argument. - tablex.deepcopy and tablex.deepcompare are now cycle aware (#262) - Installing through LuaRocks will now include the full rendered documentation - Fixed seq.last returning nil instead of an empty list when given an empty iterator (#253). - pl.template now applies tostring when substituting values in templates, avoiding errors when they are not strings or numbers (#256). - Fixed pl.import_into not importing some Penlight modules (#268). - Fixed version number stuck at 1.5.2 (#260). - Fixed types.is_empty returning true on tables containing false key (#267). - Fixed test.assertraise throwing an error when passed an array with a function to call plus its arguments (#272). - Fixed test.assertraise not throwing an error when given function does not error but instead returns a string matching given error pattern. - Fixed placeholder expressions being evaluated with wrong precedence of binary and unary negation. - Fixed placeholder expressions being evaluated assuming wrong binary operator associativity (e.g. _1-(_2+_3) was evaluated as (_1-_2)+_3. - Fixed placeholder expressions being evaluated as if unary operators take precedence over power operator (e.g. (-_1)^_2) was evaluated as -(_1^2)). - Fixed vulnerable backtracking pattern in pl.stringx.strip (#275)
#### 1.7.0 (2019-10-14) ##### New features - utils.quote_arg will now optionally take an array of arguments and escape them all into a single string. - app.parse_args now accepts a 3rd parameter with a list of valid flags and aliases - app.script_name returns the name of the current script (previously a private function) ##### Changes - Documentation updates - utils.quit: exit message is no longer required, and closes the Lua state (on 5.2+). - utils.assert_arg and utils.assert_string: now return the validated value - pl.compat: now exports the jit and jit52 flags - pretty.write: now sorts the output for easier diffs #293 ##### Fixes - utils.raise changed the global on_error-level when passing in bad arguments - utils.writefile now checks and returns errors when writing - compat.execute now handles the Windows exitcode -1 properly - types.is_empty would return true on spaces always, indepedent of the parameter - types.to_bool will now compare case-insensitive for the extra passed strings - app.require_here will now properly handle an absolute base path - stringx.split will no longer append an empty match if the number of requested elements has already been reached #295 - path.common_prefix and path.relpath return the result in the original casing (only impacted Windows) #297 - dir.copyfile, dir.movefile, and dir.makepath create the new file/path with the requested casing, and no longer force lowercase (only impacted Windows) #297 - added a missing assertion on path.getmtime #291 - stringx.rpartition returned bad results on a not-found #299 #### 1.6.0 (2018-11-23) ##### New features - pl.compat now provides unpack as table.unpack on Lua 5.1 ##### Changes - utils.unpack is now documented and respects .n field of its argument. - tablex.deepcopy and tablex.deepcompare are now cycle aware (#262) - Installing through LuaRocks will now include the full rendered documentation ##### Fixes - Fixed seq.last returning nil instead of an empty list when given an empty iterator (#253). - pl.template now applies tostring when substituting values in templates, avoiding errors when they are not strings or numbers (#256). - Fixed pl.import_into not importing some Penlight modules (#268). - Fixed version number stuck at 1.5.2 (#260). - Fixed types.is_empty returning true on tables containing false key (#267). - Fixed test.assertraise throwing an error when passed an array with a function to call plus its arguments (#272). - Fixed test.assertraise not throwing an error when given function does not error but instead returns a string matching given error pattern. - Fixed placeholder expressions being evaluated with wrong precedence of binary and unary negation. - Fixed placeholder expressions being evaluated assuming wrong binary operator associativity (e.g. _1-(_2+_3) was evaluated as (_1-_2)+_3. - Fixed placeholder expressions being evaluated as if unary operators take precedence over power operator (e.g. (-_1)^_2) was evaluated as -(_1^2)). - Fixed vulnerable backtracking pattern in pl.stringx.strip (#275)
fix(packaging) fix some build packaging errors
Implementing analytics plugin, part of #86.
Changes:
/plugins/log_serializers
. Ideally, they would live in their own repos and be dependencies of all logging plugins, and the user will chose the logging format by selecting one of them. This is a step in that direction.localhost
for testing) when the size exceeds the limit set (default to100
).log_body
log or not the request/response bodies (default false)service_token
API Analytics service token (required)batch_size
When the entries reach or exceed that size, the ALF is sent (default 100)Notes:
response.statusText
andresponse.httpVersion
because I wasn't able to find a way to read the raw response from nginx.batch_size
.Future improvements:
cookies
, andresponse.redirectURL
?