Proposed updates for release 0.92 #41
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
python-requests does not like it when it sees Unicode here.
For now, assume UTF-8 encoding since that seems to be the standard being used elsewhere.
Ensure all headers are encoded using US-ASCII encoding. Ensure all body data is encoded using UTF-8 encoding. We'll treat passwords as being UTF-8.
The authorization header is basically a hash, so US-ASCII is the most appropriate for this.
If a string object has no 'decode' method, then assume it is a Unicode object already.
- Headers should be bytes() objects - Bodies should be decoded from UTF-8
This gives us the ability to whitelist specific status codes, instead of failing normally "bad" codes such as 400, 401, 403 or 404, allowing the underlying client code to handle it.
Firefly will report error 401 if the session is invalid.
If we have a 401 error message thrown at us, assume we've been logged out. This could be delivered by way of a response or an exception. We should not raise an exception ourselves, but instead just invalidate the session.
To imply that it's not just *any* HTTP response this gets called on, but rather specific responses.
Add a test that verifies that the function *does* get called when a grid is requested.
…oaded Otherwise you get stack traces like this when pandas isn't installed: error 06-Sep-2016 14:45:22 ====================================================================== error 06-Sep-2016 14:45:22 ERROR: Attempt to add a single user error 06-Sep-2016 14:45:22 ---------------------------------------------------------------------- error 06-Sep-2016 14:45:22 Traceback (most recent call last): error 06-Sep-2016 14:45:22 File "/usr/lib/python2.7/dist-packages/nose/case.py", line 197, in runTest error 06-Sep-2016 14:45:22 self.test(*self.arg) error 06-Sep-2016 14:45:22 File "/mnt/src/tests/test_user_editor_add.py", line 58, in test_add_single_user error 06-Sep-2016 14:45:22 core.main(arguments) error 06-Sep-2016 14:45:22 File "/mnt/src/widesky_editor/user/core.py", line 35, in main error 06-Sep-2016 14:45:22 session = init_haystack(args) error 06-Sep-2016 14:45:22 File "/mnt/src/widesky_editor/common.py", line 67, in init_haystack error 06-Sep-2016 14:45:22 http_args=http_args, tagging_model=WideSkyTaggingModel) error 06-Sep-2016 14:45:22 File "/usr/lib/python2.7/dist-packages/pyhaystack/client/loader.py", line 61, in get_instance error 06-Sep-2016 14:45:22 implementation = get_implementation(implementation) error 06-Sep-2016 14:45:22 File "/usr/lib/python2.7/dist-packages/pyhaystack/client/loader.py", line 34, in get_implementation error 06-Sep-2016 14:45:22 package='pyhaystack.client') error 06-Sep-2016 14:45:22 File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module error 06-Sep-2016 14:45:22 __import__(name) error 06-Sep-2016 14:45:22 File "/usr/lib/python2.7/dist-packages/pyhaystack/client/widesky.py", line 8, in <module> error 06-Sep-2016 14:45:22 from .session import HaystackSession error 06-Sep-2016 14:45:22 File "/usr/lib/python2.7/dist-packages/pyhaystack/client/session.py", line 18, in <module> error 06-Sep-2016 14:45:22 from .ops import his as his_ops error 06-Sep-2016 14:45:22 File "/usr/lib/python2.7/dist-packages/pyhaystack/client/ops/his.py", line 801, in <module> error 06-Sep-2016 14:45:22 class MetaSeries(Series): error 06-Sep-2016 14:45:22 NameError: name 'Series' is not defined
- Handle non-semver version string - Prefer moduleVersion over productVersion; newer versions will use this over productVersion which will be the "WideSky release". (like Ubuntu's; 16.04.)
sjlongland
force-pushed
the
feature/proposed-0.92-vrt-changes
branch
from
be207da
to
9f12a6f
Compare
|
Actually, I'll pull one failed commit (be207da) out of that set… the rest is good to go. :-) |
|
Many thanks :-) |
|
I'm very lucky having you guys. A great amount of work. |
ChristianTremblay
added a commit
that referenced
this pull request
May 8, 2017
* First implementation of a browser that will help navigate the haystack server from pyhaystack.client.niagara import NiagaraHaystackSession from pyhaystack.browser.browser import Browser import logging logging.root.setLevel(logging.DEBUG) session = NiagaraHaystackSession(uri='http://ip', username='user', password='password') server = Browser(session) site = server['Site'] will return the site site['equipment'] will return an equipment Signed-off-by: Christian Tremblay, ing. <christian.tremblay@servisys.com> * Equipments and points are created Signed-off-by: Christian Tremblay, ing. <christian.tremblay@servisys.com> * New approach for high level interface Adding functions to mixins Signed-off-by: Christian Tremblay, ing. <christian.tremblay@servisys.com> * Adding helper functions to high level interface (__iter__, __getitem__, etc...) Signed-off-by: Christian Tremblay, ing. <christian.tremblay@servisys.com> * client.http.base: Send Content-Length as a string. `requests` tries to do a regex match or similar on the values of headers, and this requires the input to be a string. It barfs when you give it an `int`. * client.ops.vendor.widesky: Fix typo in create entity operation. * client.ops.entity: Add loggers, handle empty grid. * client.ops.his: Handle write of empty series. * client.ops.his: Handle no columns in write_his_frame. * WC-241: revive multi-hist mixin. * WC-241: initialize timestamp column for write grid * WC-241: Added a simple application that uses the pyhaystack library Edit by Stuart Longland <stuartl@vrt.com.au>: Removal of default credentials! * WC-237: Re-locate manual mutli-his-read-write test script. Manually run test scripts can go under tests/manual, until such time as they get automated. * WC-237: client.session: Add storage for grid cache. Some grids are unlikely to change in the course of a session. Examples of these are the 'about', 'formats' and 'ops' grids, which indicate the features that the Haystack server supports. We should cache these for a limited period of time so that the results can be used for enabling features depending on server version. * WC-237: client.ops.grid: Implement cache support * WC-237: client.session: Enable caching for about, ops, formats. * WC-237: client.mixins: Remove defunct classes from __init__ * WC-237: client.session: Fix typos * WC-237: client.session: Add "has_feature" operation. Used to detect extension features in the Haystack server. Some servers implement extension featurs that the standard Haystack API makes difficult to expose. Sometimes a feature is a "new" feature in a particular release of a server, as such, it should be disabled if the version is found to be below a certain threshold. (e.g. multi-point hisRead/hisWrite is supported in WideSky v0.5.0 but not earlier) This allows a run-time check to take place in the relevant state machines to decide how to tackle the problem. * WC-237: client.ops.feature: Add HasFeatureOperation base implementation. * WC-237: client.ops.feature: Fix some state machine bugs, typos. * WC-237: client.session: Use base class for HasFeaturesOperation * WC-237: client.ops.vendor.widesky: Add detection of WideSky extensions * WC-237: client.widesky: Enable detection of WideSky extensions * WC-237: client.ops.his: Probe multi-point hisRead * WC-237: client.ops.his: Implement probing of multi-hisWrite * WC-237: Pull in semver for WideSky version checking * WC-237: client.ops.his: Fix multi-point hisRead/hisWrite issues. * client.http.dummy: Dummy HTTP Client/Server This is a "HTTP Server" class and matching "Client" that we can drive like a puppet for writing test cases. * tests: Bring test case for 'about' method. * client.http.base: Handle 'str' input on Python3. If we're given an 'str', we might safely assume that the body is in fact decoded already. * tests.util: Add some utility functions * tests.client.test_base: Check output of about() * tests.client.test_base: Add ops test * tests.client.test_base: Add formats() test * tests.client.test_base: More tests for read by ID, etc. * getitem will now return tags or equip (for site) or points (for equip) Some bugs are still there but WIP goes well Signed-off-by: Christian Tremblay, ing. <christian.tremblay@servisys.com> * Corrected point mixin so value works Added his shortcut to his_read_series (with a default to 'today' range ex. znt.his(rng='yesterday') or znt.his() filter requests works with __getitem__ ex. zone_sensors = my_office['sensor and zone'] Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com> * Bugfix: Handle missing entities in multi-entity read requests (#26) * WC-366: client.ops.entity: Handle missing entities in read response. If an entity does not exist, and we are requesting more than one, the missing entity appears in the grid as a row of nulls: DEBUG:pyhaystack.client.WideskyHaystackSession.get_entity:Received grid: <Grid> Columns: id dis status user Row 0: id=Ref(u'test_user4', u'test_user4', True), dis=u'Test User4', status=u'inactive', user=MARKER Row 1: id=None, dis=None, status=None, user=None Row 2: id=Ref(u'test_user2', u'test_user2', True), dis=u'Test User2', status=u'inactive', user=MARKER Row 3: id=Ref(u'test_user3', u'test_user3', True), dis=u'Test User3', status=u'inactive', user=MARKER </Grid> which then results in this exception: File "pyhaystack/client/ops/entity.py", line 51, in _on_read entity_id = row.pop('id').name # Should be a Ref AttributeError: 'NoneType' object has no attribute 'name' We should skip rows that have a null 'id'. * WC-366: client entity tests: Test retrieval of entities by id. * WC-366: client entity tests: Python 3 support * client.ops.grid: Handle expired cache (#35) If the cache has expired, then we can wind up with the situation where `op` is `None`, so the `if` block falls through to the `else` branch erroniously. This causes a stack trace like the following: ``` 2017-02-17 01:49:00,039 pyhaystack.client.WideskyHaystackSession.his_write_frame[his.py: 726] WARNING Unable to probe multi-his-write support Traceback (most recent call last): File "/usr/lib/python2.7/dist-packages/pyhaystack/client/ops/his.py", line 723, in _on_probe_multi result = operation.result File "/usr/lib/python2.7/dist-packages/pyhaystack/util/state.py", line 97, in result self._result.reraise() File "/usr/lib/python2.7/dist-packages/pyhaystack/util/asyncexc.py", line 28, in reraise reraise(*self._exc_info) File "/usr/lib/python2.7/dist-packages/pyhaystack/client/ops/feature.py", line 93, in _on_got_about self._about = operation.result File "/usr/lib/python2.7/dist-packages/pyhaystack/util/state.py", line 97, in result self._result.reraise() File "/usr/lib/python2.7/dist-packages/pyhaystack/util/asyncexc.py", line 28, in reraise reraise(*self._exc_info) File "/usr/lib/python2.7/dist-packages/pyhaystack/client/ops/grid.py", line 118, in _check_auth self._state_machine.auth_ok() File "/usr/lib/python2.7/dist-packages/fysom/__init__.py", line 297, in fn self.transition() File "/usr/lib/python2.7/dist-packages/fysom/__init__.py", line 290, in _tran self._enter_state(e) File "/usr/lib/python2.7/dist-packages/fysom/__init__.py", line 341, in _enter_state return getattr(self, fnname)(e) File "/usr/lib/python2.7/dist-packages/fysom/__init__.py", line 88, in _callback return func(obj, *args, **kwargs) File "/usr/lib/python2.7/dist-packages/pyhaystack/client/ops/grid.py", line 177, in _do_check_cache op.done_sig.connect(_proxy) AttributeError: 'NoneType' object has no attribute 'done_sig' ``` We need to instead do our check while we have the mutex locked and disregard the expired cache. * skyspark V3 SCRAM authentication (#36) First Raw of Skyspark V3 SCRAM authentication * Proposed updates for release 0.92 (#41) * WC-414: Ensure HTTP header names are US-ASCII encoded. python-requests does not like it when it sees Unicode here. * WC-414: client.ops.grid: Ensure grid is encoded For now, assume UTF-8 encoding since that seems to be the standard being used elsewhere. * WC-414: client.ops.vendor.widesky: Fix encoding issues. Ensure all headers are encoded using US-ASCII encoding. Ensure all body data is encoded using UTF-8 encoding. We'll treat passwords as being UTF-8. * WC-414: client.widesky: Send authorization in US-ASCII encoding. The authorization header is basically a hash, so US-ASCII is the most appropriate for this. * WC-414: client.widesky: Assume already 'unicode' if no 'decode'. If a string object has no 'decode' method, then assume it is a Unicode object already. * WC-414: client.http.base: Use bytes object not str/unicode * WC-414: client.ops.grid: Use bytes object not str/unicode * WC-414: base client tests: Fix encoding issues - Headers should be bytes() objects - Bodies should be decoded from UTF-8 * WC-414: client entity tests: Fix more encoding issues. * FIR-214: client.http.base: Assert that we have a content type. * FIR-214: client.http.base: Add accept_status argument. This gives us the ability to whitelist specific status codes, instead of failing normally "bad" codes such as 400, 401, 403 or 404, allowing the underlying client code to handle it. * FIR-214: client.http.dummy: Add accept_status logic. * FIR-214: client.http.sync: Add accept_status logic. * FIR-214: client.ops.grid: Pass through accept_status * FIR-214: client.ops.grid: Add hook for detecting session issues. Firefly will report error 401 if the session is invalid. * FIR-214: client.widesky: Add hook to detect 401 error. If we have a 401 error message thrown at us, assume we've been logged out. This could be delivered by way of a response or an exception. We should not raise an exception ourselves, but instead just invalidate the session. * FIR-214: client: _on_http_response → _on_http_grid_response. To imply that it's not just *any* HTTP response this gets called on, but rather specific responses. * FIR-214: client.widesky tests: Add test for is_logged_in. * FIR-214: client.widesky tests: Test 401 drops the session. * FIR-214: client.session tests: Check _on_http_grid_response gets called. Add a test that verifies that the function *does* get called when a grid is requested. * FIR-214: client.widesky tests: Test other exceptions are ignored. * client.ops.his: Don't define MetaSeries/MetaDataFrame unless pandas loaded Otherwise you get stack traces like this when pandas isn't installed: error 06-Sep-2016 14:45:22 ====================================================================== error 06-Sep-2016 14:45:22 ERROR: Attempt to add a single user error 06-Sep-2016 14:45:22 ---------------------------------------------------------------------- error 06-Sep-2016 14:45:22 Traceback (most recent call last): error 06-Sep-2016 14:45:22 File "/usr/lib/python2.7/dist-packages/nose/case.py", line 197, in runTest error 06-Sep-2016 14:45:22 self.test(*self.arg) error 06-Sep-2016 14:45:22 File "/mnt/src/tests/test_user_editor_add.py", line 58, in test_add_single_user error 06-Sep-2016 14:45:22 core.main(arguments) error 06-Sep-2016 14:45:22 File "/mnt/src/widesky_editor/user/core.py", line 35, in main error 06-Sep-2016 14:45:22 session = init_haystack(args) error 06-Sep-2016 14:45:22 File "/mnt/src/widesky_editor/common.py", line 67, in init_haystack error 06-Sep-2016 14:45:22 http_args=http_args, tagging_model=WideSkyTaggingModel) error 06-Sep-2016 14:45:22 File "/usr/lib/python2.7/dist-packages/pyhaystack/client/loader.py", line 61, in get_instance error 06-Sep-2016 14:45:22 implementation = get_implementation(implementation) error 06-Sep-2016 14:45:22 File "/usr/lib/python2.7/dist-packages/pyhaystack/client/loader.py", line 34, in get_implementation error 06-Sep-2016 14:45:22 package='pyhaystack.client') error 06-Sep-2016 14:45:22 File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module error 06-Sep-2016 14:45:22 __import__(name) error 06-Sep-2016 14:45:22 File "/usr/lib/python2.7/dist-packages/pyhaystack/client/widesky.py", line 8, in <module> error 06-Sep-2016 14:45:22 from .session import HaystackSession error 06-Sep-2016 14:45:22 File "/usr/lib/python2.7/dist-packages/pyhaystack/client/session.py", line 18, in <module> error 06-Sep-2016 14:45:22 from .ops import his as his_ops error 06-Sep-2016 14:45:22 File "/usr/lib/python2.7/dist-packages/pyhaystack/client/ops/his.py", line 801, in <module> error 06-Sep-2016 14:45:22 class MetaSeries(Series): error 06-Sep-2016 14:45:22 NameError: name 'Series' is not defined * WC-9: client.ops.vendor.widesky: Version detection tweaks - Handle non-semver version string - Prefer moduleVersion over productVersion; newer versions will use this over productVersion which will be the "WideSky release". (like Ubuntu's; 16.04.) * VRT-506: client.http.base: Make headers and cookies case-insensitive. * VRT-506: client.http.base: Fix `bytes`/`str` issue, initialisation. * client.loader: Add support for aliases for known implementations. (#44) This allows us to whistle up aliases for implementations such as `ax` for Niagara AX, etc. Further implementations can be added to this file, and they will be loaded when needed. * Added support for easier session declaration using get_instance we can now use : ``` import pyhaystack pyhaystack.connect('ax', 'uri', 'username', 'password') #or import pyhaystack pyhaystack.connect('ax', uri='uri', username='username', password='password') ``` Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com> * Feature/scram (#45) * Just beginning to implement SCRAM script by Pierre into pyhaystack... Modified state machine and begun tests.... It is not working yet. Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com> * Big step ahead. Now I get to the point where the Auth Failed message appears The state machine seems ok Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com> * Able to retrieve authToken (#37) Working version retrieving the authToken, not able to send request afterwards, got response auth_attempt * Connection is working now after modifying minor things in session.py api=True by default removed cookie from the response handling Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com> * Clean up conversation comment and print statements Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com> * Started work implementing Niagara4 SCRAM Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com> * WIP I will need to access from another PC than the one running the Niagara 4 server... Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com> * file omitted Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com> * file omitted Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com> * WIP * Just beginning to implement SCRAM script by Pierre into pyhaystack... Modified state machine and begun tests.... It is not working yet. Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com> * Big step ahead. Now I get to the point where the Auth Failed message appears The state machine seems ok Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com> * Connection is working now after modifying minor things in session.py api=True by default removed cookie from the response handling Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com> * Able to retrieve authToken (#37) Working version retrieving the authToken, not able to send request afterwards, got response auth_attempt * Clean up conversation comment and print statements Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com> * Started work implementing Niagara4 SCRAM Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com> * WIP I will need to access from another PC than the one running the Niagara 4 server... Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com> * file omitted Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com> * file omitted Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com> * WIP * need rebase to benefit from headers lower case Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com> * Added note folder to ignore Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com> * We receive server first message Seems to fail with authentication We are close ! Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com> * final msg response is null.... why ? Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com> * Fighting with session cookies Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com> * modified how I set cookies... no improvements * Update niagaraV4.py (#43) Working version Niagara, able to have the remoteSignature checked with the server one. Will need to test with an other url afterwards to see if we are able to use the session to retrieve the data * Modifications to salt password added some print statements Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com> * Removed manual handling of cookies and let the session plays... * Niagara4 working We needed one more request to j_security_check, no parameter to authenticate. Now it's time to refactor, document and release !!!! Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com> * Small changes before push.... this file will be deleted Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com> * Cleanup, some comments. Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com> * Deleted scripts that helped implementing scram Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com> * Dropping Python 3.3 support Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com> * Pierre Sigwalt Special Commit Edition Added Pierre to the contributors file. He helped us with the implementation of SCRAM authentication for Skyspark and Niagara4. Nice work that was way over my skills. Thanks Pierre Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com> * First implementation of a browser that will help navigate the haystack server from pyhaystack.client.niagara import NiagaraHaystackSession from pyhaystack.browser.browser import Browser import logging logging.root.setLevel(logging.DEBUG) session = NiagaraHaystackSession(uri='http://ip', username='user', password='password') server = Browser(session) site = server['Site'] will return the site site['equipment'] will return an equipment Signed-off-by: Christian Tremblay, ing. <christian.tremblay@servisys.com> * Equipments and points are created Signed-off-by: Christian Tremblay, ing. <christian.tremblay@servisys.com> * New approach for high level interface Adding functions to mixins Signed-off-by: Christian Tremblay, ing. <christian.tremblay@servisys.com> * Adding helper functions to high level interface (__iter__, __getitem__, etc...) Signed-off-by: Christian Tremblay, ing. <christian.tremblay@servisys.com> * getitem will now return tags or equip (for site) or points (for equip) Some bugs are still there but WIP goes well Signed-off-by: Christian Tremblay, ing. <christian.tremblay@servisys.com> * Corrected point mixin so value works Added his shortcut to his_read_series (with a default to 'today' range ex. znt.his(rng='yesterday') or znt.his() filter requests works with __getitem__ ex. zone_sensors = my_office['sensor and zone'] Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com> * Small doc update Should be able to get site from session in a more direct way Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com> * Removed browser files as this is done using Mixin interfaces Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com> * Added a notice when connection fails in the first place, to use a certificate or tls_verify = False Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com> * Removed old folder Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com> * Clean up on high level functions code Merged with develop Added session.sites and session.site, shortcut to retrieve the site from a server Typical use : ``` import pyhaystack session = pyhaystack.connect('n4', uri, username, password, http_args={"tls_verify":False}) my_site = session.site my_site['UTA-1']['SF-O'] # UTA-1 is an equipment # SF-O is a point under UTA-1 my_site['UTA-1']['SF-O'].his().result # Will bring histories for this point ``` Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com> * Fix PBKDF2 ImportError on Debian Wheezy (#46) * Clean up whitespace changes. - Remove carriage returns - Remove trailing whitespace * client.ops.grid: Drop `print` We should *not* be printing stuff to `stdout` for several reasons: 1. who says the user is watching `stdout`? 2. who says the user isn't using `stdout` for something else? Randomly printing stuff has the potential to corrupt `stdout` output of any script that calls the grid operations, which is pretty much all scripts. We should not be polluting `stdout` with this data. * util.scram: Add hashbang, encoding to file. * util.scram: Trim trailing blank lines. * util.scram: Split excessively long lines. * util.scram: Use backports.pbkdf2 as fallback. If we can't load `pbkdf2` from `hashlib`, use the backports version. * Disable SubjectAltNameWarning Added more precise exception handling for SSLError in sync.py This modification allow us to remove print statement in niagara_scram.py Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com> * Modified README.rst to add new implementations (Niagara4 and Skyspark 3) * replaced main README.rst by new one * Error in nav arguments.....must be navId, not nav_id Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com> * Update to README.rst to remove "new" implementation as the implementation is now established. Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com> * Added documentation around the square bracket search for site, equip, points. Some cleanup probably needed. Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com> * docs: Clean up line endings, whitespace, etc. * docs: connect.rst: More detailed connection information. Detail the various options and their meanings. Most options are presented here for completeness. * docs: connect.rst: Use `pyhaystack.connect` This is an alias for `get_instance`… but Sphinx doesn't pick up on that fact, and so I mention both there. * docs: connect.rst: Move "from file" hint to Usage hints. * docs: site.rst: Describe operations, high-level interface. * Updated version ready to push Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com> * docs: tags.rst: Document tag CRUD operations. Some servers, notably WideSky, support making changes to the asset model through the API, and this is supported by pyhaystack. This details how to retrieve and how to update tags on an entity. * docs: site.rst: Fix formatting, tweak wording * docs: his.rst: Document timezones, range handling, etc. * docs: site.rst: Fix reference to `TzMixin` * docs: site.rst: Fix references, formatting * docs: site.rst: Further formatting tweaks * docs: connect.rst: Fix formatting * docs: tags.rst: Formatting fixes, example of `is_dirty` * docs: tags.rst: Fix broken references * pyhaystack.info: Drop out middle initial. Usually I just go by first/last name, the world doesn't need to know my middle name is James. :-) * docs: tags.rst: Fix more formatting issues Keep forgetting that newline between the `::` and the start of the block. * More details on ways to deal with CA certificates when connecting to a server Signed-off-by: Christian Tremblay, ing. <christian.tremblay@servisys.com> * Minor corrections to comments into code Removed references to LGPL and replaced by Apache 2.0 Renamed Stuart in contributors.txt Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com>
ChristianTremblay
added a commit
that referenced
this pull request
Sep 2, 2017
* First implementation of a browser that will help navigate the haystack server from pyhaystack.client.niagara import NiagaraHaystackSession from pyhaystack.browser.browser import Browser import logging logging.root.setLevel(logging.DEBUG) session = NiagaraHaystackSession(uri='http://ip', username='user', password='password') server = Browser(session) site = server['Site'] will return the site site['equipment'] will return an equipment Signed-off-by: Christian Tremblay, ing. <christian.tremblay@servisys.com> * Equipments and points are created Signed-off-by: Christian Tremblay, ing. <christian.tremblay@servisys.com> * New approach for high level interface Adding functions to mixins Signed-off-by: Christian Tremblay, ing. <christian.tremblay@servisys.com> * Adding helper functions to high level interface (__iter__, __getitem__, etc...) Signed-off-by: Christian Tremblay, ing. <christian.tremblay@servisys.com> * client.http.base: Send Content-Length as a string. `requests` tries to do a regex match or similar on the values of headers, and this requires the input to be a string. It barfs when you give it an `int`. * client.ops.vendor.widesky: Fix typo in create entity operation. * client.ops.entity: Add loggers, handle empty grid. * client.ops.his: Handle write of empty series. * client.ops.his: Handle no columns in write_his_frame. * WC-241: revive multi-hist mixin. * WC-241: initialize timestamp column for write grid * WC-241: Added a simple application that uses the pyhaystack library Edit by Stuart Longland <stuartl@vrt.com.au>: Removal of default credentials! * WC-237: Re-locate manual mutli-his-read-write test script. Manually run test scripts can go under tests/manual, until such time as they get automated. * WC-237: client.session: Add storage for grid cache. Some grids are unlikely to change in the course of a session. Examples of these are the 'about', 'formats' and 'ops' grids, which indicate the features that the Haystack server supports. We should cache these for a limited period of time so that the results can be used for enabling features depending on server version. * WC-237: client.ops.grid: Implement cache support * WC-237: client.session: Enable caching for about, ops, formats. * WC-237: client.mixins: Remove defunct classes from __init__ * WC-237: client.session: Fix typos * WC-237: client.session: Add "has_feature" operation. Used to detect extension features in the Haystack server. Some servers implement extension featurs that the standard Haystack API makes difficult to expose. Sometimes a feature is a "new" feature in a particular release of a server, as such, it should be disabled if the version is found to be below a certain threshold. (e.g. multi-point hisRead/hisWrite is supported in WideSky v0.5.0 but not earlier) This allows a run-time check to take place in the relevant state machines to decide how to tackle the problem. * WC-237: client.ops.feature: Add HasFeatureOperation base implementation. * WC-237: client.ops.feature: Fix some state machine bugs, typos. * WC-237: client.session: Use base class for HasFeaturesOperation * WC-237: client.ops.vendor.widesky: Add detection of WideSky extensions * WC-237: client.widesky: Enable detection of WideSky extensions * WC-237: client.ops.his: Probe multi-point hisRead * WC-237: client.ops.his: Implement probing of multi-hisWrite * WC-237: Pull in semver for WideSky version checking * WC-237: client.ops.his: Fix multi-point hisRead/hisWrite issues. * client.http.dummy: Dummy HTTP Client/Server This is a "HTTP Server" class and matching "Client" that we can drive like a puppet for writing test cases. * tests: Bring test case for 'about' method. * client.http.base: Handle 'str' input on Python3. If we're given an 'str', we might safely assume that the body is in fact decoded already. * tests.util: Add some utility functions * tests.client.test_base: Check output of about() * tests.client.test_base: Add ops test * tests.client.test_base: Add formats() test * tests.client.test_base: More tests for read by ID, etc. * getitem will now return tags or equip (for site) or points (for equip) Some bugs are still there but WIP goes well Signed-off-by: Christian Tremblay, ing. <christian.tremblay@servisys.com> * Corrected point mixin so value works Added his shortcut to his_read_series (with a default to 'today' range ex. znt.his(rng='yesterday') or znt.his() filter requests works with __getitem__ ex. zone_sensors = my_office['sensor and zone'] Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com> * Bugfix: Handle missing entities in multi-entity read requests (#26) * WC-366: client.ops.entity: Handle missing entities in read response. If an entity does not exist, and we are requesting more than one, the missing entity appears in the grid as a row of nulls: DEBUG:pyhaystack.client.WideskyHaystackSession.get_entity:Received grid: <Grid> Columns: id dis status user Row 0: id=Ref(u'test_user4', u'test_user4', True), dis=u'Test User4', status=u'inactive', user=MARKER Row 1: id=None, dis=None, status=None, user=None Row 2: id=Ref(u'test_user2', u'test_user2', True), dis=u'Test User2', status=u'inactive', user=MARKER Row 3: id=Ref(u'test_user3', u'test_user3', True), dis=u'Test User3', status=u'inactive', user=MARKER </Grid> which then results in this exception: File "pyhaystack/client/ops/entity.py", line 51, in _on_read entity_id = row.pop('id').name # Should be a Ref AttributeError: 'NoneType' object has no attribute 'name' We should skip rows that have a null 'id'. * WC-366: client entity tests: Test retrieval of entities by id. * WC-366: client entity tests: Python 3 support * client.ops.grid: Handle expired cache (#35) If the cache has expired, then we can wind up with the situation where `op` is `None`, so the `if` block falls through to the `else` branch erroniously. This causes a stack trace like the following: ``` 2017-02-17 01:49:00,039 pyhaystack.client.WideskyHaystackSession.his_write_frame[his.py: 726] WARNING Unable to probe multi-his-write support Traceback (most recent call last): File "/usr/lib/python2.7/dist-packages/pyhaystack/client/ops/his.py", line 723, in _on_probe_multi result = operation.result File "/usr/lib/python2.7/dist-packages/pyhaystack/util/state.py", line 97, in result self._result.reraise() File "/usr/lib/python2.7/dist-packages/pyhaystack/util/asyncexc.py", line 28, in reraise reraise(*self._exc_info) File "/usr/lib/python2.7/dist-packages/pyhaystack/client/ops/feature.py", line 93, in _on_got_about self._about = operation.result File "/usr/lib/python2.7/dist-packages/pyhaystack/util/state.py", line 97, in result self._result.reraise() File "/usr/lib/python2.7/dist-packages/pyhaystack/util/asyncexc.py", line 28, in reraise reraise(*self._exc_info) File "/usr/lib/python2.7/dist-packages/pyhaystack/client/ops/grid.py", line 118, in _check_auth self._state_machine.auth_ok() File "/usr/lib/python2.7/dist-packages/fysom/__init__.py", line 297, in fn self.transition() File "/usr/lib/python2.7/dist-packages/fysom/__init__.py", line 290, in _tran self._enter_state(e) File "/usr/lib/python2.7/dist-packages/fysom/__init__.py", line 341, in _enter_state return getattr(self, fnname)(e) File "/usr/lib/python2.7/dist-packages/fysom/__init__.py", line 88, in _callback return func(obj, *args, **kwargs) File "/usr/lib/python2.7/dist-packages/pyhaystack/client/ops/grid.py", line 177, in _do_check_cache op.done_sig.connect(_proxy) AttributeError: 'NoneType' object has no attribute 'done_sig' ``` We need to instead do our check while we have the mutex locked and disregard the expired cache. * skyspark V3 SCRAM authentication (#36) First Raw of Skyspark V3 SCRAM authentication * Proposed updates for release 0.92 (#41) * WC-414: Ensure HTTP header names are US-ASCII encoded. python-requests does not like it when it sees Unicode here. * WC-414: client.ops.grid: Ensure grid is encoded For now, assume UTF-8 encoding since that seems to be the standard being used elsewhere. * WC-414: client.ops.vendor.widesky: Fix encoding issues. Ensure all headers are encoded using US-ASCII encoding. Ensure all body data is encoded using UTF-8 encoding. We'll treat passwords as being UTF-8. * WC-414: client.widesky: Send authorization in US-ASCII encoding. The authorization header is basically a hash, so US-ASCII is the most appropriate for this. * WC-414: client.widesky: Assume already 'unicode' if no 'decode'. If a string object has no 'decode' method, then assume it is a Unicode object already. * WC-414: client.http.base: Use bytes object not str/unicode * WC-414: client.ops.grid: Use bytes object not str/unicode * WC-414: base client tests: Fix encoding issues - Headers should be bytes() objects - Bodies should be decoded from UTF-8 * WC-414: client entity tests: Fix more encoding issues. * FIR-214: client.http.base: Assert that we have a content type. * FIR-214: client.http.base: Add accept_status argument. This gives us the ability to whitelist specific status codes, instead of failing normally "bad" codes such as 400, 401, 403 or 404, allowing the underlying client code to handle it. * FIR-214: client.http.dummy: Add accept_status logic. * FIR-214: client.http.sync: Add accept_status logic. * FIR-214: client.ops.grid: Pass through accept_status * FIR-214: client.ops.grid: Add hook for detecting session issues. Firefly will report error 401 if the session is invalid. * FIR-214: client.widesky: Add hook to detect 401 error. If we have a 401 error message thrown at us, assume we've been logged out. This could be delivered by way of a response or an exception. We should not raise an exception ourselves, but instead just invalidate the session. * FIR-214: client: _on_http_response → _on_http_grid_response. To imply that it's not just *any* HTTP response this gets called on, but rather specific responses. * FIR-214: client.widesky tests: Add test for is_logged_in. * FIR-214: client.widesky tests: Test 401 drops the session. * FIR-214: client.session tests: Check _on_http_grid_response gets called. Add a test that verifies that the function *does* get called when a grid is requested. * FIR-214: client.widesky tests: Test other exceptions are ignored. * client.ops.his: Don't define MetaSeries/MetaDataFrame unless pandas loaded Otherwise you get stack traces like this when pandas isn't installed: error 06-Sep-2016 14:45:22 ====================================================================== error 06-Sep-2016 14:45:22 ERROR: Attempt to add a single user error 06-Sep-2016 14:45:22 ---------------------------------------------------------------------- error 06-Sep-2016 14:45:22 Traceback (most recent call last): error 06-Sep-2016 14:45:22 File "/usr/lib/python2.7/dist-packages/nose/case.py", line 197, in runTest error 06-Sep-2016 14:45:22 self.test(*self.arg) error 06-Sep-2016 14:45:22 File "/mnt/src/tests/test_user_editor_add.py", line 58, in test_add_single_user error 06-Sep-2016 14:45:22 core.main(arguments) error 06-Sep-2016 14:45:22 File "/mnt/src/widesky_editor/user/core.py", line 35, in main error 06-Sep-2016 14:45:22 session = init_haystack(args) error 06-Sep-2016 14:45:22 File "/mnt/src/widesky_editor/common.py", line 67, in init_haystack error 06-Sep-2016 14:45:22 http_args=http_args, tagging_model=WideSkyTaggingModel) error 06-Sep-2016 14:45:22 File "/usr/lib/python2.7/dist-packages/pyhaystack/client/loader.py", line 61, in get_instance error 06-Sep-2016 14:45:22 implementation = get_implementation(implementation) error 06-Sep-2016 14:45:22 File "/usr/lib/python2.7/dist-packages/pyhaystack/client/loader.py", line 34, in get_implementation error 06-Sep-2016 14:45:22 package='pyhaystack.client') error 06-Sep-2016 14:45:22 File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module error 06-Sep-2016 14:45:22 __import__(name) error 06-Sep-2016 14:45:22 File "/usr/lib/python2.7/dist-packages/pyhaystack/client/widesky.py", line 8, in <module> error 06-Sep-2016 14:45:22 from .session import HaystackSession error 06-Sep-2016 14:45:22 File "/usr/lib/python2.7/dist-packages/pyhaystack/client/session.py", line 18, in <module> error 06-Sep-2016 14:45:22 from .ops import his as his_ops error 06-Sep-2016 14:45:22 File "/usr/lib/python2.7/dist-packages/pyhaystack/client/ops/his.py", line 801, in <module> error 06-Sep-2016 14:45:22 class MetaSeries(Series): error 06-Sep-2016 14:45:22 NameError: name 'Series' is not defined * WC-9: client.ops.vendor.widesky: Version detection tweaks - Handle non-semver version string - Prefer moduleVersion over productVersion; newer versions will use this over productVersion which will be the "WideSky release". (like Ubuntu's; 16.04.) * VRT-506: client.http.base: Make headers and cookies case-insensitive. * VRT-506: client.http.base: Fix `bytes`/`str` issue, initialisation. * client.loader: Add support for aliases for known implementations. (#44) This allows us to whistle up aliases for implementations such as `ax` for Niagara AX, etc. Further implementations can be added to this file, and they will be loaded when needed. * Added support for easier session declaration using get_instance we can now use : ``` import pyhaystack pyhaystack.connect('ax', 'uri', 'username', 'password') #or import pyhaystack pyhaystack.connect('ax', uri='uri', username='username', password='password') ``` Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com> * Feature/scram (#45) * Just beginning to implement SCRAM script by Pierre into pyhaystack... Modified state machine and begun tests.... It is not working yet. Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com> * Big step ahead. Now I get to the point where the Auth Failed message appears The state machine seems ok Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com> * Able to retrieve authToken (#37) Working version retrieving the authToken, not able to send request afterwards, got response auth_attempt * Connection is working now after modifying minor things in session.py api=True by default removed cookie from the response handling Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com> * Clean up conversation comment and print statements Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com> * Started work implementing Niagara4 SCRAM Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com> * WIP I will need to access from another PC than the one running the Niagara 4 server... Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com> * file omitted Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com> * file omitted Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com> * WIP * Just beginning to implement SCRAM script by Pierre into pyhaystack... Modified state machine and begun tests.... It is not working yet. Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com> * Big step ahead. Now I get to the point where the Auth Failed message appears The state machine seems ok Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com> * Connection is working now after modifying minor things in session.py api=True by default removed cookie from the response handling Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com> * Able to retrieve authToken (#37) Working version retrieving the authToken, not able to send request afterwards, got response auth_attempt * Clean up conversation comment and print statements Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com> * Started work implementing Niagara4 SCRAM Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com> * WIP I will need to access from another PC than the one running the Niagara 4 server... Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com> * file omitted Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com> * file omitted Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com> * WIP * need rebase to benefit from headers lower case Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com> * Added note folder to ignore Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com> * We receive server first message Seems to fail with authentication We are close ! Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com> * final msg response is null.... why ? Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com> * Fighting with session cookies Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com> * modified how I set cookies... no improvements * Update niagaraV4.py (#43) Working version Niagara, able to have the remoteSignature checked with the server one. Will need to test with an other url afterwards to see if we are able to use the session to retrieve the data * Modifications to salt password added some print statements Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com> * Removed manual handling of cookies and let the session plays... * Niagara4 working We needed one more request to j_security_check, no parameter to authenticate. Now it's time to refactor, document and release !!!! Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com> * Small changes before push.... this file will be deleted Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com> * Cleanup, some comments. Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com> * Deleted scripts that helped implementing scram Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com> * Dropping Python 3.3 support Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com> * Pierre Sigwalt Special Commit Edition Added Pierre to the contributors file. He helped us with the implementation of SCRAM authentication for Skyspark and Niagara4. Nice work that was way over my skills. Thanks Pierre Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com> * First implementation of a browser that will help navigate the haystack server from pyhaystack.client.niagara import NiagaraHaystackSession from pyhaystack.browser.browser import Browser import logging logging.root.setLevel(logging.DEBUG) session = NiagaraHaystackSession(uri='http://ip', username='user', password='password') server = Browser(session) site = server['Site'] will return the site site['equipment'] will return an equipment Signed-off-by: Christian Tremblay, ing. <christian.tremblay@servisys.com> * Equipments and points are created Signed-off-by: Christian Tremblay, ing. <christian.tremblay@servisys.com> * New approach for high level interface Adding functions to mixins Signed-off-by: Christian Tremblay, ing. <christian.tremblay@servisys.com> * Adding helper functions to high level interface (__iter__, __getitem__, etc...) Signed-off-by: Christian Tremblay, ing. <christian.tremblay@servisys.com> * getitem will now return tags or equip (for site) or points (for equip) Some bugs are still there but WIP goes well Signed-off-by: Christian Tremblay, ing. <christian.tremblay@servisys.com> * Corrected point mixin so value works Added his shortcut to his_read_series (with a default to 'today' range ex. znt.his(rng='yesterday') or znt.his() filter requests works with __getitem__ ex. zone_sensors = my_office['sensor and zone'] Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com> * Small doc update Should be able to get site from session in a more direct way Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com> * Removed browser files as this is done using Mixin interfaces Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com> * Added a notice when connection fails in the first place, to use a certificate or tls_verify = False Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com> * Removed old folder Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com> * Clean up on high level functions code Merged with develop Added session.sites and session.site, shortcut to retrieve the site from a server Typical use : ``` import pyhaystack session = pyhaystack.connect('n4', uri, username, password, http_args={"tls_verify":False}) my_site = session.site my_site['UTA-1']['SF-O'] # UTA-1 is an equipment # SF-O is a point under UTA-1 my_site['UTA-1']['SF-O'].his().result # Will bring histories for this point ``` Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com> * Fix PBKDF2 ImportError on Debian Wheezy (#46) * Clean up whitespace changes. - Remove carriage returns - Remove trailing whitespace * client.ops.grid: Drop `print` We should *not* be printing stuff to `stdout` for several reasons: 1. who says the user is watching `stdout`? 2. who says the user isn't using `stdout` for something else? Randomly printing stuff has the potential to corrupt `stdout` output of any script that calls the grid operations, which is pretty much all scripts. We should not be polluting `stdout` with this data. * util.scram: Add hashbang, encoding to file. * util.scram: Trim trailing blank lines. * util.scram: Split excessively long lines. * util.scram: Use backports.pbkdf2 as fallback. If we can't load `pbkdf2` from `hashlib`, use the backports version. * Disable SubjectAltNameWarning Added more precise exception handling for SSLError in sync.py This modification allow us to remove print statement in niagara_scram.py Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com> * Modified README.rst to add new implementations (Niagara4 and Skyspark 3) * replaced main README.rst by new one * Error in nav arguments.....must be navId, not nav_id Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com> * Update to README.rst to remove "new" implementation as the implementation is now established. Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com> * Added documentation around the square bracket search for site, equip, points. Some cleanup probably needed. Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com> * docs: Clean up line endings, whitespace, etc. * docs: connect.rst: More detailed connection information. Detail the various options and their meanings. Most options are presented here for completeness. * docs: connect.rst: Use `pyhaystack.connect` This is an alias for `get_instance`… but Sphinx doesn't pick up on that fact, and so I mention both there. * docs: connect.rst: Move "from file" hint to Usage hints. * docs: site.rst: Describe operations, high-level interface. * Modification to doc related to pyhaystack installation in some environment This is related to issue #50 Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com> * Minor fix to support nhaystack compiled for Niagara AX 3.7 Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com> * Added beta support for eval ops for skyspark client Corrected typo in Widesky doc Incremented version number * Corrected error in function Added mixins to skyspark2 added packages in setup.py incremented version Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com> * removed slash from request issue #53 Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com> * Reverted error when merging Corrected info Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com> * Added certifi to requirements. SSL is widely used now Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com>
ChristianTremblay
added a commit
that referenced
this pull request
Sep 2, 2017
* First implementation of a browser that will help navigate the haystack server from pyhaystack.client.niagara import NiagaraHaystackSession from pyhaystack.browser.browser import Browser import logging logging.root.setLevel(logging.DEBUG) session = NiagaraHaystackSession(uri='http://ip', username='user', password='password') server = Browser(session) site = server['Site'] will return the site site['equipment'] will return an equipment Signed-off-by: Christian Tremblay, ing. <christian.tremblay@servisys.com> * Equipments and points are created Signed-off-by: Christian Tremblay, ing. <christian.tremblay@servisys.com> * New approach for high level interface Adding functions to mixins Signed-off-by: Christian Tremblay, ing. <christian.tremblay@servisys.com> * Adding helper functions to high level interface (__iter__, __getitem__, etc...) Signed-off-by: Christian Tremblay, ing. <christian.tremblay@servisys.com> * client.http.base: Send Content-Length as a string. `requests` tries to do a regex match or similar on the values of headers, and this requires the input to be a string. It barfs when you give it an `int`. * client.ops.vendor.widesky: Fix typo in create entity operation. * client.ops.entity: Add loggers, handle empty grid. * client.ops.his: Handle write of empty series. * client.ops.his: Handle no columns in write_his_frame. * WC-241: revive multi-hist mixin. * WC-241: initialize timestamp column for write grid * WC-241: Added a simple application that uses the pyhaystack library Edit by Stuart Longland <stuartl@vrt.com.au>: Removal of default credentials! * WC-237: Re-locate manual mutli-his-read-write test script. Manually run test scripts can go under tests/manual, until such time as they get automated. * WC-237: client.session: Add storage for grid cache. Some grids are unlikely to change in the course of a session. Examples of these are the 'about', 'formats' and 'ops' grids, which indicate the features that the Haystack server supports. We should cache these for a limited period of time so that the results can be used for enabling features depending on server version. * WC-237: client.ops.grid: Implement cache support * WC-237: client.session: Enable caching for about, ops, formats. * WC-237: client.mixins: Remove defunct classes from __init__ * WC-237: client.session: Fix typos * WC-237: client.session: Add "has_feature" operation. Used to detect extension features in the Haystack server. Some servers implement extension featurs that the standard Haystack API makes difficult to expose. Sometimes a feature is a "new" feature in a particular release of a server, as such, it should be disabled if the version is found to be below a certain threshold. (e.g. multi-point hisRead/hisWrite is supported in WideSky v0.5.0 but not earlier) This allows a run-time check to take place in the relevant state machines to decide how to tackle the problem. * WC-237: client.ops.feature: Add HasFeatureOperation base implementation. * WC-237: client.ops.feature: Fix some state machine bugs, typos. * WC-237: client.session: Use base class for HasFeaturesOperation * WC-237: client.ops.vendor.widesky: Add detection of WideSky extensions * WC-237: client.widesky: Enable detection of WideSky extensions * WC-237: client.ops.his: Probe multi-point hisRead * WC-237: client.ops.his: Implement probing of multi-hisWrite * WC-237: Pull in semver for WideSky version checking * WC-237: client.ops.his: Fix multi-point hisRead/hisWrite issues. * client.http.dummy: Dummy HTTP Client/Server This is a "HTTP Server" class and matching "Client" that we can drive like a puppet for writing test cases. * tests: Bring test case for 'about' method. * client.http.base: Handle 'str' input on Python3. If we're given an 'str', we might safely assume that the body is in fact decoded already. * tests.util: Add some utility functions * tests.client.test_base: Check output of about() * tests.client.test_base: Add ops test * tests.client.test_base: Add formats() test * tests.client.test_base: More tests for read by ID, etc. * getitem will now return tags or equip (for site) or points (for equip) Some bugs are still there but WIP goes well Signed-off-by: Christian Tremblay, ing. <christian.tremblay@servisys.com> * Corrected point mixin so value works Added his shortcut to his_read_series (with a default to 'today' range ex. znt.his(rng='yesterday') or znt.his() filter requests works with __getitem__ ex. zone_sensors = my_office['sensor and zone'] Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com> * Bugfix: Handle missing entities in multi-entity read requests (#26) * WC-366: client.ops.entity: Handle missing entities in read response. If an entity does not exist, and we are requesting more than one, the missing entity appears in the grid as a row of nulls: DEBUG:pyhaystack.client.WideskyHaystackSession.get_entity:Received grid: <Grid> Columns: id dis status user Row 0: id=Ref(u'test_user4', u'test_user4', True), dis=u'Test User4', status=u'inactive', user=MARKER Row 1: id=None, dis=None, status=None, user=None Row 2: id=Ref(u'test_user2', u'test_user2', True), dis=u'Test User2', status=u'inactive', user=MARKER Row 3: id=Ref(u'test_user3', u'test_user3', True), dis=u'Test User3', status=u'inactive', user=MARKER </Grid> which then results in this exception: File "pyhaystack/client/ops/entity.py", line 51, in _on_read entity_id = row.pop('id').name # Should be a Ref AttributeError: 'NoneType' object has no attribute 'name' We should skip rows that have a null 'id'. * WC-366: client entity tests: Test retrieval of entities by id. * WC-366: client entity tests: Python 3 support * client.ops.grid: Handle expired cache (#35) If the cache has expired, then we can wind up with the situation where `op` is `None`, so the `if` block falls through to the `else` branch erroniously. This causes a stack trace like the following: ``` 2017-02-17 01:49:00,039 pyhaystack.client.WideskyHaystackSession.his_write_frame[his.py: 726] WARNING Unable to probe multi-his-write support Traceback (most recent call last): File "/usr/lib/python2.7/dist-packages/pyhaystack/client/ops/his.py", line 723, in _on_probe_multi result = operation.result File "/usr/lib/python2.7/dist-packages/pyhaystack/util/state.py", line 97, in result self._result.reraise() File "/usr/lib/python2.7/dist-packages/pyhaystack/util/asyncexc.py", line 28, in reraise reraise(*self._exc_info) File "/usr/lib/python2.7/dist-packages/pyhaystack/client/ops/feature.py", line 93, in _on_got_about self._about = operation.result File "/usr/lib/python2.7/dist-packages/pyhaystack/util/state.py", line 97, in result self._result.reraise() File "/usr/lib/python2.7/dist-packages/pyhaystack/util/asyncexc.py", line 28, in reraise reraise(*self._exc_info) File "/usr/lib/python2.7/dist-packages/pyhaystack/client/ops/grid.py", line 118, in _check_auth self._state_machine.auth_ok() File "/usr/lib/python2.7/dist-packages/fysom/__init__.py", line 297, in fn self.transition() File "/usr/lib/python2.7/dist-packages/fysom/__init__.py", line 290, in _tran self._enter_state(e) File "/usr/lib/python2.7/dist-packages/fysom/__init__.py", line 341, in _enter_state return getattr(self, fnname)(e) File "/usr/lib/python2.7/dist-packages/fysom/__init__.py", line 88, in _callback return func(obj, *args, **kwargs) File "/usr/lib/python2.7/dist-packages/pyhaystack/client/ops/grid.py", line 177, in _do_check_cache op.done_sig.connect(_proxy) AttributeError: 'NoneType' object has no attribute 'done_sig' ``` We need to instead do our check while we have the mutex locked and disregard the expired cache. * skyspark V3 SCRAM authentication (#36) First Raw of Skyspark V3 SCRAM authentication * Proposed updates for release 0.92 (#41) * WC-414: Ensure HTTP header names are US-ASCII encoded. python-requests does not like it when it sees Unicode here. * WC-414: client.ops.grid: Ensure grid is encoded For now, assume UTF-8 encoding since that seems to be the standard being used elsewhere. * WC-414: client.ops.vendor.widesky: Fix encoding issues. Ensure all headers are encoded using US-ASCII encoding. Ensure all body data is encoded using UTF-8 encoding. We'll treat passwords as being UTF-8. * WC-414: client.widesky: Send authorization in US-ASCII encoding. The authorization header is basically a hash, so US-ASCII is the most appropriate for this. * WC-414: client.widesky: Assume already 'unicode' if no 'decode'. If a string object has no 'decode' method, then assume it is a Unicode object already. * WC-414: client.http.base: Use bytes object not str/unicode * WC-414: client.ops.grid: Use bytes object not str/unicode * WC-414: base client tests: Fix encoding issues - Headers should be bytes() objects - Bodies should be decoded from UTF-8 * WC-414: client entity tests: Fix more encoding issues. * FIR-214: client.http.base: Assert that we have a content type. * FIR-214: client.http.base: Add accept_status argument. This gives us the ability to whitelist specific status codes, instead of failing normally "bad" codes such as 400, 401, 403 or 404, allowing the underlying client code to handle it. * FIR-214: client.http.dummy: Add accept_status logic. * FIR-214: client.http.sync: Add accept_status logic. * FIR-214: client.ops.grid: Pass through accept_status * FIR-214: client.ops.grid: Add hook for detecting session issues. Firefly will report error 401 if the session is invalid. * FIR-214: client.widesky: Add hook to detect 401 error. If we have a 401 error message thrown at us, assume we've been logged out. This could be delivered by way of a response or an exception. We should not raise an exception ourselves, but instead just invalidate the session. * FIR-214: client: _on_http_response → _on_http_grid_response. To imply that it's not just *any* HTTP response this gets called on, but rather specific responses. * FIR-214: client.widesky tests: Add test for is_logged_in. * FIR-214: client.widesky tests: Test 401 drops the session. * FIR-214: client.session tests: Check _on_http_grid_response gets called. Add a test that verifies that the function *does* get called when a grid is requested. * FIR-214: client.widesky tests: Test other exceptions are ignored. * client.ops.his: Don't define MetaSeries/MetaDataFrame unless pandas loaded Otherwise you get stack traces like this when pandas isn't installed: error 06-Sep-2016 14:45:22 ====================================================================== error 06-Sep-2016 14:45:22 ERROR: Attempt to add a single user error 06-Sep-2016 14:45:22 ---------------------------------------------------------------------- error 06-Sep-2016 14:45:22 Traceback (most recent call last): error 06-Sep-2016 14:45:22 File "/usr/lib/python2.7/dist-packages/nose/case.py", line 197, in runTest error 06-Sep-2016 14:45:22 self.test(*self.arg) error 06-Sep-2016 14:45:22 File "/mnt/src/tests/test_user_editor_add.py", line 58, in test_add_single_user error 06-Sep-2016 14:45:22 core.main(arguments) error 06-Sep-2016 14:45:22 File "/mnt/src/widesky_editor/user/core.py", line 35, in main error 06-Sep-2016 14:45:22 session = init_haystack(args) error 06-Sep-2016 14:45:22 File "/mnt/src/widesky_editor/common.py", line 67, in init_haystack error 06-Sep-2016 14:45:22 http_args=http_args, tagging_model=WideSkyTaggingModel) error 06-Sep-2016 14:45:22 File "/usr/lib/python2.7/dist-packages/pyhaystack/client/loader.py", line 61, in get_instance error 06-Sep-2016 14:45:22 implementation = get_implementation(implementation) error 06-Sep-2016 14:45:22 File "/usr/lib/python2.7/dist-packages/pyhaystack/client/loader.py", line 34, in get_implementation error 06-Sep-2016 14:45:22 package='pyhaystack.client') error 06-Sep-2016 14:45:22 File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module error 06-Sep-2016 14:45:22 __import__(name) error 06-Sep-2016 14:45:22 File "/usr/lib/python2.7/dist-packages/pyhaystack/client/widesky.py", line 8, in <module> error 06-Sep-2016 14:45:22 from .session import HaystackSession error 06-Sep-2016 14:45:22 File "/usr/lib/python2.7/dist-packages/pyhaystack/client/session.py", line 18, in <module> error 06-Sep-2016 14:45:22 from .ops import his as his_ops error 06-Sep-2016 14:45:22 File "/usr/lib/python2.7/dist-packages/pyhaystack/client/ops/his.py", line 801, in <module> error 06-Sep-2016 14:45:22 class MetaSeries(Series): error 06-Sep-2016 14:45:22 NameError: name 'Series' is not defined * WC-9: client.ops.vendor.widesky: Version detection tweaks - Handle non-semver version string - Prefer moduleVersion over productVersion; newer versions will use this over productVersion which will be the "WideSky release". (like Ubuntu's; 16.04.) * VRT-506: client.http.base: Make headers and cookies case-insensitive. * VRT-506: client.http.base: Fix `bytes`/`str` issue, initialisation. * client.loader: Add support for aliases for known implementations. (#44) This allows us to whistle up aliases for implementations such as `ax` for Niagara AX, etc. Further implementations can be added to this file, and they will be loaded when needed. * Added support for easier session declaration using get_instance we can now use : ``` import pyhaystack pyhaystack.connect('ax', 'uri', 'username', 'password') #or import pyhaystack pyhaystack.connect('ax', uri='uri', username='username', password='password') ``` Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com> * Feature/scram (#45) * Just beginning to implement SCRAM script by Pierre into pyhaystack... Modified state machine and begun tests.... It is not working yet. Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com> * Big step ahead. Now I get to the point where the Auth Failed message appears The state machine seems ok Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com> * Able to retrieve authToken (#37) Working version retrieving the authToken, not able to send request afterwards, got response auth_attempt * Connection is working now after modifying minor things in session.py api=True by default removed cookie from the response handling Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com> * Clean up conversation comment and print statements Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com> * Started work implementing Niagara4 SCRAM Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com> * WIP I will need to access from another PC than the one running the Niagara 4 server... Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com> * file omitted Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com> * file omitted Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com> * WIP * Just beginning to implement SCRAM script by Pierre into pyhaystack... Modified state machine and begun tests.... It is not working yet. Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com> * Big step ahead. Now I get to the point where the Auth Failed message appears The state machine seems ok Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com> * Connection is working now after modifying minor things in session.py api=True by default removed cookie from the response handling Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com> * Able to retrieve authToken (#37) Working version retrieving the authToken, not able to send request afterwards, got response auth_attempt * Clean up conversation comment and print statements Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com> * Started work implementing Niagara4 SCRAM Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com> * WIP I will need to access from another PC than the one running the Niagara 4 server... Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com> * file omitted Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com> * file omitted Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com> * WIP * need rebase to benefit from headers lower case Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com> * Added note folder to ignore Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com> * We receive server first message Seems to fail with authentication We are close ! Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com> * final msg response is null.... why ? Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com> * Fighting with session cookies Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com> * modified how I set cookies... no improvements * Update niagaraV4.py (#43) Working version Niagara, able to have the remoteSignature checked with the server one. Will need to test with an other url afterwards to see if we are able to use the session to retrieve the data * Modifications to salt password added some print statements Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com> * Removed manual handling of cookies and let the session plays... * Niagara4 working We needed one more request to j_security_check, no parameter to authenticate. Now it's time to refactor, document and release !!!! Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com> * Small changes before push.... this file will be deleted Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com> * Cleanup, some comments. Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com> * Deleted scripts that helped implementing scram Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com> * Dropping Python 3.3 support Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com> * Pierre Sigwalt Special Commit Edition Added Pierre to the contributors file. He helped us with the implementation of SCRAM authentication for Skyspark and Niagara4. Nice work that was way over my skills. Thanks Pierre Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com> * First implementation of a browser that will help navigate the haystack server from pyhaystack.client.niagara import NiagaraHaystackSession from pyhaystack.browser.browser import Browser import logging logging.root.setLevel(logging.DEBUG) session = NiagaraHaystackSession(uri='http://ip', username='user', password='password') server = Browser(session) site = server['Site'] will return the site site['equipment'] will return an equipment Signed-off-by: Christian Tremblay, ing. <christian.tremblay@servisys.com> * Equipments and points are created Signed-off-by: Christian Tremblay, ing. <christian.tremblay@servisys.com> * New approach for high level interface Adding functions to mixins Signed-off-by: Christian Tremblay, ing. <christian.tremblay@servisys.com> * Adding helper functions to high level interface (__iter__, __getitem__, etc...) Signed-off-by: Christian Tremblay, ing. <christian.tremblay@servisys.com> * getitem will now return tags or equip (for site) or points (for equip) Some bugs are still there but WIP goes well Signed-off-by: Christian Tremblay, ing. <christian.tremblay@servisys.com> * Corrected point mixin so value works Added his shortcut to his_read_series (with a default to 'today' range ex. znt.his(rng='yesterday') or znt.his() filter requests works with __getitem__ ex. zone_sensors = my_office['sensor and zone'] Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com> * Small doc update Should be able to get site from session in a more direct way Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com> * Removed browser files as this is done using Mixin interfaces Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com> * Added a notice when connection fails in the first place, to use a certificate or tls_verify = False Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com> * Removed old folder Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com> * Clean up on high level functions code Merged with develop Added session.sites and session.site, shortcut to retrieve the site from a server Typical use : ``` import pyhaystack session = pyhaystack.connect('n4', uri, username, password, http_args={"tls_verify":False}) my_site = session.site my_site['UTA-1']['SF-O'] # UTA-1 is an equipment # SF-O is a point under UTA-1 my_site['UTA-1']['SF-O'].his().result # Will bring histories for this point ``` Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com> * Fix PBKDF2 ImportError on Debian Wheezy (#46) * Clean up whitespace changes. - Remove carriage returns - Remove trailing whitespace * client.ops.grid: Drop `print` We should *not* be printing stuff to `stdout` for several reasons: 1. who says the user is watching `stdout`? 2. who says the user isn't using `stdout` for something else? Randomly printing stuff has the potential to corrupt `stdout` output of any script that calls the grid operations, which is pretty much all scripts. We should not be polluting `stdout` with this data. * util.scram: Add hashbang, encoding to file. * util.scram: Trim trailing blank lines. * util.scram: Split excessively long lines. * util.scram: Use backports.pbkdf2 as fallback. If we can't load `pbkdf2` from `hashlib`, use the backports version. * Disable SubjectAltNameWarning Added more precise exception handling for SSLError in sync.py This modification allow us to remove print statement in niagara_scram.py Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com> * Modified README.rst to add new implementations (Niagara4 and Skyspark 3) * replaced main README.rst by new one * Error in nav arguments.....must be navId, not nav_id Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com> * Update to README.rst to remove "new" implementation as the implementation is now established. Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com> * Added documentation around the square bracket search for site, equip, points. Some cleanup probably needed. Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com> * docs: Clean up line endings, whitespace, etc. * docs: connect.rst: More detailed connection information. Detail the various options and their meanings. Most options are presented here for completeness. * docs: connect.rst: Use `pyhaystack.connect` This is an alias for `get_instance`… but Sphinx doesn't pick up on that fact, and so I mention both there. * docs: connect.rst: Move "from file" hint to Usage hints. * docs: site.rst: Describe operations, high-level interface. * Modification to doc related to pyhaystack installation in some environment This is related to issue #50 Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com> * Minor fix to support nhaystack compiled for Niagara AX 3.7 Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com> * Added beta support for eval ops for skyspark client Corrected typo in Widesky doc Incremented version number * Corrected error in function Added mixins to skyspark2 added packages in setup.py incremented version Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com> * removed slash from request issue #53 Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com> * Reverted error when merging Corrected info Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com> * Added certifi to requirements. SSL is widely used now Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Hi Christian,
Just bundled up all the changes that we've got in our fork that have been reviewed here and would like them considered for the next release.
WC-414:
bytesvsunicodehandling.We were getting error 500s in the API server when attempting to authenticate. The root cause turned out to be the
bcryptPython module generating password hashes that the server didn't like, but this was not before finding numerous issues in Python 2 vs 3 and strings. (a Python dev favourite!)The changes have been in production here since September last year, and so far haven't given issues in both Python 2.7 and 3.4.
FIR-214:
pyhaystackdoes not refresh expired token, causing failed writes in AMQP connectorWe use
pyhaystackto bridge between our house-built real-time data collection system (built on Python and using AMQP), and our WideSky system. We found that if the session token got revoked prematurely for any reason or if the token expired,pyhaystackwould fail to renew it, and we'd get gaps like this:https://vrt.on.widesky.cloud/dashboard/db/smartlight?from=1490512025765&to=1490717986116
The changes here allow "whitelisting" of response codes and the handling of the
401: Unauthorizedresponse to facilitate renewal of the token. (This may be of help with the SCRAM code in issue #38.)client.ops.his: Don't defineMetaSeries/MetaDataFrameunlesspandasloadedPandas is great if you're doing advanced numerical analysis, such as what
pyhaystackwas originally written for. But, it's a big library that drags innumpyand on Debian, that often pulls inmatplotliband a truck-load of X11-related packages as well.Much of the time we're dealing with headless devices (sometimes embedded ARM devices with no video hardware whatsoever) and with small amounts of storage. Every byte counts, so it's helpful to make such features optional.
WC-9: client.ops.vendor.widesky: Version detection tweaks
After discussion of where and how we'd encode the versions, we ended up making a breaking change. This moves the location of that check, and also puts some error handling in in case we get something we're not expecting.