Merge pull request #22 from nickmaccarthy/issue_21

Issue 21 fix
nickmaccarthy · Oct 2, 2020 · be87139 · be87139
2 parents b715364 + 69a313e
commit be87139
Show file tree

Hide file tree

Showing 6 changed files with 117 additions and 46 deletions.
diff --git a/.gitignore b/.gitignore
@@ -1,5 +1,6 @@
 *.pyc
 env
+venv
 arrow_test.py
 dm_pretty.py
 ranges.py

diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -1,5 +1,8 @@
 # Changelog
 
+## 1.5.2 (2020-10-01)
+* [FIX] [Issue #21](https://github.com/nickmaccarthy/python-datemath/issues/21) - Fixed an issue where if timezone offset was in a datetime string (ISO8601), the timezone of the returned datemath object would be UTC and not the timezone as specified in the datetime string.
+
 ## 1.5.1 (2020-03-25)
 
 * [FIX] [Issue #15](https://github.com/nickmaccarthy/python-datemath/issues/15) - Fixed issue with parser finding invalid timeunits and throwing correct errors

diff --git a/README.md b/README.md
@@ -1,32 +1,37 @@
 [![Build Status](https://travis-ci.org/nickmaccarthy/python-datemath.svg?branch=master)](https://travis-ci.org/nickmaccarthy/python-datemath.svg?branch=master)
 
+# Python Datemath
+
+## What?
 
-# What?
 A date math (aka datemath) parser compatiable with the elasticsearch 'date math' format
 
-# Why?
-Working with date objects in python has always been interesting.  Having a background in php, I have been looking for quite some time ( no pun intended ) for a way to do date time interpolation similar to php's ```strtotime()``` function.  While the arrow module comes close, I needed something that could turn date math type strings into datetime objects for use in tattle.io and other projects I use in elasticsearch.  I have found even more uses for it, including AWS cloudwatch and various other projects and hopefully you will too.
+## Why?
+
+Working with date objects in python has always been interesting.  Having a background in php, I have been looking for quite some time ( no pun intended ) for a way to do date time interpolation similar to php's ```strtotime()``` function.  While the arrow module comes close, I needed something that could turn date math type strings into datetime objects for use in [tattle.io](http://tattle.io) and other projects I use in elasticsearch.  I have found even more uses for it, including AWS cloudwatch and various other projects and hopefully you will too.
+
+## What is date math?
 
-# What is date math ?
 Date Math is the short hand arithmetic to find relative time to fixed moments in date and time. Similar to the SOLR date math format, Elasticsearch has its own built in format for short hand date math and this module aims to support that same coverage in python.
 
 Documentation from elasticsearch:
-http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-date-format.html#date-math
+[http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-date-format.html#date-math](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-date-format.html#date-math)
 
 > The date type supports using date math expression when using it in a query/filter (mainly makes sense in range query/filter).
-
+>
 > The expression starts with an "anchor" date, which can be either now or a date string (in the applicable format) ending with `||`.
-
+>
 > It can then follow by a math expression, supporting `+`, `-` and `/` (rounding).
-
+>
 > The units supported are `y` (year), `M` (month), `w` (week), `d` (day), `h` (hour), `m` (minute), and `s` (second).
-
+>
 > Here are some samples: `now+1h`, `now+1h+1m`, `now+1h/d`, `2012-01-01||+1M/d`.
-
+>
 > Note, when doing range type searches, and the upper value is inclusive, the rounding will properly be rounded to the ceiling instead of flooring it.
 
-# Unit Maps
-```
+## Unit Maps
+
+```yaml
 y or Y      =   'year'
 M           =   'month'
 m           =   'minute'
@@ -36,13 +41,17 @@ h or H      =   'hour'
 s or S      =   'second'
 ```
 
-# Install
+## Install
+
 ```python
 pip install python-datemath
 ```
-# Examples
-Assuming our datetime is currently: '2016-01-01T00:00:00-00:00'
-```
+
+## Examples
+
+Assuming our datetime is currently: `2016-01-01T00:00:00-00:00`
+
+```yaml
 Expression:                 Result:
 now-1h                      2015-12-31T23:00:00+00:00
 now-1y                      2015-01-01T00:00:00+00:00
@@ -62,10 +71,11 @@ now/d                       2016-01-01T23:59:59+00:00
 now/Y                       2016-12-31T23:59:59+00:00
 ```
 
-# Usage
+## Usage
+
 By default datemath return an arrow date object representing your timestamp.  
 
-```
+```python
 >>> from datemath import dm
 >>>
 >>> dm('now+1h')
@@ -92,7 +102,8 @@ By default datemath return an arrow date object representing your timestamp.
 
 If you would rather have a string, you can use arrow's ```.format()``` method.
 > For for info on string formatting, check out arrows tokens section: http://crsmithdev.com/arrow/#tokens
-```
+
+```python
 >>> from datemath import dm
 >>>
 >>> src_timestamp = dm('2016-01-01')
@@ -105,11 +116,11 @@ If you would rather have a string, you can use arrow's ```.format()``` method.
 >>>
 >>> new_timestamp.format('YYYY.MM.DD')
 u'2015.12.18'
->>>
 ```
 
 Rather have a python datetime object instead? Just pass along the 'datetime' type
-```
+
+```python
 from datemath import dm
 >>> dm('now', type='datetime')
 datetime.datetime(2016, 1, 22, 22, 58, 28, 338060, tzinfo=tzutc())
@@ -119,7 +130,8 @@ datetime.datetime(2016, 1, 24, 22, 57, 45, 394470, tzinfo=tzutc())
 ```
 
 Or you can just import the `datemath` module, this will always give us a native `datetime` object
-```
+
+```python
 >>> from datemath import datemath
 >>>
 >>> datemath('2016-01-01T16:20:00||/d', roundDown=False)
@@ -129,20 +141,28 @@ datetime.datetime(2016, 1, 1, 23, 59, 59, 999999, tzinfo=tzutc())
 >>> # roundDown=True is default and implied
 >>> datemath('2016-01-01T16:20:00||/d')
 datetime.datetime(2016, 1, 1, 0, 0, tzinfo=tzutc())
->>>
 ```
 
 If you want a Epoch timestamp back instead, we can do that.  
+
 ```python
 >>> dm('now+2d-1m', type='timestamp')
 1453676321
 ```
 
-# What timezone are my objects in?
-By default all objects returned by datemath are in UTC.  If you want them them in a different timezone, just pass along the ```tz``` argument. 
-Timezone list can be found here: https://gist.github.com/pamelafox/986163
-```
-from datemath import dm 
+## What timezone are my objects in?
+
+By default all object returned by datemath are in UTC.  
+
+If you want them them back in a different timezone, just pass along the ```tz``` argument.  Timezone list can be found here: [https://gist.github.com/pamelafox/986163](https://gist.github.com/pamelafox/986163)
+
+If you provide a timezone offset in your timestring, datemath will return your time object as that timezone offset in the string.
+
+Note - currently timestrings with a timezone offset and the usage of the ```tz``` argument will result in the time object being returned with the timezone of what was in the timezone offset in the original string
+
+```python
+>>> from datemath import dm 
+>>>
 >>> dm('now')
 <Arrow [2016-01-26T01:00:53.601088+00:00]>
 >>>
@@ -154,9 +174,26 @@ from datemath import dm
 >>>
 >>> dm('2017-10-20 09:15:20', tz='US/Pacific')
 <Arrow [2017-10-20T09:15:20.000000-08:00]>
+>>> 
+>>> # Timestring with TZ offset in the string (ISO8601 format only)
+>>> dm('2016-01-01T00:00:00-05:00')
+<Arrow [2016-01-01T00:00:00-05:00]>
+>>>
+>>> # Timestring with TZ offset with datemath added (again, TS must be in ISO8601)
+>>> dm('2016-01-01T00:00:00-05:00||+2d+3h+5m')
+<Arrow [2016-01-03T03:05:00-05:00]>
+>>>
+>>> # Note, timestrings with TZ offsets will be returned as the timezone of the offset in the string even if the "tz" option is used. 
+>>> dm('2016-01-01T00:00:00-05:00', tz='US/Central')
+<Arrow [2016-01-01T00:00:00-05:00]>
 ```
 
+## Debugging
+
+If you would like more verbose output to debug the process of what datemath is doing, simply set `export DATEMATH_DEBUG=true` in your shell then run some datemath tests.  To stop debugging, run `unset DATEMATH_DEBUG`.
+
 ## Changes
+
 See CHANGELOG.md
 
 # Happy date math'ing!

diff --git a/VERSION.txt b/VERSION.txt
@@ -1 +1 @@
-1.5.1
+1.5.2
diff --git a/datemath/helpers.py b/datemath/helpers.py
@@ -42,9 +42,11 @@
 import re
 import os
 from dateutil import tz
+import dateutil
 import sys 
+from pprint import pprint
 
-debug = True if os.environ.get('DEBUG') == 'true' else False 
+debug = True if os.environ.get('DATEMATH_DEBUG') else False 
 
 class DateMathException(Exception):
     pass 
@@ -75,7 +77,7 @@ def unitMap(c):
 
 def as_datetime(expression, now, tz='UTC'):
     '''
-        returs our datemath expression as a python datetime object
+        returns our datemath expression as a python datetime object
         note: this has been deprecated and the 'type' argument in parse is the current way
     '''
     return parse(expression, now, tz)
@@ -89,26 +91,28 @@ def parse(expression, now=None, tz='UTC', type=None, roundDown=True):
         :param type - if we are dealing with a arrow or datetime object
         :param roundDown - wether or not we should round up or round down on this.  default is roundDown=True, which means if it was 12:00:00, `/d` would be '00:00:00', and with roundDown=False, `/d` would be '29:59:59'
     '''
+    if debug: print("parse() - starting for expression: {}".format(expression))
     if now is None:
+        if debug: print("parse() - Now is None, setting now to utcnow()")
         now = arrow.utcnow()
 
-    if debug: print("Orig Expression: {0}".format(expression))
+    if debug: print("parse() - Orig Expression: {0}".format(expression))
 
     math = ''
     time = ''
 
     if 'UTC' not in tz:
-        if debug: print("will now convert tz to {0}".format(tz))
+        if debug: print("parse() - will now convert tz to {0}".format(tz))
         now = now.to(tz)
 
     if expression == 'now':
-        if debug: print("Now, no dm: {0}".format(now))
+        if debug: print("parse() - Now, no dm: {0}".format(now))
         if type:
             return getattr(now, type)
         else:
             return now
     elif re.match('\d{10,}', str(expression)):
-        if debug: print('found an epoch timestamp')
+        if debug: print('parse() - found an epoch timestamp')
         if len(str(expression)) == 13:
             raise DateMathException('Unable to parse epoch timestamps in millis, please convert to the nearest second to continue - i.e. 1451610061 / 1000')
         ts = arrow.get(int(expression))
@@ -118,7 +122,7 @@ def parse(expression, now=None, tz='UTC', type=None, roundDown=True):
         ''' parse our standard "now+1d" kind of queries '''
         math = expression[3:]
         time = now
-        if debug: print('now expression: {0}'.format(now))
+        if debug: print('parse() - now expression: {0}'.format(now))
     else:
         ''' parse out datemath with date, ex "2015-10-20||+1d"  '''
         if '||' in expression:
@@ -137,24 +141,40 @@ def parse(expression, now=None, tz='UTC', type=None, roundDown=True):
 
     if not math or math == '':
         rettime = time
+
     rettime = evaluate(math, time, tz, roundDown)
 
     if type:
         return getattr(rettime, type)
     else:
         return rettime
 
-
 def parseTime(timestamp, timezone='UTC'):
     '''
-        parses a date/time stamp and returns and arrow object
+        parses a datetime string and returns and arrow object
     '''
     if timestamp and len(timestamp) >= 4: 
         ts = arrow.get(timestamp)
-        ts = ts.replace(tzinfo=timezone)
-        return ts
+        if debug: print("parseTime() - ts = {} :: vars :: {}".format(ts, vars(ts)))
+        if debug: print("parseTime() - ts timezone = {}".format(ts.tzinfo))
+        if debug: print("parseTime() - tzinfo type = {}".format(type(ts.tzinfo)))
+        if debug: print("parseTime() - timezone that came in = {}".format(timezone))
+
+        if ts.tzinfo:
+            import dateutil
+            if isinstance(ts.tzinfo, dateutil.tz.tz.tzoffset):
+            # this means our TZ probably came in via our datetime string
+            # then lets set our tz to whatever tzoffset is
+                ts = ts.replace(tzinfo=ts.tzinfo)
+            elif isinstance(ts.tzinfo, dateutil.tz.tz.tzutc):
+            # otherwise if we are utc, then lets just set it to be as such
+                ts = ts.replace(tzinfo=timezone)
+        else: 
+            # otherwise lets just ensure its set to whatever timezone came in
+            ts = ts.replace(tzinfo=timezone)
 
-
+        return ts
+
 def roundDate(now, unit, tz='UTC', roundDown=True):
     '''
         rounds our date object
@@ -163,7 +183,7 @@ def roundDate(now, unit, tz='UTC', roundDown=True):
         now = now.floor(unit)
     else:
         now = now.ceil(unit)
-    if debug: print("roundDate Now: {0}".format(now))
+    if debug: print("roundDate() Now: {0}".format(now))
     return now
 
 def calculate(now, offsetval, unit):
@@ -175,7 +195,7 @@ def calculate(now, offsetval, unit):
         offsetval = int(offsetval)
     try:
         now = now.shift(**{unit: offsetval})
-        if debug: print("Calculate called:  now: {}, offsetval: {}, offsetval-type: {}, unit: {}".format(now, offsetval, type(offsetval), unit))
+        if debug: print("calculate() called:  now: {}, offsetval: {}, offsetval-type: {}, unit: {}".format(now, offsetval, type(offsetval), unit))
         return now
     except Exception as e:
         raise DateMathException('Unable to calculate date: now: {0}, offsetvalue: {1}, unit: {2} - reason: {3}'.format(now,offsetval,unit,e))
@@ -184,8 +204,8 @@ def evaluate(expression, now, timeZone='UTC', roundDown=True):
     '''
         evaluates our datemath style expression
     '''
-    if debug: print('Expression: {0}'.format(expression))
-    if debug: print('Now: {0}'.format(now))
+    if debug: print('evaluate() - Expression: {0}'.format(expression))
+    if debug: print('evaluate() - Now: {0}'.format(now))
     val = 0
     i = 0
     while i < len(expression):
@@ -221,7 +241,7 @@ def evaluate(expression, now, timeZone='UTC', roundDown=True):
             raise DateMathException(''''{}' is not a valid timeunit for expression: '{}' '''.format(char, expression))
 
         i += 1
-    if debug: print("Fin: {0}".format(now))
+    if debug: print("evaluate() - Finished: {0}".format(now))
     if debug: print('\n\n')
     return now
 

diff --git a/tests.py b/tests.py
@@ -5,6 +5,7 @@
 import unittest2 as unittest
 import arrow
 from datetime import datetime as pydatetime
+from datetime import timedelta
 from datemath import dm, datemath
 from datemath.helpers import DateMathException as DateMathException
 from dateutil import tz
@@ -40,6 +41,15 @@ def testParse(self):
         self.assertEqual(dm('2016-01-01', tz='US/Eastern'), pydatetime(2016, 1, 1, tzinfo=tz.gettz('US/Eastern')))
         self.assertEqual(datemath('2016-01-01T01:00:00', tz='US/Central'), pydatetime(2016, 1, 1, 1, 0, 0, tzinfo=tz.gettz('US/Central')))
         self.assertEqual(datemath('2016-01-01T02:00:00', tz='US/Eastern'), pydatetime(2016, 1, 1, 2, tzinfo=tz.gettz('US/Eastern')))
+        # TZ offset inside of date string
+        self.assertEqual(datemath('2016-01-01T16:20:00.5+12:00'), pydatetime(2016, 1, 1, 16, 20, 0, 500000, tzinfo=tz.tzoffset(None, timedelta(hours=12))))
+        self.assertEqual(datemath('2016-01-01T16:20:00.5-05:00'), pydatetime(2016, 1, 1, 16, 20, 0, 500000, tzinfo=tz.tzoffset(None, timedelta(hours=-5))))
+        self.assertEqual(datemath('2016-01-01T16:20:00.5-00:00'), pydatetime(2016, 1, 1, 16, 20, 0, 500000, tzinfo=tz.tzoffset(None, timedelta(hours=0))))
+        # TZ offset inside of date string with datemath
+        self.assertEqual(datemath('2016-01-01T16:20:00.5+12:00||+1d'), pydatetime(2016, 1, 2, 16, 20, 0, 500000, tzinfo=tz.tzoffset(None, timedelta(hours=12))))
+        self.assertEqual(datemath('2016-01-01T16:20:00.6+12:00||+2d+1h'), pydatetime(2016, 1, 3, 17, 20, 0, 600000, tzinfo=tz.tzoffset(None, timedelta(hours=12))))
+        # If a TZ offset is in a datetime string, and there is a tz param used, the TZ offset will take precedence for the returned timeobj
+        self.assertEqual(datemath('2016-01-01T16:20:00.6+12:00||+2d+1h', tz='US/Eastern'), pydatetime(2016, 1, 3, 17, 20, 0, 600000, tzinfo=tz.tzoffset(None, timedelta(hours=12))))
 
         # relitive formats
         # addition