Formatting of datetime

[chrisb-c01]

I think this target is bothered by a similar issue as reported here for which a PR was written in singer-python.

My case:

• I have a custom tap for which I would like to target stitch.
• Although the setup.py for target-stitch states "singer-python==5.0.0", I've upgrade to "singer-python>=5.0.14", to prevent this issue.

However during the execution:

CRITICAL ('Unknown string format:', '4Y-04-12T13:39:19.593113Z')
Traceback (most recent call last):
  File "../target-stitch/venv/bin/target-stitch", line 11, in <module>
    load_entry_point('target-stitch==1.7.1', 'console_scripts', 'target-stitch')()
  File "/Users/chris/Code/singer/target-stitch/venv/lib/python3.6/site-packages/target_stitch-1.7.1-py3.6.egg/target_stitch/__init__.py", line 528, in main
  File "/Users/chris/Code/singer/target-stitch/venv/lib/python3.6/site-packages/target_stitch-1.7.1-py3.6.egg/target_stitch/__init__.py", line 514, in main
  File "/Users/chris/Code/singer/target-stitch/venv/lib/python3.6/site-packages/target_stitch-1.7.1-py3.6.egg/target_stitch/__init__.py", line 507, in main_impl
  File "/Users/chris/Code/singer/target-stitch/venv/lib/python3.6/site-packages/target_stitch-1.7.1-py3.6.egg/target_stitch/__init__.py", line 413, in consume
  File "/Users/chris/Code/singer/target-stitch/venv/lib/python3.6/site-packages/target_stitch-1.7.1-py3.6.egg/target_stitch/__init__.py", line 373, in handle_line
  File "/Users/chris/Code/singer/target-stitch/venv/lib/python3.6/site-packages/singer_python-5.0.14-py3.6.egg/singer/messages.py", line 183, in parse_message
    time_extracted = dateutil.parser.parse(time_extracted)
  File "/Users/chris/Code/singer/target-stitch/venv/lib/python3.6/site-packages/python_dateutil-2.7.2-py3.6.egg/dateutil/parser/_parser.py", line 1312, in parse
    return DEFAULTPARSER.parse(timestr, **kwargs)
  File "/Users/chris/Code/singer/target-stitch/venv/lib/python3.6/site-packages/python_dateutil-2.7.2-py3.6.egg/dateutil/parser/_parser.py", line 607, in parse
    raise ValueError("Unknown string format:", timestr)
ValueError: ('Unknown string format:', '4Y-04-12T13:39:19.593113Z')

Note: Target CSV works without issues since the PR was merged.

[KAllan357]

I think you are correct, we need to bump this target's dependency on singer-python based on the code here - https://github.com/singer-io/target-stitch/blob/master/target_stitch/__init__.py#L266-L267

But its attempting to parse a string of '4Y-04-12T13:39:19.593113Z' which indicates to me that there's something fishy going on in the tap side of things. What does the code look like on your tap's side where it builds the message and sends it to stdout?

[KAllan357]

After some more research it could also be this code: https://github.com/singer-io/singer-python/blob/master/singer/messages.py#L61-L63

[chrisb-c01]

Hi @KAllan357,

I've checked the output of the tap (dumped to a file) and checked the output of target-csv and all were fine.

In the custom tap I'm loggin as such:

def format_timestamp(data, typ, schema):
    result = data
    if typ == 'string' and schema.get('format') == 'date-time':
        rfc3339_ts = rfc3339_to_timestamp(data)
        utc_dt = datetime.datetime.utcfromtimestamp(rfc3339_ts).replace(tzinfo=pytz.UTC)
        result = utils.strftime(utc_dt)
    return result

def sync(resource):
    rows = http.request(
        resource, "GET", RESOURCES[resource]["url"]
    )

    with Transformer(pre_hook=format_timestamp) as transformer:
        for row in rows:
            transformed_row = transformer.transform(row, RESOURCES[resource]["schema"])
            singer.write_record(resource, transformed_row, time_extracted=utils.now())

[dmosorast]

@ChrisCalculus I've worked up a few PRs for cases in singer-python and target-stitch that should address the possible formatting issues in them. I won't be able to do a release until later tonight at the earliest, but in the meantime, you can check these out to see if they fix your issue!

singer-python: #71
target-stitch: #50

[chrisb-c01]

Hi @dmosorast thanks for your response.

I just pulled the new versions (stitch-target=1.7.2), but was only able to get things working when correcting 'singer-python==5.0.14' to 'singer-python>=5.0.15' in the setup.py of target-stitch so that it actually uses the revised singer-python. Just so you know.

Thanks again!

[dmosorast]

Thanks for the testing and feedback! @ChrisCalculus

It didn't seem like it at first, but I suppose the target code is implictly calling asdict on the RecordMessage object at some point in time. I'll bump the singer-python version and release a 1.7.3. Much appreciation. Closing this issue.