[BEAM-5612] [BEAM-5315] Python 3 port gcs related io modules. #7650
Conversation
RobbeSneyders
force-pushed
the
gcp-io
branch
2 times, most recently
from
a3eb70f
to
5935d5b
Compare
|
R: @markflyhigh |
| lines.append(line) | ||
|
|
||
| contents = ''.join(lines) | ||
| contents = b''.join(lines) |
There was a problem hiding this comment.
Why converting to bytes works here? Can you explain a little more about the problem?
There was a problem hiding this comment.
os.urandom returns a bytestring. On Python 2, this worked using string, since a Python 2 string is actually a bytestring ('' = b''), while on Python 3 a string is actually a unicode string ('' = u''). By defining b'', we get the same behavior on both Python versions.
| @@ -512,7 +512,8 @@ def create_job(self, job): | |||
| if job_location: | |||
| gcs_or_local_path = os.path.dirname(job_location) | |||
| file_name = os.path.basename(job_location) | |||
| self.stage_file(gcs_or_local_path, file_name, io.BytesIO(job.json())) | |||
| self.stage_file(gcs_or_local_path, file_name, | |||
| io.BytesIO(job.json().encode('utf-8'))) | |||
There was a problem hiding this comment.
Pipeline works well to me without this change. Wondering what problem you want to fix here?
There was a problem hiding this comment.
io.BytesIO() expects bytes, while job.json() returns a string. Is this code path executed when running your pipeline?
There was a problem hiding this comment.
I don't think we hit this codepath before, the change makes sense to me.
This is is part of a series of PRs with goal to make Apache Beam PY3 compatible. The proposal with the outlined approach has been documented here: https://s.apache.org/beam-python-3.
This PR adds a py3-gcp test suite to tox and ports the gcs related io modules.
Post-Commit Tests Status (on master branch)