[protoc python] Relative import paths in generated python modules

[cherezov]

In order to avoid module name conflict with modules in PYTHON_PATH relative package import is introduced to generated python files.

This means that all imports like from Module1.Module2 import ProtoPackage will now looks like

from . import ProtoPackage
from .. import Module2

This will allow us to safety have another module with name 'Module1' and 'Module2' in the python path.

[grpc-kokoro]

Thanks for your pull request. The automated tests will run as soon as one of the admins verifies this change is ok for us to run on our infrastructure.

[grpc-kokoro]

Thanks for your pull request. The automated tests will run as soon as one of the admins verifies this change is ok for us to run on our infrastructure.

[googlebot]

Thanks for your pull request. It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please visit https://cla.developers.google.com/ to sign.

Once you've signed, please reply here (e.g. I signed it!) and we'll verify. Thanks.

---

• If you've already signed a CLA, it's possible we don't have your GitHub username or you're using a different email address. Check your existing CLA data and verify that your email is set on your git commits.
• If your company signed a CLA, they designated a Point of Contact who decides which employees are authorized to participate. You may need to contact the Point of Contact for your company and ask to be added to the group of authorized contributors. If you don't know who your Point of Contact is, direct the project maintainer to go/cla#troubleshoot.
• In order to pass this check, please resolve this problem and have the pull request author add another comment and the bot will run again.

[cherezov]

I signed it!

[bazel-io]

Can one of the admins verify this patch?

[googlebot]

CLAs look good, thanks!

[arthur-tacca]

(Note: I am not a Googler, or even a contributor) I think this is a dangerous pull request, because it masks mistakes in the use of protoc.

protoc creates directories containing *_pb2.py files to mirror the directories containing *.proto files. This is a bit confusing when you first see it, but is the Pythonic way to allow users to specify the packages that their *_pb2.py files should go in and import dependencies from, so long as they have correctly set the --proto_path= switch to protoc. For example, if you put all your proto files in a proto/ directory but use --proto_path=., you'll soon notice from proto import dependancy_pb2. This is a hint that you need to create an intermediate directory ./proto/package_name and pass the switch --proto_path=./proto; then you end up with the desired from package_name import dependancy_pb2

Say you are getting an error from from Package1.Package2 import ProtoModule (I have switched your example to the proper Python terminology here) due to Package1 being split over several directories in your PYTHONPATH. Then you should either (a) Make Package1 a namespace package or (b) not have two places for Package1! The first option is best avoided because namespace packages are a bit of a tricky beast. The second option is usually pretty easy, because you just make the --python_out= parameter to protoc point to the location that also contains your non-generated source code for Package1. I suspect this point is the cause of your problems.

By the way, from Package1.Package2 import ProtoModule only creates a variable ProtoModule; it does not create a variable Package2, so there is no need for the second relative import in your example.

See also a similar pull request #1511 and related bug report #1491.

[arthur-tacca]

I've realised I don't fully understand what this is for. If package_name.high_level_pb2 tries to import package_name.low_level_pb2, but gets an error because there is another package_name earlier in the PYTHON_PATH, then relative imports won't help at all because you won't be able to import package_name.high_level_pb2 in the first place the same reason! What situation would this actually help in?

[liujisi]

What if the dependency is in a different directory in the PYTHONPATH, and relative path wouldn't work? I guess you would have to reorganize your packages to avoid such conflicts.

[tlyng]

Any progress on this issue? Currently it does not generate valid python3 code.

[cherezov]

Actually no progress for now. Let's close my request at all.