Allow for the use of a proxy server during Python Language Server download #2418

d3r3kk · 2018-08-20T23:56:45Z

Title summarizes what is changing
Includes a news entry file (remember to thank yourself!)
Unit tests & code coverage are not adversely affected (within reason)
Works on all actively maintained versions of Python (e.g. Python 2.7 & the latest Python 3 release)
Works on Windows 10, macOS, and Linux (e.g. considered file system case-sensitivity)

DonJayamanne · 2018-08-21T06:09:55Z

src/client/activation/downloader.ts

-        this.platform = this.services.get<IPlatformService>(IPlatformService);
-        this.platformData = new PlatformData(this.platform, this.fs);
+    constructor(
+        private readonly output: IOutputChannel,


Why are we changing the di pattern? I thought we (you and i )agreed to keep using a simple parameter and user the service locator pattern?
Has anything changed since, to change your decision?

This particular class was not using DI, so I didn't add it. There was no new decision here, that was all there was to it 😄.

I felt that there was no need to add it to this particular class, seemed unnecessary for what it does. But I am certainly willing to have that discussion!

Should we want to expand this class's design, perhaps that's a different issue deserving of a longer conversation.

DonJayamanne · 2018-08-21T06:11:36Z

src/client/activation/downloader.ts

+        private readonly fs: IFileSystem,
+        private readonly platformData: PlatformData,
+        readonly workspace: IWorkspaceService,
+        private requestHandler: IRequestWrapper | undefined,


Optional parameter should be at the end.

Why is this optional?

Agreed, I will move it - not sure why I missed that!

LanguageServerDownloader move optional arguments to the end.

Optional in the sense that you can supply your own implementation only. If you do not supply one, one is supplied for you. Allows us to test the implementation of the request 'wrapper'.

DonJayamanne · 2018-08-21T06:15:25Z

src/client/activation/downloader.ts

-        }
+        // Set file to executable (nothing happens in Windows, as chmod has no definition there)
+        const executablePath = path.join(installFolder, this.platformData.getEngineExecutableName());
+        fileSystem.chmodSync(executablePath, '0764'); // -rwxrw-r--


Please use an async version, i.e. always use async instead of sync. This is what made node.js popular and faster. We need to remember that and not write old.net style sync code .

Again, just using what was there before. I can improve on the code of course, but this change was supposed to be a simple one.

I will add an issue to redesign the downloader class to make it more generic/improve the design.

Create a new issue to track the redesign of the LanguageServerDownloader class. We should improve upon its design by adding DI, removing any synchronous calls, and making it generally more robust for other use cases.

Make the use of chmod async. This is the right thing to do.

DonJayamanne · 2018-08-21T06:18:16Z

src/client/activation/requestWithProxy.ts

+                proxy: this.proxyUri
+            };
+        }
+        return undefined;


Just use return

Ok, will do!

RequestWithProxy::getRequestOptions remove return undefined and make it just return instead.

DonJayamanne · 2018-08-21T06:22:20Z

src/client/activation/requestWithProxy.ts

+    public downloadFileRequest(uri: string): request.Request {
+        const requestOptions = this.getRequestOptions();
+        if (requestOptions) {
+            return request(uri, requestOptions!);


! is not necessary due to type guards.
Infact you should be able to use return request(uri, requestOptions)

Ok, I'll try that. I believe the linter was complaining about this but I may be mistaken.

Remove unnecessary ! in downloadFileRequest

I was mistaken. All good.

DonJayamanne · 2018-08-21T06:24:43Z

src/client/activation/types.ts

@@ -16,3 +20,7 @@ export interface IExtensionActivator {
  activate(): Promise<boolean>;
  deactivate(): Promise<void>;
 }
+
+export interface IRequestWrapper {
+  downloadFileRequest(uri: string): RequestResult;


It provides a service of downloading a file, it does not wrap anything!
Please change the method to downliadFile.

Sure thing.

Reconsider the names IRequestWrapper (not a wrapper) and downloadFileRequest.

DonJayamanne · 2018-08-21T06:41:12Z

src/client/activation/downloader.ts

+        private engineFolder: string) {
+
+        if (!this.requestHandler) {
+            this.requestHandler = new RequestWithProxy(workspace.getConfiguration('http').get('proxy', ''));


This shouldn't be passed in as an argument. Assume you have to use this class in 10 places, now you have to read the proxy and pass it in 10 places.
It should be accessed by the downloader in the class itself.

Please inject this via DI, we shouldn't be manually constructing concrete classes, goes against IOC pattern.

Assume you have to use this class in 10 places

We use it in 1, and I didn't think this issue deserved the time to make it more generic. I feel that is a different issue.

now you have to read the proxy and pass it in 10 places

It felt odd coupling the RequestHandler to the WorkspaceService. The downloader doesn't care that you have a Workspace object, it only cares that you have a proxy setting (or not).

Please inject this via DI

...not this change. Let's push that to another.

See issue #2421

d3r3kk · 2018-08-21T15:06:29Z

@DonJayamanne please re-review.

codecov · 2018-08-21T15:07:23Z

Codecov Report

Merging #2418 into master will decrease coverage by 0.48%.
The diff coverage is 68.29%.

@@            Coverage Diff             @@
##           master    #2418      +/-   ##
==========================================
- Coverage   75.63%   75.14%   -0.49%     
==========================================
  Files         310      310              
  Lines       14511    14352     -159     
  Branches     2567     2540      -27     
==========================================
- Hits        10975    10785     -190     
- Misses       3528     3558      +30     
- Partials        8        9       +1

Flag	Coverage Δ
#MacOS	`?`
#Windows	`?`

Impacted Files	Coverage Δ
src/client/common/platform/types.ts	`100% <ø> (ø)`	⬆️
src/client/activation/types.ts	`100% <ø> (ø)`	⬆️
src/client/activation/languageServer.ts	`26.08% <0%> (+0.4%)`	⬆️
src/client/activation/platformData.ts	`85% <100%> (ø)`	⬆️
src/client/common/platform/fileSystem.ts	`70.96% <66.66%> (-0.47%)`	⬇️
src/client/activation/requestWithProxy.ts	`66.66% <66.66%> (ø)`
src/client/activation/downloader.ts	`34.56% <72.22%> (-0.8%)`	⬇️
src/client/common/platform/registry.ts	`40.42% <0%> (-51.07%)`	⬇️
src/client/common/platform/pathUtils.ts	`66.66% <0%> (-33.34%)`	⬇️
src/client/common/utils.ts	`57.89% <0%> (-16.79%)`	⬇️
... and 29 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e3a6bc2...26fbd00. Read the comment docs.

ericsnowcurrently

At a high level, LGTM. However there are a bunch of little things I noticed that might be worth cleaning up. Some of them are nits, but some should probably be addressed.

ericsnowcurrently · 2018-08-23T19:27:02Z

src/client/activation/downloader.ts

+        private readonly platformData: PlatformData,
+        readonly workspace: IWorkspaceService,
+        private engineFolder: string,
+        private requestHandler?: IDownloadFileService) {


FWIW, I find it clearer to put the opening brace on its own line:

private engineFolder: string, private requestHandler?: IDownloadFileService ) {

I think either way is good, and I have tried to follow what I see across the rest of the classes in the extension thus far.

Personally, I like the:

constructor ( some: string, params: number, here: object ) { some_code_here(); }

...way of doing things, but I'm learning to adapt.

ericsnowcurrently · 2018-08-23T19:35:27Z

src/client/activation/types.ts

@@ -16,3 +20,7 @@ export interface IExtensionActivator {
  activate(): Promise<boolean>;
  deactivate(): Promise<void>;
 }
+
+export interface IDownloadFileService {


Doesn't this need the following?

export const IDownloadFileService = Symbol('IDownloadFileService');

Our use of injectify makes use of the Symbol type to identify implementations of interfaces. Since we don't inject the IDownloadFileService into DI (yet) it doesn't need the Symbol definition.

ericsnowcurrently · 2018-08-23T19:38:08Z

src/client/activation/downloader.ts


        let localTempFilePath = '';
        try {
+


superfluous blank line

ericsnowcurrently · 2018-08-23T19:39:25Z

src/client/activation/downloader.ts

@@ -71,7 +72,7 @@ export class LanguageServerDownloader {
        const tempFile = await this.fs.createTemporaryFile(downloadFileExtension);

        const deferred = createDeferred();
-        const fileStream = fileSystem.createWriteStream(tempFile.filePath);
+        const fileStream = this.fs.createWriteStream(tempFile.filePath);


Ha! We weren't even using this.fs already?

ericsnowcurrently · 2018-08-23T19:41:18Z

src/client/activation/downloader.ts

+        private readonly output: IOutputChannel,
+        private readonly fs: IFileSystem,
+        private readonly platformData: PlatformData,
+        readonly workspace: IWorkspaceService,


This doesn't need readonly, right?

I felt this was appropriate, since I am not changing the state of that object and am only reading from it.

Ah, I didn't realize you could mark parameters as readonly. Cool!

ericsnowcurrently · 2018-08-23T20:11:06Z

src/client/activation/requestWithProxy.ts

+        if (requestOptions) {
+            return request(uri, requestOptions);
+        }
+        return request(uri);


FWIW, if both cases terminate the function then it's often clearer to use an else:

if (requestOptions) { return request(uri, requestOptions); } else { return request(uri); }

6 or 1/2 dozen.
Personally I find the current approach more readable, less nesting. I like an else for such simple statements only when using IIF (immediate ifs)

Wow @DonJayamanne jumped in between @ericsnowcurrently's comments and my responses. Fun timez.

I am going with Eric's suggestion here, I likey.

I definitely agree with Don in cases where there's a clear common path. I try to keep those less indented than the code for the "exceptional" cases. However, when they are equally likely or equally relevant I prefer to keep indentation the same. It helps inform the reader of the nature of the cases.

That said, if either case is more than a few lines it often makes sense to factor it out into a separate private function/method, so that the logic of the original function isn't obscured by handling of the different cases.

ericsnowcurrently · 2018-08-23T20:15:04Z

src/client/common/platform/fileSystem.ts


+    public chmod(filePath: string, mode: string): Promise<void> {
+        return new Promise<void>((resolve, reject) => {


Blech! This async stuff is so gross! Kudos for knowing how to do this. :)

There is also createDeferred() that simplifies things. Rough equivalent of the TaskCompletionSource in C#

I've not learned why/how to use the createDeferred method yet, and was simply following the pattern from the methods above!

ericsnowcurrently · 2018-08-23T20:34:13Z

src/test/activation/downloader.unit.test.ts

        assert.equal(link, DownloadLinks[PlatformName.Linux64Bit]);
    });
+    test('Supports download via proxy', async () => {


Maybe I'm missing something, but languageServerDownloader isn't used in this test. So doesn't the test belong in src/test/activation/requestWithProxy.unit.test.ts (or at least in its own suite)?

Also, there probably should be separate tests for LanguageServerDownloader.downloadLanguageServer() that use different languageServerDownloader.requestHandler values.

You found me out! Lazy me.

Move tests into appropriate test suites.

Let's ensure we add far more robust testing as part of #2421, as it will need a bit of fine-tuning to make it more accessible to tests.

ericsnowcurrently · 2018-08-23T20:35:46Z

src/client/activation/downloader.ts

@@ -83,7 +84,7 @@ export class LanguageServerDownloader {
            location: ProgressLocation.Window
        }, (progress) => {

-            requestProgress(request(uri))
+            requestProgress(this.requestHandler!.downloadFile(uri))


Splitting up this line might be more readable:

requestProgress( this.requestHandler!.downloadFile(uri))

ericsnowcurrently · 2018-08-23T20:37:34Z

src/test/common/platform/filesystem.unit.test.ts

@@ -88,4 +88,31 @@ suite('FileSystem', () => {
        const fileName = files[0].replace(/\\/g, '/');
        expect(fileName).to.equal(expectedFileName);
    });
+    test('Ensure creating a temporary file results in a unique temp file path', async () => {


smart tests!

MikhailArkhipov · 2018-08-23T20:56:35Z

src/client/activation/downloader.ts

-            fileSystem.chmodSync(executablePath, '0764'); // -rwxrw-r--
-        }
+        // Set file to executable (nothing happens in Windows, as chmod has no definition there)
+        const executablePath = path.join(installFolder, this.platformData.getEngineExecutableName());


It worked on Windows as is but OK

I took the opportunity to decouple this class from the IPlatformService (I know we still use PlatformData to get a hold of some names, but those two classes have separate use cases).

d3r3kk · 2018-08-24T15:31:37Z

@ericsnowcurrently, @DonJayamanne: All PR comments are addressed, if someone could find the time to re-review? Thanks!

d3r3kk · 2018-08-24T17:10:44Z

Retry the CI on VSTS - uncertain why this would fail! (Cannot reproduce it on Ubuntu linux...)

ericsnowcurrently

LGTM

I left a few comments, but they're not that big a deal.

ericsnowcurrently · 2018-08-24T16:53:23Z

src/client/activation/languageServer.ts

@@ -141,7 +142,7 @@ export class LanguageServerExtensionActivator implements IExtensionActivator {
                this.output,
                this.fs,
                this.platformData,
-                this.workspace,
+                new RequestWithProxy(this.workspace.getConfiguration('http').get('proxy', '')),


Mashing this into one line is okay if the risk of exceptions from getConfiguration() and get('proxy') are relatively nil (which AFAICS they are). Otherwise stack traces are a little harder to follow. I'll also argue that this is a bit less readable. :)

I'll leave it up to you.

ericsnowcurrently · 2018-08-24T16:56:47Z

src/client/activation/requestWithProxy.ts

        if (this.proxyUri && this.proxyUri.length > 0) {
            return {
                proxy: this.proxyUri
            };
+        } else {
+            return;


I though you were going with return {};. It's fine either way. Mostly this syntax (mixed bare return and non-bare return in the same function) weirds me out! that and the whole return-CoreOptions-OR-undefined makes the function feel much more complex than it actually is.

That said, it's not a huge deal. :) I'll leave this up to you.

ericsnowcurrently · 2018-08-24T16:57:15Z

src/client/activation/requestWithProxy.ts

-        }
-        return request(uri);
+        const requestOptions: request.CoreOptions | undefined = this.requestOptions;
+        return request(uri, requestOptions);


ericsnowcurrently · 2018-08-24T17:10:31Z

src/test/mocks/vsc/extHostedTypes.ts

@@ -570,6 +570,18 @@ export namespace vscMockExtHostedTypes {
            return edits ? edits.slice() : undefined;
        }

+        createFile(uri: vscUri.URI, options?: { overwrite?: boolean, ignoreIfExists?: boolean }): void {


Whoa! I didn't realize we had any pre-built testing doubles! (This is more of a fake than a mock, BTW.) I thought we only use mocks.

See:

https://martinfowler.com/articles/mocksArentStubs.html#TheDifferenceBetweenMocksAndStubs

https://testing.googleblog.com/2013/07/testing-on-toilet-know-your-test-doubles.html

NOTE: revert launch.json before submitting a PR

- Provide a default interface that will be used in production - Swap out the interface for testing

- Update downloader to use a request interface to handle a request - implement the interface such that it knows about proxy uris - test that proxy info is handled in the request interface

…f the list

- give it's download method a more apt name

…eEdit

- decouple LanguageServerDownloader from the workspace service

All PR comments have been addressed or forwarded to #2421.

d3r3kk requested review from DonJayamanne and ericsnowcurrently August 21, 2018 00:40

DonJayamanne previously requested changes Aug 21, 2018

View reviewed changes

d3r3kk force-pushed the issue2385_proxy_download branch from 526be81 to c7e60b0 Compare August 21, 2018 15:06

ericsnowcurrently suggested changes Aug 23, 2018

View reviewed changes

MikhailArkhipov reviewed Aug 23, 2018

View reviewed changes

d3r3kk closed this Aug 24, 2018

d3r3kk reopened this Aug 24, 2018

ericsnowcurrently approved these changes Aug 24, 2018

View reviewed changes

d3r3kk added 16 commits August 24, 2018 11:56

Updates to get downloader to become testable

a58307b

NOTE: revert launch.json before submitting a PR

Refactor the downloader to accept an interface for request handling

3bb79a8

- Provide a default interface that will be used in production - Swap out the interface for testing

Pull out request altogether from LanguageServerDownloader class.

6037f4e

- Update downloader to use a request interface to handle a request - implement the interface such that it knows about proxy uris - test that proxy info is handled in the request interface

Add news, remove grep from launch.json

2a042ac

LanguageServerDownloader::constructor move optional args to the end o…

5af6473

…f the list

Make use of chmod async

60cf4fa

Remove unnecessary bang in return line.

531ecc0

Avoid returning undefined. Instead, just use return.

9b76edb

Rename IRequestWrapper to IDownloadFileService

e0e63e6

- give it's download method a more apt name

Add testing to the filesystem implementation.

f99a676

Fix unit test for writing to a file stream

951fa27

update mock vscode file to match latest vscode interface for Workspac…

2ea42df

…eEdit

PR feedback improvements

14f467b

- decouple LanguageServerDownloader from the workspace service

Go back to using return value | undefined pattern in request class

a38ff7c

Move tests into appropriately named & focussed test files

f82f2f7

Remove my duplication of the fakes in extHostedTypes

26fbd00

d3r3kk force-pushed the issue2385_proxy_download branch from 0ff58c0 to 26fbd00 Compare August 24, 2018 18:58

d3r3kk merged commit 3ab005e into microsoft:master Aug 24, 2018

d3r3kk deleted the issue2385_proxy_download branch August 24, 2018 20:09

lock bot locked as resolved and limited conversation to collaborators Jul 31, 2019


		public chmod(filePath: string, mode: string): Promise<void> {
		return new Promise<void>((resolve, reject) => {

Allow for the use of a proxy server during Python Language Server download #2418

Allow for the use of a proxy server during Python Language Server download #2418

Conversation

d3r3kk commented Aug 20, 2018

Choose a reason for hiding this comment

d3r3kk Aug 21, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

d3r3kk Aug 21, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

d3r3kk Aug 21, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

d3r3kk Aug 21, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

d3r3kk Aug 21, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

d3r3kk Aug 21, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

d3r3kk Aug 21, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

d3r3kk commented Aug 21, 2018

codecov bot commented Aug 21, 2018 • edited Loading

Codecov Report

ericsnowcurrently left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MikhailArkhipov Aug 23, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

d3r3kk Aug 23, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

d3r3kk commented Aug 24, 2018

d3r3kk commented Aug 24, 2018

ericsnowcurrently left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

d3r3kk Aug 21, 2018 •

edited

Loading

d3r3kk Aug 21, 2018 •

edited

Loading

d3r3kk Aug 21, 2018 •

edited

Loading

d3r3kk Aug 21, 2018 •

edited

Loading

d3r3kk Aug 21, 2018 •

edited

Loading

d3r3kk Aug 21, 2018 •

edited

Loading

d3r3kk Aug 21, 2018 •

edited

Loading

codecov bot commented Aug 21, 2018 •

edited

Loading

MikhailArkhipov Aug 23, 2018 •

edited

Loading

d3r3kk Aug 23, 2018 •

edited

Loading