Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

getDocsBatch for bulk processing #2371

Draft
wants to merge 1 commit into
base: master
Choose a base branch
from
Draft

getDocsBatch for bulk processing #2371

wants to merge 1 commit into from

Conversation

Anton-Latukha
Copy link
Collaborator

@Anton-Latukha Anton-Latukha commented Nov 19, 2021 •
edited
Loading

A branch from: #2349

Holds a relevant getDocsBatch code to make bulk requests.

TODO:

  • error -> exception handling code
  • Maps be Strict
  • Recompose (functor law & firsts can be flattened)
  • Wire it in

Besides that - can not find the better-optimized version, I hope that swap ... swap & tuple passing et al. optimizes/fuses away by GHC (which should).

Also - after functor laws are applied - count the number of traverses through the [] & weigh it for a number of batch requests.

Currently in master primary use of getDocsBatch is: getDocumentationTryGhc that runs singleton element doc on: getDocumentationsTryGhc that runs getDocsBatch on singleton.

getDocsBatch internally has a modified (with removed interactive check) copy of GHCs getDocs. If that internal 1-elem non-interactive function is needed - it probably needs to be getDocsNonInteractive & so be module scope.

And then getDocsBatch is designed for what it is intended - a bulk [Name] requests.

Then I went further: if getDocsBatch would be used really in batch mode - why is to do getDocsBatch on getDocsNonInteractive - that searches & loads module interface on each element:

So, currently I do in getDocsBatch:

Load [Name].

For each Name get module it says being part of.

Then group all Names by according Module: [(Module, [Name])].

Then load module interface once & process through it in bulk [Name]s that are in that module.

For current way function is used (singleton retrieval) that would introduce + C steps for retrieval (that may be specialized-away by the compiler).

If HLS starts bulking retrieval from GHC interfaces based on [Name] - that would be optimal way.

@Anton-Latukha Anton-Latukha mentioned this pull request Nov 19, 2021
Closed
@Anton-Latukha Anton-Latukha force-pushed the 2021-11-19-bulk-getDocsBatch branch 2 times, most recently from ab947a3 to 38463da Compare November 19, 2021 18:46
@jneira

This comment has been minimized.

Sign in to view
@pepeiborra

This comment has been minimized.

Sign in to view
@Anton-Latukha

This comment has been minimized.

Sign in to view
@Anton-Latukha

This comment has been minimized.

Sign in to view
@Anton-Latukha

This comment has been minimized.

Sign in to view
@Anton-Latukha

This comment has been minimized.

Sign in to view
Anton-Latukha added a commit to Anton-Latukha/haskell-language-server that referenced this pull request Nov 26, 2021
@Anton-Latukha
ghcide: Core.Compile: add getDocsNonInteractive{',}
44de0dc 
`getDocsBatch` cuurently (& before) used only for single name retrieval
function. Use of it is in `Documentation` module `getDocumentationTryGhc` where
it wraps arg into singleton & gives to `getDocsBatch` & then recieves a Map with
1 entry & unsafely "lookups" doc in it.

This work is to supply the proper single name retrieval-optimized version to
stop that `getDocsBatch` there.

& further ideally `getDocumentationTryGhc` uses single-retrieval &
`getDocumentationsTryGhc` uses a batch mode & batch mode gets optimized along
the lines of: haskell#2371
Anton-Latukha added a commit to Anton-Latukha/haskell-language-server that referenced this pull request Nov 26, 2021
@Anton-Latukha
ghcide: Core.Compile: add getDocsNonInteractive{',}
93088b1 
`getDocsBatch` cuurently (& before) used only for single name retrieval
function. Use of it is in `Documentation` module `getDocumentationTryGhc` where
it wraps arg into singleton & gives to `getDocsBatch` & then recieves a Map with
1 entry & unsafely "lookups" doc in it.

This work is to supply the proper single name retrieval-optimized version to
stop that `getDocsBatch` there.

& further ideally `getDocumentationTryGhc` uses single-retrieval &
`getDocumentationsTryGhc` uses a batch mode & batch mode gets optimized along
the lines of: haskell#2371
Anton-Latukha added a commit to Anton-Latukha/haskell-language-server that referenced this pull request Nov 27, 2021
@Anton-Latukha
ghcide: Core.Compile: add getDocsNonInteractive{',}
f5b8b20 
`getDocsBatch` cuurently (& before) used only for single name retrieval
function. Use of it is in `Documentation` module `getDocumentationTryGhc` where
it wraps arg into singleton & gives to `getDocsBatch` & then recieves a Map with
1 entry & unsafely "lookups" doc in it.

This work is to supply the proper single name retrieval-optimized version to
stop that `getDocsBatch` there.

& further ideally `getDocumentationTryGhc` uses single-retrieval &
`getDocumentationsTryGhc` uses a batch mode & batch mode gets optimized along
the lines of: haskell#2371
Anton-Latukha added a commit to Anton-Latukha/haskell-language-server that referenced this pull request Dec 12, 2021
@Anton-Latukha
ghcide: Core.Compile: add getDocsNonInteractive{',}
af78e74 
`getDocsBatch` cuurently (& before) used only for single name retrieval
function. Use of it is in `Documentation` module `getDocumentationTryGhc` where
it wraps arg into singleton & gives to `getDocsBatch` & then recieves a Map with
1 entry & unsafely "lookups" doc in it.

This work is to supply the proper single name retrieval-optimized version to
stop that `getDocsBatch` there.

& further ideally `getDocumentationTryGhc` uses single-retrieval &
`getDocumentationsTryGhc` uses a batch mode & batch mode gets optimized along
the lines of: haskell#2371
Anton-Latukha added a commit to Anton-Latukha/haskell-language-server that referenced this pull request Dec 15, 2021
@Anton-Latukha
ghcide: Core.Compile: add getDocsNonInteractive{',}
ddf1a55 
`getDocsBatch` cuurently (& before) used only for single name retrieval
function. Use of it is in `Documentation` module `getDocumentationTryGhc` where
it wraps arg into singleton & gives to `getDocsBatch` & then recieves a Map with
1 entry & unsafely "lookups" doc in it.

This work is to supply the proper single name retrieval-optimized version to
stop that `getDocsBatch` there.

& further ideally `getDocumentationTryGhc` uses single-retrieval &
`getDocumentationsTryGhc` uses a batch mode & batch mode gets optimized along
the lines of: haskell#2371
Anton-Latukha
Anton-Latukha commented Dec 19, 2021
View reviewed changes
ghcide/src/Development/IDE/Core/Compile.hs
where
throwErrors = liftIO . throwIO . mkSrcErr
compiled n =
((_warns,errs), res) <- initTc hsc_env HsSrcFile False _mod fakeSpan $ Map.fromList <$> traverse undefined undefined
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From https://gitlab.haskell.org/ghc/ghc/-/issues/20831#note_398227:

I looked at the HLS code and you should probably rather use:
initIfaceLoad hsc_env (loadSysInterface "docs" module)
to load the interface rather than calling initTc.
Then that will cache the loaded interface so you don't load it multiple times from disk.

Copy link
Collaborator Author

@Anton-Latukha Anton-Latukha Dec 20, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(code currently is created so that it loads & opens interfaces once, but if I understand it right - the mentioned method caches interfaces between function runs).

This was referenced Dec 20, 2021
Merged
Open
Anton-Latukha added a commit to Anton-Latukha/haskell-language-server that referenced this pull request Dec 24, 2021
@Anton-Latukha
ghcide: Core.Compile: add getDocsNonInteractive{',}
b51b190 
`getDocsBatch` cuurently (& before) used only for single name retrieval
function. Use of it is in `Documentation` module `getDocumentationTryGhc` where
it wraps arg into singleton & gives to `getDocsBatch` & then recieves a Map with
1 entry & unsafely "lookups" doc in it.

This work is to supply the proper single name retrieval-optimized version to
stop that `getDocsBatch` there.

& further ideally `getDocumentationTryGhc` uses single-retrieval &
`getDocumentationsTryGhc` uses a batch mode & batch mode gets optimized along
the lines of: haskell#2371
Anton-Latukha added a commit to Anton-Latukha/haskell-language-server that referenced this pull request Dec 25, 2021
@Anton-Latukha
ghcide: Core.Compile: add getDocsNonInteractive{',}
80d00d7 
`getDocsBatch` cuurently (& before) used only for single name retrieval
function. Use of it is in `Documentation` module `getDocumentationTryGhc` where
it wraps arg into singleton & gives to `getDocsBatch` & then recieves a Map with
1 entry & unsafely "lookups" doc in it.

This work is to supply the proper single name retrieval-optimized version to
stop that `getDocsBatch` there.

& further ideally `getDocumentationTryGhc` uses single-retrieval &
`getDocumentationsTryGhc` uses a batch mode & batch mode gets optimized along
the lines of: haskell#2371
Anton-Latukha added a commit to Anton-Latukha/haskell-language-server that referenced this pull request Dec 25, 2021
@Anton-Latukha
ghcide: Core.Compile: add getDocsNonInteractive{',}
f6232ec 
`getDocsBatch` cuurently (& before) used only for single name retrieval
function. Use of it is in `Documentation` module `getDocumentationTryGhc` where
it wraps arg into singleton & gives to `getDocsBatch` & then recieves a Map with
1 entry & unsafely "lookups" doc in it.

This work is to supply the proper single name retrieval-optimized version to
stop that `getDocsBatch` there.

& further ideally `getDocumentationTryGhc` uses single-retrieval &
`getDocumentationsTryGhc` uses a batch mode & batch mode gets optimized along
the lines of: haskell#2371
Anton-Latukha added a commit to Anton-Latukha/haskell-language-server that referenced this pull request Dec 25, 2021
@Anton-Latukha
ghcide: Core.Compile: add getDocsNonInteractive{',}
d1fabad 
`getDocsBatch` cuurently (& before) used only for single name retrieval
function. Use of it is in `Documentation` module `getDocumentationTryGhc` where
it wraps arg into singleton & gives to `getDocsBatch` & then recieves a Map with
1 entry & unsafely "lookups" doc in it.

This work is to supply the proper single name retrieval-optimized version to
stop that `getDocsBatch` there.

& further ideally `getDocumentationTryGhc` uses single-retrieval &
`getDocumentationsTryGhc` uses a batch mode & batch mode gets optimized along
the lines of: haskell#2371
Anton-Latukha added a commit to Anton-Latukha/haskell-language-server that referenced this pull request Dec 25, 2021
@Anton-Latukha
ghcide: Core.Compile: add getDocsNonInteractive{',}
17df04e 
`getDocsBatch` cuurently (& before) used only for single name retrieval
function. Use of it is in `Documentation` module `getDocumentationTryGhc` where
it wraps arg into singleton & gives to `getDocsBatch` & then recieves a Map with
1 entry & unsafely "lookups" doc in it.

This work is to supply the proper single name retrieval-optimized version to
stop that `getDocsBatch` there.

& further ideally `getDocumentationTryGhc` uses single-retrieval &
`getDocumentationsTryGhc` uses a batch mode & batch mode gets optimized along
the lines of: haskell#2371
Anton-Latukha added a commit to Anton-Latukha/haskell-language-server that referenced this pull request Dec 25, 2021
@Anton-Latukha
ghcide: Core.Compile: add getDocsNonInteractive{',}
d846164 
`getDocsBatch` cuurently (& before) used only for single name retrieval
function. Use of it is in `Documentation` module `getDocumentationTryGhc` where
it wraps arg into singleton & gives to `getDocsBatch` & then recieves a Map with
1 entry & unsafely "lookups" doc in it.

This work is to supply the proper single name retrieval-optimized version to
stop that `getDocsBatch` there.

& further ideally `getDocumentationTryGhc` uses single-retrieval &
`getDocumentationsTryGhc` uses a batch mode & batch mode gets optimized along
the lines of: haskell#2371
Anton-Latukha added a commit to Anton-Latukha/haskell-language-server that referenced this pull request Dec 25, 2021
@Anton-Latukha
ghcide: Core.Compile: add getDocsNonInteractive{',}
51113a9 
`getDocsBatch` cuurently (& before) used only for single name retrieval
function. Use of it is in `Documentation` module `getDocumentationTryGhc` where
it wraps arg into singleton & gives to `getDocsBatch` & then recieves a Map with
1 entry & unsafely "lookups" doc in it.

This work is to supply the proper single name retrieval-optimized version to
stop that `getDocsBatch` there.

& further ideally `getDocumentationTryGhc` uses single-retrieval &
`getDocumentationsTryGhc` uses a batch mode & batch mode gets optimized along
the lines of: haskell#2371
Anton-Latukha added a commit to Anton-Latukha/haskell-language-server that referenced this pull request Feb 1, 2022
@Anton-Latukha
ghcide: Core.Compile: add getDocsNonInteractive{',}
1798006 
`getDocsBatch` cuurently (& before) used only for single name retrieval
function. Use of it is in `Documentation` module `getDocumentationTryGhc` where
it wraps arg into singleton & gives to `getDocsBatch` & then recieves a Map with
1 entry & unsafely "lookups" doc in it.

This work is to supply the proper single name retrieval-optimized version to
stop that `getDocsBatch` there.

& further ideally `getDocumentationTryGhc` uses single-retrieval &
`getDocumentationsTryGhc` uses a batch mode & batch mode gets optimized along
the lines of: haskell#2371
Anton-Latukha added a commit to Anton-Latukha/haskell-language-server that referenced this pull request Feb 4, 2022
@Anton-Latukha
ghcide: Core.Compile: add getDocsNonInteractive{',}
b751674 
`getDocsBatch` cuurently (& before) used only for single name retrieval
function. Use of it is in `Documentation` module `getDocumentationTryGhc` where
it wraps arg into singleton & gives to `getDocsBatch` & then recieves a Map with
1 entry & unsafely "lookups" doc in it.

This work is to supply the proper single name retrieval-optimized version to
stop that `getDocsBatch` there.

& further ideally `getDocumentationTryGhc` uses single-retrieval &
`getDocumentationsTryGhc` uses a batch mode & batch mode gets optimized along
the lines of: haskell#2371
@Anton-Latukha Anton-Latukha mentioned this pull request Feb 6, 2022
Open
@pepeiborra pepeiborra added the status: unfinished Status for PRs that have been semi-abandoned label Aug 13, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@wz1000 wz1000 Awaiting requested review from wz1000 wz1000 will be requested when the pull request is marked ready for review wz1000 is a code owner

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
status: unfinished Status for PRs that have been semi-abandoned
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@Anton-Latukha @jneira @pepeiborra