Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

internal: speed up FileInfoIdx-from-path lookup #859

Merged
merged 10 commits into from
Aug 27, 2023

Conversation

bung87
Copy link
Contributor

@bung87 bung87 commented Aug 26, 2023

Summary

Cache the result of file-info-from-absolute-path lookups, avoiding
unnecessary filesystem IO incurred by expanding file paths. This
significantly increases the speed of the incremental compilation mode
(which performs a lot of those lookups), with compile time reductions
of up to 90% (measured with the tstdlib_import_changed.nim test).

Details

Add the rawPathToIndexTbl table to MsgFormat. It associates the
unprocessed absolute file paths passed to fileInfoIdx with their
resolved-to FileInfoIdx, preventing the repeated expansion of the
same file paths.

In addition, the line overrides in the assert and jsAssert
templates are changed to use the no-argument form ({.line.}). The
latter is semantically equivalent to the previous {.line: loc.},
while not performing an unnecessary FileInfoIdx -> filename ->
FileInfoIdx round-trip.

Copy link
Collaborator

@zerbina zerbina left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Welcome, and thank you for working on this.

I've left a few requests regarding code style, data placement, and documentation. In addition, please update the PR message and title according to the contribution guidelines. The guidelines contain examples of how the title and message should look like, and for inspiration, you can also look at PRs merged previously.

compiler/front/options.nim Outdated Show resolved Hide resolved
compiler/front/options.nim Outdated Show resolved Hide resolved
compiler/front/options.nim Outdated Show resolved Hide resolved
compiler/front/options.nim Outdated Show resolved Hide resolved
lib/js/jsconsole.nim Outdated Show resolved Hide resolved
compiler/front/options.nim Outdated Show resolved Hide resolved
@zerbina zerbina added compiler General compiler tag commit style Commit message need to be fixed according to the contribution guideline labels Aug 26, 2023
bung87 and others added 3 commits August 26, 2023 23:39
Co-authored-by: zerbina <100542850+zerbina@users.noreply.github.com>
Co-authored-by: zerbina <100542850+zerbina@users.noreply.github.com>
Copy link
Collaborator

@zerbina zerbina left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. There are few things I missed during my earlier review that still need to be changed, but apart from those and the PR message/title, this is good to.


For the PR title, make sure it reads like a changelog entry (it should also not include the fixed issue) and is 50 characters in length (a few more are generally okay).

The "summary" section should provide a short overview of what changed and why it changed. The "details" section should provide details about the implementation and steps taken. In addition, each line in the message body must be no longer than 72 characters.

As it stands, most of what you have placed in the "summary" section should be part of the "details" section.

compiler/front/options.nim Outdated Show resolved Hide resolved
compiler/front/options.nim Outdated Show resolved Hide resolved
lib/js/jsconsole.nim Outdated Show resolved Hide resolved
lib/system/assertions.nim Outdated Show resolved Hide resolved
bung87 and others added 4 commits August 27, 2023 01:48
Co-authored-by: zerbina <100542850+zerbina@users.noreply.github.com>
Co-authored-by: zerbina <100542850+zerbina@users.noreply.github.com>
@bung87 bung87 changed the title fix #546;Canonicalized filepaths has a large performance impact on co… add cache layer to proc fileInfoIdx avoiding redundant file io Aug 26, 2023
@zerbina
Copy link
Collaborator

zerbina commented Aug 26, 2023

The pragmaLine procedure ignores the provided column information, which is why the t9768.nim test failed when you used the .line pragma without any argument.

I'll quickly make a separate PR with a fix, after which you can merge devel into this PR and use .line again.

@zerbina
Copy link
Collaborator

zerbina commented Aug 26, 2023

Okay, the fix is merged. After merging the devel branch into this PR, you should be able to use only {.line.}.

@bung87
Copy link
Contributor Author

bung87 commented Aug 27, 2023

now all passed, am not native English speaker, feel free to modify the title and description.

@zerbina zerbina changed the title add cache layer to proc fileInfoIdx avoiding redundant file io internal: speed up FileInfoIdx-from-path lookup Aug 27, 2023
@zerbina zerbina removed the commit style Commit message need to be fixed according to the contribution guideline label Aug 27, 2023
Copy link
Collaborator

@zerbina zerbina left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've updated both the PR message and title. Thank you seeing this through. Nice work! IC is a lot faster now.

I did remove the mention of #546, however, as the caching doesn't affect the --filenames=canonical mode more than it affects the other modes (canonicalImportAux doesn't use fileInfoIdx).

@bung87
Copy link
Contributor Author

bung87 commented Aug 27, 2023

Thank you for review my PR and make a better description. glad to hear it getting faster, nice!

@zerbina
Copy link
Collaborator

zerbina commented Aug 27, 2023

/merge

@github-actions
Copy link

Merge requested by: @zerbina

Contents after the first section break of the PR description has been removed and preserved below:


@chore-runner chore-runner bot added this pull request to the merge queue Aug 27, 2023
Merged via the queue into nim-works:devel with commit 5737047 Aug 27, 2023
18 checks passed
@saem
Copy link
Collaborator

saem commented Aug 28, 2023

That's a very nice performance improvement, thanks @bung87

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
compiler General compiler tag
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants