Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

sparse-index diff integration: initial attempt #418

Conversation

ldennington
Copy link
Collaborator

@ldennington ldennington commented Sep 1, 2021

THIS CHANGE IS NOT IN FINAL FORM AND IS CURRENTLY JUST INTENDED FOR
FEEDBACK. To understand the context laid out below I have purposefully
included both source and test changes in this initial commit. Based on
the feedback I get, I will either split this commit or create
an entirely new branch with the necessary changes when it is ready to
officially submit.

This change enables running the diff builtin command without
expanding the full index in a cone-mode sparse checkout. It is an attempt
to add the basic infrastructure to "light up" sparse index for the
command. However, based on some testing, this may need to change.

After reviewing #417, I decided to add ensure_full_index to this
initial commit in an attempt to maintain current behavior. However,
as I started building and testing with and without this update, I
noticed unexpected results.

The ensure_not_expanded tests I added are passing with
ensure_full_index both enabled and disabled. That felt wrong, so I
used TRACE2 to try and understand what was happening and have attached
the results below.

It looks as though no code paths in diff.c are actually being used.
On the bright side, however, it doesn't look as though ensure_full_index
is ever being called. I recognize my lack of knowledge wrt cache-tree
and read-cache may be the reason I'm unsure what the correct path
forward is. In light of this, any guidance is appreciated.

THIS CHANGE IS NOT IN FINAL FORM AND IS CURRENTLY JUST INTENDED FOR
FEEDBACK. To understand the context laid out below I have purposefully
included both source and test changes in this initial commit. Based on
the feedback I get, I will either split this commit or create
an entirely new branch with the necessary changes when it is ready to
officially submit.

This change enables running the `diff` builtin command without
expanding the full index in a cone-mode sparse checkout. It is an attempt
to add the basic infrastructure to "light up" sparse index for the
command. However, based on some testing, this may need to change.

After reviewing microsoft#417, I decided to add `ensure_full_index` to this
initial commit in an attempt to maintain current behavior. However,
as I started building and testing with and without this update, I
noticed unexpected results.

The `ensure_not_expanded` tests I added are passing with
`ensure_full_index` both enabled and disabled. That felt wrong, so I
used `TRACE2` to try and understand what was happening and have attached
the results.

It looks as though no code paths in `diff.c` are actually being used.
On the bright side, however, it doesn't look as though `ensure_full_index`
is ever being called. I recognize my lack of knowledge wrt `cache-tree`
and `read-cache` may be the reason I'm unsure what the correct path
forward is. In light of this, any guidance is appreciated.
@ldennington
Copy link
Collaborator Author

Here are the GIT_TRACE2_PERF results mentioned in the PR description:

13:58:58.306222 common-main.c:48             | d0 | main                     | version      |     |           |           |              | 2.33.0.vfs.0.0.42.ga5b42902ab.dirty
13:58:58.306693 common-main.c:49             | d0 | main                     | start        |     |  0.001933 |           |              | ./git diff
13:58:58.307515 git.c:523                    | d0 | main                     | cmd_name     |     |           |           |              | diff (diff)
13:58:58.307913 repository.c:132             | d0 | main                     | def_repo     | r1  |           |           |              | worktree:/Users/ldennington/repos/_git/sparse-index
13:58:58.308565 read-cache.c:2406            | d0 | main                     | region_enter | r1  |  0.003813 |           | index        | label:do_read_index .git/index
13:58:58.309187 read-cache.c:1807            | d0 | main                     | region_enter |     |  0.004436 |           | index        | ..label:read/extension/cache_tree
13:58:58.309223 cache-tree.c:659             | d0 | main                     | region_enter | r1  |  0.004471 |           | cache_tree   | ....label:read
13:58:58.309336 cache-tree.c:661             | d0 | main                     | region_leave | r1  |  0.004585 |  0.000114 | cache_tree   | ....label:read
13:58:58.309365 read-cache.c:1809            | d0 | main                     | data         |     |  0.004612 |  0.000176 | index        | ....read/extension/cache_tree/bytes:6693
13:58:58.309391 read-cache.c:1810            | d0 | main                     | region_leave |     |  0.004641 |  0.000205 | index        | ..label:read/extension/cache_tree
13:58:58.309436 read-cache.c:2360            | d0 | main                     | data         | r1  |  0.004685 |  0.000872 | index        | ..read/version:2
13:58:58.309465 read-cache.c:2362            | d0 | main                     | data         | r1  |  0.004715 |  0.000902 | index        | ..read/cache_nr:4051
13:58:58.309489 read-cache.c:2411            | d0 | main                     | region_leave | r1  |  0.004738 |  0.000925 | index        | label:do_read_index .git/index
13:58:58.309522 preload-index.c:119          | d0 | main                     | region_enter |     |  0.004771 |           | index        | label:preload
13:58:58.368915 preload-index.c:161          | d0 | main                     | data         |     |  0.064163 |  0.059392 | index        | ..preload/sum_lstat:4050
13:58:58.369001 preload-index.c:162          | d0 | main                     | region_leave |     |  0.064250 |  0.059479 | index        | label:preload
13:58:58.372920 diffcore-rename.c:1358       | d0 | main                     | region_enter | r1  |  0.068167 |           | diff         | label:setup
13:58:58.372957 diffcore-rename.c:1411       | d0 | main                     | region_leave | r1  |  0.068206 |  0.000039 | diff         | label:setup
13:58:58.372981 diffcore-rename.c:1583       | d0 | main                     | region_enter | r1  |  0.068230 |           | diff         | label:write back to queue
13:58:58.373001 diffcore-rename.c:1660       | d0 | main                     | region_leave | r1  |  0.068251 |  0.000021 | diff         | label:write back to queue
13:58:58.374551 git.c:785                    | d0 | main                     | exit         |     |  0.069800 |           |              | code:0
13:58:58.374583 trace2/tr2_tgt_perf.c:213    | d0 | main                     | atexit       |     |  0.069832 |           |              | code:0
13:59:05.684250 common-main.c:48             | d0 | main                     | version      |     |           |           |              | 2.33.0.vfs.0.0.42.ga5b42902ab.dirty
13:59:05.685126 common-main.c:49             | d0 | main                     | start        |     |  0.002344 |           |              | ./git diff --staged
13:59:05.685923 git.c:523                    | d0 | main                     | cmd_name     |     |           |           |              | diff (diff)
13:59:05.686339 repository.c:132             | d0 | main                     | def_repo     | r1  |           |           |              | worktree:/Users/ldennington/repos/_git/sparse-index
13:59:05.688553 read-cache.c:2406            | d0 | main                     | region_enter | r1  |  0.005781 |           | index        | label:do_read_index .git/index
13:59:05.689162 read-cache.c:1807            | d0 | main                     | region_enter |     |  0.006390 |           | index        | ..label:read/extension/cache_tree
13:59:05.689195 cache-tree.c:659             | d0 | main                     | region_enter | r1  |  0.006422 |           | cache_tree   | ....label:read
13:59:05.689309 cache-tree.c:661             | d0 | main                     | region_leave | r1  |  0.006537 |  0.000115 | cache_tree   | ....label:read
13:59:05.689337 read-cache.c:1809            | d0 | main                     | data         |     |  0.006563 |  0.000173 | index        | ....read/extension/cache_tree/bytes:6693
13:59:05.689359 read-cache.c:1810            | d0 | main                     | region_leave |     |  0.006587 |  0.000197 | index        | ..label:read/extension/cache_tree
13:59:05.689391 read-cache.c:2360            | d0 | main                     | data         | r1  |  0.006619 |  0.000838 | index        | ..read/version:2
13:59:05.689417 read-cache.c:2362            | d0 | main                     | data         | r1  |  0.006646 |  0.000865 | index        | ..read/cache_nr:4051
13:59:05.689438 read-cache.c:2411            | d0 | main                     | region_leave | r1  |  0.006666 |  0.000885 | index        | label:do_read_index .git/index
13:59:05.689734 unpack-trees.c:1731          | d0 | main                     | region_enter |     |  0.006960 |           | unpack_trees | label:unpack_trees
13:59:05.689771 unpack-trees.c:1735          | d0 | main                     | region_enter | r1  |  0.006999 |           | unpack_trees | ..label:unpack_trees
13:59:05.689868 unpack-trees.c:1819          | d0 | main                     | region_enter | r1  |  0.007096 |           | unpack_trees | ....label:traverse_trees
13:59:05.690040 unpack-trees.c:1821          | d0 | main                     | region_leave | r1  |  0.007268 |  0.000172 | unpack_trees | ....label:traverse_trees
13:59:05.690077 unpack-trees.c:416           | d0 | main                     | region_enter |     |  0.007305 |           | unpack_trees | ....label:check_updates
13:59:05.690102 unpack-trees.c:504           | d0 | main                     | region_leave |     |  0.007330 |  0.000025 | unpack_trees | ....label:check_updates
13:59:05.690129 unpack-trees.c:1907          | d0 | main                     | region_leave | r1  |  0.007357 |  0.000358 | unpack_trees | ..label:unpack_trees
13:59:05.690151 unpack-trees.c:1910          | d0 | main                     | data         |     |  0.007379 |  0.000419 | unpack_trees | ..unpack_trees/nr_unpack_entries:0
13:59:05.690171 unpack-trees.c:1911          | d0 | main                     | region_leave |     |  0.007399 |  0.000439 | unpack_trees | label:unpack_trees
13:59:05.690201 diffcore-rename.c:1358       | d0 | main                     | region_enter | r1  |  0.007429 |           | diff         | label:setup
13:59:05.690222 diffcore-rename.c:1411       | d0 | main                     | region_leave | r1  |  0.007451 |  0.000022 | diff         | label:setup
13:59:05.690245 diffcore-rename.c:1583       | d0 | main                     | region_enter | r1  |  0.007473 |           | diff         | label:write back to queue
13:59:05.690265 diffcore-rename.c:1660       | d0 | main                     | region_leave | r1  |  0.007493 |  0.000020 | diff         | label:write back to queue
13:59:05.690330 git.c:785                    | d0 | main                     | exit         |     |  0.007558 |           |              | code:0
13:59:05.690364 tree-walk.c:185              | d0 | main                     | data_json    | r1  |  0.007592 |  0.007592 | traverse_tre | statistics:{"traverse_trees_count":1,"traverse_trees_max_depth":1}
13:59:05.690389 trace2/tr2_tgt_perf.c:213    | d0 | main                     | atexit       |     |  0.007617 |           |              | code:0

@ldennington ldennington closed this Sep 1, 2021
@ldennington ldennington deleted the ldennington/sparse-index-diff branch September 1, 2021 21:05
@ldennington ldennington self-assigned this Sep 1, 2021
@ldennington ldennington requested review from vdye and derrickstolee and removed request for derrickstolee and vdye September 1, 2021 21:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant