Rebase to v2.38.0 #533

While using the reset --stdin feature on windows path added may have a \r at the end of the path that wasn't getting removed so didn't match the path in the index and wasn't reset. Signed-off-by: Kevin Willford <kewillf@microsoft.com>

Signed-off-by: Saeed Noursalehi <sanoursa@microsoft.com>

Since we really want to be based on a `.vfs.*` tag, let's make sure that there was a new-enough one, i.e. one that agrees with the first three version numbers of the recorded default version. This prevents e.g. v2.22.0.vfs.0.<some-huge-number>.<commit> from being used when the current release train was not yet tagged. It is important to get the first three numbers of the version right because e.g. Scalar makes decisions depending on those (such as assuming that the `git maintenance` built-in is not available, even though it actually _is_ available). Signed-off-by: Johannes Schindelin <johasc@microsoft.com>

This header file will accumulate GVFS-specific definitions. Signed-off-by: Kevin Willford <kewillf@microsoft.com>

This does not do anything yet. The next patches will add various values for that config setting that correspond to the various features offered/required by GVFS. Signed-off-by: Kevin Willford <kewillf@microsoft.com>

This takes a substantial amount of time, and if the user is reasonably sure that the files' integrity is not compromised, that time can be saved. Git no longer verifies the SHA-1 by default, anyway. Signed-off-by: Kevin Willford <kewillf@microsoft.com>

Signed-off-by: Kevin Willford <kewillf@microsoft.com>

Prevent the sparse checkout to delete files that were marked with skip-worktree bit and are not in the sparse-checkout file. This is because everything with the skip-worktree bit turned on is being virtualized and will be removed with the change of HEAD. There was only one failing test when running with these changes that was checking to make sure the worktree narrows on checkout which was expected since we would no longer be narrowing the worktree. Update 2022-04-05: temporarily set 'sparse.expectfilesoutsideofpatterns' in test (until we start disabling the "remove present-despite-SKIP_WORKTREE" behavior with 'core.virtualfilesystem' in a later commit). Signed-off-by: Kevin Willford <kewillf@microsoft.com>

This adds hard-coded call to GVFS.hooks.exe before and after each Git command runs. To make sure that this is only called on repositories cloned with GVFS, we test for the tell-tale .gvfs. 2021-10-30: Recent movement of find_hook() to hook.c required moving these changes out of run-command.c to hook.c. Signed-off-by: Ben Peart <Ben.Peart@microsoft.com>

Suggested by Ben Peart. Signed-off-by: Johannes Schindelin <johasc@microsoft.com>

Signed-off-by: Alejandro Pauly <alpauly@microsoft.com>

While performing a fetch with a virtual file system we know that there will be missing objects and we don't want to download them just because of the reachability of the commits. We also don't want to download a pack file with commits, trees, and blobs since these will be downloaded on demand. This flag will skip the first connectivity check and by returning zero will skip the upload pack. It will also skip the second connectivity check but continue to update the branches to the latest commit ids. Signed-off-by: Kevin Willford <kewillf@microsoft.com>

We need to respect that config setting even if we already know that we have a repository, but have not yet read the config. The regression test was written by Alejandro Pauly. 2021-10-30: Recent movement of find_hook() into hook.c required moving this change from run-command.c. Signed-off-by: Johannes Schindelin <johasc@microsoft.com>

Ensure all filters and EOL conversions are blocked when running under GVFS so that our projected file sizes will match the actual file size when it is hydrated on the local machine. Signed-off-by: Ben Peart <Ben.Peart@microsoft.com>

When using the sparse-checkout feature, the file might not be on disk because the skip-worktree bit is on. Signed-off-by: Kevin Willford <kewillf@microsoft.com>

The idea is to allow blob objects to be missing from the local repository, and to load them lazily on demand. After discussing this idea on the mailing list, we will rename the feature to "lazy clone" and work more on this. Signed-off-by: Ben Peart <Ben.Peart@microsoft.com>

When using the sparse-checkout feature git should not write to the working directory for files with the skip-worktree bit on. With the skip-worktree bit on the file may or may not be in the working directory and if it is not we don't want or need to create it by calling checkout_entry. There are two callers of checkout_target. Both of which check that the file does not exist before calling checkout_target. load_current which make a call to lstat right before calling checkout_target and check_preimage which will only run checkout_taret it stat_ret is less than zero. It sets stat_ret to zero and only if !stat->cached will it lstat the file and set stat_ret to something other than zero. This patch checks if skip-worktree bit is on in checkout_target and just returns so that the entry doesn't not end up in the working directory. This is so that apply will not create a file in the working directory, then update the index but not keep the working directory up to date with the changes that happened in the index. Signed-off-by: Kevin Willford <kewillf@microsoft.com>

Hydrate missing loose objects in check_and_freshen() when running virtualized. Add test cases to verify read-object hook works when running virtualized. This hook is called in check_and_freshen() rather than check_and_freshen_local() to make the hook work also with alternates. Helped-by: Kevin Willford <kewillf@microsoft.com> Signed-off-by: Ben Peart <Ben.Peart@microsoft.com>

Signed-off-by: Kevin Willford <kewillf@microsoft.com>

If we are going to write an object there is no use in calling the read object hook to get an object from a potentially remote source. We would rather just write out the object and avoid the potential round trip for an object that doesn't exist. This change adds a flag to the check_and_freshen() and freshen_loose_object() functions' signatures so that the hook is bypassed when the functions are called before writing loose objects. The check for a local object is still performed so we don't overwrite something that has already been written to one of the objects directories. Based on a patch by Kevin Willford. Signed-off-by: Johannes Schindelin <johasc@microsoft.com>

On index load, clear/set the skip worktree bits based on the virtual file system data. Use virtual file system data to update skip-worktree bit in unpack-trees. Use virtual file system data to exclude files and folders not explicitly requested. Update 2022-04-05: disable the "present-despite-SKIP_WORKTREE" file removal behavior when 'core.virtualfilesystem' is enabled. Signed-off-by: Ben Peart <benpeart@microsoft.com>

This code change makes sure that the config value for core_gvfs is always loaded before checking it. Signed-off-by: Kevin Willford <kewillf@microsoft.com>

…x has been redirected Fixes #13 Some git commands spawn helpers and redirect the index to a different location. These include "difftool -d" and the sequencer (i.e. `git rebase -i`, `git cherry-pick` and `git revert`) and others. In those instances we don't want to update their temporary index with our virtualization data. Helped-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Ben Peart <Ben.Peart@microsoft.com>

The virtual file system code incorrectly treated symlinks as directories instead of regular files. This meant symlinks were not included even if they are listed in the list of files returned by the core.virtualFilesystem hook proc. Fixes #25 Signed-off-by: Ben Peart <Ben.Peart@microsoft.com>

String formatting can be a performance issue when there are hundreds of thousands of trees. Change to stop using the strbuf_addf and just add the strings or characters individually. There are a limited number of modes so added a switch for the known ones and a default case if something comes through that are not a known one for git. In one scenario regarding a huge worktree, this reduces the time required for a `git checkout <branch>` from 44 seconds to 38 seconds, i.e. it is a non-negligible performance improvement. Signed-off-by: Kevin Willford <kewillf@microsoft.com>

Add check to see if a directory is included in the virtualfilesystem before checking the directory hashmap. This allows a directory entry like foo/ to find all untracked files in subdirectories.

The following commands and options are not currently supported when working in a GVFS repo. Add code to detect and block these commands from executing. 1) fsck 2) gc 4) prune 5) repack 6) submodule 8) update-index --split-index 9) update-index --index-version (other than 4) 10) update-index --[no-]skip-worktree 11) worktree Signed-off-by: Ben Peart <benpeart@microsoft.com>

The vfs does not correctly handle the case when there is a file that begins with the same prefix as a directory. For example, the following setup would encounter this issue: A directory contains a file named `dir1.sln` and a directory named `dir1/`. The directory `dir1` contains other files. The directory `dir1` is in the virtual file system list The contents of `dir1` should be in the virtual file system, but it is not. The contents of this directory do not have the skip worktree bit cleared as expected. The problem is in the `apply_virtualfilesystem(...)` function where it does not include the trailing slash of the directory name when looking up the position in the index to start clearing the skip worktree bit. This fix is it include the trailing slash when finding the first index entry from `index_name_pos(...)`.

We found a user who had set "core.gvfs = false" in their global config. This should not have been necessary, but it also should not have caused a problem. However, it did. The reason is that gvfs_load_config_value() is called from config.c when reading config key/value pairs from all the config files. The local config should override the global config, and this is done by config.c reading the global config first then reading the local config. However, our logic only allowed writing the core_gvfs variable once. Put the guards against multiple assignments of core_gvfs into gvfs_config_is_set() instead, because that will fix the problem _and_ keep multiple calls to gvfs_config_is_set() from slowing down. Signed-off-by: Derrick Stolee <dstolee@microsoft.com>

When our patches to support that hook were upstreamed, the hook's name was eliciting some reviewer suggestions, and it was renamed to `post-index-change`. These patches (with the new name) made it into v2.22.0. However, VFSforGit users may very well have checkouts with that hook installed under the original name. To support this, let's just introduce a hack where we look a bit more closely when we just failed to find the `post-index-change` hook, and allow any `post-indexchanged` hook to run instead (if it exists).

Teach STATUS to optionally serialize the results of a status computation to a file. Teach STATUS to optionally read an existing serialization file and simply print the results, rather than actually scanning. This is intended for immediate status results on extremely large repos and assumes the use of a service/daemon to maintain a fresh current status snapshot. 2021-10-30: packet_read() changed its prototype in ec9a37d (pkt-line.[ch]: remove unused packet_read_line_buf(), 2021-10-14). 2021-10-30: sscanf() now does an extra check that "%d" goes into an "int" and complains about "uint32_t". Replacing with "%u" fixes the compile-time error. 2021-10-30: string_list_init() was removed by abf897b (string-list.[ch]: remove string_list_init() compatibility function, 2021-09-28), so we need to initialize manually. Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com> Signed-off-by: Derrick Stolee <dstolee@microsoft.com>

Teach status serialization to take an optional pathname on the command line to direct that cache data be written there rather than to stdout. When used this way, normal status results will still be written to stdout. When no path is given, only binary serialization data is written to stdout. Usage: git status --serialize[=<path>] Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>

Teach status deserialize code to reject status cache when printing in porcelain V2 and there are unresolved conflicts in the cache file. A follow-on task might extend the cache format to include this additiona data. See code for longer explanation. Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>

Fix "git status --deserialize" to correctly report both pathnames for renames. Add a test case to verify. A change was made upstream that added an additional "rename_status" field to the "struct wt_status_change_data" structure. It is used during the various print routines to decide if 2 pathnames need to be printed. 5134ccd wt-status.c: rename rename-related fields in wt_status_change_data The fix here is to add that field to the status cache data. Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>

Changes to the global or repo-local excludes files can change the results returned by "git status" for untracked files. Therefore, it is important that the exclude-file values used during serialization are still current at the time of deserialization. Teach "git status --serialize" to report metadata on the user's global exclude file (which defaults to "$XDG_HOME/git/ignore") and for the repo-local excludes file (which is in ".git/info/excludes"). Serialize will record the pathnames and mtimes for these files in the serialization header (next to the mtime data for the .git/index file). Teach "git status --deserialize" to validate this new metadata. If either exclude file has changed since the serialization-cache-file was written, then deserialize will reject the cache file and force a full/normal status run. Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>

Teach `git status --deserialize` to either wait indefintely or immediately fail if the status serialization cache file is stale. Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>

It took this developer more than a moment to verify that was_dirty() really returns 0 (i.e. "false") if the file was not even tracked. In other words, the `dirty` variable that was initialized to 1 (i.e. "true") and then negated to be returned was not helping readability. The same holds for the final return: rather than assigning the value to return to `dirty` and then *immediately* returning that, we can simplify it to a single statement.

It took this developer quite a good while to understand why the current code cannot get a `NULL` returned by `index_file_exists()`. To un-confuse readers (and future-proof the code), let's just be safe and check before we dereference the returned pointer.

The idea of the virtual file system really is to tell Git to avoid accessing certain paths. This fixes the case where a given path is not yet included in the virtual file system and we are about to write a conflicted version of it.

With the "--untracked-files=complete" option status computes a superset of the untracked files. We use this when writing the status cache. If subsequent deserialize commands ask for either the complete set or one of the "no", "normal", or "all" subsets, it can still use the cache file because of filtering in the deserialize parser. When running status with the "-uno" option, the long format status would print a "(use -u to show untracked files)" hint. When deserializing with the "-uno" option and using a cache computed with "-ucomplete", the "nothing to commit, working tree clean" message would be printed instead of the hint. It was easy to miss because the correct hint message was printed if the cache was rejected for any reason (and status did the full fallback). The "struct wt_status des" structure was initialized with the content of the status cache (and thus defaulted to "complete"). This change sets "des.show_untracked_files" to the requested subset from the command-line or config. This allows the long format to print the hint. Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>

When using a virtual file system layer, the FSMonitor does not make sense. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

Add trace2 region around read_object_process to collect time spent waiting for missing objects to be dynamically fetched. Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>

Add trace2 region and data events describing attempts to deserialize status data using a status cache. A category:status, label:deserialize region is pushed around the deserialize code. Deserialization results when reading from a file are: category:status, path = <path> category:status, polled = <number_of_attempts> category:status, result = "ok" | "reject" When reading from STDIN are: category:status, path = "STDIN" category:status, result = "ok" | "reject" Status will fallback and run a normal status scan when a "reject" is reported (unless "--deserialize-wait=fail"). If "ok" is reported, status was able to use the status cache and avoid scanning the workdir. Additionally, a cmd_mode is emitted for each step: collection, deserialization, and serialization. For example, if deserialization is attempted and fails and status falls back to actually computing the status, a cmd_mode message containing "deserialize" is issued and then a cmd_mode for "collect" is issued. Also, if deserialization fails, a data message containing the rejection reason is emitted. Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>

Fix crash in `git status -v` by setting `des_s->repo` to a non-null value. Upstream changes to eliminate use of `the_repository` added a `repo` field to `struct status`. And calls in `wt-status.c` to `repo_init_revisions()` were changed to pass `s->repo` rather than `the_repository`. The status deserialization code was not updated to actually set `s->repo` before common code passed the value to OID routines. This caused a segfault when verbose output was requested. Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>

Add trace information around status serialization. Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>

When using fsmonitor the CE_FSMONITOR_VALID flag should be checked when wanting to know if the entry has been updated. If the flag is set the entry should be considered up to date and the same as if the CE_UPTODATE is set. In order to trust the CE_FSMONITOR_VALID flag, the fsmonitor data needs to be refreshed when the fsmonitor bitmap is applied to the index in tweak_fsmonitor. Since the fsmonitor data is kept up to date for every command, some tests needed to be updated to take that into account. istate->untracked->use_fsmonitor was set in tweak_fsmonitor when the fsmonitor bitmap data was loaded and is now in refresh_fsmonitor since that is being called in tweak_fsmonitor. refresh_fsmonitor will only be called once and any other callers should be setting it when refreshing the fsmonitor data so that code can use the fsmonitor data when checking untracked files. When writing the index, fsmonitor_last_update is used to determine if the fsmonitor bitmap should be created and the extension data written to the index. When running through unpack-trees this is not copied to the result index. This makes the next time a git command is ran do all the work of lstating all files to determine what is clean since all entries in the index are marked as dirty since there wasn't any fsmonitor data saved in the index extension. Copying the fsmonitor_last_update to the result index will cause the extension data for fsmonitor to be in the index for the next git command to use. Signed-off-by: Kevin Willford <Kevin.Willford@microsoft.com>

Report virtual filesystem summary data. Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>

The fsmonitor script that can be used for running all the git tests using watchman was causing some of the tests to fail because it wrote to stderr and created some files for debugging purposes. Add a new debug script to use with debugging and modify the other script to remove the code that would cause tests to fail. Signed-off-by: Kevin Willford <Kevin.Willford@microsoft.com>

Create trace2_initialize_clock() and call from main() to capture process start time in isolation and before other sub-systems are ready. Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>

Disable deserialization when verbose output requested. Verbose mode causes Git to print diffs for modified files. This requires the index to be loaded to have the currently staged OID values. Without loading the index, verbose output make it look like everything was deleted. Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>

Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>

Verify that `git status --deserialize=x -v` does not crash and generates the same output as a normal (scanning) status command. These issues are described in the previous 2 commits. Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>

Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>

Teach Git to not throw a fatal error when an explicitly-specified status-cache file (`git status --deserialize=<foo>`) could not be found or opened for reading and silently fallback to a traditional scan. This matches the behavior when the status-cache file is implicitly given via a config setting. Note: the current version causes a test to start failing. Mark this as an expected result for now. Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com> Signed-off-by: Derrick Stolee <dstolee@microsoft.com>

Add trace2_thread_start() and trace2_thread_exit() events to the worker threads used to read the index. This gives per-thread perf data. These workers were introduced in: abb4bb8 read-cache: load cache extensions on a worker thread 77ff112 read-cache: load cache entries on worker threads Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>

…ension Add regions around code to read and write the cache-tree extension when the index is read or written. This is an experiment and may be dropped in future releases if we don't need it anymore. This experiment demonstrates that it takes more time to parse and deserialize the cache-tree extension than it does to read the cache-entries. Commits [1] and [2] spreads cache-entry reading across N-1 cores and dedicates a single core to simultaneously read the index extensions. Local testing (on my machine) shows that reading the cache-tree extension takes ~0.28 seconds. The 11 cache-entry threads take ~0.08 seconds. The main thread is blocked for 0.15 to 0.20 seconds waiting for the extension thread to finish. Let's use this commit to gather some telemetry and confirm this. My point is that improvements, such as index V5 which makes the cache entries smaller, may improve performance, but the gains may be limited because of this extension. And that we may need to look inside the cache-tree extension to truly improve do_read_index() performance. [1] abb4bb8 read-cache: load cache extensions on a worker thread [2] 77ff112 read-cache: load cache entries on worker threads Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>

Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>

…and report_tracking() Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>

Teach subprocess_start() to use a copy of the passed `cmd` string rather than borrowing the buffer from the caller. Some callers of subprocess_start() pass the value returned from find_hook() which points to a static buffer and therefore is only good until the next call to find_hook(). This could cause problems for the long-running background processes managed by sub-process.c where later calls to subprocess_find_entry() to get an existing process will fail. This could cause more than 1 long-running process to be created. TODO Need to confirm, but if only read_object_hook() uses TODO subprocess_start() in this manner, we could drop this TODO commit when we drop support for read_object_hook(). Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>

…data

Add function to start a subprocess with an argv. Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>

Add data for the number of files created/overwritten and deleted during the checkout. Give proper category name to all events in unpack-trees.c and eliminate "exp". This is modified slightly from the original version due to interactions with 26f924d (unpack-trees: exit check_updates() early if updates are not wanted, 2020-01-07). Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com> Signed-off-by: Derrick Stolee <dstolee@microsoft.com>

Create a function to add a new object to the loose object cache after the existing odb/xx/ directory was scanned. This will be used in a later commit to keep the loose object cache fresh after dynamically fetching an individual object and without requiring the odb/xx/ directory to be rescanned. Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>

Update tracing around report_tracking() to use 'tracking' category rather than 'exp' category. Add ahead/behind results from stat_tracking_info(). Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>

Create a function to install a new packfile into the packed-git list and add it to the head of the MRU list. This function will be used in a later commit to install packfiles created by dynamic object fetching. Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>

Prevent packfile parsing from accidentally dynamically fetching each individual object found in the packfile. When index-pack parses the input packfile, it does a lookup in the ODB to test for conflicts/collisions. This can accidentally cause the object to be individually fetched when gvfs-helper (or read-object-hook or partial-clone) is enabled. Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>

Create gvfs-helper. This is a helper tool to use the GVFS Protocol REST API to fetch objects and configuration data from a GVFS cache-server or Git server. This tool uses libcurl to send object requests to either server. This tool creates loose objects and/or packfiles. Create gvfs-helper-client. This code resides within git proper and uses the sub-process API to manage gvfs-helper as a long-running background process. Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com> Signed-off-by: Derrick Stolee <dstolee@microsoft.com>

When two gvfs-helper processes are the first to create a loose object directory, the processes (A and B in the timeline below) could have the following race: 1. A sees that the directory does not exist. 2. B sees that the directory does not exist. 3. A creates the directory with success. 4. B fails to create the directory and fails. Instead of having B fail here, just check for the directory's existence before reporting an error. That solves the race and allows tests to pass. Signed-off-by: Derrick Stolee <dstolee@microsoft.com>

The config variable `gvfs.sharedCache` contains the pathname to an alternate <odb> that will be used by `gvfs-helper` to store dynamically-fetched missing objects. If this directory does not exist on disk, `prepare_alt_odb()` omits this directory from the in-memory list of alternates. This causes `git` commands (and `gvfs-helper` in particular) to fall-back to `.git/objects` for storage of these objects. This disables the shared-cache and leads to poorer performance. Teach `alt_obj_usable()` and `prepare_alt_odb()`, match up the directory named in `gvfs.sharedCache` with an entry in `.git/objects/info/alternates` and force-create the `<odb>` root directory (and the associated `<odb>/pack` directory) if necessary. If the value of `gvfs.sharedCache` refers to a directory that is NOT listed as an alternate, create an in-memory alternate entry in the odb-list. (This is similar to how GIT_ALTERNATE_OBJECT_DIRECTORIES works.) This work happens the first time that `prepare_alt_odb()` is called. Furthermore, teach the `--shared-cache=<odb>` command line option in `gvfs-helper` (which is runs after the first call to `prepare_alt_odb()`) to override the inherited shared-cache (and again, create the ODB directory if necessary). Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>

Add trace2 message for CURL and HTTP errors. Fix typo reporting network error code back to gvfs-helper-client. Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>

Fix parsing of the "loose <odb>" response from `gvfs-helper` and use the actually parsed OID when updating the loose oid cache. Previously, an uninitialized "struct oid" was used to update the cache. This did not cause any corruption, but could cause extra fetches for objects visited multiple times. Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>

Add robust-retry mechanism to automatically retry a request after network errors. This includes retry after: [] transient network problems reported by CURL. [] http 429 throttling (with associated Retry-After) [] http 503 server unavailable (with associated Retry-After) Add voluntary throttling using Azure X-RateLimit-* hints to avoid being soft-throttled (tarpitted) or hard-throttled (429) on later requests. Add global (outside of a single request) azure-throttle data to track the rate limit hints from the cache-server and main Git server independently. Add exponential retry backoff. This is used for transient network problems when we don't have a Retry-After hint. Move the call to index-pack earlier in the response/error handling sequence so that if we receive a 200 but yet the packfile is truncated/corrupted, we can use the regular retry logic to get it again. Refactor the way we create tempfiles for packfiles to use <odb>/pack/tempPacks/ rather than working directly in the <odb>/pack/ directory. Move the code to create a new tempfile to the start of a single request attempt (initial and retry attempts), rather than at the overall start of a request. This gives us a fresh tempfile for each network request attempt. This simplifies the retry mechanism and isolates us from the file ownership issues hidden within the tempfile class. And avoids the need to truncate previous incomplete results. This was necessary because index-pack was pulled into the retry loop. Minor: Add support for logging X-VSS-E2EID to telemetry on network errors. Minor: rename variable: params.b_no_cache_server --> params.b_permit_cache_server_if_defined. This variable is used to indicate whether we should try to use the cache-server when it is defined. Got rid of double-negative logic. Minor: rename variable: params.label --> params.tr2_label Clarify that this variable is only used with trace2 logging. Minor: Move the code to automatically map cache-server 400 responses to normal 401 response earlier in the response/error handling sequence to simplify later retry logic. Minor: Decorate trace2 messages with "(cs)" or "(main)" to identify the server in log messages. Add params->server_type to simplify this. Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>

Expose the differences in the semantics of GET and POST for the "gvfs/objects" API: HTTP GET: fetches a single loose object over the network. When a commit object is requested, it just returns the single object. HTTP POST: fetches a batch of objects over the network. When the oid-set contains a commit object, all referenced trees are also included in the response. gvfs-helper is updated to take "get" and "post" command line options. the gvfs-helper "server" mode is updated to take "objects.get" and "objects.post" verbs. For convenience, the "get" option and the "objects.get" verb do allow more than one object to be requested. gvfs-helper will automatically issue a series of (single object) HTTP GET requests and creating a series of loose objects. The "post" option and the "objects.post" verb will perform bulk object fetching using the batch-size chunking. Individual HTTP POST requests containing more than one object will be created as a packfile. A HTTP POST for a single object will create a loose object. This commit also contains some refactoring to eliminate the assumption that POST is always associated with packfiles. In gvfs-helper-client.c, gh_client__get_immediate() now uses the "objects.get" verb and ignores any currently queued objects. In gvfs-helper-client.c, the OIDSET built by gh_client__queue_oid() is only processed when gh_client__drain_queue() is called. The queue is processed using the "object.post" verb. Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>

During development, it was very helpful to see the gvfs-helper do its work to request a pack-file or download a loose object. When these messages appear during normal use, it leads to a very noisy terminal output. Remove all progress indicators when downloading loose objects. We know that these can be numbered in the thousands in certain kinds of history calls, and would litter the terminal output with noise. This happens during 'git fetch' or 'git pull' as well when the tip commits are checked for the new refs. Remove the "Requesting packfile with %ld objects" message, as this operation is very fast. We quickly follow up with the more valuable "Receiving packfile %ld%ld with %ld objects". When a large "git checkout" causes many pack-file downloads, it is good to know that Git is asking for data from the server. Signed-off-by: Derrick Stolee <dstolee@microsoft.com>

Signed-off-by: Derrick Stolee <dstolee@microsoft.com>

If our POST request includes a commit ID, then the the remote will send a pack-file containing the commit and all trees reachable from its root tree. With the current implementation, this causes a failure since we call install_loose() when asking for one object. Modify the condition to check for install_pack() when the response type changes. Also, create a tempfile for the pack-file download or else we will have problems! Signed-off-by: Derrick Stolee <dstolee@microsoft.com>

Teach gvfs-helper to support "/gvfs/prefetch" REST API. This includes a new `gvfs-helper prefetch --since=<t>` command line option. And a new `objects.prefetch` verb in `gvfs-helper server` mode. If `since` argument is omitted, `gvfs-helper` will search the local shared-cache for the most recent prefetch packfile and start from there. The <t> is usually a seconds-since-epoch, but may also be a "friendly" date -- such as "midnight", "yesterday" and etc. using the existing date selection mechanism. Add `gh_client__prefetch()` API to allow `git.exe` to easily call prefetch (and using the same long-running process as immediate and queued object fetches). Expanded t5799 unit tests to include prefetch tests. Test setup now also builds some commits-and-trees packfiles for testing purposes with well-known timestamps. Expanded t/helper/test-gvfs-protocol.exe to support "/gvfs/prefetch" REST API. Massive refactor of existing packfile handling in gvfs-helper.c to reuse more code between "/gvfs/objects POST" and "/gvfs/prefetch". With this we now properly name packfiles with the checksum SHA1 rather than a date string. Refactor also addresses some of the confusing tempfile setup and install_<result> code processing (introduced to handle the ambiguity of how POST works with commit objects). Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>

Create t/helper/test-gvfs-protocol.c and t/t5799-gvfs-helper.sh to test gvfs-helper. Create t/helper/test-gvfs-protocol.c as a stand-alone web server that speaks the GVFS Protocol [1] and serves loose objects and packfiles to clients. It is borrows heavily from the code in daemon.c. It includes a "mayhem" mode to cause various network and HTTP errors to test the retry/recovery ability of gvfs-helper. Create t/t5799-gvfs-helper.sh to test gvfs-helper. [1] https://github.com/microsoft/VFSForGit/blob/master/Protocol.md Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>

Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>

gvfs-helper prints a "loose <oid>" or "packfile <name>" messages after they are received to help invokers update their in-memory caches. Move the code to accumulate these messages in the result_list into the install_* functions rather than waiting until the end. POST requests containing 1 object may return a loose object or a packfile depending on whether the object is a commit or non-commit. Delaying the message generation just complicated the caller. Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>

Signed-off-by: Derrick Stolee <dstolee@microsoft.com>

Earlier versions of the test always returned a packfile in response to a POST. Now we look at the number of objects in the POST request. If > 1, always send a packfile. If = 1 and it is a commit, send a packfile. Otherwise, send a loose object. This is to better model the behavior of the GVFS server/protocol which treats commits differently. Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>

Signed-off-by: Derrick Stolee <dstolee@microsoft.com>

Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>

The gvfs-helper allows us to download prefetch packs using a simple subprocess call. The gvfs-helper-client.h method will automatically compute the timestamp if passing 0, and passing NULL for the number of downloaded packs is valid. Signed-off-by: Derrick Stolee <dstolee@microsoft.com>

It is possible that a loose object that is written from a GVFS protocol "get object" request does not match the expected hash. Error out in this case. 2021-10-30: The prototype for read_loose_object() changed in 31deb28 (fsck: don't hard die on invalid object types, 2021-10-01) and 96e41f5 (fsck: report invalid object type-path combinations, 2021-10-01). Signed-off-by: Derrick Stolee <dstolee@microsoft.com>

Teach gvfs-helper to better support the concurrent fetching of the same packfile by multiple instances. If 2 instances of gvfs-helper did a POST and requested the same set of OIDs, they might receive the exact same packfile (same checksum SHA). Both processes would then race to install their copy of the .pack and .idx files into the ODB/pack directory. This is not a problem on Unix (because of filesystem semantics). On Windows, this can cause an EBUSY/EPERM problem for the loser while the winner is holding a handle to the target files. (The existing packfile code already handled simple the existence and/or replacement case.) The solution presented here is to silently let the loser claim victory IIF the .pack and .idx are already present in the ODB. (We can't check this in advance because we don't know the packfile SHA checksum until after we receive it and run index-pack.) We avoid using a per-packfile lockfile (or a single lockfile for the `vfs-` prefix) to avoid the usual issues with stale lockfiles. Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>

Teach helper/test-gvfs-protocol to be able to send corrupted loose blobs. Add unit test for gvfs-helper to detect receipt of a corrupted loose blob. Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>

When using the GVFS protocol, we should _never_ call "git fetch-pack" to attempt downloading a pack-file via the regular Git protocol. It appears that the mechanism that prevented this in the VFS for Git world is due to the read-object hook populating the commits at the new ref tips in a different way than the gvfs-helper does. By acting as if the fetch-pack succeeds here in remote-curl, we prevent a failed fetch. Signed-off-by: Derrick Stolee <dstolee@microsoft.com>

Signed-off-by: Derrick Stolee <dstolee@microsoft.com>

Add a GitHub workflow that is triggered on the `release` event to automatically update the `microsoft-git` Homebrew Cask on the `microsoft/git` Tap. A secret `HOMEBREW_TOKEN` with push permissions to the `microsoft/homebrew-git` repository must exist. A pull request will be created at the moment to allow for last minute manual verification. Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>

When we create temp files for downloading packs, we use a name based on the current timestamp. There is no randomness in the name, so we can have collisions in the same second. Retry the temp pack names using a new "-<retry>" suffix to the name before the ".temp". Signed-off-by: Derrick Stolee <dstolee@microsoft.com>

`sparse` complains with an error message like this: gvfs-helper.c:2912:17: error: expression using sizeof on a function The culprit is this line: curl_easy_setopt(slot->curl, CURLOPT_WRITEFUNCTION, fwrite); Similar lines exist in `http-push.c` and other files that are in upstream Git, and to avoid these bogus warnings, they are already exempted from `sparse`'s tender, loving care. We simply add `gvfs-helper.c` to that list. Signed-off-by: Derrick Stolee <dstolee@microsoft.com>

Test t5516-fetch-push.sh has a test 'deny fetch unreachable SHA1, allowtipsha1inwant=true' that checks stderr for a specific error string from the remote. In some build environments the error sent over the remote connection gets mingled with the error from the die() statement. Since both signals are being output to the same file descriptor (but from parent and child processes), the output we are matching with grep gets split. To reduce the risk of this failure, follow this process instead: 1. Write an error message to stderr. 2. Write an error message across the connection. 3. exit(1). This reorders the events so the error is written entirely before the client receives a message from the remote, removing the race condition. Signed-off-by: Derrick Stolee <dstolee@microsoft.com>

For Scalar and VFS for Git, we use an alternate as a shared object cache. We need to enable the maintenance builtin to work on that shared object cache, especially in the background. 'scalar run <task>' would set GIT_OBJECT_DIRECTORY to handle this. We set GIT_OBJECT_DIRECTORY based on the gvfs.sharedCache config, but we also need the checks in pack_loose() to look at that object directory instead of the current ODB's. Signed-off-by: Derrick Stolee <dstolee@microsoft.com>

When the virtualfilesystem is enabled the previous implementation of clear_ce_flags would iterate all of the cache entries and query whether each one is in the virtual filesystem to determine whether to clear one of the SKIP_WORKTREE bits. For each cache entry, we would do a hash lookup for each parent directory in the is_included_in_virtualfilesystem function. The former approach is slow for a typical Windows OS enlistment with 3 million files where only a small percentage is in the virtual filesystem. The cost is O(n_index_entries * n_chars_per_path * n_parent_directories_per_path). In this change, we use the same approach as apply_virtualfilesystem, which iterates the set of entries in the virtualfilesystem and searches in the cache for the corresponding entries in order to clear their flags. This approach has a cost of O(n_virtual_filesystem_entries * n_chars_per_path * log(n_index_entries)). The apply_virtualfilesystem code was refactored a bit and modified to clear flags for all names that 'alias' a given virtual filesystem name when ignore_case is set. n_virtual_filesystem_entries is typically much less than n_index_entries, in which case the new approach is much faster. We wind up building the name hash for the index, but this occurs quickly thanks to the multi-threading. Signed-off-by: Neeraj Singh <neerajsi@ntdev.microsoft.com>

This adds support for releasing to Ubuntu repositories hosted at http://packages.microsoft.com/ (hosting location for Microsoft's official apt/yum repos). This allows users to install via apt-get on Hirsute/Bionic. Details to configure appropriate repos can be found here: https://docs.microsoft.com/en-us/windows-server/administration/Linux-Package-Repository-for-Microsoft-Software).

It really does not make sense to run that workflow in any fork of git-for-windows/git. Typically, it is enough to simply disable it (since it is a scheduled workflow, it is disabled by default in any new fork). However, in microsoft/git, we switch the default branch whenever we rebase to a new upstream version, and every time we do so, this scheduled workflow gets re-enabled. Let's just delete it in microsoft/git and never be bothered by it again. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

- trigger on tag matching basic "vfs" version pattern - validate tag is annotated & matches stricter checks - include `scalar` - build x86_64 & portable git installers, upload artifacts to workflow Update Apr 18, 2022: these steps are built explicitly on 'windows-2019' agents (rather than 'windows-latest') to ensure the correct version of Visual Studio is used (verified in the pipeline via 'type -p mspdb140.dll'). Additionally, due to a known (but not-yet-fixed) issue downloading the 'build-installers' flavor of the Git for Windows SDK with the 'git-for-windows/setup-git-for-windows-sdk' Action, the SDK used is the 'full' flavor.

- include `scalar` - build signed .dmg & .pkg for target OS version 10.6 - upload artifacts to workflow Update 2022-08-11: .dmg and .pkg are now signed

- include `scalar` - build & upload unsigned .deb package

Allow specification of a custom `GIT_BUILT_FROM_COMMIT` string to replace the output of `git rev-parse HEAD`. This allows a build of `git` from somewhere other than an active clone of `git` (e.g. from the archive created with `make dist`) to include commit information in `git version --build-options`. Signed-off-by: Victoria Dye <vdye@github.com>

- sign using Azure-stored certificates & client - sign on Windows agent via python script - job skipped if credentials for accessing certificate aren't present

Update `git archive` tree-ish argument from `HEAD^{tree}` to `HEAD`. By using a commit (rather than tree) reference, the commit hash will be stored as an extended pax header, extractable git `git get-tar-commit-id`. The intended use-case for this change is building `git` from the output of `make dist` - in combination with the ability to specify a fallback `GIT_BUILT_FROM_COMMIT`, a user can extract the commit ID used to build the archive and set it as `GIT_BUILT_FROM_COMMIT`. The result is fully-populated information for the commit hash in `git version --build-options`. Signed-off-by: Victoria Dye <vdye@github.com>

Just do the boilerplate stuff of making a new builtin, including documentation and integration with git.c. Signed-off-by: Derrick Stolee <dstolee@microsoft.com>

- create release & uploads artifact using Octokit - use job "if" condition to handle uploading signed *or* unsigned .deb

Set the `GIT_BUILT_FROM_COMMIT` based on the version specified in the `make dist` output archive header. This ensures the commit hash is shown in `git version --build-options`. Signed-off-by: Victoria Dye <vdye@github.com>

On Windows, we have the 'git update-git-for-windows' command. It is poorly named within the microsoft/git fork, because the script has been updated to look at the GitHub releases of microsoft/git, not git-for-windows/git. Still, it handles all the complicated details about downloading, verifying, and running the installer. Signed-off-by: Derrick Stolee <dstolee@microsoft.com>

In 'git-update-git-for-windows', there is a recently_seen variable that is loaded from Git config. This is intended to allow users to say "No, I don't want that version of Git for Windows." If users say no, then they are not reminded. Ever. We want users of microsoft/git to be notified repeately until they upgrade. The first notification might be dismissed because they don't want to interrupt their work. They should get the picture within a few reminders and upgrade in a timely fashion. Signed-off-by: Derrick Stolee <dstolee@microsoft.com>

Add basic installer validation to release pipeline for Windows, macOS, and Linux (Debian package only). Validation runs the installers/any necessary setup and checks that the installed version matches the expected version.

We have been using the default issue template from git-for-windows/git, but we should ask different questions than Git for Windows. Update the issue template to ask these helpful questions. Signed-off-by: Derrick Stolee <derrickstolee@github.com>

In particular when multiple processes want to write to the config simultaneously, it would come in handy to not fail immediately when another process locked the config, but to gently try again. This will help with Scalar's functional test suite which wants to register multiple repositories for maintenance semi-simultaneously. As not all code paths calling this function read the config (e.g. `git config`), we have to read the config setting via `git_config_get_ulong()`. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

We have long inherited the pull request template from git-for-windows/git, but we should probably do a better job of specifying the need for why a PR in microsoft/git exists instead of an upstream contribution. Signed-off-by: Derrick Stolee <derrickstolee@github.com>

The 'git maintenance unregister' subcommand has a step that removes the current repository from the multi-valued maitenance.repo config key. This fails if the repository is not listed in that key. This makes running 'git maintenance unregister' twice result in a failure in the second instance. Make this task idempotent, since the end result is the same in both cases: maintenance will no longer run on this repository. Signed-off-by: Derrick Stolee <derrickstolee@github.com>

By default, Git fails immediately when locking a config file for writing fails due to an existing lock. With this change, Scalar-registered repositories will fall back to trying a couple times within a 150ms timeout. Signed-off-by: Johannes Schindelin <johasc@microsoft.com>

The steps to update the microsoft-git cask are: 1. brew update 2. brew upgrade --cask microsoft-git This is adapted from the UpgradeVerb within microsoft/scalar. There is one important simplification: Scalar needed to check 'brew list --cask' to find out if the 'scalar' cask or the 'scalar-azrepos' cask was installed (which determined if the 'microsoft-git' cask was a necessary dependency). We do not need that here, since we are already in the microsoft-git cask. Signed-off-by: Derrick Stolee <dstolee@microsoft.com>

Microsoft's fork of Git is not quite Git for Windows, therefore we want to tell the keen reader all about it. :-) Signed-off-by: Derrick Stolee <dstolee@microsoft.com>

This bug fix is required so that the `scalar unregister` tests pass again (this fix is required due to a change in the Scalar code that was done in response to reviewers' feedback). Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

These docs have been altered to fit the version implemented in C within microsoft/git. This means in particular that the advanced.md file no longer applied at all. Some other areas were removed or significantly edited. Signed-off-by: Derrick Stolee <dstolee@microsoft.com>

Allow concurrent `scalar register` and `scalar unregister` calls to be more collaborative when trying to lock the global Git config at the very same time. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

Adding the extra documentation from the Scalar project. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

Git traditionally uses those, not backslashes, ever. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

Use a fixed 3 tries total to see how that increases our chances of success for subcommands such as 'git fetch'. Signed-off-by: Derrick Stolee <dstolee@microsoft.com>

The .NET version supported running `scalar config` to reconfigure the current enlistment, and now the C port does, too. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

We actually only need the FSMonitor... Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

This merges the upstreamable part of the Scalar patches. Minor merge conflicts (caused by the gvfs-helper) were resolved trivially. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

Upstream, a20f704 (add: warn when asked to update SKIP_WORKTREE entries, 2021-04-08) modified how 'git add <pathspec>' works with cache entries marked with the SKIP_WORKTREE bit. The intention is to prevent a user from accidentally adding a path that is outside their sparse-checkout definition but somehow matches an existing index entry. A similar change for 'git rm' happened in d5f4b82 (rm: honor sparse checkout patterns, 2021-04-08). This breaks when using the virtual filesystem in VFS for Git. It is rare, but we could be in a scenario where the user has staged a change and then the file is projected away. If the user re-adds the file, then this warning causes the command to fail with the advise message. Disable this logic when core_virtualfilesystem is enabled. Signed-off-by: Derrick Stolee <dstolee@microsoft.com>

The clean_tracked_sparse_directories() method deletes the tracked directories that go out of scope when the sparse-checkout cone changes, at least in cone mode. This is new behavior, but is recommended based on our understanding of how users are interacting with the feature in most cases. It is possible that some users will object to the new behavior, so create a new configuration option 'index.deleteSparseDirectories' that can be set to 'false' to make clean_tracked_sparse_directories() do nothing. This will keep all untracked files in the working tree and cause performance problems with the sparse index, but those trade-offs are for the user to decide. Signed-off-by: Derrick Stolee <dstolee@microsoft.com>

The diff_populate_filespec() method is used to describe the diff after a merge operation is complete, especially when a conflict appears. In order to avoid expanding a sparse index, the reuse_worktree_file() needs to be adapted to ignore files that are outside of the sparse-checkout cone. The file names and OIDs used for this check come from the merged tree in the case of the ORT strategy, not the index, hence the ability to look into these paths without having already expanded the index. Signed-off-by: Derrick Stolee <dstolee@microsoft.com>

… virtual Upstream, a20f704 (add: warn when asked to update SKIP_WORKTREE entries, 04-08-2021) modified how 'git add <pathspec>' works with cache entries marked with the SKIP_WORKTREE bit. The intention is to prevent a user from accidentally adding a path that is outside their sparse-checkout definition but somehow matches an existing index entry. This breaks when using the virtual filesystem in VFS for Git. It is rare, but we could be in a scenario where the user has staged a change and then the file is projected away. If the user re-adds the file, then this warning causes the command to fail with the advise message. Disable this logic when core_virtualfilesystem is enabled. This should allow the VFS for Git functional tests to pass (at least the ones in the default run). I'll create a `-pr` installer build to check before merging this.

Signed-off-by: Derrick Stolee <dstolee@microsoft.com>

There is some strangeness when expanding a sparse-index that exists within a submodule. We will need to resolve that later, but for now, let's do a better job of explicitly disabling the sparse-index when requested, and do so in t7817. Signed-off-by: Derrick Stolee <dstolee@microsoft.com>

``` 6e74958 p2000: add 'git checkout -' test and decrease depth 3e1d03c p2000: compress repo names cd94f82 commit: integrate with sparse-index 65e79b8 sparse-index: recompute cache-tree e9a9981 checkout: stop expanding sparse indexes 4b801c8 t1092: document bad 'git checkout' behavior 71e3015 unpack-trees: resolve sparse-directory/file conflicts 5e96df4 t1092: test merge conflicts outside cone defab1b add: allow operating on a sparse-only index 9fc4313 pathspec: stop calling ensure_full_index 0ec03ab add: ignore outside the sparse-checkout in refresh() adf5b15 add: remove ensure_full_index() with --renormalize ``` These commits are equivalent to those already in `next` via gitgitgadget#999. ``` 80b8d6c Merge branch 'sparse-index/add' into stolee/sparse-index/add ``` This merge resolves conflicts with some work that happened in parallel, but is already in upstream `master`. ``` c407b2c t7519: rewrite sparse index test 9dad0d2 sparse-index: silently return when not using cone-mode patterns 2974920 sparse-index: silently return when cache tree fails e7cdaa0 unpack-trees: fix nested sparse-dir search 347410c sparse-checkout: create helper methods 4537233 attr: be careful about sparse directories 5282a86 sparse-index: add SPARSE_INDEX_MEMORY_ONLY flag 3a2f316 sparse-checkout: clear tracked sparse dirs fb47b56 sparse-checkout: add config to disable deleting dirs ``` These commits are the ones under review as of gitgitgadget#1009. Recent review made this less stable. It's a slightly different and more robust version of msysgit#396. > Note: I'm still not done with the feedback for upstream, but the remaining feedback is "can we add tests that cover these tricky technical bits?" and in `microsoft/git` these are already covered by the Scalar functional tests (since that's how they were found). ``` 080b02c diff: ignore sparse paths in diffstat d91a647 merge: make sparse-aware with ORT df49b5f merge-ort: expand only for out-of-cone conflicts cdecb85 t1092: add cherry-pick, rebase tests 0c1ecfb sequencer: ensure full index if not ORT strategy 406dfbe sparse-index: integrate with cherry-pick and rebase ``` These commits integrate with `git merge`, `git cherry-pick`, `git revert`, and `git rebase` as of gitgitgadget#1019. This got some feedback that changed how the tests were working so they are more robust. This led to a new commit (0c1ecfb). ``` cbb0ab3 Merge branch 'sparse-index/merge' into vfs-2.33.0 acb8623 t7524: test no longer fails ``` Finally, the commits are merged into `vfs-2.33.0` and also we include a fix to a `microsoft/git` test that is no longer broken.

This verifies that `diff` and `diff --staged` behave the same in sparse index repositories in the following partially-staged scenarios (i.e. the index, HEAD, and working directory differ at a given path): 1. Path is within sparse-checkout cone. 2. Path is outside sparse-checkout cone. 3. A merge conflict exists for paths outside sparse-checkout cone. Signed-off-by: Lessley Dennington <lessleydennington@gmail.com>

This branch is exactly msysgit#410, but with one more commit: enabling the sparse index by default in d59110a. Having this in the `vfs-2.33.0` branch helps build confidence that the sparse index is doing what it should be doing by running in the Scalar functional tests and in our test branches. If we want to cut a new `microsoft/git` release without enabling the sparse index, we can simply revert this commit.

Test cases specific to handling untracked files in `git stash` a) ensure that files outside the sparse checkout definition are handled as-expected and b) document the index expansion inside of `git stash -u`. Note that, in b), it is not the full repository index that is expanded - it is the temporary, standalone index containing the stashed untracked files only. Signed-off-by: Victoria Dye <vdye@github.com>

diff: enable and test the sparse index

During a run of the Scalar functional tests, we hit a case where the inexact rename detection of a 'git cherry-pick' command slowed to the point of writing its delayed progress, failing the test because stderr differed from the control case. Showing progress like this when stderr is not a terminal is non-standard for Git, so inject an isatty(2) when initializing the progress option in sequencer.c. Unfortunately, there is no '--quiet' option in 'git cherry-pick' currently wired up. This could be considered in the future, and the isatty(2) could be moved to that position. This would also be needed for commands like 'git rebase', so we leave that for another time. Reported-by: Victoria Dye <vdye@github.com> Signed-off-by: Derrick Stolee <dstolee@microsoft.com>

Sparse index: integrate with `clean` and `stash -u`

…n stderr is redirected During a run of the Scalar functional tests, we hit a case where the inexact rename detection of a 'git cherry-pick' command slowed to the point of writing its delayed progress, failing the test because stderr differed from the control case. Showing progress like this when stderr is not a terminal is non-standard for Git, so inject an isatty(2) when initializing the progress option in sequencer.c. Unfortunately, there is no '--quiet' option in 'git cherry-pick' currently wired up. This could be considered in the future, and the isatty(2) could be moved to that position. This would also be needed for commands like 'git rebase', so we leave that for another time.

The maintenance.lock file exists to prevent concurrent maintenance processes from writing to a repository at the same time. However, it has the downside of causing maintenance to start failing without recovery if there is any reason why the maintenance command failed without cleaning up the lock-file. This change makes it such that maintenance will delete a lock file that was modified over 6 hours ago. This will auto-heal repositories that are stuck with failed maintenance (and maybe it will fail again, but we will get a message other than the lock file exists). Signed-off-by: Derrick Stolee <dstolee@microsoft.com>

* t1092: remove the 'git update-index' test that currently fails because the command ignores the bad path, but doesn't return a failure. * dir.c: prevent matching against sparse-checkout patterns when the virtual filesystem is enabled. Should prevent some corner case issues. * t1092: add quiet mode for some rebase tests because the stderr output can change in some of the modes. Signed-off-by: Derrick Stolee <dstolee@microsoft.com>

…k files, fix loose-objects task The maintenance.lock file exists to prevent concurrent maintenance processes from writing to a repository at the same time. However, it has the downside of causing maintenance to start failing without recovery if there is any reason why the maintenance command failed without cleaning up the lock-file. This change makes it such that maintenance will delete a lock file that was modified over 6 hours ago. This will auto-heal repositories that are stuck with failed maintenance (and maybe it will fail again, but we will get a message other than the lock file exists). The intention here is to get users out of a bad state without weakening the maintenance lock _too_ much. I'm open to other thoughts, including expanding the timeframe from 6 to 24 hours. --- The `fixup!` commit fixes the use of the `gvfs.sharedcache` config value around the loose-objects task. Specifically, the loose-objects step might fail because it is calling `git pack-objects {garbage}/pack/loose` which fails.

During the 2.35.0 rebase, we ejected 570f64b (Fix reset when using the sparse-checkout feature., 2017-03-15) because of a similar change upstream that actually works with the expected behavior of sparse-checkout. That commit only ever existed in microsoft/git, but when it was considered for upstream we realized that it behaved strangely for a sparse-checkout scenario. The root problem is that during a mixed reset, 'git reset <commit>' updates the index to aggree with <commit> but leaves the worktree the same as it was before. The issue with sparse-checkout is that some files might not be in the worktree and thus the information from those files would be "lost". The upstream decision was to leave these files as ignored, because that's what the SKIP_WORKTREE bit means: don't put these files in the worktree and ignore their contents. If there already were files in the worktree, then Git does not change them. The case for "losing" data is if a committed change outside of the sparse-checkout was in the previous HEAD position. However, this information could be recovered from the reflog. The case where this is different is in a virtualized filesystem. The virtualization is projecting the index contents onto the filesystem, so we need to do something different here. In a virtual environment, every file is considered "important" and we abuse the SKIP_WORKTREE bit to indicate that Git does not need to process a projected file. When a file is populated, the virtual filesystem hook provides the information for removing the SKIP_WORKTREE bit. In the case of these mixed resets, we have the issue where we change the projection of the worktree for these cache entries that change. If a file is populated in the worktree, then the populated file will persist and appear in a follow-up 'git status'. However, if the file is not populated and only projected, we change the projection from the current value to the new value, leaving a clean 'git status'. The previous version of this commit includes a call to checkout_entry(), which populates the file. This causes the file to be actually in the working tree and no longer projected. To make this work with the upstream changes, stop setting the skip-worktree bit for the new cache entry. This seemed to work fine without this change, but it's likely due to some indirection with the virtual filesystem. Better to do the best-possible thing here so we don't hide a corner-case bug by accident. Helped-by: Victoria Dye <vdye@github.com> Signed-off-by: Kevin Willford <kewillf@microsoft.com> Signed-off-by: Derrick Stolee <derrickstolee@github.com>

These were done in private, before microsoft/git. Signed-off-by: Derrick Stolee <dstolee@microsoft.com>

…sing virtual filesystem This replaces git-for-windows#493 (can't reopen a PR after a force-push...). I updated this commit with a more firm version of the fix. This hopefully answers Victoria's excellent concerns with the previous approach. I did not manage to get an automated test for this, but I did carefully verify this manually with a few commits in a VFS for Git enlistment (with different files every time). I updated the commit message with more details about why this works. --- This fork contains changes specific to monorepo scenarios. If you are an external contributor, then please detail your reason for submitting to this fork: * [X] This change only applies to the virtualization hook and VFS for Git. Resolves git-for-windows#490.

Most of these were done in private before microsoft/git. However, the following pull requests modified the core feature: msysgit#85 msysgit#89 msysgit#91 msysgit#98 msysgit#243 msysgit#263 Signed-off-by: Derrick Stolee <dstolee@microsoft.com>

Add virtual file system settings and hook proc. On index load, clear/set the skip worktree bits based on the virtual file system data. Use virtual file system data to update skip-worktree bit in unpack-trees. Use virtual file system data to exclude files and folders not explicitly requested. The hook was first contributed in private, but was extended via the following pull requests: #15 #27 #33 msysgit#70 Signed-off-by: Ben Peart <Ben.Peart@microsoft.com> Signed-off-by: Derrick Stolee <dstolee@microsoft.com>

Includes these pull requests: #1 #6 #10 #11 msysgit#157 msysgit#212 msysgit#260 msysgit#270 Signed-off-by: Derrick Stolee <dstolee@microsoft.com>

Includes gvfs-specific commits from these pull requests: msysgit#158 msysgit#159 msysgit#160 msysgit#164 Signed-off-by: Derrick Stolee <dstolee@microsoft.com>

Includes commits from these pull requests: msysgit#191 msysgit#205 msysgit#206 msysgit#207 msysgit#208 msysgit#215 msysgit#220 msysgit#221 Signed-off-by: Derrick Stolee <dstolee@microsoft.com>

Includes these pull requests: msysgit#227 msysgit#228 msysgit#229 msysgit#231 msysgit#240 Signed-off-by: Derrick Stolee <dstolee@microsoft.com>

Includes commits from these pull requests: msysgit#188 Signed-off-by: Derrick Stolee <dstolee@microsoft.com>

…tream This PR updates our `vfs-2.29.0` branch's version of `git maintenance` to match the latest in upstream. Unfortunately, not all of these commits made it to the `2.30` release candidate, but there are more commits from the series making it in. They will cause a conflict in the `vfs-2.30.0` rebase, so merge them in here. This also includes the `fixup!` reverts of the earlier versions. Finally, I also noticed that we started depending on `git maintenance start` in Scalar for macOS, but we never checked that this worked with the shared object cache. It doesn't! 😨 The very tip commit of this PR includes logic to make `git maintenance run` care about `gvfs.sharedCache`. Functional test updates in Scalar will follow.

…ve efficiency of clear_ce_flags When the virtualfilesystem is enabled the previous implementation of clear_ce_flags would iterate all of the cache entries and query whether each one is in the virtual filesystem to determine whether to clear one of the SKIP_WORKTREE bits. For each cache entry, we would do a hash lookup for each parent directory in the is_included_in_virtualfilesystem function. The former approach is slow for a typical Windows OS enlistment with 3 million files where only a small percentage is in the virtual filesystem. The cost is O(n_index_entries * n_chars_per_path * n_parent_directories_per_path). In this change, we use the same approach as apply_virtualfilesystem, which iterates the set of entries in the virtualfilesystem and searches in the cache for the corresponding entries in order to clear their flags. This approach has a cost of O(n_virtual_filesystem_entries * n_chars_per_path * log(n_index_entries)). The apply_virtualfilesystem code was refactored a bit and modified to clear flags for all names that 'alias' a given virtual filesystem name when ignore_case is set. n_virtual_filesystem_entries is typically much less than n_index_entries, in which case the new approach is much faster. We wind up building the name hash for the index, but this occurs quickly thanks to the multi-threading.

Adding a few workflows to publish releases! 🥳

…ponents Disable the `monitor-components` workflow in msft-git

Implement workflow to create GitHub release with attached `git` installers

Fixes for MacOS release build & build options

This adds a new builtin, `git update-microsoft-git`, that executes the platform-specific upgrade steps to get the latest version of `microsoft-git`. On Windows, this means running `git update-git-for-windows` which was updated to use the `microsoft/git` releases page, when appropriate. See msysgit#321 for details. On macOS, this means running a sequence of `brew` commands. These are adapted from the `UpgradeVerb` in `microsoft/scalar`, with an important simplification: we don't need to differentiate between the `scalar` and `scalar-azrepos` cask.

Signed-off-by: Derrick Stolee <dstolee@microsoft.com>

No grown-up C project comes without their own JSON parser. Just kidding! We need to parse a JSON result when determining which cache server to use. It would appear that searching for needles `"CacheServers":[`, `"Url":"` and `"GlobalDefault":true` _happens_ to work right now, it is fragile as it depends on no whitespace padding and on the order of the fields remaining as-is. Let's implement a super simple JSON parser (at the cost of being slightly inefficient) for that purpose. To avoid allocating a ton of memory, we implement a callback-based one. And to save on complexity, let's not even bother validating the input properly (we will just go ahead and instead rely on Azure Repos to produce correct JSON). Note: An alternative would have been to use existing solutions such as JSON-C, CentiJSON or JSMN. However, they are all a lot larger than the current solution; The smallest, JSMN, which does not even provide parsed string values (something we actually need) weighs in with 471 lines, while we get away with 182 + 29 lines for the C and the header file, respectively. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

With this change, we come a big step closer to feature parity with Scalar: this allows cloning from Azure Repos (which do not support partial clones at time of writing). We use the just-implemented JSON parser to parse the response we got from the `gvfs/config` endpoint; Please note that this response might, or might not, contain information about a cache server. The presence or absence of said cache server, however, has nothing to do with the ability to speak the GVFS protocol (but the presence of the `gvfs/config` endpoint does that). An alternative considered during the development of this patch was to perform simple string matching instead of parsing the JSON-formatted data; However, this would have been fragile, as the response contains free-form text (e.g. the repository's description) which might contain parts that would confuse a simple string matcher (but not a proper JSON parser). Note: we need to limit the re-try logic in `git clone` to handle only the non-GVFS case: the call to `set_config()` to un-set the partial clone settings would otherwise fail because those settings would not exist in the GVFS protocol case. This will at least give us a clearer reason why such a fetch fails. Co-authored-by: Derrick Stolee <dstolee@microsoft.com> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Derrick Stolee <dstolee@microsoft.com>

This comes in handy, as we want to verify that `scalar clone` also works against a GVFS-enabled remote repository. Note that we have to set `MSYS2_ENV_CONV_EXCL` to prevent MSYS2 from mangling `PATH_TRANSLATED`: The value _does_ look like a Unix-style path, but no, MSYS2 must not be allowed to convert that into a Windows path: `http-backend` needs it in the unmodified form. (The MSYS2 runtime comes in when `git` is run via `bin-wrappers/git`, which is a shell script.) Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

We already have the `config` command that accesses the `gvfs/config` endpoint. To implement `scalar`, we also need to be able to access the `vsts/info` endpoint. Let's add a command to do precisely that. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

On Windows, both the forward slash and the backslash are directory separators. Which means that `a\b\c` really is inside `a/b`. Therefore, we need to special-case the directory separators in the helper function `cmp_icase()` that is used in the loop in `dir_inside_of()`. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

Modified to remove call to is_unattended() that has not been implemented yet. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

This finalizes the port of the `QueryVstsInfo()` function: we already taught `gvfs-helper` to access the `vsts/info` endpoint on demand, we implemented proper JSON parsing, and now it is time to hook it all up. To that end, we also provide a default local cache root directory. It works the same way as the .NET version of Scalar: it uses C:\scalarCache on Windows, ~/.scalarCache/ on macOS and ~/.cache/scalar on Linux Modified to include call to is_unattended() that was removed from a previous commit. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

Well, technically also the http:// protocol is allowed _when testing_... Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

Azure Repos does not support partial clones at the moment, but it does support the GVFS protocol. To that end, the Microsoft fork of Git has a `gvfs-helper` command that is optionally used to perform essentially the same functionality as partial clone. Let's verify that `scalar clone` detects that situation and enables the GVFS helper. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

This allows setting the GVFS-enabled cache server, or listing the one(s) associated with the remote repository. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

In Scalar's functional tests, we do not do anything with authentication. Therefore, we do want to avoid accessing the `vsts/info` endpoint because it requires authentication even on otherwise public repositories. Let's introduce the environment variable `SCALAR_TEST_SKIP_VSTS_INFO` which can be set to `true` to simply skip that step (and force the `url_*` style repository IDs instead of `id_*` whenever possible). Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

Scalar in Microsoft's Git fork can do a little more than Scalar in upstream Git: in Microsoft's Git, it supports the GVFS protocol so that Scalar can clone from Azure DevOps. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

Signed-off-by: Derrick Stolee <dstolee@microsoft.com>

Those tests specifically verify that the `.zip` file path is shown on `stdout`. Let's do that again, under the assumption that there are scripts out there that rely on this behavior. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

…atibility This option does not do anything anymore, though. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

Scalar's Functional Test suite is pretty comprehensive, and caught more than just one bug in the built-in FSMonitor that was missed by Git's own test suite. To benefit from this test suite, automatically run it on the `vfs-*` and `features/*` branches. Note: for simplicity, we're building Git from scratch in all matrix jobs. Also note: for speed, we are using `git-sdk-64-minimal`, even if it lacks the `/bin/install` that we need to install Git's files; We're providing a minimal shell script shim instead. Also, we do not need to bother with the Tcl/Tk parts, therefore we're skipping them, too. Finally, we use GIT_FORCE_UNTRACKED_CACHE in the functional tests, to give the untracked cache a thorough work-out. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

Prepare `scalar` to use the GVFS protocol instead of partial clone (required to support Azure Repos). Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

There are many reasons why discovering a Git directory may fail. In particular, 8959555 (setup_git_directory(): add an owner check for the top-level directory, 2022-03-02) added ownership checks as a security precaution. Callers attempting to set up a Git directory may want to inform the user about the reason for the failure. For that, expose the enum discovery_result from within setup.c and into cache.h where discover_git_directory() is defined. I initially wanted to change the return type of discover_git_directory() to be this enum, but several callers rely upon the "zero means success". The two problems with this are: 1. The zero value of the enum is actually GIT_DIR_NONE, so nonpositive results are errors. 2. There are multiple successful states, so some positive results are successful. Instead of updating all callers immediately, add a new method, discover_git_directory_reason(), and convert discover_git_directory() to be a thin shim on top of it. Because there are extra checks that discover_git_directory_reason() does after setup_git_directory_gently_1(), there are other modes that can be returned for failure states. Add these modes to the enum, but be sure to explicitly add them as BUG() states in the switch of setup_git_directory_gently(). Signed-off-by: Derrick Stolee <derrickstolee@github.com>

When FSMonitor was upstreamed, the 'core.useBuiltinFSMonitor' config was deprecated and replaced with an overload of the 'core.fsmonitor' config (i.e., if a boolean value was specified in 'core.fsmonitor', it is treated the way 'core.useBuiltinFSMonitor' originally was). Because 'scalar register' actively sets that config, use it to upgrade the deprecated config setting. Co-authored-by: Johannes Schindelin <Johannes.Schindelin@gmx.de> Signed-off-by: Victoria Dye <vdye@github.com>

Run the comprehensive Scalar Functional Tests as part of CI and PR builds. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

When running 'scalar reconfigure -a', such as at install time, Scalar has warning messages about the repository missing (or not containing a .git directory). Failures can also happen while trying to modify the repository-local config for that repository. These warnings may seem confusing to users who don't understand what they mean or how to stop them. Add a warning that instructs the user how to remove the warning in future installations. Signed-off-by: Derrick Stolee <derrickstolee@github.com>

This allows fixing settings after a Scalar upgrade, or after botching the enlistments configuration. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

…s remove buggy repos When running 'scalar reconfigure -a', such as at install time, Scalar has warning messages about the repository missing (or not containing a .git directory). Failures can also happen while trying to modify the repository-local config for that repository. These warnings may seem confusing to users who don't understand what they mean or how to stop them. Add a warning that instructs the user how to remove the warning in future installations.

…-and-fix-built-in-fsmonitor Fix the built-in FSMonitor, and run Scalar's Functional Tests as part of the automated builds

This is random stuff that probably all got upstream in the meantime.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Rebase to v2.38.0 #533

Rebase to v2.38.0 #533

Commits on Oct 3, 2022

Commits on Oct 4, 2022