Include Windows-specific maintenance and headless-git #2974
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The existing schedule mechanism using 'cron' is supported by POSIX platforms, but not Windows. It also works slightly differently on macOS to significant detriment of the user experience. To allow for new implementations on these platforms, extract a method that performs the platform-specific scheduling mechanism. This will be swapped at compile time with new implementations on specialized platforms. As we add this generality, rename GIT_TEST_CRONTAB to GIT_TEST_MAINT_SCHEDULER. Further, this variable is now parsed as "<scheduler>:<command>" so we can test platform-specific scheduling logic even when not on the correct platform. By specifying the <scheduler> in this string, we will be able to test all three sets of Git logic from a Linux machine. Co-authored-by: Eric Sunshine <sunshine@sunshineco.com> Signed-off-by: Eric Sunshine <sunshine@sunshineco.com> Signed-off-by: Derrick Stolee <dstolee@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
Advanced and expert users may want to know how 'git maintenance start' schedules background maintenance in order to customize their own schedules beyond what the maintenance.* config values allow. Start a new set of sections in git-maintenance.txt that describe how 'cron' is used to run these tasks. This is particularly valuable for users who want to inspect what Git is doing or for users who want to customize the schedule further. Having a baseline can provide a way forward for users who have never worked with cron schedules. Helped-by: Eric Sunshine <sunshine@sunshineco.com> Signed-off-by: Derrick Stolee <dstolee@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
The existing mechanism for scheduling background maintenance is done through cron. The 'crontab -e' command allows updating the schedule while cron itself runs those commands. While this is technically supported by macOS, it has some significant deficiencies: 1. Every run of 'crontab -e' must request elevated privileges through the user interface. When running 'git maintenance start' from the Terminal app, it presents a dialog box saying "Terminal.app would like to administer your computer. Administration can include modifying passwords, networking, and system settings." This is more alarming than what we are hoping to achieve. If this alert had some information about how "git" is trying to run "crontab" then we would have some reason to believe that this dialog might be fine. However, it also doesn't help that some scenarios just leave Git waiting for a response without presenting anything to the user. I experienced this when executing the command from a Bash terminal view inside Visual Studio Code. 2. While cron initializes a user environment enough for "git config --global --show-origin" to show the correct config file information, it does not set up the environment enough for Git Credential Manager Core to load credentials during a 'prefetch' task. My prefetches against private repositories required re-authenticating through UI pop-ups in a way that should not be required. The solution is to switch from cron to the Apple-recommended [1] 'launchd' tool. [1] https://developer.apple.com/library/archive/documentation/MacOSX/Conceptual/BPSystemStartup/Chapters/ScheduledJobs.html The basics of this tool is that we need to create XML-formatted "plist" files inside "~/Library/LaunchAgents/" and then use the 'launchctl' tool to make launchd aware of them. The plist files include all of the scheduling information, along with the command-line arguments split across an array of <string> tags. For example, here is my plist file for the weekly scheduled tasks: <?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd"> <plist version="1.0"><dict> <key>Label</key><string>org.git-scm.git.weekly</string> <key>ProgramArguments</key> <array> <string>/usr/local/libexec/git-core/git</string> <string>--exec-path=/usr/local/libexec/git-core</string> <string>for-each-repo</string> <string>--config=maintenance.repo</string> <string>maintenance</string> <string>run</string> <string>--schedule=weekly</string> </array> <key>StartCalendarInterval</key> <array> <dict> <key>Day</key><integer>0</integer> <key>Hour</key><integer>0</integer> <key>Minute</key><integer>0</integer> </dict> </array> </dict> </plist> The schedules for the daily and hourly tasks are more complicated since we need to use an array for the StartCalendarInterval with an entry for each of the six days other than the 0th day (to avoid colliding with the weekly task), and each of the 23 hours other than the 0th hour (to avoid colliding with the daily task). The "Label" value is currently filled with "org.git-scm.git.X" where X is the frequency. We need a different plist file for each frequency. The launchctl command needs to be aligned with a user id in order to initialize the command environment. This must be done using the 'launchctl bootstrap' subcommand. This subcommand is new as of macOS 10.11, which was released in September 2015. Before that release the 'launchctl load' subcommand was recommended. The best source of information on this transition I have seen is available at [2]. The current design does not preclude a future version that detects the available fatures of 'launchctl' to use the older commands. However, it is best to rely on the newest version since Apple might completely remove the deprecated version on short notice. [2] https://babodee.wordpress.com/2016/04/09/launchctl-2-0-syntax/ To remove a schedule, we must run 'launchctl bootout' with a valid plist file. We also need to 'bootout' a task before the 'bootstrap' subcommand will succeed, if such a task already exists. The need for a user id requires us to run 'id -u' which works on POSIX systems but not Windows. Further, the need for fully-qualitifed path names including $HOME behaves differently in the Git internals and the external test suite. The $HOME variable starts with "C:\..." instead of the "/c/..." that is provided by Git in these subcommands. The test therefore has a prerequisite that we are not on Windows. The cross- platform logic still allows us to test the macOS logic on a Linux machine. We can verify the commands that were run by 'git maintenance start' and 'git maintenance stop' by injecting a script that writes the command-line arguments into GIT_TEST_MAINT_SCHEDULER. An earlier version of this patch accidentally had an opening "<dict>" tag when it should have had a closing "</dict>" tag. This was caught during manual testing with actual 'launchctl' commands, but we do not want to update developers' tasks when running tests. It appears that macOS includes the "xmllint" tool which can verify the XML format. This is useful for any system that might contain the tool, so use it whenever it is available. We strive to make these tests work on all platforms, but Windows caused some headaches. In particular, the value of getuid() called by the C code is not guaranteed to be the same as `$(id -u)` invoked by a test. This is because `git.exe` is a native Windows program, whereas the utility programs run by the test script mostly utilize the MSYS2 runtime, which emulates a POSIX-like environment. Since the purpose of the test is to check that the input to the hook is well-formed, the actual user ID is immaterial, thus we can work around the problem by making the the test UID-agnostic. Another subtle issue is the $HOME environment variable being a Windows-style path instead of a Unix-style path. We can be more flexible here instead of expecting exact path matches. Helped-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Co-authored-by: Eric Sunshine <sunshine@sunshineco.com> Signed-off-by: Eric Sunshine <sunshine@sunshineco.com> Signed-off-by: Derrick Stolee <dstolee@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
Git's background maintenance uses cron by default, but this is not available on Windows. Instead, integrate with Task Scheduler. Tasks can be scheduled using the 'schtasks' command. There are several command-line options that can allow for some advanced scheduling, but unfortunately these seem to all require authenticating using a password. Instead, use the "/xml" option to pass an XML file that contains the configuration for the necessary schedule. These XML files are based on some that I exported after constructing a schedule in the Task Scheduler GUI. These options only run background maintenance when the user is logged in, and more fields are populated with the current username and SID at run-time by 'schtasks'. Since the GIT_TEST_MAINT_SCHEDULER environment variable allows us to specify 'schtasks' as the scheduler, we can test the Windows-specific logic on other platforms. Thus, add a check that the XML file written by Git is valid when xmllint exists on the system. Since we use a temporary file for the XML files sent to 'schtasks', we prefix the random characters with the frequency so it is easier to examine the proper file during tests. Instead of an exact match on the 'args' file, we 'grep' for the arguments other than the filename. There is a deficiency in the current design. Windows has two kinds of applications: GUI applications that start by "winmain()" and console applications that start by "main()". Console applications are attached to a new Console window if they are not already associated with a GUI application. This means that every hour the scheudled task launches a command window for the scheduled tasks. Not only is this visually obtrusive, but it also takes focus from whatever else the user is doing! A simple fix would be to insert a GUI application that acts as a shim between the scheduled task and Git. This is currently possible in Git for Windows by setting the <Command> tag equal to C:\Program Files\Git\git-bash.exe with options "--hide --no-needs-console --command=cmd\git.exe" followed by the arguments currently used. Since git-bash.exe is not included in Windows builds of core Git, I chose to leave out this feature. My plan is to submit a small patch to Git for Windows that converts the use of git.exe with this use of git-bash.exe in the short term. In the long term, we can consider creating this GUI shim application within core Git, perhaps in contrib/. Co-authored-by: Eric Sunshine <sunshine@sunshineco.com> Signed-off-by: Eric Sunshine <sunshine@sunshineco.com> Signed-off-by: Derrick Stolee <dstolee@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
On Windows, there are two kinds of executables, console ones and non-console ones. Git's executables are all console ones. When launching the former e.g. in a scheduled task, a CMD window pops up. This is not what we want for the tasks installed via the `git maintenance` command. To work around this, let's introduce `headless-git.exe`, which is a non-console program that does _not_ pop up any window. All it does is to re-launch `git.exe`, suppressing that console window, passing through all command-line arguments as-are. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
We just introduced a helper to avoid showing a console window when the scheduled task runs `git.exe`. Let's actually use it. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
|
Don't we need dc86de5 to let the test suite pass? |
This instance of `GIT_TEST_CRONTAB` seems to have been forgotten. Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
|
Should we also do something about gitgitgadget#776 (comment)? |
I'm hoping that this doesn't result in a re-roll, but let's wait and see. |
In any case, I think we can already merge this, as it is a commit message-only change (which will easily be "rebased away" if it hits upstream). |
4 tasks
dscho
added a commit
that referenced
this pull request
Jan 14, 2021
Include Windows-specific maintenance and headless-git
dscho
added a commit
that referenced
this pull request
Jan 14, 2021
Include Windows-specific maintenance and headless-git
dscho
added a commit
that referenced
this pull request
Jan 14, 2021
Include Windows-specific maintenance and headless-git
dscho
added a commit
that referenced
this pull request
Jan 14, 2021
Include Windows-specific maintenance and headless-git
dscho
added a commit
that referenced
this pull request
Jan 14, 2021
Include Windows-specific maintenance and headless-git
dscho
added a commit
that referenced
this pull request
Jan 15, 2021
Include Windows-specific maintenance and headless-git
dscho
added a commit
that referenced
this pull request
Jan 15, 2021
Include Windows-specific maintenance and headless-git
dscho
added a commit
that referenced
this pull request
Jan 15, 2021
Include Windows-specific maintenance and headless-git
dscho
added a commit
that referenced
this pull request
Jan 15, 2021
Include Windows-specific maintenance and headless-git
dscho
added a commit
that referenced
this pull request
Jan 15, 2021
Include Windows-specific maintenance and headless-git
dscho
added a commit
that referenced
this pull request
Jan 15, 2021
Include Windows-specific maintenance and headless-git
dscho
added a commit
that referenced
this pull request
Jan 18, 2021
Include Windows-specific maintenance and headless-git
dscho
added a commit
that referenced
this pull request
Jan 18, 2021
Include Windows-specific maintenance and headless-git
dscho
added a commit
that referenced
this pull request
Jan 18, 2021
Include Windows-specific maintenance and headless-git
dscho
added a commit
that referenced
this pull request
Oct 11, 2024
Include Windows-specific maintenance and headless-git
dscho
added a commit
that referenced
this pull request
Oct 20, 2024
Include Windows-specific maintenance and headless-git
dscho
added a commit
that referenced
this pull request
Oct 20, 2024
Include Windows-specific maintenance and headless-git
dscho
added a commit
that referenced
this pull request
Oct 21, 2024
Include Windows-specific maintenance and headless-git
git-for-windows-ci
pushed a commit
that referenced
this pull request
Oct 21, 2024
Include Windows-specific maintenance and headless-git
git-for-windows-ci
pushed a commit
that referenced
this pull request
Oct 21, 2024
Include Windows-specific maintenance and headless-git
git-for-windows-ci
pushed a commit
that referenced
this pull request
Oct 22, 2024
Include Windows-specific maintenance and headless-git
git-for-windows-ci
pushed a commit
that referenced
this pull request
Oct 22, 2024
Include Windows-specific maintenance and headless-git
git-for-windows-ci
pushed a commit
that referenced
this pull request
Oct 22, 2024
Include Windows-specific maintenance and headless-git
git-for-windows-ci
pushed a commit
that referenced
this pull request
Oct 22, 2024
Include Windows-specific maintenance and headless-git
git-for-windows-ci
pushed a commit
that referenced
this pull request
Oct 22, 2024
Include Windows-specific maintenance and headless-git
git-for-windows-ci
pushed a commit
that referenced
this pull request
Oct 22, 2024
Include Windows-specific maintenance and headless-git
dscho
added a commit
that referenced
this pull request
Oct 23, 2024
Include Windows-specific maintenance and headless-git
git-for-windows-ci
pushed a commit
that referenced
this pull request
Oct 25, 2024
Include Windows-specific maintenance and headless-git
git-for-windows-ci
pushed a commit
that referenced
this pull request
Oct 25, 2024
Include Windows-specific maintenance and headless-git
git-for-windows-ci
pushed a commit
that referenced
this pull request
Oct 25, 2024
Include Windows-specific maintenance and headless-git
git-for-windows-ci
pushed a commit
that referenced
this pull request
Oct 30, 2024
Include Windows-specific maintenance and headless-git
git-for-windows-ci
pushed a commit
that referenced
this pull request
Nov 1, 2024
Include Windows-specific maintenance and headless-git
git-for-windows-ci
pushed a commit
that referenced
this pull request
Nov 6, 2024
Include Windows-specific maintenance and headless-git
dscho
added a commit
that referenced
this pull request
Nov 22, 2024
Include Windows-specific maintenance and headless-git
dscho
added a commit
that referenced
this pull request
Nov 22, 2024
Include Windows-specific maintenance and headless-git
dscho
added a commit
that referenced
this pull request
Nov 22, 2024
Include Windows-specific maintenance and headless-git
dscho
added a commit
that referenced
this pull request
Nov 22, 2024
Include Windows-specific maintenance and headless-git
dscho
added a commit
that referenced
this pull request
Nov 22, 2024
Include Windows-specific maintenance and headless-git
git-for-windows-ci
pushed a commit
that referenced
this pull request
Nov 25, 2024
Include Windows-specific maintenance and headless-git
git-for-windows-ci
pushed a commit
that referenced
this pull request
Nov 25, 2024
Include Windows-specific maintenance and headless-git
git-for-windows-ci
pushed a commit
that referenced
this pull request
Nov 25, 2024
Include Windows-specific maintenance and headless-git
git-for-windows-ci
pushed a commit
that referenced
this pull request
Nov 25, 2024
Include Windows-specific maintenance and headless-git
dscho
added a commit
to dscho/git
that referenced
this pull request
Nov 25, 2024
…e-and-headless Include Windows-specific maintenance and headless-git
git-for-windows-ci
pushed a commit
that referenced
this pull request
Nov 25, 2024
Include Windows-specific maintenance and headless-git
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
These six patches complete the background maintenance story for Git for Windows.
The first four patches were recently merged into
nextin upstream. These update the platform-specific background maintenance to useschtaskson Windows.The last two patches create a
headless-git.exethat uses awinmain()instead of amain()to allow the background maintenance to not create a console window with focus in front of a user.