When two `git maintenance` processes try to write the `.plist` file, we
need to help them with serializing their efforts.

The 150ms time-out value was determined from thin air.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

On macOS, we use launchctl to manage the background maintenance
schedule. This uses a set of .plist files to describe the schedule, but
these files are also registered with 'launchctl bootstrap'. If multiple
'git maintenance start' commands run concurrently, then they can collide
replacing these schedule files and registering them with launchctl.

To avoid extra launchctl commands, do a check for the .plist files on
disk and check if they are registered using 'launchctl list <name>'.
This command will return with exit code 0 if it exists, or exit code 113
if it does not.

We can test this behavior using the GIT_TEST_MAINT_SCHEDULER environment
variable.

Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

Over the course of Scalar's development, it became obvious that there is
a need for a command that can gather all kinds of useful information
that can help identify the most typical problems with large
worktrees/repositories.

The `diagnose` command is the culmination of this hard-won knowledge: it
gathers the installed hooks, the config, a couple statistics describing
the data shape, among other pieces of information, and then wraps
everything up in a tidy, neat `.zip` archive.

Note: in the .NET version we have the luxury of a comprehensive standard
library that includes basic functionality such as writing a `.zip` file.
In the C version, we lack such a commodity. Rather than introducing a
dependency on, say, libzip, we slightly abuse Git's `archive` command:
instead of writing the `.zip` file directly, we stage the file contents
in a Git index of a temporary, bare repository, only to let `git
archive` have at it, and finally removing the temporary repository.

Also note: Due to the frequent spawned `git hash-object` processes, this
command is quite a bit slow on Windows. Should it turn out to be a big
problem, the lack of a batch mode of the `hash-object` command could
potentially be worked around via using `git fast-import` with a crafted
`stdin`.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

Let's start implementing the `register` command. With this commit,
recommended settings are configured upon `scalar register`.

Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

This implements Scalar's opinionated `clone` command: it tries to use a
partial clone and sets up a sparse checkout by default. In contrast to
`git clone`, `scalar clone` sets up the worktree in the `src/`
subdirectory, to encourage a separation between the source files and the
build output (which helps Git tremendously because it avoids untracked
files that have to be specifically ignored when refreshing the index).

Also, it registers the repository for regular, scheduled maintenance,
and configures a slur of configuration settings based on the experience
of the Microsoft Windows and the Microsoft Office development teams.

Note: We intentionally use a slightly wasteful `set_config()` function
(which does not reuse a single `strbuf`, for example, though performance
_really_ does not matter here) because it is very, very convenient.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

Arguably, the biggest learning from the Scalar project is that scheduled
maintenance is crucial to keep large repositories in a good shape.

With this commit, `scalar register` starts those scheduled maintenance
tasks, and `scalar unregister` stops them.

Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

This commit adds a simple regression test, modeled after Git's own
test suite.

A more comprehensive functional (or: integration) test suite can be
found at https://github.com/microsoft/scalar; There is no intention to
port that fuller test suite to `contrib/scalar/`; Instead, it will still
be used to verify the `scalar` functionality in Microsoft's Git fork.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

This comes in handy during Scalar upgrades, or when config settings were
messed up by mistake.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

Teach the `scalar diagnose` command to gather file size information
about pack files.

Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>

With this patch, we start the journey from the C# project at
https://github.com/microsoft/scalar to move what is left to Git's own
`contrib/` directory.

The idea of Scalar, and before that VFS for Git, has always been to
prove that Git _can_ scale, and to upstream whatever strategies have
been demonstrated to help.

For example, while the virtual filesystem provided by VFS for Git helped
the team developing the Windows operating system to move onto Git, it is
not really an upstreamable strategy: getting it to work, and the
required server-side support, make this not quite feasible.

The Scalar project learned from that and tackled the problem with
different tactics: instead of pretending to Git that the working
directory is fully populated, it _specifically_ teaches Git about
partial clone (which is based on VFS for Git's cache server), about
sparse checkout (which VFS for Git tried to do transparently, in the
file system layer), and regularly runs maintenance tasks to keep the
repository in a healthy state.

With partial clone, sparse checkout and `git maintenance` having been
upstreamed, there is little left that `scalar.exe` does that which
`git.exe` cannot do. One such thing is that `scalar clone <url>` will
automatically set up a partial, sparse clone, and configure
known-helpful settings from the start.

Let's bring this convenience directly into Git's tree.

The idea here is that you can (optionally) build Scalar via

	make -C contrib/scalar/Makefile

This will build the `scalar` executable and put it into the
contrib/scalar/ subdirectory.

The slightly awkward addition of the `contrib/scalar/*` bits to the
top-level `Makefile` are actually really required: we want to link to
`libgit.a`, which means that we will need to use the very same `CFLAGS`
and `LDFLAGS` as the rest of Git.

An early development version of this patch tried to replicate the
respective conditionals in `contrib/scalar/Makefile` (just like
`contrib/svn-fe/Makefile` tried to do). It turned out to be quite the
whack-a-mole game: the SHA-1-related flags, the flags enabling/disabling
`compat/poll/`, `compat/regex/`, `compat/win32mmap.c` etc based on the
current platform... To put it mildly: it was a major mess.

Instead, this patch makes minimal changes to the top-level `Makefile` so
that the bits in `contrib/scalar/` can be compiled and linked, and
adds a `contrib/scalar/Makefile` that uses the top-level `Makefile` in a
most minimal way to do the actual compiling.

Note: With this commit, we only establish the infrastructure, no
Scalar functionality is implemented yet; We will do that incrementally
over the next few commits.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

The list is simply those registered under the multi-valued scalar.repo
config setting.

Signed-off-by: Derrick Stolee <dstolee@microsoft.com>

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

For example after a Scalar upgrade, it can come in really handy if there
is an easy way to reconfigure all Scalar enlistments. This new option
offers this functionality.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

Teach the `scalar diagnose` command to gather loose object counts.

Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>

... which does not do much, yet...

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

When a user deleted an enlistment manually, let's be generous and
_still_ unregister it.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

This is mostly just a shim for `git maintenance`, mapping task names
from the way Scalar called them to the way Git calls them.

Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

The .NET version supported running `scalar config` to reconfigure the
current enlistment, and now the C port does, too.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

This commit establishes the infrastructure to build the manual page for
te `scalar` command.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

Let's populate the manual page of `scalar` a bit.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

With this commit, `git help scalar` will open the appropriate manual
or HTML page (instead of looking for `gitscalar`).

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

Using the built-in FSMonitor makes many common commands quite a bit
faster. So let's teach the `scalar register` command to enable the
built-in FSMonitor and kick-start the fsmonitor--daemon process (for
convenience).

For simplicity, we only support the built-in FSMonitor (and no external
file system monitor such as e.g. Watchman).

Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

Continuing the documentation journey.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

In particular when multiple processes want to write to the config
simultaneously, it would come in handy to not fail immediately when
another process locked the config, but to gently try again.

This will help with Scalar's functional test suite which wants to
register multiple repositories for maintenance semi-simultaneously.

As not all code paths calling this function read the config (e.g. `git
config`), we have to read the config setting via
`git_config_get_ulong()`.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

It is merely handing off to `git help scalar`.

Signed-off-by: Johannes Schindelin <johasc@microsoft.com>

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

No grown-up C project comes without their own JSON parser.

Just kidding!

We need to parse a JSON result when determining which cache server to
use. It would appear that searching for needles `"CacheServers":[`,
`"Url":"` and `"GlobalDefault":true` _happens_ to work right now, it is
fragile as it depends on no whitespace padding and on the order of the
fields remaining as-is.

Let's implement a super simple JSON parser (at the cost of being
slightly inefficient) for that purpose. To avoid allocating a ton of
memory, we implement a callback-based one. And to save on complexity,
let's not even bother validating the input properly (we will just go
ahead and instead rely on Azure Repos to produce correct JSON).

Note: An alternative would have been to use existing solutions such as
JSON-C, CentiJSON or JSMN. However, they are all a lot larger than the
current solution; The smallest, JSMN, which does not even provide parsed
string values (something we actually need) weighs in with 471 lines,
while we get away with 182 + 29 lines for the C and the header file,
respectively.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

Delete an enlistment by first unregistering the repository and then
deleting the enlisment directory (the directory containing the worktree
`src` directory).

On Windows, if the current directory is inside the enlismtnet directory
then change to the parent of the enlistment directory, to allow us to
delete the enlistment.

Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

The .NET version of Scalar has a `version` command. This was necessary
because it was versioned independently of Git.

Since Scalar is now tightly coupled with Git, it does not make sense for
them to show different versions. Therefore, it shows the same output as
`git versions`. For backwards-compatibility with the .NET version,
`scalar version` prints to `stderr`, though (`git version` prints to
`stdout` instead).

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

The `git` executable has these two very useful options:

-C <directory>:
	switch to the specified directory before performing any actions

-c <key>=<value>:
	temporarily configure this setting for the duration of the
	specified scalar subcommand

With this commit, we teach the `scalar` executable the same trick.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

Especially on Windows, we will need to stop that daemon, just in case
that the directory needs to be removed (the daemon would otherwise hold
a handle to that directory, preventing it from being deleted).

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

This completes the manual page for the `scalar` command.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

By default, Git fails immediately when locking a config file for writing
fails due to an existing lock. With this change, Scalar-registered
repositories will fall back to trying a couple times within a 150ms
timeout.

Signed-off-by: Johannes Schindelin <johasc@microsoft.com>

The Scalar project started out as a separate project. As designated
successor of VFS for Git, its goal has been from the start to apply the
learnings from the VFS for Git project to allow Git to scale. In
contrast to VFS for Git, Scalar very much wants Git to be aware that the
checkout is sparse and the clone is partial (as opposed to VFS for Git
which tried to pretend that everything is there via a virtual file
system driver).

The intention has always been to integrate as much as possible of Scalar
into Git proper. And this has been the case: the partial clone feature,
the sparse checkout (cone mode), the commit graph, multi-pack index
files, scheduled maintenance tasks, all of these wonderful things can
now be enjoyed by Git users _without_ having to install Scalar.

So what remains of Scalar was ported to C, and put into
`contrib/scalar/`. And to make it easier for folks to build and install
it along with Git, this here patch makes it possible to simply drop
`INCLUDE_SCALAR = YesPlease` into the `config.mak` file to make it so.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

Sadly, this is a bit trickier than merely flipping the
`INCLUDE_SCALAR=YesPlease` switch: The Windows tests are run in a very
different way.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

With this change, we come a big step closer to feature parity with
Scalar: this allows cloning from Azure Repos (which do not support
partial clones at time of writing).

We use the just-implemented JSON parser to parse the response we got
from the `gvfs/config` endpoint; Please note that this response might,
or might not, contain information about a cache server. The presence or
absence of said cache server, however, has nothing to do with the
ability to speak the GVFS protocol (but the presence of the
`gvfs/config` endpoint does that).

An alternative considered during the development of this patch was to
perform simple string matching instead of parsing the JSON-formatted
data; However, this would have been fragile, as the response contains
free-form text (e.g. the repository's description) which might contain
parts that would confuse a simple string matcher (but not a proper JSON
parser).

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

This comes in handy, as we want to verify that `scalar clone` also works
against a GVFS-enabled remote repository.

Note that we have to set `MSYS2_ENV_CONV_EXCL` to prevent MSYS2 from
mangling `PATH_TRANSLATED`: The value _does_ look like a Unix-style
path, but no, MSYS2 must not be allowed to convert that into a Windows
path: `http-backend` needs it in the unmodified form. (The MSYS2 runtime
comes in when `git` is run via `bin-wrappers/git`, which is a shell
script.)

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

We already have the `config` command that accesses the `gvfs/config`
endpoint.

To implement `scalar`, we also need to be able to access the `vsts/info`
endpoint. Let's add a command to do precisely that.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

On Windows, both the forward slash and the backslash are directory
separators. Which means that `a\b\c` really is inside `a/b`. Therefore,
we need to special-case the directory separators in the helper function
`cmp_icase()` that is used in the loop in `dir_inside_of()`.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

This finalizes the port of the `QueryVstsInfo()` function: we already
taught `gvfs-helper` to access the `vsts/info` endpoint on demand, we
implemented proper JSON parsing, and now it is time to hook it all up.

To that end, we also provide a default local cache root directory. It
works the same way as the .NET version of Scalar: it uses

    C:\scalarCache on Windows,

    ~/.scalarCache/ on macOS and

    ~/.cache/scalar on Linux

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

Well, technically also the http:// protocol is allowed _when testing_...

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

Azure Repos does not support partial clones at the moment, but it does
support the GVFS protocol. To that end, the Microsoft fork of Git has a
`gvfs-helper` command that is optionally used to perform essentially the
same functionality as partial clone.

Let's verify that `scalar clone` detects that situation and enables the
GVFS helper.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

This allows setting the GVFS-enabled cache server, or listing the one(s)
associated with the remote repository.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

In Scalar's functional tests, we do not do anything with authentication.
Therefore, we do want to avoid accessing the `vsts/info` endpoint
because it requires authentication even on otherwise public
repositories.

Let's introduce the environment variable `SCALAR_TEST_SKIP_VSTS_INFO`
which can be set to `true` to simply skip that step (and force the
`url_*` style repository IDs instead of `id_*` whenever possible).

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

…hen plist is registered

This adds the bare minimum to compile the `scalar` executable.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

This implements the subcommands `register`, `unregister` and `list`.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

This implements `clone`.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

This implements `scalar run`.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

This allows fixing settings after a Scalar upgrade, or after botching
the enlistments configuration.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

This implements the `diagnose` subcommand.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

This is a convenient shortcut for `scalar unregister <enlistment> &&
rm -rf <enlistment>`.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

This implements the `version` command for backwards-compatibility with
the .NET version of Scalar.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

For convenience, this ports the `git -c <key>=<value> -C <dir>
<command>` functionality to `scalar`, allowing config settings and
workig directories to be set for the duration of the Scalar invocation.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

Make `contrib/scalar/` work nicely with the built-in FSMonitor.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

Document the whole thing.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

Allow concurrent `scalar register` and `scalar unregister` calls to be
more collaborative when trying to lock the global Git config at the very
same time.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

This topic branch offers to include `scalar` in a regular Git build,
simply by setting `INCLUDE_SCALAR=YesPlease`.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

For ease of development, build and test `scalar`, too.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

Prepare `scalar` to use the GVFS protocol instead of partial clone
(required to support Azure Repos).

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

Adjust the Scalar Functional Tests CI run to adjust to Scalar/C.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

Apparently, there _might_ be left-over Git processes even outside of
using the built-in FSMonitor... (for example, see
https://github.com/microsoft/git/pull/366/checks?check_run_id=2786318460#step:19:20)

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

As Scalar/C ignores `feature.scalar`, let's not run essentially the same
thing twice.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>