Detect false-positive conflicts between local and remote when initializing source tracking

[mmindenhall]

Summary

To speed up our CI/CD pipelines and dev/admin workflows, our team is implementing scratch org pools (to be clear, we're using the pool tools provided by DX@Scale, but we don't use DX@Scale otherwise). We are creating two separate pools: one for validation and running tests within CI/CD pipelines, and one for use by our devs & admins.

Here's how we envisioned this working for the dev/admin scratch org pool:

1. A scheduled pipeline runs the command to create/replenish the scratch org pool.
2. After a new scratch org has been created, the pipeline executes a post-deployment script that installs all of our managed package dependencies, and runs a sf force source push on all of our DX packages (38 and counting) in the correct order. This takes 15-20 minutes to complete.
3. A dev or admin later "fetches" (claims) a scratch org from the pool, and runs a script to push all packages from their local feature branch to "catch up" the scratch org with changes in their branch.

Where did we go wrong?

We did not understand how source tracking works for scratch orgs!

We thought that since we "pre-pushed" all packages to the scratch orgs after creating them in the pipeline, that when a developer claimed one and pushed from their local branch, only the diffs would need to be pushed, so this would be very fast.

Instead, when the developer claims the scratch org, no local source tracking for that scratch org exists. Running sf project deploy preview initializes source tracking, but flags nearly all metadata within the package as conflicts. Then running sf force source push --force-overwrite to push and resolve all conflicts takes just as long as the initial push after the scratch org was created. However, all (or nearly all) of the reported conflicts are false-positives, because the metadata in the local source tree is identical to what is deployed in the scratch org.

What can be done?

The code that initializes source tracking for a new remote should be smart enough to detect and automatically resolve these false-positive conflicts when local and remote metadata are identical.

In case it's useful, here's an idea for how that could work.

1. Use a hashing algorithm to calculate a hash of the source representation (i.e., XML or Apex source) when deploying to an org, and store that hash in the org.
2. When initializing source tracking in a local tree for a new remote, fetch the remote hashes for all metadata.
3. For any metadata that exists both in the local tree and in the remote (i.e., what would currently be "in conflict"), generate a hash from the local source representation. Of course this would require consistent formatting of metadata to avoid different hashes due to whitespace differences.
4. Compare the local and remote hashes. If the hashes match, ignore the conflict and update the local source tracking to match the remote (i.e., set lastRetrievedFromServer to match serverRevisionCounter).

Why should anyone care?

This will save developers tons of time when using scratch orgs that have been pre-allocated.

For example, when running the script to push all of our packages (not including managed package versions) to a scratch org:

• When source tracking has not been initialized: 12-14 minutes
• When source tracking is up to date: 30-40 seconds

[shetzel]

This is not exactly what you're talking about, but if you know what the source tracking state should be you can script it with something like:
sf data query --query "SELECT MAX(RevisionCounter) FROM SourceMember " -t --json --> grab the number and use it in the tracking reset command like:
sf project tracking reset --revision <revNum> -p

[mmindenhall]

Hi @shetzel, thanks for the idea!

However, we can't reset all tracking to a specific revision, because that would clear all conflicts. In the use case I'm describing, there would be a small number of actual conflicts, and a much larger number of false-positives as described above. Here's why:

1. The scratch org in the pool will contain a snapshot of metadata in main at the time the scratch org was created and added to the pool.
2. Devs/admins continue to work on tickets, and merge changes into main several times each day.
3. When someone creates their feature branch (from latest in main) and claims the scratch org, their branch will contain all of the commits that have happened in main since the scratch org was created/populated. Those changes could be to any metadata files in any packages.

So yes, resetting the tracking to the highest revision in the scratch org would clear the false-positives, but would also clear actual conflicts (turning the into false-negatives!).

[shetzel]

I understand you're describing something more advanced than my basic reset suggestion. I was sharing it just in case it was new information.

Another idea to solve this without CLI changes would be for the org pool to know about the associated source control branch/commit of the metadata deployed to it. If the developer ensures the same branch/commit is used when getting an org from the pool, then switches to their development branch, the source tracking state should be correct. This isn't a perfect solution either and has its problems but mentioning it as a possibility.

I think you have some nice ideas as well and I don't want to take anything away from the discussion. I'm just throwing some things out there that might work well enough in the current CLI.

[mmindenhall]

Thanks again! That's a great idea for a workaround. It might be a while, but I'll report back here with results if I get something working.

[aheber]

Not a simple answer but in the past I've solved this by storing the commit hash it was built at in the org, once an org was claimed from the pool I then pulled down the commit hash and used sfdx-git-delta to push all changes from that commit hash until current into the org.

At that point I can be confident the org is up to date and I would reset the push state.

Not an easy process or intuitive but if you're looking to get by it might be good enough.

I had a JavaScript file that could be executed via NPM script so my team could "sync" with the push of a button after claiming an org.

[mmindenhall]

Thanks @aheber! I didn't know about sfdx-git-delta, so I'll check it out when I can.

[mmindenhall]

Just to close the loop on this, I did get this working as suggested by @aheber. Here are some additional details in case anyone stumbles across this and wants to try it.

Background

• We implemented 2 scratch org pools: one for CI/CD (Github Actions), one for devs/admins. The main difference is CI/CD scratch orgs expire after 2 days, whereas dev scratch orgs expire in 30 days, but are deleted from the pool if not claimed within 5 days.

• Rather than "trunk-based development" (which seems to be popular with SFDC teams), we use feature branches and 3 long-lived branches:

  • develop: merge triggers deploy to partial-copy sandbox
  • staging: merge triggers deploy to full-copy sandbox
  • main: merge triggers deploy to production

• Approvals aren't required until merging into staging, but devs/admins usually create PRs for each merge.

• We only use source push to deploy metadata to scratch orgs. We don't build and install packages until PRs are merged to be deployed to a sandbox or production.

• We structure our DX packages in a non-standard way

  • We have a lot of packages (38 currently, but plan to consolidate to around 10-15), and they all live in a single github "monorepo".
  • For reasons I no longer remember, each of our 38 packages is created as a folder that is lives in a top-level folder in the monorepo that is also a DX project (i.e., contains the following structure):

  dx-package-folder
  ├── config
  ├── force-app
  │   └── main
  │       └── default
  │           ├── ...
  └── sfdx-project.json
  

Scratch org pool creation / replenishment

We're using the DX@Scale sfpowertools to create/replenish the scratch orgs in a scheduled CI/CD pipeline (runs hourly during business hours).

DX@Scale intended the scratch org pool tools to build and install packages in scratch orgs. We need the tool to just create empty scratch orgs, which is accomplished as follows:

1. Create a temporary empty DX project (no dependencies): sf project generate --name orgpool --template empty

2. Run the sfp orchestrator:prepare command from the empty DX project folder with a pool config file that disables installation of any package dependencies. For example:

   {
       "tag": "cicd",
       "waitTime": 60,
       "maxAllocation": 20,
       "expiry": 2,
       "batchSize": 10,
       "configFilePath": "../common/config/project-scratch-def.json",
       "postDeploymentScriptPath": "../scripts/orgpool/post_deploy.sh",
       "relaxAllIPRanges": true,
       "installAll": false,
       "enableSourceTracking": true,
       "retryOnFailure": true,
       "succeedOnDeploymentErrors": false
   }
   

After empty scratch orgs are created, the sfp orchestrator:prepare command invokes our post_deploy.sh script for each new org. That script does all of the following:

1. Runs git rev-list --no-merges -n 1 HEAD to get the latest non-merge commit in main.
2. Uses the sf data create record command to save the commit hash in the scratch org. (Note to avoid the need for any custom object or field, we're just creating an Account with a known Name, and saving the hash in the SicDesc field).
3. Walks our monorepo to generate a correctly ordered and de-duplicated list of all of our packages and their dependencies.
4. Iterates through the list running sf force source push to push our DX packages, and sf package install to install managed package dependencies.

When the post_deploy.sh script has finished for all new orgs in the pool, each contains all of our packages and the 3rd party managed packages that we use. This pipeline is currently taking anywhere from 25-50 minutes to complete when scratch orgs need to be added to the pool.

Scratch org retrieval and sync

We have a CI/CD pipeline that runs when a PR is created or updated with new commits. In that pipeline, we run a script to retrieve a scratch org from the pool, and push all changes in the PR branch since the scratch org was created. That script does the following:

1. Executes sfp pool:fetch to claim a scratch org from the pool
2. Runs sf data query to get the git hash stored when the scratch org was created
3. Walks our repo to generate a list of all of our packages and their dependencies, then de-duplicates the list.
4. For each of our DX packages in the list:
   1. Creates a temp folder
   2. Runs this sfdx-git-delta plugin command to generate deltas (if any) for the package since the scratch org was created: sf sgd source delta --from {from_git_hash} --source {pkg_folder} --output {temp_dir} --ignore {forceignore_file} --generate-delta --json
   3. If the previous command generates deltas, runs sf force source deploy --sourcepath {temp_dir} --json to deploy them.
   4. Runs sf data query --json --use-tooling-api --query "select revisioncounter from sourcemember order by revisioncounter desc limit 1" to retrieve the latest source tracking Revision Counter from the scratch org.
   5. Runs sf force source tracking reset --no-prompt --revision {revision_counter} --json to reset source tracking for the local package to the latest in the scratch org.

All of the above takes 2-4 minutes to complete in the pipeline. When the script has finished running, all changes in the local branch (possibly across multiple packages) will have been pushed to the scratch org. Our CI/CD pipeline then proceeds to run all tests in the scratch org, and at the end runs sfp pool:org:delete to delete the scratch org.

Results

We create PRs to merge into each environment, so each user story goes through the PR validation pipeline at least 3 times to reach production. This impacts our overall team velocity, as well as the minimum amount of time it takes for a fix to get deployed to production.

• Before the scratch org pool, this pipeline took 20-35 minutes (optimized to push/install only packages with changed files, plus their dependencies)
• Using the scratch org pool this pipeline is taking 12-15 minutes