Add `dotEnv` to `turbo.json` #4870

nathanhammond · 2023-05-08T17:34:27Z

Enable configuring .env files inside of turbo.json. These are addition-only to file hashes, and enables specifying load order for cache distinctions.

vercel · 2023-05-08T17:34:35Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
examples-basic-web	🔄 Building (Inspect)			May 19, 2023 10:28am
examples-vite-web	🔄 Building (Inspect)			May 19, 2023 10:28am
turbo-site	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	May 19, 2023 10:28am

8 Ignored Deployments

Name	Status	Updated (UTC)
examples-cra-web	⬜️ Ignored (Inspect)	May 19, 2023 10:28am
examples-designsystem-docs	⬜️ Ignored (Inspect)	May 19, 2023 10:28am
examples-gatsby-web	⬜️ Ignored (Inspect)	May 19, 2023 10:28am
examples-kitchensink-blog	⬜️ Ignored (Inspect)	May 19, 2023 10:28am
examples-native-web	⬜️ Ignored (Inspect)	May 19, 2023 10:28am
examples-nonmonorepo	⬜️ Ignored (Inspect)	May 19, 2023 10:28am
examples-svelte-web	⬜️ Ignored (Inspect)	May 19, 2023 10:28am
examples-tailwind-web	⬜️ Ignored (Inspect)	May 19, 2023 10:28am

vercel · 2023-05-08T17:34:37Z

Someone is attempting to deploy a commit to the Vercel Team on Vercel.

A member of the Team first needs to authorize it.

cli/internal/fs/turbo_json.go

cli/internal/hashing/package_deps_hash.go

mehulkar

Is there any difference between using this key and the inputs key?
do we want to throw an error if it's specified in both places?
What happens if you add inputs: [!.env]?

cli/internal/fs/turbo_json.go

cli/internal/fs/turbo_json_test.go

cli/internal/graph/graph.go

nathanhammond · 2023-05-11T11:06:09Z

@mehulkar

Is there any difference between using this key and the inputs key?

Yes, specifying the first glob in inputs toggles off the "include everything in the git index inside this package's folder" default. (caveats apply, but they're bugs: see Slack thread)

Having an alternative method to specify this does a few things:

By simple existence causes people to actively consider their .env files. Sometimes simply naming a thing or pattern makes the UX better. (For context: you could do 100% of what Ember Services could do when introduced using vanilla Ember things if you knew the incantation; same applies here.)
Creates a configuration location that does not toggle off the default behavior.
Gives us some rails to ride on for future "help me not screw up .env files" behavior.

do we want to throw an error if it's specified in both places?

No, because it doesn't make any material difference if you include it in both places (they get combined into the same list of hashed files).

What happens if you add inputs: [!.env]?

Given this configuration:

{
  "inputs": ["**", "!.env*"],
  "dotEnv": [".env.local", ".env"]
}

The file hash will include .env and .env.local (if present on disk). This is a conservative default; it results in over-partitioning in the event of misconfiguration.

In one proposed possible future variant turbo --dotenv will load environment variables from those locations and not merge the files into the task hash; instead encouraging you to specify your environment variable dependencies using env (and passthroughEnv). This would make the configuration you describe appear sensible, but would already be configured for you by turning all of dotEnv to excludes when that flag is present.

(Including the entire file and the entire fallback chain is already over-partitioning; this would enable precise targeting.)

nathanhammond

Reviewer's Guide!

nathanhammond · 2023-05-11T13:41:10Z

cli/internal/fs/turbo_json.go

+	if task.DotEnv != nil {
+		btd.definedFields.Add("DotEnv")
+
+		// Going to _at least_ be an empty array.
+		btd.TaskDefinition.DotEnv = make(turbopath.AnchoredUnixPathArray, 0, len(task.DotEnv))
+
+		// Port the raw dotEnv values in.
+		for _, dotEnvPath := range task.DotEnv {
+			typeCheckedPath, err := turbopath.CheckedToAnchoredUnixPath(dotEnvPath)
+			if err != nil {
+				return err
+			}
+
+			// These are _explicitly_ not sorted.
+			btd.TaskDefinition.DotEnv = append(btd.TaskDefinition.DotEnv, typeCheckedPath)
+		}
+	}


This is the basic logic for reading the inputs into the tasks.

nathanhammond · 2023-05-11T13:41:41Z

cli/internal/fs/turbo_json.go

+	// Port the raw globalDotEnv values in.
+	if raw.GlobalDotEnv != nil {
+		tj.GlobalDotEnv = make(turbopath.AnchoredUnixPathArray, 0, len(raw.GlobalDotEnv))
+
+		for _, dotEnvPath := range raw.GlobalDotEnv {
+			typeCheckedPath, err := turbopath.CheckedToAnchoredUnixPath(dotEnvPath)
+			if err != nil {
+				return err
+			}
+			tj.GlobalDotEnv = append(tj.GlobalDotEnv, typeCheckedPath)
+		}
+	}


And this is the basic logic for reading the inputs into the global state.

Combined with the task work, this covers the input. Everything else is plumbing.

nathanhammond · 2023-05-11T13:42:04Z

cli/internal/fs/turbo_json_test.go

@@ -27,6 +27,150 @@ func assertIsSorted(t *testing.T, arr []string, msg string) {
 	}
 }

+func Test_ReadTurboConfigDotEnvUndefined(t *testing.T) {


These tests are just one each of:

{}

{ "(global)DotEnv": null }

{ "(global)DotEnv": [] }

{ "(global)DotEnv": ["populated"] }

They also check that it can serialize back out to ensure that we don't break prune.

cli/internal/graph/graph.go

nathanhammond · 2023-05-11T13:47:50Z

cli/internal/hashing/package_deps_hash.go

+// does not error if a file does not exist, then
+// returns a map of normalized path to hash.
+// This map is suitable for cross-platform caching.
+func GetHashesForExistingFiles(rootPath turbopath.AbsoluteSystemPath, files []turbopath.AnchoredSystemPath) (map[turbopath.AnchoredUnixPath]string, error) {


The existing hash mechanism hard errors for nonexistent files; that's because we already do an existence check prior to attempting to hash with the glob walk.

The dotEnv field does not support globbing, we don't want this to error for files that don't exist, and we can't rely on git since a fair chunk of .env files will not be committed, so we plumb it directly through to manual hashing.

cli/internal/run/global_hash.go

nathanhammond · 2023-05-11T13:50:09Z

cli/internal/run/global_hash.go

+	// Make sure we include specified .env files in the file hash.
+	// Handled separately because these are not globs!
+	if len(dotEnv) > 0 {
+		dotEnvObject, err := hashing.GetHashesForExistingFiles(rootpath, dotEnv.ToSystemPathArray())
+		if err != nil {
+			return GlobalHashableInputs{}, fmt.Errorf("error hashing files: %w", err)
+		}
+
+		// Add the dotEnv files into the file hash object.
+		for key, value := range dotEnvObject {
+			globalFileHashMap[key] = value
+		}
+	}


These files get dropped into the files that already get hashed from globalDeps.

nathanhammond · 2023-05-11T13:51:32Z

cli/internal/taskhash/taskhash.go

+				// Make sure we include specified .env files in the file hash.
+				// Handled separately because these are not globs!
+				if len(packageFileHashInputs.taskDefinition.DotEnv) > 0 {
+					packagePath := pkg.Dir.RestoreAnchor(repoRoot)
+					dotEnvObject, err := hashing.GetHashesForExistingFiles(packagePath, packageFileHashInputs.taskDefinition.DotEnv.ToSystemPathArray())
+					if err != nil {
+						return err
+					}
+
+					// Add the dotEnv files into the file hash object.
+					for key, value := range dotEnvObject {
+						hashObject[key] = value
+					}
+				}


And we repeat the task for individual task hashes.

nathanhammond · 2023-05-11T13:52:36Z

cli/internal/turbopath/anchored_unix_path.go

 	"path"
 	"path/filepath"
 )

 // AnchoredUnixPath is a path stemming from a specified root using Unix `/` separators.
 type AnchoredUnixPath string

+// CheckedToAnchoredUnixPath inspects a string and determines if it is a relative path.
+func CheckedToAnchoredUnixPath(s string) (AnchoredUnixPath, error) {


Makes sure that the specified path is not an absolute path.

mehulkar

The code looks good, but I'm having trouble getting on board with special casing a certain kind of file. Have we considered any alternative ways of doing this? For example an $auto option came up a while ago, maybe a similar inputs: ['$dotenv'] could work? This is not as discoverable as what you have here, but thinking out loud.

Specifically, I'm concerned about:

Two affordances for specifying file inputs
Other env var declaration tools than dotenv (now or in the future)
Other ways of specifying dotenv config that don't match these patterns
Abusing this feature to "always include a file" that has nothing to do with dotenv
API expansion in general

I'm also thinking of the auto inclusion of package.json and turbo.json when inputs are specified. That's another exception to inputs that applies under "some but not all" circumstances, and is a totally different mechanism than the one for dotenv files.

In any case, I don't know of any conversations or RFCs where we decided on this implementation, so I don't love having this conversation after the code is already done. I'll assume that you've already done due diligence, but wanted to give a "real" review.

nathanhammond requested a review from a team as a code owner May 8, 2023 17:34

nathanhammond requested review from mehulkar and NicholasLYang May 8, 2023 17:34

turbo-orchestrator bot added area: docs Improvements or additions to documentation team: turborepo labels May 8, 2023

NicholasLYang reviewed May 8, 2023

View reviewed changes

cli/internal/fs/turbo_json.go Outdated Show resolved Hide resolved

cli/internal/hashing/package_deps_hash.go Outdated Show resolved Hide resolved

mehulkar reviewed May 8, 2023

View reviewed changes

cli/internal/fs/turbo_json.go Outdated Show resolved Hide resolved

cli/internal/fs/turbo_json_test.go Show resolved Hide resolved

cli/internal/graph/graph.go Outdated Show resolved Hide resolved

nathanhammond commented May 11, 2023

View reviewed changes

nathanhammond requested review from mehulkar and NicholasLYang May 11, 2023 17:02

mehulkar approved these changes May 12, 2023

View reviewed changes

nathanhammond force-pushed the dotenv branch from 30ea5b1 to d531f23 Compare May 19, 2023 07:53

nathanhammond requested a review from anthonyshew as a code owner May 19, 2023 07:53

nathanhammond force-pushed the dotenv branch from d531f23 to 0280047 Compare May 19, 2023 08:24

vercel bot deployed to Preview – turbo-site May 19, 2023 08:29 View deployment

Nathan Hammond added 10 commits May 19, 2023 18:11

Add dotenv support.

234dda9

dotenv docs and tests and stuff

9d498ac

Constrain to non-absolute paths for now.

e87a2d6

Identified during review.

6a7e539

Correct existence check.

656a88e

Update integration tests.

8719048

Update the hashes again.

9721ef6

Fix formatting.

cb0074f

Correct a link.

f4ed7c5

Improve the documentation for the schema.

5bb666c

nathanhammond force-pushed the dotenv branch from 1925f1f to 5bb666c Compare May 19, 2023 10:11

vercel bot deployed to Preview – turbo-site May 19, 2023 10:28 View deployment

nathanhammond added the pr: automerge Kodiak will merge these automatically after checks pass label May 19, 2023

kodiakhq bot merged commit f1fae9a into vercel:main May 19, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add `dotEnv` to `turbo.json` #4870

Add `dotEnv` to `turbo.json` #4870

nathanhammond commented May 8, 2023 •

edited

Loading

vercel bot commented May 8, 2023 •

edited

Loading

vercel bot commented May 8, 2023

mehulkar left a comment

nathanhammond commented May 11, 2023

nathanhammond left a comment

nathanhammond May 11, 2023

nathanhammond May 11, 2023

nathanhammond May 11, 2023

nathanhammond May 11, 2023

nathanhammond May 11, 2023

nathanhammond May 11, 2023

nathanhammond May 11, 2023

mehulkar left a comment

Add dotEnv to turbo.json #4870

Add dotEnv to turbo.json #4870

Conversation

nathanhammond commented May 8, 2023 • edited Loading

vercel bot commented May 8, 2023 • edited Loading

vercel bot commented May 8, 2023

mehulkar left a comment

Choose a reason for hiding this comment

nathanhammond commented May 11, 2023

nathanhammond left a comment

Choose a reason for hiding this comment

nathanhammond May 11, 2023

Choose a reason for hiding this comment

nathanhammond May 11, 2023

Choose a reason for hiding this comment

nathanhammond May 11, 2023

Choose a reason for hiding this comment

nathanhammond May 11, 2023

Choose a reason for hiding this comment

nathanhammond May 11, 2023

Choose a reason for hiding this comment

nathanhammond May 11, 2023

Choose a reason for hiding this comment

nathanhammond May 11, 2023

Choose a reason for hiding this comment

mehulkar left a comment

Choose a reason for hiding this comment

Add `dotEnv` to `turbo.json` #4870

Add `dotEnv` to `turbo.json` #4870

nathanhammond commented May 8, 2023 •

edited

Loading

vercel bot commented May 8, 2023 •

edited

Loading