[BUG] secrets from environment not resolved on included projects

[inakimalerba]

Description

I'm trying to include a compose file into my main compose which contains a secret defined on it's own environment file, but compose fails to resolve it.

The main compose includes a child compose which has a secret loaded on it's own .env file.

When moving app/.env to .env it works fine, which means include is not respecting project_directory nor env_file for the secret resolving.

Is there any way I can avoid this besides copying the .env to the root or adding the --env-file app/.env flag to the docker compose up command? (which also works)

Steps To Reproduce

The project structure is as follows:

├── app
│   └── .env
│   └── docker-compose.yml
└── docker-compose.yml

• docker-compose.yml

---
include:
  - path: app/docker-compose.yml
    project_directory: app
    env_file: app/.env

services:
  main:
    image: nginx:1.25.4-alpine

• app/docker-compose.yml

---
services:
  app:
    image: nginx:1.25.4-alpine
    secrets:
      - passwd

secrets:
  passwd:
    environment: PASSWD

• app/.env

PASSWD=secret

When trying to run the previous stack, I get the following error:

$ docker compose up
[+] Running 1/3
 ✔ Network demo_default   Created                                                                                                                                                                                                            0.1s 
 ⠋ Container demo-app-1   Creating                                                                                                                                                                                                           0.1s 
 ⠋ Container demo-main-1  Creating                                                                                                                                                                                                           0.1s 
environment variable "PASSWD" required by file "demo_passwd" is not set

Compose Version

Docker Compose version v2.28.1

Docker Environment

Client: Docker Engine - Community
 Version:    27.0.1
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.15.1
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.28.1
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Anything else?

No response

[ndeloof]

env_file set in includes file is used to process interpolation (i.e replace ${VARIABLE} in yaml), not to set environment values. As includes parse another compose file to be added to your compose application, while interpolation takes place, the local environment is the one you run locally

There's unfortunately a major misconception in compose between variables used by interpolation and environment variables. I hardly can imagine a way we could support this use-case, need to think twice about this.