KrakenD adding unknown headers to requests?

[qtpi-bonding]

Environment info:

• KrakenD version: Version: 2.7.2
• System info: docker
• Backend technology: SeaweedFS
• Additional environment information: SeaweedFS & KrakenD are in docker compose.

Describe what are you trying to do:
Retrieve an image that exists in SeaweedFS (S3 compatible bucket) by going client -> KrakenD -> SeaweedFS.

ALL INPUT HEADERS:

{
  "$schema": "https://www.krakend.io/schema/krakend.json",
  "version": 3,
  "name": "KrakenD - API Gateway",
  "timeout": "3000ms",
  "cache_ttl": "300s",
  "output_encoding": "json",
  "debug_endpoint": true,
  "echo_endpoint": true,
  "tls": {
    "keys": [
      {
        "public_key": "/path/fullchain.pem",
        "private_key": "/path/privkey.pem"
      }
    ]
  },
  "endpoints": [
    {
      "endpoint": "/debug/seaweedfs/{segment1}/{segment2}/{segment3}/",
      "method": "GET",
      "output_encoding": "no-op",
      "input_headers": [
        "*"
      ],
      "backend": [
        {
          "method": "GET",
          "encoding": "no-op",
          "host": [
            "https://HOSTNAME"
          ],
          "url_pattern": "/__debug/default"
        }
      ]
    },
    {
      "endpoint": "/seaweedfs/{segment1}/{segment2}/{segment3}/",
      "method": "GET",
      "output_encoding": "no-op",
      "input_headers": [
        "*"
      ],
      "backend": [
        {
          "method": "GET",
          "encoding": "no-op",
          "url_pattern": "/{segment1}/{segment2}/{segment3}/",
          "host": [
            "http://seaweedfs:8333"
          ]
        }
      ]
    }
  ]
}

NO INPUT HEADERS:

{
  "$schema": "https://www.krakend.io/schema/krakend.json",
  "version": 3,
  "name": "KrakenD - API Gateway",
  "timeout": "3000ms",
  "cache_ttl": "300s",
  "output_encoding": "json",
  "debug_endpoint": true,
  "echo_endpoint": true,
  "tls": {
    "keys": [
      {
        "public_key": "/path/fullchain.pem",
        "private_key": "/path/privkey.pem"
      }
    ]
  },
  "endpoints": [
    {
      "endpoint": "/debug/seaweedfs/{segment1}/{segment2}/{segment3}/",
      "method": "GET",
      "output_encoding": "no-op",
      "backend": [
        {
          "method": "GET",
          "encoding": "no-op",
          "host": [
            "https://HOSTNAME"
          ],
          "url_pattern": "/__debug/default"
        }
      ]
    },
    {
      "endpoint": "/seaweedfs/{segment1}/{segment2}/{segment3}/",
      "method": "GET",
      "output_encoding": "no-op",
      "backend": [
        {
          "method": "GET",
          "encoding": "no-op",
          "url_pattern": "/{segment1}/{segment2}/{segment3}/",
          "host": [
            "http://seaweedfs:8333"
          ]
        }
      ]
    }
  ]
}

Configuration check output:
Result of krakend check -dtc krakend.json --lint command

Parsing configuration file: /etc/krakend/krakend.json
Global settings
	Name: KrakenD - API Gateway
	Version: 3
	Address: 
	Port: 8080
	TLS Disabled: false
	TLS Public key: 
	TLS Private key: 
	TLS Enable MTLS: false
	TLS Disable System CA Pool: false
2 API endpoint(s):
	- GET /debug/seaweedfs/:segment1/:segment2/:segment3/
	Timeout: 3s
	Connecting to 1 backend(s):
		[+] GET /__debug/default
		Timeout: 3s
		Hosts: [https://hostname]

	- GET /seaweedfs/:segment1/:segment2/:segment3/
	Timeout: 3s
	Connecting to 1 backend(s):
		[+] GET /{{.Segment1}}/{{.Segment2}}/{{.Segment3}}/
		Timeout: 3s
		Hosts: [http://seaweedfs:8333]

0 async agent(s):
Syntax OK!

Commands used:
How did you start the software?

docker-compose -f docker-compose.yml up -d

Docker Compose

volumes:
  seaweed_data: {}


services:
  seaweedfs:
    image: chrislusf/seaweedfs
    container_name: container-seaweedfs
    ports:
      - "9333:9333" # Master port
      - "8080:8080" # Volume server port
      - "8333:8333" # S3 API port
      - "8888:8888" # Filer port
    volumes:
      - seaweed_data:/data
    command: server -s3

  krakend:
    build:
      context: ./krakend
      dockerfile: ./Dockerfile
    image: image-krakend
    container_name: container-krakend
    ports:
      - '443:8080'
    volumes:
      - /path-to-certs/:ro

Dockerfile

FROM devopsfaith/krakend:2.7
COPY ./krakend.json /etc/krakend/krakend.json

# Copy the entrypoint script into the container
COPY ./entrypoint.sh /entrypoint.sh

# Uncomment with Enterprise image:
# COPY LICENSE /etc/krakend/LICENSE

RUN chmod +x /entrypoint.sh

# Set the entrypoint to our script (which sets permissions and starts KrakenD)
ENTRYPOINT ["/entrypoint.sh"]

# Set the command to run KrakenD with the specified config file
CMD ["krakend", "run", "--config", "/etc/krakend/krakend.json"]

Logs:
Curl from outside of docker:

curl -vL "https://HOSTNAME/seaweedfs/default-bucket/media/simple_image_1.png?AWSAccessKeyId=TEST&Signature=SIGNATURE&Expires=EXPIRATION


<Error><Code>NotImplemented</Code><Message>A header you provided implies functionality that is not implemented</Message><Resource>

Debug KrakenD Endpoint:

KRAKEND DEBUG: [ENDPOINT: /__debug/*] Method: GET
KRAKEND DEBUG: [ENDPOINT: /__debug/*] URL: /__debug/default
KRAKEND DEBUG: [ENDPOINT: /__debug/*] Query: map[]
KRAKEND DEBUG: [ENDPOINT: /__debug/*] Params: [{param /default}]
KRAKEND DEBUG: [ENDPOINT: /__debug/*] Headers: map[Accept:[*/*] Accept-Encoding:[gzip] User-Agent:[curl/7.81.0] X-Forwarded-For:[] X-Forwarded-Host:[HOSTNAME] X-Forwarded-Via:[KrakenD Version 2.7.2]]
2024/12/16 06:30:27 KRAKEND DEBUG: [ENDPOINT: /__debug/*] Body: 
[GIN] 2024/12/16 - 06:30:27 | 200 |     131.484µs |    185.28.23.70 | GET      "/__debug/default"
[GIN] 2024/12/16 - 06:30:27 | 200 |    31.07885ms |    185.28.23.70 | GET      "/debug/seaweedfs

Curl from inside KrakenD docker container (successful):

curl -vL -H "Accept: */*" -H "Accept-Encoding: gzip" -H "User-Agent: curl/7.81.0" -H "X-Forwarded-For: " -H "X-Forwarded-Host:HOSTNAME" -H "X-Forwarded-Via: KrakenD Version 2.7.2" "https://HOSTNAME/seaweedfs/default-bucket/media/simple_image_1.png?AWSAccessKeyId=TEST&Signature=SIGNATURE&Expires=EXPIRATION

HTTP/1.1 200 OK
Accept-Ranges: bytes
Access-Control-Expose-Headers: Content-Disposition
Content-Disposition: inline; filename="simple_image_1.png"
Content-Length: 75
Content-Type: image/png
Date: Mon, 16 Dec 2024 07:50:03 GMT
Etag: ""
Last-Modified: Mon, 16 Dec 2024 06:55:58 GMT
Server: SeaweedFS 30GB 3.79

Additional comments:
The error I get from SeaweedFS by going through KrakenD is A header you provided implies functionality that is not implemented.
This error occurs when I put "input_headers": ["*"] and when I don't put any input_headers at all.
When I go into KrakenD docker container and do a curl request everything works.
When I go into KrakenD docker container and do a curl request and add all the headers that KrakenD puts (found via debugging endpoint) it works.
Therefore, I suspect that KrakenD adds more headers than it is listing.

[alombarte]

All the headers KrakenD sends to a backend are shown in the __debug output you pasted. There's nothing else that is sent. You can verify this by looking into the source code or implementing an echo functionality if you don't believe it. The __echo endpoint does the same thing using a different code base.

You pasted the output of curl -v but I don't see the request headers (the lines starting with >).

I would start by removing the * from the input headers, which I suspect is the culprit. Then, try one by one which headers you want to pass.

[qtpi-bonding]

Thank you for your help and reply. I appreciate it greatly.

I don't pass any explicit headers in the curl command:
curl -vL URL

Here is the curl:
`

GET /seaweedfs/default-bucket
Host: HOSTNAME
user-agent: curl/7.81.0
accept: /

• TLSv1.2 (IN), TLS header, Supplemental data (23):
• TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
• TLSv1.2 (IN), TLS header, Supplemental data (23):
• Connection state changed (MAX_CONCURRENT_STREAMS == 250)!
• TLSv1.2 (OUT), TLS header, Supplemental data (23):
• TLSv1.2 (IN), TLS header, Supplemental data (23):
• TLSv1.2 (IN), TLS header, Supplemental data (23):
• TLSv1.2 (IN), TLS header, Supplemental data (23):
  < HTTP/2 301
  < content-type: text/html; charset=utf-8
  < location: /seaweedfs/default-bucket/media/
  < content-length: 273
  < date: Mon, 16 Dec 2024 09:17:55 GMT
  <

GET /seaweedfs/default-bucket/media/
user-agent: curl/7.81.0
accept: /

< accept-ranges: bytes
< content-type: application/xml
< date: Mon, 16 Dec 2024 09:17:55 GMT
< server: SeaweedFS 30GB 3.79
< x-amz-request-id:
< x-krakend: Version 2.7.2
< x-krakend-completed: false
< content-length: 416
<

NotImplementedA header you provided implies functionality that is not implemented/default-bucket/media/
`

Here is the curl that works (directly to SeaweedFS):

`

GET /default-bucket/media/
Host: localhost
User-Agent: curl/7.81.0
Accept: /

• Mark bundle as not supporting multiuse
  < HTTP/1.1 200 OK
  < Accept-Ranges: bytes
  < Access-Control-Expose-Headers: Content-Disposition
  < Content-Disposition: inline; filename="FILE"
  < Content-Length: 75
  < Content-Type: image/png
  < Date: Mon, 16 Dec 2024 09:17:55 GMT
  < Etag:
  < Last-Modified: Mon, 16 Dec 2024 09:10:06 GMT
  < Server: SeaweedFS 30GB 3.79
  <
  Warning: Binary output can mess up your terminal. Use "--output -" to tell
  Warning: curl to output it to your terminal anyway, or consider "--output
  Warning: " to save to a file.
  `

I have completely removed the "input_headers": ["*"] (which should not pass any headers?) from the configuration file and it still gives the <Error><Code>NotImplemented</Code><Message>A header you provided implies functionality that is not implemented</Message><Resource>

[qtpi-bonding]

Could it be the query strings? Or would no-op be fine for those? I am confused because the error refers to the headers but I don't know what header it could be if I completely remove "input_headers": ["*"].

https://HOSTNAME/seaweedfs/default-bucket/media/FILENAME?AWSAccessKeyId=ACCESS_CODE&Signature=SIGNATURE&Expires=EXPIRATION

[alombarte]

If you don't pass the query strings, these do not reach the backend. Add them in the endpoint:

"input_query_strings ": ["AWSAccessKeyId","Signature","Expires"]

You can also use * but is not recommended.

The no-op specifies how to treat the response (output), not the request.