Issue with invoking Lambda using plugin.

[mynameisgv]

Hello
I have a kong setup in aws with ELB. (https).
I tried to use the lambda plugin.
when I change the loadbalancer to http i do not see any issue. But with https I get
"curl: (92) HTTP/2 stream 1 was not closed cleanly: PROTOCOL_ERROR (err 1)"
Can I get some help or pointers on how to fix this. i tried using the http port from elb to kong instance as well. It does not work
How can i have https enabled

My Setup
3 kong ec2 boxes 0.11.0
One Application loadbalancer with https listner sending traffic to port 8443 in each box.
I also tried sending traffic to 8000(http).
But i get the same error with curl. i tried to call from postman and request just timesout.
with firefox or chrome, i see the response but the request does not get disconnected.
if i add --http1.1 option to curl it works.

Thanks

[kikito]

Hello, please provide the debug log of curl, (the options are -v or --trace)

[mynameisgv]

Here is the output.

== Info:   Trying 10.8.138.51...
== Info: TCP_NODELAY set
== Info: Connected to kong.lab.expts.net (10.8.138.51) port 443 (#0)
== Info: Initializing NSS with certpath: sql:/etc/pki/nssdb
== Info:   CAfile: none
  CApath: none
== Info: loaded libnssckbi.so
== Info: ALPN, server accepted to use h2
== Info: SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
== Info: Server certificate:
== Info:   subject: CN=*.lab.expts.net
== Info:   start date: Jan 31 00:00:00 2017 GMT
== Info:   expire date: Feb 28 12:00:00 2018 GMT
== Info:   common name: *.lab.expts.net
== Info:   issuer: CN=Amazon,OU=Server CA 1B,O=Amazon,C=US
== Info: Using HTTP2, server supports multi-use
== Info: Connection state changed (HTTP/2 confirmed)
== Info: Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
== Info: Using Stream ID: 1 (easy handle 0x55f205016e10)
=> Send header, 143 bytes (0x8f)
0000: 47 45 54 20 2f 6d 61 75 69 64 65 76 2f 66 6d 73 GET /mauidev/fms
0010: 2f 76 31 2f 70 75 72 63 68 61 73 65 6f 72 64 65 /v1/purchaseorde
0020: 72 2f 69 64 2f 73 65 61 72 63 68 3f 62 6f 6f 6b r/id/search?book
0030: 69 6e 67 49 74 65 6d 49 44 73 3d 37 38 31 31 36 ingItemIDs=78116
0040: 35 38 36 31 20 48 54 54 50 2f 32 0d 0a 48 6f 73 5861 HTTP/2..Hos
0050: 74 3a 20 6b 6f 6e 67 2e 6c 61 62 2e 65 78 70 74 t: kong.lab.expt
0060: 73 2e 6e 65 74 0d 0a 55 73 65 72 2d 41 67 65 6e s.net..User-Agen
0070: 74 3a 20 63 75 72 6c 2f 37 2e 35 33 2e 31 0d 0a t: curl/7.53.1..
0080: 41 63 63 65 70 74 3a 20 2a 2f 2a 0d 0a 0d 0a    Accept: /....
== Info: Connection state changed (MAX_CONCURRENT_STREAMS updated)!
<= Recv header, 13 bytes (0xd)
0000: 48 54 54 50 2f 32 20 32 30 30 20 0d 0a          HTTP/2 200 ..
<= Recv header, 37 bytes (0x25)
0000: 64 61 74 65 3a 20 4d 6f 6e 2c 20 33 30 20 4f 63 date: Mon, 30 Oc
0010: 74 20 32 30 31 37 20 31 36 3a 34 38 3a 35 37 20 t 2017 16:48:57
0020: 47 4d 54 0d 0a                                  GMT..
<= Recv header, 32 bytes (0x20)
0000: 63 6f 6e 74 65 6e 74 2d 74 79 70 65 3a 20 61 70 content-type: ap
0010: 70 6c 69 63 61 74 69 6f 6e 2f 6a 73 6f 6e 0d 0a plication/json..
<= Recv header, 20 bytes (0x14)
0000: 63 6f 6e 74 65 6e 74 2d 6c 65 6e 67 74 68 3a 20 content-length:
0010: 33 30 0d 0a                                     30..
<= Recv header, 35 bytes (0x23)
0000: 78 2d 61 6d 7a 6e 2d 72 65 6d 61 70 70 65 64 2d x-amzn-remapped-
0010: 63 6f 6e 74 65 6e 74 2d 6c 65 6e 67 74 68 3a 20 content-length:
0020: 30 0d 0a                                        0..
<= Recv header, 69 bytes (0x45)
0000: 78 2d 61 6d 7a 6e 2d 74 72 61 63 65 2d 69 64 3a x-amzn-trace-id:
0010: 20 72 6f 6f 74 3d 31 2d 35 39 66 37 35 37 66 39  root=1-59f757f9
0020: 2d 33 61 61 63 33 62 66 33 35 63 30 34 61 35 30 -3aac3bf35c04a50
0030: 37 31 66 39 33 63 65 31 32 3b 73 61 6d 70 6c 65 71f93ce12;sample
0040: 64 3d 30 0d 0a                                  d=0..
<= Recv header, 56 bytes (0x38)
0000: 78 2d 61 6d 7a 6e 2d 72 65 71 75 65 73 74 69 64 x-amzn-requestid
0010: 3a 20 33 38 65 30 30 36 31 65 2d 62 64 39 32 2d : 38e0061e-bd92-
0020: 31 31 65 37 2d 62 37 38 62 2d 38 64 35 35 31 61 11e7-b78b-8d551a
0030: 33 62 35 33 66 39 0d 0a                         3b53f9..
<= Recv header, 21 bytes (0x15)
0000: 73 65 72 76 65 72 3a 20 6b 6f 6e 67 2f 30 2e 31 server: kong/0.1
0010: 31 2e 30 0d 0a                                  1.0..
<= Recv header, 2 bytes (0x2)
0000: 0d 0a                                           ..
== Info: HTTP/2 stream 1 was not closed cleanly: PROTOCOL_ERROR (err 1)
== Info: Closing connection 0
curl: (92) HTTP/2 stream 1 was not closed cleanly: PROTOCOL_ERROR (err 1)
Show less

[mynameisgv]

If I use postman it does not even work.
In curl if i pass option --http1.1 it get a proper output.

[mynameisgv]

1. Mac curl version 7.54.0 (x86_64-apple-darwin16.0) libcurl/7.54.0 SecureTransport zlib/1.2.8 works
2. centos curk verl curl 7.53.1 (x86_64-redhat-linux-gnu) libcurl/7.53.1 NSS/3.32.1 zlib/1.2.11 libidn2/2.0.4 libpsl/0.18.0 (+libidn2/2.0.3) libssh2/1.8.0 nghttp2/1.21.1 does not work with out the http1.1 option
3. Java clients do nto work
4. postman does not work
5. chrome and firefox work, but the connection does not terminate.

[kikito]

Thanks for testing this so thoroughly. By your own results it seems to be a problem with curl 7.53.0 (fixed in 7.54.0). If you need it to work in environments where 7.54.0 is not available then my suggestion would be adding the http1.1 option or changing to a different client.

[mynameisgv]

Hello Kikito

How abt clients like postman or standard java http spring libraries. They also show similar problem.

Thanks

[pkorohod]

We are exposing this API to different clients, and most of them do not work. Browser-based hang because the read does not terminate, Java Apache HttpClient clients timeout most likely due to the same problem (unless the HTTP 1.0 protocol is forced).
This is not a problem with the client. Most likely it is related to the Kong - ELB pair.

[aaronhmiller]

Workaround is to pass in HTTP 1.1 protocol using request transformer. Should still be fixed.

[adamlc]

@aaronhmiller I'm also have this issue using curl and safari (chrome seems ok).

Can you provide an example of what I need to do in the request transformer?

Edit: So using curl verbosely I get the following:

* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
* http2 error: Invalid HTTP header field was received: frame type: 1, stream: 1, name: [connection], value: [keep-alive]
* HTTP/2 stream 1 was not closed cleanly: PROTOCOL_ERROR (err 1)
* Closing connection 0
* TLSv1.2 (OUT), TLS alert, Client hello (1):
curl: (92) HTTP/2 stream 1 was not closed cleanly: PROTOCOL_ERROR (err 1)

So it looks like the Connection header is getting converted back to http2, which isn't allowed. I can use the response transformer to remove this, but I unfortunately it still doesn't work correctly.

* HTTP/2 stream 1 was not closed cleanly: PROTOCOL_ERROR (err 1)
* Closing connection 0
* TLSv1.2 (OUT), TLS alert, Client hello (1):
curl: (92) HTTP/2 stream 1 was not closed cleanly: PROTOCOL_ERROR (err 1)

[gaffney]

@kikito I am encountering this with my first simple "Hello World" attempt. Hangs in Chrome and won't even load in Safari - definitely not a curl-specific issue.

* HTTP/2 stream 1 was not closed cleanly: PROTOCOL_ERROR (err 1)
* Closing connection 0
curl: (92) HTTP/2 stream 1 was not closed cleanly: PROTOCOL_ERROR (err 1)