Compatibility with Kong 1.x

[NikoGrano]

KongProxyController request error { name: 'invalid size',
  message: 'size must be an integer between 1 and 1000',
  code: 9 }

Using Konga 12.2 and 13.

Name        : kong-community-edition
Arch        : noarch
Version     : 1.0.0rc1
Release     : 1
Size        : 46 M
Repo        : installed
From repo   : bintray--kong-kong-community-edition-rpm
Summary     : Kong is a distributed gateway for APIs and Microservices, focused on
            : high performance and reliability.
URL         : https://getkong.org/
License     : ASL 2.0
Description : Kong is a distributed gateway for APIs and Microservices, focused on
            : high performance and reliability.

[NikoGrano]

This is caused by Kong 1.0.0. Please implement support for Kong 1.0.0. Thank You! 🥇

[jeremyjpj0916]

Same, testing Kong 1.0rc2 some today and see similar errors on all service/route/consumer/plugins/upstreams/certificates pages in our dev environment.

My guess is they don't return size in the way the admin API did in the past?

[jeremyjpj0916]

@pantsel I know the problem :P , Kong caught onto your max int size tricks and finally now limits you to 1000 per request.

Implement something recursive like such as this with the pagination logic(my java client code), and Konga should be all set!:

MAX_SIZE = 1000

public Consumers getKongConsumers()
    {
        Consumers kongConsumers = new Consumers();

        request = new Request.Builder()
                .header("Authorization", "Bearer " + jwt_token)
                .url(base_admin_url + "/consumers?size=" + MAX_SIZE) 
                .build();
        try
        {
            response = client.newCall(request).execute();
            //Got a valid response body?
            if(response != null && response.body() != null && response.isSuccessful())
            {
                kongConsumers = gson.fromJson(response.body().string(), Consumers.class);

                if(kongConsumers != null && kongConsumers.getOffset() != null)
                {
                    kongConsumers = getKongConsumers(kongConsumers.getOffset(), kongConsumers);
                }
            }
            else if(response != null && "401".equals(Integer.toString(response.code())))
            {
                renewJwtToken();
                return getKongConsumers();
            }
            else
            {
                //Empty response body, how do we want to handle this
            }
        }
        catch(IOException e)
        {
            //Send an email
            System.out.println("GetKongConsumers" + " Error: " + e.getMessage());
        }

        return kongConsumers;
    }

    public Consumers getKongConsumers(String offset, Consumers kongConsumers)
    {
        Consumers paginatedConsumers = new Consumers();
        try {
            request = new Request.Builder()
                    .header("Authorization", "Bearer " + jwt_token)
                    .url(base_admin_url + "/consumers?offset=" + URLEncoder.encode(offset, "UTF-8") + "&size=" + MAX_SIZE)
                    .build();
        } catch (UnsupportedEncodingException e) {
            e.printStackTrace();
        }
        try
        {
            response = client.newCall(request).execute();
            //Got a valid response body?
            if(response != null && response.body() != null && response.isSuccessful())
            {
                paginatedConsumers = gson.fromJson(response.body().string(), Consumers.class);
                kongConsumers.getData().addAll(paginatedConsumers.getData());
                if(paginatedConsumers.getOffset() != null)
                {
                    kongConsumers = getKongConsumers(paginatedConsumers.getOffset(), kongConsumers);
                }
            }
            else
            {
                //Empty response body, how do we want to handle this
            }
        }
        catch(IOException e)
        {
            //Send an email
            System.out.println("GetKongConsumers" + " Error: " + e.getMessage());
        }

        return kongConsumers;
    }

So no more of this in your js 😄
/assets/js/app/routes/00_routes.js

 'ServiceModel', function resolve(ServiceModel) {
                      return ServiceModel.load({
                        size : 4294967295
                      })

[NikoGrano]

Nice! Could you submit PR for this?

[pantsel]

@jeremyjpj0916 , yup, noticed that.
I just need to find some time to deal with the fix.
I believe that I will manage to get to it in the next couple of days

[jeremyjpj0916]

Excited to test it out! Once you get that out I will continue testing to see if there are any other compatibility issues between Kong 1.0 and the 0.14.x series Konga is already compatible with. Hopefully no more surprises hah.

[rsbrisci]

Bumping this for attention, we could also use this feature as Kong 1.0 will be released soon.

[NikoGrano]

Kong 1.0 will be released soon.
Isn't it released already? At least on my server Kong 1.0 is waiting to be updated and due this I need to hold that update and make YUM black magic to exclude it from automatic updates.

[nerdshack604]

Kong 1.0 GA dropped today -> https://konghq.com/blog/kong-1-0-ga/

[wilderchen]

I have encountered the same problem, anyone can make a PR to solve this issue?

[jeremyjpj0916]

@pantsel Did hack around and make Konga work with 1.0 just by forcing the 1000 limit on query in konga(without pagination because we do not have over 1k) as a quick hack. Seemed services/routes loaded. But the plugins pages and loading were messed up because of schema change violations so that will need some rework too I think, correct me if any of this is wrong @rsbrisci (I was not personally testing this because I am on a trip now, only relaying the info).

[wilderchen]

It seems that the konga isn't compatible with kong 1.0 right now. I found there are a lot of "{{humanizeLabel(key)}}" symbols when I trying to add a plugin

[dsteinkopf]

see also #315

[pantsel]

Hello guys, I know I've been absent for a while but I had some RL issues to address while waiting for Kong to finalize their 1.x version.

They've changed a number of things Konga was depending upon, and a s a result, issues like this and #315 started occuring.

The changes that need to be done to the project in order to achieve 1.x compatibility are not trivial and I need your help to verify that things are working as expected.

Kong 1.x compatibility is being worked on the next branch. Feel free try it out and drop some feedback. If you're using docker, pull pantsel/konga:next.

Bare in mind that while there may still be some broken features, the basic Kong API integration should be working just fine.

I'm yet to decide if Konga 0.14.x will be backwards compatible to Kong < 1.x. In case it won't, I will keep a branch of Konga 0.13.x active for the ones that don't want to upgrade their Kong installations just yet.

Cheers.

[jeremyjpj0916]

Cross posting from the gitter chat:
"
@pantsel Starting to look good, pagination seems to work on service/routes/plugins pages. Plugins page opening up the schema and fields seem to load fine. Consumers, their acl's and their creds seem to load fine(acl has been needing pagination and searchability for awhile but thats another issue for another day lol). So far 1 bug I have found. If you go into an individual service or an individual route, right now the plugins tab on either loads ALL plugins and not just the ones meant for that given service or route. Probably a simple fix not querying down on what needs to be.
"

[wilderchen]

Hello guys, I know I've been absent for a while but I had some RL issues to address while waiting for Kong to finalize their 1.x version.

They've changed a number of things Konga was depending upon, and a s a result, issues like this and #315 started occuring.

The changes that need to be done to the project in order to achieve 1.x compatibility are not trivial and I need your help to verify that things are working as expected.

Kong 1.x compatibility is being worked on the next branch. Feel free try it out and drop some feedback. If you're using docker, pull pantsel/konga:next.

Bare in mind that while there may still be some broken features, the basic Kong API integration should be working just fine.

I'm yet to decide if Konga 0.14.x will be backwards compatible to Kong < 1.x. In case it won't, I will keep a branch of Konga 0.13.x active for the ones that don't want to upgrade their Kong installations just yet.

Cheers.

After debugging into the konga project, I found that the kong 1.0 version bring quite a lot break changes. For example, the service schema interface response change the "fields" from dict/map/hash to list. It seems that it may need quite a lot time to follow up the new version of Kong.

[pantsel]

@wilderchen , have you noticed something concerning the services entity that is not working as expected on Konga:next due to that change? It seems to be working fine on my end.

[pantsel]

@jeremyjpj0916 ,

right now the plugins tab on either loads ALL plugins and not just the ones meant for that given service or route

This happens because their API stopped working with query strings for some reason.

/plugins?service_id={service_id} returns all plugins

only the /services/{service_id}/plugins is valid now.

For the exact same reason, the Eligible consumers on routes and services are not returning correct results.

/services/{service_id}/plugins?enabled=true is not working.

That's why it's hard to keep up with this project. They're breaking everything on nearly every release with no apparent reason. Almost feels like they're doing it on purpose.

Anyways, I'll make the needed changes for the plugins but I will have to wait and see if they will bring query string filters back before i start meddling with the eligible consumers logic.

Thanks again for the feedback.

[jeremyjpj0916]

@pantsel Good stuff! Yeah I think they are changing from query parameter to path based because its more "restful" or they consider it to be better taxonomy. I am not entirely sure. Maybe they are doing it just to make more work for yah 😆 . Either way will give your latest next branch commits a go and see if its looking good!

[pantsel]

Well IMHO that's neither more 'restful' nor better taxonomy. Quite the contrary, it's ver un-restful, plus apart from the results you can get through paths, now there's no way for example to only get a list of only enabled plugins for a service/route?

That doesn't make sense to me...

[jeremyjpj0916]

@pantsel

Creating an upstream resource seems to work against Kong 1.0, creating a target(ip:port) does not seem to work against Kong 1.0. Also snapshots does not work(but I consider it to be the lowest of priorities).

Edit - actually unconfirmed on the ^ with targets, my vpn disconnected from the network i was on when i claimed that lol. Will update later.

-Jeremy

[jeremyjpj0916]

@pantsel

Seems my upstream edit throws an interesting error, Images are as follows top / middle / bottom. All I really want is the active healthcheck by itself with no passive healthcheck anyways... Error in red at the top when i tried to save after deleting all the http statuses out of the auto-populated passive healthchecks section after my initial submit:

Edit -

Also Konga is lacking fields active/passive health checks expose(things like validate the certificate for https healthchecks):
https://docs.konghq.com/1.0.x/admin-api/#add-upstream

A few that seem important are:
healthchecks.active.type
healthchecks.active.https_verify_certificate

The actual targets page looks pretty solid in general, I like the heart next to the targets indicating health which i imagine is coming from the upstreams/{name or id}/health api call for the page 👍 . But adding the parameters that will help support https like I mentioned will be good as it seems right now I can't setup an HTTPS targets in Konga because I can't disable the ssl verify in UI(we are using self-signed certs).

Another point to make is Kong's active vs passive healthchecks documentation is confusing.. It seems to basically disable the use of one of them you need to keep tcp/http_failures / timeouts parameter to the default of 0 ?? I wish in the Konga UI I could do a toggle at the high level those sections(active vs passive) of (enable/disable, and disable won't even let the konga UI do the drop down action on that given active or passive path) and you let Konga internally do the heavy lifting to ensure these fields are defaulting properly so active health-checks are only running or say passive health-checks are only running. Maybe thats just a IMO, but right now it feels tricky having to think about the 0's and such, giving me a headache just thinking about it mate 😆 .

[jeremyjpj0916]

@pantsel I see you did some commits on Christmas 👍 , coding Santa over there lol. So new issue. ACL whitelist to a Service or Route does not seem to work at the moment:

EDIT , also seems the add JWT to service/route seems a bit off. Notice the Object in brackets bit for cookie names:

Edit Edit - tried posting too and noticed claims to verify I could not bubble up say the jwt exp claim as a field or anything in the UI, and the error at the top due to this is: "{"uri_param_names":"expected a set","claims_to_verify":"expected a set"} "

You are doing the lords work man, keep going strong towards this 1.0 compatibility as you are on a roll!

[pantsel]

I'll try to handle it somehow, maybe by populating everything there is in the schema regardless of whether the property exists in their reponse or not

[dsteinkopf]

Ok fine. But as you said, this is not the ideal solution...

[pantsel]

We work with what we have :)

[pantsel]

Strange, this is what Konga is sending to Kong on my env.
As you can see there is no config.blacklist defined.

{
  "created_at": 1546179319,
  "id": "ddf25733-116d-4dbc-b5a9-490c98b4d04d",
  "service": null,
  "enabled": true,
  "run_on": "first",
  "consumer": null,
  "route": null,
  "name": "acl",
  "config": {
    "whitelist": [
      "test"
    ],
    "hide_groups_header": false
  }
}

Still, it updates with no probs.

Im using Kong 1.0.0 from https://github.com/Kong/docker-kong/tree/master/alpine

Question, are you by any chance using Casandra as a datastore? Im using Pg here.

[dsteinkopf]

THAT is strange. I am also using kong 1.0.0 which I built myself from https://github.com/Kong/docker-kong (alpine directory) because kong 1.0 isn't yet available from https://hub.docker.com/_/kong.

Did you try to use curl and PATCH this JSON (without blacklist) ?

And no - I am also using pg.

[dsteinkopf]

BTW. What is your knowledge of the transformer plugin problem? (I am currently trying to add a request-transformer without success.)

[pantsel]

That is a Kong issue. The schema is messed up on 1.0.0
Kong/kong#4137

[jeremyjpj0916]

@pantsel Noted on the oauth change here for their docs: Kong/docs.konghq.com#1092 , if I get time later I will fix it myself, thanks for fixing it in Konga!

[jeremyjpj0916]

@pantsel Okay tested OAuth2.0 client cred creation now, seems to be working good. Did some regression testing and it seems the request termination plugin created on a route breaks once again xD -

To apply it to Kong via curl I did this:

curl -X POST http://localhost:8001/routes/c358bcf8-16ea-45cc-a1a3-0e0b6bddd006/plugins --data "name=request-termination"  --data "config.status_code=200" --data "config.message=Success"

And the json the above produces from Konga view:

{
  "created_at": 1546280442,
  "consumer": null,
  "id": "7fb1ff6b-a7c4-4c62-91a9-bc40100d7f7a",
  "service": null,
  "enabled": true,
  "run_on": "first",
  "name": "request-termination",
  "route": {
    "id": "c358bcf8-16ea-45cc-a1a3-0e0b6bddd006"
  },
  "config": {
    "status_code": 200,
    "content_type": null,
    "body": null,
    "message": "Success"
  }
}

I can taste the finish lineeeee.

[jeremyjpj0916]

Confirmed ^ problem was fixed by the commit. Thx.

[jeremyjpj0916]

@pantsel

So I wiped out my 500+service/route keyspace Kong sandbox and now went back to super barebones for some mem leak testing of Kong. I setup a keyspace and added 2 service/route pairs and that was absolutely it. I set the KONG_PLUGINS env field to "bundled" just since that is the Kong default. I have no custom plugins or anything now. Snapshots under these conditions now is throwing a MYSQL error it seems:

[jeremyjpj0916]

@pantsel ^ seems to have fixed the issue. I did try a restore snapshot and I have a oauth2 and jwt cred assigned to a user. I didn't restore it to an empty node but just clicked restore on a kong environment that already had that snapshot data and I see these errors:

I imagine if it successfully checks they are already present it should just count as a success and keep going right and add +1 to the imported row? Since services/plugins/routes/cousmers/acls seem to have taken that same action successfully.

Edit -
I also made a Kong 1.0 node with 5k service/route pairs with oaut2 and acl added to route. Then 5k consumers with jwt/oauth creds and a white-list group to them. I attempted a Konga backup of that and it ran spinning a really long time before stopping, I imagine a built in timeout? Does not seem to have made the backup after I waited minutes though.

[jeremyjpj0916]

Update, now testing 1.0.2, seems the request transformation plugin page works like a charm out of the box 👍 .

[dsteinkopf]

@jeremyjpj0916 , I can confirm this.

[pantsel]

So I guess we're ready to merge next to master?
I'll do some final tests and publish the new release.

Meanwhile, if you guys see anything unexpected, let me know.

[jeremyjpj0916]

@pantsel I would say it is certainly good enough to release, I would test snapshots and consumer eligible routes at scale with plugins and all enabled on the routes(see my earlier comment a few up with a snapshot import that failed).

Thanks!

[amitofile]

@Niko9911 I got same error on every page services, routes, certificates,

KongProxyController request error { name: 'invalid size',
message: 'size must be an integer between 1 and 1000',
code: 9 }

So I changed the value for,
size : 4294967295 to 999 (in assets/js/app/routes/00_routes.js)
itemsFetchSize: 4294967295 to 999 (in assets/js/app/core/services/ListConfigService.js)

This fixed the alert and started showing lists.
I hope this is fine.

Kong v1.0.2
konga v0.13.0

[pantsel]

@amitofile , Konga 0.13.0 is not compatible with Kong 1+. The change you made won't save the day:).

We're working on Kong 1+ compatibility on the next branch.

It's 99.9% ready and fully compatible with Kong 1+.

You might want to try that instead.

[NikoGrano]

@amitofile as pantsel said.... It's not about getting one thing working. If I fix that error, there will be 10 things to fix. Please do not spam this ticket. Let's focus developing full support for Kong 1.0.

Could you contribute to this issue @amitofile by downloading next branch and testing it against some edge cases? Would appropriate it a lot! 🥇

[amitofile]

@pantsel @Niko9911
Yes. next branch nailed it 👍
It also solved this #315
Thanks.

[pantsel]

Closing this issue due to housekeeping since Konga 0.14.1 is fully compatible with Kong 1,x.

Feel free to reopen if you stumble upon anything weird.