-
Notifications
You must be signed in to change notification settings - Fork 1.1k
/
SPEC.md
1205 lines (982 loc) · 53.5 KB
/
SPEC.md
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
# Container Network Interface (CNI) Specification
- [Container Network Interface (CNI) Specification](#container-network-interface-cni-specification)
- [Version](#version)
- [Released versions](#released-versions)
- [Overview](#overview)
- [Summary](#summary)
- [Section 1: Network configuration format](#section-1-network-configuration-format)
- [Configuration format](#configuration-format)
- [Plugin configuration objects:](#plugin-configuration-objects)
- [Example configuration](#example-configuration)
- [Version considerations](#version-considerations)
- [Section 2: Execution Protocol](#section-2-execution-protocol)
- [Overview](#overview-1)
- [Parameters](#parameters)
- [Errors](#errors)
- [CNI operations](#cni-operations)
- [`ADD`: Add container to network, or apply modifications](#add-add-container-to-network-or-apply-modifications)
- [`DEL`: Remove container from network, or un-apply modifications](#del-remove-container-from-network-or-un-apply-modifications)
- [`CHECK`: Check container's networking is as expected](#check-check-containers-networking-is-as-expected)
- [`STATUS`: Check plugin status](#status-check-plugin-status)
- [`VERSION`: probe plugin version support](#version-probe-plugin-version-support)
- [`GC`: Clean up any stale resources](#gc-clean-up-any-stale-resources)
- [Section 3: Execution of Network Configurations](#section-3-execution-of-network-configurations)
- [Lifecycle \& Ordering](#lifecycle--ordering)
- [Attachment Parameters](#attachment-parameters)
- [Adding an attachment](#adding-an-attachment)
- [Deleting an attachment](#deleting-an-attachment)
- [Checking an attachment](#checking-an-attachment)
- [Garbage-collecting a network](#garbage-collecting-a-network)
- [Deriving request configuration from plugin configuration](#deriving-request-configuration-from-plugin-configuration)
- [Deriving `runtimeConfig`](#deriving-runtimeconfig)
- [Section 4: Plugin Delegation](#section-4-plugin-delegation)
- [Delegated Plugin protocol](#delegated-plugin-protocol)
- [Delegated plugin execution procedure](#delegated-plugin-execution-procedure)
- [Section 5: Result Types](#section-5-result-types)
- [ADD Success](#add-success)
- [Delegated plugins (IPAM)](#delegated-plugins-ipam)
- [VERSION Success](#version-success)
- [Error](#error)
- [Version](#version-1)
- [Appendix: Examples](#appendix-examples)
- [Add example](#add-example)
- [Check example](#check-example)
- [Delete example](#delete-example)
## Version
This is CNI **spec** version **1.1.0**.
Note that this is **independent from the version of the CNI library and plugins** in this repository (e.g. the versions of [releases](https://github.com/containernetworking/cni/releases)).
#### Released versions
Released versions of the spec are available as Git tags.
| tag | spec permalink | major changes |
| ------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------- | --------------------------------- |
| [`spec-v1.0.0`](https://github.com/containernetworking/cni/releases/tag/spec-v1.0.0) | [spec at v1.0.0](https://github.com/containernetworking/cni/blob/spec-v1.0.0/SPEC.md) | Removed non-list configurations; removed `version` field of `interfaces` array |
| [`spec-v0.4.0`](https://github.com/containernetworking/cni/releases/tag/spec-v0.4.0) | [spec at v0.4.0](https://github.com/containernetworking/cni/blob/spec-v0.4.0/SPEC.md) | Introduce the CHECK command and passing prevResult on DEL |
| [`spec-v0.3.1`](https://github.com/containernetworking/cni/releases/tag/spec-v0.3.1) | [spec at v0.3.1](https://github.com/containernetworking/cni/blob/spec-v0.3.1/SPEC.md) | none (typo fix only) |
| [`spec-v0.3.0`](https://github.com/containernetworking/cni/releases/tag/spec-v0.3.0) | [spec at v0.3.0](https://github.com/containernetworking/cni/blob/spec-v0.3.0/SPEC.md) | rich result type, plugin chaining |
| [`spec-v0.2.0`](https://github.com/containernetworking/cni/releases/tag/spec-v0.2.0) | [spec at v0.2.0](https://github.com/containernetworking/cni/blob/spec-v0.2.0/SPEC.md) | VERSION command |
| [`spec-v0.1.0`](https://github.com/containernetworking/cni/releases/tag/spec-v0.1.0) | [spec at v0.1.0](https://github.com/containernetworking/cni/blob/spec-v0.1.0/SPEC.md) | initial version |
*Do not rely on these tags being stable. In the future, we may change our mind about which particular commit is the right marker for a given historical spec version.*
## Overview
This document proposes a generic plugin-based networking solution for application containers on Linux, the _Container Networking Interface_, or _CNI_.
For the purposes of this proposal, we define three terms very specifically:
- _container_ is a network isolation domain, though the actual isolation technology is not defined by the specification. This could be a [network namespace][namespaces] or a virtual machine, for example.
- _network_ refers to a group of endpoints that are uniquely addressable that can communicate amongst each other. This could be either an individual container (as specified above), a machine, or some other network device (e.g. a router). Containers can be conceptually _added to_ or _removed from_ one or more networks.
- _runtime_ is the program responsible for executing CNI plugins.
- _plugin_ is a program that applies a specified network configuration.
This document aims to specify the interface between "runtimes" and "plugins". The key words "must", "must not", "required", "shall", "shall not", "should", "should not", "recommended", "may" and "optional" are used as specified in [RFC 2119][rfc-2119].
[namespaces]: http://man7.org/linux/man-pages/man7/namespaces.7.html
[rfc-2119]: https://www.ietf.org/rfc/rfc2119.txt
## Summary
The CNI specification defines:
1. A format for administrators to define network configuration.
2. A protocol for container runtimes to make requests to network plugins.
3. A procedure for executing plugins based on a supplied configuration.
4. A procedure for plugins to delegate functionality to other plugins.
5. Data types for plugins to return their results to the runtime.
## Section 1: Network configuration format
CNI defines a network configuration format for administrators. It contains
directives for both the container runtime as well as the plugins to consume. At
plugin execution time, this configuration format is interpreted by the runtime and
transformed in to a form to be passed to the plugins.
In general, the network configuration is intended to be static. It can conceptually
be thought of as being "on disk", though the CNI specification does not actually
require this.
### Configuration format
A network configuration consists of a JSON object with the following keys:
- `cniVersion` (string): [Semantic Version 2.0](https://semver.org) of CNI specification to which this configuration list and all the individual configurations conform. Currently "1.1.0"
- `cniVersions` (string list): List of all CNI versions which this configuration supports. See [version selection](#version-selection) below.
- `name` (string): Network name. This should be unique across all network configurations on a host (or other administrative domain). Must start with an alphanumeric character, optionally followed by any combination of one or more alphanumeric characters, underscore, dot (.) or hyphen (-). Must not contain characters disallowed in file paths.
- `disableCheck` (boolean): Either `true` or `false`. If `disableCheck` is `true`, runtimes must not call `CHECK` for this network configuration list. This allows an administrator to prevent `CHECK`ing where a combination of plugins is known to return spurious errors.
- `disableGC` (boolean): Either `true` or `false`. If `disableGC` is `true`, runtimes must not call `GC` for this network configuration list. This allows an administrator to prevent `GC`ing when it is known that garbage collection may have undesired effects (e.g. shared configuration between multiple runtimes).
- `loadOnlyInlinedPlugins` (boolean): Either `true` or `false`. If `false` (default), indicates [plugin configuration objects](#plugin-configuration-objects) can be aggregated from multiple sources. Any valid plugin configuration objects aggregated from other sources must be appended to the final list of `plugins` for that network name. If set to `true`, indicates that valid plugin configuration objects aggregated from sources other than the main network configuration will be ignored. If `plugins` is not present in the network configuration, `loadOnlyInlinedPlugins` cannot be set to `true`.
- `plugins` (list): A list of inlined [plugin configuration objects](#plugin-configuration-objects). If this key is populated with inlined plugin objects, and `loadOnlyInlinedPlugins` is true, the final set of plugins for a network must consist of all the plugin objects in this list, merged with all the plugins loaded from the sibling folder with the same name as the network.
#### Plugin configuration objects:
Runtimes may aggregate plugin configuration objects from multiple sources, and must unambiguously associate each loaded plugin configuration object with a single, valid network configuration. All aggregated plugin configuration objects must be validated, and each plugin with a valid configuration object must be invoked.
Plugin configuration objects may contain additional fields beyond the ones defined here. The runtime MUST pass through these fields, unchanged, to the invoked plugin, as defined in section 3.
**Required keys:**
- `type` (string): Matches the name of the CNI plugin binary on disk. Must not contain characters disallowed in file paths for the system (e.g. / or \\).
**Optional keys, used by the protocol:**
- `capabilities` (dictionary): Defined in [section 3](#Deriving-runtimeConfig)
**Reserved keys, used by the protocol:**
These keys are generated by the runtime at execution time, and thus should not be used in configuration.
- `runtimeConfig`
- `args`
- Any keys starting with `cni.dev/`
**Optional keys, well-known:**
These keys are not used by the protocol, but have a standard meaning to plugins.
Plugins that consume any of these configuration keys should respect their intended semantics.
- `ipMasq` (boolean): If supported by the plugin, sets up an IP masquerade on the host for this network. This is necessary if the host will act as a gateway to subnets that are not able to route to the IP assigned to the container.
- `ipam` (dictionary): Dictionary with IPAM (IP Address Management) specific values:
- `type` (string): Refers to the filename of the IPAM plugin executable. Must not contain characters disallowed in file paths for the system (e.g. / or \\).
- `dns` (dictionary, optional): Dictionary with DNS specific values:
- `nameservers` (list of strings, optional): list of a priority-ordered list of DNS nameservers that this network is aware of. Each entry in the list is a string containing either an IPv4 or an IPv6 address.
- `domain` (string, optional): the local domain used for short hostname lookups.
- `search` (list of strings, optional): list of priority ordered search domains for short hostname lookups. Will be preferred over `domain` by most resolvers.
- `options` (list of strings, optional): list of options that can be passed to the resolver
**Other keys:**
Plugins may define additional fields that they accept and may generate an error if called with unknown fields. Runtimes must preserve unknown fields in plugin configuration objects when transforming for execution.
#### Example configuration
The following is an example JSON representation of a network configuration `dbnet` with three plugin configurations (`bridge`, `tuning`, and `portmap`).
```jsonc
{
"cniVersion": "1.1.0",
"cniVersions": ["0.3.1", "0.4.0", "1.0.0", "1.1.0"],
"name": "dbnet",
"plugins": [
{
"type": "bridge",
// plugin specific parameters
"bridge": "cni0",
"keyA": ["some more", "plugin specific", "configuration"],
"ipam": {
"type": "host-local",
// ipam specific
"subnet": "10.1.0.0/16",
"gateway": "10.1.0.1",
"routes": [
{"dst": "0.0.0.0/0"}
]
},
"dns": {
"nameservers": [ "10.1.0.1" ]
}
},
{
"type": "tuning",
"capabilities": {
"mac": true
},
"sysctl": {
"net.core.somaxconn": "500"
}
},
{
"type": "portmap",
"capabilities": {"portMappings": true}
}
]
}
```
### Version considerations
CNI runtimes, plugins, and network configurations may support multiple CNI specification versions independently. Plugins indicate their set of supported versions through the VERSION command, while network configurations indicate their set of supported versions through the `cniVersion` and `cniVersions` fields.
CNI runtimes MUST select the highest supported version from the set of network configuration versions given by the `cniVersion` and `cniVersions` fields. Runtimes MAY consider the set of supported plugin versions as reported by the VERSION command when determining available versions.
The CNI protocol follows Semantic Versioning principles, so the configuration format MUST remain backwards and forwards compatible within major versions.
## Section 2: Execution Protocol
### Overview
The CNI protocol is based on execution of binaries invoked by the container runtime. CNI defines the protocol between the plugin binary and the runtime.
A CNI plugin is responsible for configuring a container's network interface in some manner. Plugins fall in to two broad categories:
* "Interface" plugins, which create a network interface inside the container and ensure it has connectivity.
* "Chained" plugins, which adjust the configuration of an already-created interface (but may need to create more interfaces to do so).
The runtime passes parameters to the plugin via environment variables and configuration. It supplies configuration via stdin. The plugin returns
a [result](#Section-5-Result-Types) on stdout on success, or an error on stderr if the operation fails. Configuration and results are encoded in JSON.
Parameters define invocation-specific settings, whereas configuration is, with some exceptions, the same for any given network.
The runtime must execute the plugin in the runtime's networking domain. (For most cases, this means the root network namespace / `dom0`).
### Parameters
Protocol parameters are passed to the plugins via OS environment variables.
- `CNI_COMMAND`: indicates the desired operation; `ADD`, `DEL`, `CHECK`, `GC`, or `VERSION`.
- `CNI_CONTAINERID`: Container ID. A unique plaintext identifier for a container, allocated by the runtime. Must not be empty. Must start with an alphanumeric character, optionally followed by any combination of one or more alphanumeric characters, underscore (), dot (.) or hyphen (-).
- `CNI_NETNS`: A reference to the container's "isolation domain". If using network namespaces, then a path to the network namespace (e.g. `/run/netns/[nsname]`)
- `CNI_IFNAME`: Name of the interface to create inside the container; if the plugin is unable to use this interface name it must return an error.
- `CNI_ARGS`: Extra arguments passed in by the user at invocation time. Alphanumeric key-value pairs separated by semicolons; for example, "FOO=BAR;ABC=123"
- `CNI_PATH`: List of paths to search for CNI plugin executables. Paths are separated by an OS-specific list separator; for example ':' on Linux and ';' on Windows
### Errors
A plugin must exit with a return code of 0 on success, and non-zero on failure. If the plugin encounters an error, it should output an ["error" result structure](#Error) (see below).
### CNI operations
CNI defines 5 operations: `ADD`, `DEL`, `CHECK`, `GC`, and `VERSION`. These are passed to the plugin via the `CNI_COMMAND` environment variable.
#### `ADD`: Add container to network, or apply modifications
A CNI plugin, upon receiving an `ADD` command, should either
- create the interface defined by `CNI_IFNAME` inside the container at `CNI_NETNS`, or
- adjust the configuration of the interface defined by `CNI_IFNAME` inside the container at `CNI_NETNS`.
If the CNI plugin is successful, it must output a [result structure](#Success) (see below) on standard out. If the plugin was supplied a `prevResult` as part of its input configuration, it MUST handle `prevResult` by either passing it through, or modifying it appropriately.
If an interface of the requested name already exists in the container, the CNI plugin MUST return with an error.
A runtime should not call `ADD` twice (without an intervening DEL) for the same `(CNI_CONTAINERID, CNI_IFNAME)` tuple. This implies that a given container ID may be added to a specific network more than once only if each addition is done with a different interface name.
**Input:**
The runtime will provide a JSON-serialized plugin configuration object (defined below) on standard in.
Required environment parameters:
- `CNI_COMMAND`
- `CNI_CONTAINERID`
- `CNI_NETNS`
- `CNI_IFNAME`
Optional environment parameters:
- `CNI_ARGS`
- `CNI_PATH`
#### `DEL`: Remove container from network, or un-apply modifications
A CNI plugin, upon receiving a `DEL` command, should either
- delete the interface defined by `CNI_IFNAME` inside the container at `CNI_NETNS`, or
- undo any modifications applied in the plugin's `ADD` functionality
Plugins should generally complete a `DEL` action without error even if some resources are missing. For example, an IPAM plugin should generally release an IP allocation and return success even if the container network namespace no longer exists, unless that network namespace is critical for IPAM management. While DHCP may usually send a 'release' message on the container network interface, since DHCP leases have a lifetime this release action would not be considered critical and no error should be returned if this action fails. For another example, the `bridge` plugin should delegate the DEL action to the IPAM plugin and clean up its own resources even if the container network namespace and/or container network interface no longer exist.
Plugins MUST accept multiple `DEL` calls for the same (`CNI_CONTAINERID`, `CNI_IFNAME`) pair, and return success if the interface in question, or any modifications added, are missing.
**Input:**
The runtime will provide a JSON-serialized plugin configuration object (defined below) on standard in.
Required environment parameters:
- `CNI_COMMAND`
- `CNI_CONTAINERID`
- `CNI_IFNAME`
Optional environment parameters:
- `CNI_NETNS`
- `CNI_ARGS`
- `CNI_PATH`
#### `CHECK`: Check container's networking is as expected
`CHECK` is a way for a runtime to probe the status of an existing container.
Plugin considerations:
- The plugin must consult the `prevResult` to determine the expected interfaces and addresses.
- The plugin must allow for a later chained plugin to have modified networking resources, e.g. routes, on `ADD`.
- The plugin should return an error if a resource included in the CNI Result type (interface, address or route) was created by the plugin, and is listed in `prevResult`, but is missing or in an invalid state.
- The plugin should return an error if other resources not tracked in the Result type such as the following are missing or are in an invalid state:
- Firewall rules
- Traffic shaping controls
- IP reservations
- External dependencies such as a daemon required for connectivity
- etc.
- The plugin should return an error if it is aware of a condition where the container is generally unreachable.
- The plugin must handle `CHECK` being called immediately after an `ADD`, and therefore should allow a reasonable convergence delay for any asynchronous resources.
- The plugin should call `CHECK` on any delegated (e.g. IPAM) plugins and pass any errors on to its caller.
Runtime considerations:
- A runtime must not call `CHECK` for a container that has not been `ADD`ed, or has been `DEL`eted after its last `ADD`.
- A runtime must not call `CHECK` if `disableCheck` is set to `true` in the [configuration](#configuration-format).
- A runtime must include a `prevResult` field in the network configuration containing the `Result` of the immediately preceding `ADD` for the container. The runtime may wish to use libcni's support for caching `Result`s.
- A runtime may choose to stop executing `CHECK` for a chain when a plugin returns an error.
- A runtime may execute `CHECK` from immediately after a successful `ADD`, up until the container is `DEL`eted from the network.
- A runtime may assume that a failed `CHECK` means the container is permanently in a misconfigured state.
**Input:**
The runtime will provide a json-serialized plugin configuration object (defined below) on standard in.
Required environment parameters:
- `CNI_COMMAND`
- `CNI_CONTAINERID`
- `CNI_NETNS`
- `CNI_IFNAME`
Optional environment parameters:
- `CNI_ARGS`
- `CNI_PATH`
All parameters, with the exception of `CNI_PATH`, must be the same as the corresponding `ADD` for this container.
#### `STATUS`: Check plugin status
`STATUS` is a way for a runtime to determine the readiness of a network plugin.
A plugin must exit with a zero (success) return code if the plugin is ready to service ADD requests. If the plugin knows that it is not able to service ADD requests, it must exit with a non-zero return code and output an error on standard out (see below).
For example, if a plugin relies on an external service or daemon, it should return an error to `STATUS` if that service is unavailable. Likewise, if a plugin has a limited number of resources (e.g. IP addresses, hardware queues), it should return an error if those resources are exhausted and no new `ADD` requests can be serviced.
The following error codes are defined in the context of `STATUS`:
- 50: The plugin is not available (i.e. cannot service `ADD` requests)
- 51: The plugin is not available, and existing containers in the network may have limited connectivity.
Plugin considerations:
- Status is purely informational. A plugin MUST NOT rely on `STATUS` being called.
- Plugins should always expect other CNI operations (like `ADD`, `DEL`, etc) even if `STATUS` returns an error. `STATUS` does not prevent other runtime requests.
- If a plugin relies on a delegated plugin (e.g. IPAM) to service `ADD` requests, it must also execute a `STATUS` request to that plugin when it receives a `STATUS` request for itself. If the delegated plugin return an error result, the executing plugin should return an error result.
**Input:**
The runtime will provide a json-serialized plugin configuration object (defined below) on standard in.
Optional environment parameters:
- `CNI_PATH`
#### `VERSION`: probe plugin version support
The plugin should output via standard-out a json-serialized version result object (see below).
**Input:**
A json-serialized object, with the following key:
- `cniVersion`: The version of the protocol in use.
Required environment parameters:
- `CNI_COMMAND`
#### `GC`: Clean up any stale resources
The GC command provides a way for runtimes to specify the expected set of attachments to a network.
The network plugin may then remove any resources related to attachments that do not exist in this set.
Resources may, for example, include:
- IPAM reservations
- Firewall rules
A plugin SHOULD remove as many stale resources as possible. For example, a plugin should remove any IPAM reservations associated with attachments not in the provided list. The plugin MAY assume that the isolation domain (e.g. network namespace) has been deleted, and thus any resources (e.g. network interfaces) therein have been removed.
Plugins should generally complete a `GC` action without error. If an error is encountered, a plugin should continue; removing as many resources as possible, and report the errors back to the runtime.
Plugins MUST, additionally, forward any GC calls to delegated plugins they are configured to use (see section 4).
The runtime MUST NOT use GC as a substitute for DEL. Plugins may be unable to clean up some resources from GC that they would have been able to clean up from DEL.
**Input:**
The runtime must provide a JSON-serialized plugin configuration object (defined below) on standard in. It contains an additional key;
- `cni.dev/valid-attachments` (array of objects): The list of **still valid** attachments to this network:
- `containerID` (string): the value of CNI_CONTAINERID as provided during the CNI ADD operation
- `ifname` (string): the value of CNI_IFNAME as provided during the CNI ADD operation
Required environment parameters:
- `CNI_COMMAND`
- `CNI_PATH`
**Output:**
No output on success, ["error" result structure](#Error) on error.
## Section 3: Execution of Network Configurations
This section describes how a container runtime interprets a network configuration (as defined in section 1) and executes plugins accordingly. A runtime may wish to _add_, _delete_, or _check_ a network configuration in a container. This results in a series of plugin `ADD`, `DELETE`, or `CHECK` executions, correspondingly. This section also defines how a network configuration is transformed and provided to the plugin.
The operation of a network configuration on a container is called an _attachment_. An attachment may be uniquely identified by the `(CNI_CONTAINERID, CNI_IFNAME)` tuple.
### Lifecycle & Ordering
- The container runtime must create a new network namespace for the container before invoking any plugins.
- The container runtime must not invoke parallel operations for the same container, but is allowed to invoke parallel operations for different containers. This includes across multiple attachments.
- **Exception**: The runtime must exclusively execute either _gc_ or _add_ and _delete_. The runtime must ensure that no _add_ or _delete_ operations are in progress before executing _gc_, and must wait for _gc_ to complete before issuing new _add_ or _delete_ commands.
- Plugins must handle being executed concurrently across different containers. If necessary, they must implement locking on shared resources (e.g. IPAM databases).
- The container runtime must ensure that _add_ is eventually followed by a corresponding _delete_. The only exception is in the event of catastrophic failure, such as node loss. A _delete_ must still be executed even if the _add_ fails.
- _delete_ may be followed by additional _deletes_.
- The network configuration should not change between _add_ and _delete_.
- The network configuration should not change between _attachments_.
- The container runtime is responsible for cleanup of the container's network namespace.
### Attachment Parameters
While a network configuration should not change between _attachments_, there are certain parameters supplied by the container runtime that are per-attachment. They are:
- **Container ID:** A unique plaintext identifier for a container, allocated by the runtime. Must not be empty. Must start with an alphanumeric character, optionally followed by any combination of one or more alphanumeric characters, underscore (), dot (.) or hyphen (-). During execution, always set as the `CNI_CONTAINERID` parameter.
- **Namespace**: A reference to the container's "isolation domain". If using network namespaces, then a path to the network namespace (e.g. `/run/netns/[nsname]`). During execution, always set as the `CNI_NETNS` parameter.
- **Container interface name**: Name of the interface to create inside the container. During execution, always set as the `CNI_IFNAME` parameter.
- **Generic Arguments**: Extra arguments, in the form of key-value string pairs, that are relevant to a specific attachment. During execution, always set as the `CNI_ARGS` parameter.
- **Capability Arguments**: These are also key-value pairs. The key is a string, whereas the value is any JSON-serializable type. The keys and values are defined by [convention](CONVENTIONS.md).
Furthermore, the runtime must be provided a list of paths to search for CNI plugins. This must also be provided to plugins during execution via the `CNI_PATH` environment variable.
### Adding an attachment
For every configuration defined in the `plugins` key of the network configuration,
1. Look up the executable specified in the `type` field. If this does not exist, then this is an error.
2. Derive request configuration from the plugin configuration, with the following parameters:
- If this is the first plugin in the list, no previous result is provided,
- For all additional plugins, the previous result is the result of the previous plugins.
3. Execute the plugin binary, with `CNI_COMMAND=ADD`. Provide parameters defined above as environment variables. Supply the derived configuration via standard in.
4. If the plugin returns an error, halt execution and return the error to the caller.
The runtime must store the result returned by the final plugin persistently, as it is required for _check_ and _delete_ operations.
### Deleting an attachment
Deleting a network attachment is much the same as adding, with a few key differences:
- The list of plugins is executed in **reverse order**
- The previous result provided is always the final result of the _add_ operation.
For every plugin defined in the `plugins` key of the network configuration, *in reverse order*,
1. Look up the executable specified in the `type` field. If this does not exist, then this is an error.
2. Derive request configuration from the plugin configuration, with the previous result from the initial _add_ operation.
3. Execute the plugin binary, with `CNI_COMMAND=DEL`. Provide parameters defined above as environment variables. Supply the derived configuration via standard in.
4. If the plugin returns an error, halt execution and return the error to the caller.
If all plugins return success, return success to the caller.
### Checking an attachment
The runtime may also ask every plugin to confirm that a given attachment is still functional. The runtime must use the same attachment parameters as it did for the _add_ operation.
Checking is similar to add with two exceptions:
- the previous result provided is always the final result of the _add_ operation.
- If the network configuration defines `disableCheck`, then always return success to the caller.
For every plugin defined in the `plugins` key of the network configuration,
1. Look up the executable specified in the `type` field. If this does not exist, then this is an error.
2. Derive request configuration from the plugin configuration, with the previous result from the initial _add_ operation.
3. Execute the plugin binary, with `CNI_COMMAND=CHECK`. Provide parameters defined above as environment variables. Supply the derived configuration via standard in.
4. If the plugin returns an error, halt execution and return the error to the caller.
If all plugins return success, return success to the caller.
### Garbage-collecting a network
The runtime may also ask every plugin in a network configuration to clean up any stale resources via the _GC_ command.
When garbage-collecting a configuration, there are no [Attachment Parameters](#attachment-parameters).
For every plugin defined in the `plugins` key of the network configuration,
1. Look up the executable specified in the `type` field. If this does not exist, then this is an error.
2. Derive request configuration from the plugin configuration.
3. Execute the plugin binary, with `CNI_COMMAND=GC`. Supply the derived configuration via standard in.
4. If the plugin returns an error, **continue** with execution, returning all errors to the caller.
If all plugins return success, return success to the caller.
### Deriving request configuration from plugin configuration
The network configuration format (which is a list of plugin configurations to execute) must be transformed to a format understood by the plugin (which is a single plugin configuration). This section describes that transformation.
The request configuration for a single plugin invocation is also JSON. It consists of the plugin configuration, primarily unchanged except for the specified additions and removals.
The following fields are always to be inserted into the request configuration by the runtime:
- `cniVersion`: the protocol version selected by the runtime - the string "1.1.0"
- `name`: taken from the `name` field of the network configuration
For attachment-specific operations (ADD, DEL, CHECK), additional field requirements apply:
- `runtimeConfig`: the runtime must insert an object consisting of the union of capabilities provided by the plugin and requested by the runtime (more details below).
- `prevResult`: the runtime must insert consisting of the result type returned by the "previous" plugin. The meaning of "previous" is defined by the specific operation (_add_, _delete_, or _check_). This field must not be set for the first _add_ in a chain.
- `capabilities`: must not be set
For GC operations:
- `cni.dev/valid-attachments`: as specified in section 2.
All other fields not prefixed with `cni.dev/` should be passed through unaltered.
#### Deriving `runtimeConfig`
Whereas CNI_ARGS are provided to all plugins, with no indication if they are going to be consumed, _Capability arguments_ need to be declared explicitly in configuration. The runtime, thus, can determine if a given network configuration supports a specific _capability_. Capabilities are not defined by the specification - rather, they are documented [conventions](CONVENTIONS.md).
As defined in section 1, the plugin configuration includes an optional key, `capabilities`. This example shows a plugin that supports the `portMapping` capability:
```json
{
"type": "myPlugin",
"capabilities": {
"portMappings": true
}
}
```
The `runtimeConfig` parameter is derived from the `capabilities` in the network configuration and the _capability arguments_ generated by the runtime. Specifically, any capability supported by the plugin configuration and provided by the runtime should be inserted in the `runtimeConfig`.
Thus, the above example could result in the following being passed to the plugin as part of the execution configuration:
```json
{
"type": "myPlugin",
"runtimeConfig": {
"portMappings": [ { "hostPort": 8080, "containerPort": 80, "protocol": "tcp" } ]
}
...
}
```
## Section 4: Plugin Delegation
There are some operations that, for whatever reason, cannot reasonably be implemented as a discrete chained plugin. Rather, a CNI plugin may wish to delegate some functionality to another plugin. One common example of this is IP address management.
As part of its operation, a CNI plugin is expected to assign (and maintain) an IP address to the interface and install any necessary routes relevant for that interface. This gives the CNI plugin great flexibility but also places a large burden on it. Many CNI plugins would need to have the same code to support several IP management schemes that users may desire (e.g. dhcp, host-local). A CNI plugin may choose to delegate IP management to another plugin.
To lessen the burden and make IP management strategy be orthogonal to the type of CNI plugin, we define a third type of plugin -- IP Address Management Plugin (IPAM plugin), as well as a protocol for plugins to delegate functionality to other plugins.
It is however the responsibility of the CNI plugin, rather than the runtime, to invoke the IPAM plugin at the proper moment in its execution. The IPAM plugin must determine the interface IP/subnet, Gateway and Routes and return this information to the "main" plugin to apply. The IPAM plugin may obtain the information via a protocol (e.g. dhcp), data stored on a local filesystem, the "ipam" section of the Network Configuration file, etc.
### Delegated Plugin protocol
Like CNI plugins, delegated plugins are invoked by running an executable. The executable is searched for in a predefined list of paths, indicated to the CNI plugin via `CNI_PATH`. The delegated plugin must receive all the same environment variables that were passed in to the CNI plugin. Just like the CNI plugin, delegated plugins receive the network configuration via stdin and output results via stdout.
Delegated plugins are provided the *complete network configuration* passed to the "upper" plugin. In other words, in the IPAM case, not just the `ipam` section of the configuration.
Success is indicated by a zero return code and a _Success_ result type output to stdout.
### Delegated plugin execution procedure
When a plugin executes a delegated plugin, it should:
- Look up the plugin binary by searching the directories provided in `CNI_PATH` environment variable.
- Execute that plugin with the same environment and configuration that it received.
- Ensure that the delegated plugin's stderr is output to the calling plugin's stderr.
If a plugin is executed with `CNI_COMMAND=CHECK`, `DEL`, or `GC`, it must also execute any delegated plugins. If any of the delegated plugins return error, error should be returned by the upper plugin.
If, on `ADD`, a delegated plugin fails, the "upper" plugin should execute again with `DEL` before returning failure.
## Section 5: Result Types
For certain operations, plugins must output result information. The output should be serialized as JSON on standard out.
### ADD Success
Plugins must output a JSON object with the following keys upon a successful `ADD` operation:
- `cniVersion`: The same version supplied on input - the string "1.1.0"
- `interfaces`: An array of all interfaces created by the attachment, including any host-level interfaces:
- `name` (string): The name of the interface.
- `mac` (string): The hardware address of the interface (if applicable).
- `mtu`: (uint) The MTU of the interface (if applicable).
- `sandbox` (string): The isolation domain reference (e.g. path to network namespace) for the interface, or empty if on the host. For interfaces created inside the container, this should be the value passed via `CNI_NETNS`.
- `socketPath` (string, optional): An absolute path to a socket file corresponding to this interface, if applicable.
- `pciID` (string, optional): The platform-specific identifier of the PCI device corresponding to this interface, if applicable.
- `ips`: IPs assigned by this attachment. Plugins may include IPs assigned external to the container.
- `address` (string): an IP address in CIDR notation (eg "192.168.1.3/24").
- `gateway` (string): the default gateway for this subnet, if one exists.
- `interface` (uint): the index into the `interfaces` list for a [CNI Plugin Result](#result) indicating which interface this IP configuration should be applied to.
- `routes`: Routes created by this attachment:
- `dst`: The destination of the route, in CIDR notation
- `gw`: The next hop address. If unset, a value in `gateway` in the `ips` array may be used.
- `mtu` (uint): The MTU (Maximum transmission unit) along the path to the destination.
- `advmss` (uint): The MSS (Maximal Segment Size) to advertise to these destinations when establishing TCP connections.
- `priority` (uint): The priority of route, lower is higher.
- `table` (uint): The table to add the route to.
- `scope` (uint): The scope of the destinations covered by the route prefix (global (0), link (253), host (254)).
- `dns`: a dictionary consisting of DNS configuration information
- `nameservers` (list of strings): list of a priority-ordered list of DNS nameservers that this network is aware of. Each entry in the list is a string containing either an IPv4 or an IPv6 address.
- `domain` (string): the local domain used for short hostname lookups.
- `search` (list of strings): list of priority ordered search domains for short hostname lookups. Will be preferred over `domain` by most resolvers.
- `options` (list of strings): list of options that can be passed to the resolver.
Plugins provided a `prevResult` key as part of their request configuration must output it as their result, with any possible modifications made by that plugin included. If a plugin makes no changes that would be reflected in the _Success result_ type, then it must output a result equivalent to the provided `prevResult`.
#### Delegated plugins (IPAM)
Delegated plugins may omit irrelevant sections.
Delegated IPAM plugins must return an abbreviated _Success_ object. Specifically, it is missing the `interfaces` array, as well as the `interface` entry in `ips`.
### VERSION Success
Plugins must output a JSON object with the following keys upon a `VERSION` operation:
- `cniVersion`: The value of `cniVersion` specified on input
- `supportedVersions`: A list of supported specification versions
Example:
```json
{
"cniVersion": "1.0.0",
"supportedVersions": [ "0.1.0", "0.2.0", "0.3.0", "0.3.1", "0.4.0", "1.0.0" ]
}
```
### Error
Plugins should output a JSON object with the following keys if they encounter an error:
- `cniVersion`: The protocol version in use - "1.1.0"
- `code`: A numeric error code, see below for reserved codes.
- `msg`: A short message characterizing the error.
- `details`: A longer message describing the error.
Example:
```json
{
"cniVersion": "1.1.0",
"code": 7,
"msg": "Invalid Configuration",
"details": "Network 192.168.0.0/31 too small to allocate from."
}
```
Error codes 0-99 are reserved for well-known errors. Values of 100+ can be freely used for plugin specific errors.
Error Code|Error Description
---|---
`1`|Incompatible CNI version
`2`|Unsupported field in network configuration. The error message must contain the key and value of the unsupported field.
`3`|Container unknown or does not exist. This error implies the runtime does not need to perform any container network cleanup (for example, calling the `DEL` action on the container).
`4`|Invalid necessary environment variables, like CNI_COMMAND, CNI_CONTAINERID, etc. The error message must contain the names of invalid variables.
`5`|I/O failure. For example, failed to read network config bytes from stdin.
`6`|Failed to decode content. For example, failed to unmarshal network config from bytes or failed to decode version info from string.
`7`|Invalid network config. If some validations on network configs do not pass, this error will be raised.
`11`|Try again later. If the plugin detects some transient condition that should clear up, it can use this code to notify the runtime it should re-try the operation later.
In addition, stderr can be used for unstructured output such as logs.
### Version
Plugins must output a JSON object with the following keys upon a `VERSION` operation:
- `cniVersion`: The value of `cniVersion` specified on input
- `supportedVersions`: A list of supported specification versions
Example:
```json
{
"cniVersion": "1.1.0",
"supportedVersions": [ "0.1.0", "0.2.0", "0.3.0", "0.3.1", "0.4.0", "1.0.0", "1.1.0" ]
}
```
## Appendix: Examples
We assume the network configuration [shown above](#Example-configuration) in section 1. For this attachment, the runtime produces `portmap` and `mac` capability args, along with the generic argument "argA=foo".
The examples uses `CNI_IFNAME=eth0`.
### Add example
The container runtime would perform the following steps for the `add` operation.
1) Call the `bridge` plugin with the following JSON, `CNI_COMMAND=ADD`:
```json
{
"cniVersion": "1.1.0",
"name": "dbnet",
"type": "bridge",
"bridge": "cni0",
"keyA": ["some more", "plugin specific", "configuration"],
"ipam": {
"type": "host-local",
"subnet": "10.1.0.0/16",
"gateway": "10.1.0.1"
},
"dns": {
"nameservers": [ "10.1.0.1" ]
}
}
```
The bridge plugin, as it delegates IPAM to the `host-local` plugin, would execute the `host-local` binary with the exact same input, `CNI_COMMAND=ADD`.
The `host-local` plugin returns the following result:
```json
{
"ips": [
{
"address": "10.1.0.5/16",
"gateway": "10.1.0.1"
}
],
"routes": [
{
"dst": "0.0.0.0/0"
}
],
"dns": {
"nameservers": [ "10.1.0.1" ]
}
}
```
The bridge plugin returns the following result, configuring the interface according to the delegated IPAM configuration:
```json
{
"ips": [
{
"address": "10.1.0.5/16",
"gateway": "10.1.0.1",
"interface": 2
}
],
"routes": [
{
"dst": "0.0.0.0/0"
}
],
"interfaces": [
{
"name": "cni0",
"mac": "00:11:22:33:44:55"
},
{
"name": "veth3243",
"mac": "55:44:33:22:11:11"
},
{
"name": "eth0",
"mac": "99:88:77:66:55:44",
"sandbox": "/var/run/netns/blue"
}
],
"dns": {
"nameservers": [ "10.1.0.1" ]
}
}
```
2) Next, call the `tuning` plugin, with `CNI_COMMAND=ADD`. Note that `prevResult` is supplied, along with the `mac` capability argument. The request configuration passed is:
```json
{
"cniVersion": "1.1.0",
"name": "dbnet",
"type": "tuning",
"sysctl": {
"net.core.somaxconn": "500"
},
"runtimeConfig": {
"mac": "00:11:22:33:44:66"
},
"prevResult": {
"ips": [
{
"address": "10.1.0.5/16",
"gateway": "10.1.0.1",
"interface": 2
}
],
"routes": [
{
"dst": "0.0.0.0/0"
}
],
"interfaces": [
{
"name": "cni0",
"mac": "00:11:22:33:44:55"
},
{
"name": "veth3243",
"mac": "55:44:33:22:11:11"
},
{
"name": "eth0",
"mac": "99:88:77:66:55:44",
"sandbox": "/var/run/netns/blue"
}
],
"dns": {
"nameservers": [ "10.1.0.1" ]
}
}
}
```
The plugin returns the following result. Note that the **mac** has changed.
```json
{
"ips": [
{
"address": "10.1.0.5/16",
"gateway": "10.1.0.1",
"interface": 2
}
],
"routes": [
{
"dst": "0.0.0.0/0"
}
],
"interfaces": [
{
"name": "cni0",
"mac": "00:11:22:33:44:55"
},
{
"name": "veth3243",
"mac": "55:44:33:22:11:11"
},
{
"name": "eth0",
"mac": "00:11:22:33:44:66",
"sandbox": "/var/run/netns/blue"
}
],
"dns": {
"nameservers": [ "10.1.0.1" ]
}
}
```
3) Finally, call the `portmap` plugin, with `CNI_COMMAND=ADD`. Note that `prevResult` matches that returned by `tuning`:
```json
{
"cniVersion": "1.1.0",
"name": "dbnet",
"type": "portmap",
"runtimeConfig": {
"portMappings" : [
{ "hostPort": 8080, "containerPort": 80, "protocol": "tcp" }
]
},
"prevResult": {
"ips": [
{
"address": "10.1.0.5/16",
"gateway": "10.1.0.1",
"interface": 2
}
],
"routes": [
{
"dst": "0.0.0.0/0"
}
],
"interfaces": [
{
"name": "cni0",
"mac": "00:11:22:33:44:55"
},
{
"name": "veth3243",
"mac": "55:44:33:22:11:11"
},
{
"name": "eth0",
"mac": "00:11:22:33:44:66",
"sandbox": "/var/run/netns/blue"
}
],
"dns": {
"nameservers": [ "10.1.0.1" ]
}
}
}
```
The `portmap` plugin outputs the exact same result as that returned by `bridge`, as the plugin has not modified anything that would change the result (i.e. it only created iptables rules).
### Check example
Given the previous _Add_, the container runtime would perform the following steps for the _Check_ action:
1) First call the `bridge` plugin with the following request configuration, including the `prevResult` field containing the final JSON response from the _Add_ operation, **including the changed mac**. `CNI_COMMAND=CHECK`
```json
{
"cniVersion": "1.1.0",
"name": "dbnet",
"type": "bridge",
"bridge": "cni0",
"keyA": ["some more", "plugin specific", "configuration"],
"ipam": {
"type": "host-local",
"subnet": "10.1.0.0/16",
"gateway": "10.1.0.1"
},
"dns": {
"nameservers": [ "10.1.0.1" ]
},
"prevResult": {
"ips": [
{
"address": "10.1.0.5/16",
"gateway": "10.1.0.1",
"interface": 2
}
],
"routes": [
{
"dst": "0.0.0.0/0"
}
],
"interfaces": [
{
"name": "cni0",
"mac": "00:11:22:33:44:55"
},
{
"name": "veth3243",
"mac": "55:44:33:22:11:11"
},
{
"name": "eth0",
"mac": "00:11:22:33:44:66",
"sandbox": "/var/run/netns/blue"
}
],
"dns": {
"nameservers": [ "10.1.0.1" ]
}
}
}
```
The `bridge` plugin, as it delegates IPAM, calls `host-local`, `CNI_COMMAND=CHECK`. It returns no error.
Assuming the `bridge` plugin is satisfied, it produces no output on standard out and exits with a 0 return code.
2) Next call the `tuning` plugin with the following request configuration:
```json
{
"cniVersion": "1.1.0",
"name": "dbnet",
"type": "tuning",
"sysctl": {
"net.core.somaxconn": "500"
},
"runtimeConfig": {
"mac": "00:11:22:33:44:66"
},
"prevResult": {
"ips": [
{
"address": "10.1.0.5/16",
"gateway": "10.1.0.1",
"interface": 2
}
],
"routes": [
{
"dst": "0.0.0.0/0"
}
],
"interfaces": [
{
"name": "cni0",
"mac": "00:11:22:33:44:55"
},
{
"name": "veth3243",
"mac": "55:44:33:22:11:11"
},
{
"name": "eth0",
"mac": "00:11:22:33:44:66",
"sandbox": "/var/run/netns/blue"
}
],
"dns": {
"nameservers": [ "10.1.0.1" ]
}
}
}
```