Nvmeof rdma qp shutdown

.gitignore

@@ -18,12 +18,13 @@ tags
 cscope.out
 dpdk-*
 CUnit-Memory-Dump.xml
-config.h
+include/spdk/config.h
 CONFIG.local
 *VC.db
 .vscode
 .project
 .cproject
 .settings
 mk/cc.mk
+mk/config.mk
 PYTHON_COMMAND

CHANGELOG.md

@@ -4,6 +4,10 @@
 
 ### nvme
 
+spdk_nvme_ctrlr_cmd_security_send() and spdk_nvme_ctrlr_cmd_security_receive()
+were added to support sending or receiving security protocol data to or from
+nvme controller.
+
 spdk_nvme_ns_get_extended_sector_size() was added.  This function includes
 the metadata size per sector (if any).  spdk_nvme_ns_get_sector_size() still
 returns only the data size per sector, not including metadata.
@@ -22,7 +26,32 @@ does not at this time confer any SPDK ABI compatibility claims.
 
 spdk_bdev_alias_del_all() was added to delete all alias from block device.
 
-### iscsi
+A new virtual bdev module has been added to perform at rest data encryption using the DPDK CryptoDev
+Framework.  The module initially uses a software AESNI CBC cipher with experimental support for the
+Intel QAT hardware accelerator also currently implemented with support for CBC cipher. Future work
+may include additional ciphers as well as consideration for authentication. NOTE: this module is
+currently marked as experimental.  Do not use in production.
+
+The RAID virtual bdev module is now always enabled by default.  The configure --with-raid and
+--without-raid options are now ignored and deprecated and will be removed in the next release.
+
+Enforcement of bandwidth limits for quality of service (QoS) has been added to the bdev layer.
+See the new [set_bdev_qos_limit](http://www.spdk.io/doc/jsonrpc.html#rpc_set_bdev_qos_limit)
+documentation for more details. The previous set_bdev_qos_limit_iops RPC method introduced at
+18.04 release has been deprecated. The new set_bdev_qos_limit RPC method can support both
+bandwidth and IOPS limits.
+
+### Environment Abstraction Layer and Event Framework
+
+The size parameter of spdk_mem_map_translate is now a pointer. This allows the
+function to report back the actual size of the translation relative to the original
+request made by the user.
+
+A new structure spdk_mem_map_ops has been introduced to hold memory map related
+callbacks. This structure is now passed as the second argument of spdk_mem_map_alloc
+in lieu of the notify callback.
+
+### iscsi target
 
 Parameter names of `set_iscsi_options` and `get_iscsi_global_params` RPC
 method for CHAP authentication in discovery sessions have been changed to
@@ -40,6 +69,12 @@ specified. This default has been removed. Users must now explicitly specify
 the location of this file to load CHAP shared secrets from a file, or use
 the related iSCSI RPC methods to add them at runtime.
 
+### iscsi initiator
+
+The SPDK iSCSI initiator is no longer considered experimental and becomes
+a first-class citizen among bdev modules. The basic usage has been briefly
+described in the bdev user guide: [iSCSI bdev](https://spdk.io/doc/bdev.html#bdev_config_iscsi)
+
 ### Miscellaneous
 
 The ReactorMask config file parameter has been deprecated.  Users should
@@ -50,6 +85,50 @@ Default config file pathnames have been removed from iscsi_tgt, nvmf_tgt
 and vhost.  Config file pathnames may now only be specified using the
 -c command line option.
 
+Users may no longer set DPDK_DIR in their environment to specify the
+location of the DPDK installation used to build SPDK.  Using DPDK_DIR
+has not been the documented nor recommended way to specify the DPDK
+location for several releases, but removing it ensures no unexpected
+surprises for users who may have DPDK_DIR defined for other reasons.
+Users should just use the "configure" script to specify the DPDK
+location before building SPDK.
+
+Although we know that many developers still use Python 2 we are officially
+switching to Python3 with requirement that all new code must be valid also
+for Python 2 up to the EOL which is year 2020.
+
+Invoking interpreter explicitly is forbidden for executable scripts. There
+is no need to use syntax like "python ./scripts/rpc.py". All executable
+scripts must contain proper shebang pointing to the right interpreter.
+Scripts without shebang musn't be executable.
+
+A Python script has been added to enable conversion of old INI config file
+to new JSON-RPC config file format. This script can be found at
+scripts/config_converter.py. Example how this script can be used:
+~~~{.sh}
+cat old_format.ini | scripts/config_converter.py > new_json_format.json
+~~~
+
+### Sock
+
+Two additional parameters were added to spdk_sock_get_addr() for the server
+port and client port. These parameters are named "sport" and "cport"
+respectively.
+
+### Virtio
+
+The following RPC commands have been deprecated:
+ - construct_virtio_user_scsi_bdev
+ - construct_virtio_pci_scsi_bdev
+ - construct_virtio_user_blk_bdev
+ - construct_virtio_pci_blk_bdev
+ - remove_virtio_scsi_bdev
+
+The `construct_virtio_*` ones were replaced with a single `construct_virtio_dev`
+command that can create any type of Virtio bdev(s). `remove_virtio_scsi_bdev`
+was replaced with `remove_virtio_bdev` that can delete both Virtio Block and SCSI
+devices.
+
 ## v18.07:
 
 ### bdev

CONFIG

@@ -32,81 +32,86 @@
 #
 
 # Installation prefix
-CONFIG_PREFIX?=/usr/local
+CONFIG_PREFIX="/usr/local"
 
 # Build with debug logging. Turn off for performance testing and normal usage
-CONFIG_DEBUG?=n
+CONFIG_DEBUG=n
 
-# Show backtrace when logging message at level <= lvl (ERROR, WARN, NOTICE, DEBUG)
-#CONFIG_LOG_BACKTRACE?=lvl
+# Build with support of backtrace printing in log messages. Requires libunwind.
+CONFIG_LOG_BACKTRACE=n
 
 # Treat warnings as errors (fail the build on any warning).
-CONFIG_WERROR?=n
+CONFIG_WERROR=n
 
 # Build with link-time optimization.
-CONFIG_LTO?=n
+CONFIG_LTO=n
 
 # Build with code coverage instrumentation.
-CONFIG_COVERAGE?=n
+CONFIG_COVERAGE=n
 
 # Build with Address Sanitizer enabled
-CONFIG_ASAN?=n
+CONFIG_ASAN=n
 
 # Build with Undefined Behavior Sanitizer enabled
-CONFIG_UBSAN?=n
+CONFIG_UBSAN=n
 
 # Build with Thread Sanitizer enabled
-CONFIG_TSAN?=n
+CONFIG_TSAN=n
 
 # Build tests
-CONFIG_TESTS?=y
+CONFIG_TESTS=y
 
 # Directory that contains the desired SPDK environment library.
 # By default, this is implemented using DPDK.
-CONFIG_ENV?=$(SPDK_ROOT_DIR)/lib/env_dpdk
+CONFIG_ENV=
 
 # This directory should contain 'include' and 'lib' directories for your DPDK
-# installation. Alternatively you can specify this on the command line
-# with 'make DPDK_DIR=/path/to/dpdk'. This is only a valid entry
-# when using the default SPDK environment library.
-CONFIG_DPDK_DIR?=$(SPDK_ROOT_DIR)/dpdk/build
+# installation.
+CONFIG_DPDK_DIR=
 
-# Build SPDK FIO plugin. Requires FIO_SOURCE_DIR set to a valid
+# Build SPDK FIO plugin. Requires CONFIG_FIO_SOURCE_DIR set to a valid
 # fio source code directory.
-CONFIG_FIO_PLUGIN?=n
+CONFIG_FIO_PLUGIN=n
 
 # This directory should contain the source code directory for fio
 # which is required for building the SPDK FIO plugin.
-FIO_SOURCE_DIR?=/usr/src/fio
+CONFIG_FIO_SOURCE_DIR=/usr/src/fio
 
 # Enable RDMA support for the NVMf target.
 # Requires ibverbs development libraries.
-CONFIG_RDMA?=n
+CONFIG_RDMA=n
+CONFIG_RDMA_SEND_WITH_INVAL=n
 
 # Build Ceph RBD support in bdev modules
 # Requires librbd development libraries
-CONFIG_RBD?=n
+CONFIG_RBD=n
 
 # Build vhost library.
-CONFIG_VHOST?=y
+CONFIG_VHOST=y
 
 # Build vhost initiator (Virtio) driver.
-CONFIG_VIRTIO?=y
+CONFIG_VIRTIO=y
 
 # Build with PMDK backends
-CONFIG_PMDK?=n
+CONFIG_PMDK=n
+CONFIG_PMDK_DIR=
 
 # Build with VPP
-CONFIG_VPP?=n
+CONFIG_VPP=n
+CONFIG_VPP_DIR=
 
 # Requires libiscsi development libraries.
-CONFIG_ISCSI_INITIATOR?=n
-
-# Build with raid
-CONFIG_RAID?=n
+CONFIG_ISCSI_INITIATOR=n
 
 # Enable the dependencies for building the crypto vbdev
-CONFIG_CRYPTO?=n
+CONFIG_CRYPTO=n
 
 # Build spdk shared libraries in addition to the static ones.
-CONFIG_SHARED?=n
+CONFIG_SHARED=n
+
+# Build with VTune suport.
+CONFIG_VTUNE=n
+CONFIG_VTUNE_DIR=
+
+# Build the dpdk igb_uio driver
+CONFIG_IGB_UIO_DRIVER=n

Makefile

@@ -41,7 +41,8 @@ DIRS-$(CONFIG_SHARED) += shared_lib
 DIRS-y += examples app include
 DIRS-$(CONFIG_TESTS) += test
 
-.PHONY: all clean $(DIRS-y) config.h CONFIG.local mk/cc.mk cc_version cxx_version
+.PHONY: all clean $(DIRS-y) include/spdk/config.h mk/config.mk mk/cc.mk \
+	cc_version cxx_version .libs_only_other .ldflags ldflags
 
 ifeq ($(SPDK_ROOT_DIR)/lib/env_dpdk,$(CONFIG_ENV))
 ifeq ($(CURDIR)/dpdk/build,$(CONFIG_DPDK_DIR))
@@ -52,33 +53,42 @@ endif
 endif
 endif
 
+ifeq ($(CONFIG_SHARED),y)
+LIB = shared_lib
+else
+LIB = lib
+endif
+
 all: $(DIRS-y)
 clean: $(DIRS-y)
 	$(Q)rm -f mk/cc.mk
-	$(Q)rm -f config.h
+	$(Q)rm -f include/spdk/config.h
 
 install: all
 	$(Q)echo "Installed to $(DESTDIR)$(CONFIG_PREFIX)"
 
 shared_lib: lib
 lib: $(DPDKBUILD)
-app: lib
-test: lib
-examples: lib
+app: $(LIB)
+test: $(LIB)
+examples: $(LIB)
 pkgdep:
 	sh ./scripts/pkgdep.sh
 
-$(DIRS-y): mk/cc.mk config.h
+$(DIRS-y): mk/cc.mk include/spdk/config.h
 
 mk/cc.mk:
 	$(Q)scripts/detect_cc.sh --cc=$(CC) --cxx=$(CXX) --lto=$(CONFIG_LTO) > $@.tmp; \
 	cmp -s $@.tmp $@ || mv $@.tmp $@ ; \
 	rm -f $@.tmp
 
-config.h: CONFIG CONFIG.local scripts/genconfig.py
+include/spdk/config.h: mk/config.mk scripts/genconfig.py
 	$(Q)PYCMD=$$(cat PYTHON_COMMAND 2>/dev/null) ; \
 	test -z "$$PYCMD" && PYCMD=python ; \
-	$$PYCMD scripts/genconfig.py $(MAKEFLAGS) > $@.tmp; \
+	echo "#ifndef SPDK_CONFIG_H" > $@.tmp; \
+	echo "#define SPDK_CONFIG_H" >> $@.tmp; \
+	$$PYCMD scripts/genconfig.py $(MAKEFLAGS) >> $@.tmp; \
+	echo "#endif /* SPDK_CONFIG_H */" >> $@.tmp; \
 	cmp -s $@.tmp $@ || mv $@.tmp $@ ; \
 	rm -f $@.tmp
 
@@ -88,4 +98,16 @@ cc_version: mk/cc.mk
 cxx_version: mk/cc.mk
 	$(Q)echo "SPDK using CXX=$(CXX)"; $(CXX) -v
 
+.libs_only_other:
+	$(Q)echo -n '$(SYS_LIBS) '
+	$(Q)if [ "$(CONFIG_SHARED)" = "y" ]; then \
+		echo -n '-lspdk '; \
+	fi
+
+.ldflags:
+	$(Q)echo -n '$(LDFLAGS) '
+
+ldflags: .ldflags .libs_only_other
+	$(Q)echo ''
+
 include $(SPDK_ROOT_DIR)/mk/spdk.subdirs.mk

README.md

@@ -103,20 +103,20 @@ Details on the Vagrant setup can be found in the
 ## Advanced Build Options
 
 Optional components and other build-time configuration are controlled by
-settings in two Makefile fragments in the root of the repository. `CONFIG`
-contains the base settings. Running the `configure` script generates a new
-file, `CONFIG.local`, that contains overrides to the base `CONFIG` file. For
-advanced configuration, there are a number of additional options to `configure`
-that may be used, or `CONFIG.local` can simply be created and edited by hand. A
-description of all possible options is located in `CONFIG`.
+settings in the Makefile configuration file in the root of the repository. `CONFIG`
+contains the base settings for the `configure` script. This script generates a new
+file, `mk/config.mk`, that contains final build settings. For advanced configuration,
+there are a number of additional options to `configure` that may be used, or
+`mk/config.mk` can simply be created and edited by hand. A description of all
+possible options is located in `CONFIG`.
 
 Boolean (on/off) options are configured with a 'y' (yes) or 'n' (no). For
 example, this line of `CONFIG` controls whether the optional RDMA (libibverbs)
 support is enabled:
 
 	CONFIG_RDMA?=n
 
-To enable RDMA, this line may be added to `CONFIG.local` with a 'y' instead of
+To enable RDMA, this line may be added to `mk/config.mk` with a 'y' instead of
 'n'. For the majority of options this can be done using the `configure` script.
 For example:
 
@@ -132,8 +132,10 @@ make CONFIG_RDMA=y
 ~~~
 
 Users may wish to use a version of DPDK different from the submodule included
-in the SPDK repository.  To specify an alternate DPDK installation, run
-configure with the --with-dpdk option.  For example:
+in the SPDK repository.  Note, this includes the ability to build not only
+from DPDK sources, but also just with the includes and libraries
+installed via the dpdk and dpdk-devel packages.  To specify an alternate DPDK
+installation, run configure with the --with-dpdk option.  For example:
 
 Linux:
 
@@ -150,10 +152,9 @@ gmake
 ~~~
 
 The options specified on the `make` command line take precedence over the
-default values in `CONFIG` and `CONFIG.local`. This can be useful if you, for
-example, generate a `CONFIG.local` using the `configure` script and then have
-one or two options (i.e. debug builds) that you wish to turn on and off
-frequently.
+values in `mk/config.mk`. This can be useful if you, for example, generate
+a `mk/config.mk` using the `configure` script and then have one or two
+options (i.e. debug builds) that you wish to turn on and off frequently.
 
 <a id="huge"></a>
 ## Hugepages and Device Binding

app/spdk_tgt/spdk_tgt.c

@@ -33,6 +33,7 @@
 
 #include "spdk/stdinc.h"
 
+#include "spdk/config.h"
 #include "spdk/env.h"
 #include "spdk/event.h"
 #include "spdk/vhost.h"