curl: functions module for shellfire

This module provides a simple framework for using curl with a shellfire application. It wraps up the common API needed for using REST over HTTP(S). In particular, it provides a secure way of passing passwords, OAuth credentials and other sensitive headers and URLs to curl without exposing them to ps or via environment variables. It also integrates with .netrc and .curlrc so site-specific proxy settings and credentials can be kept separate from source control (and from users, where necessary) used without constantly reinventing the wheel.

An example user is the github api module. It is useful when combined with the urlencode module (eg for URL Templates) and the jsonwriter and jsonreader modules. The latter provides a fully JSON compliant event (ie SAX like) module to parse any arbitary JSON graph in shell script.

Headers are set to sensible defaults wherever possible, but can be overridden as necessary.

Settings are automatically adjusted when older versions of curl are detected (eg on CentOS 6.5).

Compatibility

• Tag release_2015.0117.1750-1 is compatible with shellfire release release_2015.0117.1750-1.

Overview

The main function to use is curl_http(). For example, to google's home page to standard out (-) with no user:-

local curl_httpVersion
local curl_httpStatusCode
local curl_httpDescription
curl_http none '' GET 'https://google.com/' '-'

my_callback()
{
	echo "Header field name: $fieldName"
	echo "Header field value: $fieldValue"
}

local headerFieldCount=0
my_other_callback()
{
	headerFieldCount=$((headerFieldCount+1))
}

curl_iterateHeaders my_callback my_other_callback

echo "There were $headerFieldCount headers"

Importing

To import this module, add a git submodule to your repository. From the root of your git repository in the terminal, type:-

mkdir -p lib/shellfire
cd lib/shellfire
git submodule add "https://github.com/shellfire-dev/curl.git"
cd -
git submodule init --update

You may need to change the url https://github.com/shellfire-dev/curl.git above if using a fork.

You will also need to add paths - include the module paths.d.

You will also need to import the version module.

Namespace curl

To use in code

If calling from another shellfire module, add to your shell code the line

core_usesIn curl

in the global scope (ie outside of any functions). A good convention is to put it above any function that depends on functions in this module. If using it directly in a program, put this line inside the _program() function:-

_program()
{
	core_usesIn curl
	…
}

Functions

---

curl_http

Parameter	Value	Optional
authentication	Authentication method to use	No
credentials	Use the function called configure_validate_${configurationValidationFunction} to validate configuration. Passed securely to curl	No
verb	HTTP verb to use.	No
url	URL to use, including query string. If needed, can be constructed from a url template using the urlencode module.	No
outputFilePath	File to write received body to. Use - for standard out (not advised, as you'd need to pipeline this call and so can't handle errors) or /dev/null if it's never wanted (again, not advised, as output might be an error moessage).	No
…	Key-Value pairs for headers, eg Content-Type text/plain;charset=utf-8 Content-Language en	Yes

The functions sets the following variables on exit; normally you should set these to local variables before calling curl_http().

Parameter	Value
curl_parsedHeadersFilePath	Use this with the function curl_iterateHeaders()
curl_httpVersion	The HTTP version string (eg HTTP/1.1) from the final set of headers received
curl_httpStatus	The HTTP status code (eg 200) from the final set of headers received
curl_httpReasonPhrase	The HTTP description (eg OK) from the final set of headers received

The values of authentication may be:-

Value	Description
none	No authentication. credentials must be empty ('')
basic	HTTP Basic authentication. credentials can be user (some_user) or user and password (some_user:some_password)
digest	HTTP Digest authentication credentials can be user (some_user) or user and password (some_user:some_password)
ntlm	Windows NTLM authentication, if curl supports it
kerberos	Windows Kerberos authentication, if curl supports it

The values of verb may be:-

Value	Description
HEAD
GET
DELETE
POST	Set curl_uploadFile to the file for the body to send. Contents must already be encoded. Given that web forms are rare for REST APIs, you might want to use the jsonwriter module to create JSON.
PATCH	Set curl_uploadFile to the file for the body to send. Contents must already be encoded. Given that web forms are rare for REST APIs, you might want to use the jsonwriter module to create JSON.
PUT	Set curl_uploadFile to the file for the body to send. Contents must already be encoded. Given that web forms are rare for REST APIs, you might want to use the jsonwriter module to create JSON.

In addition, various out-of-bound parameters may be used to modify behaviour be set be specifiying variables local before calling `curl_http:_

Out-of-band Parameter	Value
curl_uploadFile	Set this to a file to POST, PATCH or PUT. No form encoding is done (ie we use --data-binary to curl)
curl_range	Set this to values supported by --range
curl_continueAt	Set this to values supported by --continue-at

Also, there are several global settings that can be overridden per-call (by using local variables, as above) or by configuration (see below). These are initially set by the internal _curl_initialise() function, which is called on first-use by curl_http().

Global Setting	Value	Default
curl_userAgent	Value passed as User-Agent header	shellfire
curl_maximumRedirects	Maximum redirects	5
curl_retries	Maximum retries	10
curl_retryDelay	Delay in seconds between retries	0
curl_retryMaximumDelay	Maximum delay in seconds between retries	0
curl_failHard	Boolean. If true, then a non-zero curl exit code causes the application to exit	false
curl_makeTlsInsecure*	Boolean	false for curl versions 7.19.7 and newer. true for older versions.

Where necessary, older versions of curl are detected and settings that won't work are not used.

* This effectively disables SSL/TLS certificate validation to workaround curl bugs with subjectAltName validation.

Using .netrc

The function will search for a .netrc file (useful for specifying passwords outside of the program) in the following locations*:-

• /etc/shellfire/netrc†
• /etc/${_program_name}/netrc†
• ${CURL_HOME}/.netrc
• ${HOME}/.netrc
• ${HOME}/shellfire.netrc
• ${HOME}/${_program_name}.netrc

* Not done on CentOS 6.5 and CentOS 5 because they use a very old version of curl. Instead only $HOME/.netrc is used if present. † Actually, whatever path has been set inside the program for /etc - such as /usr/local/etc.

Using .curlrc

.curlrc lets you override most of the configuration choices made by curl_http() (but not those that affect URL use, file upload, etc). It is an ideal place to put site proxy credentials. The function will search for a .curlrc file in the the following locations:-

• /etc/shellfire/curlrc*
• /etc/${_program_name}/curlrc*
• ${CURL_HOME}/.curlrc
• ${HOME}/.curlrc
• ${HOME}/shellfire.curlrc
• ${HOME}/${_program_name}.curlrc

* Actually, whatever path has been set inside the program for /etc - such as /usr/local/etc.

Tips

When expecting a reply, or even if there's the possibility of a reply, you'll want to keep outputFilePath. A good idea is to use a temporary file. Likewise, when using POST, PUT or PATCH repeatedly, it might be a good idea to create a temporary file and reuse it. For example:-

local TMP_FILE
core_temporaryFiles_newFileToRemoveOnExit
curl_uploadFile="$TMP_FILE"
…
jsonwriter_object \
	string name "$tagName" \
	string label "$commitish" \
	>"$curl_uploadFile"

If you do this, remember to truncate the file afterwards, eg:-

printf '' >"$curl_uploadFile"

---

curl_iterateHeaders

Parameter	Value	Optional
…	Zero or more callback functions (function names)	Yes

Iterates over header fields after a call to curl_http(), calling each callback for each header field. Each callback will be able to look at the value of fieldName and fieldValue. Folded header fields are supported. fieldValue will have had leading and trailing whitespace removed; interior whitespace, however, isn't normalised. Useful for extracting headers such as Location or Link.