Skip to content

pfioh command: pushPath (keystore) using zip

Jump to bottom
Dan McPherson edited this page Aug 20, 2018 · 5 revisions

pfioh command: pushPath (keystore) (zip)

Abstract

This page describes the pushPath command to pfioh. It is used to push a directory tree to a remote server. In this example, the tree is zipped up at the source, and then unzipped to a location on the remote server handled internally by pfioh and tagged to a keystore mechanism. For an example of using absolute addressing, see here.

Using a keystore mechanism means that the client does not need any specific knowledge about the filespace topology in the remote location. Essentially, the client sends a key (just a string of text) that pfioh associates with some location in its filespace. Subsequently, the client then refers to the remote storage location using this key.

Preconditions

  • This page assumes that pfioh is listening on: 172.17.0.2:5055.
  • Make sure that pfioh has been started (see here for more info):
pfioh --forever --httpResponse --storeBase /tmp

msg summary

First, the client needs to send a string of text that the service will associate with an internal storage location of its own choosing:

{
    "action":   "internalctl",
    "meta": {
           "var":          "key2address",
           "compute":      "jid-1234"
       }
}

Here, the string jid-1234 is the key text.

Once set, the client uses {"key": "jid-1234"} to refer to this location in the remote filesystem space:

{  "action": "pushPath",
    "meta": {
        "remote": {
            "key":         "jid-1234"
        },
        "local": {
            "path":         "/usr/sbin"
        },
        "transport": {
            "mechanism":    "compress",
            "compress": {
                "archive":  "zip",
                "unpack":   true,
                "cleanup":  true
            }
        }
    }
}

pushPath

pfurl calling syntax

Assuming satisfied preconditions, let's push a directory tree from the machine/host running purl to the remote machine:

pfurl --verb POST --raw --http 172.17.0.2:5055/api/v1/cmd --msg \
'{  "action": "pushPath",
    "meta": {
        "remote": {
            "key":         "jid-1234"
        },
        "local": {
            "path":         "/usr/sbin"
        },
        "transport": {
            "mechanism":    "compress",
            "compress": {
                "archive":  "zip",
                "unpack":   true,
                "cleanup":  true
            }
        }
    }
}' --quiet --jsonpprintindent 4

Using the dockerized version of pfurl

To use the dockerized version of pfurl, and assuming a pfioh on the given IP:

Using the source repo

If you have cloned the source repo, you can cd to the root directory and execute the docker helper scripts in the docker-bin directory.

docker-bin/pfurl --verb POST --raw --http 172.17.0.2:5055/api/v1/cmd --msg \
'{  "action": "pushPath",
    "meta": {
        "remote": {
            "key":         "jid-1234"
        },
        "local": {
            "path":         "/usr/sbin"
        },
        "transport": {
            "mechanism":    "compress",
            "compress": {
                "archive":  "zip",
                "unpack":   true,
                "cleanup":  true
            }
        }
    }
}' --quiet --jsonpprintindent 4
Calling docker directly

The helper script just creates a docker run command line string. You can run this string directly without using the helper script and directly calling the docker app:

docker run --name pfurl -v /home:/Users --rm -ti fnndsc/pfurl --verb POST --raw --http 172.17.0.2:5055/api/v1/cmd --msg \
'{  "action": "pushPath",
    "meta": {
        "remote": {
            "key":         "jid-1234"
        },
        "local": {
            "path":         "/usr/sbin"
        },
        "transport": {
            "mechanism":    "compress",
            "compress": {
                "archive":  "zip",
                "unpack":   true,
                "cleanup":  true
            }
        }
    }
}' --quiet --jsonpprintindent 4

This will push the local /usr/sbin directory out to the remote machine and unpack to the remote machine's /tmp directory. Note that the zip compression will not preserve file mode bits (executable files need to have their execute bit explicitly set again).

return payload

The above call returns the JSON string:

{
    "stdout": {
        "remoteCheck": {
            "size": "48",
            "msg": "Check on remote path successful.",
            "response": {
                "isdir": true,
                "isfile": false,
                "status": true
            },
            "timestamp": "2017-03-13 13:36:09.146058",
            "status": true
        },
        "msg": "push OK.",
        "localCheck": {
            "check": {
                "isfile": false,
                "isdir": true,
                "status": true
            },
            "status": true,
            "timestamp": "2017-03-13 13:36:09.143503",
            "msg": "Check on local path successful."
        },
        "compress": {
            "msg": "push OK.",
            "local": {
                "zip": {
                    "fileProcessed": "a20eebf8-9788-456b-8ea9-d33b842ca60e.zip",
                    "filesize": "1,028,644",
                    "msg": "zip operation successful",
                    "path": "/usr/sbin",
                    "zipmode": "w",
                    "status": true
                }
            },
            "status": true,
            "remoteServer": {
                "stdout": {
                    "User-agent": "PycURL/7.43.0 libcurl/7.47.0 GnuTLS/3.4.10 zlib/1.2.8 libidn/1.32 librtmp/2.3",
                    "decode": {},
                    "unzip": {
                        "fileProcessed": "/tmp/sbin.zip",
                        "filesize": "1,028,644",
                        "msg": "unzip operation successful",
                        "path": "/tmp",
                        "zipmode": "r",
                        "status": true
                    },
                    "msg": "unzip operation successful",
                    "status": true
                },
                "msg": "push OK.",
                "status": true
            }
        },
        "status": true
    }
}

oneShot

If --oneShot is passed to pfurl, then a server control message is transmitted after the file IO event that effectively shuts down the remote server:

pfurl --verb POST --http 172.17.0.2:5055/api/v1/cmd/ --oneShot --msg \

--30--

Clone this wiki locally