Adds 'faas-cli template pull' command #201

ericstoekl · 2017-11-03T05:16:28Z

Pull language templates from any repo that has a 'template/' directory
in the root with 'faas-cli template pull '.

Also allows 'faas-cli new --list' to list available languages

Signed-off-by: Eric Stoekl ems5311@gmail.com
Signed-off-by: Minh-Quan TRAN account@itscaro.me

Description

This PR is to add the ability to use faas-cli template pull <github repo>. For example:

$ faas-cli template pull https://github.com/itscaro/openfaas-template-php
Fetch templates from repository: https://github.com/itscaro/openfaas-template-php
2017/11/02 21:06:55 HTTP GET https://github.com/itscaro/openfaas-template-php/archive/master.zip
2017/11/02 21:06:56 Writing 8Kb to master.zip

2017/11/02 21:06:56 Attempting to expand templates from master.zip
2017/11/02 21:06:56 Fetched 2 template(s) : [php php5] from https://github.com/itscaro/openfaas-template-php
2017/11/02 21:06:56 Cleaning up zip file...
$ ls template/php5
composer.json  Dockerfile  function  index.php  php-extension.sh  template.yml
$ faas-cli new php_t1 --lang php
Folder: php_t1 created.
  ___                   _____           ____  
 / _ \ _ __   ___ _ __ |  ___|_ _  __ _/ ___| 
| | | | '_ \ / _ \ '_ \| |_ / _` |/ _` \___ \ 
| |_| | |_) |  __/ | | |  _| (_| | (_| |___) |
 \___/| .__/ \___|_| |_|_|  \__,_|\__,_|____/ 
      |_|                                     


Function created in folder: php_t1
Stack file written: php_t1.yml
$ cat php_t1.yml 
provider:
  name: faas
  gateway: http://localhost:8080

functions:
  php_t1:
    lang: php
    handler: ./php_t1
    image: php_t1
$ ls php_t1
composer.json  Handler.php  php-extension.sh

Check this asciicinema to see it in action: https://asciinema.org/a/ieyznKidFlvimpeiS5BFMG3ep

Here's a breakdown of what was changed:

Main Files

builder/build.go - Uses IsValidTemplate() to tell whether or not it can build the given language
commands/build.go - Update to pass overwrite parameter to fetchTemplates()
commands/deploy.go - Get Fprocess from the template/<lang>/template.yml file instead of hard-coded
commands/fetch_templates.go - fetchTemplates() method pulls from openfaas/faas-cli github repo by default, custom repo URL can be passed in. Implements --overwrite functionality
commands/new_function.go - Lists available languages from the template dir with --list option
commands/template_pull.go - Make sure the passed-in valid, then passes it to fetchTemplates()
`proxy/deploy.go - Remove hard-coded validation of available language templates
stack/language_template.go - Adds ParseYAMLForLanguageTemplate(), which will parse the template/<lang>/template.yml file, which contains info like Fprocess
stack/schema.go - Adds LanguageTemplate type
guide/TEMPLATE.md - Adds a guide for usage of faas-cli template pull
- template/csharp/template.yml
- template/go-armhf/template.yml
- template/go/template.yml
- template/node-arm64/template.yml
- template/node-armhf/template.yml
- template/node/template.yml
- template/python-armhf/template.yml
- template/python/template.yml
- template/python3/template.yml
- template/ruby/template.yml

The rest of the files are for testing, and are discussed below.

Motivation and Context

I have raised an issue to propose this change (required)
First discussed in issue Proposal: third-party templates via GitHub #85, originally work was done in Create faas-cli add-template --url <URL> command #87.

How Has This Been Tested?

In addition to the following files being added for testing, @burtonr reported success when manually testing: #87 (comment)

commands/fetch_templates_test.go - Simplify tests, add test for fetchTemplates()
commands/new_function_test.go - Test --list option, and add more tests for new
commands/template_pull_test.go - Test template pull functionality
commands/testdata/master_test.zip - Minimal template zip for testing, served by the mock test server
commands/testdata/new_function/template/csharp/function/.gitignore - Add directories
commands/testdata/new_function/template/go-armhf/.gitignore
commands/testdata/new_function/template/go/function/.gitignore
commands/testdata/new_function/template/node-arm64/function/.gitignore
commands/testdata/new_function/template/node-armhf/.gitignore
commands/testdata/new_function/template/node/function/.gitignore
commands/testdata/new_function/template/python-armhf/.gitignore
commands/testdata/new_function/template/python/function/.gitignore
commands/testdata/new_function/template/python3/function/.gitignore
commands/testdata/new_function/template/ruby/function/.gitignore
stack/language_template_test.go - Test ParseYAMLForLanguageTemplate() against different types of YAML file

Types of changes

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

My code follows the code style of this project.
My change requires a change to the documentation.
I have updated the documentation accordingly.
I've read the CONTRIBUTION guide
I have signed-off my commits with git commit -s
I have added tests to cover my changes.
All new and existing tests passed.

itscaro · 2017-11-05T20:24:27Z

@ericstoekl new --list instead of new --lang in your PR description

faas-cli template list could be an alias of faas-cli new --list

alexellis · 2017-11-06T17:35:16Z

$ mkdir -p ./template/bash

$ ./faas-cli new --lang bash abc
Folder: abc created.
2017/11/06 17:33:51 open ./template/bash/function/: no such file or directory

@ericstoekl perhaps we should not make languages available where there is no template.yml file in the directory.

alexellis · 2017-11-06T17:42:01Z

I've tested the PR and really liked the experience, this will be a great addition to our CLI.

For follow-up work post-merge, let's talk about:

How do we shrink-wrap templates?
How do we pin a specific commit/SHA or version of a template?
What happens at CI time? Do we do faas-cli template pull, or store this data alongside stack.yml or inside $HOME/openfaas.yml?

alexellis · 2017-11-06T17:54:15Z

commands/fetch_templates.go

+		if languageSplit := strings.Split(relativePath, "/"); len(languageSplit) > 2 {
+			language = languageSplit[1]
+
+			if len(languageSplit) == 3 && relativePath[len(relativePath)-1:] == "/" {


Pre-squash I asked for these magic numbers to be documented / split out either into a new function or named consts. If I had to work on this I wouldn't know what to do.

alexellis · 2017-11-06T17:55:16Z

commands/fetch_templates.go

@@ -137,3 +186,29 @@ func createPath(relativePath string, perms os.FileMode) error {
 	err := os.MkdirAll(dir, perms)
 	return err
 }
+
+// Tells whether the language can be expanded from the zip or not


Comment should start "canWriteLanguage tells whether"

alexellis · 2017-11-06T17:55:48Z

commands/fetch_templates.go

+func canWriteLanguage(availableLanguages *map[string]bool, language string, overwrite bool) bool {
+	canWrite := false
+	if len(language) > 0 {
+		if _, ok := (*availableLanguages)[language]; ok {


Please add if availableLanguages != nil and change ok to exists or found

alexellis · 2017-11-06T17:56:52Z

commands/fetch_templates_test.go

+}
+
+func Test_fetchTemplates(t *testing.T) {
+	defer tearDown_fetch_templates(t)


This should be run at the start and at the end - because you may have had an inconsistent file-system if the files were written out and the code panicked and crashed mid-way.

alexellis · 2017-11-06T17:57:54Z

commands/new_function.go

@@ -44,15 +44,23 @@ language or type in --list for a list of languages available.`,
 }

 func runNewFunction(cmd *cobra.Command, args []string) {


We're changing the commands to return (error) so we can handle the exit codes during CI / scripting. If that's beyond scope of the PR please raise an issue instead to log the work.

I will take you up on the second option: #210

alexellis · 2017-11-06T17:58:23Z

commands/new_function.go

-		found = true
+func printAvailableTemplates(availableTemplates []string) string {
+	var result string
+	for _, template := range availableTemplates {


Can we sort "printAvailableTemplates" deterministically?

alexellis · 2017-11-06T17:59:50Z

commands/template_pull.go

@@ -0,0 +1,80 @@
+// Copyright (c) Alex Ellis, Eric Stoekl 2017. All rights reserved.


License should be:

// Copyright (c) Alex Ellis 2017. All rights reserved.

or

// Copyright (c) OpenFaaS Project 2017. All rights reserved.

Please see: https://github.com/openfaas/faas/blob/master/CONTRIBUTING.md#license

alexellis · 2017-11-06T18:00:00Z

commands/template_pull_test.go

@@ -0,0 +1,144 @@
+// Copyright (c) Alex Ellis, Eric Stoekl 2017. All rights reserved.


https://github.com/openfaas/faas/blob/master/CONTRIBUTING.md#license

alexellis · 2017-11-06T18:00:34Z

guide/TEMPLATE.md

@@ -0,0 +1,56 @@
+# Using template from external repository


I wonder if this should be in the guides in the faas repo?

alexellis · 2017-11-06T18:01:02Z

stack/language_template.go

+	return ParseYAMLDataForLanguageTemplate(fileData)
+}
+
+// iParseYAMLDataForLanguageTemplate parses YAML data into language template


Typo of "i" at start of comment.

alexellis · 2017-11-06T18:01:43Z

stack/schema.go

@@ -47,3 +47,8 @@ type Services struct {
 	Functions map[string]Function `yaml:"functions,omitempty"`
 	Provider  Provider            `yaml:"provider,omitempty"`
 }
+
+type LanguageTemplate struct {


Comment:

// LanguageTemplate read from template.yml within root of a language template folder

alexellis · 2017-11-06T18:03:14Z

commands/deploy.go

@@ -160,6 +160,28 @@ func runDeploy(cmd *cobra.Command, args []string) {
 				log.Fatalln(envErr)
 			}

+			// Get FProcess to use from the ./template/template.yml, if a template is being used
+			if function.Language != "" && function.Language != "Dockerfile" && function.Language != "dockerfile" {


Extract to named method.

Change to this or similar:

len(function.Language) > 0 && strings.Lower(function.Language) != "dockerfile"

alexellis

Very close to finished. Please address comments and licensing then add a new commit, do not squash.

1) Moves code that merges zip file contents into template/ dir into expandTemplatesFromZip() function. Removes hard-coded constant. 2) Sort list of languages found with 'faas-cli new --list' before printing 3) Fixes for typos, more descriptive comments, and remove incorrect license attribution Signed-off-by: Eric Stoekl <ems5311@gmail.com>

ericstoekl · 2017-11-07T06:41:50Z

Hi @alexellis , thanks for your comments. I've attempted to address all of them except the comment about the guide needing to go into the faas repo. Maybe we can have the guide here for now, and move it in a future PR?

I've created a new function in fetch_templates.go called expandTemplatesFromZip(). The goal is to encapsulate the unwieldy blob of code in (the for loop formerly in fetchTemplates()) that was there to unpack the template zip file, and merge it with the existing template/ dir, with the option of overwriting.

I've added sort functionality to the faas-cli new --list command, to sort the list of available language templates before displaying them. This is done by implementing the Len(), Swap(), and Less() functions, as is described by the docs: https://golang.org/pkg/sort/

ericstoekl · 2017-11-07T07:29:00Z

@alexellis , I have pushed another commit to address this comment: #201 (comment)

This is to disallow creation of a function when template.yml is not found in the template directory.

If we want to split this off into a future PR, I'm okay with that as well.

alexellis · 2017-11-09T14:57:50Z

stack/language_template_test.go

+		},
+	}
+
+	for k, i := range dataProvider {


I don't know what dataProvider means.

alexellis · 2017-11-09T14:59:30Z

template/node-armhf/template.yml

@@ -0,0 +1,2 @@
+language: node-armhf
+fprocess: node index.js


Does the node template contain a full template or does it still only have the Dockerfile patch? cc @rgee0

alexellis · 2017-11-09T15:01:46Z

commands/fetch_templates.go

 				}
 			}
+		} else {
+			// template/
+			continue


I don't like the half-dozen continue statements, how can we re-write this to be clearer?

alexellis · 2017-11-09T15:02:10Z

commands/fetch_templates.go

+			continue
+		}
+
+		if pathSplit := strings.Split(relativePath, "/"); len(pathSplit) > 2 {


Perhaps this all needs to be a function of its own. Extract a new function

alexellis · 2017-11-09T15:08:50Z

commands/fetch_templates.go

+}
+
+// Takes a language input (e.g. "node"), tells whether or not it is OK to download
+func checkLanguage(language string, overwrite bool) bool {


templateFolderExists / targetExists?

alexellis · 2017-11-09T15:09:29Z

commands/fetch_templates.go

+			isDirectory = relativePath[len(relativePath)-1:] == "/"
+
+			// Check if this is the root directory for a language (at ./template/lang)
+			if len(pathSplit) == rootLanguageDirSplitCount && isDirectory {


What are we doing here exactly?

We unzip and then don't overwrite if a template/node folder exists?

Yep, don't overwrite if the template/<lang> folder already exists. Unless --overwrite flag is active.

Pull language templates from any repo that has a 'template/' directory in the root with 'faas-cli template pull <github repo>'. Also allows 'faas-cli new --lang' to list available languages Signed-off-by: Eric Stoekl <ems5311@gmail.com> Signed-off-by: Minh-Quan TRAN <account@itscaro.me>

1) Moves code that merges zip file contents into template/ dir into expandTemplatesFromZip() function. Removes hard-coded constant. 2) Sort list of languages found with 'faas-cli new --list' before printing 3) Fixes for typos, more descriptive comments, and remove incorrect license attribution Signed-off-by: Eric Stoekl <ems5311@gmail.com>

is not present in language template directory Needs to add a 'template.yml' to each fake language template dir in commands/testdata/new_function/template/<lang> in order to have current tests pass. Deletes some unnecessary .gitignore files as a result. Signed-off-by: Eric Stoekl <ems5311@gmail.com>

1) Improve readability by adding canExpandTemplateData() function, which returns an enum telling what to do with the data found in the zip archive. 2) Add function data to ARMHF directories in template/ dir 3) More descriptive naming for test array in language_template_test.go and checkLanguage() function changed to templateFolderExists() Signed-off-by: Eric Stoekl <ems5311@gmail.com>

ericstoekl · 2017-11-09T16:45:50Z

@alexellis , just made those requested updates. I've refactored the part in the loop with continue usage to use a separate function which returns an enum that is then used in a switch statement to tell whether to continue with the loop or not. I've also changed the naming of several identifiers as requested, and included the function data into the go-armhf, node-armhf, and python-armhf directories. Let me know what you think. Thanks!

alexellis · 2017-11-11T16:46:03Z

Needs a rebase.

1) Moves code that merges zip file contents into template/ dir into expandTemplatesFromZip() function. Removes hard-coded constant. 2) Sort list of languages found with 'faas-cli new --list' before printing 3) Fixes for typos, more descriptive comments, and remove incorrect license attribution Signed-off-by: Eric Stoekl <ems5311@gmail.com>

alexellis · 2017-11-11T17:20:04Z

Rebasing and carrying out some review / clean-up over in #220.

1) Moves code that merges zip file contents into template/ dir into expandTemplatesFromZip() function. Removes hard-coded constant. 2) Sort list of languages found with 'faas-cli new --list' before printing 3) Fixes for typos, more descriptive comments, and remove incorrect license attribution Signed-off-by: Eric Stoekl <ems5311@gmail.com>

ericstoekl mentioned this pull request Nov 3, 2017

Create faas-cli add-template --url <URL> command #87

Closed

11 tasks

alexellis force-pushed the master branch from da0b406 to 5cd3ea2 Compare November 3, 2017 13:05

itscaro force-pushed the addTemplateCmd_mc1 branch 3 times, most recently from 7f00b90 to b83a368 Compare November 5, 2017 19:29

itscaro force-pushed the addTemplateCmd_mc1 branch from b83a368 to 1e80b83 Compare November 5, 2017 20:36

alexellis mentioned this pull request Nov 6, 2017

Proposal: Add support for PHP #55

Closed

alexellis requested a review from johnmccabe November 6, 2017 17:52

alexellis reviewed Nov 6, 2017

View reviewed changes

alexellis requested changes Nov 6, 2017

View reviewed changes

alexellis removed the request for review from johnmccabe November 6, 2017 18:07

alexellis assigned ericstoekl Nov 6, 2017

ericstoekl mentioned this pull request Nov 7, 2017

Return non-zero error code when faas-cli new command fails #210

Closed

alexellis reviewed Nov 9, 2017

View reviewed changes

ericstoekl added 4 commits November 9, 2017 08:38

ericstoekl force-pushed the addTemplateCmd_mc1 branch from 03c1bf4 to 927400a Compare November 9, 2017 16:41

alexellis mentioned this pull request Nov 11, 2017

Adds 'faas-cli template pull' command (rebase/review PR) #220

Merged

alexellis closed this Nov 11, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adds 'faas-cli template pull' command #201

Adds 'faas-cli template pull' command #201

ericstoekl commented Nov 3, 2017 •

edited

Loading

itscaro commented Nov 5, 2017 •

edited

Loading

alexellis commented Nov 6, 2017

alexellis commented Nov 6, 2017 •

edited

Loading

alexellis Nov 6, 2017

alexellis Nov 6, 2017

alexellis Nov 6, 2017

alexellis Nov 6, 2017

alexellis Nov 6, 2017

ericstoekl Nov 7, 2017

alexellis Nov 6, 2017

alexellis Nov 6, 2017

alexellis Nov 6, 2017

alexellis Nov 6, 2017

alexellis Nov 6, 2017

alexellis Nov 6, 2017

alexellis Nov 6, 2017

alexellis left a comment

ericstoekl commented Nov 7, 2017 •

edited

Loading

ericstoekl commented Nov 7, 2017 •

edited

Loading

alexellis Nov 9, 2017

alexellis Nov 9, 2017

alexellis Nov 9, 2017

alexellis Nov 9, 2017

alexellis Nov 9, 2017

alexellis Nov 9, 2017

ericstoekl Nov 9, 2017

ericstoekl commented Nov 9, 2017

alexellis commented Nov 11, 2017

alexellis commented Nov 11, 2017

		@@ -44,15 +44,23 @@ language or type in --list for a list of languages available.`,
		}

		func runNewFunction(cmd *cobra.Command, args []string) {

		@@ -0,0 +1,80 @@
		// Copyright (c) Alex Ellis, Eric Stoekl 2017. All rights reserved.

		@@ -0,0 +1,144 @@
		// Copyright (c) Alex Ellis, Eric Stoekl 2017. All rights reserved.

Adds 'faas-cli template pull' command #201

Adds 'faas-cli template pull' command #201

Conversation

ericstoekl commented Nov 3, 2017 • edited Loading

Description

Main Files

Motivation and Context

How Has This Been Tested?

Types of changes

Checklist:

itscaro commented Nov 5, 2017 • edited Loading

alexellis commented Nov 6, 2017

alexellis commented Nov 6, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alexellis left a comment

Choose a reason for hiding this comment

ericstoekl commented Nov 7, 2017 • edited Loading

ericstoekl commented Nov 7, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ericstoekl commented Nov 9, 2017

alexellis commented Nov 11, 2017

alexellis commented Nov 11, 2017

ericstoekl commented Nov 3, 2017 •

edited

Loading

itscaro commented Nov 5, 2017 •

edited

Loading

alexellis commented Nov 6, 2017 •

edited

Loading

ericstoekl commented Nov 7, 2017 •

edited

Loading

ericstoekl commented Nov 7, 2017 •

edited

Loading