Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor!(loader): download git repo archive instead of clone & pull #47

Merged
merged 49 commits into from
Mar 7, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
49 commits
Select commit Hold shift + click to select a range
bce6fc1
refactor(loader): download git repo archive
tassja Oct 20, 2021
d2f3ff4
style: ameba flags
tassja Oct 20, 2021
0219dbb
Merge branch 'master' into refactor-git
tassja Oct 20, 2021
7e02f1f
refactor(loader): move Github methods to class
tassja Oct 29, 2021
9b1f112
fix(Github): raise exception if repo not found
tassja Oct 31, 2021
1f01189
refactor: Remote hiearchy
tassja Nov 4, 2021
c284842
refactor(github): improve Exception rescue
tassja Nov 4, 2021
ce645a5
refactor(loader): update `repeating_update` method
tassja Nov 4, 2021
919c125
refactor(github): perform methods in actioner
tassja Nov 10, 2021
caea59a
refactor: use `HashFile` for repo metadata
tassja Nov 12, 2021
09b537a
refactor(github): save path in ref struct
tassja Nov 12, 2021
2310d09
feat(gitlab): add support for GitLab repos
tassja Nov 14, 2021
a505fb3
refactor: use `Remote` effectively
tassja Nov 15, 2021
cadb64e
refactor(remote): init with URL
tassja Nov 16, 2021
bf23cc9
feat(githubremote): use Octokit client
tassja Nov 18, 2021
67b76b8
feat(gitlab): use gitlab client
tassja Nov 19, 2021
1e75da9
refactor(remote): add basic auth for clients
tassja Nov 19, 2021
17bb6ec
reafactor(remote): common functions to abstract
tassja Nov 30, 2021
e7db5f1
refactor(remote): lock metadata writes
tassja Dec 1, 2021
243bec8
refactor(loader): get remote from ref
tassja Dec 2, 2021
3778382
refactor: use hash store for metadata
tassja Dec 3, 2021
bb606de
refactor(laoder): use remote hash
tassja Dec 6, 2021
870c20b
refactor: add remote methods
tassja Dec 7, 2021
f9b80f3
feat(repos): releases endpoint
tassja Dec 7, 2021
1c85058
feat(github): download release asset
tassja Dec 9, 2021
6e7a15d
feat(github): download asset on flag
tassja Dec 14, 2021
f89610a
chore(shard): bump
tassja Dec 14, 2021
6cd5e86
chore(shard): bump
tassja Dec 14, 2021
52d2785
Merge branch 'master' into refactor-git
tassja Dec 14, 2021
1388f47
chore(shard): bump
tassja Dec 14, 2021
56424ab
feat(remotes): add API
tassja Dec 16, 2021
626dbfe
refactor(remote): remove repo init
tassja Dec 17, 2021
7e716ea
refactor(remote): don't write Directories as file
tassja Dec 20, 2021
7633831
feat(client): add remote endpoints
tassja Dec 20, 2021
3af96dc
refactor(client): encode URL
tassja Dec 20, 2021
ddc5c1a
refactor(github): don't rely on model for info
tassja Feb 3, 2022
11e6e04
Merge branch 'master' into refactor-git
tassja Feb 3, 2022
af371e9
chore(shards): bump
tassja Feb 3, 2022
f27fbd2
feat(github): drop github client
tassja Feb 8, 2022
8a8f9a7
refactor(remote): use git cli
tassja Feb 15, 2022
57163e9
fix(github): use new commit args
tassja Feb 15, 2022
483c7ac
refactor(github): use cli
tassja Feb 17, 2022
379ca1b
caspian: git refactor changes (#52)
caspiano Feb 18, 2022
e57d967
refactor(github): use api for releases
tassja Feb 28, 2022
fdd8aa7
Merge branch 'master' into refactor-git
caspiano Mar 1, 2022
4fe1598
fix(repositories): correct controller name
tassja Mar 4, 2022
5bb6de9
chore(shard): bump
tassja Mar 4, 2022
a30f3e1
Merge branch 'master' into refactor-git
tassja Mar 4, 2022
787951e
style(shard.lock): remove whitespace
tassja Mar 4, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion Dockerfile.test
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,9 @@ ARG PLACE_COMMIT="DEV"

# - Add trusted CAs for communicating with external services
# - Add watchexec for running tests on change
RUN apk add --no-cache \
RUN apk upgrade --no-cache apk \
&& \
apk add --no-cache \
bash \
ca-certificates \
&& \
Expand Down
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -53,3 +53,4 @@ See [`CONTRIBUTING.md`](./CONTRIBUTING.md).
## Contributors

- [Caspian Baska](https://github.com/caspiano) - creator and maintainer
- [Tassja Kriek](https://github.com/tassja) - contributor and maintainer
32 changes: 32 additions & 0 deletions shard.lock
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,10 @@ shards:
git: https://github.com/maiha/crc16.cr.git
version: 0.1.0

crest:
git: https://github.com/mamantoha/crest.git
version: 1.2.1

cron_parser:
git: https://github.com/kostya/cron_parser.git
version: 0.4.0
Expand All @@ -37,6 +41,10 @@ shards:
git: https://github.com/spider-gazelle/crunits.git
version: 1.1.0

crystar:
git: https://github.com/naqvis/crystar.git
version: 0.2.0

db:
git: https://github.com/crystal-lang/crystal-db.git
version: 0.11.0
Expand Down Expand Up @@ -65,18 +73,38 @@ shards:
git: https://github.com/crystal-community/future.cr.git
version: 1.0.0

gitlab:
git: https://github.com/icyleaf/gitlab.cr.git
version: 0.7.0

habitat:
git: https://github.com/luckyframework/habitat.git
version: 0.4.7

halite:
git: https://github.com/icyleaf/halite.git
version: 0.12.0

hash_file:
git: https://github.com/tassja/hash_file.git
version: 0.4

hound-dog:
git: https://github.com/place-labs/hound-dog.git
version: 2.9.0

http-client-digest_auth:
git: https://github.com/mamantoha/http-client-digest_auth.git
version: 0.6.0

http-params-serializable:
git: https://github.com/place-labs/http-params-serializable.git
version: 0.5.0

http_proxy:
git: https://github.com/mamantoha/http_proxy.git
version: 0.9.0

ipaddress:
git: https://github.com/sija/ipaddress.cr.git
version: 0.2.1
Expand Down Expand Up @@ -105,6 +133,10 @@ shards:
git: https://github.com/spider-gazelle/neuroplastic.git
version: 1.11.0

octokit:
git: https://github.com/place-labs/octokit.cr.git
version: 0.1.6

open_api:
git: https://github.com/elbywan/open_api.cr.git
version: 1.3.0
Expand Down
18 changes: 17 additions & 1 deletion shard.yml
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ crystal: ">= 1.1.1"
license: MIT
authors:
- Caspian Baska <caspian@place.tech>
- Tassja Kriek <tassja@place.tech>

targets:
frontends:
Expand All @@ -14,6 +15,22 @@ dependencies:
github: spider-gazelle/action-controller
version: ~> 4.4

crest:
github: mamantoha/crest

crystar:
github: naqvis/crystar

gitlab:
github: icyleaf/gitlab.cr

hash_file:
github: tassja/hash_file

octokit:
github: place-labs/octokit.cr
version: ~> 0.1.2

placeos-compiler:
github: placeos/compiler
version: ~> 4.8
Expand All @@ -35,7 +52,6 @@ dependencies:

tasker:
github: spider-gazelle/tasker

development_dependencies:
ameba:
github: crystal-ameba/ameba
Expand Down
36 changes: 36 additions & 0 deletions spec/api/remotes_spec.cr
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
require "../helper"

module PlaceOS::FrontendLoader::Api
describe Remotes do
with_server do
remotes_base = "/api/frontend-loader/v1/remotes"
it "lists releases for a given repository" do
encoded_url = URI.encode_www_form("https://www.github.com/PlaceOS/frontend-loader")
route = "#{remotes_base}/#{encoded_url}/releases"
result = curl("GET", route)
Array(String).from_json(result.body).includes?("v0.11.2").should be_true
end

it "lists commits for a given repository" do
encoded_url = URI.encode_www_form("https://www.github.com/PlaceOS/frontend-loader")
route = "#{remotes_base}/#{encoded_url}/commits"
result = curl("GET", route)
Array(PlaceOS::FrontendLoader::Remote::Commit).from_json(result.body).should_not be_empty
end

it "lists branches for a given repository" do
encoded_url = URI.encode_www_form("https://www.github.com/PlaceOS/frontend-loader")
route = "#{remotes_base}/#{encoded_url}/branches"
result = curl("GET", route)
Array(String).from_json(result.body).includes?("master").should be_true
end

it "lists tags for a given repository" do
encoded_url = URI.encode_www_form("https://www.github.com/PlaceOS/frontend-loader")
route = "#{remotes_base}/#{encoded_url}/tags"
result = curl("GET", route)
Array(String).from_json(result.body).includes?("v1.3.0").should be_true
end
end
end
end
23 changes: 20 additions & 3 deletions spec/api/repositories_spec.cr
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,16 @@ module PlaceOS::FrontendLoader::Api
commits.should_not be_empty
end

it "gets the default branch if not master" do
repository = example_repository(TEST_FOLDER, uri: "https://www.github.com/placeos/backoffice")

loader = Loader.new
loader.process_resource(:created, repository).success?.should be_true
branch = Api::Repositories.default_branch(repository.folder_name, loader: loader)
commits = Api::Repositories.commits(repository.folder_name, branch, loader: loader).not_nil!
commits.should_not be_empty
end

it "lists branches for a loaded repository" do
repository = example_repository(TEST_FOLDER)
loader = Loader.new
Expand All @@ -30,6 +40,14 @@ module PlaceOS::FrontendLoader::Api
loaded[repository.folder_name].should_not eq("HEAD")
end

it "lists releases for a loaded repository" do
repository = example_repository(TEST_FOLDER)
loader = Loader.new
loader.process_resource(:created, repository).success?.should be_true
releases = Api::Repositories.releases(repository.folder_name, loader: loader).not_nil!
releases.should_not be_empty
end

describe "query" do
it "does not mutate the managed repositories" do
branch = "test-fixture"
Expand All @@ -43,12 +61,11 @@ module PlaceOS::FrontendLoader::Api
expected_path = File.join(loader.content_directory, folder)

Dir.exists?(expected_path).should be_true
Compiler::Git.current_repository_commit(folder, loader.content_directory).should eq checked_out_commit
Api::Repositories.current_commit(expected_path).should eq checked_out_commit
Api::Repositories.branches(folder, loader).not_nil!.should_not be_empty
Api::Repositories.commits(folder, branch, loader: loader).not_nil!.should_not be_empty
Api::Repositories.commits(folder, "master", loader: loader).not_nil!.should_not be_empty

Compiler::Git.current_repository_commit(folder, loader.content_directory).should eq checked_out_commit
Api::Repositories.current_commit(expected_path).should eq checked_out_commit
end
end
end
Expand Down
19 changes: 19 additions & 0 deletions spec/gitlab_spec.cr
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
require "./helper"

module PlaceOS::FrontendLoader
LAB_TEST_FOLDER = "test-gitlab"
describe GitLab do
before_each do
reset
end
repository = example_repository(LAB_TEST_FOLDER, uri: "https://gitlab.com/bdowney/ansible-demo/")
expected_path = File.join(TEST_DIR, repository.folder_name)

it "downloads a GitLab archive" do
ref = PlaceOS::FrontendLoader::Remote::Reference.new(repository.uri, branch: "master")
actioner = PlaceOS::FrontendLoader::GitLab.new
actioner.download(ref: ref, path: expected_path)
Dir.exists?(File.join(expected_path, "decks")).should be_true
end
end
end
5 changes: 3 additions & 2 deletions spec/helper.cr
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ TEST_FOLDER = "test-repo"

def example_repository(
folder_name : String = UUID.random.to_s[0..8],
uri : String = "https://github.com/placeos/compiler",
uri : String = "https://www.github.com/placeos/compiler",
commit : String = "HEAD",
branch : String = "master"
)
Expand All @@ -35,6 +35,7 @@ def example_repository(
existing.uri = uri unless existing.uri == uri
existing.branch = branch unless existing.branch == branch
existing.commit_hash = commit unless existing.commit_hash == commit
existing.save!
existing
else
PlaceOS::Model::Generator.repository(type: :interface).tap do |repository|
Expand All @@ -43,6 +44,6 @@ def example_repository(
repository.folder_name = folder_name
repository.commit_hash = commit
repository.branch = branch
end
end.save!
end
end
31 changes: 23 additions & 8 deletions spec/loader_spec.cr
Original file line number Diff line number Diff line change
Expand Up @@ -85,7 +85,7 @@ module PlaceOS::FrontendLoader
end

it "supports changing a uri" do
expected_uri = "https://github.com/placeOS/private-drivers"
expected_uri = "https://www.github.com/placeOS/private-drivers"
repository.username = "robot@place.tech"

loader = Loader.new
Expand All @@ -94,12 +94,12 @@ module PlaceOS::FrontendLoader

repository.clear_changes_information
repository.uri = expected_uri
repository.save!
loader.process_resource(:updated, repository).success?.should be_true

Dir.exists?(expected_path).should be_true

url = Compiler::Git.run_git(expected_path, {"remote", "get-url", "origin"}).output.to_s
url.strip.should end_with("private-drivers")
Api::Repositories.current_repo(expected_path).should end_with("private-drivers")
File.exists?("/app/test-www/test-repo/README.md").should be_true
end

describe "branches" do
Expand All @@ -123,14 +123,29 @@ module PlaceOS::FrontendLoader

loader.process_resource(:created, repository).success?.should be_true
Dir.exists?(expected_path).should be_true
Compiler::Git.current_branch(expected_path).should eq branch

Api::Repositories.current_branch(expected_path).should eq branch
repository.clear_changes_information
repository.branch = updated_branch

loader.process_resource(:updated, repository).success?.should be_true
Dir.exists?(expected_path).should be_true
Compiler::Git.current_branch(expected_path).should eq updated_branch
Api::Repositories.current_branch(expected_path).should eq updated_branch
end

it "downloads a release asset" do
actioner = PlaceOS::FrontendLoader::Github.new
actioner.download_latest_asset("tassja/octokit.cr", Dir.current.to_s)
File.exists?("new_file.txt").should be_true
end

it "downloads the release asset on repo flag" do
repository = example_repository(LAB_TEST_FOLDER, uri: "https://www.github.com/tassja/octokit.cr")
repository.release = true
repository.save!
ref = PlaceOS::FrontendLoader::Remote::Reference.from_repository(repository)
expected_path = File.join(TEST_DIR, repository.folder_name)
actioner = PlaceOS::FrontendLoader::Github.new
actioner.download(ref: ref, path: expected_path)
File.exists?(File.join(expected_path, "new_file.txt")).should be_true
end
end
end
Expand Down
8 changes: 6 additions & 2 deletions src/constants.cr
Original file line number Diff line number Diff line change
Expand Up @@ -17,8 +17,12 @@ module PlaceOS::FrontendLoader
# settings for `./placeos-frontend-loader/loader.cr`
WWW = ENV["PLACE_LOADER_WWW"]? || "www"
CRON = ENV["PLACE_LOADER_CRON"]? || "0 * * * *"
GIT_USER = ENV["PLACE_LOADER_GIT_USER"]?
GIT_PASS = ENV["PLACE_LOADER_GIT_PASS"]?
GIT_USER = ENV["PLACE_LOADER_GIT_USER"]? || ""
GIT_PASS = ENV["PLACE_LOADER_GIT_PASS"]? || ""

GITLAB_TOKEN = ENV["GITLAB_TOKEN"]? || ""

BASE_REF = "https://www.github.com/PlaceOS/www-core"

# NOTE:: following used in `./placeos-frontend-loader/client.cr`
# URI.parse(ENV["PLACE_LOADER_URI"]? || "http://127.0.0.1:3000")
Expand Down
Loading