Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: support config dir #1324

Merged
merged 11 commits into from
Dec 12, 2022
Merged

feat: support config dir #1324

merged 11 commits into from
Dec 12, 2022

Conversation

aFlyBird0
Copy link
Member

@aFlyBird0 aFlyBird0 commented Dec 9, 2022 •
edited
Loading

Pre-Checklist

Note: please complete ALL items in the following checklist.

  • I have read through the CONTRIBUTING.md documentation.
  • My code has the necessary comments and documentation (if needed).
  • I have added relevant tests

Description

  • support config dir
  • duplicated tools, apps and pipeline templates check
  • create state file related to config file
  • update related docs
  • refactor flags of commands

Related Issues

close #1328

New Behavior (screenshots if needed)

1. -f flag is required

image

2. When pipeline templates are duplicated

image

3. When apps or tools are duplicated

image

4. When config is a directory

image

image

image

image

./dtm apply -f configdir --debug
2022-12-09 23:43:56 ℹ [INFO]  Log level is: debug.
2022-12-09 23:43:56 ℹ [INFO]  Apply started.
2022-12-09 23:43:56 λ [DEBUG]  renderConfigWithVariables got str: 
  - name: helm-installer
    instanceID: argocd-001

2022-12-09 23:43:56 λ [DEBUG]  Vars: ---
2022-12-09 23:43:56 λ [DEBUG]  repoOwner: RepoOwner
2022-12-09 23:43:56 λ [DEBUG]  repoTemplateBaseURL: github.com/devstream-io
2022-12-09 23:43:56 λ [DEBUG]  imageRepoOwner: ImageRepoOwner
2022-12-09 23:43:56 λ [DEBUG]  ---
2022-12-09 23:43:56 λ [DEBUG]  renderConfigWithVariables got str: 
  - name: myapp1
    spec:
      language: python
      framework: django
    repo:
      url: github.com/[[ repoOwner ]]/myapp1
    repoTemplate:
      url: [[ repoTemplateBaseURL ]]/dtm-scaffolding-python
      vars:
        imageRepo: [[ imageRepoOwner ]]/myapp1
    ci:
      - type: template
        templateName: ci-github-actions
      - type: argocdapp
  - name: myapp2
    spec:
      framework: gin
      language: golang
    repo:
      owner: [[ repoOwner ]]
      scmType: github
    repoTemplate:
      url: [[ repoTemplateBaseURL ]]/dtm-scaffolding-golang
      vars:
        imageRepo: [[ imageRepoOwner ]]/myapp2
    ci:
      - type: template
        templateName: ci-github-actions
    cd:
      - type: argocdapp

2022-12-09 23:43:56 λ [DEBUG]  Vars: ---
2022-12-09 23:43:56 λ [DEBUG]  repoOwner: RepoOwner
2022-12-09 23:43:56 λ [DEBUG]  repoTemplateBaseURL: github.com/devstream-io
2022-12-09 23:43:56 λ [DEBUG]  imageRepoOwner: ImageRepoOwner
2022-12-09 23:43:56 λ [DEBUG]  ---
2022-12-09 23:43:56 λ [DEBUG]  renderConfigWithVariables got str: name: ci-github-actions
type: github-actions
options:
    imageRepo:
        user: '[[ imageRepoOwner ]]'

2022-12-09 23:43:56 λ [DEBUG]  Vars: ---
2022-12-09 23:43:56 λ [DEBUG]  language: python
2022-12-09 23:43:56 λ [DEBUG]  framework: django
2022-12-09 23:43:56 λ [DEBUG]  repoOwner: RepoOwner
2022-12-09 23:43:56 λ [DEBUG]  repoTemplateBaseURL: github.com/devstream-io
2022-12-09 23:43:56 λ [DEBUG]  imageRepoOwner: ImageRepoOwner
2022-12-09 23:43:56 λ [DEBUG]  ---
2022-12-09 23:43:56 λ [DEBUG]  Have got 3 tools from app myapp1.
2022-12-09 23:43:56 λ [DEBUG]  renderConfigWithVariables got str: name: ci-github-actions
type: github-actions
options:
    imageRepo:
        user: '[[ imageRepoOwner ]]'

2022-12-09 23:43:56 λ [DEBUG]  Vars: ---
2022-12-09 23:43:56 λ [DEBUG]  language: golang
2022-12-09 23:43:56 λ [DEBUG]  framework: gin
2022-12-09 23:43:56 λ [DEBUG]  repoOwner: RepoOwner
2022-12-09 23:43:56 λ [DEBUG]  repoTemplateBaseURL: github.com/devstream-io
2022-12-09 23:43:56 λ [DEBUG]  imageRepoOwner: ImageRepoOwner
2022-12-09 23:43:56 λ [DEBUG]  ---
2022-12-09 23:43:56 λ [DEBUG]  Have got 3 tools from app myapp2.
2022-12-09 23:43:56 λ [DEBUG]  Plugin directory: ~/.devstream/plugins.
2022-12-09 23:43:56 λ [DEBUG]  real path: /Users/lhp/.devstream/plugins.
2022-12-09 23:43:56 λ [DEBUG]  Real plugin directory: /Users/lhp/.devstream/plugins.
2022-12-09 23:43:58 λ [DEBUG]  The global manager m is not initialized.
2022-12-09 23:43:58 ℹ [INFO]  Using local backend. State file: devstream.state.
2022-12-09 23:43:58 λ [DEBUG]  TOOL helm-installer.argocd-001 dependency already solved

2022-12-09 23:43:58 λ [DEBUG]  TOOL github-actions.myapp1 dependency NOT solved

2022-12-09 23:43:58 λ [DEBUG]  TOOL github-actions myapp1 false

2022-12-09 23:43:58 λ [DEBUG]  TOOL argocdapp.myapp1 dependency NOT solved

2022-12-09 23:43:58 λ [DEBUG]  TOOL argocdapp myapp1 false

2022-12-09 23:43:58 λ [DEBUG]  TOOL repo-scaffolding.myapp1 dependency NOT solved

2022-12-09 23:43:58 λ [DEBUG]  TOOL repo-scaffolding myapp1 false

2022-12-09 23:43:58 λ [DEBUG]  TOOL github-actions.myapp2 dependency NOT solved

2022-12-09 23:43:58 λ [DEBUG]  TOOL github-actions myapp2 false

2022-12-09 23:43:58 λ [DEBUG]  TOOL argocdapp.myapp2 dependency NOT solved

2022-12-09 23:43:58 λ [DEBUG]  TOOL argocdapp myapp2 false

2022-12-09 23:43:58 λ [DEBUG]  TOOL repo-scaffolding.myapp2 dependency NOT solved

2022-12-09 23:43:58 λ [DEBUG]  TOOL repo-scaffolding myapp2 false

2022-12-09 23:43:58 λ [DEBUG]  BATCH: [name: helm-installer
instanceID: argocd-001
dependsOn: []
options:
    instanceID: argocd-001
]
2022-12-09 23:43:58 λ [DEBUG]  TOOL github-actions.myapp1 dependency NOT solved

2022-12-09 23:43:58 λ [DEBUG]  TOOL github-actions myapp1 false

2022-12-09 23:43:58 λ [DEBUG]  TOOL argocdapp.myapp1 dependency NOT solved

2022-12-09 23:43:58 λ [DEBUG]  TOOL argocdapp myapp1 false

2022-12-09 23:43:58 λ [DEBUG]  TOOL repo-scaffolding.myapp1 dependency NOT solved

2022-12-09 23:43:58 λ [DEBUG]  TOOL repo-scaffolding myapp1 true

2022-12-09 23:43:58 λ [DEBUG]  TOOL repo-scaffolding.myapp1 dependency already solved

2022-12-09 23:43:58 λ [DEBUG]  TOOL github-actions.myapp2 dependency NOT solved

2022-12-09 23:43:58 λ [DEBUG]  TOOL github-actions myapp2 false

2022-12-09 23:43:58 λ [DEBUG]  TOOL argocdapp.myapp2 dependency NOT solved

2022-12-09 23:43:58 λ [DEBUG]  TOOL argocdapp myapp2 false

2022-12-09 23:43:58 λ [DEBUG]  TOOL repo-scaffolding.myapp2 dependency NOT solved

2022-12-09 23:43:58 λ [DEBUG]  TOOL repo-scaffolding myapp2 true

2022-12-09 23:43:58 λ [DEBUG]  TOOL repo-scaffolding.myapp2 dependency already solved

2022-12-09 23:43:58 λ [DEBUG]  BATCH: [name: repo-scaffolding
instanceID: myapp1
dependsOn:
    - helm-installer.argocd-001
options:
    destinationRepo:
        branch: main
        needAuth: true
        owner: RepoOwner
        repo: myapp1
        repoType: github
        url: github.com/RepoOwner/myapp1
    instanceID: myapp1
    sourceRepo:
        branch: main
        owner: devstream-io
        repo: dtm-scaffolding-python
        repoType: github
        url: github.com/devstream-io/dtm-scaffolding-python
    vars: {}
 name: repo-scaffolding
instanceID: myapp2
dependsOn:
    - helm-installer.argocd-001
options:
    destinationRepo:
        branch: main
        needAuth: true
        owner: RepoOwner
        repo: myapp2
        repoType: github
        url: https://github.com/RepoOwner/myapp2
    instanceID: myapp2
    sourceRepo:
        branch: main
        owner: devstream-io
        repo: dtm-scaffolding-golang
        repoType: github
        url: github.com/devstream-io/dtm-scaffolding-golang
    vars: {}
]
2022-12-09 23:43:58 λ [DEBUG]  TOOL github-actions.myapp1 dependency NOT solved

2022-12-09 23:43:58 λ [DEBUG]  TOOL github-actions.myapp1 dependency NOT solved

2022-12-09 23:43:58 λ [DEBUG]  TOOL github-actions myapp1 true

2022-12-09 23:43:58 λ [DEBUG]  TOOL github-actions.myapp1 dependency already solved

2022-12-09 23:43:58 λ [DEBUG]  TOOL argocdapp.myapp1 dependency NOT solved

2022-12-09 23:43:58 λ [DEBUG]  TOOL argocdapp.myapp1 dependency NOT solved

2022-12-09 23:43:58 λ [DEBUG]  TOOL argocdapp myapp1 true

2022-12-09 23:43:58 λ [DEBUG]  TOOL argocdapp.myapp1 dependency already solved

2022-12-09 23:43:58 λ [DEBUG]  TOOL github-actions.myapp2 dependency NOT solved

2022-12-09 23:43:58 λ [DEBUG]  TOOL github-actions.myapp2 dependency NOT solved

2022-12-09 23:43:58 λ [DEBUG]  TOOL github-actions myapp2 true

2022-12-09 23:43:58 λ [DEBUG]  TOOL github-actions.myapp2 dependency already solved

2022-12-09 23:43:58 λ [DEBUG]  TOOL argocdapp.myapp2 dependency NOT solved

2022-12-09 23:43:58 λ [DEBUG]  TOOL argocdapp.myapp2 dependency NOT solved

2022-12-09 23:43:58 λ [DEBUG]  TOOL argocdapp myapp2 true

2022-12-09 23:43:58 λ [DEBUG]  TOOL argocdapp.myapp2 dependency already solved

2022-12-09 23:43:58 λ [DEBUG]  BATCH: [name: github-actions
instanceID: myapp1
dependsOn:
    - repo-scaffolding.myapp1
    - helm-installer.argocd-001
options:
    instanceID: myapp1
    pipeline:
        configLocation: https://raw.githubusercontent.com/devstream-io/ci-template/main/github-actions/workflows/main.yml
        imageRepo:
            user: ImageRepoOwner
        language:
            framework: django
            name: python
    scm:
        name: myapp1
        url: https://github.com/RepoOwner/myapp1
 name: argocdapp
instanceID: myapp1
dependsOn:
    - repo-scaffolding.myapp1
    - helm-installer.argocd-001
options:
    app:
        name: myapp1
        namespace: argocd
    destination:
        namespace: default
        server: https://kubernetes.default.svc
    imageRepo:
        user: ImageRepoOwner
    instanceID: myapp1
    source:
        path: helm/myapp1
        repoURL: github.com/RepoOwner/myapp1
        valuefile: values.yaml
 name: github-actions
instanceID: myapp2
dependsOn:
    - repo-scaffolding.myapp2
    - helm-installer.argocd-001
options:
    instanceID: myapp2
    pipeline:
        configLocation: https://raw.githubusercontent.com/devstream-io/ci-template/main/github-actions/workflows/main.yml
        imageRepo:
            user: ImageRepoOwner
        language:
            framework: gin
            name: golang
    scm:
        name: myapp2
        owner: RepoOwner
        scmType: github
 name: argocdapp
instanceID: myapp2
dependsOn:
    - repo-scaffolding.myapp2
    - helm-installer.argocd-001
options:
    app:
        name: myapp2
        namespace: argocd
    destination:
        namespace: default
        server: https://kubernetes.default.svc
    imageRepo:
        user: ImageRepoOwner
    instanceID: myapp2
    source:
        path: helm/myapp2
        repoURL: https://github.com/RepoOwner/myapp2
        valuefile: values.yaml
]
2022-12-09 23:43:58 λ [DEBUG]  Changes for the plan:
2022-12-09 23:43:58 λ [DEBUG]  Change - 1/7 -> 
{
  ActionName: Create,
  Tool: {Name: helm-installer, InstanceID: argocd-001}}
}
2022-12-09 23:43:58 λ [DEBUG]  Change - 2/7 -> 
{
  ActionName: Create,
  Tool: {Name: repo-scaffolding, InstanceID: myapp1}}
}
2022-12-09 23:43:58 λ [DEBUG]  Change - 3/7 -> 
{
  ActionName: Create,
  Tool: {Name: repo-scaffolding, InstanceID: myapp2}}
}
2022-12-09 23:43:58 λ [DEBUG]  Change - 4/7 -> 
{
  ActionName: Create,
  Tool: {Name: github-actions, InstanceID: myapp1}}
}
2022-12-09 23:43:58 λ [DEBUG]  Change - 5/7 -> 
{
  ActionName: Create,
  Tool: {Name: argocdapp, InstanceID: myapp1}}
}
2022-12-09 23:43:58 λ [DEBUG]  Change - 6/7 -> 
{
  ActionName: Create,
  Tool: {Name: github-actions, InstanceID: myapp2}}
}
2022-12-09 23:43:58 λ [DEBUG]  Change - 7/7 -> 
{
  ActionName: Create,
  Tool: {Name: argocdapp, InstanceID: myapp2}}
}
2022-12-09 23:43:58 ℹ [INFO]  Tool (helm-installer/argocd-001) found in config but doesn't exist in the state, will be created.
2022-12-09 23:43:58 ℹ [INFO]  Tool (repo-scaffolding/myapp1) found in config but doesn't exist in the state, will be created.
2022-12-09 23:43:58 ℹ [INFO]  Tool (repo-scaffolding/myapp2) found in config but doesn't exist in the state, will be created.
2022-12-09 23:43:58 ℹ [INFO]  Tool (github-actions/myapp1) found in config but doesn't exist in the state, will be created.
2022-12-09 23:43:58 ℹ [INFO]  Tool (argocdapp/myapp1) found in config but doesn't exist in the state, will be created.
2022-12-09 23:43:58 ℹ [INFO]  Tool (github-actions/myapp2) found in config but doesn't exist in the state, will be created.
2022-12-09 23:43:58 ℹ [INFO]  Tool (argocdapp/myapp2) found in config but doesn't exist in the state, will be created.
Continue? [y/n]

or you can see ut for more information

/pkg/util/file/yaml_test.go/"ReadYamls func"

5. When keys in the config are duplicated or invalid

See ut for more information

/internal/pkg/configmanager/rawconfig_test.go/"newRawConfigFromConfigText func"

6. refactor flags of commands

image

image

@aFlyBird0
refactor: receiver of renderInstanceIDtoOptions
372c153 
Signed-off-by: Bird <aflybird0@gmail.com>
@aFlyBird0
feat: check duplicate tools and pipeline templates
c8c3548 
Signed-off-by: Bird <aflybird0@gmail.com>
@aFlyBird0
test: update ut of configmanager
76d2ed8 
Signed-off-by: Bird <aflybird0@gmail.com>
@aFlyBird0
feat: config support directory
73025d4 
Signed-off-by: Bird <aflybird0@gmail.com>
@aFlyBird0
feat: create state file relative to config file
3a7e07b 
Signed-off-by: Bird <aflybird0@gmail.com>
@aFlyBird0
test: update uts of local backend
6459e49 
Signed-off-by: Bird <aflybird0@gmail.com>
@aFlyBird0
docs: update docs of config files
ec18070 
Signed-off-by: Bird <aflybird0@gmail.com>
@aFlyBird0
test: fix ut
f10318f 
Signed-off-by: Bird <aflybird0@gmail.com>
@aFlyBird0
chore: remove useless files
6632eec 
Signed-off-by: Bird <aflybird0@gmail.com>
steinliber
steinliber reviewed Dec 9, 2022
View reviewed changes
internal/pkg/configmanager/tool.go Outdated Show resolved Hide resolved
internal/pkg/configmanager/tool.go Outdated Show resolved Hide resolved
@aFlyBird0
refactor: flags of commands
d864684 
Signed-off-by: Bird <aflybird0@gmail.com>
@aFlyBird0 aFlyBird0 marked this pull request as ready for review December 9, 2022 15:31
@aFlyBird0 aFlyBird0 requested review from IronCore864 and a team as code owners December 9, 2022 15:31
@aFlyBird0
fix: fail to get base dir when config is a directory
6cb5699 
Signed-off-by: Bird <aflybird0@gmail.com>
daniel-hutao
daniel-hutao approved these changes Dec 12, 2022
View reviewed changes
Copy link
Member

@daniel-hutao daniel-hutao left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good job!

@daniel-hutao daniel-hutao merged commit 78f0e9f into devstream-io:main Dec 12, 2022
@aFlyBird0 aFlyBird0 deleted the feat-config-dir branch January 5, 2023 16:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@daniel-hutao daniel-hutao daniel-hutao approved these changes

@steinliber steinliber steinliber approved these changes

@IronCore864 IronCore864 Awaiting requested review from IronCore864

Assignees

@steinliber steinliber

@daniel-hutao daniel-hutao

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

🍀 Proposal: Support Config Dir
3 participants
@aFlyBird0 @steinliber @daniel-hutao