Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add push to remote mirror repository #15157

Merged
merged 73 commits into from
Jun 14, 2021
Merged
Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
73 commits
Select commit Hold shift + click to select a range
4e0e107
Added push mirror model.
KN4CK3R Mar 22, 2021
5797f96
Integrated push mirror into queue.
KN4CK3R Mar 24, 2021
00bf264
Moved methods into own file.
KN4CK3R Mar 25, 2021
000cf56
Added basic implementation.
KN4CK3R Mar 26, 2021
5e701eb
Mirror wiki too.
KN4CK3R Mar 27, 2021
ecbbc67
Removed duplicated method.
KN4CK3R Mar 28, 2021
f6f4dba
Get url for different remotes.
KN4CK3R Mar 28, 2021
2365760
Added migration.
KN4CK3R Mar 31, 2021
aff13d6
Unified remote url access.
KN4CK3R Mar 31, 2021
ff4b4ab
Add/Remove push mirror remotes.
KN4CK3R Mar 31, 2021
382be07
Prevent hangs with missing credentials.
KN4CK3R Mar 31, 2021
bf22cd1
Moved code between files.
KN4CK3R Apr 1, 2021
5604087
Lint
KN4CK3R Apr 1, 2021
84a9bd8
Changed sanitizer interface.
KN4CK3R Apr 3, 2021
ca14354
Added push mirror backend methods.
KN4CK3R Apr 3, 2021
a9b1961
Fix
KN4CK3R Apr 3, 2021
ca2c530
Only update the mirror remote.
KN4CK3R Apr 4, 2021
84d1352
Limit refs on push.
KN4CK3R Apr 5, 2021
806408a
Added UI part.
KN4CK3R Apr 5, 2021
4eed55d
Merge branch 'master' of https://github.com/go-gitea/gitea into featu…
KN4CK3R Apr 5, 2021
79201d7
Added missing table.
KN4CK3R Apr 5, 2021
b78b83e
Delete mirror if repository gets removed.
KN4CK3R Apr 6, 2021
96dab54
Merge branch 'master' of https://github.com/go-gitea/gitea into featu…
KN4CK3R Apr 9, 2021
07ad389
Merge branch 'master' of https://github.com/go-gitea/gitea into featu…
KN4CK3R Apr 13, 2021
bafeba1
Merge branch 'master' of https://github.com/go-gitea/gitea into featu…
KN4CK3R Apr 17, 2021
c0d3c78
Changed signature. Handle object errors.
KN4CK3R Apr 17, 2021
20de69c
Added upload method.
KN4CK3R Apr 17, 2021
92e2573
Added "upload" unit tests.
KN4CK3R Apr 18, 2021
0d9c384
Added transfer adapter unit tests.
KN4CK3R Apr 18, 2021
5746665
Merge branch 'master' of https://github.com/go-gitea/gitea into featu…
KN4CK3R Apr 18, 2021
3bc2300
Send correct headers.
KN4CK3R Apr 18, 2021
c4ff5d2
Added pushing of LFS objects.
KN4CK3R Apr 19, 2021
0012f34
Go 1.14
KN4CK3R Apr 19, 2021
bacae71
Added more logging.
KN4CK3R Apr 20, 2021
0b3e91a
Simpler body handling.
KN4CK3R Apr 20, 2021
f2cbd62
Process files in batches to reduce HTTP calls.
KN4CK3R Apr 20, 2021
3688138
Merge branch 'master' into feature-push-mirror
6543 Apr 21, 2021
656c347
Fixed unit tests.
KN4CK3R Apr 21, 2021
a6088cb
Added created timestamp.
KN4CK3R Apr 21, 2021
1923b51
Fixed name.
KN4CK3R Apr 21, 2021
1eb9c74
Merge branch 'master' of https://github.com/go-gitea/gitea into featu…
KN4CK3R May 1, 2021
5739008
Fixed invalid column name.
KN4CK3R May 12, 2021
980548a
Changed name to prevent xorm auto setting.
KN4CK3R May 12, 2021
310fdb9
Remove table header im empty.
KN4CK3R May 12, 2021
33d0244
Strip exit code from error message.
KN4CK3R May 12, 2021
ffaf55c
Added docs page about mirroring.
KN4CK3R May 13, 2021
180a61c
Fixed date.
KN4CK3R May 18, 2021
835f228
Merge branch 'main' of https://github.com/go-gitea/gitea into feature…
KN4CK3R May 18, 2021
b83497b
Use new method.
KN4CK3R May 18, 2021
2a01584
Use jsoniter.
KN4CK3R May 18, 2021
d92a8ba
Merge branch 'main' of https://github.com/go-gitea/gitea into feature…
KN4CK3R May 30, 2021
882e958
Merge branch 'main' of https://github.com/go-gitea/gitea into feature…
KN4CK3R May 31, 2021
902a730
Fixed merge errors.
KN4CK3R May 31, 2021
68e0c00
Use syncPullMirror in test.
KN4CK3R May 31, 2021
ac879f8
Fixed merge error.
KN4CK3R May 31, 2021
65f47da
Moved test to integrations.
KN4CK3R May 31, 2021
2bc6102
Added push mirror test.
KN4CK3R Jun 1, 2021
d7a3719
Removed NextUpdateUnix.
KN4CK3R Jun 2, 2021
dad9adf
and here
KN4CK3R Jun 2, 2021
c876f05
Merge branch 'main' into feature-push-mirror
6543 Jun 2, 2021
687302a
Merge branch 'main' of https://github.com/go-gitea/gitea into feature…
KN4CK3R Jun 5, 2021
c950877
Merge branch 'main' into feature-push-mirror
6543 Jun 6, 2021
c76cdc6
Merge branch 'main' into feature-push-mirror
6543 Jun 7, 2021
ec872c0
Merge branch 'main' into feature-push-mirror
6543 Jun 9, 2021
4e39abb
Apply suggestions from code review
6543 Jun 11, 2021
e482136
Merge branch 'main' into feature-push-mirror
6543 Jun 11, 2021
19d7911
Fixed sql error.
KN4CK3R Jun 11, 2021
444b96c
Added test.
KN4CK3R Jun 11, 2021
27b30d6
Merge branch 'main' of https://github.com/go-gitea/gitea into feature…
KN4CK3R Jun 14, 2021
567dccc
untouch models/migrations/v182.go
6543 Jun 14, 2021
c305bca
Merge branch 'main' into feature-push-mirror
6543 Jun 14, 2021
9a66e72
Merge branch 'master' into feature-push-mirror
6543 Jun 14, 2021
2c05692
`xorm:"text"`
6543 Jun 14, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
113 changes: 113 additions & 0 deletions models/repo_pushmirror.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,113 @@
// Copyright 2021 The Gitea Authors. All rights reserved.
// Use of this source code is governed by a MIT-style
// license that can be found in the LICENSE file.

package models

import (
"errors"
"time"

"code.gitea.io/gitea/modules/log"
"code.gitea.io/gitea/modules/timeutil"

"xorm.io/xorm"
)

var (
// ErrPushMirrorNotExist mirror does not exist error
ErrPushMirrorNotExist = errors.New("PushMirror does not exist")
6543 marked this conversation as resolved.
Show resolved Hide resolved
)

// PushMirror represents mirror information of a repository.
type PushMirror struct {
ID int64 `xorm:"pk autoincr"`
RepoID int64 `xorm:"INDEX"`
Repo *Repository `xorm:"-"`
RemoteName string

Interval time.Duration
UpdatedUnix timeutil.TimeStamp `xorm:"INDEX"`
NextUpdateUnix timeutil.TimeStamp `xorm:"INDEX"`
LastError string
KN4CK3R marked this conversation as resolved.
Show resolved Hide resolved
}

// BeforeInsert will be invoked by XORM before inserting a record
func (m *PushMirror) BeforeInsert() {
if m != nil {
m.UpdatedUnix = timeutil.TimeStampNow()
m.NextUpdateUnix = timeutil.TimeStampNow()
}
}

// AfterLoad is invoked from XORM after setting the values of all fields of this object.
func (m *PushMirror) AfterLoad(session *xorm.Session) {
6543 marked this conversation as resolved.
Show resolved Hide resolved
if m == nil {
return
}

var err error
m.Repo, err = getRepositoryByID(session, m.RepoID)
if err != nil {
log.Error("getRepositoryByID[%d]: %v", m.ID, err)
}
}

// InsertPushMirror inserts a push-mirror to database
func InsertPushMirror(m *PushMirror) error {
_, err := x.Insert(m)
return err
}

// UpdatePushMirror updates the push-mirror
func UpdatePushMirror(m *PushMirror) error {
_, err := x.ID(m.ID).AllCols().Update(m)
return err
}

// DeletePushMirrorByID deletes a push-mirrors by ID
func DeletePushMirrorByID(ID int64) error {
_, err := x.Delete(&PushMirror{ID: ID})
6543 marked this conversation as resolved.
Show resolved Hide resolved
return err
}

// DeletePushMirrorsByRepoID deletes all push-mirrors by repoID
func DeletePushMirrorsByRepoID(repoID int64) error {
_, err := x.Delete(&PushMirror{RepoID: repoID})
return err
}

// GetPushMirrorByID returns push-mirror information.
func GetPushMirrorByID(ID int64) (*PushMirror, error) {
m := &PushMirror{ID: ID}
has, err := x.Get(m)
6543 marked this conversation as resolved.
Show resolved Hide resolved
if err != nil {
return nil, err
} else if !has {
return nil, ErrPushMirrorNotExist
}
return m, nil
}

// GetPushMirrorsByRepoID returns push-mirror informations of a repository.
func GetPushMirrorsByRepoID(repoID int64) ([]*PushMirror, error) {
mirrors := make([]*PushMirror, 0, 10)
return mirrors, x.Where("repo_id=?", repoID).Find(&mirrors)
}

// PushMirrorsIterate iterates all push-mirror repositories.
func PushMirrorsIterate(f func(idx int, bean interface{}) error) error {
return x.
Where("next_update_unix<=?", time.Now().Unix()).
And("next_update_unix!=0").
Iterate(new(PushMirror), f)
}

// ScheduleNextUpdate calculates and sets next update time.
func (m *PushMirror) ScheduleNextUpdate() {
if m.Interval != 0 {
m.NextUpdateUnix = timeutil.TimeStampNow().AddDuration(m.Interval)
} else {
m.NextUpdateUnix = 0
}
}
Loading