chore: barretenberg in the monorepo

[ludamad]

Description

This implements "Integrate barretenberg code and CI situated at circuits/cpp/barretenberg" in #1126. The motion to include barretenberg in the monorepo has not had major opposition, though for sure there was hesitation around the disruption and effort. Minor opposition was noted around the repo barrier and CI times, but overall we believe we can solve these issues.

Note about "Ensure barretenberg circuits/cpp/barretenberg is either a viable path or move it" in #1126:
One non-ideal feature of this PR is that barretenberg is nested in circuits/cpp/barrretenberg. This might make more sense for the circuits team's current setup than someone trying to find barretenberg to use standalone. However,

• There's an interest to get onto a monorepo quickly to have a small cost in disruption
• Moving barretenberg from the location in this PR will not cause issues with the git blame function in e.g. vscode

Future improvements are welcome, although there isn't necessarily a 'correct place' for barretenberg.

Process:

set -xue
# Clone both repositories
git clone https://github.com/AztecProtocol/barretenberg.git
git clone https://github.com/AztecProtocol/aztec-packages.git

# Enter barretenberg repository and filter it
cd barretenberg
git filter-repo --to-subdirectory-filter barretenberg
git filter-repo --message-callback """
  import re

  def replace_hash_with_url(text):
       url_format = 'https://github.com/AztecProtocol/barretenberg/issues/{}'
       # the regular expression pattern r'#(\d+)' matches a '#' followed by one or more digits (\d+)
       return re.sub(r'#(\d+)', lambda match: url_format.format(match.group(1)), text)
  message = message.decode('utf-8') # Decode from bytes to string
  message = message.replace('(#', '(https://github.com/AztecProtocol/barretenberg/pull/')
  message = message.replace('Merge pull request #', 'Merge pull request https://github.com/AztecProtocol/barretenberg/pull/')
  return replace_hash_with_url(message).encode('utf-8') # Encode back into bytes
"""

# Enter aztec-packages repository
cd ../aztec-packages

# Create a new branch for the combined history
git checkout -b combined_history

# Add barretenberg as a remote and fetch it
git remote add barretenberg ../barretenberg
git fetch barretenberg

# Merge the histories, preserving the directory structure
git merge --no-commit --allow-unrelated-histories barretenberg/master

# Commit the changes
git commit -m "chore: barretenberg in the monorepo" -m "See #1126 for motivation."

Checklist:

• [Sorry, no] I have reviewed my diff in github, line by line.
• [*] Every change is related to the PR description.
• [*] I have linked this pull request to the issue(s) that it resolves.
• [*] There are no unexpected formatting changes, superfluous debug logs, or commented-out code.
• [*] The branch has been merged or rebased against the head of its merge target.
• [*] I'm happy for the PR to be merged at the reviewer's next convenience.

[ludamad]

@dbanks12 @spalladino @charlielye @codygunton This is now a viable base that I think can be merged. I have some followup stuff (github action, subrepo file, team-based job filters) but given that I want every commit to go into master here, I don't want to add too much more to this one

[codygunton]

Can you explain how you made this PR? If you just delete the submodule definition, I guess all of Bb shows up as new files. but then there's no history for those files, right? But we see the full history. What was the workflow?

[ludamad]

We will have full history now, this I checked. Github is chugging with this PR so you might not see good details. This is the process

set -xue
# Clone both repositories
git clone https://github.com/AztecProtocol/barretenberg.git
git clone https://github.com/AztecProtocol/aztec-packages.git

# Enter barretenberg repository and filter it
cd barretenberg
git filter-repo --to-subdirectory-filter barretenberg
git filter-repo --message-callback """
  import re

  def replace_hash_with_url(text):
       url_format = 'https://github.com/AztecProtocol/barretenberg/issues/{}'
       # the regular expression pattern r'#(\d+)' matches a '#' followed by one or more digits (\d+)
       return re.sub(r'#(\d+)', lambda match: url_format.format(match.group(1)), text)
  message = message.decode('utf-8') # Decode from bytes to string
  message = message.replace('(#', '(https://github.com/AztecProtocol/barretenberg/pull/')
  message = message.replace('Merge pull request #', 'Merge pull request https://github.com/AztecProtocol/barretenberg/pull/')
  return replace_hash_with_url(message).encode('utf-8') # Encode back into bytes
"""

# Enter aztec-packages repository
cd ../aztec-packages

# Create a new branch for the combined history
git checkout -b combined_history

# Add barretenberg as a remote and fetch it
git remote add barretenberg ../barretenberg
git fetch barretenberg

# Merge the histories, preserving the directory structure
git merge --no-commit --allow-unrelated-histories barretenberg/master

# Commit the changes
git commit -m "chore: barretenberg in the monorepo" -m "See #1126 for motivation."

[codygunton]

I tried out continuing work on my branch cg/simulate-spike in the new monorepo. Workflow: clone fresh, open barretenberg.code-workspace in the embedded copy of Bb, check out my branch. Result: VS code restarts; no more files the embedded barretenberg folder. What's the intended workflow to continue existing work?

[ludamad]

As I mentioned privately,

1. you need to try ad/bb-monorepo/github-action to try this out as that PR adds .gitrepo info
2. open workspace as you did
3. ./scripts/git_subrepo.sh pull circuits/cpp/barretenberg --branch=cg/simulate-spike
4. there's a merge conflict with instructions how to fix (avoidable if you first pull master in your bb branch?)

See the result #1167

[codygunton]

OK, looks like a reasonable workflow after your followup. I checked out cg/example and see that we lose the PR branch history, but I think that's acceptable, especially if you accept the temporary role of git tech support for a week or so 😅

[codygunton]

Looks good, excited for the saner workflow.

[ludamad]

OK, looks like a reasonable workflow after your followup. I checked out cg/example and see that we lose the PR branch history, but I think that's acceptable, especially if you accept the temporary role of git tech support for a week or so 😅

Yeah I think given that we squash PRs anyway we should settle for a squashed pull request if we're porting. Let's just link to the PR in bb if it has moved and call it a day (it won't appear in git blame either way, so we don't lose much, and when I do investigate as far as the PR I don't mind clicking one more link). Now I do want to replay any history that makes it to master - you can leave that to me.

[dbanks12]

LGTM. Some small comments.

[dbanks12]

why is this and other e2e tests removed?

[ludamad]

Good catch, shouldn't have been.

[dbanks12]

Where is this change from? Just a small fix you included?

[ludamad]

Surprised by this, I'll double check any changes outside of barretenberg

[dbanks12]

What is this diff doing?

[ludamad]

Nothing good!

[netlify]

✅ Deploy Preview for preeminent-bienenstitch-606ad0 canceled.

Name	Link
🔨 Latest commit	c7fb04e
🔍 Latest deploy log	https://app.netlify.com/sites/preeminent-bienenstitch-606ad0/deploys/64bef34542d39f00081e5f38