Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

report_results.csh: clone Test-Results.wiki with --depth=1 #492

Closed
phil-blain opened this issue Jul 20, 2020 · 18 comments
Closed

report_results.csh: clone Test-Results.wiki with --depth=1 #492

phil-blain opened this issue Jul 20, 2020 · 18 comments

Comments

@phil-blain
Copy link
Member

It would be nice if the report_results.csh script would allow using an already existing clone of Test-Results.wiki repo, and just fetch the latests commits instead of always cloning the full repo.

Or, if we stick of a full clone, just clone the latest commit (--depth=1) since we do not need the full history.

Also, it would be nice if there was an option to use the SSH url git@github.com:CICE-Consortium/Test-Results.wiki.git so that pushing would not required a password.

@apcraig
Copy link
Contributor

apcraig commented Jul 21, 2020

@phil-blain, what is the motivation for implementing a "shortcut" to the clone of the test-results wiki. That takes a few seconds in my experience. Having said that, I agree we don't need the full history and maybe setting --depth=1 would be an improvement.

With regard to the password, I've handled this for my logins by adding

[credential "https://github.com/CICE-Consortium/Test-Results.wiki.git"]
helper = store

to my .gitconfig file on several platforms. This remembers the password after the first time. That is supporting all the weekend automated testing and automatically pushes the results to the wiki page quite nicely.

@phil-blain
Copy link
Member Author

I guess the motivation would be that as time passes the clone inevitably takes more and more time. I think --depth=1 is an easy and quick improvement.

I've never used any Git credential helpers, I always use SSH... Reading https://git-scm.com/docs/git-credential-store it appears your username and password are stored unencrypted on the filesystem, so I guess if you trust root on all systems that is OK... (and using unencrypted SSH keys also means you trust root so the situation would be the same with an SSH URL)...

@apcraig
Copy link
Contributor

apcraig commented Jul 21, 2020

I agree there are questions of security, and it's something I think about. I do think the ssh version has the same issues as the credentials. I am inclined to stick with the current implementation and use credentials as the workaround. Making ssh the default would create some extra work during setup. And adding an ssh option to cice.setup seems to add complexity without a lot of benefit, but I am open. It would not be that hard to do.

@phil-blain
Copy link
Member Author

Yes. I agree: no need to add complexity. I'll refocus the title of the issue to just mention --depth=1.

@phil-blain phil-blain changed the title report_results.csh: allow using a pre-existing clone and use SSH url report_results.csh: clone with --depth=1 Jul 21, 2020
@phil-blain phil-blain changed the title report_results.csh: clone with --depth=1 report_results.csh: clone Test-Results.wiki with --depth=1 Jul 21, 2020
@apcraig apcraig self-assigned this Jul 22, 2020
apcraig added a commit to apcraig/CICE that referenced this issue Jul 23, 2020
Add warning messages when multiple set options overlap (CICE-Consortium#243)
Fix namelist output diagnostic for atmbndy (CICE-Consortium#493)
Modify git clone in script to use --depth=1 (CICE-Consortium#492)
apcraig added a commit that referenced this issue Jul 31, 2020
…ut, script warning messages for --set conflicts (#494)

* Add support for .cice_set (#464)
Add warning messages when multiple set options overlap (#243)
Fix namelist output diagnostic for atmbndy (#493)
Modify git clone in script to use --depth=1 (#492)

* update doc precedence

* update documentation, add info about .cice_set format

Co-authored-by: Philippe Blain <levraiphilippeblain@gmail.com>
@apcraig
Copy link
Contributor

apcraig commented Jul 31, 2020

Added --depth=1 in #494 for git clone when used in test results reporting as well as in code coverage stuff.

@apcraig apcraig closed this as completed Jul 31, 2020
@phil-blain
Copy link
Member Author

For posterity: you can make Git use an SSH URL even if the hardcoded URL in report_results.csh uses https://:

git config --global url.git@github.com:.insteadOf https://github.com/

This affects all repository cloned from GitHub.

@dabail10 dabail10 reopened this Mar 19, 2024
@dabail10
Copy link
Contributor

Given the recent changes with the authentication at github. Does this still work? I tried committing some results from my laptop, but I don't have the right authentication here and I was not allowed to use the standard username / password. What is the latest on this?

@apcraig
Copy link
Contributor

apcraig commented Mar 19, 2024

@dabail10, what recent changes and what problems are you having? I generally don't have problems pushing test results to the wiki without an ssh key. Is that the problem you're having?

@dabail10
Copy link
Contributor

I think the problem is that I have an ssh key on derecho somewhere and that still works. I do not have one on my laptop. I was confused about how to generate this again. I wondered if I could just copy from derecho?

@apcraig
Copy link
Contributor

apcraig commented Mar 19, 2024

I don't think you can copy from Derecho. github documentation about ssh keys is

https://docs.github.com/en/authentication/connecting-to-github-with-ssh

I am testing myself from my desktop, no ssh key. This has worked in the past, lets see if it doesn't now. Will also check to see if write permission is set differently for us in the repo.

@apcraig
Copy link
Contributor

apcraig commented Mar 19, 2024

report_results works fine for me, just tested from my mac desktop now (conda macos), https://github.com/CICE-Consortium/Test-Results/wiki/cice_by_hash. I don't have an ssh key defined. Also, I checked permissions, I think you should be able to write to the test results wiki. I am running the test suite with

./cice.setup --suite travis_suite -m conda -e macos --testid t01 --bgen default --bcmp default --bdir ${basedir} --report

What error are you getting?

@dabail10
Copy link
Contributor

Interesting. I am running the report_results.csh script manually. Then github is asking for my userid and passwd, but it won't allow that. So, perhaps there is something else done when you do the --report option?

@dabail10
Copy link
Contributor

Maybe it is related to what @phil-blain was saying above.

@apcraig
Copy link
Contributor

apcraig commented Mar 19, 2024

I don't think the --report is doing anything special. report_results.csh manually should work too. What error are you getting? And what issue are you referring to from above? Also, I checkout with

git clone https://github.com/cice-consortium/CICE cice.master.${date} --recursive

and I have the following in .gitconfig,

[push]
default = simple

[credential "https://github.com/CICE-Consortium/Test-Results.wiki.git"]
helper = store

[credential]
helper = store

Maybe it's how you're checking out the repo or your config settings?

@dabail10
Copy link
Contributor

All I have in .gitconfig is my email and name. This is the error. Does it ask you for a userid and password? I think you have to have keys stored.

./report_results.csh: Running results.csh
Cloning into 'Test-Results.wiki'...
remote: Enumerating objects: 3071, done.
remote: Counting objects: 100% (3071/3071), done.
remote: Compressing objects: 100% (754/754), done.
remote: Total 3071 (delta 2967), reused 2343 (delta 2317), pack-reused 0
Receiving objects: 100% (3071/3071), 11.83 MiB | 22.56 MiB/s, done.
Resolving deltas: 100% (2967/2967), done.
./report_results.csh: writing to Test-Results.wiki/icepack_dev/f6ff8f7c4d.conda.macos.24-02-29.210513.0.md
sed: 1: "Test-Results.wiki/icepa ...": invalid command code T
[master db75771] update f6ff8f7c4d4cb6feabe3651b13204cf43fc948e3 conda
2 files changed, 29 insertions(+)
create mode 100644 icepack_dev/f6ff8f7c4d.conda.macos.24-02-29.210513.0.md
Username for 'https://github.com': dabail10
Password for 'https://dabail10@github.com':
remote: Support for password authentication was removed on August 13, 2021.
remote: Please see https://docs.github.com/get-started/getting-started-with-git/about-remote-repositories#cloning-with-https-urls for information on currently recommended modes of authentication.
fatal: Authentication failed for 'https://github.com/CICE-Consortium/Test-Results.wiki.git/'

@phil-blain
Copy link
Member Author

As the message says, support for password authentication for Git HTTPS operations was removed by GitHub a while ago. You need to create a personal access token, which you put in instead of your password. It's explained on the linked help page.

Probably the script works for Tony because the token is what is recorded by the "store" credential-helper.

@dabail10
Copy link
Contributor

That's what I was thinking. I was trying to remember what I did on derecho / cheyenne awhile back.

@dabail10
Copy link
Contributor

I just figured it out! I coped .gitconfig and .git-credentials from derecho over to my Mac and it works now!

@apcraig apcraig closed this as completed Mar 20, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants