Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Setting autocrlf and safecrlf on workers #1453

Closed
12 tasks done
joyeecheung opened this issue Aug 15, 2018 · 20 comments
Closed
12 tasks done

Setting autocrlf and safecrlf on workers #1453

joyeecheung opened this issue Aug 15, 2018 · 20 comments
Labels

Comments

@joyeecheung
Copy link
Member

joyeecheung commented Aug 15, 2018

For context, see nodejs/reliability#12 and the minutes of the last build meeting #1448 It was agreed to fix those workers by configuring git to not fix the line breaks.

I'll fix the nodes manually by running

git config --global core.autocrlf false
git config --global core.safecrlf false

on nodes with trouble checking out files with CRLF.

Feel free to add nodes in the following list if you find a new worker with this issue or have fixed a worker

  • test-packetnet-ubuntu1604-arm64-2
  • test-softlayer-ubuntu1604_sharedlibs_container-x64-2
  • test-softlayer-ubuntu1604_sharedlibs_container-x64-5
  • test-softlayer-ubuntu1604_sharedlibs_container-x64-1
  • test-linuxonecc-rhel72-s390x-2
  • test-packetnet-centos7-arm64-2
  • test-joyent-smartos16-x64-1
  • test-digitalocean-ubuntu1604_sharedlibs_container-x64-1
  • test-packetnet-centos7-arm64-1
  • test-linuxonecc-rhel72-s390x-3
  • test-digitalocean-ubuntu1604_sharedlibs_container-x64-3
  • test-digitalocean-ubuntu1604_sharedlibs_container-x64-10
@joyeecheung
Copy link
Member Author

Maybe we should just put the configuration in https://github.com/nodejs/build/blob/master/jenkins/scripts/node-test-commit-pre.sh instead?

@richardlau
Copy link
Member

+1 for configuration changes in scripts rather than manually done.

@joyeecheung
Copy link
Member Author

Also, as far as I can tell the containers have their configs set in the Jenkins instead of using the script

@joaocgreis
Copy link
Member

autocrlf should not be changed in the Windows machines, we want to ensure new contributors can compile with a default installation of Git (we already ask for the Unix tools, but the less of those little details the better and some people might not want to change it). I understand this does not include Windows hosts for now, but there is support to run the V8 tests on Windows and the ChakraCore nightlies seem to be failing because of this.

@joyeecheung can we instead use https://github.com/nodejs/node/blob/master/.gitattributes to specify that that file is always CRLF? This is already done for vcbuild.bat.

@joyeecheung
Copy link
Member Author

joyeecheung commented Aug 15, 2018

@joaocgreis The failures in CI are caused by incorrect checkouts (not a fresh clone of nodejs/node), I don't see changing settings on our CI makes a difference since that's not how new contributors set up their environment anyway? For instance, developers don't see the modified: deps/v8/third_party/jinja2/LICENSE error, but we do.

@joyeecheung
Copy link
Member Author

joyeecheung commented Aug 15, 2018

can we instead use https://github.com/nodejs/node/blob/master/.gitattributes to specify that that file is always CRLF? This is already done for vcbuild.bat.

Does that work with https://github.com/nodejs/node/blob/8a41470c85536119a6dbeaf4b40dc9ebc4c453c4/deps/v8/.gitattributes#L1-L2 ? It's a file in deps/v8 but not actually in V8's upstream source, and V8 actually uses LF for all its own text files in its own .gitattributes

@joaocgreis
Copy link
Member

I don't see changing settings on our CI makes a difference since that's not how new contributors set up their environment anyway?

It should be, Windows CI machines are set up in the same way that we recommend contributors to set their own (with a few additions for Jenkins). Using a different autocrlf setting in CI can cause tests to fail because of different line endings.

Please let me take a better look at this, I'd much rather find a solution that doesn't involve changing machine configuration.

@joyeecheung
Copy link
Member Author

joyeecheung commented Aug 15, 2018

As it's a file maintained by us and not in V8's source, maybe we can just patch it to use LF? That way it won't conflict with deps/v8/.gitattributes

@joaocgreis
Copy link
Member

As it's a file maintained by us and not in V8's source, maybe we can just patch it to use LF?

That sounds good to me, if @targos is ok with it.

@joaocgreis
Copy link
Member

I confirmed this is only a problem depending on the Git version. Happens with older versions (tested 1.9.1) but not with newer (2.18.0). nodejs/node#22340 (thanks for opening @joyeecheung!) should fix the problem, but in any case I'd be ok with changing the crlf settings only in machines where Git can't be updated.

@refack
Copy link
Contributor

refack commented Aug 15, 2018

we want to ensure new contributors can compile with a default installation of Git

That was my motivation as well. Setting git to not touch the files in any way would break Windows build in case file are improperly checked in (e.g. .bat file with just LF). So we will be actually testing if the code can compile even from tarball.

@refack refack added bug incident ci-change PSA of configuration changes labels Aug 15, 2018
@richardlau
Copy link
Member

So we will be actually testing if the code can compile even from tarball.

Given that we have actually broken this in the past this would be better off as a separate job (that actually unpacks the tarball and attempts to compile/test it).

@refack
Copy link
Contributor

refack commented Aug 15, 2018

Given that we have actually broken this in the past this would be better off as a separate job (that actually unpacks the tarball and attempts to compile/test it).

Agreed WRT to the "packed" tarballs. Here I'm referring to GitHub repo tarballs. For example ATM inspecting the GitHub tarball, checking files in that directory LICENSE is CR+LF, while AUTHORS is LF
image

refack pushed a commit to joyeecheung/node that referenced this issue Aug 17, 2018
`deps/v8/third_party/jinja2/LICENSE` is not in upstream v8,
and contains CRLF, which is in conflict with `deps/v8.gitattributes`
which sets all text files to use LF.
This has caused failures in CI workers with older versions of Git.
This patch manually fixes up the file to use LF to resolve
the conflict.

The file has already been fixed in upstream jinja2,
which is pull into our repo when we update V8 so it should
be fixed the next time we update V8.

PR-URL: nodejs#22340
Refs: nodejs/build#1443
Refs: nodejs/reliability#12
Refs: nodejs/build#1453
Refs: https://chromium-review.googlesource.com/c/993812/
Reviewed-By: Richard Lau <riclau@uk.ibm.com>
Reviewed-By: João Reis <reis@janeasystems.com>
Reviewed-By: Matheus Marchini <matheus@sthima.com>
Reviewed-By: Luigi Pinca <luigipinca@gmail.com>
Reviewed-By: Jon Moss <me@jonathanmoss.me>
Reviewed-By: Ruben Bridgewater <ruben@bridgewater.de>
Reviewed-By: James M Snell <jasnell@gmail.com>
joyeecheung added a commit to nodejs/node that referenced this issue Aug 17, 2018
`deps/v8/third_party/jinja2/LICENSE` is not in upstream v8,
and contains CRLF, which is in conflict with `deps/v8/.gitattributes`
which sets all text files to use LF.
This has caused failures in CI workers with older versions of Git.
This patch manually fixes up the file to use LF to resolve
the conflict.

The file has already been fixed in upstream jinja2,
which is pull into our repo when we update V8 so it should
be fixed the next time we update V8.

PR-URL: #22340
Refs: nodejs/build#1443
Refs: nodejs/reliability#12
Refs: nodejs/build#1453
Refs: https://chromium-review.googlesource.com/c/993812/
Reviewed-By: Richard Lau <riclau@uk.ibm.com>
Reviewed-By: João Reis <reis@janeasystems.com>
Reviewed-By: Matheus Marchini <matheus@sthima.com>
Reviewed-By: Luigi Pinca <luigipinca@gmail.com>
Reviewed-By: Jon Moss <me@jonathanmoss.me>
Reviewed-By: Ruben Bridgewater <ruben@bridgewater.de>
Reviewed-By: James M Snell <jasnell@gmail.com>
@refack
Copy link
Contributor

refack commented Aug 18, 2018

I run this groovy script:

import hudson.util.RemotingDiagnostics
import jenkins.model.Jenkins
 
String agent_name = 'test-digitalocean-ubuntu1604_sharedlibs_container-x64-3'
//groovy script you want executed on an agent
String groovy_script = '''
println "git config --global core.autocrlf false".execute().text
println "git config --global core.safecrlf false".execute().text
println "git config --global -l".execute().text
'''.trim()
 
Jenkins.instance.slaves.findAll { agent ->
    agent.name ==~ /(?i).*container.*/
}.forEach { agent ->
    println "################## ${agent.name} #################"
    println  RemotingDiagnostics.executeGroovy(groovy_script, agent.channel)
    println '####################'
}

to get:

################## test-digitalocean-alpine37_container-x64-1 #################
core.autocrlf=false
core.safecrlf=false


####################
################## test-digitalocean-alpine37_container-x64-2 #################
core.autocrlf=false
core.safecrlf=false


####################
################## test-digitalocean-alpine38_container-x64-1 #################
core.autocrlf=false
core.safecrlf=false


####################
################## test-digitalocean-alpine38_container-x64-2 #################
core.autocrlf=false
core.safecrlf=false


####################
################## test-digitalocean-ubuntu1604_container-x64-1 #################
core.autocrlf=false
core.safecrlf=false


####################
################## test-digitalocean-ubuntu1604_container-x64-2 #################
core.autocrlf=false
core.safecrlf=false


####################
################## test-digitalocean-ubuntu1604_sharedlibs_container-x64-1 #################
core.autocrlf=false
core.safecrlf=false


####################
################## test-digitalocean-ubuntu1604_sharedlibs_container-x64-10 #################
core.autocrlf=false
core.safecrlf=false


####################
################## test-digitalocean-ubuntu1604_sharedlibs_container-x64-2 #################
core.autocrlf=false
core.safecrlf=false


####################
################## test-digitalocean-ubuntu1604_sharedlibs_container-x64-3 #################
core.autocrlf=false
core.safecrlf=false


####################
################## test-digitalocean-ubuntu1604_sharedlibs_container-x64-4 #################
core.autocrlf=false
core.safecrlf=false


####################
################## test-digitalocean-ubuntu1604_sharedlibs_container-x64-5 #################
core.autocrlf=false
core.safecrlf=false


####################
################## test-digitalocean-ubuntu1604_sharedlibs_container-x64-6 #################
core.autocrlf=false
core.safecrlf=false


####################
################## test-digitalocean-ubuntu1604_sharedlibs_container-x64-7 #################
core.autocrlf=false
core.safecrlf=false


####################
################## test-digitalocean-ubuntu1604_sharedlibs_container-x64-8 #################
core.autocrlf=false
core.safecrlf=false


####################
################## test-digitalocean-ubuntu1604_sharedlibs_container-x64-9 #################
core.autocrlf=false
core.safecrlf=false


####################
################## test-digitalocean-ubuntu1804_container-x64-1 #################
core.autocrlf=false
core.safecrlf=false


####################
################## test-digitalocean-ubuntu1804_container-x64-2 #################
core.autocrlf=false
core.safecrlf=false


####################
################## test-joyent-alpine37_container-x64-1 #################
core.autocrlf=false
core.safecrlf=false


####################
################## test-joyent-alpine38_container-x64-1 #################
core.autocrlf=false
core.safecrlf=false


####################
################## test-joyent-ubuntu1604_sharedlibs_container-x64-1 #################
core.autocrlf=false
core.safecrlf=false


####################
################## test-joyent-ubuntu1604_sharedlibs_container-x64-2 #################
core.autocrlf=false
core.safecrlf=false


####################
################## test-joyent-ubuntu1604_sharedlibs_container-x64-3 #################
core.autocrlf=false
core.safecrlf=false


####################
################## test-joyent-ubuntu1604_sharedlibs_container-x64-4 #################
core.autocrlf=false
core.safecrlf=false


####################
################## test-joyent-ubuntu1604_sharedlibs_container-x64-5 #################
core.autocrlf=false
core.safecrlf=false


####################
################## test-joyent-ubuntu1804_container-x64-1 #################
core.autocrlf=false
core.safecrlf=false


####################
################## test-softlayer-alpine37_container-x64-1 #################
core.autocrlf=false
core.safecrlf=false


####################
################## test-softlayer-alpine38_container-x64-1 #################
core.autocrlf=false
core.safecrlf=false


####################
################## test-softlayer-ubuntu1604_sharedlibs_container-x64-1 #################
core.autocrlf=false
core.safecrlf=false


####################
################## test-softlayer-ubuntu1604_sharedlibs_container-x64-2 #################
core.autocrlf=false
core.safecrlf=false


####################
################## test-softlayer-ubuntu1604_sharedlibs_container-x64-3 #################
core.autocrlf=false
core.safecrlf=false


####################
################## test-softlayer-ubuntu1604_sharedlibs_container-x64-4 #################
core.autocrlf=false
core.safecrlf=false


####################
################## test-softlayer-ubuntu1604_sharedlibs_container-x64-5 #################
core.autocrlf=false
core.safecrlf=false


####################
################## test-softlayer-ubuntu1804_container-x64-1 #################
core.autocrlf=false
core.safecrlf=false


####################

@refack refack closed this as completed Aug 18, 2018
@refack
Copy link
Contributor

refack commented Aug 18, 2018

Also for good measure run with agent.name ==~ /(?i).*linuxone.*/:

################## test-linuxonecc-rhel72-s390x-1 #################


core.autocrlf=false
core.safecrlf=false


####################
################## test-linuxonecc-rhel72-s390x-2 #################


core.autocrlf=false
core.safecrlf=false


####################
################## test-linuxonecc-rhel72-s390x-3 #################


core.autocrlf=false
core.safecrlf=false


####################

targos pushed a commit to nodejs/node that referenced this issue Aug 19, 2018
`deps/v8/third_party/jinja2/LICENSE` is not in upstream v8,
and contains CRLF, which is in conflict with `deps/v8/.gitattributes`
which sets all text files to use LF.
This has caused failures in CI workers with older versions of Git.
This patch manually fixes up the file to use LF to resolve
the conflict.

The file has already been fixed in upstream jinja2,
which is pull into our repo when we update V8 so it should
be fixed the next time we update V8.

PR-URL: #22340
Refs: nodejs/build#1443
Refs: nodejs/reliability#12
Refs: nodejs/build#1453
Refs: https://chromium-review.googlesource.com/c/993812/
Reviewed-By: Richard Lau <riclau@uk.ibm.com>
Reviewed-By: João Reis <reis@janeasystems.com>
Reviewed-By: Matheus Marchini <matheus@sthima.com>
Reviewed-By: Luigi Pinca <luigipinca@gmail.com>
Reviewed-By: Jon Moss <me@jonathanmoss.me>
Reviewed-By: Ruben Bridgewater <ruben@bridgewater.de>
Reviewed-By: James M Snell <jasnell@gmail.com>
@refack
Copy link
Contributor

refack commented Aug 20, 2018

Help, this can't be killed:

[iojs@test-linuxonecc-rhel72-s390x-3 rhel72-s390x]$ git status
# HEAD detached at 4b7cd4b
# Changes not staged for commit:
#   (use "git add/rm <file>..." to update what will be committed)
#   (use "git checkout -- <file>..." to discard changes in working directory)
#
#       deleted:    deps/v8/third_party/jinja2/LICENSE
#
# Untracked files:
#   (use "git add <file>..." to include in what will be committed)
#
#       build/
no changes added to commit (use "git add" and/or "git commit -a")
[iojs@test-linuxonecc-rhel72-s390x-3 rhel72-s390x]$ git checkout deps/v8/third_party/jinja2/LICENSE
[iojs@test-linuxonecc-rhel72-s390x-3 rhel72-s390x]$ git status
# HEAD detached at 4b7cd4b
# Changes not staged for commit:
#   (use "git add <file>..." to update what will be committed)
#   (use "git checkout -- <file>..." to discard changes in working directory)
#
#       modified:   deps/v8/third_party/jinja2/LICENSE
#
# Untracked files:
#   (use "git add <file>..." to include in what will be committed)
#
#       build/
no changes added to commit (use "git add" and/or "git commit -a")
[iojs@test-linuxonecc-rhel72-s390x-3 rhel72-s390x]$ git --version
git version 1.8.3.1
[iojs@test-linuxonecc-rhel72-s390x-3 rhel72-s390x]$ git config --global -l
core.autocrlf=false
core.safecrlf=false
core.eol=lf
[iojs@test-linuxonecc-rhel72-s390x-3 rhel72-s390x]$

@refack refack reopened this Aug 20, 2018
@refack
Copy link
Contributor

refack commented Aug 20, 2018

So apparently with git 1.8 we need to delete the workspace for the settings to work.

@joyeecheung
Copy link
Member Author

I thought the errors would not appear if you are checking out from a commit without the CRLF file and not rebasing from somewhere that contains it? (but yeah, it'll appear the next time we have a file like that in deps/v8)

@refack
Copy link
Contributor

refack commented Aug 20, 2018

I thought the errors would not appear if you are checking out from a commit without the CRLF file and not rebasing from somewhere that contains it? (but yeah, it'll appear the next time we have a file like that in deps/v8)

I was testing explicitly with "bad" commits:

GIT_REMOTE_REF	4b7cd4bd60623b9c69d791112f4500575a3e9e7d
REBASE_ONTO		2ce03804a6a2e55a034814ca40fafac654321e33

https://ci.nodejs.org/job/node-test-commit-linuxone/4190/

When everything is setup right, it passes.

targos pushed a commit to nodejs/node that referenced this issue Sep 3, 2018
`deps/v8/third_party/jinja2/LICENSE` is not in upstream v8,
and contains CRLF, which is in conflict with `deps/v8/.gitattributes`
which sets all text files to use LF.
This has caused failures in CI workers with older versions of Git.
This patch manually fixes up the file to use LF to resolve
the conflict.

The file has already been fixed in upstream jinja2,
which is pull into our repo when we update V8 so it should
be fixed the next time we update V8.

PR-URL: #22340
Refs: nodejs/build#1443
Refs: nodejs/reliability#12
Refs: nodejs/build#1453
Refs: https://chromium-review.googlesource.com/c/993812/
Reviewed-By: Richard Lau <riclau@uk.ibm.com>
Reviewed-By: João Reis <reis@janeasystems.com>
Reviewed-By: Matheus Marchini <matheus@sthima.com>
Reviewed-By: Luigi Pinca <luigipinca@gmail.com>
Reviewed-By: Jon Moss <me@jonathanmoss.me>
Reviewed-By: Ruben Bridgewater <ruben@bridgewater.de>
Reviewed-By: James M Snell <jasnell@gmail.com>
@github-actions
Copy link

github-actions bot commented Mar 8, 2020

This issue is stale because it has been open many days with no activity. It will be closed soon unless the stale label is removed or a comment is made.

@github-actions github-actions bot added the stale label Mar 8, 2020
@github-actions github-actions bot closed this as completed Apr 8, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

4 participants