Setting `autocrlf` and `safecrlf` on workers #1453

joyeecheung · 2018-08-15T16:13:55Z

For context, see nodejs/reliability#12 and the minutes of the last build meeting #1448 It was agreed to fix those workers by configuring git to not fix the line breaks.

I'll fix the nodes manually by running

git config --global core.autocrlf false
git config --global core.safecrlf false

on nodes with trouble checking out files with CRLF.

Feel free to add nodes in the following list if you find a new worker with this issue or have fixed a worker

The text was updated successfully, but these errors were encountered:

joyeecheung · 2018-08-15T16:32:30Z

Maybe we should just put the configuration in https://github.com/nodejs/build/blob/master/jenkins/scripts/node-test-commit-pre.sh instead?

richardlau · 2018-08-15T16:37:33Z

+1 for configuration changes in scripts rather than manually done.

joyeecheung · 2018-08-15T16:41:33Z

Also, as far as I can tell the containers have their configs set in the Jenkins instead of using the script

joaocgreis · 2018-08-15T16:46:06Z

autocrlf should not be changed in the Windows machines, we want to ensure new contributors can compile with a default installation of Git (we already ask for the Unix tools, but the less of those little details the better and some people might not want to change it). I understand this does not include Windows hosts for now, but there is support to run the V8 tests on Windows and the ChakraCore nightlies seem to be failing because of this.

@joyeecheung can we instead use https://github.com/nodejs/node/blob/master/.gitattributes to specify that that file is always CRLF? This is already done for vcbuild.bat.

joyeecheung · 2018-08-15T16:51:43Z

@joaocgreis The failures in CI are caused by incorrect checkouts (not a fresh clone of nodejs/node), I don't see changing settings on our CI makes a difference since that's not how new contributors set up their environment anyway? For instance, developers don't see the modified: deps/v8/third_party/jinja2/LICENSE error, but we do.

joyeecheung · 2018-08-15T16:57:52Z

can we instead use https://github.com/nodejs/node/blob/master/.gitattributes to specify that that file is always CRLF? This is already done for vcbuild.bat.

Does that work with https://github.com/nodejs/node/blob/8a41470c85536119a6dbeaf4b40dc9ebc4c453c4/deps/v8/.gitattributes#L1-L2 ? It's a file in deps/v8 but not actually in V8's upstream source, and V8 actually uses LF for all its own text files in its own .gitattributes

joaocgreis · 2018-08-15T17:03:43Z

I don't see changing settings on our CI makes a difference since that's not how new contributors set up their environment anyway?

It should be, Windows CI machines are set up in the same way that we recommend contributors to set their own (with a few additions for Jenkins). Using a different autocrlf setting in CI can cause tests to fail because of different line endings.

Please let me take a better look at this, I'd much rather find a solution that doesn't involve changing machine configuration.

joyeecheung · 2018-08-15T17:12:13Z

As it's a file maintained by us and not in V8's source, maybe we can just patch it to use LF? That way it won't conflict with deps/v8/.gitattributes

joaocgreis · 2018-08-15T17:16:24Z

As it's a file maintained by us and not in V8's source, maybe we can just patch it to use LF?

That sounds good to me, if @targos is ok with it.

joaocgreis · 2018-08-15T18:21:40Z

I confirmed this is only a problem depending on the Git version. Happens with older versions (tested 1.9.1) but not with newer (2.18.0). nodejs/node#22340 (thanks for opening @joyeecheung!) should fix the problem, but in any case I'd be ok with changing the crlf settings only in machines where Git can't be updated.

refack · 2018-08-15T19:42:27Z

we want to ensure new contributors can compile with a default installation of Git

That was my motivation as well. Setting git to not touch the files in any way would break Windows build in case file are improperly checked in (e.g. .bat file with just LF). So we will be actually testing if the code can compile even from tarball.

richardlau · 2018-08-15T19:48:03Z

So we will be actually testing if the code can compile even from tarball.

Given that we have actually broken this in the past this would be better off as a separate job (that actually unpacks the tarball and attempts to compile/test it).

refack · 2018-08-15T19:52:44Z

Given that we have actually broken this in the past this would be better off as a separate job (that actually unpacks the tarball and attempts to compile/test it).

Agreed WRT to the "packed" tarballs. Here I'm referring to GitHub repo tarballs. For example ATM inspecting the GitHub tarball, checking files in that directory LICENSE is CR+LF, while AUTHORS is LF

`deps/v8/third_party/jinja2/LICENSE` is not in upstream v8, and contains CRLF, which is in conflict with `deps/v8.gitattributes` which sets all text files to use LF. This has caused failures in CI workers with older versions of Git. This patch manually fixes up the file to use LF to resolve the conflict. The file has already been fixed in upstream jinja2, which is pull into our repo when we update V8 so it should be fixed the next time we update V8. PR-URL: nodejs#22340 Refs: nodejs/build#1443 Refs: nodejs/reliability#12 Refs: nodejs/build#1453 Refs: https://chromium-review.googlesource.com/c/993812/ Reviewed-By: Richard Lau <riclau@uk.ibm.com> Reviewed-By: João Reis <reis@janeasystems.com> Reviewed-By: Matheus Marchini <matheus@sthima.com> Reviewed-By: Luigi Pinca <luigipinca@gmail.com> Reviewed-By: Jon Moss <me@jonathanmoss.me> Reviewed-By: Ruben Bridgewater <ruben@bridgewater.de> Reviewed-By: James M Snell <jasnell@gmail.com>

`deps/v8/third_party/jinja2/LICENSE` is not in upstream v8, and contains CRLF, which is in conflict with `deps/v8/.gitattributes` which sets all text files to use LF. This has caused failures in CI workers with older versions of Git. This patch manually fixes up the file to use LF to resolve the conflict. The file has already been fixed in upstream jinja2, which is pull into our repo when we update V8 so it should be fixed the next time we update V8. PR-URL: #22340 Refs: nodejs/build#1443 Refs: nodejs/reliability#12 Refs: nodejs/build#1453 Refs: https://chromium-review.googlesource.com/c/993812/ Reviewed-By: Richard Lau <riclau@uk.ibm.com> Reviewed-By: João Reis <reis@janeasystems.com> Reviewed-By: Matheus Marchini <matheus@sthima.com> Reviewed-By: Luigi Pinca <luigipinca@gmail.com> Reviewed-By: Jon Moss <me@jonathanmoss.me> Reviewed-By: Ruben Bridgewater <ruben@bridgewater.de> Reviewed-By: James M Snell <jasnell@gmail.com>

refack · 2018-08-18T22:21:29Z

I run this groovy script:

import hudson.util.RemotingDiagnostics
import jenkins.model.Jenkins
 
String agent_name = 'test-digitalocean-ubuntu1604_sharedlibs_container-x64-3'
//groovy script you want executed on an agent
String groovy_script = '''
println "git config --global core.autocrlf false".execute().text
println "git config --global core.safecrlf false".execute().text
println "git config --global -l".execute().text
'''.trim()
 
Jenkins.instance.slaves.findAll { agent ->
    agent.name ==~ /(?i).*container.*/
}.forEach { agent ->
    println "################## ${agent.name} #################"
    println  RemotingDiagnostics.executeGroovy(groovy_script, agent.channel)
    println '####################'
}

to get:

################## test-digitalocean-alpine37_container-x64-1 #################
core.autocrlf=false
core.safecrlf=false


####################
################## test-digitalocean-alpine37_container-x64-2 #################
core.autocrlf=false
core.safecrlf=false


####################
################## test-digitalocean-alpine38_container-x64-1 #################
core.autocrlf=false
core.safecrlf=false


####################
################## test-digitalocean-alpine38_container-x64-2 #################
core.autocrlf=false
core.safecrlf=false


####################
################## test-digitalocean-ubuntu1604_container-x64-1 #################
core.autocrlf=false
core.safecrlf=false


####################
################## test-digitalocean-ubuntu1604_container-x64-2 #################
core.autocrlf=false
core.safecrlf=false


####################
################## test-digitalocean-ubuntu1604_sharedlibs_container-x64-1 #################
core.autocrlf=false
core.safecrlf=false


####################
################## test-digitalocean-ubuntu1604_sharedlibs_container-x64-10 #################
core.autocrlf=false
core.safecrlf=false


####################
################## test-digitalocean-ubuntu1604_sharedlibs_container-x64-2 #################
core.autocrlf=false
core.safecrlf=false


####################
################## test-digitalocean-ubuntu1604_sharedlibs_container-x64-3 #################
core.autocrlf=false
core.safecrlf=false


####################
################## test-digitalocean-ubuntu1604_sharedlibs_container-x64-4 #################
core.autocrlf=false
core.safecrlf=false


####################
################## test-digitalocean-ubuntu1604_sharedlibs_container-x64-5 #################
core.autocrlf=false
core.safecrlf=false


####################
################## test-digitalocean-ubuntu1604_sharedlibs_container-x64-6 #################
core.autocrlf=false
core.safecrlf=false


####################
################## test-digitalocean-ubuntu1604_sharedlibs_container-x64-7 #################
core.autocrlf=false
core.safecrlf=false


####################
################## test-digitalocean-ubuntu1604_sharedlibs_container-x64-8 #################
core.autocrlf=false
core.safecrlf=false


####################
################## test-digitalocean-ubuntu1604_sharedlibs_container-x64-9 #################
core.autocrlf=false
core.safecrlf=false


####################
################## test-digitalocean-ubuntu1804_container-x64-1 #################
core.autocrlf=false
core.safecrlf=false


####################
################## test-digitalocean-ubuntu1804_container-x64-2 #################
core.autocrlf=false
core.safecrlf=false


####################
################## test-joyent-alpine37_container-x64-1 #################
core.autocrlf=false
core.safecrlf=false


####################
################## test-joyent-alpine38_container-x64-1 #################
core.autocrlf=false
core.safecrlf=false


####################
################## test-joyent-ubuntu1604_sharedlibs_container-x64-1 #################
core.autocrlf=false
core.safecrlf=false


####################
################## test-joyent-ubuntu1604_sharedlibs_container-x64-2 #################
core.autocrlf=false
core.safecrlf=false


####################
################## test-joyent-ubuntu1604_sharedlibs_container-x64-3 #################
core.autocrlf=false
core.safecrlf=false


####################
################## test-joyent-ubuntu1604_sharedlibs_container-x64-4 #################
core.autocrlf=false
core.safecrlf=false


####################
################## test-joyent-ubuntu1604_sharedlibs_container-x64-5 #################
core.autocrlf=false
core.safecrlf=false


####################
################## test-joyent-ubuntu1804_container-x64-1 #################
core.autocrlf=false
core.safecrlf=false


####################
################## test-softlayer-alpine37_container-x64-1 #################
core.autocrlf=false
core.safecrlf=false


####################
################## test-softlayer-alpine38_container-x64-1 #################
core.autocrlf=false
core.safecrlf=false


####################
################## test-softlayer-ubuntu1604_sharedlibs_container-x64-1 #################
core.autocrlf=false
core.safecrlf=false


####################
################## test-softlayer-ubuntu1604_sharedlibs_container-x64-2 #################
core.autocrlf=false
core.safecrlf=false


####################
################## test-softlayer-ubuntu1604_sharedlibs_container-x64-3 #################
core.autocrlf=false
core.safecrlf=false


####################
################## test-softlayer-ubuntu1604_sharedlibs_container-x64-4 #################
core.autocrlf=false
core.safecrlf=false


####################
################## test-softlayer-ubuntu1604_sharedlibs_container-x64-5 #################
core.autocrlf=false
core.safecrlf=false


####################
################## test-softlayer-ubuntu1804_container-x64-1 #################
core.autocrlf=false
core.safecrlf=false


####################

refack · 2018-08-18T22:47:18Z

Also for good measure run with agent.name ==~ /(?i).*linuxone.*/:

################## test-linuxonecc-rhel72-s390x-1 #################


core.autocrlf=false
core.safecrlf=false


####################
################## test-linuxonecc-rhel72-s390x-2 #################


core.autocrlf=false
core.safecrlf=false


####################
################## test-linuxonecc-rhel72-s390x-3 #################


core.autocrlf=false
core.safecrlf=false


####################

`deps/v8/third_party/jinja2/LICENSE` is not in upstream v8, and contains CRLF, which is in conflict with `deps/v8/.gitattributes` which sets all text files to use LF. This has caused failures in CI workers with older versions of Git. This patch manually fixes up the file to use LF to resolve the conflict. The file has already been fixed in upstream jinja2, which is pull into our repo when we update V8 so it should be fixed the next time we update V8. PR-URL: #22340 Refs: nodejs/build#1443 Refs: nodejs/reliability#12 Refs: nodejs/build#1453 Refs: https://chromium-review.googlesource.com/c/993812/ Reviewed-By: Richard Lau <riclau@uk.ibm.com> Reviewed-By: João Reis <reis@janeasystems.com> Reviewed-By: Matheus Marchini <matheus@sthima.com> Reviewed-By: Luigi Pinca <luigipinca@gmail.com> Reviewed-By: Jon Moss <me@jonathanmoss.me> Reviewed-By: Ruben Bridgewater <ruben@bridgewater.de> Reviewed-By: James M Snell <jasnell@gmail.com>

refack · 2018-08-20T14:28:22Z

Help, this can't be killed:

[iojs@test-linuxonecc-rhel72-s390x-3 rhel72-s390x]$ git status
# HEAD detached at 4b7cd4b
# Changes not staged for commit:
#   (use "git add/rm <file>..." to update what will be committed)
#   (use "git checkout -- <file>..." to discard changes in working directory)
#
#       deleted:    deps/v8/third_party/jinja2/LICENSE
#
# Untracked files:
#   (use "git add <file>..." to include in what will be committed)
#
#       build/
no changes added to commit (use "git add" and/or "git commit -a")
[iojs@test-linuxonecc-rhel72-s390x-3 rhel72-s390x]$ git checkout deps/v8/third_party/jinja2/LICENSE
[iojs@test-linuxonecc-rhel72-s390x-3 rhel72-s390x]$ git status
# HEAD detached at 4b7cd4b
# Changes not staged for commit:
#   (use "git add <file>..." to update what will be committed)
#   (use "git checkout -- <file>..." to discard changes in working directory)
#
#       modified:   deps/v8/third_party/jinja2/LICENSE
#
# Untracked files:
#   (use "git add <file>..." to include in what will be committed)
#
#       build/
no changes added to commit (use "git add" and/or "git commit -a")
[iojs@test-linuxonecc-rhel72-s390x-3 rhel72-s390x]$ git --version
git version 1.8.3.1
[iojs@test-linuxonecc-rhel72-s390x-3 rhel72-s390x]$ git config --global -l
core.autocrlf=false
core.safecrlf=false
core.eol=lf
[iojs@test-linuxonecc-rhel72-s390x-3 rhel72-s390x]$

refack · 2018-08-20T14:42:12Z

So apparently with git 1.8 we need to delete the workspace for the settings to work.

joyeecheung · 2018-08-20T15:08:08Z

I thought the errors would not appear if you are checking out from a commit without the CRLF file and not rebasing from somewhere that contains it? (but yeah, it'll appear the next time we have a file like that in deps/v8)

refack · 2018-08-20T15:12:09Z

I thought the errors would not appear if you are checking out from a commit without the CRLF file and not rebasing from somewhere that contains it? (but yeah, it'll appear the next time we have a file like that in deps/v8)

I was testing explicitly with "bad" commits:

GIT_REMOTE_REF	4b7cd4bd60623b9c69d791112f4500575a3e9e7d
REBASE_ONTO		2ce03804a6a2e55a034814ca40fafac654321e33

https://ci.nodejs.org/job/node-test-commit-linuxone/4190/

When everything is setup right, it passes.

`deps/v8/third_party/jinja2/LICENSE` is not in upstream v8, and contains CRLF, which is in conflict with `deps/v8/.gitattributes` which sets all text files to use LF. This has caused failures in CI workers with older versions of Git. This patch manually fixes up the file to use LF to resolve the conflict. The file has already been fixed in upstream jinja2, which is pull into our repo when we update V8 so it should be fixed the next time we update V8. PR-URL: #22340 Refs: nodejs/build#1443 Refs: nodejs/reliability#12 Refs: nodejs/build#1453 Refs: https://chromium-review.googlesource.com/c/993812/ Reviewed-By: Richard Lau <riclau@uk.ibm.com> Reviewed-By: João Reis <reis@janeasystems.com> Reviewed-By: Matheus Marchini <matheus@sthima.com> Reviewed-By: Luigi Pinca <luigipinca@gmail.com> Reviewed-By: Jon Moss <me@jonathanmoss.me> Reviewed-By: Ruben Bridgewater <ruben@bridgewater.de> Reviewed-By: James M Snell <jasnell@gmail.com>

github-actions · 2020-03-08T00:11:56Z

This issue is stale because it has been open many days with no activity. It will be closed soon unless the stale label is removed or a comment is made.

joyeecheung mentioned this issue Aug 15, 2018

test: mark flaky tests as flaky nodejs/node#22330

Closed

joyeecheung mentioned this issue Aug 15, 2018

CI failures: 20180815 nodejs/reliability#13

Closed

joyeecheung mentioned this issue Aug 15, 2018

deps: fix CRLF in text file not present in upstream nodejs/node#22340

Closed

2 tasks

refack added bug incident ci-change PSA of configuration changes labels Aug 15, 2018

refack closed this as completed Aug 18, 2018

refack mentioned this issue Aug 20, 2018

Immediate failures on smartos, linux-containered, arm, and linuxone #1467

Closed

refack added the help wanted label Aug 20, 2018

refack reopened this Aug 20, 2018

refack mentioned this issue Sep 4, 2018

Node.js Foundation Build WorkGroup Meeting 2018-09-04 #1478

Closed

github-actions bot added the stale label Mar 8, 2020

github-actions bot closed this as completed Apr 8, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Setting `autocrlf` and `safecrlf` on workers #1453

Setting `autocrlf` and `safecrlf` on workers #1453

joyeecheung commented Aug 15, 2018 •

edited by refack

Loading

joyeecheung commented Aug 15, 2018

richardlau commented Aug 15, 2018

joyeecheung commented Aug 15, 2018

joaocgreis commented Aug 15, 2018

joyeecheung commented Aug 15, 2018 •

edited

Loading

joyeecheung commented Aug 15, 2018 •

edited

Loading

joaocgreis commented Aug 15, 2018

joyeecheung commented Aug 15, 2018 •

edited

Loading

joaocgreis commented Aug 15, 2018

joaocgreis commented Aug 15, 2018

refack commented Aug 15, 2018

richardlau commented Aug 15, 2018

refack commented Aug 15, 2018

refack commented Aug 18, 2018

refack commented Aug 18, 2018

refack commented Aug 20, 2018

refack commented Aug 20, 2018

joyeecheung commented Aug 20, 2018

refack commented Aug 20, 2018

github-actions bot commented Mar 8, 2020

Setting autocrlf and safecrlf on workers #1453

Setting autocrlf and safecrlf on workers #1453

Comments

joyeecheung commented Aug 15, 2018 • edited by refack Loading

joyeecheung commented Aug 15, 2018

richardlau commented Aug 15, 2018

joyeecheung commented Aug 15, 2018

joaocgreis commented Aug 15, 2018

joyeecheung commented Aug 15, 2018 • edited Loading

joyeecheung commented Aug 15, 2018 • edited Loading

joaocgreis commented Aug 15, 2018

joyeecheung commented Aug 15, 2018 • edited Loading

joaocgreis commented Aug 15, 2018

joaocgreis commented Aug 15, 2018

refack commented Aug 15, 2018

richardlau commented Aug 15, 2018

refack commented Aug 15, 2018

refack commented Aug 18, 2018

refack commented Aug 18, 2018

refack commented Aug 20, 2018

refack commented Aug 20, 2018

joyeecheung commented Aug 20, 2018

refack commented Aug 20, 2018

github-actions bot commented Mar 8, 2020

Setting `autocrlf` and `safecrlf` on workers #1453

Setting `autocrlf` and `safecrlf` on workers #1453

joyeecheung commented Aug 15, 2018 •

edited by refack

Loading

joyeecheung commented Aug 15, 2018 •

edited

Loading

joyeecheung commented Aug 15, 2018 •

edited

Loading

joyeecheung commented Aug 15, 2018 •

edited

Loading