Compares current with previous test-status.json

[FriedrichRober]

High-level status dashboard

Transfer this into the README.md after this pull-request is merged.
Replace FriedrichRober with gap-system.

Test	GAP master	GAP 4.11.1
Released packages

Python Scripts

• tools/generate-report.py creates a report.md that compares the test results with the previous run.
  It also generates a test-status-diff.json which we could use for writing the comment as suggested in Automatically add (resp. update) PR comments with test results #194.
  We could also directly use report.md as the comment. We could use test-status-diff.json in order to figure out, if we should write notifacations to slack or if we can automerge pull-request.
• tools/update-latest-report.py updates the latest report symlink
  and the corresponding html-redirects and badges.

[fingolfin]

Thank you for working on this! I left some remarks

[fingolfin]

I think we should add a function like this to utils.py, as we have similar code in soooo many places

def read_json(path):
    with open(path, 'rb') as f:
        return json.load(f)

Note that I used 'rb', to read as binary, with the hope that we can ignore encodings this way; but it may not work... Of course in the four places we call json.load right now, we do...

• open(fname, 'r')
• open(fname, "rb")
• open(fname, "r", encoding="utf-8")
• open(fname, 'r', encoding='utf-8', errors='ignore')

So actually unifying this may be difficult sigh. But at least within this file, we could do it...

[fingolfin]

... aaaand also a write_json helper might be useful as well

[fingolfin]

We need an empty gh-pages and data branch

There is no such thing as an "empty" branch, but I've created dummy gh-pages and data branches. I'll have a few more questions and comments later. Thanks for working on this

[fingolfin]

It is not quite clear to me what needs to be done / what the status of this is. Perhaps we can discuss it a bit tomorrow?

[FriedrichRober]

@fingolfin, also after you merged some new package updates, the test-all.yml workflow does not work, since we cannot load all packages.

Edit: Sometimes it seems to randomly work, and sometimes it fails. Really weird.

Edit 2: It seems, that sometimes the first run on a commit fails. The next workflow that runs on the same commit succeeds after that.

Edit 3: The problem seems to have disappeared

[FriedrichRober]

@fingolfin, could you configure our GAP bot somehow, so that it has a picture and we could click on its name in the commit history? Currently this is not possible, but it was possible with the Github Actions Bot.

Edit: For the pull-request this seems to work, but once you merge, the name seems to change? At least in the commit history the name (Gap Package Distribution Bot) is not clickable, but in the pull-request tab the name (gap-package-distribution-bot) is clickable.

Edit 2: Yeah, this is quite weird. The author/committer in our actions is maybe set wrong. For example in this pull-request you can click on our bot. But if you click on the first force-push commit, you cannot click on the author (which should be our bot, something is fishy here).

Edit 3: Ok, the commiter is definitely set wrong. I maybe have found a solution here

Edit 4: Going here shows the id of our bot

Edit 5: Fixed wrong commiter for our bot in all actions.

[FriedrichRober]

@fingolfin, the pull-request is finished. For now, I scheduled the daily tests for 2 am (I hope).

I am not sure if splitting the daily tests into different YML workflows for different GAP versions (like Wilf did in gap-infra/integration) is somehow more efficient than calling workflows in parallel as I did now.

---

Here is a dynamic dashboard that we could use in the README.md later:

Test	GAP master	GAP 4.11.1
Released packages

---

Here is the report for master. (For some reason the package hapcryst is now succeeding.)
You can access the reports also by clicking on the above badges:

Package Evaluation Report

Job Properties

Testing: master/2022-03-30-19:10:47-84bdcfd8 vs master/2022-03-30-18:22:10-3ff817cf

Generated by Workflow: https://github.com/FriedrichRober/PackageDistro/actions/runs/2066703509

In total, 152 packages were tested, out of which 114 succeeded, 17 failed and 21 were skipped.

✔️ ✔️ Packages now succeeding

1 package(s) succeeded tests only on the current version.

Click to show package(s)!

• hapcryst 0.1.14 vs hapcryst 0.1.14 (failure)
  

❗ Packages still failing

17 package(s) failed tests also on the previous version.

Click to show package(s)!

• ctbllib 1.3.3
  
• fining 1.4.1
  
• francy 1.2.4
  
• groupoids 1.69
  
• hap 1.38
  
• normalizinterface 1.3.2
  
• openmath 11.5.0
  
• packagemanager 1.2
  
• rcwa 4.6.4
  
• recog 1.3.2
  
• semigroups 4.0.0
  
• transgrp 3.5
  
• ugaly 4.0.1
  
• unitlib 4.0.0
  
• wedderga 4.10.1
  
• xmod 2.86
  
• yangbaxter 0.9.0
  

✔️ Packages still succeeding

113 package(s) succeeded tests also on the previous version.

Click to show package(s)!

• ace 5.4
  
• aclib 1.3.2
  
• agt 0.2
  
• alnuth 3.2.0
  
• anupq 3.2.6
  
• atlasrep 2.1.1
  
• autodoc 2022.03.10
  
• automata 1.15
  
• automgrp 1.3.2
  
• autpgrp 1.10.2
  
• cap 2022.03-06
  
• caratinterface 2.3.3
  
• cddinterface 2020.06.24
  
• circle 1.6.4
  
• cohomolo 1.6.9
  
• congruence 1.2.3
  
• crime 1.6
  
• crisp 1.4.5
  
• crypting 0.10
  
• cryst 4.1.24
  
• crystcat 1.1.9
  
• cubefree 1.19
  
• curlinterface 2.2.2
  
• cvec 2.7.5
  
• datastructures 0.2.7
  
• deepthought 1.0.5
  
• design 1.7
  
• difsets 2.3.1
  
• digraphs 1.5.1
  
• edim 1.3.5
  
• example 4.3.0
  
• factint 1.6.3
  
• ferret 1.0.6
  
• fga 1.4.0
  
• float 1.0.3
  
• format 1.4.3
  
• forms 1.2.7
  
• fplsa 1.2.5
  
• fr 2.4.8
  
• fwtree 1.3
  
• gbnp 1.0.5
  
• generalizedmorphismsforcap 2022.03-03
  
• genss 1.6.6
  
• gradedringforhomalg 2022.03-01
  
• grape 4.8.5
  
• grpconst 2.6.2
  
• guarana 0.96.3
  
• guava 3.15
  
• hecke 1.5.3
  
• help 3.5
  
• idrel 2.43
  
• images 1.3.1
  
• intpic 0.2.4
  
• io 4.7.2
  
• irredsol 1.4.3
  
• json 2.1.0
  
• jupyterkernel 1.4.1
  
• jupyterviz 1.5.1
  
• kan 1.34
  
• kbmag 1.5.9
  
• laguna 3.9.4
  
• liealgdb 2.2.1
  
• liepring 1.9.2
  
• liering 2.4.2
  
• linearalgebraforcap 2022.03-04
  
• loops 3.4.1
  
• lpres 1.0.3
  
• majoranaalgebras 1.4
  
• mapclass 1.4.5
  
• matgrp 0.64
  
• modisom 2.5.1
  
• modulepresentationsforcap 2022.03-02
  
• monoidalcategories 2022.03-02
  
• nconvex 2020.11-04
  
• nilmat 1.4.1
  
• nock 1.5
  
• nq 2.5.7
  
• numericalsgps 1.3.0
  
• orb 4.8.4
  
• patternclass 2.4.2
  
• permut 2.0.4
  
• polenta 1.3.10
  
• polymaking 0.8.6
  
• primgrp 3.4.1
  
• profiling 2.5.0
  
• qpa 1.33
  
• quagroup 1.8.3
  
• radiroot 2.9
  
• rds 1.8
  
• repndecomp 1.2.1
  
• repsn 3.1.0
  
• resclasses 4.7.2
  
• scscp 2.3.1
  
• sglppow 2.1
  
• sgpviz 0.999.5
  
• simpcomp 2.1.14
  
• singular 2020.12.18
  
• sla 1.5.3
  
• smallgrp 1.4.2
  
• smallsemi 0.6.13
  
• sonata 2.9.3
  
• sophus 1.25
  
• spinsym 1.5.2
  
• symbcompcc 1.3.2
  
• thelma 1.3
  
• tomlib 1.2.9
  
• toric 1.9.5
  
• unipot 1.5
  
• utils 0.72
  
• uuid 0.7
  
• walrus 0.9991
  
• xmodalg 1.18
  
• zeromqinterface 0.13
  

➖ Packages that still skipped

21 package(s) skipped tests also on the previous version.

Click to show package(s)!

• 4ti2interface 2022.03-01
  
• browse 1.8.14
  
• corelg 1.55
  
• examplesforhomalg 2022.03-01
  
• gapdoc 1.6.5
  
• gauss 2022.03-01
  
• gaussforhomalg 2022.03-01
  
• gradedmodules 2022.03-01
  
• homalg 2022.03-01
  
• homalgtocas 2022.03-01
  
• io_forhomalg 2022.03-01
  
• itc 1.5.1
  
• localizeringforhomalg 2022.03-01
  
• matricesforhomalg 2022.03-02
  
• modules 2022.03-01
  
• polycyclic 2.16
  
• ringsforhomalg 2022.03-01
  
• sco 2022.03-01
  
• toolsforhomalg 2022.03-01
  
• toricvarieties 2022.03.23
  
• xgap 4.31
  

[fingolfin]

Awesome! Let's see how it fares :-)