r.mapcalc: Add explicit dependency on mapcalc.tab.h

[bmwiedemann]

Add explicit dependency on mapcalc.tab.h
and use a .META file to tell make that
both .c and .h are created by one call
to allow for reproducible builds.

Without this change, r.mapcalc and r3.mapcalc
would be rebuilt a 2nd time in a make -j1 run
with a slightly changed order of mapcalc.yy.o and mapcalc.tab.o

This patch was done while working on reproducible builds for openSUSE.

[echoix]

Thanks! I think I found one of these missing dependencies in issue #3406, but I didn't have the knowledge to finish debugging and writing a fix. I know using gnu make 4.4 with make --shuffle -j $(nproc) fails at different places for seemingly different reasons, but this is definitely one of the problematic modules.

[bmwiedemann]

For debugging I ran make -d with -j1 and -j4 to create http://rb.zq1.de/other/grass/build-log-1.txt.zstd http://rb.zq1.de/other/grass/build-log-4.txt.zstd (10+15 MB compressed, 600+630 MB uncompressed)

In the -j1 version you can see that such

Prerequisite 'mapcalc.tab.h' is newer than target 'OBJ.x86_64-pc-linux-gnu/column_shift.o'.
../../include/Make/Compile.make:32: update target 'OBJ.x86_64-pc-linux-gnu/column_shift.o' due to: mapcalc.tab.h

appear and trigger rebuilds of r.mapcalc and r3.mapcalc .

But there might still be other issues left such as random missing .png files:
https://rb.zq1.de/compare.factory-20240131/diffs/grass-compare.out

[echoix]

For debugging I ran make -d with -j1 and -j4 to create http://rb.zq1.de/other/grass/build-log-1.txt.zstd http://rb.zq1.de/other/grass/build-log-4.txt.zstd (10+15 MB compressed, 600+630 MB uncompressed)

In the -j1 version you can see that such

Prerequisite 'mapcalc.tab.h' is newer than target 'OBJ.x86_64-pc-linux-gnu/column_shift.o'.

../../include/Make/Compile.make:32: update target 'OBJ.x86_64-pc-linux-gnu/column_shift.o' due to: mapcalc.tab.h

appear and trigger rebuilds of r.mapcalc and r3.mapcalc .

But there might still be other issues left such as random missing .png files:

https://rb.zq1.de/compare.factory-20240131/diffs/grass-compare.out

I think you are describing another bug we caught earlier this year, but wasn't pinned down. I'm sure the solution will be through a makefile fix

#3038

[echoix]

Just for personal curiosity, is adding an extra file the best way to define extra dependencies in targets? Is the file copied out to the installed directories or included in the bindist?

Other than that, I would approve if I was confident enough with that level of Makefile, which isn't the case. Is anyone one else more knowledgeable for this?

[neteler]

Just for personal curiosity, is adding an extra file the best way to define extra dependencies in targets? Is the file copied out to the installed directories or included in the bindist?

As we moved it to OBJ.$ARCH the .META file is not copied into bindist but just generated in the "compile space" subdirectory which is eventually cleaned up.

[echoix]

Ok I see. Then who has the knowledge to approve and merge this PR?

[wenzeslaus]

Given @bmwiedemann's activity elsewhere, I feel like I can confidently approve this contribution even with my low confidence in Makefiles.

The only heads up I see is that touch fails on systems with no access to touch which is apparently some HPC systems. However, we are using touch already, so no change there.

[neteler]

Thanks for your work on reproducible builds of GRASS GIS!

[neteler]

I tagged it as "backport to 8.2" but we will not release 8.2.x any more - however, this PR might go into the upcoming 8.3.2 release (https://github.com/OSGeo/grass/milestone/24).

[echoix]

What's the relation with using an extra file vs .INTERMEDIATE or .SECONDARY special targets?

[bmwiedemann]

I experimented some more and found a way to make it do the right thing without the extra .META file and with less overall changes.

[wenzeslaus]

Thanks! This one is indeed much nicer!

[echoix]

@neteler Would you want to merge this new revision?

[nilason]

But there might still be other issues left such as random missing .png files: https://rb.zq1.de/compare.factory-20240131/diffs/grass-compare.out

That looks like #3038.

I have not been able to reproduce either with -d -j1 or -d -j8 on Mac. Cannot produce anything like:

Prerequisite 'mapcalc.tab.h' is newer than target 'OBJ.x86_64-pc-linux-gnu/column_shift.o'.

[neteler]

Backported to

• G8.3 in 7bcd38c (to be released in 8.3.2, see https://github.com/OSGeo/grass/milestone/24)
• G8.2 in c8be723 (will not be released)

[nilason]

Suggested change

	$(OBJDIR)/*.o: mapcalc.tab.h mapcalc.tab.c
	$(OBJDIR)/mapcalc.yy.o: mapcalc.tab.h mapcalc.tab.c
	$(OBJDIR)/*.o $(OBJDIR)/mapcalc.yy.o: mapcalc.tab.h mapcalc.tab.c

While it bothers me not to be able to reproduce this issue (and to full depth understand what's going on), I think this solution is good. If my suggested simplification works I'd personally prefer to have this in one line (as they are logically interconnected). Or even better, would the following solve the issue

$(OBJDIR)/*.o: mapcalc.tab.h mapcalc.tab.c

[nilason]

Too late :)

[neteler]

@nilason I'm sorry, I just merged it (didn't understand that you wanted to keep it open and modify further). So, a new PR will be needed...

[nilason]

@nilason I'm sorry, I just merged it (didn't understand that you wanted to keep it open and modify further). So, a new PR will be needed...

No worries! The one line solution I suggested should work (as it is basically the same). I can't test the latter suggestion (just adding mapcalc.tab.c as a pre-requisite to $(OBJDIR)/*.o targets) for solving the issue at hand, perhaps @bmwiedemann can chip in. The end-effect should be the same, but the code more concise.

[bmwiedemann]

$(OBJDIR)/*.o: mapcalc.tab.h mapcalc.tab.c alone did not work - my guess is that the wildcard only matches existing files. Combining both in one line should work - I'm giving it a test-run now and should have results in an hour.

[nilason]

$(OBJDIR)/*.o: mapcalc.tab.h mapcalc.tab.c alone did not work - my guess is that the wildcard only matches existing files. Combining both in one line should work - I'm giving it a test-run now and should have results in an hour.

Thanks for testing and for your contribution!

[bmwiedemann]

$(OBJDIR)/*.o $(OBJDIR)/mapcalc.yy.o: mapcalc.tab.h mapcalc.tab.c worked fine.

[nilason]

$(OBJDIR)/*.o $(OBJDIR)/mapcalc.yy.o: mapcalc.tab.h mapcalc.tab.c worked fine.

Thought so, thanks! I'll put up a PR, simplifying this.

Would you have been able to reproduce this issue you reported by only (re-)building in this module (directory)? E.g.:

cd raster/r.mapcalc
make clean
make -j1

I'm asking because I try figure out my inability to reproduce.

[bmwiedemann]

My tooling does 2 whole package builds from scratch via our spec file. Not that easy to adapt to test a partial build. My guess is that there are some cross-dependencies from outside that dir that caused the issue.

[nilason]

My tooling does 2 whole package builds from scratch via our spec file. Not that easy to adapt to test a partial build. My guess is that there are some cross-dependencies from outside that dir that caused the issue.

I did take a look at your tools, I'll see if I can make use of it on the Mac (definitely won't work as is).

Studying your log file, I realised the default GNU Make is quite old (3.81), but trying with 4.4.1 didn't make any difference.

Also I happened to notice:

 [   31s] configure: WARNING: unrecognized options: --enable-socket, --with-curses, --with-motif, --with-python, --with-wxwidgets

the listed options may all be removed from configure.

Thanks for your feedback!