Cleanup remains from previous installations after upgrade

[karolyi]

Hello,

after having found no previously built solution to my problem, and no one answering to my question regarding this on irc, I've built a script that will delete files belonging to a previous, recently upgraded installation of nextcloud after an upgrade. The issue is, files health check (self-check) will fail if you have a recently upgraded nextcloud, and those files are not deleted automatically.

Requirements are: jq, a command line json parser.
This script is built for my FreeBSD install, so you might want to modify paths within for your installation.
Also, this script won't exit with an error code in case a file is not found (and mentioned as superfluous) but will print it, so you have to check the output, instead of the exit code. A little update for using exit code might be an improvement, I can live with this version as is.

Here's the script, do what you want with it. It can be a good starting point for something similar in the future.

#!/usr/bin/env sh

NEXTCLOUD_ROOT="/usr/local/www/nextcloud"

cleanupModule() {
  MODULE="${1}"
  OUTPUT=$(su -m nextcloud -c "php -d memory_limit=512M ./occ integrity:check-app -n --no-warnings --output=json '${MO
DULE}'" | jq -r 'to_entries[] | select(.key="EXTRA_FILE") | [.value] | .[] | keys[]')
  for PATH_SUFFIX in ${OUTPUT}
  do
    FILEPATH="${NEXTCLOUD_ROOT}/apps-pkg/${MODULE}/${PATH_SUFFIX}"
    test -e "${FILEPATH}" && rm "${FILEPATH}" || echo FILE DOES NOT EXIST: ${FILEPATH}
  done
}

cleanupCore() {
  OUTPUT=$(su -m nextcloud -c "php -d memory_limit=512M ./occ integrity:check-core -n --no-warnings --output=json" | jq -r 'to_entries[] | select(.key="EXTRA_FILE") | [.value] | .[] | keys[]')
  for PATH_SUFFIX in ${OUTPUT}
  do
    FILEPATH="${NEXTCLOUD_ROOT}/${PATH_SUFFIX}"
    test -e "${FILEPATH}" && rm "${FILEPATH}" || echo FILE DOES NOT EXIST: ${FILEPATH}
  done
}

cd "${NEXTCLOUD_ROOT}"
IFS="
"

cleanupCore

MODULES=$(su -m nextcloud -c 'php -d memory_limit=512M ./occ app:list -n --no-warnings --output=json --shipped true'|jq -r '.enabled,.disabled | keys[]')

for MODULE in ${MODULES}
do
  cleanupModule "${MODULE}"
done

# vim: sw=2

[joshtrichards]

after having found no previously built solution to my problem, and no one answering to my question regarding this on irc, I've built a script that will delete files belonging to a previous, recently upgraded installation of nextcloud after an upgrade. The issue is, files health check (self-check) will fail if you have a recently upgraded nextcloud, and those files are not deleted automatically.

Can you elaborate on how you're updating your instance?

If I'm understanding the problem you're attempting to fix (a bunch of extra files being around after doing an update), this shouldn't happen.

Are you using the Web or command line Updater? Or a manual update perhaps?

[karolyi]

Hey,

I have a FreeBSD jail that has a nextcloud mount where previously installed files will get overwritten upon installation of a new version. This causes the residual files, since I use nullfs mounts (bind in linux terms) for the /usr/local/www/nextcloud mount to store the source.

I could just delete all files there, but in my experience there are files that are part of the configuration and extra installed modules, despite having specified datadirectory' => '/home/nextcloud/data' in the config.php (that is also within the installation).

Does this help?

[karolyi]

Are you using the Web or command line Updater? Or a manual update perhaps?

I'm using the FreeBSD ports package: https://www.freshports.org/www/nextcloud

[joshtrichards]

I have a FreeBSD jail that has a nextcloud mount where previously installed files will get overwritten upon installation of a new version

Okay so this has nothing to do with our Updater. :-)

I could just delete all files there, but in my experience there are files that are part of the configuration and extra installed modules, despite having specified datadirectory' => '/home/nextcloud/data' in the config.php (that is also within the installation).

You can find hints about the folders/files to retain by reviewing our manual updating process in the Maintenance section of the Admin Manual (https://docs.nextcloud.com). That might help since there sort of what you're doing it sounds like. It's basically data/, apps/ (and any additional apps paths if configured), and config/.

I'm going to close this issue out here since there isn't anything actionable for us to do in this end.

[karolyi]

You meant this URL, right?

https://docs.nextcloud.com/server/latest/admin_manual/maintenance/manual_upgrade.html

[joshtrichards]

Correct.

[karolyi]

The methods outlined there might be useful if you only care about the 'main installation' files, that is no 3rd part applications.

My script also takes care the extra files of those, in case their automatic update would leave residual files.

The only thing that makes me wonder is, sometimes more runs are necessary to entirely clean up extraneous files, so the output of the integritycheck doesn't emit all files every time, seemingly.

[joshtrichards]

The methods outlined there might be useful if you only care about the 'main installation' files, that is no 3rd part applications.

Step 10 covers custom app folders in the manual process.

My script also takes care the extra files of those, in case their automatic update would leave residual files.

The Updater takes doesn't leave extra files around unless there's a bug. There is a known issue, which seems more common in FreeBSD environments that impacts the file/folder iterator. It seems particularly easy to trigger with FreeBSD + NFS. There may be some progress on that soon (#158).

Keep in mind we don't officially support FreeBSD and the manual update approach is always an option regardless.

The only thing that makes me wonder is, sometimes more runs are necessary to entirely clean up extraneous files, so the output of the integritycheck doesn't emit all files every time, seemingly.

It should/ does unless something weird is going on.

Technically we use a similar iterator in the integrity checker so it's quite possible if impacted by the previously mentioned issue that it would impact that as well in your environment.

[karolyi]

I was thinking on actually putting together a non-bash-script based cleanup script, that is, native PHP, but I didn't make enough effort to find and extract the code that lists the extraneous files in the codebase.

Now with this information on the updater, I have a hint as to where to look for it. :) I might put together something 'more native' in the future for this.

[karolyi]

Step 10 covers custom app folders in the manual process.

Yeah, I have read it, but it only says to move folders from the old installation to the new. Which is fine, but still doesn't remove the extra files in there, that I get, unfortunately.