Skip to content

Commit

Permalink
Adds support for local paths and project bundles (LMMS#5735)
Browse files Browse the repository at this point in the history
* Adds a baseDir for the local path

	Adds a new Base for paths called "local:", which will translate to the dir where the currently opened project file is at. In the future this will allow us to make project bundles and make it easier to export and transfer projects to other people without breaking the paths to samples, presets, plugins and others.

* Starts implementing the makeBundle functionality

	For now, to make a bundle LMMS has to be run through CLI with the makeBundle/--makeBundle command, followed by an input file and an output file ('lmms --makeBundle input.mmp output.mmp'). DataFile::writeBundle() is then called. For now, it only saves the mmp/mmpz file normally and also creates a "resources" folder if it doesn't exists. Later it will also manipulate the DataFile so all paths are local and copy all files to the resources folder.

TODO:
	-Remove warnings.
	-Implement the logic to manipulate the DataFile and copy files.

* Starts implementing logic to go through resources

	Starts implementing the logic that will go through all the resources of the project file and add them to the bundle. We use a std::map of QString to std::vector<QString>: The first string is the DOM element tagname that is going to be searched for. The vector of strings holds all attributes this element can have that accesses resources. For now we just print those to the screen.

* Adds logic to copy files and update the project

	The raw logic for creating the bundle is finished. It now copies the resource files and update the project to use "local:" paths to the new file now.
	Now it's a matter of organizing things and adding safety checks for file operation errors basically.

* Makes the writeBundle method more organized

	Improves comments and debugging warnings to make the writeBundle a bit more organized for review.

* Adds a project bundle folder

	Adds a project bundle folder, inside which the bundles will be created. Instead of receiving an output project file name, the makeBundle command now receives a bundle name that will be used as the name of the bundle's folder.
	Uses a typedef for the std::map with the tags and attributes with resources.

TODO:
	- Fix the local: prefix so it works when we don't have the project file open (for CLI usage, or find another way to deal with it).
	- Sanitize the bundle name.
	- Allow overwriting bundles?

* Handles local paths when a project isn't open

	The PathUtil base prefix conversion for "local:" uses the loaded song file name. When we are running the makebundle command from the CLI there isn't a loaded project, so those prefixes aren't converted properly. Now, when there isn't a project open PathUtil will return "local:" again when it tries to convert this base prefix. DataFile can then check if the base prefix is still there, and if it is it knows the conversion wasn't possible, so it does the conversion itself. To do that, a member called m_fileName was added to DataFile, which will hold the file being manipulated if that's where the DataFile originated from, and the local path can be retrieved from it.

* Sanitizes the bundle name

	The bundle name is now sanitized. Since it's going to be used as a folder name, we need to keep the user from giving invalid folder names as a bundle's name. The rules for the name are:
	1) It must start with a word character (either a digit or letter)
	2) It can be followed by any number of letters, digits, whitespaces or hyphens
	3) It must end with a word character (either a digit or letter)

	A Regexp is used to check for the name validity.

* Moves away from projectbundle folder concept

	This commit regresses some functionality. Project bundles will be saved just as any other project, except they will also have the resources folder. It will be up to the user to organize the bundles on their own folders. It's currently not allowed to save a bundle on a folder where there's one already though (if there's a resources folder already). Later it might be allowed to overwrite bundles in that case.
	The projectbundles folder was dropped. The user can save project bundles anywhere in the system.
	The DataFile::writeBundle was removed. It's functionality was merged into the DataFile::writeFile method, by adding a boolean on the parameters defining whether it should be saved with resources. The logic of copying the resource files and changing the paths inside the project DataFile was moved to DataFile::copyResources, making the methods a little bit less dense.

* Adds an option to save project as bundle

	The "Save As" dialog now has an option to save project as a project bundle (with resources), which will save the file as a bundle.

Known bug:
	- Because the "local:" base prefix is translated to the filename from the Engine::getSong(), it breaks when Song::guiSaveProjectAs is called, because that method changes the project name before saving. Urgent fix!

* Fix local: prefix saving bug

	There was a bug where "local:" prefixes weren't resolved properly during saving because Song::guiSaveProjectAs() changed the project name to the destiny file name before saving. This resulted in the local paths using the destination file as a reference. Both Song::guiSaveProject() and Song::guiSaveProjectAs() were rewritten, and now they only rename the project after it's saved.

* Adds a warning message box

	When the user tries to save a project bundle on a folder that already has a project bundle (contains a resources folder) a message box pops up telling the user it's not permitted and that another path should be chosen.

* Removes unused header

	Forgot to remove <QRegExp> header when I removed the code that used it.

* Removes Vestige plugins bundling

	For safety reasons, remove the possibility to bundle VSTs loaded
through vestige. Also runs a safety check during the project being
loaded (Song::loadProject) to check if either Vestige plugins or effect
plugins are using local paths, and abort the project load if so. That is
to avoid malicious code being run because of bad DLLs being shipped with
a project file.

* Extracts code from loadProject to another method

	Extracts code that checks if a DataFile contains local paths to
plugins to another method inside DataFile.

* Removes debug warnings

	Removes warnings previously used for debugging. Improves a
warning message on PathUtil.

* Fixes small bug with error logging

	Fixes small bug, where a QMessageBox was being used to prompt an
error without checking if the gui is loaded first. Now we check for the
GUI and if we are in CLI mode we use a QTextStream instead.

* Saves the bundle in a newly created folder

	Now a folder with the project name is created inside which the
bundle will be saved. This makes the process more convenient.
	Some save errors that previously only triggered qWarnings now
trigger message boxes to warn the user of what happened (using a lambda
function that either shows message boxes or trigger qWarnings depending
whether a gui is present).
	Makes it so saving a bundle doesn't change the loaded project
path, that way the user won't be able to accidentally "Save" over a
bundle which should not be done for now.

* Enhances the name conflict workaround

	Now, instead of replacing the resource names with meaningless
numbers, the bundle save will just append a counter to the end of
filenames that have been repeated.

* Starts addressing Johannes review

* Adds makebundle action to bash completion file

	Adds the bash completion code for the made bundle action.

* Improves safety check on project files

	Now, instead of checking certain XML tags for local paths,
DataFile::hasLocalPlugin() will return true if ANY tag that isn't on the
RESOURCE_ELEMENTS list contains an attribute that starts with "local:".
	The method is now recursive so it can go through all XML tags
during this check.

* Addresses Spekular change request

	Uses basePrefix(Base::LocalDir) instead of "local:" on the
return of unresolved local paths.

* Makes hasLocalPlugins method const

* Replaces literal uses of "local:"

	Instead of using "local:" we are now retrieving the base prefix
from PathUtil, so if we change the prefix on the future we don't need to
replace every mention to it as well.

* Fix some comments on the header and cpp file

* Changes variable on PathUtil to const

	Changes the retrieved pointer to the song object to a const
pointer.

* Leave doxygen comment on CPP file only

	There was 2 doxygen comments for the same method, on the header
and CPP file. The latter was kept since it goes into more details about
the functionality of the method.

* Fix doxygen comment @param

	Fixes the doxygen comment from hasLocalPlugin().

* Remove assert statements

	Some assert statements were being done wrong and are probably
even unnecessary for that piece of code, so they were removed.

* Skips local paths when looking for shortest path

	PathUtil::toShortestRelative() was including the local paths on
the candidate paths, which could lead to a unallowed resource (i.e.:
vst plugin) to be assigned a local path even on a regular save.
	The local paths are now skipped when looking for the shortest
relative path, since they should only be used by the bundle save on the
allowed resources.

* Address Spekular's review

	Changes some of the PathUtil methods to allow a boolean pointer
to be used to return the status of the method, setting it to false if it
failed somewhere.
	Also adds a parameter to toShortestRelative to either allow or
forbid local paths in the search for the shortest relative path.

* Replaces "ok" with "error"
  • Loading branch information
IanCaio authored and devnexen committed Apr 10, 2021
1 parent 4898da8 commit f49a3c0
Show file tree
Hide file tree
Showing 10 changed files with 418 additions and 65 deletions.
13 changes: 12 additions & 1 deletion doc/bash-completion/lmms
Original file line number Diff line number Diff line change
Expand Up @@ -89,7 +89,7 @@ _lmms()
pars_render=(--float --bitrate --format --interpolation)
pars_render+=(--loop --mode --output --profile)
pars_render+=(--samplerate --oversampling)
actions=(dump compress render rendertracks upgrade)
actions=(dump compress render rendertracks upgrade makebundle)
actions_old=(-d --dump -r --render --rendertracks -u --upgrade)
shortargs+=(-a -b -c -f -h -i -l -m -o -p -s -v -x)

Expand Down Expand Up @@ -250,6 +250,17 @@ _lmms()
filemode="files"
filetypes="$savefiletypes"
fi
elif [ "$action_found" == "makebundle" ]
then
if [ "$prev" == "makebundle" ]
then
filemode="existing_files"
filetypes="$savefiletypes"
elif [ "$prev2" == "makebundle" ]
then
filemode="files"
filetypes="$savefiletypes"
fi
elif [[ "$action_found" =~ render(tracks)? ]]
then
if [[ "$prev" =~ render(tracks)? ]]
Expand Down
10 changes: 9 additions & 1 deletion include/DataFile.h
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@
#ifndef DATA_FILE_H
#define DATA_FILE_H

#include <map>
#include <QDomDocument>

#include "lmms_export.h"
Expand Down Expand Up @@ -72,7 +73,9 @@ class LMMS_EXPORT DataFile : public QDomDocument
QString nameWithExtension( const QString& fn ) const;

void write( QTextStream& strm );
bool writeFile( const QString& fn );
bool writeFile(const QString& fn, bool withResources = false);
bool copyResources(const QString& resourcesDir); //!< Copies resources to the resourcesDir and changes the DataFile to use local paths to them
bool hasLocalPlugins(QDomElement parent = QDomElement(), bool firstCall = true) const;

QDomElement& content()
{
Expand Down Expand Up @@ -123,6 +126,10 @@ class LMMS_EXPORT DataFile : public QDomDocument
// List of ProjectVersions for the legacyFileVersion method
static const std::vector<ProjectVersion> UPGRADE_VERSIONS;

// Map with DOM elements that access resources (for making bundles)
typedef std::map<QString, std::vector<QString>> ResourcesMap;
static const ResourcesMap ELEMENTS_WITH_RESOURCES;

void upgrade();

void loadData( const QByteArray & _data, const QString & _sourceFile );
Expand All @@ -135,6 +142,7 @@ class LMMS_EXPORT DataFile : public QDomDocument
} ;
static typeDescStruct s_types[TypeCount];

QString m_fileName; //!< The origin file name or "" if this DataFile didn't originate from a file
QDomElement m_content;
QDomElement m_head;
Type m_type;
Expand Down
24 changes: 16 additions & 8 deletions include/PathUtil.h
Original file line number Diff line number Diff line change
Expand Up @@ -8,12 +8,18 @@
namespace PathUtil
{
enum class Base { Absolute, ProjectDir, FactorySample, UserSample, UserVST, Preset,
UserLADSPA, DefaultLADSPA, UserSoundfont, DefaultSoundfont, UserGIG, DefaultGIG };
UserLADSPA, DefaultLADSPA, UserSoundfont, DefaultSoundfont, UserGIG, DefaultGIG,
LocalDir };

//! Return the directory associated with a given base as a QString
QString LMMS_EXPORT baseLocation(const Base base);
//! Return the directory associated with a given base as a QDir
QDir LMMS_EXPORT baseQDir (const Base base);
//! Optionally, if a pointer to boolean is given the method will
//! use it to indicate whether the prefix could be resolved properly
//! or not.
QString LMMS_EXPORT baseLocation(const Base base, bool* error = nullptr);
//! Return the directory associated with a given base as a QDir.
//! Optional pointer to boolean to indicate if the prefix could
//! be resolved properly.
QDir LMMS_EXPORT baseQDir (const Base base, bool* error = nullptr);
//! Return the prefix used to denote this base in path strings
QString LMMS_EXPORT basePrefix(const Base base);
//! Check the prefix of a path and return the base it corresponds to
Expand All @@ -28,13 +34,15 @@ namespace PathUtil
//! Upgrade prefix-less relative paths to the new format
QString LMMS_EXPORT oldRelativeUpgrade(const QString & input);

//! Make this path absolute
QString LMMS_EXPORT toAbsolute(const QString & input);
//! Make this path absolute. If a pointer to boolean is given
//! it will indicate whether the path was converted successfully
QString LMMS_EXPORT toAbsolute(const QString & input, bool* error = nullptr);
//! Make this path relative to a given base, return an absolute path if that fails
QString LMMS_EXPORT relativeOrAbsolute(const QString & input, const Base base);
//! Make this path relative to any base, choosing the shortest if there are
//! multiple options. Defaults to an absolute path if all bases fail.
QString LMMS_EXPORT toShortestRelative(const QString & input);
//! multiple options. allowLocal defines whether local paths should be considered.
//! Defaults to an absolute path if all bases fail.
QString LMMS_EXPORT toShortestRelative(const QString & input, bool allowLocal = false);

}

Expand Down
9 changes: 7 additions & 2 deletions include/Song.h
Original file line number Diff line number Diff line change
Expand Up @@ -72,9 +72,14 @@ class LMMS_EXPORT Song : public TrackContainer
* Should we discard MIDI ControllerConnections from project files?
*/
BoolModel discardMIDIConnections{false};
/**
* Should we save the project as a project bundle? (with resources)
*/
BoolModel saveAsProjectBundle{false};

void setDefaultOptions() {
discardMIDIConnections.setValue(false);
saveAsProjectBundle.setValue(false);
}
};

Expand Down Expand Up @@ -282,8 +287,8 @@ class LMMS_EXPORT Song : public TrackContainer
void createNewProjectFromTemplate( const QString & templ );
void loadProject( const QString & filename );
bool guiSaveProject();
bool guiSaveProjectAs( const QString & filename );
bool saveProjectFile( const QString & filename );
bool guiSaveProjectAs(const QString & filename);
bool saveProjectFile(const QString & filename, bool withResources = false);

const QString & projectFileName() const
{
Expand Down
1 change: 1 addition & 0 deletions include/VersionedSaveDialog.h
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,7 @@ class SaveOptionsWidget : public QWidget {

private:
LedCheckBox *m_discardMIDIConnectionsCheckbox;
LedCheckBox *m_saveAsProjectBundleCheckbox;
};

class VersionedSaveDialog : public FileDialog
Expand Down
Loading

0 comments on commit f49a3c0

Please sign in to comment.