Enable Merging of BibDatabases

[DominikVoigt]

This PR adds a method to the BibDatabases that allows instances to be merged with other instances.
This merging will not introduce duplicates if an entry is contained in both databases.

• Change in CHANGELOG.md described (if applicable)
• Tests created for changes (if applicable)
• Manually tested changed features in running JabRef (always required)
• Screenshots added in PR description (for UI changes)
• Checked documentation: Is the information available and up to date? If not created an issue at https://github.com/JabRef/user-documentation/issues or, even better, submitted a pull request to the documentation repository.

[tobiasdiez]

Good that you have a look at the merge functionality as well.

We already have two merge methods:

jabref/src/main/java/org/jabref/gui/importer/ImportEntriesViewModel.java

Lines 106 to 195 in 7cc5747

	public void importEntries(List<BibEntry> entriesToImport, boolean downloadFiles) {
	// Check if we are supposed to warn about duplicates.
	// If so, then see if there are duplicates, and warn if yes.
	if (preferences.shouldWarnAboutDuplicatesForImport()) {
	BackgroundTask.wrap(() -> entriesToImport.stream()
	.anyMatch(this::hasDuplicate)).onSuccess(duplicateFound -> {
	if (duplicateFound) {
	boolean continueImport = dialogService.showConfirmationDialogWithOptOutAndWait(Localization.lang("Duplicates found"),
	Localization.lang("There are possible duplicates (marked with an icon) that haven't been resolved. Continue?"),
	Localization.lang("Continue with import"),
	Localization.lang("Cancel import"),
	Localization.lang("Disable this confirmation dialog"),
	optOut -> preferences.setShouldWarnAboutDuplicatesForImport(!optOut));
	if (!continueImport) {
	dialogService.notify(Localization.lang("Import canceled"));
	} else {
	buildImportHandlerThenImportEntries(entriesToImport);
	}
	} else {
	buildImportHandlerThenImportEntries(entriesToImport);
	}
	}).executeWith(Globals.TASK_EXECUTOR);
	} else {
	buildImportHandlerThenImportEntries(entriesToImport);
	}
	if (downloadFiles) {
	for (BibEntry bibEntry : entriesToImport) {
	for (LinkedFile linkedFile : bibEntry.getFiles()) {
	LinkedFileViewModel linkedFileViewModel = new LinkedFileViewModel(linkedFile, bibEntry, databaseContext, taskExecutor, dialogService, preferences.getXMPPreferences(), preferences.getFilePreferences(), ExternalFileTypes.getInstance());
	linkedFileViewModel.download();
	}
	}
	}
	NamedCompound namedCompound = new NamedCompound(Localization.lang("Import file"));
	namedCompound.addEdit(new UndoableInsertEntries(databaseContext.getDatabase(), entriesToImport));
	// merge strings into target database
	for (BibtexString bibtexString : parserResult.getDatabase().getStringValues()) {
	String bibtexStringName = bibtexString.getName();
	if (databaseContext.getDatabase().hasStringByName(bibtexStringName)) {
	String importedContent = bibtexString.getContent();
	String existingContent = databaseContext.getDatabase().getStringByName(bibtexStringName).get().getContent();
	if (!importedContent.equals(existingContent)) {
	LOGGER.warn("String contents differ for {}: {} != {}", bibtexStringName, importedContent, existingContent);
	// TODO: decide what to do here (in case the same string exits)
	}
	} else {
	databaseContext.getDatabase().addString(bibtexString);
	// FIXME: this prevents this method to be moved to logic - we need to implement a new undo/redo data model
	namedCompound.addEdit(new UndoableInsertString(databaseContext.getDatabase(), bibtexString));
	}
	}
	// copy content selectors to target database
	MetaData targetMetada = databaseContext.getMetaData();
	parserResult.getMetaData()
	.getContentSelectorList()
	.forEach(contentSelector -> targetMetada.addContentSelector(contentSelector));
	// TODO undo of content selectors (currently not implemented)
	// copy groups to target database
	parserResult.getMetaData().getGroups().ifPresent(
	newGroupsTreeNode -> {
	if (targetMetada.getGroups().isPresent()) {
	GroupTreeNode groupTreeNode = targetMetada.getGroups().get();
	newGroupsTreeNode.moveTo(groupTreeNode);
	namedCompound.addEdit(
	new UndoableAddOrRemoveGroup(
	new GroupTreeNodeViewModel(groupTreeNode),
	new GroupTreeNodeViewModel(newGroupsTreeNode),
	UndoableAddOrRemoveGroup.ADD_NODE));
	} else {
	// target does not contain any groups, so we can just use the new groups
	targetMetada.setGroups(newGroupsTreeNode);
	namedCompound.addEdit(
	new UndoableAddOrRemoveGroup(
	new GroupTreeNodeViewModel(newGroupsTreeNode),
	new GroupTreeNodeViewModel(newGroupsTreeNode),
	UndoableAddOrRemoveGroup.ADD_NODE));
	}
	}
	);
	namedCompound.end();
	Globals.undoManager.addEdit(namedCompound);
	JabRefGUI.getMainFrame().getCurrentBasePanel().markBaseChanged();
	}

jabref/src/main/java/org/jabref/gui/importer/ImportAction.java

Lines 129 to 189 in 7cc5747

	private ParserResult mergeImportResults(List<ImportFormatReader.UnknownFormatImport> imports) {
	BibDatabase resultDatabase = new BibDatabase();
	ParserResult result = new ParserResult(resultDatabase);
	for (ImportFormatReader.UnknownFormatImport importResult : imports) {
	if (importResult == null) {
	continue;
	}
	ParserResult parserResult = importResult.parserResult;
	List<BibEntry> entries = parserResult.getDatabase().getEntries();
	resultDatabase.insertEntries(entries);
	if (ImportFormatReader.BIBTEX_FORMAT.equals(importResult.format)) {
	// additional treatment of BibTeX
	// merge into existing database
	// Merge strings
	for (BibtexString bibtexString : parserResult.getDatabase().getStringValues()) {
	String bibtexStringName = bibtexString.getName();
	if (resultDatabase.hasStringByName(bibtexStringName)) {
	String importedContent = bibtexString.getContent();
	String existingContent = resultDatabase.getStringByName(bibtexStringName).get().getContent();
	if (!importedContent.equals(existingContent)) {
	LOGGER.warn("String contents differ for {}: {} != {}", bibtexStringName, importedContent, existingContent);
	// TODO: decide what to do here (in case the same string exits)
	}
	} else {
	resultDatabase.addString(bibtexString);
	}
	}
	// Merge groups
	// Adds the specified node as a child of the current root. The group contained in <b>newGroups </b> must not be of
	// type AllEntriesGroup, since every tree has exactly one AllEntriesGroup (its root). The <b>newGroups </b> are
	// inserted directly, i.e. they are not deepCopy()'d.
	parserResult.getMetaData().getGroups().ifPresent(newGroups -> {
	// ensure that there is always only one AllEntriesGroup in the resulting database
	// "Rename" the AllEntriesGroup of the imported database to "Imported"
	if (newGroups.getGroup() instanceof AllEntriesGroup) {
	// create a dummy group
	try {
	// This will cause a bug if the group already exists
	// There will be group where the two groups are merged
	String newGroupName = importResult.parserResult.getFile().map(File::getName).orElse("unknown");
	ExplicitGroup group = new ExplicitGroup("Imported " + newGroupName, GroupHierarchyType.INDEPENDENT,
	Globals.prefs.getKeywordDelimiter());
	newGroups.setGroup(group);
	group.add(parserResult.getDatabase().getEntries());
	} catch (IllegalArgumentException e) {
	LOGGER.error("Problem appending entries to group", e);
	}
	}
	result.getMetaData().getGroups().ifPresent(newGroups::moveTo);
	});
	for (ContentSelector selector : parserResult.getMetaData().getContentSelectorList()) {
	result.getMetaData().addContentSelector(selector);
	}
	}
	// TODO: collect errors into ParserResult, because they are currently ignored (see caller of this method)
	}

It would be good to refactor and combine them (with your newly added method as well).

Side remark: depending on your envisioned applications, the ImportEntriesDialog might be helpful.

[koppor]

The linked code refs #6488

Currently, I understand the new code better than the looong linked code.

We also have BibDatabaseDiff, which doesn't seem to be referenced in any code. (Example use: https://github.com/koppor/jabref/pull/442/files#diff-58f2d74f80b59a9ba6468243b879c428R68)

[koppor]

We merged the merge methods at the cose of "some" undo/redo (at the whole import 😟)

undo/redo refs JabRef#453

[koppor]

Commit issues resolved. LGTM.

[tobiasdiez]

Thanks. The code looks mostly good to me. I've a few comments and suggestions for improvement.

[tobiasdiez]

Since this is not related to writing bibtex (as the rest of the package), I suggest to move it together with the duplication check to a new logic.database package in parallel to the existing model.database. @JabRef/developers opinions?

[DominikVoigt]

I moved it to the recommended package :).

[tobiasdiez]

Please reduce these examples to a minimum. I don't think you need to have 4 entries with full information

[DominikVoigt]

I reduced the number of entries and their information content.

[tobiasdiez]

Please move these merge metadata methods to the database merger class. There you can also add a merge method operating on BibDatabaseContext objects, to have an easy way to merge two databases including all their metadata.

[tobiasdiez]

Please also include tests for the merged groups.

[DominikVoigt]

Thanks for your comments :). I implemented the requested changes!