Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Checksum truncation, click to copy + file metadata fragment code consolidation #7312

Merged
merged 41 commits into from
Feb 3, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
41 commits
Select commit Hold shift + click to select a range
2903b08
Consolidated file metadata on dataset pg by using file-info-fragment …
mheppler Oct 7, 2020
f094b97
Fixed download count error on file-info-fragment, as well as cleaned …
mheppler Oct 8, 2020
56dc4c7
Removed additional editMode render logic not needed in file table on …
mheppler Oct 8, 2020
44722f5
Removed commented out isDuplicate render logic from checksum in datas…
mheppler Oct 9, 2020
9eb7f9a
Additional render logic clean up on file-info-fragment [ref #7081]
mheppler Oct 9, 2020
81384ba
Added checksum truncation function WIP [ref #7081 #6685 #5210]
mheppler Oct 15, 2020
aefcfad
Merge branch '7081-kebab-file-options-icon-btn' into 7081-file-metada…
mheppler Oct 18, 2020
da6e364
Merge branch 'develop' into 7081-file-metadata-include
mheppler Dec 17, 2020
3b90d1c
Commented out code in file table include to clean up UI [ref #6685 #5…
mheppler Dec 17, 2020
35982b7
Checksum truncate and click to copy added to file table on dataset pg…
mheppler Jan 5, 2021
b9052ce
Merge branch 'develop' into 7081-file-metadata-include
mheppler Jan 5, 2021
3ea71dd
Checksum/unf truncation, click to copy, for dataset pg and dataverse …
mheppler Jan 8, 2021
d039784
Applied truncation to UNF, moved tabular metadata labels to bundle [r…
mheppler Jan 11, 2021
d16b6ee
This is the reason search cards were not fully initialized for tabula…
landreev Jan 11, 2021
1b27ec3
Separate unf display from variables + observations on dataverse pg re…
mheppler Jan 11, 2021
16a3efa
Overwrite css from primefaces component.css for overflow property to …
mheppler Jan 11, 2021
aae9925
Fixed float UI issue in file table on dataset pg [ref #5210, #6685]
mheppler Jan 11, 2021
d85806e
Merge branch 'develop' into 7081-file-metadata-include
mheppler Jan 11, 2021
6b4fb24
Undo on bug testing revision mistakenly left in java function [ref #5…
mheppler Jan 12, 2021
2ee739a
File table responsive clean up, added file date labels [ref #5210, #6…
mheppler Jan 19, 2021
23e8f5c
Merge branch 'develop' into 7081-file-metadata-include
mheppler Jan 19, 2021
c7f7782
Updated metadata labels in dataset version differences popup, added t…
mheppler Jan 20, 2021
0706720
Mistakenly committed dev workaround, removed emuns [ref #5210, #6685]
mheppler Jan 21, 2021
3587fe9
Hide file metadata download count in file tables if dataset unpublish…
mheppler Jan 21, 2021
dfa367e
Removed parantesis from Tags label [ref #5210]
mheppler Jan 21, 2021
0a14b86
Merge branch 'develop' into 7081-file-metadata-include
mheppler Jan 25, 2021
f46a87f
Removed unnecessary editMode render logic, refactored other logic [re…
mheppler Jan 26, 2021
191bdfc
Code clean up and comments to doc renders and params for file-info [r…
mheppler Jan 26, 2021
5961033
Undoing mistakenly committed bug workaround hack for notifications gi…
mheppler Jan 26, 2021
c0dc967
Update file-info-fragment.xhtml
scolapasta Jan 26, 2021
0ebc23f
Update dataset.xhtml
scolapasta Jan 26, 2021
714b288
Update filesFragment.xhtml
scolapasta Jan 26, 2021
7f97d01
Update dataset.xhtml
scolapasta Jan 26, 2021
63eb5bd
Update DatasetPage.java
scolapasta Jan 26, 2021
0d08717
Merge branch 'develop' into 7081-file-metadata-include
mheppler Jan 27, 2021
814c005
Cleaned up copy to clipboard code for checksums and tokens [ref #6039…
mheppler Jan 27, 2021
89023ba
Fix checksum flashing and rebinding issues on download, search [ref #…
mheppler Jan 29, 2021
85755ba
Fix checksum flashing and rebinding issues on download [ref #7081]
mheppler Jan 29, 2021
a6d46bb
Added copy to clipboard success ok icon for checksums [ref #7081]
mheppler Feb 2, 2021
e17300f
Merge branch 'develop' into 7081-file-metadata-include
mheppler Feb 2, 2021
e9e99b4
Fixed double click issue on click to copy [ref #7081, #6685, #5210]
mheppler Feb 3, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -497,7 +497,7 @@ public DataFile findCheapAndEasy(Long id) {
// If content type indicates it's tabular data, spend 2 extra queries
// looking up the data table and tabular tags objects:

if (MIME_TYPE_TSV.equalsIgnoreCase(contentType)) {
if (MIME_TYPE_TSV.equalsIgnoreCase(contentType) || MIME_TYPE_TSV_ALT.equalsIgnoreCase(contentType)) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If I understand correctly (e.g. from the code links below), MIME_TYPE_TSV_ALT should be the only mimetype that will have variable metadata. So - can you also drop the first MIME_TYPE_TSV in this if statement?

Ingested tabular file mimetype set to MIME_TYPE_TSV_ALT:

dataFile.setContentType(FileUtil.MIME_TYPE_INGESTED_FILE);

Incoming files of type MIME_TYPE_TSV_ALT changed to MIME_TYPE_TSV

} else if (FileUtil.MIME_TYPE_INGESTED_FILE.equals(dataFile.getContentType())) {
// Make sure no *uningested* tab-delimited files are saved with the type "text/tab-separated-values"!
// "text/tsv" should be used instead:
dataFile.setContentType(FileUtil.MIME_TYPE_TSV);

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought about it, but I believe it is entirely possible for some legacy installations to still have some ingested tabular files with the other mime types. So I figured I would leave it in place to be safe. The worst possible drawback here is 1 wasted custom query, if this not an ingested tab. file.
(We make sure NOT to rely on the mime type, for identifying tabular data files elsewhere, and instead check for the presence of the DataTable object; but this is one situation where we have to - because we only have the search result, but not the entity).

Object[] dtResult;
try {
dtResult = (Object[]) em.createNativeQuery("SELECT ID, UNF, CASEQUANTITY, VARQUANTITY, ORIGINALFILEFORMAT, ORIGINALFILESIZE FROM dataTable WHERE DATAFILE_ID = " + id).getSingleResult();
Expand Down
19 changes: 1 addition & 18 deletions src/main/java/edu/harvard/iq/dataverse/DatasetPage.java
Original file line number Diff line number Diff line change
Expand Up @@ -1310,15 +1310,6 @@ public boolean isSuperUser(){
}
return false;
}
/*
TODO/OPTIMIZATION: This is still costing us N SELECT FROM GuestbookResponse queries,
where N is the number of files. This could of course be replaced by a query that'll
look up all N at once... Not sure if it's worth it; especially now that N
will always be 10, for the initial page load. -- L.A. 4.2.1
*/
public Long getGuestbookResponseCount(FileMetadata fileMetadata) {
return guestbookResponseService.getCountGuestbookResponsesByDataFileId(fileMetadata.getDataFile().getId());
}
/**
* Check Dataset related permissions
*
Expand Down Expand Up @@ -5107,15 +5098,7 @@ public String getPrivateUrlLink(PrivateUrl privateUrl) {
}


public FileDownloadHelper getFileDownloadHelper() {
return fileDownloadHelper;
}

public void setFileDownloadHelper(FileDownloadHelper fileDownloadHelper) {
this.fileDownloadHelper = fileDownloadHelper;
}


// todo: we should be able to remove - this is passed in the html pages to other fragments, but they could just access this service bean directly.
public FileDownloadServiceBean getFileDownloadService() {
return fileDownloadService;
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,6 @@ public final class DatasetVersionDifference {
private List<String[]> changedTermsAccess = new ArrayList<>();
private List<Object[]> summaryDataForNote = new ArrayList<>();
private List<Object[]> blockDataForNote = new ArrayList<>();
String noFileDifferencesFoundLabel = "";

private VariableMetadataUtil variableMetadataUtil;

Expand Down Expand Up @@ -104,7 +103,6 @@ public DatasetVersionDifference(DatasetVersion newVersion, DatasetVersion origin
addToSummary(null, dsfn);
}
}


// TODO: ?
// It looks like we are going through the filemetadatas in both versions,
Expand Down Expand Up @@ -493,7 +491,6 @@ private void addToSummary(DatasetField dsfo, DatasetField dsfn) {
}

private void updateBlockSummary(DatasetField dsf, int added, int deleted, int changed) {

boolean addedToAll = false;
for (Object[] blockList : blockDataForNote) {
DatasetField dsft = (DatasetField) blockList[0];
Expand All @@ -512,7 +509,6 @@ private void updateBlockSummary(DatasetField dsf, int added, int deleted, int ch
newArray[3] = changed;
blockDataForNote.add(newArray);
}

}

private void addToNoteSummary(DatasetField dsfo, int added, int deleted, int changed) {
Expand Down Expand Up @@ -552,8 +548,6 @@ private boolean compareVarGroup(FileMetadata fmdo, FileMetadata fmdn) {
}
}



private boolean compareFileMetadatas(FileMetadata fmdo, FileMetadata fmdn) {

if (!StringUtils.equals(fmdo.getDescription(), fmdn.getDescription())) {
Expand Down Expand Up @@ -786,7 +780,6 @@ private void initDatasetFilesDifferencesList() {
datasetFilesDiffList = new ArrayList<>();
datasetFilesReplacementList = new ArrayList <>();


// Study Files themselves are version-less;
// In other words, 2 different versions can have different sets of
// study files, but the files themselves don't have versions.
Expand All @@ -798,13 +791,6 @@ private void initDatasetFilesDifferencesList() {
// same study file, the file metadatas ARE version-specific, so some of
// the fields there (filename, etc.) may be different. If this is the
// case, we want to display these differences as well.

if (originalVersion.getFileMetadatas().isEmpty() && newVersion.getFileMetadatas().isEmpty()) {
noFileDifferencesFoundLabel = "No data files in either version of the study";
return;
}



int i = 0;
int j = 0;
Expand All @@ -825,7 +811,6 @@ private void initDatasetFilesDifferencesList() {

if (!replacedFiles.isEmpty()) {


replacedFiles.stream().map((replacedPair) -> {
FileMetadata replacedFile = replacedPair[0];
FileMetadata newFile = replacedPair[1];
Expand All @@ -851,7 +836,6 @@ private void initDatasetFilesDifferencesList() {

}


Collections.sort(fileMetadatasOriginal, (FileMetadata l1, FileMetadata l2) -> {
FileMetadata fm3 = l1; //(DatasetField[]) l1.get(0);
FileMetadata fm4 = l2;
Expand All @@ -871,7 +855,6 @@ private void initDatasetFilesDifferencesList() {
// added file. Since we are only doing this for the purposes of generating
// version differences, this should be OK.
// -- L.A. Aug. 2014


Collections.sort(fileMetadatasNew, (FileMetadata l1, FileMetadata l2) -> {
FileMetadata fm3 = l1; //(DatasetField[]) l1.get(0);
Expand Down Expand Up @@ -964,14 +947,6 @@ private void initDatasetFilesDifferencesList() {

j++;
}





if (datasetFilesDiffList.isEmpty()) {
noFileDifferencesFoundLabel = "These study versions have identical sets of data files";
}
}

private boolean fileMetadataIsDifferent(FileMetadata fm1, FileMetadata fm2) {
Expand Down Expand Up @@ -1080,12 +1055,8 @@ private boolean fileMetadataIsDifferent(FileMetadata fm1, FileMetadata fm2) {
return true;
}

//File restrictions

value1 = fm1.isRestricted() ? "Restricted" : "Not Restricted";
value2 = fm2.isRestricted() ? "Restricted" : "Not Restricted";

return !value1.equals(value2);
// File restrictions
return fm1.isRestricted() != fm2.isRestricted();
}

private datasetFileDifferenceItem selectFileMetadataDiffs(FileMetadata fm1, FileMetadata fm2) {
Expand All @@ -1109,7 +1080,7 @@ private datasetFileDifferenceItem selectFileMetadataDiffs(FileMetadata fm1, File
}

fdi.setFileProvFree1(fm1.getProvFreeForm());
fdi.setFileRest1(fm1.isRestricted() ? "Restricted" : "Not Restricted");
fdi.setFileRest1(fm1.isRestricted() ? BundleUtil.getStringFromBundle("restricted") : BundleUtil.getStringFromBundle("public"));
fdi.setFile2Empty(true);

} else if (fm1 == null) {
Expand All @@ -1125,7 +1096,7 @@ private datasetFileDifferenceItem selectFileMetadataDiffs(FileMetadata fm1, File
fdi.setFileCat2(fm2.getCategoriesByName().toString());
}
fdi.setFileProvFree2(fm2.getProvFreeForm());
fdi.setFileRest2(fm2.isRestricted() ? "Restricted" : "Not Restricted");
fdi.setFileRest2(fm2.isRestricted() ? BundleUtil.getStringFromBundle("restricted") : BundleUtil.getStringFromBundle("public"));
} else {
// Both are non-null metadata objects.
// We simply go through the 5 metadata fields, if any are
Expand Down Expand Up @@ -1206,11 +1177,9 @@ private datasetFileDifferenceItem selectFileMetadataDiffs(FileMetadata fm1, File
}

// file restricted:
value1 = fm1.isRestricted() ? "Restricted" : "Not Restricted";
value2 = fm2.isRestricted() ? "Restricted" : "Not Restricted";
if (!value1.equals(value2)) {
fdi.setFileRest1(value1);
fdi.setFileRest2(value2);
if (fm1.isRestricted() != fm2.isRestricted() ) {
fdi.setFileRest1(fm1.isRestricted() ? BundleUtil.getStringFromBundle("restricted") : BundleUtil.getStringFromBundle("public"));
fdi.setFileRest2(fm2.isRestricted() ? BundleUtil.getStringFromBundle("restricted") : BundleUtil.getStringFromBundle("public"));
}
}
return fdi;
Expand Down Expand Up @@ -1314,7 +1283,7 @@ public String getEditSummaryForLog() {
}

if (item.fileRest1 != null || item.fileRest2 != null) {
itemDiff += System.lineSeparator() + " " + BundleUtil.getStringFromBundle("file.viewDiffDialog.restricted") + ": ";
itemDiff += System.lineSeparator() + " " + BundleUtil.getStringFromBundle("file.viewDiffDialog.fileAccess") + ": ";
itemDiff += item.fileRest1 != null ? item.fileRest1 : BundleUtil.getStringFromBundle("file.viewDiffDialog.notAvailable");
itemDiff += " : ";
itemDiff += item.fileRest2 != null ? item.fileRest2 : BundleUtil.getStringFromBundle("file.viewDiffDialog.notAvailable") + " ";
Expand Down Expand Up @@ -1356,7 +1325,7 @@ public String getEditSummaryForLog() {
itemDiff += item.fdi.fileProvFree1 != null ? item.fdi.fileProvFree1 : BundleUtil.getStringFromBundle("file.viewDiffDialog.notAvailable");
itemDiff += " : ";
itemDiff += item.fdi.fileProvFree2 != null ? item.fdi.fileProvFree2 : BundleUtil.getStringFromBundle("file.viewDiffDialog.notAvailable") + " ";
itemDiff += System.lineSeparator() + " " + BundleUtil.getStringFromBundle("file.viewDiffDialog.restricted") + ": ";
itemDiff += System.lineSeparator() + " " + BundleUtil.getStringFromBundle("file.viewDiffDialog.fileAccess") + ": ";
itemDiff += item.fdi.fileRest1 != null ? item.fdi.fileRest1 : BundleUtil.getStringFromBundle("file.viewDiffDialog.notAvailable");
itemDiff += " : ";
itemDiff += item.fdi.fileRest2 != null ? item.fdi.fileRest2 : BundleUtil.getStringFromBundle("file.viewDiffDialog.notAvailable") + " ";
Expand All @@ -1377,12 +1346,11 @@ public String getEditSummaryForLog() {
retVal +=termsOfUseDiff;
}


return retVal;
}


public class DifferenceSummaryGroup{
public class DifferenceSummaryGroup {

private String displayName;
private String type;
Expand Down Expand Up @@ -1411,8 +1379,6 @@ public List<DifferenceSummaryItem> getDifferenceSummaryItems() {
public void setDifferenceSummaryItems(List<DifferenceSummaryItem> differenceSummaryItems) {
this.differenceSummaryItems = differenceSummaryItems;
}


}

public class DifferenceSummaryItem {
Expand Down Expand Up @@ -1470,9 +1436,6 @@ public boolean isMultiple() {
public void setMultiple(boolean multiple) {
this.multiple = multiple;
}



}

public class datasetReplaceFileItem {
Expand Down Expand Up @@ -1733,7 +1696,6 @@ public String getFileChecksumValue() {
public void setFileChecksumValue(String fileChecksumValue) {
this.fileChecksumValue = fileChecksumValue;
}

}

public List<datasetFileDifferenceItem> getDatasetFilesDiffList() {
Expand All @@ -1744,11 +1706,4 @@ public void setDatasetFilesDiffList(List<datasetFileDifferenceItem> datasetFiles
this.datasetFilesDiffList = datasetFilesDiffList;
}

public String getNoFileDifferencesFoundLabel() {
return noFileDifferencesFoundLabel;
}

public void setNoFileDifferencesFoundLabel(String noFileDifferencesFoundLabel) {
this.noFileDifferencesFoundLabel = noFileDifferencesFoundLabel;
}
}
14 changes: 0 additions & 14 deletions src/main/java/edu/harvard/iq/dataverse/EditDatafilesPage.java
Original file line number Diff line number Diff line change
Expand Up @@ -2453,20 +2453,6 @@ public boolean isLocked() {
return false;
}

public boolean isThumbnailAvailable(FileMetadata fileMetadata) {
// new and optimized logic:
// - check download permission here (should be cached - so it's free!)
// - only then ask the file service if the thumbnail is available/exists.
// the service itself no longer checks download permissions.
if (!fileDownloadHelper.canDownloadFile(fileMetadata)) {
return false;
}

return datafileService.isThumbnailAvailable(fileMetadata.getDataFile());
}
scolapasta marked this conversation as resolved.
Show resolved Hide resolved



private Boolean lockedFromEditsVar;

public boolean isLockedFromEdits() {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -1201,53 +1201,52 @@ public SearchException getSearchException() {
}

public String tabularDataDisplayInfo(DataFile datafile) {
String ret = "";
String tabInfo = "";

if (datafile == null) {
return null;
return "";
}

if (datafile.isTabularData() && datafile.getDataTable() != null) {
DataTable datatable = datafile.getDataTable();
String unf = datatable.getUnf();
Long varNumber = datatable.getVarQuantity();
Long obsNumber = datatable.getCaseQuantity();
if (varNumber != null && varNumber.intValue() != 0) {
ret = ret.concat(varNumber + " Variables");
tabInfo = tabInfo.concat(varNumber + " " + BundleUtil.getStringFromBundle("file.metaData.dataFile.dataTab.variables"));
if (obsNumber != null && obsNumber.intValue() != 0) {
ret = ret.concat(", " + obsNumber + " Observations");
tabInfo = tabInfo.concat(", " + obsNumber + " " + BundleUtil.getStringFromBundle("file.metaData.dataFile.dataTab.observations"));
}
ret = ret.concat(" - ");
}
if (unf != null && !unf.equals("")) {
ret = ret.concat("UNF: " + unf);
}
}

return ret;
return tabInfo;
}

public String tabularDataUnfDisplay(DataFile datafile) {
String tabUnf = "";

public String dataFileSizeDisplay(DataFile datafile) {
if (datafile == null) {
return "";
}

return datafile.getFriendlySize();
if (datafile.isTabularData() && datafile.getDataTable() != null) {
DataTable datatable = datafile.getDataTable();
String unf = datatable.getUnf();
if (unf != null && !unf.equals("")) {
tabUnf = tabUnf.concat(unf);
}
}

return tabUnf;
}

public String dataFileChecksumDisplay(DataFile datafile) {
public String dataFileSizeDisplay(DataFile datafile) {
if (datafile == null) {
return "";
}

if (datafile.getChecksumValue() != null && !StringUtils.isEmpty(datafile.getChecksumValue())) {
if (datafile.getChecksumType() != null) {
return " " + datafile.getChecksumType() + ": " + datafile.getChecksumValue() + " ";
}
}
return datafile.getFriendlySize();

return "";
}

public void setDisplayCardValues() {
Expand Down
Loading