Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

2976 feat search for datasets esgf era5 terarium #3033

Merged
merged 60 commits into from
Mar 19, 2024
Merged
Show file tree
Hide file tree
Changes from 48 commits
Commits
Show all changes
60 commits
Select commit Hold shift + click to select a range
d17a77a
feat [hmi-client]: search for Datasets (ESGF, ERA5, Terarium)
dvince2 Mar 11, 2024
eeb2bc1
feat [hmi-client]: search for Datasets (ESGF, ERA5, Terarium)
dvince2 Mar 12, 2024
ac3f68d
Merge remote-tracking branch 'origin/main' into 2976-feat-search-for-…
dvince2 Mar 12, 2024
5f9a909
enable scheduling
bigglesandginger Mar 13, 2024
09c889f
initial wip check status echeduled task
bigglesandginger Mar 13, 2024
bd02643
schedule looking up status of long running tasks
bigglesandginger Mar 13, 2024
15fc6f0
pointless spelling correction
bigglesandginger Mar 13, 2024
dd21a0a
Climate Data Task object for storage in database
bigglesandginger Mar 13, 2024
139d79f
store ClimateData id in database for status checking
bigglesandginger Mar 13, 2024
a8a04dd
refactoring with storing ideas
bigglesandginger Mar 13, 2024
583ed05
add job result and error
bigglesandginger Mar 13, 2024
f419572
more camel-case friendly
bigglesandginger Mar 13, 2024
420d3ba
more camel-case friendly
bigglesandginger Mar 13, 2024
13647ec
feat [hmi-client]: search for Datasets (ESGF, ERA5, Terarium)
dvince2 Mar 13, 2024
b25cc85
Merge branch '2976-feat-search-for-datasets-esgf-era5-terarium' of gi…
bigglesandginger Mar 13, 2024
e1e1ac0
fix merge
bigglesandginger Mar 13, 2024
2d2957a
refactor datasetId to esfgId
bigglesandginger Mar 13, 2024
9e640eb
use esgfId
bigglesandginger Mar 13, 2024
ee7ded7
store preview image in S3
bigglesandginger Mar 13, 2024
5b7439c
added more fleshed out idea of the Preview response workflow
bigglesandginger Mar 14, 2024
27cf5ce
feat [hmi-client]: search for Datasets (ESGF, ERA5, Terarium)
dvince2 Mar 14, 2024
3049fe6
add constructors and png url
bigglesandginger Mar 14, 2024
cdb125c
feat [hmi-client]: search for Datasets (ESGF, ERA5, Terarium)
dvince2 Mar 14, 2024
8e285a2
deal with mulitple preview images
bigglesandginger Mar 14, 2024
26196cb
Merge branch '2976-feat-search-for-datasets-esgf-era5-terarium' of gi…
bigglesandginger Mar 14, 2024
2840f68
Merge remote-tracking branch 'origin/main' into 2976-feat-search-for-…
dvince2 Mar 14, 2024
1d134e5
deal with fail to extract png error
bigglesandginger Mar 14, 2024
704dba3
Merge remote-tracking branch 'origin/main' into 2976-feat-search-for-…
dvince2 Mar 14, 2024
88df7b1
feat [hmi-client]: search for Datasets (ESGF, ERA5, Terarium)
dvince2 Mar 14, 2024
ab945e2
subset generation
bigglesandginger Mar 14, 2024
1c6c4bb
remove useless comment
bigglesandginger Mar 14, 2024
5c1fec7
feat [hmi-client]: search for Datasets (ESGF, ERA5, Terarium)
dvince2 Mar 14, 2024
4ed76d2
functioning subsets
bigglesandginger Mar 14, 2024
d96ff22
correct null check
bigglesandginger Mar 14, 2024
1c9a80b
Merge branch '2976-feat-search-for-datasets-esgf-era5-terarium' of gi…
bigglesandginger Mar 14, 2024
09fa67a
dataset_id
bigglesandginger Mar 14, 2024
89f1eaa
Merge remote-tracking branch 'origin/main' into 2976-feat-search-for-…
dvince2 Mar 15, 2024
7522133
allow returning non CSVs
bigglesandginger Mar 15, 2024
df9ae78
Subset ideas finilizing
bigglesandginger Mar 15, 2024
8ea430f
Merge branch '2976-feat-search-for-datasets-esgf-era5-terarium' of gi…
bigglesandginger Mar 15, 2024
614a21f
change dataset/preview to be NOT_IMPLEMENTED
bigglesandginger Mar 18, 2024
b19f8a5
documentation
bigglesandginger Mar 18, 2024
e2b2691
feat [hmi-client]: search for Datasets (ESGF, ERA5, Terarium)
dvince2 Mar 18, 2024
752aba0
feat [hmi-client]: search for Datasets (ESGF, ERA5, Terarium)
dvince2 Mar 18, 2024
396efd8
Merge remote-tracking branch 'origin/main' into 2976-feat-search-for-…
dvince2 Mar 18, 2024
9715ab6
feat [hmi-client]: search for Datasets (ESGF, ERA5, Terarium)
dvince2 Mar 18, 2024
faf43da
feat [hmi-client]: search for Datasets (ESGF, ERA5, Terarium)
dvince2 Mar 18, 2024
9b90031
Merge branch 'main' into 2976-feat-search-for-datasets-esgf-era5-tera…
dvince2 Mar 18, 2024
41e13a8
Merge branch 'main' into 2976-feat-search-for-datasets-esgf-era5-tera…
dvince2 Mar 18, 2024
f463957
Merge remote-tracking branch 'origin/main' into 2976-feat-search-for-…
dvince2 Mar 19, 2024
425f967
feat [hmi-client]: search for Datasets (ESGF, ERA5, Terarium)
dvince2 Mar 19, 2024
5d7d761
Merge remote-tracking branch 'origin/main' into 2976-feat-search-for-…
dvince2 Mar 19, 2024
869e6ad
feat [hmi-client]: search for Datasets (ESGF, ERA5, Terarium)
dvince2 Mar 19, 2024
f580457
feat [hmi-client]: search for Datasets (ESGF, ERA5, Terarium)
dvince2 Mar 19, 2024
c5d6279
Merge branch 'main' into 2976-feat-search-for-datasets-esgf-era5-tera…
dvince2 Mar 19, 2024
aa3dac2
feat [hmi-client]: search for Datasets (ESGF, ERA5, Terarium)
dvince2 Mar 19, 2024
c56216e
Keeping the original error code and throwing response exceptions on.
dvince2 Mar 19, 2024
8951506
Merge remote-tracking branch 'origin/main' into 2976-feat-search-for-…
dvince2 Mar 19, 2024
747cfda
Merge branch 'main' into 2976-feat-search-for-datasets-esgf-era5-tera…
dvince2 Mar 19, 2024
78a8f19
Merge branch 'main' into 2976-feat-search-for-datasets-esgf-era5-tera…
dvince2 Mar 19, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -25,13 +25,16 @@
@enriched="fetchAsset"
/>
</AccordionTab>
<AccordionTab header="Column information">
<AccordionTab header="Column information" v-if="!dataset?.esgfId">
dvince2 marked this conversation as resolved.
Show resolved Hide resolved
<tera-dataset-overview-table
v-if="dataset"
:dataset="dataset"
@update-dataset="(dataset: Dataset) => emit('update-dataset', dataset)"
/>
</AccordionTab>
<AccordionTab header="Preview" v-if="dataset?.esgfId">
dvince2 marked this conversation as resolved.
Show resolved Hide resolved
<img :src="image" alt="" />
</AccordionTab>
</Accordion>
</template>

Expand All @@ -50,6 +53,7 @@ import TeraDatasetOverviewTable from './tera-dataset-overview-table.vue';

const props = defineProps<{
dataset: Dataset | null;
image?: string | undefined;
highlight?: string;
featureConfig?: FeatureConfig;
rawContent: CsvAsset | null;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -34,11 +34,12 @@
<tera-dataset-description
tabName="Description"
:dataset="dataset"
:image="image"
:raw-content="rawContent"
@update-dataset="(dataset: Dataset) => updateAndFetchDataset(dataset)"
/>
</section>
<section class="tab data-tab" tabName="Data">
<section class="tab data-tab" tabName="Data" v-if="rawContent">
<tera-dataset-datatable :rows="100" :raw-content="rawContent" />
</section>
</template>
Expand All @@ -48,7 +49,13 @@
import { onUpdated, PropType, Ref, ref, watch } from 'vue';
import * as textUtil from '@/utils/text';
import { cloneDeep, isString } from 'lodash';
import { downloadRawFile, getDataset, updateDataset } from '@/services/dataset';
import {
downloadRawFile,
getClimateDataset,
getClimateDatasetPreview,
getDataset,
updateDataset
} from '@/services/dataset';
import { AssetType, type CsvAsset, type Dataset, type DatasetColumn } from '@/types/Types';
import TeraDatasetDatatable from '@/components/dataset/tera-dataset-datatable.vue';
import TeraAsset from '@/components/asset/tera-asset.vue';
Expand All @@ -58,6 +65,7 @@ import InputText from 'primevue/inputtext';
import ContextMenu from 'primevue/contextmenu';
import Button from 'primevue/button';
import { logger } from '@/utils/logger';
import { DatasetSource } from '@/types/Dataset';
import TeraDatasetDescription from './tera-dataset-description.vue';
import { enrichDataset } from './utils';

Expand All @@ -79,6 +87,10 @@ const props = defineProps({
highlight: {
type: String,
default: null
},
datasetSource: {
type: String as PropType<DatasetSource>,
default: DatasetSource.TERARIUM
dvince2 marked this conversation as resolved.
Show resolved Hide resolved
}
});

Expand All @@ -89,9 +101,8 @@ const isRenamingDataset = ref(false);
const rawContent: Ref<CsvAsset | null> = ref(null);
const isDatasetLoading = ref(false);
const selectedTabIndex = ref(0);

const view = ref(DatasetView.DESCRIPTION);

const image: Ref<string | undefined> = ref(undefined);
dvince2 marked this conversation as resolved.
Show resolved Hide resolved
// Highlight strings based on props.highlight
function highlightSearchTerms(text: string | undefined): string {
if (!!props.highlight && !!text) {
Expand Down Expand Up @@ -163,17 +174,38 @@ async function updateAndFetchDataset(ds: Dataset) {
}

const fetchDataset = async () => {
const datasetTemp: Dataset | null = await getDataset(props.assetId);

if (datasetTemp) {
// We are assuming here there is only a single csv file. This may change in the future as the API allows for it.
rawContent.value = await downloadRawFile(props.assetId, datasetTemp?.fileNames?.[0] ?? '');
Object.entries(datasetTemp).forEach(([key, value]) => {
if (isString(value)) {
datasetTemp[key] = highlightSearchTerms(value);
switch (props.datasetSource) {
case DatasetSource.TERARIUM: {
const datasetTemp = await getDataset(props.assetId);
if (datasetTemp) {
if (datasetTemp.esgfId) {
image.value = await getClimateDatasetPreview(datasetTemp.esgfId);
rawContent.value = null;
} else {
// We are assuming here there is only a single csv file. This may change in the future as the API allows for it.
image.value = undefined;
rawContent.value = await downloadRawFile(
props.assetId,
datasetTemp?.fileNames?.[0] ?? ''
);
Object.entries(datasetTemp).forEach(([key, value]) => {
if (isString(value)) {
datasetTemp[key] = highlightSearchTerms(value);
}
});
}
dataset.value = enrichDataset(datasetTemp);
}
});
dataset.value = enrichDataset(datasetTemp);
break;
}
case DatasetSource.ESGF: {
dataset.value = await getClimateDataset(props.assetId);
if (dataset.value?.esgfId)
image.value = await getClimateDatasetPreview(dataset.value?.esgfId);
break;
}
default:
break;
}
};

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,17 @@
</span>
<tera-filter-bar :topic-options="topicOptions" @filter-changed="executeNewQuery" />
</div>
<div class="toggles" v-if="assetType === AssetType.Dataset">
<span>
<label class="mr-2">Source</label>
<Dropdown
v-model="chosenDatasetSource"
:options="datasetSourceOptions"
@change="executeNewQuery"
/>
</span>
<tera-filter-bar :topic-options="topicOptions" @filter-changed="executeNewQuery" />
</div>
</nav>
<tera-searchbar
ref="searchBarRef"
Expand Down Expand Up @@ -68,13 +79,14 @@
:source="chosenSource"
:resource-type="resourceType"
:search-term="searchTerm"
:dataset-source="chosenDatasetSource"
/>
</section>
</main>
</template>

<script setup lang="ts">
import { computed, onMounted, onUnmounted, ref, watch } from 'vue';
import { computed, onMounted, onUnmounted, Ref, ref, watch } from 'vue';
import TeraSliderPanel from '@/components/widgets/tera-slider-panel.vue';
import SelectButton from 'primevue/selectbutton';
import Dropdown from 'primevue/dropdown';
Expand All @@ -93,6 +105,7 @@ import {
FACET_FIELDS as XDD_FACET_FIELDS,
GITHUB_URL,
XDD_RESULT_DEFAULT_PAGE_SIZE,
XDDSearchParams,
YEAR
} from '@/types/XDD';
import useQueryStore from '@/stores/query';
Expand All @@ -106,6 +119,8 @@ import TeraFacetsPanel from '@/page/data-explorer/components/tera-facets-panel.v
import TeraSearchResultsList from '@/page/data-explorer/components/tera-search-results-list.vue';
import { AssetType, XDDFacetsItemResponse } from '@/types/Types';
import TeraSearchbar from '@/components/navbar/tera-searchbar.vue'; // import Chip from 'primevue/chip';
import { DatasetSearchParams, DatasetSource } from '@/types/Dataset';
import { ModelSearchParams } from '@/types/Model';
import { useSearchByExampleOptions } from './search-by-example';
import TeraFilterBar from './components/tera-filter-bar.vue';

Expand Down Expand Up @@ -171,6 +186,9 @@ const topicOptions = ref([
const sourceOptions = ref(['xDD', 'Terarium']);
const chosenSource = ref('xDD');

const datasetSourceOptions: Ref<DatasetSource[]> = ref(Object.values(DatasetSource));
const chosenDatasetSource: Ref<DatasetSource> = ref(DatasetSource.TERARIUM);

const sliderWidth = computed(() =>
isSliderFacetsOpen.value ? 'calc(50% - 120px)' : 'calc(50% - 20px)'
);
Expand Down Expand Up @@ -261,7 +279,10 @@ const executeSearch = async () => {
known_entities: 'url_extractions,askem_object'
},
model: {},
dataset: {}
[ResourceType.DATASET]: {
source: chosenDatasetSource.value,
topic: 'covid-19'
}
};

// handle the search-by-example for finding related documents, models, and/or datasets
Expand Down Expand Up @@ -298,12 +319,12 @@ const executeSearch = async () => {
assetType.value = AssetType.Dataset;
}
}
const searchParamsWithFacetFilters = cloneDeep(searchParams);
const searchParamsWithFacetFilters: SearchParameters = cloneDeep(searchParams);

//
// extend search parameters by converting facet filters into proper search parameters
//
const xddSearchParams = searchParamsWithFacetFilters?.[ResourceType.XDD] || {};
const xddSearchParams: XDDSearchParams = searchParamsWithFacetFilters?.[ResourceType.XDD] || {};
// transform facet filters into xdd search parameters
clientFilters.value.clauses.forEach((clause) => {
if (XDD_FACET_FIELDS.includes(clause.field)) {
Expand Down Expand Up @@ -334,17 +355,21 @@ const executeSearch = async () => {
}
});

let modelSearchParams;
let modelSearchParams: ModelSearchParams;
if (searchParamsWithFacetFilters?.[ResourceType.MODEL]?.filters) {
modelSearchParams = searchParamsWithFacetFilters[ResourceType.MODEL];
} else {
modelSearchParams = { filters: clientFilters.value };
}
let datasetSearchParams;
let datasetSearchParams: DatasetSearchParams;
if (searchParamsWithFacetFilters?.[ResourceType.DATASET]?.filters) {
datasetSearchParams = searchParamsWithFacetFilters[ResourceType.MODEL];
datasetSearchParams = searchParamsWithFacetFilters[ResourceType.DATASET];
} else {
datasetSearchParams = { filters: clientFilters.value };
datasetSearchParams = {
filters: clientFilters.value,
source: chosenDatasetSource.value,
topic: 'covid-19' // TODO - this should be dynamic
};
}

// update search parameters object
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@
:asset-id="previewItemId"
:highlight="searchTerm"
:feature-config="{ isPreview: true }"
:dataset-source="datasetSource"
@close-preview="closePreview"
/>
<tera-model
Expand All @@ -42,14 +43,16 @@
</template>

<script setup lang="ts">
import { PropType, computed, ref, watch } from 'vue';
import { ResultType, ResourceType } from '@/types/common';
import { computed, PropType, ref, watch } from 'vue';
import { ResourceType, ResultType } from '@/types/common';
import { isDocument } from '@/utils/data-util';
import TeraModel from '@/components/model/tera-model.vue';
import TeraDataset from '@/components/dataset/tera-dataset.vue';
import TeraSlider from '@/components/widgets/tera-slider.vue';
import TeraDocumentAsset from '@/components/documents/tera-document-asset.vue';
import TeraExternalPublication from '@/components/documents/tera-external-publication.vue';
import { DatasetSource } from '@/types/Dataset';
import { Dataset } from '@/types/Types';

const props = defineProps({
// slider props
Expand Down Expand Up @@ -80,6 +83,10 @@ const props = defineProps({
source: {
type: String,
default: 'xDD'
},
datasetSource: {
dvince2 marked this conversation as resolved.
Show resolved Hide resolved
type: String as PropType<DatasetSource>,
default: DatasetSource.TERARIUM
}
});

Expand All @@ -92,6 +99,13 @@ const previewItemId = computed(() => {
if (isDocument(previewItemState.value)) {
return previewItemState.value.gddId;
}
if (
previewItemResourceType.value === ResourceType.DATASET &&
props.datasetSource === DatasetSource.ESGF
) {
const dataset: Dataset = previewItemState.value as Dataset;
return dataset.esgfId as string;
}
return previewItemState.value.id as string;
});

Expand Down
Loading
Loading