Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

On Study View handle larger number of resource files #5034

Merged
merged 10 commits into from
Nov 1, 2024
Merged

On Study View handle larger number of resource files #5034

Changes from 5 commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
4c2f184
Modify FilesAndLinks table to use the new resource-data-all endpoint.
hweej Oct 27, 2024
e6da0c7
Add filtering logic based off studyview sample selection
hweej Oct 28, 2024
cf69080
Merge branch 'master' into reduce-api-calls-files-and-links-tab
hweej Oct 28, 2024
4ef97bf
Add missing typescript types
hweej Oct 28, 2024
a417dac
Merge branch 'reduce-api-calls-files-and-links-tab' of github.com:hwe…
hweej Oct 28, 2024
e029bb0
Remove clinicalData request, rename variables, add relevant comments on
hweej Oct 29, 2024
c4f1df8
Update linear array lookup to a key Map lookup for optimization. Replace
hweej Oct 30, 2024
f8aff81
Exchange string for actual property for typesafety.
hweej Oct 30, 2024
07f04bb
Remove unnecessary Map instantiation
hweej Oct 31, 2024
42ea827
Add latest typescript api definition files
hweej Nov 1, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
67 changes: 59 additions & 8 deletions src/pages/studyView/resources/FilesAndLinks.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,20 @@ class FilesLinksTableComponent extends LazyMobXTable<{

const RECORD_LIMIT = 500;

function getResourceDataOfEntireStudy(studyClinicalData: {
[uniqueSampleKey: string]: ClinicalData[];
}) {
// Need a better way to get just the studyId
const studyId = Object.values(studyClinicalData)[0][0]['studyId'];
const allResources = internalClient.getAllStudyResourceDataInStudyPatientSampleUsingGET(
{
studyId: studyId,
projection: 'DETAILED',
}
);
return allResources;
}

function getResourceDataOfPatients(studyClinicalData: {
[uniqueSampleKey: string]: ClinicalData[];
}) {
Expand Down Expand Up @@ -97,7 +111,8 @@ function buildItemsAndResources(resourceData: {

async function fetchFilesLinksData(
filters: StudyViewFilter,
sampleIdResourceData: { [sampleId: string]: ResourceData[] },
selectedSamples: Array<any>,
// sampleIdResourceData: { [sampleId: string]: ResourceData[] },
searchTerm: string | undefined,
sortAttributeId: string | undefined,
sortDirection: 'asc' | 'desc' | undefined,
Expand All @@ -112,13 +127,47 @@ async function fetchFilesLinksData(
0
);

const resourcesForPatients = await getResourceDataOfPatients(
const selectedSamplesList = selectedSamples.map(item => item.sampleId);
hweej marked this conversation as resolved.
Show resolved Hide resolved
const selectedPatientsList = [
hweej marked this conversation as resolved.
Show resolved Hide resolved
...new Set(selectedSamples.map(item => item.patientId)),
];

const resourcesForEntireStudy = await getResourceDataOfEntireStudy(
hweej marked this conversation as resolved.
Show resolved Hide resolved
studyClinicalDataResponse.data
);
const resourcesForPatientsAndSamples: { [key: string]: ResourceData[] } = {
...sampleIdResourceData,
...resourcesForPatients,
};

const resourcesForPatientsAndSamples = resourcesForEntireStudy
hweej marked this conversation as resolved.
Show resolved Hide resolved
.filter(
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

one more thing here: this is operation is Rsamples+Rpatients. could easily get to billion. easily optimized by putting selectedSamples in a map so you can use IN operator.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have replaced this with Maps instead of arrays to remove the linear lookup. I don't think you can use the IN operator with Maps so we're opting for the .has('key') expression instead.

(resource: ResourceData) =>
selectedSamplesList.includes(resource.sampleId) ||
selectedPatientsList.includes(resource.patientId)
)
.reduce(
(
idMap: { [key: string]: ResourceData[] },
resource: ResourceData
) => {
const { resourceType } = resource.resourceDefinition;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this whole operation would be more clearly handled by lodash's keyBy function. we like to use these kind of named helpers because they make the operation more explicit.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm, I couldn't make lodash keyBy function to work the way I wanted it to for this use case. In python this would be a case for collections.defaultDict. But I couldn't make this work where I could create new arrays and append to them.

LMK if you want me to work on this part some more.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@hweej sorry, i mean _.groupBy

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have replaced these lines with a single lodash groupBy('patientId'). Thanks @alisman !

const { patientId, sampleId } = resource;

if (resourceType == 'PATIENT') {
if (idMap.hasOwnProperty(patientId)) {
idMap[patientId].push(resource);
} else {
idMap[patientId] = [resource];
}
} else if (resourceType == 'SAMPLE') {
if (idMap.hasOwnProperty(sampleId)) {
idMap[sampleId].push(resource);
} else {
idMap[sampleId] = [resource];
}
}

return idMap;
},
{}
);

// we create objects with the necessary properties for each resource
// calculate the total number of resources per patient.
Expand Down Expand Up @@ -197,7 +246,7 @@ export class FilesAndLinks extends React.Component<IFilesLinksTable, {}> {
await: () => [
this.props.store.selectedSamples,
this.props.store.resourceDefinitions,
this.props.store.sampleResourceData,
// this.props.store.sampleResourceData,
],
onError: () => {},
invoke: async () => {
Expand All @@ -207,12 +256,14 @@ export class FilesAndLinks extends React.Component<IFilesLinksTable, {}> {

const resources = await fetchFilesLinksData(
this.props.store.filters,
this.props.store.sampleResourceData.result!,
this.props.store.selectedSamples.result,
// this.props.store.sampleResourceData.result!,
this.searchTerm,
'patientId',
'asc',
RECORD_LIMIT
);

return Promise.resolve(resources);
},
});
Expand Down
Loading