Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ts/2509 task update entity similarity api #2791

Merged
merged 8 commits into from
Feb 22, 2024
Merged

Ts/2509 task update entity similarity api #2791

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
c53bc52
updating type
Tom-Szendrey Feb 5, 2024
0e49617
Merge branch 'main' of https://github.com/DARPA-ASKEM/TERArium into T…
Tom-Szendrey Feb 16, 2024
40853f0
Merge branch 'main' of https://github.com/DARPA-ASKEM/TERArium into T…
Tom-Szendrey Feb 21, 2024
23e649f
updating dataset format parser
Tom-Szendrey Feb 21, 2024
8d9ea90
Merge branch 'main' of https://github.com/DARPA-ASKEM/TERArium into T…
Tom-Szendrey Feb 21, 2024
c21bc73
getting dataoptions grounding better
Tom-Szendrey Feb 22, 2024
419d271
Merge branch 'main' of https://github.com/DARPA-ASKEM/TERArium into T…
Tom-Szendrey Feb 22, 2024
4b513f2
still need to make this 1:1 i know
Tom-Szendrey Feb 22, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 18 additions & 9 deletions packages/client/hmi-client/src/services/calibrate-workflow.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -124,7 +124,7 @@ export const autoCalibrationMapping = async (
const result = [] as CalibrateMap[];
const allModelGroundings: string[] = [];
const allDataGroundings: string[] = [];
const acceptableDistance = 0.7;
const acceptableDistance = 0.5;
// Get all model groundings
modelOptions.forEach((state) => {
if (state.grounding?.identifiers) {
Expand All @@ -135,31 +135,40 @@ export const autoCalibrationMapping = async (
});
// Get all data column groundings
datasetOptions.forEach((col) => {
const dataGroundingList = Object.keys(col.metadata?.groundings?.identifiers);
dataGroundingList.forEach((ele) => allDataGroundings.push(ele));
if (col.metadata?.groundings?.identifiers) {
Object.entries(col.metadata?.groundings?.identifiers)
.map((ele) => ele.join(':'))
.forEach((ele) => allDataGroundings.push(ele));
}
});

// take out duplicates:
const distinctModelGroundings = [...new Set(allModelGroundings)];
const distinctDataGroundings = [...new Set(allDataGroundings)];
const allSimilarity = await getEntitySimilarity(distinctModelGroundings, distinctDataGroundings);
if (!allSimilarity) return result;

const filteredSim = allSimilarity.filter((ele) => ele.distance < acceptableDistance);
// normalized cosine similarities
Tom-Szendrey marked this conversation as resolved.
Show resolved Hide resolved
const filteredSim = allSimilarity.filter((ele) => ele.similarity > acceptableDistance);
filteredSim.forEach((sim) => {
// Find all states assosiated with this sim
const validStates = modelOptions.filter((state) => {
if (state.grounding?.identifiers) {
const modelTemp = Object.entries(state.grounding?.identifiers);
const modelGroundingList = modelTemp.map((ele) => ele.join(':'));
const modelGroundingList = Object.entries(state.grounding?.identifiers).map((ele) =>
ele.join(':')
);
return modelGroundingList.includes(sim.source);
}
return false;
});
// Find all columns assosiated with this sim
const validCols = datasetOptions.filter((col) => {
const dataGroundingList = Object.keys(col.metadata?.groundings?.identifiers);
return dataGroundingList.includes(sim.target);
if (col.metadata?.groundings?.identifiers) {
const dataGroundingList = Object.entries(col.metadata?.groundings?.identifiers).map((ele) =>
ele.join(':')
);
return dataGroundingList.includes(sim.target);
}
return false;
});
// For all states and columns that have short distances throw them into results
validStates.forEach((state) => {
Expand Down
2 changes: 1 addition & 1 deletion packages/client/hmi-client/src/types/Types.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -677,7 +677,7 @@ export interface DKG {
export interface EntitySimilarityResult {
source: string;
target: string;
distance: number;
similarity: number;
}

export interface PermissionGroup {
Expand Down
2 changes: 1 addition & 1 deletion ...c/main/java/software/uncharted/terarium/hmiserver/models/mira/EntitySimilarityResult.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,5 +10,5 @@
public class EntitySimilarityResult {
private String source;
private String target;
private Float distance;
private Float similarity;
}
Loading