[NL Interface] Making medium_ft the default #2920

jehangiramjad · 2023-07-11T21:06:29Z

In this PR, we have the following:

Making the medium_ft embeddings and the finetuned model to be the default.
Updating integration tests with medium_ft embeddings (and associated finetuned model)

Below is the manually checked report (for each integration test scenario) for the diffs between the old (small index embeddings) and new (medium_ft index embeddings) for what shows up on the frontend (it may be hard to parse the changes from the config jsons):

textbox_sample:

'family earnings in california':
- old: https://screenshot.googleplex.com/8PEgL9mhDvAqQGV
- new: https://screenshot.googleplex.com/AAGvbreEjgVKcoM
- same top chart
- [good] "individual income" drop out which is good

demo_feb2023:

'What are the projected temperature extremes across California',
- same top few charts
- Now we see more charts (Mean, Min, Max temperature, Heat Events etc show up as well).
  Not necessarily a good/bad thing. Bad is that the Mean/Min/Max temps are not projected.
'Where were the major fires in the last year',
- Same chart and data
'Tell me about Placer County',
- Same
'What were the most common jobs there',
- Same chart and data
'Which jobs have grown the most',
- Same top charts shown
'What are the most common health issues there',
- Same chart and data
'Which counties in california have the highest levels of blood pressure',
- old: https://screenshot.googleplex.com/9jP6qakxXWqnEpt
- new: https://screenshot.googleplex.com/8ChbDUcK6dfktLB
- Same top chart
- New one shows one more chart (which is also kind of relevant)
- New debug matched TVs: https://screenshot.googleplex.com/BzUgo8dFHFyAnpW
- Note: after the previous medium_ft, some fine-tuning was done to disassociate "blood pressure" with "diseases of blood" and those SVs have all dropped back out. The two SVs now are both squarely related to blood pressure. If this is not desired, we can also try to fine tune "medication" away from "blood pressure" but that feels to be overfitting.
'Which counties in the USA have the highest levels of blood pressure',
- Same as above (same top chart but one more charts now being shown which isn't terrible).
'How does this correlate with income',
- Same top correlation chart.
- Due to the second chart in steps above, there are more correlation charts shown in for medium_ft but with the limit on the number of charts shown, the top four appear to be slightly different between small vs medium_ft.
'What is the meaning of life',
- Both show the same thing: no result with no place in query message.

demo2_cities_feb2023:

'How big are the public schools in Sunnyvale',
- Same rankings
'What is the prevalence of asthma there',
- old: https://screenshot.googleplex.com/6JAnFhtt2BWbHxF
- new: https://screenshot.googleplex.com/GAqNMeui3u6Ub4o
- Improvement: "Pulmonary Disease" shows up now and Smoking SVs no longer show up.
'What is the commute pattern there',
- Same chart
'How does that compare with San Bruno',
- Same chart
'Which cities in the SF Bay Area have the highest larceny',
- Same charts (which is an improvement compared to the previous medium_ft index where other crimes were also showing up very highly).
'What countries in Africa had the greatest increase in life expectancy',
- Same top few charts
- Improvement: in the small index there are two charts about Death per Capita which are not extremely relevant. This is also an improvement over the old medium_ft index which was showing mortality SVs (this is due to the fine-tuning to disassociate death with life expectancy).

demo_fallback:

'Number of Shakespeare fans in San Francisco and Chicago.',
- Same chart
'Crime in California and Florida',
- Pretty much the same charts (one chart re-ordering): small has SV about "rape" show up but medium_ft does not.
'counties in California with highest crime',
- Same behavior in both.
- Nothing comes up for "counties" and the suggestion about using California produces the same starts (same as above).
- Note that "cities in California..." Works.
'obesity in California',
- Same as above.
- Improvement: SV about overweight population among 4 years or younger shows up now.
'GDP of countries in the US',
- Top charts are the same and except for some re-ordering below, the charts are mostly the same.

multisv:

"Poverty vs. Obesity in California",
- Same chart
"Poverty vs. Obesity in California and Florida",
- Same top chart
- Different charts and ordering after that in small vs medium_ft
- Improvement: the small index brings up many "above poverty" SVs whereas the medium_ft now only brings up one.
- Small matches these SVs: https://screenshot.googleplex.com/6Fqomb2amuwjFUL
- Medium_ft matches these SVs: https://screenshot.googleplex.com/4yUkYbhpnoqCdq2
- Improvement: small index matches many races with poverty but they mostly disappear from the medium_ft
- Improvement over previous medium_ft index: "Count_Person_PovertyStatusDetermined" is not showing (due to fine tuning).
"California cities with hispanic population over 10000",
- Same chart
"Prevalence of Asthma in California cities with hispanic population over 10000",
- Same top chart
- Both now show just one chart (whereas previously medium_ft was showing more).
- One thing to potentially fix is that even though we don't see the chart, the top there SVs matched in medium_ft also have race info (Count_Death_DiseasesOfTheRespiratorySystem_BlackOrAfricanAmerican and Percent_Person_18To64Years_NoHealthInsurance_HispanicOrLatino).
  Could be improved with some finetuning.

demo_climatetrace:

Which countries emit the most greenhouse gases?
- Same top chart.
- Not great: Similar SVs and charts but different orderings with medium_ft bringing up many more "emissions" SVS which is not great.

place_detection_e2e:

'tell me about palo alto',
- same chart
'US states which have that the cheapest houses',
- Same top few charts
- Improvement: some charts about Race drop out in medium_ft and several other more household/housing SV charts show up
- Old: https://screenshot.googleplex.com/BRbfT4eN8uCwjTC
- New: https://screenshot.googleplex.com/9KogGWMyLHMazBX
'what about in florida',
- Similar as above
'compare with california and new york state and washington state',
- Similar as above
'show me the population of mexico city',
- Same top charts
- Female and Male population charts/SVs no longer show in medium_ft (which is OK but not ideal)
'counties in the US with the most poverty',
- Same top chart
- Different charts and ordering after that in small vs medium_ft
- Improvement: the small index brings up many "above poverty" SVs whereas the medium_ft now only brings up one.
- Improvement: small index matches many races with poverty but they mostly disappear from the medium_ft
- Improvement over previous medium_ft index: "Count_Person_PovertyStatusDetermined" is not showing (due to fine tuning).

international:

'Where are the most rural districts in India',
- Same top charts
- medium_ft now has a lot more rural SVs showing (not necessarily good or bad):
- Old SVs matched: https://screenshot.googleplex.com/7koU847XCaS5WsW
- New SVs matched: https://screenshot.googleplex.com/AAwppynsrf8Tt8N
'Life expectancy across provinces of China',
- Same top 3 charts
- Improvement: SV about deaths drop out in medium_ft (due to finetuning).
'GDP of counties in the United Kingdom',
- Smart charts at top
- Medium_ft now also showing the PerCapita SV (and then taking the per capita of that SV which is bad)
'Districts in Turkey with the highest fertility rate',
- Old: https://screenshot.googleplex.com/4gsGWXhuCG4nuAQ
- New: https://screenshot.googleplex.com/3EaUFpNritufEQf
- Improvement: only showing "birth" SVs and not ones about Death.
'Floods in Brazil',
- Same chart
'Drought in Africa',
- Same chart

usa_map_types:

'which cities in the Santa Clara County have the highest larceny?',
- - Same charts (which is an improvement compared to the previous medium_ft index where other crimes were also showing up very highly).
'household median income across tracts of Placer County',
- Same top two charts.
- Old: https://screenshot.googleplex.com/B7KZjCiLjkJxmUA
- New: https://screenshot.googleplex.com/3e82PjdRmKhawpB
- Improvement: SVs about "Asian" and "Hispanic" households drop out.
'how many people are unemployed in zip codes of washington state?'
- Same charts
- Improvement over previous medium_ft where extra unnecessary charts were also showing up.

sdg:

'tell me about poverty in africa',
- Same charts
'which countries have show the greatest reduction?',
- Same charts
'health in the world',
- Same charts

pradh · 2023-07-12T03:48:18Z

Thanks Jehangir! Taking a look...

pradh · 2023-07-12T03:49:01Z

Thanks Jehangir! Taking a look...

(Sorry sent early) Do you have the diff report between small vs. medium_ft?

jehangiramjad · 2023-07-12T05:14:30Z

Here is the differ report in the previous PR (#2908) description:

Linked is the differ report (html) and spreadsheet (link) which show the comparison vs the currently deployed "small" index and also makes several comments to the differences compared to a "medium" index which did not use the finetuned model.

jehangiramjad · 2023-07-12T05:38:46Z

Just updated the Python test goldens. One problem which will need to be fixed in the next batch of embedding refresh is this (should will be a TODO):

In the multi-sv test for "hispanic women phd", we now get this SV to show up:

Count_Death_DiseasesOfTheNervousSystem_White

Upon further investigation, seems like this SV has the following description and alternatives:

25% of hispanics live in low-poverty neighborhoods;
Count of Mortality Event: G00-G98 (Diseases of The Nervous System), White;
Hispanic Population With Live in Low Poverty Neighborhood,25th Percentile;
a quarter of hispanics live in low-poverty neighborhoods;
about one-fourth of hispanics live in low-poverty neighborhoods;
one in four hispanics lives in a low-poverty neighborhood

We need to fix these sentences because the word "hispanic" should not feature here.

pradh · 2023-07-12T03:50:35Z

static/js/apps/nl_interface/nl_options.tsx

              }}
            />
-            Medium-5K (experimental)
+            Medium-FT-5K (experimental)


Call it Medium-FineTuned ? Drop (experimental)??

pradh · 2023-07-12T03:50:49Z

static/js/constants/app/nl_interface_constants.ts

@@ -38,6 +38,7 @@ export const NL_URL_PARAMS = {
 export const NL_INDEX_VALS = {
  SMALL: "small",
  MEDIUM: "medium",


Drop MEDIUM?

Could you also change the checkbox default to MEDIUM_FT:

website/static/js/apps/nl_interface/app.tsx

Lines 45 to 47 in f734137

const [indexType, setIndexType] = useState(

getUrlTokenOrDefault(NL_URL_PARAMS.IDX, NL_INDEX_VALS.SMALL)

);

pradh

Should we fix the long-tail SVs for "blood pressure" query? https://screenshot.googleplex.com/3aQgF2er4bwtec2 That makes 3 queries on the go/dcnl-main-demo look odd

…e in the embeddings PR

…update

pradh · 2023-07-19T00:36:55Z

Should we fix the long-tail SVs for "blood pressure" query? https://screenshot.googleplex.com/3aQgF2er4bwtec2 That makes 3 queries on the go/dcnl-main-demo look odd

Maybe we fix the chart title for those to be a bit more readable and then its good.

pradh · 2023-07-19T00:37:07Z

Slightly wonder if we can do something about the climate trace GHG query? Because that is a very common query and dc/topic/GHGEmissionsBySource has too many SVs that push the main one out of the screen: https://screenshot.googleplex.com/C5b8nF4xjEaF85J

This is resolved now.

pradh · 2023-07-19T00:37:29Z

In go/dcnl-ct-demo, with medium_ft, the [How does that compare with russia?] page has many more charts (https://screenshot.googleplex.com/9DqPsPxG9ZasUMt) than the [top sources of co2 emissions in USA] page. That's because the "ranking-across-vars" doesn't get inherited thru context. We can improve on the fullfillment side.

This is also perhaps OK for now. We can have a bug?

pradh · 2023-07-19T00:39:10Z

This is also fixed now!

jehangiramjad · 2023-07-19T04:47:08Z

Should we fix the long-tail SVs for "blood pressure" query? https://screenshot.googleplex.com/3aQgF2er4bwtec2 That makes 3 queries on the go/dcnl-main-demo look odd

Maybe we fix the chart title for those to be a bit more readable and then its good.

Updated the chart title.

jehangiramjad · 2023-07-19T04:49:40Z

3. In go/dcnl-ct-demo, with medium_ft, the [How does that compare with russia?] page has many more charts (https://screenshot.googleplex.com/9DqPsPxG9ZasUMt) than the [top sources of co2 emissions in USA] page. That's because the "ranking-across-vars" doesn't get inherited thru context. We can improve on the fullfillment side.

Opened #2964 to track

pradh

I think this looks good!

Thanks for patiently working through this!

pradh · 2023-07-19T05:35:53Z

server/integration_tests/test_data/demo_climatetrace/query_1/chart_config.json

                    ],
-                    "title": "Greenhouse Gas Emissions from Agriculture in the World (${date})",
+                    "title": "Annual Amount of Emissions: Other Energy Use, Greenhouse Gas in the World (${date})",
                    "type": "MAP"


TODO: we can remove CLimateTrace_OtherEnergyUse

pradh · 2023-07-19T05:38:10Z

server/config/nl_page/chart_titles_by_sv.json

@@ -1061,7 +1061,7 @@
  "NumberOfMonths_WetBulbTemperature_35COrMore_RCP85_MeanRelativeHumidity": "Number of Months Reaching Wet Bulb Temperature based on RCP 8.5 Mean Relative Humidity",
  "NumberOfMonths_WetBulbTemperature_35COrMore_RCP85_MinRelativeHumidity": "Number of Months Reaching Wet Bulb Temperature based on RCP 8.5 Min Relative Humidity",
  "PalmerDroughtSeverityIndex_Atmosphere": "Palmer Drought Severity Index",
-  "Percent_Person_18OrMoreYears_WithHighBloodPressure_ReceivedTakingBloodPressureMedication": "Percent of  Population With High Blood Pressure Who Are Taking Blood Pressure Medication  Among Population Who Are 18 Years or Older in Age;",
+  "Percent_Person_18OrMoreYears_WithHighBloodPressure_ReceivedTakingBloodPressureMedication": "Percent of Adults Taking Medication Among Those With High Blood Pressure",


nit: Percent of Adults with Blood Pressure Taking Medication

jehangiramjad added 20 commits June 30, 2023 19:15

differ report with final finetuned model without intermediate

78fa246

Merge branch 'master' into finetune

98f8b7b

not updating the yaml files in this PR

cf8e75d

lint fix

6c4f502

finetuned model + embeddings update

2bf3504

merge

6ebe8e1

merging

1404213

updating the finetuned model + medium index using the finetuned model

b64d057

removing the updated model and embeddings from the yaml files for now

fee804b

Merge branch 'master' into model_embeddings_update

d2033ed

Merge branch 'master' into model_embeddings_update

9f6ad04

also updating the model.yaml and embeddings.yaml

7cb8896

Merge branch 'master' into model_embeddings_update

522df57

updating integration test goldens and using medium_ft index

5a8897f

Merge branch 'master' into embeddings

927a98d

making the medium-fit index default

7090a3d

merging

4adbdf7

Merge branch 'master' into embeddings

163c905

updating integration test goldens?

b65346d

merging

3f82aaf

jehangiramjad requested a review from pradh July 11, 2023 21:06

jehangiramjad added 2 commits July 11, 2023 22:15

Merge branch 'master' into embeddings

9975ceb

updating python test goldens for multi sv

8fe64a9

pradh reviewed Jul 12, 2023

View reviewed changes

Merge branch 'master' into embeddings

6debf0c

jehangiramjad added 14 commits July 12, 2023 14:41

fixing the local cache bug

1d05376

lint fix

e8c6448

Merge branch 'cache_fix' into embeddings

49f8034

not making the embeddings model switch in this commit/PR -- to be don…

a9ad2f8

…e in the embeddings PR

Merge branch 'cache_fix' into embeddings

97e6123

when loading the finetuned embeddings, also use the finetuned model

32b3830

pairing finetuned index with the finetuned model for the embeddings test

d3904d3

Merge branch 'master' into embeddings

f658c7f

loader update

a9b6b92

Merge branch 'master' into embeddings

c6ce2dd

merging

0dbdbdd

updating the integration tests after the new medium_ft index + model …

562de35

…update

updating some nl tests

ea50cde

Merge branch 'master' into embeddings

9fde716

jehangiramjad requested a review from pradh July 18, 2023 22:06

jehangiramjad added 3 commits July 18, 2023 21:41

updating tests

c7bdcd3

Merge branch 'master' into embeddings

9337ee9

one more test data update

2770228

jehangiramjad mentioned this pull request Jul 19, 2023

[NL] "ranking-across-vars' does not get inherited through context #2964

Open

one more update to the tests after merging etc

6c26f45

pradh approved these changes Jul 19, 2023

View reviewed changes

lint fix

9bfe07f

jehangiramjad merged commit f2779f5 into datacommonsorg:master Jul 19, 2023
9 checks passed

jehangiramjad deleted the embeddings branch July 19, 2023 06:02

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[NL Interface] Making medium_ft the default #2920

[NL Interface] Making medium_ft the default #2920

jehangiramjad commented Jul 11, 2023 •

edited

Loading

pradh commented Jul 12, 2023

pradh commented Jul 12, 2023

jehangiramjad commented Jul 12, 2023

jehangiramjad commented Jul 12, 2023

pradh Jul 12, 2023

pradh Jul 12, 2023

pradh Jul 12, 2023

pradh left a comment

pradh commented Jul 19, 2023

pradh commented Jul 19, 2023

pradh commented Jul 19, 2023

pradh commented Jul 19, 2023

jehangiramjad commented Jul 19, 2023

jehangiramjad commented Jul 19, 2023

pradh left a comment

pradh Jul 19, 2023

pradh Jul 19, 2023

	const [indexType, setIndexType] = useState(
	getUrlTokenOrDefault(NL_URL_PARAMS.IDX, NL_INDEX_VALS.SMALL)
	);

[NL Interface] Making medium_ft the default #2920

[NL Interface] Making medium_ft the default #2920

Conversation

jehangiramjad commented Jul 11, 2023 • edited Loading

pradh commented Jul 12, 2023

pradh commented Jul 12, 2023

jehangiramjad commented Jul 12, 2023

jehangiramjad commented Jul 12, 2023

pradh Jul 12, 2023

Choose a reason for hiding this comment

pradh Jul 12, 2023

Choose a reason for hiding this comment

pradh Jul 12, 2023

Choose a reason for hiding this comment

pradh left a comment

Choose a reason for hiding this comment

pradh commented Jul 19, 2023

pradh commented Jul 19, 2023

pradh commented Jul 19, 2023

pradh commented Jul 19, 2023

jehangiramjad commented Jul 19, 2023

jehangiramjad commented Jul 19, 2023

pradh left a comment

Choose a reason for hiding this comment

pradh Jul 19, 2023

Choose a reason for hiding this comment

pradh Jul 19, 2023

Choose a reason for hiding this comment

jehangiramjad commented Jul 11, 2023 •

edited

Loading