Add make sprint-data-import and issue-data-import to import github sprint and issue data to database #84

aplybeah · 2024-06-14T22:00:23Z

Summary

Fixes #46

Time to review: x mins

Changes proposed

added sprint-db-data-import to Makefile
added export_json_to_database

Context for reviewers

One strategy would be to keep the make sprint-data-export and issue-data-export and create make sprint-db-data-import and issue-data-db-import so that the data is exported to JSON and then imported into the database.

A single make command could then be created to run the the export and then import files.

Additional information

Sample data in database

coilysiren · 2024-06-20T19:47:33Z

analytics/config.py

+        Validator("SLACK_BOT_TOKEN", must_exist=False), #disabled for testing
+        Validator("REPORTING_CHANNEL_ID", must_exist=False), #disabled for testing


Billy wrote some end to end tests that require these tokens. So these need to stay enabled if we want to run the full end to end test suite, unfortunately.

That said, I don't think we should have ever written end to end tests that require these tokens to be set. So I'm willing to accept this change.

coilysiren · 2024-06-20T19:51:05Z

analytics/Makefile

+sprint-db-data-import:
+	@echo "=> Importing project data to the database"
+	@echo "====================================================="
+	$(POETRY) analytics export db_export \


⭐⭐⭐ (must change before approval) The makefile command says db-import whereas the python command is db-export. You'll want to pick one or the other. I'm thinking import?

I'm also thinking import

Please update with import. Please also create a separate import app:

import_app = typer.Typer() ... app.add_typer(import_app, name="import", help="Import data into the database")

acouch · 2024-06-20T23:07:43Z

Looks good, can you fix the lint test?

acouch · 2024-06-21T11:59:02Z

When I run this locally I get:

(analytics)(import-sprint-data)$make sprint-db-data-import
=> Importing project data to the database
=====================================================
docker-compose run -e GH_TOKEN --rm grants-analytics poetry run analytics export db_export \
	--owner HHS \
	--project 13 \
	--sprint-file data/sprint-data.json \
	--issue-file data/issue-data.json
WARN[0000] /Users/partisan/workshop/grantsgov/nava-simpler/analytics/docker-compose.yml: `version` is obsolete
[+] Creating 1/0
 ✔ Container grants-analytics-db  Running                                                                                                                                           0.0s
Warning: 'analytics' is an entry point defined in pyproject.toml, but it's not installed as a script. You may get improper `sys.argv[0]`.

The support to run uninstalled scripts will be removed in a future release.

Run `poetry install` to resolve and get rid of this message.

Usage: analytics export db_export [OPTIONS]
Try 'analytics export db_export --help' for help.
╭─ Error ───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ No such option: --owner                                                                                                                                                               │
╰───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
make: *** [sprint-db-data-import] Error 2

acouch · 2024-06-21T13:37:52Z

Missing the issue data export, unless you plan to-do another PR for that one.

acouch · 2024-06-24T14:37:28Z

analytics/src/analytics/cli.py

+    )
+
+    BaseDataset.to_sql(
+        output_table=task_data,


The output_table argument is a string for the table name. Can you recheck this locally? Did you have a working version here?

@acouch it worked in the sense that the command ran locally, but I haven't been able to find the data in the database

Try instantiating the class, then applying the .to_sql method :

deliverables = DeliverableTasks.load_from_json_files( sprint_file=sprint_file, issue_file=issue_file, ) deliverables.to_sql( output_table="github_project_data", engine=connection, replace_table=True )

acouch · 2024-06-24T15:08:10Z

analytics/poetry.lock

Make sure to add an update to the pyproject.toml if you are updating the poetry.lock file.

…e comments

aplybeah · 2024-06-25T19:31:04Z

Expected Failures:

FAILED tests/integrations/test_slack.py::test_fetch_slack_channels - TypeError: 'NoneType' object is not subscriptable
FAILED tests/integrations/test_slack.py::test_upload_files_to_slack_channel - TypeError: 'NoneType' object is not subscriptable

The tests expect slack auth. Turning the validators off ended up interrupting the full suite

coilysiren

👍🏼! But you probably want to follow-up to disable those failing tests

acouch · 2024-06-26T14:14:27Z

analytics/settings.toml

@@ -1,4 +1,4 @@
 POSTGRES_NAME = "app"
-POSTGRES_HOST = "0.0.0.0"
+POSTGRES_HOST = "grants-analytics-db"
 POSTGRES_USER = "app"
 POSTGRES_PORT = 5432


I believe you would need a password here as well, can you confirm?

Theres a secrets.toml that's in the gitignore that has the password

coilysiren

👍🏼 !

acouch · 2024-06-26T18:28:37Z

Thanks for the updates. Can you add the SprintBoard data as well? https://github.com/navapbc/simpler-grants-gov/pull/84/files#diff-9ebe0c30e3ac3922cd1dd64b982265f1732a5d801148e6cdbf7a3ff5daa399a6R12

#113) ## Summary Fixes #100 ### Time to review: __1 mins__ ## Changes proposed * Added documentation about local database import ## Context for reviewers > The current analytics documentation is focused on the slack integration. This task is to add the work from #84 to the documentation and include the local Metabase steps. ## Additional information > Screenshots, GIF demos, code examples or output to help show the changes working as expected.

…Analytics (#136) ## Summary Fixes #107 Fixes #115 ### Time to review: __5 mins__ ## Changes proposed * removed the `*.toml` files related to dynaconf * removed references to Dynaconf (e.g. in Docstrings, gitignore) * use Pydantic for loading ## Context for reviewers > After getting feedback for #107, the consensus was to reevaluate the way the database loader works for more uniformity. Changes will need to be made primarily in db.py and cli.py > > With the PR #84 , the env settings for the db are stored in settings.toml. The config settings should be updated to use the existing local.env file ## Additional information > Screenshots, GIF demos, code examples or output to help show the changes working as expected.

…rint and issue data to database (#84) Fixes #46 * added `sprint-db-data-import` to Makefile * added `export_json_to_database` > One strategy would be to keep the make sprint-data-export and issue-data-export and create make sprint-db-data-import and issue-data-db-import so that the data is exported to JSON and then imported into the database. > > A single make command could then be created to run the the export and then import files. Sample data in database <img width="1133" alt="Screen Shot 2024-06-26 at 3 38 47 PM" src="https://github.com/navapbc/simpler-grants-gov/assets/37313082/34c962d6-a78e-4963-be15-ef0f7de3bccf">

#113) Fixes #100 * Added documentation about local database import > The current analytics documentation is focused on the slack integration. This task is to add the work from #84 to the documentation and include the local Metabase steps. > Screenshots, GIF demos, code examples or output to help show the changes working as expected.

…Analytics (#136) Fixes #107 Fixes #115 * removed the `*.toml` files related to dynaconf * removed references to Dynaconf (e.g. in Docstrings, gitignore) * use Pydantic for loading > After getting feedback for #107, the consensus was to reevaluate the way the database loader works for more uniformity. Changes will need to be made primarily in db.py and cli.py > > With the PR #84 , the env settings for the db are stored in settings.toml. The config settings should be updated to use the existing local.env file > Screenshots, GIF demos, code examples or output to help show the changes working as expected.

…rint and issue data to database (#84) Fixes #46 * added `sprint-db-data-import` to Makefile * added `export_json_to_database` > One strategy would be to keep the make sprint-data-export and issue-data-export and create make sprint-db-data-import and issue-data-db-import so that the data is exported to JSON and then imported into the database. > > A single make command could then be created to run the the export and then import files. Sample data in database <img width="1133" alt="Screen Shot 2024-06-26 at 3 38 47 PM" src="https://github.com/navapbc/simpler-grants-gov/assets/37313082/34c962d6-a78e-4963-be15-ef0f7de3bccf">

#113) Fixes #100 * Added documentation about local database import > The current analytics documentation is focused on the slack integration. This task is to add the work from #84 to the documentation and include the local Metabase steps. > Screenshots, GIF demos, code examples or output to help show the changes working as expected.

…Analytics (#136) Fixes #107 Fixes #115 * removed the `*.toml` files related to dynaconf * removed references to Dynaconf (e.g. in Docstrings, gitignore) * use Pydantic for loading > After getting feedback for #107, the consensus was to reevaluate the way the database loader works for more uniformity. Changes will need to be made primarily in db.py and cli.py > > With the PR #84 , the env settings for the db are stored in settings.toml. The config settings should be updated to use the existing local.env file > Screenshots, GIF demos, code examples or output to help show the changes working as expected.

…rint and issue data to database (navapbc#84) Fixes #46 * added `sprint-db-data-import` to Makefile * added `export_json_to_database` > One strategy would be to keep the make sprint-data-export and issue-data-export and create make sprint-db-data-import and issue-data-db-import so that the data is exported to JSON and then imported into the database. > > A single make command could then be created to run the the export and then import files. Sample data in database <img width="1133" alt="Screen Shot 2024-06-26 at 3 38 47 PM" src="https://github.com/navapbc/simpler-grants-gov/assets/37313082/34c962d6-a78e-4963-be15-ef0f7de3bccf">

navapbc#113) Fixes #100 * Added documentation about local database import > The current analytics documentation is focused on the slack integration. This task is to add the work from navapbc#84 to the documentation and include the local Metabase steps. > Screenshots, GIF demos, code examples or output to help show the changes working as expected.

…Analytics (navapbc#136) Fixes #107 Fixes #115 * removed the `*.toml` files related to dynaconf * removed references to Dynaconf (e.g. in Docstrings, gitignore) * use Pydantic for loading > After getting feedback for navapbc#107, the consensus was to reevaluate the way the database loader works for more uniformity. Changes will need to be made primarily in db.py and cli.py > > With the PR navapbc#84 , the env settings for the db are stored in settings.toml. The config settings should be updated to use the existing local.env file > Screenshots, GIF demos, code examples or output to help show the changes working as expected.

aplybeah added 2 commits June 13, 2024 13:59

initial commands

a0f8f3f

update export cmd

4a8a832

github-actions bot added ci/cd python analytics labels Jun 14, 2024

aplybeah added 4 commits June 20, 2024 12:03

update cli command to accept files

81f480c

Merge branch 'main' into import-sprint-data

b17fda9

lint

a26de2c

linting again

1d5bce6

aplybeah requested review from acouch and chouinar June 20, 2024 17:44

aplybeah marked this pull request as ready for review June 20, 2024 17:45

aplybeah requested a review from coilysiren as a code owner June 20, 2024 17:45

coilysiren reviewed Jun 20, 2024

View reviewed changes

acouch mentioned this pull request Jun 21, 2024

[Task]: Document the new steps for the analytics local database import #100

Closed

1 task

acouch reviewed Jun 24, 2024

View reviewed changes

aplybeah added 4 commits June 25, 2024 13:05

update pyproject; cli and settings to create working connection

2f9d957

remove unused

4663874

Merge branch 'main' into import-sprint-data

3d1f2c2

reenable validator checks to appease testing; update analytics packag…

7c415ee

…e comments

aplybeah requested review from acouch and coilysiren June 25, 2024 19:31

coilysiren approved these changes Jun 25, 2024

View reviewed changes

skip e2e tests that require slack auth

b4db830

lint

b2dfe45

acouch reviewed Jun 26, 2024

View reviewed changes

This was referenced Jun 26, 2024

[Task]: Update analytics db to use local.env #107

Closed

[Task]: Add db connection test #108

Closed

[Task]: Report on the success of the database import and the number of records imported #109

Closed

match naming conventions;add import app

bc4553f

coilysiren approved these changes Jun 26, 2024

View reviewed changes

add sprint board data

09667a0

acouch approved these changes Jun 26, 2024

View reviewed changes

aplybeah merged commit 817a1d3 into main Jun 26, 2024
28 checks passed

aplybeah deleted the import-sprint-data branch June 26, 2024 20:15

aplybeah mentioned this pull request Jun 27, 2024

[Task]: Document the new steps for the analytics local database import #113

Merged

This was referenced Jul 3, 2024

[Task]: Report on the success of the database import and the number of records imported #133

Closed

[Task] Update analytics db to use local.env and Remove Dynaconf from Analytics #136

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add make sprint-data-import and issue-data-import to import github sprint and issue data to database #84

Add make sprint-data-import and issue-data-import to import github sprint and issue data to database #84

aplybeah commented Jun 14, 2024 •

edited

Loading

coilysiren Jun 20, 2024

coilysiren Jun 20, 2024

aplybeah Jun 24, 2024

acouch Jun 26, 2024

acouch commented Jun 20, 2024

acouch commented Jun 21, 2024 •

edited

Loading

acouch commented Jun 21, 2024

acouch Jun 24, 2024

aplybeah Jun 24, 2024

acouch Jun 24, 2024

acouch Jun 24, 2024

aplybeah commented Jun 25, 2024

coilysiren left a comment

acouch Jun 26, 2024

aplybeah Jun 26, 2024

coilysiren left a comment

acouch commented Jun 26, 2024

		Validator("SLACK_BOT_TOKEN", must_exist=False), #disabled for testing
		Validator("REPORTING_CHANNEL_ID", must_exist=False), #disabled for testing

Add make sprint-data-import and issue-data-import to import github sprint and issue data to database #84

Add make sprint-data-import and issue-data-import to import github sprint and issue data to database #84

Conversation

aplybeah commented Jun 14, 2024 • edited Loading

Summary

Time to review: x mins

Changes proposed

Context for reviewers

Additional information

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

acouch commented Jun 20, 2024

acouch commented Jun 21, 2024 • edited Loading

acouch commented Jun 21, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aplybeah commented Jun 25, 2024

coilysiren left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

coilysiren left a comment

Choose a reason for hiding this comment

acouch commented Jun 26, 2024

aplybeah commented Jun 14, 2024 •

edited

Loading

acouch commented Jun 21, 2024 •

edited

Loading