Add pypi github project example (#14145)

## Summary & Motivation

This is an example project for the upcoming analytics workshop

---------

Co-authored-by: yuhan <yuhan@elementl.com>

examples/project_pypi_github/.gitignore

@@ -0,0 +1,161 @@
+# Byte-compiled / optimized / DLL files
+*.db
+__pycache__/
+*.py[cod]
+*$py.class
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+share/python-wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+
+# PyInstaller
+#  Usually these files are written by a python script from a template
+#  before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.nox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+*.py,cover
+.hypothesis/
+.pytest_cache/
+cover/
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+local_settings.py
+db.sqlite3
+db.sqlite3-journal
+
+# Flask stuff:
+instance/
+.webassets-cache
+
+# Scrapy stuff:
+.scrapy
+
+# Sphinx documentation
+docs/_build/
+
+# PyBuilder
+.pybuilder/
+target/
+
+# Jupyter Notebook
+.ipynb_checkpoints
+
+# IPython
+profile_default/
+ipython_config.py
+
+# pyenv
+#   For a library or package, you might want to ignore these files since the code is
+#   intended to run in multiple environments; otherwise, check them in:
+# .python-version
+
+# pipenv
+#   According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
+#   However, in case of collaboration, if having platform-specific dependencies or dependencies
+#   having no cross-platform support, pipenv may install dependencies that don't work, or not
+#   install all needed dependencies.
+#Pipfile.lock
+
+# poetry
+#   Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control.
+#   This is especially recommended for binary packages to ensure reproducibility, and is more
+#   commonly ignored for libraries.
+#   https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control
+#poetry.lock
+
+# pdm
+#   Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control.
+#pdm.lock
+#   pdm stores project-wide configurations in .pdm.toml, but it is recommended to not include it
+#   in version control.
+#   https://pdm.fming.dev/#use-with-ide
+.pdm.toml
+
+# PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm
+__pypackages__/
+
+# Celery stuff
+celerybeat-schedule
+celerybeat.pid
+
+# SageMath parsed files
+*.sage.py
+
+# Environments
+.env
+.venv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+
+# Spyder project settings
+.spyderproject
+.spyproject
+
+# Rope project settings
+.ropeproject
+
+# mkdocs documentation
+/site
+
+# mypy
+.mypy_cache/
+.dmypy.json
+dmypy.json
+
+# Pyre type checker
+.pyre/
+
+# pytype static type analyzer
+.pytype/
+
+# Cython debug symbols
+cython_debug/
+
+# PyCharm
+#  JetBrains specific template is maintained in a separate JetBrains.gitignore that can
+#  be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore
+#  and can be added to the global gitignore or merged into this file.  For a more nuclear
+#  option (not recommended) you can uncomment the following to ignore the entire idea folder.
+#.idea/

examples/project_pypi_github/README.md

@@ -0,0 +1,95 @@
+# Dagster Pypi / Github
+
+This starter kit shows how to build the Dagster's
+Software-Defined Assets for an Analytics ETL Workflow with
+different deployments for a local and prod environment.
+
+![](assets/asset_graph.png)
+
+Locally, this project uses DuckDB, dbt, and CSV files to load sample
+data and generate example models.
+
+In production, this project uses BigQuery along with the pandas
+io manager to fetch data on Pypi downloads from the Bigquery public
+dataset, as well as Steampipe to fetch the latest Github data
+for stars, forks, and other metrics.
+
+## Overview
+
+The idea around this project is to capture Python package usage data
+as well as Github metrics for a given package. This data is then
+aggregated and used to generate a model that can be used to
+visualize the growth of a package over time.
+
+This project demonstrates the use of partitions and software-defined assets.
+Daily partitions are created for fetching data related to both
+
+## Prequisites
+
+### Local Development
+
+To run locally, all you need is to install this package locally:
+
+```bash
+pip install -e .
+```
+
+Then start dagit
+
+```bash
+dagit dev
+```
+
+### Production
+
+The production version requires a few things to run:
+
+- A BigQuery project with [credentials configured](https://docs.dagster.io/integrations/bigquery/reference#providing-credentials-as-configuration)
+- A running [Steampipe](https://steampipe.io/downloads) service with [Github credentials](https://hub.steampipe.io/plugins/turbot/github#credentials)
+  configured for API access.
+- A Hex account with an API key and a project to run
+
+Be mindful of costs of running this model, especially if you choose to backfill.
+It is recommended that you only run for a few partition dates to test.
+
+## Using environment variables to handle secrets
+
+Dagster allows using environment variables to handle sensitive information. You can define various configuration options and access environment variables through them. This also allows you to parameterize your pipeline without modifying code.
+
+- BIGQUERY
+  - `BIGQUERY_PROJECT` e.g. `my-bigquery-project`
+- HEX
+  - `HEX_API_KEY` e.g. `super-secret-api-key`
+  - `HEX_PROJECT_ID` e.g. `abc-123-def`
+- STEAMPIPE
+  - `STEAMPIPE_CONN` e.g. `postgresql://steampipe:hunter42@localhost:9193/steampipe`
+
+You can declare environment variables in various ways:
+- **Local development**: [Using `.env` files to load env vars into local environments](https://docs.dagster.io/guides/dagster/using-environment-variables-and-secrets#declaring-environment-variables)
+- **Dagster Cloud**: [Using the Dagster Cloud UI](https://docs.dagster.io/master/dagster-cloud/developing-testing/environment-variables-and-secrets#using-the-dagster-cloud-ui) to manage environment variables
+- **Dagster Open Source**: How environment variables are set for Dagster projects deployed on your infrastructure depends on where Dagster is deployed. Read about how to declare environment variables [here](https://docs.dagster.io/master/guides/dagster/using-environment-variables-and-secrets#declaring-environment-variables).
+
+Check out [Using environment variables and secrets guide](https://docs.dagster.io/guides/dagster/using-environment-variables-and-secrets) for more info and examples.
+
+## Steampipe Setup
+
+If you wish to run Steampipe locally on a Mac you can
+run `brew install steampipe`
+
+Then install the Github plugin:
+
+```bash
+steampipe plugin install github
+```
+
+You'll need to create a [Github token](https://hub.steampipe.io/plugins/turbot/github#documentation:~:text=must%20create%20a-,personal%20access%20token,-and%20assign%20the)
+and update the steampipe confing file: `~/.steampipe/config/github.spc`
+See the [docs](https://hub.steampipe.io/plugins/turbot/github#documentation)
+for more information
+
+Finally, run `steampipe service start` to start the steampipe service.
+Typically you can use the following connection string
+`postgresql://steampipe@localhost:9193/steampipe`
+
+Note that it is `postgresql` and not `postgres`.
+

examples/project_pypi_github/dagster_pypi/__init__.py

@@ -0,0 +1,28 @@
+import os
+
+from dagster import (
+    Definitions,
+    ScheduleDefinition,
+    define_asset_job,
+    load_assets_from_modules,
+)
+
+from . import assets, resources
+
+ENV = os.getenv("DAGSTER_ENV", "LOCAL")
+
+
+daily_schedule = ScheduleDefinition(
+    job=define_asset_job(name="dagster_pypi_job"),
+    cron_schedule="0 0 * * *",
+)
+
+all_assets = load_assets_from_modules([assets])
+
+print("Loading definitions for environment: ", ENV)
+
+defs = Definitions(
+    assets=all_assets,
+    schedules=[daily_schedule],
+    resources=resources.resource_def[ENV.upper()],
+)

examples/project_pypi_github/dagster_pypi/assets.py

@@ -0,0 +1,98 @@
+import os
+
+import pandas as pd
+from dagster import DailyPartitionsDefinition, MetadataValue, asset
+from dagster_dbt import load_assets_from_dbt_project
+
+from .resources import (
+    DBT_PROFILE_DIR,
+    DBT_PROJECT_DIR,
+    HEX_PROJECT_ID,
+    GithubResource,
+    PyPiResource,
+)
+
+dbt_assets = load_assets_from_dbt_project(
+    project_dir=DBT_PROJECT_DIR,
+    profiles_dir=DBT_PROFILE_DIR,
+)
+
+START_DATE = "2023-04-10"
+
+
+@asset(
+    key_prefix=["dagster_pypi"],
+    partitions_def=DailyPartitionsDefinition(start_date=START_DATE),
+    metadata={"partition_expr": "download_date"},
+)
+def raw_pypi_downloads(context, pypi: PyPiResource) -> pd.DataFrame:
+    df = pypi.get_pypi_download_counts(context.partition_key)
+    context.add_output_metadata(
+        {
+            "num_records": len(df),
+            "preview": MetadataValue.md(df.head().to_markdown()),
+        }
+    )
+    return df
+
+
+@asset(
+    key_prefix=["dagster_pypi"],
+    partitions_def=DailyPartitionsDefinition(start_date=START_DATE),
+    metadata={"partition_expr": "date"},
+)
+def raw_github_stars(context, github: GithubResource) -> pd.DataFrame:
+    df = github.get_github_stars(context.partition_key)
+    context.add_output_metadata(
+        {
+            "num_records": len(df),
+            "preview": MetadataValue.md(df.head().to_markdown()),
+        }
+    )
+
+    return df
+
+
+@asset(
+    key_prefix=["dagster_pypi"],
+    partitions_def=DailyPartitionsDefinition(start_date=START_DATE),
+    metadata={"partition_expr": "download_date"},
+)
+def pypi_downloads(context, raw_pypi_downloads) -> pd.DataFrame:
+    df = raw_pypi_downloads
+    # Here we could perform some pandas transformations on data
+    context.add_output_metadata(
+        {
+            "num_records": len(df),
+            "preview": MetadataValue.md(df.head().to_markdown()),
+        }
+    )
+    return df
+
+
+@asset(
+    key_prefix=["dagster_pypi"],
+    partitions_def=DailyPartitionsDefinition(start_date=START_DATE),
+    metadata={"partition_expr": "date"},
+)
+def github_stars(context, raw_github_stars) -> pd.DataFrame:
+    df = raw_github_stars
+    context.add_output_metadata(
+        {
+            "num_records": len(df),
+            "preview": MetadataValue.md(df.head().to_markdown()),
+        }
+    )
+
+    return df
+
+
+@asset(
+    non_argument_deps={"weekly_agg_activity", "daily_agg_activity", "monthly_agg_activity"},
+    required_resource_keys={"hex"},
+)
+def hex_notebook(context) -> None:
+    if os.getenv("DAGSTER_ENV") == "PROD":
+        context.resources.hex.run_and_poll(project_id=HEX_PROJECT_ID, inputs=[])
+    else:
+        print("Skipping hex notebook in non-prod environment")

examples/project_pypi_github/dagster_pypi/dbt_project/.gitignore

@@ -0,0 +1,4 @@
+duckdb.db
+target/
+dbt_packages/
+logs/