Skip to content

Commit

Permalink
Clean up after moving to LFAI (#665)
Browse files Browse the repository at this point in the history
* Clean up after moving to LFAI

Clean up after moving to LFAI

* Update README.md
  • Loading branch information
xiaoyongzhu authored Sep 14, 2022
1 parent 6de1d60 commit a54180b
Show file tree
Hide file tree
Showing 18 changed files with 64 additions and 62 deletions.
2 changes: 1 addition & 1 deletion .github/pull_request_template.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
## Description
<!--
Hey! Thank you for the contribution! Please go through https://github.com/linkedin/feathr/blob/main/docs/dev_guide/pull_request_guideline.md for more information.
Hey! Thank you for the contribution! Please go through https://github.com/feathr-ai/feathr/blob/main/docs/dev_guide/pull_request_guideline.md for more information.
Describe what changes to make and why you are making these changes.
Expand Down
4 changes: 2 additions & 2 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ As a contributor, you represent that the code you submit is your original work o

# Responsible Disclosure of Security Vulnerabilities

Please do not file reports on Github for security issues. Please review the guidelines on at (link to more info). Reports should be encrypted using PGP (link to PGP key) and sent to security@linkedin.com preferably with the title "Github linkedin/ - ".
Please do not file reports on Github for security issues. Please review the guidelines on at (link to more info). Reports should be encrypted using PGP (link to PGP key) and sent to feathr-security@lists.lfaidata.foundation.

# Contribution Process

Expand All @@ -14,7 +14,7 @@ The Feathr community welcome everyone, and encourage a friendly and positive env

Please read existing Github issues or development work that is in progress or in the backlog to avoid duplication. If you are interested in those existing ones, you can leave a comment in the Github issues and the community will try to involve you. If you are not sure if it's duplicated, just create a Github issue and ask!

If it's a simple bug fix(less than 20 lines) or documentation change, you can just submit your pull request(PR) without Github issues. For any other PRs, a Github issue is required.
If it's a simple bug fix (less than 20 lines) or documentation change, you can just submit your pull request(PR) without Github issues. For any other PRs, a Github issue is required.

If you want to contribute something new and it's not tracked in existing Github issues, please create a new Github issue and the community will help review the idea. Please state `why` in your Github issue. If you already have a short design in mind, you can provide a one pager in the Github issue. If the idea in general make sense, then we can proceed to the design or development work. If the change is not small, an [RFC](https://en.wikipedia.org/wiki/Request_for_Comments) should be reviewed and approved by the team.

Expand Down
38 changes: 20 additions & 18 deletions docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,23 +6,25 @@
An enterprise-grade, high-performance feature store
</h3>
<h3 align="center">
Support channels:
Important Links:
<a href="https://join.slack.com/t/feathrai/shared_invite/zt-1ffva5u6v-voq0Us7bbKAw873cEzHOSg">Slack</a> &
<a href="https://github.com/feathr-ai/feathr/discussions">Discussions</a>
<a href="https://github.com/feathr-ai/feathr/discussions">Discussions</a>.
<a href="https://feathr-ai.github.io/feathr/">Docs</a>.
</h3>
</html>

#

[![License](https://img.shields.io/badge/License-Apache%202.0-blue)](https://github.com/linkedin/feathr/blob/main/LICENSE)
[![GitHub Release](https://img.shields.io/github/v/release/linkedin/feathr.svg?style=flat&sort=semver&color=blue)](https://github.com/linkedin/feathr/releases)

[![License](https://img.shields.io/badge/License-Apache%202.0-blue)](https://github.com/feathr-ai/feathr/blob/main/LICENSE)
[![GitHub Release](https://img.shields.io/github/v/release/feathr-ai/feathr.svg?style=flat&sort=semver&color=blue)](https://github.com/feathr-ai/feathr/releases)
[![Docs Latest](https://img.shields.io/badge/docs-latest-blue.svg)](https://feathr-ai.github.io/feathr/)
[![Python API](https://img.shields.io/readthedocs/feathr?label=Python%20API)](https://feathr.readthedocs.io/en/latest/)
[![CII Best Practices](https://bestpractices.coreinfrastructure.org/projects/6457/badge)](https://bestpractices.coreinfrastructure.org/projects/6457)

## What is Feathr?

Feathr is the feature store that is used in production in LinkedIn for many years and was open sourced in April 2022. Read our announcement on [Open Sourcing Feathr](https://engineering.linkedin.com/blog/2022/open-sourcing-feathr---linkedin-s-feature-store-for-productive-m) and [Feathr on Azure](https://azure.microsoft.com/en-us/blog/feathr-linkedin-s-feature-store-is-now-available-on-azure/).
Feathr is the feature store that is used in production in LinkedIn for many years and was open sourced in April 2022. It is currently a project under [LF AI & Data Foundation](https://lfaidata.foundation/blog/2022/09/12/feathr-joins-lf-ai-data-as-new-sandbox-project/).

Read our announcement on [Open Sourcing Feathr](https://engineering.linkedin.com/blog/2022/open-sourcing-feathr---linkedin-s-feature-store-for-productive-m) and [Feathr on Azure](https://azure.microsoft.com/en-us/blog/feathr-linkedin-s-feature-store-is-now-available-on-azure/), as well as the [announcement from LF AI & Data Foundation](https://lfaidata.foundation/blog/2022/09/12/feathr-joins-lf-ai-data-as-new-sandbox-project/).

Feathr lets you:

Expand Down Expand Up @@ -72,7 +74,7 @@ pip install feathr
Or use the latest code from GitHub:

```bash
pip install git+https://github.com/linkedin/feathr.git#subdirectory=feathr_project
pip install git+https://github.com/feathr-ai/feathr.git#subdirectory=feathr_project
```

## 🔡 Feathr Highlighted Capabilities
Expand Down Expand Up @@ -170,17 +172,17 @@ Follow the [quick start Jupyter Notebook](./samples/product_recommendation_demo.

![Architecture Diagram](./images/architecture.png)

| Feathr component | Cloud Integrations |
| ------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- |
| Offline store – Object Store | Azure Blob Storage, Azure ADLS Gen2, AWS S3 |
| Offline store – SQL | Azure SQL DB, Azure Synapse Dedicated SQL Pools, Azure SQL in VM, Snowflake |
| Streaming Source | Kafka, EventHub |
| Feathr component | Cloud Integrations |
| ------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Offline store – Object Store | Azure Blob Storage, Azure ADLS Gen2, AWS S3 |
| Offline store – SQL | Azure SQL DB, Azure Synapse Dedicated SQL Pools, Azure SQL in VM, Snowflake |
| Streaming Source | Kafka, EventHub |
| Online store | Redis, [Azure Cosmos DB](https://feathr-ai.github.io/feathr/how-to-guides/jdbc-cosmos-notes.html#using-cosmosdb-as-the-online-store), Aerospike (coming soon) |
| Feature Registry and Governance | Azure Purview, ANSI SQL such as Azure SQL Server |
| Compute Engine | Azure Synapse Spark Pools, Databricks |
| Machine Learning Platform | Azure Machine Learning, Jupyter Notebook, Databricks Notebook |
| File Format | Parquet, ORC, Avro, JSON, Delta Lake, CSV |
| Credentials | Azure Key Vault |
| Feature Registry and Governance | Azure Purview, ANSI SQL such as Azure SQL Server |
| Compute Engine | Azure Synapse Spark Pools, Databricks |
| Machine Learning Platform | Azure Machine Learning, Jupyter Notebook, Databricks Notebook |
| File Format | Parquet, ORC, Avro, JSON, Delta Lake, CSV |
| Credentials | Azure Key Vault |

## 🚀 Roadmap

Expand Down
2 changes: 1 addition & 1 deletion docs/concepts/feature-registry.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ Feathr UI and Feathr Registry are two optional components to use Feathr, but usu

Please follow the `Provision Azure Resources using ARM Template` part in the [Azure Resource Provisioning document](../how-to-guides/azure-deployment-arm.md#provision-azure-resources-using-arm-template) to provision corresponding the Azure resources. After completing those steps, you should have a set of resources that can be used for Feature Registry.

In case you want to do it in a more customized way, you can use [this Dockerfile](https://github.com/linkedin/feathr/blob/main/FeathrRegistry.Dockerfile) to deploy the REST API and UI. This docker image is more for illustration purpose, and you can customize it further (like building the REST API and UI in separate docker images).
In case you want to do it in a more customized way, you can use [this Dockerfile](https://github.com/feathr-ai/feathr/blob/main/FeathrRegistry.Dockerfile) to deploy the REST API and UI. This docker image is more for illustration purpose, and you can customize it further (like building the REST API and UI in separate docker images).

If you use [Azure Resource Provisioning document](../how-to-guides/azure-deployment-arm.md#provision-azure-resources-using-arm-template) to provision the resources, you should be able to access the UI in this website:

Expand Down
2 changes: 1 addition & 1 deletion docs/concepts/get-offline-features.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ For example, the dataset is like below, where there are 3 tables that feature pr

For feature consumers, they will usually use a central dataset ("observation data", `user_observation_mock_data` in this case) which contains a couple of IDs (`user_id` and `product_id` in this case), timestamps, and other columns. Feature consumers will use this "observation data" to query from different feature tables (using `Feature Query` below).

![Feature Flow](https://github.com/linkedin/feathr/blob/main/docs/images/product_recommendation_advanced.jpg?raw=true)
![Feature Flow](https://github.com/feathr-ai/feathr/blob/main/docs/images/product_recommendation_advanced.jpg?raw=true)

As we can see, the use case for getting offline features using Feathr is straightforward. Feature consumers want to get a few features - for a particular user, what's the gift card balance? What's the total purchase in the last 90 days; Feature consumers can also get a few features for other entities in the same `Feature Query`. For example, in the meanwhile, feature consumers can also query the product feature such as product quantity and price.

Expand Down
2 changes: 1 addition & 1 deletion docs/dev_guide/cloud_integration_testing.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ parent: Developer Guides
---
# Cloud Integration Test/CI Pipeline

We use [GitHub Actions](https://github.com/linkedin/feathr/tree/main/.github/workflows) to do cloud integration test. Currently the integration test has 4 jobs:
We use [GitHub Actions](https://github.com/feathr-ai/feathr/tree/main/.github/workflows) to do cloud integration test. Currently the integration test has 4 jobs:

- running `sbt test` to verify if the scala/spark related code has passed all the test
- running `flake8` to lint python scripts and make sure there are no obvious syntax errors
Expand Down
2 changes: 1 addition & 1 deletion docs/dev_guide/cloud_resource_provision.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ Get-AzLocation | select displayname,location

```powershell
iwr https://raw.githubusercontent.com/linkedin/feathr/main/docs/how-to-guides/deployFeathr.ps1 -outfile ./deployFeathr.ps1; ./deployFeathr.ps1 -AzureRegion '{Assign Your Region}'
iwr https://raw.githubusercontent.com/feathr-ai/feathr/main/docs/how-to-guides/deployFeathr.ps1 -outfile ./deployFeathr.ps1; ./deployFeathr.ps1 -AzureRegion '{Assign Your Region}'
```

Expand Down
26 changes: 13 additions & 13 deletions docs/dev_guide/feathr_overall_release_guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,33 +21,33 @@ This document describes all the release process for the development team.

## Writing Release Note

Write a release note following past examples [here](https://github.com/linkedin/feathr/releases).
Read through the [commit log](https://github.com/linkedin/feathr/commits/main) to identify the commits after last release to include in the release note. Here are the major things to include
Write a release note following past examples [here](https://github.com/feathr-ai/feathr/releases).
Read through the [commit log](https://github.com/feathr-ai/feathr/commits/main) to identify the commits after last release to include in the release note. Here are the major things to include

- highlights of the release
- improvements and changes of this release
- new contributors of this release

## Code Changes
Before the release is made, the version needs to be updated in following places
- [build.sbt](https://github.com/linkedin/feathr/blob/main/build.sbt#L3) - For Maven release version
- [setup.py](https://github.com/linkedin/feathr/blob/main/feathr_project/setup.py#L10) - For PyPi release version
- [conf.py](https://github.com/linkedin/feathr/blob/main/feathr_project/docs/conf.py#L27) - For documentation version
- [feathr_config.yaml](https://github.com/linkedin/feathr/blob/main/feathr_project/test/test_user_workspace/feathr_config.yaml#L84) - To set the spark runtime location for Azure Synapse and Azure Databricks used by test suite.
- [constants.py](https://github.com/linkedin/feathr/blob/73656fe4a57219e99ff6fede10d51a000ae90fa1/feathr_project/feathr/constants.py#L31) - To set the default maven artifact version
- [build.sbt](https://github.com/feathr-ai/feathr/blob/main/build.sbt#L3) - For Maven release version
- [setup.py](https://github.com/feathr-ai/feathr/blob/main/feathr_project/setup.py#L10) - For PyPi release version
- [conf.py](https://github.com/feathr-ai/feathr/blob/main/feathr_project/docs/conf.py#L27) - For documentation version
- [feathr_config.yaml](https://github.com/feathr-ai/feathr/blob/main/feathr_project/test/test_user_workspace/feathr_config.yaml#L84) - To set the spark runtime location for Azure Synapse and Azure Databricks used by test suite.
- [constants.py](https://github.com/feathr-ai/feathr/blob/73656fe4a57219e99ff6fede10d51a000ae90fa1/feathr_project/feathr/constants.py#L31) - To set the default maven artifact version

## Triggering automated release pipelines
Our goal is to automate the release process as much as possible. So far, we have automated the following steps
1. Automated [workflow](https://github.com/linkedin/feathr/blob/main/.github/workflows/docker-publish.yml) to build and publish for our UI and API container to [dockerhub](https://hub.docker.com/r/feathrfeaturestore/feathr-registry/tags).
1. Automated [workflow](https://github.com/feathr-ai/feathr/blob/main/.github/workflows/docker-publish.yml) to build and publish for our UI and API container to [dockerhub](https://hub.docker.com/r/feathrfeaturestore/feathr-registry/tags).
**Triggers** - Nightly, branch with name pattern "releases/*"

1. Automated [workflow](https://github.com/linkedin/feathr/blob/main/.github/workflows/publish-to-pypi.yml) for publishing Python package to [PyPi](https://pypi.org/project/feathr/).
1. Automated [workflow](https://github.com/feathr-ai/feathr/blob/main/.github/workflows/publish-to-pypi.yml) for publishing Python package to [PyPi](https://pypi.org/project/feathr/).

**Triggers** - branch with name pattern "releases/*"

1. Automated Maven workflow - Coming soon.

**PLEASE NOTE: To trigger the above workflows as part of release, create a new branch with pattern releases/v0.x.0**. See past release branches [here](https://github.com/linkedin/feathr/branches/all?query=releases).
**PLEASE NOTE: To trigger the above workflows as part of release, create a new branch with pattern releases/v0.x.0**. See past release branches [here](https://github.com/feathr-ai/feathr/branches/all?query=releases).


## Release Maven
Expand All @@ -59,13 +59,13 @@ See [Developer Guide for publishing to maven](publish_to_maven.md)
Run the command to generate the Java jar. After the jar is generated, please upload to [Azure storage](https://ms.portal.azure.com/#view/Microsoft_Azure_Storage/ContainerMenuBlade/~/overview/storageAccountId/%2Fsubscriptions%2Fa6c2a7cc-d67e-4a1a-b765-983f08c0423a%2FresourceGroups%2Fazurefeathrintegration%2Fproviders%2FMicrosoft.Storage%2FstorageAccounts%2Fazurefeathrstorage/path/public/etag/%220x8D9E6F64D62D599%22/defaultEncryptionScope/%24account-encryption-key/denyEncryptionScopeOverride//defaultId//publicAccessVal/Container) for faster access.

## Release PyPi
The automated workflow should take care of this, you can check under [actions](https://github.com/linkedin/feathr/actions/workflows/publish-to-pypi.yml) to see the triggered run and results. For manual steps, see [Python Package Release Note](https://feathr-ai.github.io/feathr/dev_guide/python_package_release.html)
The automated workflow should take care of this, you can check under [actions](https://github.com/feathr-ai/feathr/actions/workflows/publish-to-pypi.yml) to see the triggered run and results. For manual steps, see [Python Package Release Note](https://feathr-ai.github.io/feathr/dev_guide/python_package_release.html)

## Updating docker image for API and Registry
The automated workflow should take care of this as well, you can check under [actions](https://github.com/linkedin/feathr/actions/workflows/docker-publish.yml) to see the triggered run and results. For manual steps, see [Feathr Registry docker image](https://feathr-ai.github.io/feathr/dev_guide/build-and-push-feathr-registry-docker-image.html)
The automated workflow should take care of this as well, you can check under [actions](https://github.com/feathr-ai/feathr/actions/workflows/docker-publish.yml) to see the triggered run and results. For manual steps, see [Feathr Registry docker image](https://feathr-ai.github.io/feathr/dev_guide/build-and-push-feathr-registry-docker-image.html)

## Testing
Run one of the sample [notebook](https://github.com/linkedin/feathr/blob/main/docs/samples/product_recommendation_demo.ipynb) as it uses the latest package from Maven and PyPi.
Run one of the sample [notebook](https://github.com/feathr-ai/feathr/blob/main/docs/samples/product_recommendation_demo.ipynb) as it uses the latest package from Maven and PyPi.

## Announcement

Expand Down
Loading

0 comments on commit a54180b

Please sign in to comment.