Skip to content

Commit

Permalink
Merged ChatGPT attempts 2 and 3 with manual enhancements
Browse files Browse the repository at this point in the history
  • Loading branch information
sveinugu committed Jan 19, 2025
1 parent c547b3b commit 3cf33df
Show file tree
Hide file tree
Showing 2 changed files with 32 additions and 19 deletions.
4 changes: 2 additions & 2 deletions docs/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,11 +18,11 @@
///

/// html | div.indented_paragraphs
--8<-- "readme.md:10:49"
--8<-- "readme.md:10:61"
///

/// html | div.left_column
--8<-- "readme.md:50:"
--8<-- "readme.md:62:"
///

/// html | div.right_column
Expand Down
47 changes: 30 additions & 17 deletions docs/readme.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,39 +13,52 @@ Omnipy is a type-driven Python library for:
**Dataflows, Not Workflows**

Traditional workflows rely on command-line tools and intermediate files, adding complexity to data
pipelines. Omnipy offers dataflows that operate directly on data in memory or standard formats like
JSON or CSV. By extending Pydantic models, it simplifies parsing, conversion, and serialization with
structured, reusable models.
pipelines. Omnipy offers dataflows that operate directly on data in memory, or on standard formats
like JSON or CSV if there is a need. It expands the functionality of Pydantic data models to
simplify parsing, conversion, and serialization of data.

**"It's Static Typing!"… "It's Dynamic!"… "It's Omnipy!"**

Omnipy blends Python’s dynamic typing with runtime type safety. Its models behave like native Python
structures, ensuring type adherence while allowing flexibility. You get type guarantees without the
rigidity of static typing, and models are as general or specific as you need.
structures, but with a bonus: you get type guarantees without the rigidity of static typing! As
Omnipy models are defined through normal Python code, they can be as general or specific as you
need.

**Parse, Don’t Validate**

Strict validation often breaks pipelines when dealing with messy data. Inspired by the "Parse, Don’t
Validate" principle, Omnipy dynamically transforms imperfect input into structured output, following
the Robustness Principle: "be liberal in what you accept, and conservative in what you send."
Strict validation often breaks pipelines when dealing with messy data. Inspired by the newer
slogan "Parse, Don’t Validate", Omnipy data models dynamically transforms imperfect input into
structured output that stays structured. This approach also lines up the well-known Robustness
Principle: "be liberal in what you accept, and conservative in what you send."

**Self-Constraining Data Models**

Omnipy models aren’t just one-time validators—they enforce data integrity dynamically. A
`Model[list[int]]()` behaves like a list, but every modification ensures type safety, rolling back
to a valid state if needed.
`Model[list[int]]()` behaves like a list, but one that only contains integers. Every modification
parses data and ensures type safety, rolling back to a valid state if needed.

**Modular, Reusable, and Scalable**
**Omnify Your Data Pipelines**

Omnipy encourages modular pipelines built from interoperable components. Dataflows, tasks, and
models are designed for reuse and extensibility, whether in research or general analytics. With
seamless integration with tools like Pandas and NumPy, Omnipy simplifies everything from parsing to
advanced data wrangling.
Omnipy invites you to “omnify” your pipelines — break them into reusable, universal components. With
dataflows defined by structured input and output models, Omnipy simplifies code reuse, encourages
modularity, and promotes a puzzle-piece approach to building robust and extensible pipelines.
Omnipy's structure-by-default approach encourages good coding practices such as "design by
contract" and "separation of concerns", making your code more maintainable as projects grow.

**Catalog of Components for Interoperability**

Omnipy includes ready-to-use components for tasks like asynchronous API requests with built-in rate
limiting, parsing JSON and tabular file formats, and flattening nested data into relational tables.
It allows seamless integration with other tools and services, such as Pandas, making it easy to
tackle data wrangling and interoperability challenges with minimal effort. The catalog will expand
as the Omnipy community grows.

**Built to Scale**

With built-in support for Prefect, Omnipy scales effortlessly from local experimentation to
distributed pipelines, making it perfect for projects of any size.
Omnipy’s hierarchical `Dataset` structure simplifies batch processing. With features like batch
parsing, serialization, and metadata handling, you can effortlessly scale from small datasets to
large directory-based pipelines. With built-in support for Prefect, Omnipy scales effortlessly from
local experimentation to distributed deployment, making it perfect for projects of any size.

## Generic functionality

Expand Down

0 comments on commit 3cf33df

Please sign in to comment.