Skip to content

Releases: mage-ai/mage-ai

0.9.74 | KAOS

19 Sep 18:55
@wangxiaoyou1993 wangxiaoyou1993
0.9.74
a64e162
This commit was created on github.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Learn about vigilant mode.
Compare
Choose a tag to compare
0.9.74 | KAOS Latest
Latest

What's Changed

🎉 Exciting New Features

☁️ Google cloud storage source

You can now effortlessly configure your data integration source blocks to Google Cloud Storage using this snippet. It specifies the destination bucket, sets the file type to Parquet, and provides the path to your service account credentials needed for authentication.

by @TalaatHasanin in #5334

🌬️ 🍽️ Airtable Integration

Effortlessly streamline your data data integration source blocks with Airtable using this tailored script. It establishes a connection to your Airtable base, designates the specific table for integration, and securely integrates your Airtable API key for access.

by @TalaatHasanin in #5404

✂️ Add trim reformat action to transformer block

A new "trim reformat" action has been added to the Python transformer block, allowing for the removal of leading and trailing whitespace from specified text columns. This enhancement ensures cleaner and more consistent data formatting by automatically stripping unnecessary spaces around text entries.

by @cristopheridlc in #5321

🐻‍❄️ Enable polars dataframe in GCS data exporter

This update enhances the export method of the GCS IO module, expanding its functionality to support exporting Polars DataFrames in addition to the previously supported Pandas DataFrames. This new capability allows users to seamlessly work with both data formats, offering greater flexibility in managing and exporting data from their workflows.

by @TalaatHasanin in #5348

Add support to scheduler name on k8s executor

This update adds the ability to customize Kubernetes scheduler options when using the Kubernetes Executor. Users can now directly configure scheduler settings, providing increased flexibility and control over how pods are scheduled within Kubernetes environments.

by @messerzen in #5412

🐛 Bug Fixes

💅 Enhancements & Polish

Other Changes

New Contributors

Full Changelog: 0.9.73...0.9.74

Contributors

tommydangerous, matrixstone, and 15 other contributors
Assets 2
Loading

0.9.73 | Deadpool & Wolverine 🥷🐺

02 Aug 14:59
@thomaschung408 thomaschung408
0.9.73
1de0a45
This commit was created on github.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Learn about vigilant mode.
Compare
Choose a tag to compare
0.9.73 | Deadpool & Wolverine 🥷🐺

What's Changed

🎉 Exciting New Features

🧠 Memory management upgrade

This feature enhances system performance and stability by optimizing allocation, deallocation, and overall utilization of RAM resources, resulting in improved efficiency and reduced memory-related errors.

Check out the doc for details.


🔄 Dynamic blocks 2.0

Introducing an enhanced Dynamic blocks feature for creating dynamic content blocks that can adapt based on user input or data changes, offering improved flexibility and interactivity for users. This update aims to streamline workflows and enhance the overall user experience by making content more responsive and customizable.

Learn more in this doc.

🔐 Azure DB connection via Key Vault

Securely retrieve a database connection URL stored in Azure Key Vault using environment variables for authentication.

AZURE_KEY_VAULT_URL
AZURE_CLIENT_ID
AZURE_CLIENT_SECRET
AZURE_TENANT_ID

by @wangxiaoyou1993 in #5302

👁️ Workspace monitoring

Add an "Overview" page and "Pipeline runs" page to the Workspace Management UI. This provides some overall monitoring for all of the pipeline runs in the different workspaces without having to individually open up each workspace instance.


by @johnson-mage in #5311

🐛 Bug Fixes

💅 Enhancements & Polish

✨ Other Changes

🆕 New Contributors

  • @tk26 made their first contribution in #5235
  • @gracia10 made their first contribution in #5218
  • @github-actions made their first contribution in #5318

Full Changelog: 0.9.72...0.9.73

Contributors

tommydangerous, tk26, and 4 other contributors
Assets 2
Loading

0.9.72 | House Targaryen 🐉

21 Jun 14:32
@thomaschung408 thomaschung408
0.9.72
553f569
This commit was created on github.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Learn about vigilant mode.
Compare
Choose a tag to compare
0.9.72 | House Targaryen 🐉

What's Changed

🎉 Exciting New Features

🎨 Canvas v2

Navigate large or complex layouts with a new canvas that allows developers to zoom, pan, drag, and drop visual features.

by @tommydangerous in #5193

🐍 Support custom python sink in Streaming pipeline

Support for custom Python sinks in streaming pipelines allows users to write their own streaming sink implementations, providing greater flexibility in how processed data is handled at the end of the pipeline.

custom python sink

by @lgarcia11 in #5174

Allow configuring custom owner credentials

This feature allows users to configure custom owner credentials for their application by setting specific environment variables:

DEFAULT_OWNER_EMAIL
DEFAULT_OWNER_PASSWORD
DEFAULT_OWNER_USERNAME

by @wangxiaoyou1993 in #5165

🐛 Bug Fixes

💅 Enhancements & Polish

✨ Other Changes

🆕 New Contributors

Full Changelog: 0.9.71...0.9.72

Contributors

tommydangerous, hugabora, and 6 other contributors
Assets 2
Loading

0.9.71 | X-Men ❌

31 May 22:55
@wangxiaoyou1993 wangxiaoyou1993
0.9.71
afb9e37
This commit was created on github.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Learn about vigilant mode.
Compare
Choose a tag to compare
0.9.71 | X-Men ❌

What's Changed

🎉 Exciting New Features

📡 Add Streaming Oracle Destination

This feature adds a new custom exporter option for OracleDB allowing developers to easily export their streaming data to Oracle DB.

oracledb-exporter

by @matrixstone in #4896

👤 Multi Project Platform Support

This feature adds support for each user to have their own active project within a multi project platform that keeps track of every project for every user.

by @dy46 in #4865

⚙️ Support Job Parameters in K8s Executor

Adds support to help manage environment variables, and control job execution and cleanup policies within the Kubernetes cluster.

k8s_executor_config:
  container_config:
    image: mageai/mageai:0.9.7
    env:
    - name: USER_CODE_PATH
      value: /home/src/k8s_project
  job:
    active_deadline_seconds: 120
    backoff_limit: 3
    ttl_seconds_after_finished: 86400

by @artche in #5044

🔑 Support OIDC_ROLES_MAPPING and user_roles

This feature is used to map roles obtained from an OpenID Connect (OIDC) identity provider to roles used within your application allowing for a standardized way to manage user roles across different identity providers.

by @mihaivint in #5053 and #4899

💅 Enhancements & Polish

🐛 Bug Fixes

✨ New Contributors

Full Changelog: 0.9.70...0.9.71

Contributors

tommydangerous, richardlhughes, and 13 other contributors
Assets 2
Loading

0.9.70 | Fallout ☢️

26 Apr 23:53
@wangxiaoyou1993 wangxiaoyou1993
0.9.70
1e805ef
This commit was created on github.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Learn about vigilant mode.
Compare
Choose a tag to compare
0.9.70 | Fallout ☢️

What's Changed

🎉 Exciting New Features

💥 Add "Last run failed" filter to Pipeline Runs table

This feature adds a new filter option to the Pipeline Runs table, allowing users to easily view and filter pipeline runs that have failed in their most recent execution. This makes it simpler to identify and troubleshoot problematic pipelines.
pipeline runs page

by @johnson-mage in #4926

🛠️ Add workspace update features

This feature allows users to update the workspace settings (environment variables) and update the workspace main container config from the UI. Being able to update the container config will allow users to update the workspace image as well.


by @dy46 in #4951

🔄 Add upsert to MySQL, BQ, and Snowflake

Add ability to upsert data in MySQL, BigQuery, and Snowflake IO classes.
It can be achieved by specifying the following params in the export method:

            unique_conflict_method='UPDATE',
            unique_constraints=['col'],

by @dy46 in #4769

⚠️ Support alerts and limited retries in streaming pipeline

Streaming pipeline used to be retried automatically and infinitely.
With this feature, when retry_config is set in streaming pipeline's metadata.yaml at pipeline level, only retry the streaming pipeline with limited times and send alerts when pipeline fails.
Example retry config

retry_config:
  # Number of retry times
  retries: 3
  # Initial delay (in seconds) before retry. If exponential_backoff is true,
  # the delay time is multiplied by 2 for the next retry
  delay: 5
  # Maximum time between the first attempt and the last retry
  max_delay: 60
  # Whether to use exponential backoff retry
  exponential_backoff: true

by @wangxiaoyou1993 in #4902

🔑 Add setting to update user roles on login

This feature adds the UPDATE_ROLES_ON_LOGIN setting which will tell Mage to update the roles every time a user logs in using LDAP or OAuth (that supports roles mapping). If LDAP or OAuth returns no roles, the roles for the user will not be updated.

by @dy46 in #4905

🐛 Bug Fixes

💅 Enhancements & Polish

Read more

Contributors

lqc, tommydangerous, and 10 other contributors
Assets 2
Loading

0.9.68 | Invincible 🦸‍♂️

28 Mar 15:35
@wangxiaoyou1993 wangxiaoyou1993
0.9.68
8ddf586
This commit was created on github.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Learn about vigilant mode.
Compare
Choose a tag to compare
0.9.68 | Invincible 🦸‍♂️

What's Changed

🎉 Exciting New Features

🐍 Custom python source in Streaming pipeline

This feature allows users to use custom Python source code in their Streaming pipelines, which provides more flexibility and control for advanced users who need to incorporate custom logic or functionality into their streaming pipelines. Here is the code template:

from mage_ai.streaming.sources.base_python import BasePythonSource
from typing import Callable

if 'streaming_source' not in globals():
    from mage_ai.data_preparation.decorators import streaming_source


@streaming_source
class CustomSource(BasePythonSource):
    def init_client(self):
        """
        Implement the logic of initializing the client.
        """

    def batch_read(self, handler: Callable):
        """
        Batch read the messages from the source and use handler to process the messages.
        """
        while True:
            records = []
            # Implement the logic of fetching the records
            if len(records) > 0:
                handler(records)

by @wangxiaoyou1993 in #4763

📦 OracleDB exporter

Add OracleDB exporter template to standard batch pipeline.
image

by @matrixstone in #4785

🔧 Server logging format template overriding

This feature enables users to override the default server logging format template in Mage. This provides more flexibility in customizing the logging output, which can be useful for troubleshooting, monitoring, or integrating with external logging systems.
Here is an example setting:

export SERVER_LOGGING_TEMPLATE=$'%(asctime)s\t[%(name)25.25s]\t%(levelname)5s: %(message)s'

by @afal007 in #4772

🔄 Update pipeline name before pipeline creation

Allow user to edit the pipeline name and add a description/tags before creating a new pipeline. Previously, a randomized name was always used for new pipelines, and then users had to go to the pipeline's settings to update the name.
image

by @johnson-mage in #4790

🐛 Bug Fixes

💅 Enhancements & Polish

✨ New Contributors

Full Changelog: 0.9.66...0.9.68

Contributors

matrixstone, afal007, and 4 other contributors
Assets 2
Loading

0.9.66 | Shogun 🎌

14 Mar 00:08
@wangxiaoyou1993 wangxiaoyou1993
0.9.66
b3f8e23
This commit was created on github.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Learn about vigilant mode.
Compare
Choose a tag to compare
0.9.66 | Shogun 🎌

What's Changed

🎉 Exciting New Features

🗑️ Delete backfills

  • Add delete button to backfills table that allows user to delete backfills individually.
  • Prevent user from setting a backfill interval unit value less than 1, which could cause issues loading the backfills.
    image

by @johnson-mage in #4726

🪪 Support AzureDevOps using Entra ID/AD authentication

If you have an Azure DevOps organization connected to your Active Directory/Microsoft Entra ID tenant, you can enable DevOps in the version control app.

  • First, you will need to enable the Active Directory integration in order for Mage to authenticate your user.
  • Once the Active Directory integration is enabled, you simply need to add your DevOps organization as an environment variable, and you should be able to authenticate with DevOps in the version control app
    • export AZURE_DEVOPS_ORGANIZATION=<organization_name>

by @dy46 in #4721

📅  Scheduler status endpoint

Add /api/status endpoint to scheduler when running scheduler in standalone mode.
This endpoint can be used to check the liveness of the scheduler.


by @wangxiaoyou1993 in #4736

🔐 Read authentication settings from AWS secrets manager

Add support for reading settings from AWS secrets manager. The existing authentication settings have been moved to using the new Settings class.

In order to set the settings backend, the user will need to add a section to the project metadata. Here is an example to set the setting backend to AWS secrets manager:

settings_backend:
  backend_type: aws_secrets_manager
  prefix: default_repo/settings/

Read the doc for details.

by @dy46 in #4674

💅 Enhancements & Polish

🐛 Bug Fixes

✨ New Contributors

Full Changelog: 0.9.65...0.9.66

Contributors

Jawahars, ckfear, and 7 other contributors
Assets 2
Loading

0.9.65 | Demon Slayer 🐗 ⚔️

28 Feb 23:59
@wangxiaoyou1993 wangxiaoyou1993
0.9.65
563e3b0
This commit was created on github.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Learn about vigilant mode.
Compare
Choose a tag to compare
0.9.65 | Demon Slayer 🐗 ⚔️

What's Changed

🎉 Exciting New Features

🔔 Telegram notification


Support for setting up Telegram notification channel for alerts is now available.
Check out the doc for details

by @luthfir-rahman in #4632

🔄 Runtime variables and concurrency config in Backfills

Backfills support configuring global variables override and concurrency limit is now available.


by @dy46 in #4618

☁️ Export raw files to S3 and GCS

Using S3 and GCS data exporter templates, you can now directly upload raw files in addition to dataframes.

    bucket_name = 'test-bucket'
    object_key = 'test_file_copy.csv'

    S3.with_config(ConfigFileLoader(config_path, config_profile)).export(
        '/home/src/test_file.csv',
        bucket_name,
        object_key,
    )
    GoogleCloudStorage.with_config(ConfigFileLoader(config_path, config_profile)).export(
        '/home/src/test_file.csv',
        bucket_name,
        object_key,
    )

by @wangxiaoyou1993 in #4607

🐛 Bug Fixes

💅 Enhancements & Polish

New Contributors

Full Changelog: 0.9.64...0.9.65

Contributors

hugabora, carillpower, and 7 other contributors
Assets 2
Loading

0.9.64 | Maestro 🎵

16 Feb 02:43
@wangxiaoyou1993 wangxiaoyou1993
0.9.64
dcb9b0d
This commit was created on github.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Learn about vigilant mode.
Compare
Choose a tag to compare
0.9.64 | Maestro 🎵

What's Changed

🎉 Exciting New Features

🔭 OpenTelemetry integration

  • Added OpenTelemetry Libraries: Integrated OpenTelemetry Python libraries to instrument the application for telemetry data collection.
    • opentelemetry-api: Provides the API for tracing and metrics.
    • opentelemetry-sdk: The core SDK for OpenTelemetry in Python.
    • opentelemetry-instrumentation: General instrumentation library.
    • opentelemetry-instrumentation-sqlalchemy: Specific for instrumenting SQLAlchemy operations.
    • opentelemetry-instrumentation-tornado: For monitoring Tornado-based HTTP server activities.
  • Environment Variables: Added new environment variables (OTEL_EXPORTER_OTLP_HTTP_ENDPOINT and OTEL_EXPORTER_OTLP_ENDPOINT) for configuring the OpenTelemetry Collector endpoints.
  • Database and HTTP Server Instrumentation: Configured instrumentation for both SQLAlchemy database operations and Tornado HTTP server activities to gather traces and metrics.
  • Check out this doc for more details

by @mfreeman451 in #4476

🦊 Add GitLab authentication to version control

Gitlab

Support authenticating with GitLab in version control app.
The feature can be enabled by configuring the following environment variables:

  • GITLAB_HOST
  • GITLAB_CLIENT_ID
  • GITLAB_CLIENT_SECRET

by @dy46 in #4534

🔔 Discord notification integration

Discord

Support setting up Discord notification channel for alerts.
Check out the doc for details

by @luthfir-rahman in #4478

🔑 Add OIDC generic provider

This feature allows users to hook up their OIDC server (e.g. Keycloak ) and use it to authenticate with Mage.
It can be enabled by configuring the following environment variables:

  • OIDC_CLIENT_ID
  • OIDC_CLIENT_SECRET
  • OIDC_DISCOVERY_URL

by @dy46 in #4563

🐛 Bug Fixes

💅 Enhancements & Polish

New Contributors

Full Changelog: 0.9.63...0.9.64

Contributors

mfreeman451, Luishfs, and 4 other contributors
Assets 2
Loading

0.9.63 | Halo 👾

08 Feb 04:06
@wangxiaoyou1993 wangxiaoyou1993
0.9.63
ba3714d
This commit was created on github.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Learn about vigilant mode.
Compare
Choose a tag to compare
0.9.63 | Halo 👾

What's Changed

🎉 Exciting New Features

🦆 MotherDuck Support


This one's for all the ducklings out there! In addition to supporting DuckDB, Mage now supports MotherDuck destinations!

By specifying a MOTHERDUCK_TOKEN and adding a md: prefix to your DuckDB database, you can read/write from/to MotherDuck locations! Check it out and get started here.

by @wangxiaoyou1993 in #4533

🤖 Support Thick mode in OracleDB


We like our OracleDB connections like we like our pancakes, THICK 🥞. By default, Mage's Oracle client runs in a "Thin" mode which connects directly to Oracle Database— this mode does not need Oracle Client libraries. However, some additional functionality is available when they're used.

Now, you can use the "Thick" mode in Mage to connect to OracleDB using the Oracle Client libraries!

Check out our docs to get started or read more about the differences between "Thin" and "Thick" modes here.

by @matrixstone in #4421

🗄️ Show unused block files in file browser

This is one of our most requested features and we're excited to announce that it's finally here! 🎉

You can now see which files are not being used in your pipeline! This is a great way to clean up your projects and remove any unnecessary files. Check out the gif below to see it in action!


Head over to our docs to learn more!

by @johnson-mage in #4449

🤐 Import functionality for pipeline zips

Like to share? So do we! You can now import pipelines (via .zip files) in your Mage projects! We're optimistic that this simple improvement will make it easier to share your pipelines or even borrow pipelines from your friends!


by @johnson-mage in #4453

🐛 Bug Fixes

💅 Enhancements & Polish

  • MSSQL data integration source - add support for DATETIMEOFFSET type by @hugabora in #4499
  • Fix: handle unprivileged user for postgres by @jdbranham in #4357
  • Optimize pipeline schedule queries by @dy46 in #4188
  • Remove pipeline's updated_at attribute by @johnson-mage in #4521
  • Create and view workspaces in different namespaces by @dy46 in #4513
  • Select status for pipeline runs that time out by @dy46 in #4519

New Contributors

Full Changelog: 0.9.62...0.9.63

Contributors

tommydangerous, hugabora, and 8 other contributors
Assets 2
Loading