Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[perflint] implement quick-fix for manual-list-comprehension (PERF401) #13919

Merged
merged 24 commits into from
Nov 11, 2024

Conversation

w0nder1ng
Copy link
Contributor

Summary

I implemented a fix to rewrite for-loops where PERF401 applies into list.extends. Since the target is guaranteed to be a list, and the lint (seemingly) guarantees that the iterator doesn't reference itself, this should be valid for all for-loops where the lint is applicable. It would be nice to implement this to rewrite the entire for-loop/variable assignment into a single list comprehensions, but I thought of several cases where this would affect how the code works, so this would need more consideration to write.

Test Plan

I tried this on local files and it seemed to work, but I couldn't find where tests for ruff check --fix lints go. I'd be happy to add tests if someone points me in the right direction.

@MichaReiser
Copy link
Member

MichaReiser commented Oct 25, 2024

Nice, thank you.
Would you mind adding a few tests showing the fix? Are there cases where a comprehension would be preferred over
list.extend. If so, then I don't think we should provide a code fix.

@MichaReiser MichaReiser added the fixes Related to suggested fixes for violations label Oct 25, 2024
@w0nder1ng
Copy link
Contributor Author

Again, I'm not sure where to add tests for fixes.

@MichaReiser
Copy link
Member

You can extend the existing tests. The testing framework runs your lint rule against the file and snapshots all created diagnostics with their fixes.

#[test_case(Rule::ManualListComprehension, Path::new("PERF401.py"))]

Copy link
Contributor

github-actions bot commented Oct 28, 2024

ruff-ecosystem results

Linter (stable)

ℹ️ ecosystem check detected linter changes. (+49 -49 violations, +0 -0 fixes in 4 projects; 50 projects unchanged)

apache/airflow (+31 -31 violations, +0 -0 fixes)

ruff check --no-cache --exit-zero --ignore RUF9 --output-format concise --no-preview --select ALL

+ dev/breeze/src/airflow_breeze/commands/ci_commands.py:374:25: PERF401 Use `list.extend` to create a transformed list
- dev/breeze/src/airflow_breeze/commands/ci_commands.py:374:25: PERF401 Use a list comprehension to create a transformed list
+ dev/breeze/src/airflow_breeze/commands/release_management_commands.py:3055:9: PERF401 Use `list.extend` to create a transformed list
- dev/breeze/src/airflow_breeze/commands/release_management_commands.py:3055:9: PERF401 Use a list comprehension to create a transformed list
+ dev/breeze/src/airflow_breeze/commands/sbom_commands.py:973:13: PERF401 Use `list.extend` to create a transformed list
- dev/breeze/src/airflow_breeze/commands/sbom_commands.py:973:13: PERF401 Use a list comprehension to create a transformed list
+ dev/breeze/src/airflow_breeze/params/shell_params.py:385:13: PERF401 Use `list.extend` to create a transformed list
- dev/breeze/src/airflow_breeze/params/shell_params.py:385:13: PERF401 Use a list comprehension to create a transformed list
+ dev/breeze/src/airflow_breeze/utils/exclude_from_matrix.py:32:9: PERF401 Use `list.extend` to create a transformed list
- dev/breeze/src/airflow_breeze/utils/exclude_from_matrix.py:32:9: PERF401 Use a list comprehension to create a transformed list
+ dev/breeze/src/airflow_breeze/utils/packages.py:326:9: PERF401 Use `list.extend` to create a transformed list
- dev/breeze/src/airflow_breeze/utils/packages.py:326:9: PERF401 Use a list comprehension to create a transformed list
+ dev/breeze/src/airflow_breeze/utils/reproducible.py:124:17: PERF401 Use `list.extend` to create a transformed list
- dev/breeze/src/airflow_breeze/utils/reproducible.py:124:17: PERF401 Use a list comprehension to create a transformed list
+ dev/breeze/src/airflow_breeze/utils/selective_checks.py:1073:17: PERF401 Use `list.extend` to create a transformed list
- dev/breeze/src/airflow_breeze/utils/selective_checks.py:1073:17: PERF401 Use a list comprehension to create a transformed list
+ docs/exts/docs_build/fetch_inventories.py:103:9: PERF401 Use `list.extend` to create a transformed list
- docs/exts/docs_build/fetch_inventories.py:103:9: PERF401 Use a list comprehension to create a transformed list
+ docs/exts/docs_build/fetch_inventories.py:111:9: PERF401 Use `list.extend` to create a transformed list
- docs/exts/docs_build/fetch_inventories.py:111:9: PERF401 Use a list comprehension to create a transformed list
+ docs/exts/docs_build/fetch_inventories.py:119:9: PERF401 Use `list.extend` to create a transformed list
- docs/exts/docs_build/fetch_inventories.py:119:9: PERF401 Use a list comprehension to create a transformed list
+ providers/src/airflow/providers/amazon/aws/hooks/s3.py:486:21: PERF401 Use `list.extend` to create a transformed list
- providers/src/airflow/providers/amazon/aws/hooks/s3.py:486:21: PERF401 Use a list comprehension to create a transformed list
+ providers/src/airflow/providers/amazon/aws/hooks/s3.py:669:21: PERF401 Use `list.extend` to create a transformed list
- providers/src/airflow/providers/amazon/aws/hooks/s3.py:669:21: PERF401 Use a list comprehension to create a transformed list
+ providers/src/airflow/providers/amazon/aws/log/s3_task_handler.py:122:17: PERF401 Use `list.extend` to create a transformed list
- providers/src/airflow/providers/amazon/aws/log/s3_task_handler.py:122:17: PERF401 Use a list comprehension to create a transformed list
+ providers/src/airflow/providers/amazon/aws/operators/sagemaker.py:1210:17: PERF401 Use `list.extend` to create a transformed list
- providers/src/airflow/providers/amazon/aws/operators/sagemaker.py:1210:17: PERF401 Use a list comprehension to create a transformed list
+ providers/src/airflow/providers/amazon/aws/operators/sagemaker.py:379:17: PERF401 Use `list.extend` to create a transformed list
... 31 additional changes omitted for project

apache/superset (+11 -11 violations, +0 -0 fixes)

ruff check --no-cache --exit-zero --ignore RUF9 --output-format concise --no-preview --select ALL

+ scripts/benchmark_migration.py:128:21: PERF401 Use `list.extend` to create a transformed list
- scripts/benchmark_migration.py:128:21: PERF401 Use a list comprehension to create a transformed list
+ superset/db_engine_specs/base.py:107:9: PERF401 Use `list.extend` to create a transformed list
- superset/db_engine_specs/base.py:107:9: PERF401 Use a list comprehension to create a transformed list
+ superset/tasks/cache.py:192:17: PERF401 Use `list.extend` to create a transformed list
- superset/tasks/cache.py:192:17: PERF401 Use a list comprehension to create a transformed list
+ superset/utils/core.py:1184:17: PERF401 Use `list.extend` to create a transformed list
- superset/utils/core.py:1184:17: PERF401 Use a list comprehension to create a transformed list
+ tests/integration_tests/charts/api_tests.py:353:13: PERF401 Use `list.extend` to create a transformed list
- tests/integration_tests/charts/api_tests.py:353:13: PERF401 Use a list comprehension to create a transformed list
+ tests/integration_tests/charts/api_tests.py:467:13: PERF401 Use `list.extend` to create a transformed list
... 11 additional changes omitted for project

bokeh/bokeh (+7 -7 violations, +0 -0 fixes)

ruff check --no-cache --exit-zero --ignore RUF9 --output-format concise --no-preview --select ALL

+ src/bokeh/layouts.py:475:25: PERF401 Use `list.extend` to create a transformed list
- src/bokeh/layouts.py:475:25: PERF401 Use a list comprehension to create a transformed list
+ src/bokeh/plotting/_figure.py:479:17: PERF401 Use `list.extend` to create a transformed list
- src/bokeh/plotting/_figure.py:479:17: PERF401 Use a list comprehension to create a transformed list
+ src/bokeh/plotting/_figure.py:485:17: PERF401 Use `list.extend` to create a transformed list
- src/bokeh/plotting/_figure.py:485:17: PERF401 Use a list comprehension to create a transformed list
+ src/bokeh/server/contexts.py:310:17: PERF401 Use `list.extend` to create a transformed list
- src/bokeh/server/contexts.py:310:17: PERF401 Use a list comprehension to create a transformed list
+ src/bokeh/settings.py:780:21: PERF401 Use `list.extend` to create a transformed list
- src/bokeh/settings.py:780:21: PERF401 Use a list comprehension to create a transformed list
... 4 additional changes omitted for project

pandas-dev/pandas (+0 -0 violations, +0 -0 fixes)


Changes by rule (1 rules affected)

code total + violation - violation + fix - fix
PERF401 98 49 49 0 0

Linter (preview)

ℹ️ ecosystem check detected linter changes. (+49 -49 violations, +0 -0 fixes in 3 projects; 51 projects unchanged)

apache/airflow (+31 -31 violations, +0 -0 fixes)

ruff check --no-cache --exit-zero --ignore RUF9 --output-format concise --preview --select ALL

+ dev/breeze/src/airflow_breeze/commands/ci_commands.py:374:25: PERF401 Use `list.extend` to create a transformed list
- dev/breeze/src/airflow_breeze/commands/ci_commands.py:374:25: PERF401 Use a list comprehension to create a transformed list
+ dev/breeze/src/airflow_breeze/commands/release_management_commands.py:3055:9: PERF401 Use `list.extend` to create a transformed list
- dev/breeze/src/airflow_breeze/commands/release_management_commands.py:3055:9: PERF401 Use a list comprehension to create a transformed list
+ dev/breeze/src/airflow_breeze/commands/sbom_commands.py:973:13: PERF401 Use `list.extend` to create a transformed list
- dev/breeze/src/airflow_breeze/commands/sbom_commands.py:973:13: PERF401 Use a list comprehension to create a transformed list
+ dev/breeze/src/airflow_breeze/params/shell_params.py:385:13: PERF401 Use `list.extend` to create a transformed list
- dev/breeze/src/airflow_breeze/params/shell_params.py:385:13: PERF401 Use a list comprehension to create a transformed list
+ dev/breeze/src/airflow_breeze/utils/exclude_from_matrix.py:32:9: PERF401 Use `list.extend` to create a transformed list
- dev/breeze/src/airflow_breeze/utils/exclude_from_matrix.py:32:9: PERF401 Use a list comprehension to create a transformed list
+ dev/breeze/src/airflow_breeze/utils/packages.py:326:9: PERF401 Use `list.extend` to create a transformed list
- dev/breeze/src/airflow_breeze/utils/packages.py:326:9: PERF401 Use a list comprehension to create a transformed list
+ dev/breeze/src/airflow_breeze/utils/reproducible.py:124:17: PERF401 Use `list.extend` to create a transformed list
- dev/breeze/src/airflow_breeze/utils/reproducible.py:124:17: PERF401 Use a list comprehension to create a transformed list
+ dev/breeze/src/airflow_breeze/utils/selective_checks.py:1073:17: PERF401 Use `list.extend` to create a transformed list
- dev/breeze/src/airflow_breeze/utils/selective_checks.py:1073:17: PERF401 Use a list comprehension to create a transformed list
+ docs/exts/docs_build/fetch_inventories.py:103:9: PERF401 Use `list.extend` to create a transformed list
- docs/exts/docs_build/fetch_inventories.py:103:9: PERF401 Use a list comprehension to create a transformed list
+ docs/exts/docs_build/fetch_inventories.py:111:9: PERF401 Use `list.extend` to create a transformed list
- docs/exts/docs_build/fetch_inventories.py:111:9: PERF401 Use a list comprehension to create a transformed list
+ docs/exts/docs_build/fetch_inventories.py:119:9: PERF401 Use `list.extend` to create a transformed list
- docs/exts/docs_build/fetch_inventories.py:119:9: PERF401 Use a list comprehension to create a transformed list
+ providers/src/airflow/providers/amazon/aws/hooks/s3.py:486:21: PERF401 Use `list.extend` to create a transformed list
- providers/src/airflow/providers/amazon/aws/hooks/s3.py:486:21: PERF401 Use a list comprehension to create a transformed list
+ providers/src/airflow/providers/amazon/aws/hooks/s3.py:669:21: PERF401 Use `list.extend` to create a transformed list
- providers/src/airflow/providers/amazon/aws/hooks/s3.py:669:21: PERF401 Use a list comprehension to create a transformed list
+ providers/src/airflow/providers/amazon/aws/log/s3_task_handler.py:122:17: PERF401 Use `list.extend` to create a transformed list
- providers/src/airflow/providers/amazon/aws/log/s3_task_handler.py:122:17: PERF401 Use a list comprehension to create a transformed list
+ providers/src/airflow/providers/amazon/aws/operators/sagemaker.py:1210:17: PERF401 Use `list.extend` to create a transformed list
- providers/src/airflow/providers/amazon/aws/operators/sagemaker.py:1210:17: PERF401 Use a list comprehension to create a transformed list
+ providers/src/airflow/providers/amazon/aws/operators/sagemaker.py:379:17: PERF401 Use `list.extend` to create a transformed list
... 31 additional changes omitted for project

apache/superset (+11 -11 violations, +0 -0 fixes)

ruff check --no-cache --exit-zero --ignore RUF9 --output-format concise --preview --select ALL

+ scripts/benchmark_migration.py:128:21: PERF401 Use `list.extend` to create a transformed list
- scripts/benchmark_migration.py:128:21: PERF401 Use a list comprehension to create a transformed list
+ superset/db_engine_specs/base.py:107:9: PERF401 Use `list.extend` to create a transformed list
- superset/db_engine_specs/base.py:107:9: PERF401 Use a list comprehension to create a transformed list
+ superset/tasks/cache.py:192:17: PERF401 Use `list.extend` to create a transformed list
- superset/tasks/cache.py:192:17: PERF401 Use a list comprehension to create a transformed list
+ superset/utils/core.py:1184:17: PERF401 Use `list.extend` to create a transformed list
- superset/utils/core.py:1184:17: PERF401 Use a list comprehension to create a transformed list
+ tests/integration_tests/charts/api_tests.py:353:13: PERF401 Use `list.extend` to create a transformed list
- tests/integration_tests/charts/api_tests.py:353:13: PERF401 Use a list comprehension to create a transformed list
+ tests/integration_tests/charts/api_tests.py:467:13: PERF401 Use `list.extend` to create a transformed list
... 11 additional changes omitted for project

bokeh/bokeh (+7 -7 violations, +0 -0 fixes)

ruff check --no-cache --exit-zero --ignore RUF9 --output-format concise --preview --select ALL

+ src/bokeh/layouts.py:475:25: PERF401 Use `list.extend` to create a transformed list
- src/bokeh/layouts.py:475:25: PERF401 Use a list comprehension to create a transformed list
+ src/bokeh/plotting/_figure.py:479:17: PERF401 Use `list.extend` to create a transformed list
- src/bokeh/plotting/_figure.py:479:17: PERF401 Use a list comprehension to create a transformed list
+ src/bokeh/plotting/_figure.py:485:17: PERF401 Use `list.extend` to create a transformed list
- src/bokeh/plotting/_figure.py:485:17: PERF401 Use a list comprehension to create a transformed list
+ src/bokeh/server/contexts.py:310:17: PERF401 Use `list.extend` to create a transformed list
- src/bokeh/server/contexts.py:310:17: PERF401 Use a list comprehension to create a transformed list
+ src/bokeh/settings.py:780:21: PERF401 Use `list.extend` to create a transformed list
- src/bokeh/settings.py:780:21: PERF401 Use a list comprehension to create a transformed list
... 4 additional changes omitted for project

Changes by rule (1 rules affected)

code total + violation - violation + fix - fix
PERF401 98 49 49 0 0

@w0nder1ng
Copy link
Contributor Author

One thing that I wasn't sure how to handle is cases where there is control flow between the binding and the loop.

def f(early_return):
    result = []
    if early_return:
        return
    for i in range(10):
        result.append(i+1)

Currently, this fix offers to rewrite the code:

def f(early_return):
    result = [i+1 for i in range(10)]
    if early_return:
        return

How do you think this case ought to be handled?

@w0nder1ng w0nder1ng changed the title [PERF401] implement quick-fix for manual-list-comprehension (PERF401) [perflint] implement quick-fix for manual-list-comprehension (PERF401) Oct 28, 2024
Copy link
Member

@MichaReiser MichaReiser left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks great. Thanks for working on this fix.

I have a few smaller code improvements that are up to you. The one change we should make is to gate the fix behind preview to give us a chance to catch any errors before rolling it out to all users.

#[derive_message_formats]
fn message(&self) -> String {
let ManualListComprehension { is_async } = self;
let ManualListComprehension { is_async, .. } = self;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's adjust the message to make use of the new comprehension_type as well to avoid inconsistency between the message and the fix title.

@w0nder1ng
Copy link
Contributor Author

I've modified the code to preserve comments and not leave a newline when removing the for loop. The tests seem to be failing because the fix is now gated behind a preview--is there a way to allow the test to set the preview flag, or should I temporarily change the fix availability to "sometimes"?

@dhruvmanila
Copy link
Member

The tests seem to be failing because the fix is now gated behind a preview--is there a way to allow the test to set the preview flag, or should I temporarily change the fix availability to "sometimes"?

@w0nder1ng You'll need to add a new test case where the preview flag is enabled. Refer to the following as an example:

#[test_case(Rule::ZipInsteadOfPairwise, Path::new("RUF007.py"))]
fn preview_rules(rule_code: Rule, path: &Path) -> Result<()> {
let snapshot = format!(
"preview__{}_{}",
rule_code.noqa_code(),
path.to_string_lossy()
);
let diagnostics = test_path(
Path::new("ruff").join(path).as_path(),
&settings::LinterSettings {
preview: PreviewMode::Enabled,
..settings::LinterSettings::for_rule(rule_code)
},
)?;
assert_messages!(snapshot, diagnostics);
Ok(())
}

@w0nder1ng
Copy link
Contributor Author

How should I deal with the existing test case?

@MichaReiser
Copy link
Member

How should I deal with the existing test case?

You would have to mark the rule as sometimes fixable. I suggest to add a TODO comment to the preview check that says that the rule should be changed to always fixable when promoting the fix to stable.

Copy link
Member

@MichaReiser MichaReiser left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is great

@w0nder1ng
Copy link
Contributor Author

I changed the diagnostic message, does the new one make more sense? Also, do you know why the tests aren't running?

@dhruvmanila
Copy link
Member

@w0nder1ng there are merge conflicts which needs to be resolved to get the CI running

@MichaReiser MichaReiser added the preview Related to preview mode features label Nov 8, 2024
Copy link
Member

@MichaReiser MichaReiser left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is great and very impressive comment handling!

I pushed a small change that uses the improved message even in stable mode

@MichaReiser
Copy link
Member

Hmm, what's up with the ecosystem check... Let's re-run to get a better sense of the changes

@MichaReiser
Copy link
Member

I suspect that the ecosystem check is panicking during the fix generation. Let me run it locally

Copy link
Member

@MichaReiser MichaReiser left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here's an example that currently panics:

# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements.  See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership.  The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License.  You may obtain a copy of the License at
#
#   http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied.  See the License for the
# specific language governing permissions and limitations
# under the License.
"""migrate_old_annotation_layers

Revision ID: 21e88bc06c02
Revises: 67a6ac9b727b
Create Date: 2017-12-17 11:06:30.180267

"""

from alembic import op
from sqlalchemy import Column, Integer, or_, String, Text
from sqlalchemy.ext.declarative import declarative_base

from superset import db
from superset.utils import json

# revision identifiers, used by Alembic.
revision = "21e88bc06c02"
down_revision = "67a6ac9b727b"

Base = declarative_base()


class Slice(Base):
    __tablename__ = "slices"
    id = Column(Integer, primary_key=True)
    viz_type = Column(String(250))
    params = Column(Text)


def upgrade():
    bind = op.get_bind()
    session = db.Session(bind=bind)

    for slc in session.query(Slice).filter(
        or_(Slice.viz_type.like("line"), Slice.viz_type.like("bar"))
    ):
        params = json.loads(slc.params)
        layers = params.get("annotation_layers", [])
        if layers:
            new_layers = []
            for layer in layers:
                new_layers.append(
                    {
                        "annotationType": "INTERVAL",
                        "style": "solid",
                        "name": f"Layer {layer}",
                        "show": True,
                        "overrides": {"since": None, "until": None},
                        "value": layer,
                        "width": 1,
                        "sourceType": "NATIVE",
                    }
                )
            params["annotation_layers"] = new_layers
            slc.params = json.dumps(params)
            session.commit()
    session.close()


def downgrade():
    bind = op.get_bind()
    session = db.Session(bind=bind)

    for slc in session.query(Slice).filter(
        or_(Slice.viz_type.like("line"), Slice.viz_type.like("bar"))
    ):
        params = json.loads(slc.params)
        layers = params.get("annotation_layers", [])
        if layers:
            params["annotation_layers"] = [layer["value"] for layer in layers]
            slc.params = json.dumps(params)
            session.commit()
    session.close()

@w0nder1ng could you take a look at what's the source of the panic?

@w0nder1ng
Copy link
Contributor Author

The issue was that having no comments inside the for loop caused an empty insert, and that made a debug assert panic. The reason I didn't catch it before was because every test case for the fix has # PERF401 following the lint target! I added a new test case to hopefully spot this type of error.

@MichaReiser
Copy link
Member

Ah, that makes sense. This is great. Thank you so much for working on the fix

@MichaReiser MichaReiser merged commit 5a3886c into astral-sh:main Nov 11, 2024
20 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
fixes Related to suggested fixes for violations preview Related to preview mode features
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants