Include .conda packages

[jakirkham]

It would be helpful to include both .conda & .tar.bz2 packages. Particularly as more of the former and less of the latter are produced. May also help to track these separately to track the transition to the newer format

[jakirkham]

cc @beckermr @wolfv

[jezdez]

Looking into this with @cappadona

[dopplershift]

@jezdez Did that go anywhere? I was working on collecting some download numbers for my library and right now 2023 shows minimal downloads due to the transition to .conda.

[jakirkham]

@jezdez did this issue get solved more broadly?

Saw the python packages were fixed recently: #41

Is there a path for fixing the other packages? Or did this already happen?

[cappadona]

@jakirkham @dopplershift. Apologies for the delay.

We have not yet addressed .conda packages missing from this data set. This work is on our backlog, and we should be able to get this done in November. We will provide updates here, but please don't hesitate to reach out with questions.

[jakirkham]

Thanks Nick! 🙏

[cappadona]

Hi @jakirkham @dopplershift. Quick update on the status of this issue.

We're working on finalizing a new pipeline that will source this public data set and include .conda packages moving forward. We expect to have it ready by the end of March 2024 and will post an update here when it is available.

[leofang]

Hi @cappadona Thanks for the update! Q: Would it be possible to also update the past statistics when the new pipeline is up?

[cappadona]

@leofang At the moment we're not planning to replace any existing files in the bucket and only implement the fix for future data.

[jakirkham]

cc @aterrel @chenghlee (as we discussed this earlier)

[leofang]

Hi @cappadona @jezdez Friendly nudge for updates 🙂 This has impacted several statistics tracking tools and caused confusion. I've heard jabbering about "no one is using conda" as they looked at the download counts from, say, condastats, but it is simply not true.

[cappadona]

Hi @leofang. Thanks for checking in. We are on track to include .conda packages in the dataset by the end of the month.

[jakirkham]

Just wanted to check in, @cappadona how are things looking here?

[wolfv]

Still looks reaaaally flat: https://prefix.dev/channels/conda-forge/packages/aesara (picked a random package)

[jakirkham]

To be fair, Nick said end of the month originally. So end of next week

Though would be good to learn if that is still true or if this is likely to slip

[jakirkham]

@cappadona how are things looking?

[cappadona]

@jakirkham Sorry I missed your earlier message. Thanks for checking in. We're looking good and the March 2024 data published to the s3 bucket later this week will include .conda packages.

I will post an update to this thread once the March data is available.

[jakirkham]

Thanks Nick! 🙏

[cappadona]

Hi all. Quick update. We're just about there. Finalizing QA with the rest of the team, including a colleague who returns next week. Here are a couple examples for March 2024.

[jakirkham]

Thanks Nick! 🙏

With numpy this includes some older versions like 1.9.2, are these coming from defaults? Asking as conda-forge jumped to numpy version 1.9.3 (in the 1.9 series). Or is this an amalgamation of different channel statistics?

aesara is only in conda-forge AFAIK. So am guessing the top sheet is based on conda-forge data. Is that right?

[cappadona]

Hi @jakirkham. The screenshot is an aggregation of multiple channels, which are usually identified in the final dataset via the data_source column. I did confirm that conda-forge is the only data sources for aesara.

[jakirkham]

How are things looking @cappadona ?

[jakirkham]

@cappadona are there any updates here?

Also as a side note, users are also asking about March data in this issue: #51

[cappadona]

Hi @jakirkham. Monthly and hourly data for March and April 2024, which includes .conda packages, are now available in the bucket.

Thank you all for your patience.

[jezdez]

@cappadona Do you think we could update the old files as well, since .conda files had been hosted for a while? Should we keep this ticket open until we fix that?

[wolfv]

So just to get it right, the format of the parquet files changed?

[jakirkham]

Tried generating my own script to parse through the data. Am seeing the following download counts for cudatoolkit (legacy package for CUDA 11 and earlier) and cuda-version (used in CUDA 12 and later)

#!/usr/bin/env python


import packaging
import sys

from packaging.version import InvalidVersion, Version

import matplotlib.pyplot as plt
import pandas as pd


plt.rcParams["figure.figsize"] = (22, 5)


def main(*argv):
    pkgs = [
        ("cudatoolkit", lambda v: Version("11.2") <= v < Version("12")),
        ("cuda-version", lambda v: Version("12") <= v and str(v) != "12.0.0"),
    ]

    for each_pkg, keep_filter in pkgs:
        year = "2024"
        month = "04"
        df = pd.read_parquet(f"{year}-{month}.parquet")

        df_pkg = df[df["pkg_name"] == each_pkg]

        pkg_vers = []
        for v in df_pkg["pkg_version"].unique():
            try:
                v = Version(v)
            except InvalidVersion:
                # Skip invalid version formats
                continue
            pkg_vers.append(v)
        pkg_vers = sorted(pkg_vers)

        pkg_vers_filt = list(filter(keep_filter, pkg_vers))

        df_pkg_sorted = pd.concat(
            [df_pkg[df_pkg["pkg_version"] == str(v)] for v in pkg_vers_filt]
        )

        df_pkg_plot = df_pkg_sorted[["pkg_version", "counts"]]
        df_pkg_plot["counts"] = df_pkg_plot["counts"] / 1e6

        plt.clf()

        plt.bar(df_pkg_plot["pkg_version"], df_pkg_plot["counts"])
        plt.title(f"{each_pkg} versions vs. Downloads (millions) for {year}-{month}")
        plt.xlabel(f"{each_pkg} versions")
        plt.ylabel("Downloads (millions)")

        plt.savefig(f"{each_pkg}_download_count.svg")

    return 0


if __name__ == "__main__":
    sys.exit(main(*sys.argv))

Here are the results it shows (note values are in millions):

Admittedly this is only one month

Plus some packages built with CUDA support link to the driver (like Arrow); so, may not pull in either of these packages at install time (despite building with CUDA support)

Also it would be better to group the cudatoolkit patch versions together like how cuda-version is handled

Nevertheless this is a good rough test of the data. It does seem to be picking up download counts for these packages that were missed in prior months (which had been off by a couple orders of magnitude in the worst case)

Edit: Fix issue where 12.0 got cutoff

[cappadona]

Thanks @jakirkham. May 2024 data was made available this past Saturday, June 1st.

As of today, the .conda packages are included in the data for the following months:

• 2024-03
• 2024-04
• 2024-05

Based on the community feedback thus far, we're considering replacing data for additional prior months, and updating them to also include .conda packages. Stay tuned.

cc @jezdez

[jakirkham]

Based on the community feedback thus far, we're considering replacing data for additional prior months, and updating them to also include .conda packages. Stay tuned.

Thanks Nick! 🙏

This would be incredibly helpful 🙂

[h-vetinari]

It would be amazing to pull these updates back to the introduction of .conda artefacts, both for having a correct history and an accurate total number of downloads. The conda-forge landing page currently prominently displays the latter, and I think we're still not counting over a year of .conda downloads.

If one goes and executes by-the-numbers notebook linked from the conda-forge landing page (with some minor adaptations to update the loop over which years we're interested in), we get the following for 2021-2023:

While there's undoubtedly some variability in the monthly data, to my understanding that sharp drop-off is related to the introduction of .conda around November 2022.

[jezdez]

I agree with @h-vetinari, let’s make this available for the whole time period, doesn’t make sense otherwise IMO.

[wolfv]

Did something happen with the timestamps? For some reason, we seem to have some new entries at "epoch 0" (ie. somewhere in 1970)

I'll delete/filter them from our data but just wanted to check if anyone knows what's up?

[jakirkham]

@cappadona , hope you had a good weekend! 😀

Do you have thoughts on the questions above? To summarize...

1. Can we backport the .conda count fix to earlier dates?
2. Do we know how the timestamps are being created (seeing 1970 references)?

Also would add one more...

3. How are the anaconda.org numbers generated relative to these? Seeing some differences ( Difference in numbers from condastats and anaconda.org conda-incubator/condastats#18 )

Thanks for your help! 🙏

[modouldemba]

June and July data is still not available. Is there an issue? From: jakirkham ***@***.***> Sent: Monday, July 15, 2024 1:07 AM To: ContinuumIO/anaconda-package-data ***@***.***> Cc: Subscribed ***@***.***> Subject: Re: [ContinuumIO/anaconda-package-data] Include `.conda` packages (Issue #45) @cappadona<https://github.com/cappadona> , hope you had a good weekend! 😀 Do you have thoughts on the questions above? To summarize... 1. Can we backport the .conda count fix to earlier dates? 2. Do we know how the timestamps are being created (seeing 1970 references)? Also would add one more... 1. How are the anaconda.org numbers generated relative to these? Seeing some differences ( conda-incubator/condastats#18<conda-incubator/condastats#18> ) Thanks for your help! 🙏 — Reply to this email directly, view it on GitHub<#45 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AZKHWNUDRGW5XPEWEZBUNSTZMN7L5AVCNFSM6AAAAAA3GO2T3WVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMRXHEYTSNJSG4>. You are receiving this because you are subscribed to this thread.Message ID: ***@***.******@***.***>>

[wolfv]

We are removing download counts from our website for now since it doesn't seem to be very reliable and looks just bad :(

[jakirkham]

@cappadona could you please help us with the issues posted above?

Notably:

• Odd 1970 data points
• Prior months with .conda
• Missing recent months (June & July)
• Differences between this data and Anaconda.org's numbers
  • Anything wrong with the data from March to today ? #52
  • Difference in numbers from condastats and anaconda.org conda-incubator/condastats#18

[jezdez]

@jakirkham FTR, this has been prioritized and get more attention again

[jakirkham]

Thanks Jannis! 🙏

Please let us know if you need more info from us or need us to test anything 🙂

[wolfv]

The 1970 issues were actually issues in our code. Sorry about that!

[wolfv]

We just fixed things on our end, but it appaears that the pipeline to produce this data is not really working anymore?

The latest data is 2024-06...

[jezdez]

Huh, I'd check with @cappadona about it, he was working on an analysis

[cappadona]

Hi all. We've been running some analysis on the dataset in response to everyone's feedback and will share our findings when this is complete.

In the interim, responding to some of the recent questions in this thread...

---

@wolfv

We just fixed things on our end, but it appaears that the pipeline to produce this data is not really working anymore?

The latest data is 2024-06...

The latest data available in the s3 bucket is for 2024-05, which was made available in June. We have temporarily paused publishing new data until we complete the QA.

The 1970 issues were actually issues in our code. Sorry about that!

Thank you. This is one issue that we haven't been able to reproduce.

---

@jakirkham @phwuil @nicrie

Notably:

• Odd 1970 data points

False alarm -- addressed by Wolf

• Prior months with .conda

This is the main focus of our QA effort and we're tentatively planning to replace data beginning in 2022-06 to address the undercounting.

• Missing recent months (June & July)

Temporarily paused publishing new data (see my response above)

• Differences between this data and Anaconda.org's numbers
  • Anything wrong with the data from March to today ? #52
  • Difference in numbers from condastats and anaconda.org conda-incubator/condastats#18

We still need to dig into the download counter displayed on anaconda.org. I will also comment on each of those issues.

[wolfv]

We've dropped the faulty data from our end. Any chance you are going to backfill data from the past? it looks pretty weird now, because some packages that had releases only have 1 measuring point.

[wolfv]

[wolfv]

Lastly, while it appears you fixed the .conda, is it possible that .tar.bz2 are not accounted for anymore?

https://prefix.dev/channels/conda-forge/packages/_libgcc_mutex

[cappadona]

We've dropped the faulty data from our end. Any chance you are going to backfill data from the past? it looks pretty weird now, because some packages that had releases only have 1 measuring point.

Hi Wolf, yes we are planning to backfill past data and we will be sharing details at this week's conda community sync.

[cappadona]

Hi @wolfv I'm unable to reproduce this dropoff for _libgcc_mutex when using condastats

[wolfv]

OK, then we might have an issue on our end again :) Thanks!

[jakirkham]

@cappadona is this working correctly for other channels?

Think it would be good to double check these are all handled correctly (others may have suggestions):

• bioconda
• defaults
• nvidia
• pytorch
• rapidsai
• rapidsai-nightly

[jakirkham]

Also worth noting RAPIDS is switching to publishing .conda packages. So we will want to make sure they are picked up in the statistics here

[h-vetinari]

We've dropped the faulty data from our end. Any chance you are going to backfill data from the past? it looks pretty weird now, because some packages that had releases only have 1 measuring point.

Hi Wolf, yes we are planning to backfill past data and we will be sharing details at this week's conda community sync.

Any updates on the backfill?

I tried to run the by-the-numbers binder again, and

dd.read_parquet("s3://anaconda-package-data/conda/hourly/2024/06/2024-06-*.parquet",storage_options={'anon': True})

returns an empty data frame, and so do all months after June (whereas the months up until May 2024 are fine).

I've loosened the match to

dd.read_parquet("s3://anaconda-package-data/conda/hourly/2024/06/*.parquet",storage_options={'anon': True})

and still nothing.

[jakirkham]

Asked about this at the Conda community meeting earlier this week and it sounds like they are working through some issues