Review and reduce dependency list with package extensions?

[juliohm]

Distributions.jl is one of the heaviest dependencies in my packages.

Is there an initiative to review and reduce the dependencies of the package? Maybe with package extensions?

julia> PkgDependency.tree("Distributions")
 Distributions v0.25.98                                                      
━━━━━━━━━━━━━━━━━━━━━━━━                                                     
           │                                                                 
           ├── Statistics v1.9.0                                             
           ├── PDMats v0.11.17                                               
           ├── StatsBase v0.33.21                                            
           │   ├── Statistics v1.9.0 (*)                                     
           │   ├── LogExpFunctions v0.3.24                                   
           │   │   ├── IrrationalConstants v0.2.2                            
           │   │   └── DocStringExtensions v0.9.3                            
           │   ├── SortingAlgorithms v1.1.1                                  
           │   │   └── DataStructures v0.18.14                               
           │   │       ├── Compat v4.7.0                                     
           │   │       └── OrderedCollections v1.6.2                         
           │   ├── Missings v1.1.0                                           
           │   │   └── DataAPI v1.15.0                                       
           │   ├── DataStructures v0.18.14 (*)                               
           │   ├── StatsAPI v1.6.0                                           
           │   └── DataAPI v1.15.0 (*)                                       
           ├── StatsAPI v1.6.0 (*)                                           
           ├── StatsFuns v1.3.0                                              
           │   ├── IrrationalConstants v0.2.2 (*)                            
           │   ├── HypergeometricFunctions v0.3.21                           
           │   │   ├── DualNumbers v0.6.8                                    
           │   │   │   ├── SpecialFunctions v2.3.0                           
           │   │   │   │   ├── IrrationalConstants v0.2.2 (*)                
           │   │   │   │   ├── LogExpFunctions v0.3.24 (*)                   
           │   │   │   │   ├── OpenLibm_jll v0.8.1+0                         
           │   │   │   │   └── OpenSpecFun_jll v0.5.5+0                      
           │   │   │   │       ├── JLLWrappers v1.4.1                        
           │   │   │   │       │   └── Preferences v1.4.0                    
           │   │   │   │       │       └── TOML v1.0.3                       
           │   │   │   │       ├── Pkg v1.9.2                                
           │   │   │   │       │   ├── Downloads v1.6.0                      
           │   │   │   │       │   │   ├── LibCURL v0.6.3                    
           │   │   │   │       │   │   │   ├── MozillaCACerts_jll v2022.10.11
           │   │   │   │       │   │   │   └── LibCURL_jll v7.84.0+0         
           │   │   │   │       │   │   │       ├── LibSSH2_jll v1.10.2+0     
           │   │   │   │       │   │   │       │   └── MbedTLS_jll v2.28.2+0 
           │   │   │   │       │   │   │       ├── MbedTLS_jll v2.28.2+0 (*) 
           │   │   │   │       │   │   │       ├── nghttp2_jll v1.48.0+0     
           │   │   │   │       │   │   │       └── Zlib_jll v1.2.13+0        
           │   │   │   │       │   │   ├── NetworkOptions v1.2.0             
           │   │   │   │       │   │   └── ArgTools v1.1.1                   
           │   │   │   │       │   ├── Tar v1.10.0                           
           │   │   │   │       │   │   ├── SHA v0.7.0                        
           │   │   │   │       │   │   └── ArgTools v1.1.1 (*)               
           │   │   │   │       │   ├── SHA v0.7.0 (*)                        
           │   │   │   │       │   ├── p7zip_jll v17.4.0+0                   
           │   │   │   │       │   └── TOML v1.0.3 (*)                       
           │   │   │   │       └── CompilerSupportLibraries_jll v1.0.5+0     
           │   │   │   ├── Calculus v0.5.1                                   
           │   │   │   └── NaNMath v1.0.2                                    
           │   │   │       └── OpenLibm_jll v0.8.1+0 (*)                     
           │   │   ├── SpecialFunctions v2.3.0 (*)                           
           │   │   └── OpenLibm_jll v0.8.1+0 (*)                             
           │   ├── LogExpFunctions v0.3.24 (*)                               
           │   ├── SpecialFunctions v2.3.0 (*)                               
           │   ├── Rmath v0.7.1                                              
           │   │   └── Rmath_jll v0.4.0+0                                    
           │   │       ├── JLLWrappers v1.4.1 (*)                            
           │   │       └── Pkg v1.9.2 (*)                                    
           │   └── Reexport v1.2.2                                           
           ├── QuadGK v2.8.2                                                 
           │   └── DataStructures v0.18.14 (*)                               
           ├── SpecialFunctions v2.3.0 (*)                                   
           └── FillArrays v0.13.11                                           
               └── Statistics v1.9.0 (*) 

[ParadaCarleton]

Julia packages often have a ton of small dependencies that make it look like they're "bigger" than they actually are. The only substantial dependency here is OpenSpecFun, a Fortran binary for special functions.

julia> @time_imports using Distributions
      0.7 ms  DataAPI
      0.2 ms  Compat
      0.1 ms  Compat → CompatLinearAlgebraExt
      5.1 ms  OrderedCollections
     39.6 ms  DataStructures
      0.3 ms  SortingAlgorithms
      7.3 ms  Missings
      0.8 ms  DocStringExtensions
      3.2 ms  IrrationalConstants
      0.2 ms  LogExpFunctions
      0.4 ms  Statistics
      0.2 ms  StatsAPI
     11.2 ms  StatsBase
      0.1 ms  SuiteSparse
      7.9 ms  PDMats
      0.1 ms  Reexport
      0.1 ms  OpenLibm_jll
      8.5 ms  Preferences
      0.3 ms  JLLWrappers
      5.0 ms  CompilerSupportLibraries_jll 92.32% compilation time
    167.1 ms  OpenSpecFun_jll 99.68% compilation time (99% recompilation)
      8.5 ms  SpecialFunctions
      0.4 ms  Rmath_jll
      0.4 ms  Rmath
      0.2 ms  NaNMath
      0.7 ms  Calculus
      7.1 ms  DualNumbers
      0.5 ms  HypergeometricFunctions
      2.4 ms  StatsFuns
      1.6 ms  QuadGK
     43.3 ms  FillArrays
    155.2 ms  Distributions

[devmotion]

Is there an initiative to review and reduce the dependencies of the package? Maybe with package extensions?

We already use package extensions for optional dependencies such as ChainRulesCore and DensityInterface. Both have almost only stdlibs as dependencies but it still avoids compilation and loading of quite a few definitions, both in Distributions and these packages.

[devmotion]

Julia packages often have a ton of small dependencies that make it look like they're "bigger" than they actually are.

Note also that the print-out in the OP contains many dependencies multiple times and includes stdlibs such as Statistics, so IMO it looks worse than it is.

[juliohm]

I think that part of the issue is caused by the external binary dependencies coming from SpecialFunctions.jl

Nowadays we have packages such as Bessels.jl that are purely written in Julia, are faster and support autodiff, but SpecialFunctions.jl is widespread. Any initiative to review the external binary dependencies and replace them with native Julia versions?

[devmotion]

No, I think this should be handled upstream first and not by Distributions. There are discussions in SpecialFunctions but there is no agreement on such changes yet (a while ago e.g. compilation times were much worse with Bessels). A large "part" of the SpecialFunctions dependency is also only indirectly via StatsFuns.

[juliohm]

Fair point. I will close the issue and hope that the upstream packages can replace these binaries at some point for a leaner ecosystem.