Port dependencies to Missing and CategoricalArray

[nalimilan]

This issue lists packages which depend on DataFrames, with links to PRs which port them to the new Missing and CategoricalArray based DataFrames. Please add new PRs as they are opened to track progress, and check boxes once they have been merged. Some packages may not need any adjustments and could be moved to a separate section.

Packages with open PRs (tick the box once PR is merged):

• CSV: Move to DataStreams 0.2.0 CSV.jl#95
• CovarianceMatrices: gragusa/CovarianceMatrices.jl@0d12f70
• DataFramesMeta: DataFrames 0.11 support DataFramesMeta.jl#79
• ExcelFiles
• ExcelReaders
• Feather: Port to Nulls.jl Feather.jl#48
• FixedEffectModels: https://github.com/matthieugomez/FixedEffectModels.jl/pull/20
• FreqTables: Port to CategoricalArrays and Missings nalimilan/FreqTables.jl#16
• Gadfly: [WIP] work towards updating to DataFrames v0.11+ framework GiovineItalia/Gadfly.jl#1088 DataFrames 0.11+ compat via IndirectArrays GiovineItalia/Gadfly.jl#1090
• GeoStats
• GeoStatsDevTools
• IterableTables: Add support for Nulls.jl based DataFrames.jl queryverse/IterableTables.jl#60, WIP: Port to Nulls queryverse/IterableTables.jl#62
• LogParser
• Microeconometrics: lbittarello/Microeconometrics.jl@ef496df
• Mimi
• MixedModels: Convert to using DataFrames v0.11, CSV, CategoricalArrays and StatsModels JuliaStats/MixedModels.jl#103
• MySQL: DataFrames 0.11+ compatibility  JuliaDatabases/MySQL.jl#91 Support DataFrames 0.11 JuliaDatabases/MySQL.jl#92
• ODBC: Port to DataStreams 0.2 JuliaDatabases/ODBC.jl#173
• PlotlyJS: use Requires.jl JuliaPlots/PlotlyJS.jl#145
• Quandl: milktrader/Quandl.jl#111
• Query: WIP: Port to Nulls.jl queryverse/Query.jl#155
• QueryOperators: WIP: Port to Nulls queryverse/QueryOperators.jl#2
• RCall
• RData: Update to DataFrames v0.11 (CategoricalArrays + Missings) RData.jl#28
• RDatasets: Update to DataFrames v0.11, CSV 0.2.1, RData 0.3 JuliaStats/RDatasets.jl#48 (package works with 0.11 already with deprecation warnings)
• ReadStat
• ScikitLearn: cstjean/ScikitLearn.jl@9555a43
• SQLite: Move to DataStreams 0.2.0 JuliaDatabases/SQLite.jl#131
• StatFiles: queryverse/StatFiles.jl@a171896
• StatsModels: Move back to DataFrames and port from Nullable to Missings JuliaStats/StatsModels.jl#30
• Taro: Fix deprecations, update for DataFrames 0.11 aviks/Taro.jl#38
• UAParser

Packages which do not need any changes:

• StatPlots
• Twitter

Packages which need new PRs:

• BeaData
• Bedgraph
• BedgraphFiles
• Benchmark
• BioMedQuery
• BlsData
• Bootstrap
• Brim
• COMTRADE
• Celeste
• CrossfilterCharts
• DBFTables
• DSGE
• DataCubes
• Deldir
• DimensionalityReduction
• Diversity
• EEG
• EasyPhys
• EconDatasets
• EconModels
• EventHistory
• ExperimentalAnalysis
• FaSTLMM
• FinancialMarkets
• FredData
• GeoIP
• Gillespie
• GoogleCharts
• Graft
• GraphGLRM
• Hive
• InteractiveFixedEffectModels
• JWAS
• JuliaFEM
• LazyQuery
• LifeTable
• LinguisticData
• LowRankModels
• MLDataUtils
• MachineLearning
• Mads
• mPulseAPI
• MultidimensionalTables
• NLOptControl
• NLreg
• OdsIO
• OpenGene
• OpenSecrets
• OptiMimi
• PGFPlots
• PhyloNetworks
• PrettyPlots
• ProjectTemplate
• PySide
• Q
• ROCAnalysis
• Resampling
• Rif
• Robotlib
• RobustStats
• SloanDigitalSkySurvey
• Sparrow
• SpatialEcology
• SpeedDate
• StackedNets
• TermWin
• TextAnalysis
• TimeData
• TimeSeriesIO
• ValueOrientedRiskManagementInsurance
• WorldBankData
• XSim
• ZipCode

Abandoned packages:

• kNN

Packages that already did not load on Julia 0.6:

• Augur
• BIGUQ
• BayesNets
• BayesianDataFusion

[ararslan]

Oh man.

[rofinn]

I can probably take a look at Bootstrap and Resampling next week. I've been wanting to transfer Bootstrap over to JuliaStats anyways (assuming the author is open to that).

[quinnj]

I think we should get an initial tag out of the port; trying to update all this code and have travis runs isn't possible unless we're able to test against current DataFrames. We don't need to announce anything yet and make sure we have really tight bounds on everything, but I think we need to tag sooner rather than later.

[nalimilan]

I think we should get an initial tag out of the port; trying to update all this code and have travis runs isn't possible unless we're able to test against current DataFrames.

That's possible by adding an explicit call to Pkg.checkout("DataFrames") in .travis.yml (see JuliaData/RData.jl#28). Not pretty, but as a temporary way of checking the PR it works.

I think we should have the discussion about when to tag the port somewhere else, maybe in #1209 since it has started there?

[dmbates]

I think it would be useful to write up some guidelines on how to convert a dependent package to the new version of DataFrames, CategoricalArrays, etc. To even be able to install the master version of DataFrames I ended up starting from a clean package directory by setting the environment variable JULIA_PKGDIR. I don't know if this is a good approach or not but I do know that it is easy to find yourself in a "twisty maze of passages" (so who is old enough to know where that phrase originated?).

[dmbates]

Carrying on with this "clean slate" approach (I can move this to another issue if desired). After setting the environment variable JULIA_PKGDIR to, say, ~/.juliaDF11 and starting julia, run

Pkg.init()
Pkg.add("DataFrames")
Pkg.checkout("DataFrames")
versioninfo(true)

the output on packages is

Package Directory: /home/bates/.juliaDF11/v0.6
1 required packages:
 - DataFrames                    0.10.1+            master
13 additional packages:
 - BinDeps                       0.6.0
 - CategoricalArrays             0.2.0
 - Compat                        0.31.0
 - DataStreams                   0.2.0
 - DataStructures                0.7.1
 - Nulls                         0.0.6
 - Reexport                      0.0.3
 - SHA                           0.5.1
 - SortingAlgorithms             0.1.1
 - SpecialFunctions              0.3.3
 - StatsBase                     0.19.0
 - URIParser                     0.2.0
 - WeakRefStrings                0.3.1

Do any of the additional packages need to have a Pkg.checkout before I continue?

[quinnj]

That looks right to me.

[piever]

StatPlots is compatible with DataFrames via the IterableTables package, and given that IterableTables has already updated, it doesn't need a PR.

[jmkuhn]

Quandl: milktrader/Quandl.jl#111 has been merged into master.

[juliohm]

Please mark GeoStatsDevToosl.jl as fixed. I am just waiting for the new tag to be merged on METADATA.jl and then I will tag a new version of GeoStats.jl. It is currently broken with the update.

[juliohm]

GeoStats.jl fixed as well: JuliaLang/METADATA.jl#12225

[Datseris]

If anyone is seeing this: If you can't install DataFrames 11, it may be because of DataTables. For me that was what was holding my upgrade! After I removed DataTables Dataframes updated!

[randyzwitch]

UAParser.jl updated for missings, just needs to be merged on METADATA

[randyzwitch]

LogParser.jl update is at METADATA, just needs to be merged

[randyzwitch]

Per johnmyleswhite/kNN.jl#20 (comment), kNN.jl is abandoned

[randyzwitch]

Twitter.jl technically already works, but updates coming with randyzwitch/Twitter.jl#17 to make DataFrames 0.11 a requirement

[davidanthoff]

ReadStat.jl no longer depends on DataFrames, so it can be removed from the list.

IterableTables.jl, QueryOperators.jl and Query.jl should also all work with DataFrames v0.11 at this point, so no further work needed there.

[randyzwitch]

PR in for Taro.jl aviks/Taro.jl#38

[matthieugomez]

I am trying to migrate but it's more complicated than what I expected.
Could someone give me a hand with this particular function?
https://discourse.julialang.org/t/tanslate-code-from-pooleddataarray-to-categoricalarray/8646

[Ken-B]

Could you uncheck the Gadfly box as the PR remains unmerged?
It took me a while to find out that it was the one holding back my update. Thanks.

[nalimilan]

It's not checked, an IIRC it wasn't a few hours ago either, was it?

[Ken-B]

I would have sworn it wasn't, but perhaps I'm mistaken, in which case sorry for the noise.

[davidanthoff]

Alright, ExcelFiles.jl and ExcelReaders.jl can (finally) be marked as resolved on this list. ExcelReaders.jl no longer depends on DataFrames.jl, and ExceFiles.jl no longer on DataTables.jl, and both packages should work fine with the new v0.11 now.

[davidanthoff]

Mimi.jl can also be marked as solved.

[nalimilan]

Closing since packages which haven't been ported don't work on Julia 1.0, so they have more serious problems.