Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Error: Cannot allocate vector of size 4.6 Mb In addition: Warning message: In read_zipdata(file, “.sav$”, haven::read_spss, user_na = TRUE) : Multiple file names match pattern ‘.sav$’ in zip file ‘COOD61SV.ZIP’. Returning file ‘CITOLOGIA.SAV’. #117

Open
Gerritdhs opened this issue May 2, 2021 · 11 comments
Labels
bug Something isn't working

Comments

@Gerritdhs
Copy link

Session info - Session info ------------------------------------------------------------------------------------------------------ setting value version R version 4.0.2 (2020-06-22) os Windows 10 x64 system x86_64, mingw32 ui RStudio language (EN) collate German_Germany.1252 ctype German_Germany.1252 tz Europe/Berlin date 2021-05-02
  • Packages ----------------------------------------------------------------------------------------------------------
    package * version date lib source
    assertthat 0.2.1 2019-03-21 [1] CRAN (R 4.0.2)
    backports 1.1.10 2020-09-15 [1] CRAN (R 4.0.2)
    brio 1.1.2 2021-04-23 [1] CRAN (R 4.0.5)
    broom 0.7.6 2021-04-05 [1] CRAN (R 4.0.5)
    cachem 1.0.4 2021-02-13 [1] CRAN (R 4.0.5)
    callr 3.7.0 2021-04-20 [1] CRAN (R 4.0.5)
    cellranger 1.1.0 2016-07-27 [1] CRAN (R 4.0.2)
    cli 2.5.0 2021-04-26 [1] CRAN (R 4.0.5)
    codetools 0.2-16 2018-12-24 [2] CRAN (R 4.0.2)
    colorspace 1.4-1 2019-03-18 [1] CRAN (R 4.0.2)
    crayon 1.4.1 2021-02-08 [1] CRAN (R 4.0.5)
    crosstalk 1.1.1 2021-01-12 [1] CRAN (R 4.0.5)
    curl 4.3 2019-12-02 [1] CRAN (R 4.0.2)
    data.table 1.13.0 2020-07-24 [1] CRAN (R 4.0.2)
    DBI 1.1.1 2021-01-15 [1] CRAN (R 4.0.5)
    dbplyr 2.1.1 2021-04-06 [1] CRAN (R 4.0.5)
    desc 1.3.0 2021-03-05 [1] CRAN (R 4.0.5)
    devtools * 2.4.0 2021-04-07 [1] CRAN (R 4.0.5)
    digest 0.6.27 2020-10-24 [1] CRAN (R 4.0.5)
    dplyr * 1.0.5 2021-03-05 [1] CRAN (R 4.0.5)
    ellipsis 0.3.2 2021-04-29 [1] CRAN (R 4.0.2)
    fansi 0.4.2 2021-01-15 [1] CRAN (R 4.0.5)
    fastmap 1.1.0 2021-01-25 [1] CRAN (R 4.0.5)
    forcats * 0.5.1 2021-01-27 [1] CRAN (R 4.0.5)
    foreign * 0.8-81 2020-12-22 [1] CRAN (R 4.0.3)
    fs 1.5.0 2020-07-31 [1] CRAN (R 4.0.2)
    gapminder 0.3.0 2017-10-31 [1] CRAN (R 4.0.3)
    generics 0.1.0 2020-10-31 [1] CRAN (R 4.0.5)
    getPass 0.2-2 2017-07-21 [1] CRAN (R 4.0.3)
    ggplot2 * 3.3.3 2020-12-30 [1] CRAN (R 4.0.5)
    ggrepel * 0.9.1 2021-01-15 [1] CRAN (R 4.0.5)
    glue 1.4.2 2020-08-27 [1] CRAN (R 4.0.2)
    gridExtra 2.3 2017-09-09 [1] CRAN (R 4.0.2)
    gtable 0.3.0 2019-03-25 [1] CRAN (R 4.0.2)
    haven * 2.4.1 2021-04-23 [1] CRAN (R 4.0.5)
    hexbin 1.28.1 2020-02-03 [1] CRAN (R 4.0.3)
    hms 1.0.0 2021-01-13 [1] CRAN (R 4.0.5)
    htmltools 0.5.0 2020-06-16 [1] CRAN (R 4.0.2)
    htmlwidgets 1.5.3 2020-12-10 [1] CRAN (R 4.0.5)
    httr 1.4.2 2020-07-20 [1] CRAN (R 4.0.2)
    jsonlite 1.7.2 2020-12-09 [1] CRAN (R 4.0.5)
    Lahman 9.0-0 2021-04-09 [1] CRAN (R 4.0.5)
    lattice 0.20-41 2020-04-02 [2] CRAN (R 4.0.2)
    leaflet 2.0.4.1 2021-01-07 [1] CRAN (R 4.0.5)
    lifecycle 1.0.0 2021-02-15 [1] CRAN (R 4.0.5)
    lubridate 1.7.10 2021-02-26 [1] CRAN (R 4.0.5)
    magrittr 2.0.1 2020-11-17 [1] CRAN (R 4.0.5)
    maps 3.3.0 2018-04-03 [1] CRAN (R 4.0.3)
    memoise 2.0.0 2021-01-26 [1] CRAN (R 4.0.5)
    microbenchmark 1.4-7 2019-09-24 [1] CRAN (R 4.0.3)
    modelr 0.1.8 2020-05-19 [1] CRAN (R 4.0.3)
    munsell 0.5.0 2018-06-12 [1] CRAN (R 4.0.2)
    nycflights13 1.0.2 2021-04-12 [1] CRAN (R 4.0.5)
    openxlsx * 4.2.3 2020-10-27 [1] CRAN (R 4.0.5)
    pillar 1.6.0 2021-04-13 [1] CRAN (R 4.0.5)
    pkgbuild 1.2.0 2020-12-15 [1] CRAN (R 4.0.5)
    pkgconfig 2.0.3 2019-09-22 [1] CRAN (R 4.0.2)
    pkgload 1.2.1 2021-04-06 [1] CRAN (R 4.0.5)
    prettyunits 1.1.1 2020-01-24 [1] CRAN (R 4.0.2)
    processx 3.5.2 2021-04-30 [1] CRAN (R 4.0.2)
    pryr 0.1.4 2018-02-18 [1] CRAN (R 4.0.3)
    ps 1.3.4 2020-08-11 [1] CRAN (R 4.0.2)
    purrr * 0.3.4 2020-04-17 [1] CRAN (R 4.0.2)
    qdapRegex 0.7.2 2017-04-09 [1] CRAN (R 4.0.3)
    r4ds * 0.1 2020-12-25 [1] Github (hadley/r4ds@2d1db88)
    R6 2.5.0 2020-10-28 [1] CRAN (R 4.0.5)
    rappdirs 0.3.3 2021-01-31 [1] CRAN (R 4.0.5)
    Rcpp 1.0.5 2020-07-06 [1] CRAN (R 4.0.2)
    rdhs * 0.7.2 2021-04-28 [1] CRAN (R 4.0.2)
    readr * 1.4.0 2020-10-05 [1] CRAN (R 4.0.2)
    readxl 1.3.1 2019-03-13 [1] CRAN (R 4.0.2)
    remotes 2.3.0 2021-04-01 [1] CRAN (R 4.0.5)
    reprex 2.0.0 2021-04-02 [1] CRAN (R 4.0.5)
    rio * 0.5.26 2021-03-01 [1] CRAN (R 4.0.5)
    rlang 0.4.11 2021-04-30 [1] CRAN (R 4.0.2)
    rprojroot 1.3-2 2018-01-03 [1] CRAN (R 4.0.2)
    rstudioapi 0.13 2020-11-12 [1] CRAN (R 4.0.5)
    rvest 1.0.0 2021-03-09 [1] CRAN (R 4.0.5)
    scales 1.1.1 2020-05-11 [1] CRAN (R 4.0.2)
    sessioninfo 1.1.1 2018-11-05 [1] CRAN (R 4.0.2)
    storr 1.2.5 2020-12-01 [1] CRAN (R 4.0.5)
    stringi 1.5.3 2020-09-09 [1] CRAN (R 4.0.2)
    stringr * 1.4.0 2019-02-10 [1] CRAN (R 4.0.2)
    testthat 3.0.2 2021-02-14 [1] CRAN (R 4.0.5)
    tibble * 3.1.1 2021-04-18 [1] CRAN (R 4.0.5)
    tidyr * 1.1.3 2021-03-03 [1] CRAN (R 4.0.5)
    tidyselect 1.1.0 2020-05-11 [1] CRAN (R 4.0.2)
    tidyverse * 1.3.1 2021-04-15 [1] CRAN (R 4.0.5)
    tinytex 0.26 2020-09-22 [1] CRAN (R 4.0.2)
    usethis * 2.0.1 2021-02-10 [1] CRAN (R 4.0.5)
    utf8 1.2.1 2021-03-12 [1] CRAN (R 4.0.5)
    vctrs 0.3.8 2021-04-29 [1] CRAN (R 4.0.2)
    viridis 0.5.1 2018-03-29 [1] CRAN (R 4.0.2)
    viridisLite 0.3.0 2018-02-01 [1] CRAN (R 4.0.2)
    withr 2.4.2 2021-04-18 [1] CRAN (R 4.0.5)
    xfun 0.17 2020-09-09 [1] CRAN (R 4.0.2)
    xml2 1.3.2 2020-04-23 [1] CRAN (R 4.0.2)
    yaml 2.2.1 2020-02-01 [1] CRAN (R 4.0.2)
    zip 2.1.1 2020-08-27 [1] CRAN (R 4.0.2)

First of all, thank you for making this great package it has been very useful for me so and for others working with DHS data.

I have tried to to follow your instruction to download surveys (I requested permission for all) and I get the console output from the images, for some reason I cant copy paste from my console.

image

... and after proceeding to download some survey data

image

in the end end get_datasets() function does not return a vector with a file path

@Gerritdhs Gerritdhs changed the title Error: cannont allocate vector of size 4.6 Mb In addition: Warning message: In read_zipdata(file, “.sav$”, haven::read_spss, user_na = TRUE) : Multiple file names match pattern ‘.sav$’ in zip file ‘COOD61SV.ZIP’. Returning file ‘CITOLOGIA.SAV’. Error: can not allocate vector of size 4.6 Mb In addition: Warning message: In read_zipdata(file, “.sav$”, haven::read_spss, user_na = TRUE) : Multiple file names match pattern ‘.sav$’ in zip file ‘COOD61SV.ZIP’. Returning file ‘CITOLOGIA.SAV’. May 2, 2021
@Gerritdhs Gerritdhs changed the title Error: can not allocate vector of size 4.6 Mb In addition: Warning message: In read_zipdata(file, “.sav$”, haven::read_spss, user_na = TRUE) : Multiple file names match pattern ‘.sav$’ in zip file ‘COOD61SV.ZIP’. Returning file ‘CITOLOGIA.SAV’. Error: Cannot allocate vector of size 4.6 Mb In addition: Warning message: In read_zipdata(file, “.sav$”, haven::read_spss, user_na = TRUE) : Multiple file names match pattern ‘.sav$’ in zip file ‘COOD61SV.ZIP’. Returning file ‘CITOLOGIA.SAV’. May 2, 2021
@jeffeaton
Copy link
Collaborator

Hello @Gerritdhs,

From the above errors, it appears that you are downloading the SPSS formatted files (.SAV). I usually recommend using the 'flat' file format for importing into R, specified by using the argument fileFormat = "flat" (or equivalently fileFormat = "FL") in the function dhs_datasets(...).

For example, to download the household recode from the 2016 South Africa DHS?

> surveys <- dhs_datasets(countryIds = "ZA", surveyYear = 2016, 
                          fileFormat = "flat", fileType = "HR")
> surveys
              FileFormat FileSize     DatasetType SurveyNum  SurveyId
1 Flat ASCII data (.dat)  6150178 Survey Datasets       390 ZA2016DHS
          FileType      FileDateLastModified SurveyType SurveyYearLabel
1 Household Recode January, 29 2019 21:21:49        DHS            2016
  SurveyYear DHS_CountryCode     FileName  CountryName
1       2016              ZA ZAHR71FL.ZIP South Africa
> get_datasets(surveys)
$ZAHR71FL
[1] "/Users/jeff/Library/Caches/rdhs/datasets/ZAHR71FL.rds"

If that does not address the issue, could you please also share the R commands that you are using that are returning these errors?

Thanks,
Jeff

@Gerritdhs
Copy link
Author

@jeffeaton I tried again using FL as format:

--survs.wpv$SurveyId is just a long vector of SurveyIds I have Identified relevant for my research--

datasets.wpv <- dhs_datasets(surveyIds =survs.wpv$SurveyId,
fileFormat = "FL")

Logging into DHS website...
Creating Download url list from DHS website...
The following requested dataset file names are used
by more than one dataset:

KEKR42FL.ZIP
KEIR42FL.ZIP
KEHR42FL.ZIP
KEBR42FL.ZIP

By default the above datasets will be downloaded accordingto the country code indicated by the first 2 letters of thesedatasets. If you wished for the the above datatasets to bedownloaded not based on just their first 2 letters then pleaseprovide the desired rows from the output of dhs_datasets() forthe datasets argument.See introductory vignette for more info about this issue.
These requested datasets are not available from your DHS login credentials:

Surveys AFFW71FL.ZIP, AOAR71FL.ZIP, AOGE71FL.zip, AOGC72FL.ZIP, AMGC72FL.ZIP, AMFW72FL.ZIP, AMGE71FL.ZIP, BJGC72FL.ZIP, BJGE71FL.ZIP, BJFW71FL.ZIP, BFGC62FL.ZIP, BFGE61FL.zip, BFAR61FL.ZIP, BUGC72FL.ZIP, BUGE71FL.ZIP, BUAR71FL.ZIP, BUFW71FL.ZIP, KHGC42FL.ZIP, KHGE43FL.zip, KHGE51FL.zip, khar51fl.zip, KHGC52FL.ZIP, KHGC72FL.ZIP, KHGE71FL.zip, CMar42fl.zip, CMGE42FL.zip, CMGC42FL.ZIP, CMGE61FL.zip, CMAR61FL.ZIP, CMGC62FL.ZIP, CMFW71FL.ZIP, CMGE71FL.ZIP, CMAR71FL.ZIP, CMGC72FL.ZIP, TDAR71FL.ZIP, TDGE71FL.zip, TDGC72FL.ZIP, COGE61FL.ZIP, COGC62FL.ZIP, KMGC62FL.ZIP, KMGE61FL.ZIP, CDOB61FL.ZIP, CDGE61FL.zip, CDAR61FL.ZIP, CDGC62FL.ZIP, CIGC62FL.ZIP, CIAR61FL.ZIP, CIGE61FL.ZIP, DRHT4AFL.ZIP, DRXP4FFL.ZIP, DRGC52FL.ZIP, DRGE52FL.zip, DRar51fl.zip, DRAR61FL.ZIP, DRGE61FL.ZIP, DRGC62FL.ZIP, EGGC62FL.ZIP, EGGE61FL.zip, ETFW71FL.ZIP, ETGE71FL.ZIP, ETGC72FL.ZIP, ETAR71FL.ZIP, GAGC62FL.ZIP, GAAR61FL.ZIP, GAGE61FL.ZIP, GMAR61FL.ZIP, GHGC5BFL.ZIP, GHGE5AFL.zip, GUGC72FL.ZIP, GUGE71FL.zip, HTGC42FL.ZIP, HTGE42FL.zip, HTGE52FL.zip, htar51fl.zip, HTGC52FL.ZIP, HTGC62FL.ZIP, HTAR61FL.ZIP, HTGE61FL.ZIP, HTGC72FL.ZIP, HTFW71FL.ZIP, HTAR71FL.ZIP, HTGE71FL.ZIP, HNGC62FL.ZIP, HNGE61FL.ZIP, iaar51fl.zip, IAGC72FL.ZIP, IAAR72FL.ZIP, IAGE71FL.ZIP, JOGC72FL.ZIP, JOGE71FL.ZIP, KEGC42FL.ZIP, KEar42fl.zip, KEGE43FL.zip, KEAR51FL.ZIP, KEGE52FL.zip, KEGC52FL.ZIP, KEGC72FL.ZIP, KEGE71FL.zip, KYGE61FL.zip, KYGC62FL.ZIP, LBGC52FL.ZIP, LBGE52FL.zip, lbar51fl.zip, LBGE7AFL.ZIP, LBFW7AFL.ZIP, LBGC7BFL.ZIP, MWGC4BFL.ZIP, MWar4Afl.zip, MWGE4BFL.zip, MWAR61FL.zip, MWGE62FL.zip, MWGC62FL.ZIP, MWGC7BFL.ZIP, MWAR7AFL.ZIP, MWFW7AFL.ZIP, MWGE7AFL.zip, MWOB7AFL.ZIP, MLGC52FL.ZIP, MLar51fl.zip, MLGE52FL.zip, MLAR6AFL.ZIP, MLGC6BFL.ZIP, MLGE6BFL.zip, MLGC7BFL.ZIP, MLGE7AFL.ZIP, MLFW7AFL.ZIP, MBGE52FL.zip, MBGC52FL.ZIP, MZGC62FL.ZIP, MZGE61FL.ZIP, MZGE71FL.ZIP, MZGC72FL.ZIP, MZAR72FL.ZIP, MZFW71FL.ZIP, MMFW71FL.ZIP, MMGC72FL.ZIP, MMGE71FL.zip, NMGC62FL.ZIP, NMGE61FL.ZIP, NMAR61FL.ZIP, NPGE61FL.ZIP, NPGC62FL.ZIP, NPGC7BFL.ZIP, NPFW7AFL.ZIP, NPGE7AFL.ZIP, NGGC52FL.ZIP, NGGE52FL.zip, NGGE6AFL.ZIP, NGGC6BFL.ZIP, NGFW7AFL.ZIP, NGGE7BFL.ZIP, NGGC7BFL.ZIP, PKFW71FL.ZIP, PKGC72FL.ZIP, PKGE71FL.ZIP, PGBR71FL.ZIP, PGCR71FL.ZIP, PGHR71FL.ZIP, PGIR71FL.ZIP, PGKR71FL.ZIP, PGMR71FL.ZIP, PGPR71FL.ZIP, PEGE52FL.zip, PEGE5BFL.ZIP, PEGE5IFL.ZIP, PEGC5JFL.ZIP, PHGE52FL.zip, PHGC52FL.ZIP, PHGC72FL.ZIP, PHFW71FL.ZIP, PHGE71FL.ZIP, RWGC52FL.ZIP, RWGE54FL.zip, rwar51fl.zip, RWGE72FL.zip, RWAR71FL.ZIP, RWGC72FL.ZIP, STAR51FL.ZIP, SLGC62FL.ZIP, SLAR61FL.ZIP, SLGE61FL.zip, SLGC7BFL.ZIP, SLAR7AFL.ZIP, SLGE7AFL.ZIP, SLFW7AFL.ZIP, ZAGE71FL.ZIP, ZAFW71FL.ZIP, ZAAR71FL.ZIP, ZAAH71FL.ZIP, ZAGC72FL.ZIP, TJGE61FL.ZIP, TJGC62FL.ZIP, TJGC72FL.ZIP, TJFW71FL.ZIP, TJGE71FL.ZIP, TZOB61FL.ZIP, TZGE61FL.ZIP, TZGC62FL.ZIP, TZGC7BFL.ZIP, TZGE7AFL.zip, TZOB7AFL.ZIP, TLGE62FL.zip, TLGC62FL.ZIP, TLGE71FL.ZIP, TLGC72FL.ZIP, TLFW71FL.ZIP, TGAR61FL.ZIP, TGGE62FL.zip, TGGC62FL.ZIP, UGGC52FL.ZIP, UGGE53FL.zip, UGGC62FL.ZIP, UGGE61FL.zip, UGOB7AFL.ZIP, UGFW7BFL.ZIP, UGGE7AFL.ZIP, UGGC7BFL.ZIP, ZMGC52FL.ZIP, ZMGE52FL.zip, zmar51fl.zip, ZMOB51FL.ZIP, ZMGE61FL.zip, ZMGC62FL.ZIP, ZMAR63FL.ZIP, ZMGE71FL.ZIP, ZMAR71FL.ZIP, ZMGC72FL.ZIP, ZMFW71FL.ZIP, ZWGE61FL.ZIP, ZWAR61FL.ZIP, ZWGC62FL.ZIP, ZWFW71FL.ZIP, ZWGC72FL.ZIP, ZWGE72FL.zip, ZWAR71FL.ZIP
--- Please request permission for these datasets from the DHS website to be able to download them Downloading: Afghanistan 2015 DHS Births Recode Flat ASCII data (.dat) [AFBR71FL.ZIP] Dataset download finished Downloading:
dowloads Afghanistan 2015 DHS Couples' Recode Flat ASCII data (.dat) [AFCR71FL.ZIP] Dataset download finished Downloading: Afghanistan 2015 DHS Household Recode Flat ASCII data (.dat) [AFHR71FL.ZIP] Dataset download finished Downloading: Afghanistan 2015 DHS Individual Recode Flat ASCII data (.dat) [AFIR71FL.ZIP] Dataset download finished Downloading: Afghanistan 2015 DHS Children's Recode Flat ASCII data (.dat) [AFKR71FL.ZIP] Dataset download finished Downloading: Afghanistan 2015 DHS Men's Recode Flat ASCII data (.dat) [AFMR71FL.ZIP] Dataset download finished Downloading: Afghanistan 2015 DHS Household Member Recode Flat ASCII data (.dat) [AFPR71FL.ZIP] Dataset download finished Downloading: Angola 2015 DHS Births Recode Flat ASCII data (.dat) [AOBR71FL.ZIP] Dataset download finished Downloading: Angola 2015 DHS Couples' Recode Flat ASCII data (.dat) [AOCR71FL.ZIP] Dataset download finished Downloading: Angola 2015 DHS Household Recode Flat ASCII data (.dat) [AOHR71FL.ZIP] Dataset download finished Downloading: Angola 2015 DHS Individual Recode Flat ASCII data (.dat) [AOIR71FL.ZIP] Dataset download finished Downloading: Angola 2015 DHS Children's Recode Flat ASCII data (.dat) [AOKR71FL.ZIP] Dataset download finished Downloading: Angola 2015 DHS Men's Recode Flat ASCII data (.dat) [AOMR71FL.ZIP] Dataset download finished Downloading: Angola 2015 DHS Household Member Recode Flat ASCII data (.dat) [AOPR71FL.ZIP] Dataset download finished Downloading: Armenia 2016 DHS Couples' Recode Flat ASCII data (.dat) [AMCR72FL.ZIP] Dataset download finished Downloading: Armenia 2016 DHS Household Recode Flat ASCII data (.dat) [AMHR72FL.ZIP] Dataset download finished Downloading: Armenia 2016 DHS Individual Recode Flat ASCII data (.dat) [AMIR72FL.ZIP] Dataset download finished Downloading: Armenia 2016 DHS Children's Recode Flat ASCII data (.dat) [AMKR72FL.ZIP] Dataset download finished Downloading: Armenia 2016 DHS Men's Recode Flat ASCII data (.dat) [AMMR72FL.ZIP] Dataset download finished Downloading: Armenia 2016 DHS Household Member Recode Flat ASCII data (.dat) [AMPR72FL.ZIP] Dataset download finished Downloading: Armenia 2016 DHS Births Recode Flat ASCII data (.dat) [AMBR72FL.ZIP] Dataset download finished Downloading: Azerbaijan 2006 DHS Births Recode Flat ASCII data (.dat) [AZBR52FL.ZIP] Dataset download finished Downloading: Azerbaijan 2006 DHS Couples' Recode Flat ASCII data (.dat) [AZCR52FL.ZIP] Dataset download finished Downloading: Azerbaijan 2006 DHS Household Recode Flat ASCII data (.dat) [AZHR52FL.ZIP] Dataset download finished Downloading: Azerbaijan 2006 DHS Individual Recode Flat ASCII data (.dat) [AZIR52FL.ZIP] Dataset download finished Downloading: Azerbaijan 2006 DHS Children's Recode Flat ASCII data (.dat) [AZKR52FL.ZIP] Dataset download finished Downloading: Azerbaijan 2006 DHS Men's Recode Flat ASCII data (.dat) [AZMR52FL.ZIP] Dataset download finished Downloading: Azerbaijan 2006 DHS Household Member Recode Flat ASCII data (.dat) [AZPR52FL.ZIP] Dataset download finished Downloading: Benin 2017 DHS Births Recode Flat ASCII data (.dat) [BJBR71FL.ZIP] Dataset download finished Downloading: Benin 2017 DHS Couples' Recode Flat ASCII data (.dat) [BJCR71FL.ZIP] Dataset download finished Downloading: Benin 2017 DHS Household Recode Flat ASCII data (.dat) [BJHR71FL.ZIP] Dataset download finished Downloading: Benin 2017 DHS Individual Recode Flat ASCII data (.dat) [BJIR71FL.ZIP] Dataset download finished Downloading: Benin 2017 DHS Children's Recode Flat ASCII data (.dat) [BJKR71FL.ZIP] Dataset download finished Downloading: Benin 2017 DHS Men's Recode Flat ASCII data (.dat) [BJMR71FL.ZIP] Dataset download finished Downloading: Benin 2017 DHS Household Member Recode Flat ASCII data (.dat) [BJPR71FL.ZIP] Dataset download finished Downloading: Burkina Faso 2010 DHS Individual Recode Flat ASCII data (.dat) [BFIR62FL.ZIP] Dataset download finished Downloading: Burkina Faso 2010 DHS Children's Recode Flat ASCII data (.dat) [BFKR62FL.ZIP] Dataset download finished Downloading: Burkina Faso 2010 DHS Men's Recode Flat ASCII data (.dat) [BFMR62FL.ZIP] Dataset download finished Downloading: Burkina Faso 2010 DHS Household Member Recode Flat ASCII data (.dat) [BFPR62FL.ZIP] Dataset download finished Downloading: Burkina Faso 2010 DHS Couples' Recode Flat ASCII data (.dat) [BFCR62FL.ZIP] Dataset download finished Downloading: Burkina Faso 2010 DHS Household Recode Flat ASCII data (.dat) [BFHR62FL.ZIP] Dataset download finished Downloading: Burkina Faso 2010 DHS Births Recode Flat ASCII data (.dat) [BFBR62FL.ZIP] Dataset download finished Downloading: Burundi 2016 DHS Births Recode Flat ASCII data (.dat) [BUBR71FL.ZIP] Dataset download finished Downloading: Burundi 2016 DHS Couples' Recode Flat ASCII data (.dat) [BUCR71FL.ZIP] Dataset download finished Downloading: Burundi 2016 DHS Men's Recode Flat ASCII data (.dat) [BUMR71FL.ZIP] Dataset download finished Downloading: Burundi 2016 DHS Household Member Recode Flat ASCII data (.dat) [BUPR71FL.ZIP] Dataset download finished Downloading: Burundi 2016 DHS Household Recode Flat ASCII data (.dat) [BUHR71FL.ZIP] Dataset download finished Downloading: Burundi 2016 DHS Individual Recode Flat ASCII data (.dat) [BUIR71FL.ZIP] Dataset download finished Downloading: Burundi 2016 DHS Children's Recode Flat ASCII data (.dat) [BUKR71FL.ZIP] Dataset download finished Downloading: Cambodia 2000 DHS Births Recode Flat ASCII data (.dat) [KHBR42FL.zip] Dataset download finished Downloading: Cambodia 2000 DHS Household Recode Flat ASCII data (.dat) [KHHR42FL.zip] Dataset download finished Downloading: Cambodia 2000 DHS Wealth Index Flat ASCII data (.dat) [KHWI41FL.ZIP] Dataset download finished Downloading: Cambodia 2000 DHS Height and Weight Scores - WHO Child Growth Standards Flat ASCII data (.dat) [KHHW42FL.ZIP] Dataset download finished Downloading: Cambodia 2000 DHS Individual Recode Flat ASCII data (.dat) [KHIR42FL.zip] Dataset download finished Downloading: Cambodia 2000 DHS Children's Recode Flat ASCII data (.dat) [KHKR42FL.zip] Dataset download finished Downloading: Cambodia 2000 DHS Household Member Recode Flat ASCII data (.dat) [KHPR42FL.zip] Dataset download finished Downloading: Cambodia 2005 DHS Household Member Recode Flat ASCII data (.dat) [KHPR51FL.zip] Dataset download finished Downloading: Cambodia 2005 DHS Children's Recode Flat ASCII data (.dat) [KHKR51FL.zip] Dataset download finished Downloading: Cambodia 2005 DHS Men's Recode Flat ASCII data (.dat) [KHMR51FL.zip] Dataset download finished Downloading: Cambodia 2005 DHS Individual Recode Flat ASCII data (.dat) [KHIR51FL.zip] Dataset download finished Downloading: Cambodia 2005 DHS Height and Weight Scores - WHO Child Growth Standards Flat ASCII data (.dat) [KHHW51FL.ZIP] Dataset download finished Downloading: Cambodia 2005 DHS Household Recode Flat ASCII data (.dat) [KHHR51FL.zip] Dataset download finished Downloading: Cambodia 2005 DHS Births Recode Flat ASCII data (.dat) [KHBR51FL.zip] Dataset download finished Downloading: Cambodia 2005 DHS Couples' Recode Flat ASCII data (.dat) [KHCR51FL.zip] Dataset download finished Downloading: Cambodia 2014 DHS Children's Recode Flat ASCII data (.dat) [KHKR73FL.ZIP] Dataset download finished Downloading: Cambodia 2014 DHS Household Member Recode Flat ASCII data (.dat) [KHPR73FL.ZIP] Dataset download finished Downloading: Cambodia 2014 DHS Men's Recode Flat ASCII data (.dat) [KHMR72FL.ZIP] Dataset download finished Downloading: Cambodia 2014 DHS Births Recode Flat ASCII data (.dat) [KHBR73FL.ZIP] Dataset download finished Downloading: Cambodia 2014 DHS Couples' Recode Flat ASCII data (.dat) [KHCR73FL.ZIP] Dataset download finished Downloading: Cambodia 2014 DHS Household Recode Flat ASCII data (.dat) [KHHR73FL.ZIP] Dataset download finished Downloading: Cambodia 2014 DHS Individual Recode Flat ASCII data (.dat) [KHIR73FL.ZIP] Dataset download finished Downloading: Cameroon 2004 DHS Men's Recode Flat ASCII data (.dat) [CMMR44FL.zip] Dataset download finished Downloading: Cameroon 2004 DHS Children's Recode Flat ASCII data (.dat) [CMKR44FL.zip] Dataset download finished Downloading: Cameroon 2004 DHS Household Member Recode Flat ASCII data (.dat) [CMPR45FL.ZIP] Dataset download finished Downloading: Cameroon 2004 DHS Births Recode Flat ASCII data (.dat) [CMBR44FL.zip] Dataset download finished Downloading: Cameroon 2004 DHS Couples' Recode Flat ASCII data (.dat) [CMCR44FL.zip] Dataset download finished Downloading: Cameroon 2004 DHS Household Recode Flat ASCII data (.dat) [CMHR44FL.zip] Dataset download finished Downloading: Cameroon 2004 DHS Height and Weight Scores - WHO Child Growth Standards Flat ASCII data (.dat) [CMHW44FL.ZIP] Dataset download finished Downloading: Cameroon 2004 DHS Individual Recode Flat ASCII data (.dat) [CMIR44FL.zip] Dataset download finished Downloading: Cameroon 2011 DHS Household Recode Flat ASCII data (.dat) [CMHR61FL.ZIP] Dataset download finished Downloading: Cameroon 2011 DHS Individual Recode Flat ASCII data (.dat) [CMIR61FL.ZIP] Dataset download finished Downloading: Cameroon 2011 DHS Children's Recode Flat ASCII data (.dat) [CMKR61FL.ZIP] Dataset download finished Downloading: Cameroon 2011 DHS Couples' Recode Flat ASCII data (.dat) [CMCR61FL.ZIP] Dataset download finished Downloading: Cameroon 2011 DHS Births Recode Flat ASCII data (.dat) [CMBR61FL.ZIP] Dataset download finished Downloading: Cameroon 2011 DHS Men's Recode Flat ASCII data (.dat) [CMMR61FL.ZIP] Dataset download finished Downloading: Cameroon 2011 DHS Household Member Recode Flat ASCII data (.dat) [CMPR61FL.ZIP] Dataset download finished Downloading: Cameroon 2018 DHS Individual Recode Flat ASCII data (.dat) [CMIR71FL.ZIP] Dataset download finished Downloading: Cameroon 2018 DHS Household Recode Flat ASCII data (.dat) [CMHR71FL.ZIP] Dataset download finished Downloading: Cameroon 2018 DHS Births Recode Flat ASCII data (.dat) [CMBR71FL.ZIP] Dataset download finished Downloading: Cameroon 2018 DHS Couples' Recode Flat ASCII data (.dat) [CMCR71FL.ZIP] Dataset download finished Downloading: Cameroon 2018 DHS Children's Recode Flat ASCII data (.dat) [CMKR71FL.ZIP] Dataset download finished Downloading: Cameroon 2018 DHS Men's Recode Flat ASCII data (.dat) [CMMR71FL.ZIP] Dataset download finished Downloading: Cameroon 2018 DHS Household Member Recode Flat ASCII data (.dat) [CMPR71FL.ZIP] Dataset download finished Downloading: Chad 2014 DHS Births Recode Flat ASCII data (.dat) [TDBR71FL.ZIP] Dataset download finished Downloading: Chad 2014 DHS Couples' Recode Flat ASCII data (.dat) [TDCR71FL.ZIP] Dataset download finished Downloading: Chad 2014 DHS Household Recode Flat ASCII data (.dat) [TDHR71FL.ZIP] Dataset download finished Downloading: Chad 2014 DHS Individual Recode Flat ASCII data (.dat) [TDIR71FL.ZIP] Dataset download finished Downloading: Chad 2014 DHS Children's Recode Flat ASCII data (.dat) [TDKR71FL.ZIP] Dataset download finished Downloading: Chad 2014 DHS Men's Recode Flat ASCII data (.dat) [TDMR71FL.ZIP] Dataset download finished Downloading: Chad 2014 DHS Household Member Recode Flat ASCII data (.dat) [TDPR71FL.ZIP] Dataset download finished Downloading: Colombia 2000 DHS Height and Weight Scores - WHO Child Growth Standards Flat ASCII data (.dat) [COHW41FL.ZIP] Dataset download finished Downloading: Colombia 2000 DHS Individual Recode Flat ASCII data (.dat) [COIR41FL.ZIP] Dataset download finished Downloading: Colombia 2000 DHS Wealth Index Flat ASCII data (.dat) [COWI41FL.ZIP] Dataset download finished Downloading: Colombia 2000 DHS Household Member Recode Flat ASCII data (.dat) [COPR41FL.ZIP] Dataset download finished Downloading: Colombia 2000 DHS Household Recode Flat ASCII data (.dat) [COHR41FL.ZIP] Dataset download finished Downloading: Colombia 2000 DHS Children's Recode Flat ASCII data (.dat) [COKR41FL.ZIP] Dataset download finished Downloading: Colombia 2000 DHS Births Recode Flat ASCII data (.dat) [COBR41FL.zip] Dataset download finished Downloading: Colombia 2010 DHS Births Recode Flat ASCII data (.dat) [COBR61FL.ZIP] Dataset download finished Downloading: Colombia 2010 DHS Children's Recode Flat ASCII data (.dat) [COKR61FL.ZIP] Dataset download finished Downloading: Colombia 2010 DHS Other Data Flat ASCII data (.dat) [COOD61FL.ZIP] Dataset download finished Downloading: Colombia 2010 DHS Household Recode Flat ASCII data (.dat) [COHR61FL.ZIP] Dataset download finished Downloading: Colombia 2010 DHS Household Member Recode Flat ASCII data (.dat) [COPR61FL.ZIP] Dataset download finished Downloading: Colombia 2010 DHS Individual Recode Flat ASCII data (.dat) [COIR61FL.ZIP] Dataset download finished Downloading: Comoros 2012 DHS Births Recode Flat ASCII data (.dat) [KMBR61FL.ZIP] Dataset download finished Downloading: Comoros 2012 DHS Couples' Recode Flat ASCII data (.dat) [KMCR61FL.ZIP] Dataset download finished Downloading: Comoros 2012 DHS Household Recode Flat ASCII data (.dat) [KMHR61FL.ZIP] Dataset download finished Downloading: Comoros 2012 DHS Individual Recode Flat ASCII data (.dat) [KMIR61FL.ZIP] Dataset download finished Downloading: Comoros 2012 DHS Children's Recode Flat ASCII data (.dat) [KMKR61FL.ZIP] Dataset download finished Downloading: Comoros 2012 DHS Men's Recode Flat ASCII data (.dat) [KMMR61FL.ZIP] Dataset download finished Downloading: Comoros 2012 DHS Household Member Recode Flat ASCII data (.dat) [KMPR61FL.ZIP] Dataset download finished Downloading: Congo Democratic Republic 2013 DHS Births Recode Flat ASCII data (.dat) [CDBR61FL.ZIP] Dataset download finished Downloading: Congo Democratic Republic 2013 DHS Couples' Recode Flat ASCII data (.dat) [CDCR61FL.ZIP] Dataset download finished Downloading: Congo Democratic Republic 2013 DHS Household Recode Flat ASCII data (.dat) [CDHR61FL.ZIP] Dataset download finished Downloading: Congo Democratic Republic 2013 DHS Individual Recode Flat ASCII data (.dat) [CDIR61FL.ZIP] Dataset download finished Downloading: Congo Democratic Republic 2013 DHS Children's Recode Flat ASCII data (.dat) [CDKR61FL.ZIP] Dataset download finished Downloading: Congo Democratic Republic 2013 DHS Men's Recode Flat ASCII data (.dat) [CDMR61FL.ZIP] Dataset download finished Downloading: Congo Democratic Republic 2013 DHS Household Member Recode Flat ASCII data (.dat) [CDPR61FL.ZIP] Dataset download finished Downloading: Cote d'Ivoire 2012 DHS Births Recode Flat ASCII data (.dat) [CIBR62FL.ZIP] Dataset download finished Downloading: Cote d'Ivoire 2012 DHS Couples' Recode Flat ASCII data (.dat) [CICR62FL.ZIP] Dataset download finished Downloading: Cote d'Ivoire 2012 DHS Household Recode Flat ASCII data (.dat) [CIHR62FL.ZIP] Dataset download finished Downloading: Cote d'Ivoire 2012 DHS Individual Recode Flat ASCII data (.dat) [CIIR62FL.ZIP] Dataset download finished Downloading: Cote d'Ivoire 2012 DHS Children's Recode Flat ASCII data (.dat) [CIKR62FL.ZIP] Dataset download finished Downloading: Cote d'Ivoire 2012 DHS Men's Recode Flat ASCII data (.dat) [CIMR62FL.ZIP] Dataset download finished Downloading: Cote d'Ivoire 2012 DHS Household Member Recode Flat ASCII data (.dat) [CIPR62FL.ZIP] Dataset download finished Downloading: Dominican Republic 2002 DHS Men's Recode Flat ASCII data (.dat) [DRMR4BFL.ZIP] Dataset download finished Downloading: Dominican Republic 2002 DHS Individual Recode Flat ASCII data (.dat) [DRIR4BFL.ZIP] Dataset download finished Downloading: Dominican Republic 2002 DHS Wealth Index Flat ASCII data (.dat) [DRWI4AFL.ZIP] Dataset download finished Downloading: Dominican Republic 2002 DHS Height and Weight Scores - WHO Child Growth Standards Flat ASCII data (.dat) [DRHW4AFL.ZIP] Dataset download finished Downloading: Dominican Republic 2002 DHS Births Recode Flat ASCII data (.dat) [DRBR4BFL.ZIP] Dataset download finished Downloading: Dominican Republic 2002 DHS Couples' Recode Flat ASCII data (.dat) [DRCR4BFL.ZIP] Dataset download finished Downloading: Dominican Republic 2002 DHS Household Recode Flat ASCII data (.dat) [DRHR4BFL.ZIP] Dataset download finished Downloading: Dominican Republic 2002 DHS Children's Recode Flat ASCII data (.dat) [DRKR4BFL.ZIP] Dataset download finished Downloading: Dominican Republic 2002 DHS Household Member Recode Flat ASCII data (.dat) [DRPR4BFL.ZIP] Dataset download finished Downloading: Dominican Republic 2007 DHS Births Recode Flat ASCII data (.dat) [DRBR52FL.zip] Dataset download finished Downloading: Dominican Republic 2007 DHS Couples' Recode Flat ASCII data (.dat) [DRCR52FL.zip] Dataset download finished Downloading: Dominican Republic 2007 DHS Men's Recode Flat ASCII data (.dat) [DRMR52FL.ZIP] Dataset download finished Downloading: Dominican Republic 2007 DHS Children's Recode Flat ASCII data (.dat) [DRKR52FL.zip] Dataset download finished Downloading: Dominican Republic 2007 DHS Household Recode Flat ASCII data (.dat) [DRHR52FL.zip] Dataset download finished Downloading: Dominican Republic 2007 DHS Individual Recode Flat ASCII data (.dat) [DRIR52FL.ZIP] Dataset download finished Downloading: Dominican Republic 2007 DHS Household Member Recode Flat ASCII data (.dat) [DRPR52FL.zip] Dataset download finished Downloading: Dominican Republic 2013 DHS Household Member Recode Flat ASCII data (.dat) [DRPR61FL.ZIP] Dataset download finished Downloading: Dominican Republic 2013 DHS Individual Recode Flat ASCII data (.dat) [DRIR61FL.ZIP] Dataset download finished Downloading: Dominican Republic 2013 DHS Household Recode Flat ASCII data (.dat) [DRHR61FL.ZIP] Dataset download finished Downloading: Dominican Republic 2013 DHS Children's Recode Flat ASCII data (.dat) [DRKR61FL.ZIP] Dataset download finished Downloading: Dominican Republic 2013 DHS Men's Recode Flat ASCII data (.dat) [DRMR61FL.ZIP] Dataset download finished Downloading: Dominican Republic 2013 DHS Births Recode Flat ASCII data (.dat) [DRBR61FL.ZIP] Dataset download finished Downloading: Dominican Republic 2013 DHS Couples' Recode Flat ASCII data (.dat) [DRCR61FL.ZIP] Dataset download finished Downloading: Egypt 2014 DHS Children's Recode Flat ASCII data (.dat) [EGKR61FL.ZIP] Dataset download finished Downloading: Egypt 2014 DHS Household Member Recode Flat ASCII data (.dat) [EGPR61FL.ZIP] Dataset download finished Downloading: Egypt 2014 DHS Births Recode Flat ASCII data (.dat) [EGBR61FL.ZIP] Dataset download finished Downloading: Egypt 2014 DHS Household Recode Flat ASCII data (.dat) [EGHR61FL.ZIP] Dataset download finished Downloading: Egypt 2014 DHS Individual Recode Flat ASCII data (.dat) [EGIR61FL.ZIP] Dataset download finished Downloading: Ethiopia 2016 DHS Births Recode Flat ASCII data (.dat) [ETBR71FL.ZIP] Dataset download finished Downloading: Ethiopia 2016 DHS Couples' Recode Flat ASCII data (.dat) [ETCR71FL.ZIP] Dataset download finished Downloading: Ethiopia 2016 DHS Household Recode Flat ASCII data (.dat) [ETHR71FL.ZIP] Dataset download finished Downloading: Ethiopia 2016 DHS Individual Recode Flat ASCII data (.dat) [ETIR71FL.ZIP] Dataset download finished Downloading: Ethiopia 2016 DHS Children's Recode Flat ASCII data (.dat) [ETKR71FL.ZIP] Dataset download finished Downloading: Ethiopia 2016 DHS Men's Recode Flat ASCII data (.dat) [ETMR71FL.ZIP] Dataset download finished Downloading: Ethiopia 2016 DHS Household Member Recode Flat ASCII data (.dat) [ETPR71FL.ZIP] Dataset download finished Downloading: Gabon 2012 DHS Men's Recode Flat ASCII data (.dat) [GAMR61FL.ZIP] Dataset download finished Downloading: Gabon 2012 DHS Household Member Recode Flat ASCII data (.dat) [GAPR61FL.ZIP] Dataset download finished Downloading: Gabon 2012 DHS Couples' Recode Flat ASCII data (.dat) [GACR61FL.ZIP] Dataset download finished Downloading: Gabon 2012 DHS Births Recode Flat ASCII data (.dat) [GABR61FL.ZIP] Dataset download finished Downloading: Gabon 2012 DHS Household Recode Flat ASCII data (.dat) [GAHR61FL.ZIP] Dataset download finished Downloading: Gabon 2012 DHS Individual Recode Flat ASCII data (.dat) [GAIR61FL.ZIP] Dataset download finished Downloading: Gabon 2012 DHS Children's Recode Flat ASCII data (.dat) [GAKR61FL.ZIP] Dataset download finished Downloading: Gambia 2013 DHS Couples' Recode Flat ASCII data (.dat) [GMCR61FL.ZIP] Dataset download finished Downloading: Gambia 2013 DHS Men's Recode Flat ASCII data (.dat) [GMMR61FL.ZIP] Dataset download finished Downloading: Gambia 2013 DHS Household Member Recode Flat ASCII data (.dat) [GMPR61FL.ZIP] Dataset download finished Downloading: Gambia 2013 DHS Births Recode Flat ASCII data (.dat) [GMBR61FL.ZIP] Dataset download finished Downloading: Gambia 2013 DHS Household Recode Flat ASCII data (.dat) [GMHR61FL.ZIP] Dataset download finished Downloading: Gambia 2013 DHS Individual Recode Flat ASCII data (.dat) [GMIR61FL.ZIP] Dataset download finished Downloading: Gambia 2013 DHS Children's Recode Flat ASCII data (.dat) [GMKR61FL.ZIP] Dataset download finished Downloading: Ghana 2008 DHS Births Recode Flat ASCII data (.dat) [GHBR5AFL.ZIP] Dataset download finished Downloading: Ghana 2008 DHS Men's Recode Flat ASCII data (.dat) [GHMR5AFL.ZIP] Dataset download finished Downloading: Ghana 2008 DHS Household Member Recode Flat ASCII data (.dat) [GHPR5AFL.ZIP] Dataset download finished Downloading: Ghana 2008 DHS Verbal Autopsy Flat ASCII data (.dat) [GHVA5AFL.ZIP]

Error in (varlst_idx + 1):(min(endblock[endblock > varlst_idx]) - 1) :
result would be too long a vector
In addition: Warning messages:
1: In read_zipdata(zfile, "\.DCF$", brio::read_lines) :
Multiple file names match pattern '.DCF$' in zip file 'COOD61FL.ZIP'. Returning file 'CITOLOGIA.DCF'.
2: In read_zipdata(zfile, "\.DAT$", iotools::input.file, formatter = iotools::dstrfw, :
Multiple file names match pattern '.DAT$' in zip file 'COOD61FL.ZIP'. Returning file 'CITOLOGIA.DAT'.
3: In min(endblock[endblock > varlst_idx]) :
no non-missing arguments to min; returning Inf

@jeffeaton
Copy link
Collaborator

Hi @Gerritdhs,

It looks like the problematic dataset COOD61FL.ZIP is an "Other Data" dataset from the Columbia 2010 DHS. The footnote describing is:

Other Data (COOD60FL.ZIP) - Contains data collected on Cytology and the Elderly.

I am not familiar with this dataset. I am also getting an error when I try to load it, but a slightly different one:

> library(rdhs)
> get_datasets("COOD61FL.ZIP")
Downloading: 
Colombia 2010 DHS Other Data Flat ASCII data (.dat) [COOD61FL.ZIP]
Logging into DHS website...
Error in unzip(tf, list = TRUE) : 
  zip file '/var/folders/fy/vmz3dx2x5bg1f8vy3z0yh5fr0000gn/T//RtmpxL6ymW/file11e5f2b2b1933' cannot be opened

Do you need this cytology dataset specifically? I can look at it more to see if there is an easy fix to parse it with rdhs, but it might be more expedient to simply download it separately.

This line is select all of the data set types from the requested list of surveys:

datasets.wpv <- dhs_datasets(surveyIds =survs.wpv$SurveyId, fileFormat = "FL")

I typically filter only the datasets I need for an analysis. For example, for the individual recode datasets:

datasets_ir <- dhs_datasets(surveyIds =survs.wpv$SurveyId, fileFormat = "FL", fileType = "IR")

And for the household member (person) recode:

datasets_ir <- dhs_datasets(surveyIds =survs.wpv$SurveyId, fileFormat = "FL", fileType = "PR")

Thanks,
Jeff

@Gerritdhs
Copy link
Author

Thank you for looking into this, I limited the fileType to IR and removed the india 2015 survey from the list,

now I just get another error:

datasets.wpv <- dhs_datasets(surveyIds =survs.wpv$SurveyId,
fileFormat = "FL", fileType = "IR")

noindia <- datasets.wpv[-c(35),] # remove india 2015 as it causes errors for dhs unzip due to file size

downloads <- get_datasets(dataset_filenames = noindia$FileName)

Downloading:
Jordan 2017 DHS Individual Recode Flat ASCII data (.dat) [JOIR73FL.ZIP]
Error in curl::curl_fetch_disk(url, x$path, handle = handle) :
Failed to open file C:\Users\Gerrit\AppData\Local\Temp\Rtmpi4ndO7\file3fd44795463e.

I dont understand what its trying to grab, that file doesnt exist

@OJWatson OJWatson added the bug Something isn't working label Sep 1, 2021
@grasshoppermouse
Copy link

I got a similar warning:

Downloading:
Afghanistan 2010 OTH Household Raw Stata dataset (.dta) [AFHH66DT.ZIP]
Dataset download finished
Warning message:
In read_zipdata(zfile, "\.dta$", haven::read_dta, ...) :
Multiple file names match pattern '.dta$' in zip file 'AFHH66DT.ZIP'. Returning file 'HHDEATHS.DTA'.

Code:

gotten <- get_datasets(
  dataset_filenames = mydatasets$FileName[1:3], 
  output_dir_root = '~/Desktop/DHS/dhs2022/'
  )

Here are the contents of the AFHH66DT.ZIP file:

Screen Shot 2022-03-28 at 6 15 10 AM

@jeffeaton
Copy link
Collaborator

Thanks -- similarly, I am not familiar with the details of this survey and unfortunately I don't have access to this data set on my user account, so I get test or debug it immediately. I can request it and try to see what is possible.

The reason for this warning is that the read_zipdata() function is looking for a single .DTA file within the zipped archive to return. From the output you've posted, it looks like it has returned the Household Deaths dataset. Is that correct and is that returned in a useable format?

What are you ultimately trying to achieve here—reading the other / all of the .DTA files in that archive? I am not sure off the top of my head how easy it would be to add support that, but I can investigate. It might be most straightforward to download this survey manually rather than through rdhs given the custom format.

Thanks,
Jeff

@grasshoppermouse
Copy link

What are you ultimately trying to achieve here—reading the other / all of the .DTA files in that archive?

Yes.

I see that read_zipdata() is used in several places, and it's probably assumed that it returns a single file, so changing its behavior might be difficult. Possible alternatives:

  • some kind of log so folks can see which zip files had multiple datasets (important when downloading many files)?
  • if there are multiple datasets in the zip file, keep the zip file for manual extraction, regardless of download_option?

@jeffeaton
Copy link
Collaborator

Many thanks. Selecting a specific .DTA file with read_zipdata() is straightforward / supported, e.g. something like this is what I would use to work with the data file if I downloaded it manually:

hhdeaths <- rdhs::read_zipdata("AFHH66DT.ZIP", "HHDEATHS.DTA$", haven::read_dta)
households <- rdhs::read_zipdata("AFHH66DT.ZIP", "HOUSEHOLDS.DTA$", haven::read_dta)
outmovers <- rdhs::read_zipdata("AFHH66DT.ZIP", "OUTMOVERS.DTA$", haven::read_dta)
outpatients <- rdhs::read_zipdata("AFHH66DT.ZIP", "OUTPATIENTS.DTA$", haven::read_dta)

What is more difficult is how to manage extraction and caching of such data sets and meta data within the "one file, one dataset" model that rdhs uses to index the cache.

Regarding a log for which zip files has multiple datasets, this is quite infrequent as the main survey datasets are very standardised. I think most/all cases are "special" cases such as this "Afghanistan Other" survey. The package gives you a warning when this has happened. Is that sufficient log?

We'll think about if custom option to cache these as ZIP files if feasible. They are pretty few cases I think though.

@grasshoppermouse
Copy link

I appreciate your rapid response to this issue. I will be using rdhs to automate downloading a large number of files, which might be different than the typical user. Copy-pasting the warnings from the console is less than ideal, but I can easily fork the project to log the warnings to a file in a way that's a bit more convenient to systematically deal with, so no need to add a feature that most folks won't use.

@jeffeaton
Copy link
Collaborator

Thanks, yes, that is quite a common use case for our analyses. But most analyses where we want a large number of datasets tend to involve the 'standard' recode datasets (individual recode, household recode, men's recode, etc.), and I usually subset to select the relevant data sets I need before calling get_datasets() to download them.

Outline of a common workflow for me:

## All surveys since year 1995
surveys <- dhs_surveys(surveyYearStart = 1995)

## All individual recode datasets, flat file format
ir_datasets <- dhs_datasets(surveyIds = surveys$SurveyId, fileType = "IR", fileFormat = "FL")

## Download and return paths to the relevant data sets
ir_paths <- get_datasets(ir_datasets)

Any custom data sets, such as the special Afghanistan survey, would need to be post-processed separately anyway because of custom formatting, and so it makes sense for me to download and read them separately from the core data sets that are relatively standardized.

I'm not sure if that helps or not regarding organizing workflow, but sharing in case.

@grasshoppermouse
Copy link

That helps, thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

4 participants