correct_compl_confluences when too many complex confluences

[p-schaefer]

correct_compl_confluences() seems to fail when there are too many complex confluences. Specifically, the where statement in this chunk fails:

prev_str <- paste0("(", paste(sort(prev_str), collapse = ","), ")", sep = "")

execGRASS("v.extract", flags = c("overwrite", "quiet"), 
                parameters = list(
                  input = "streams_v", 
                  output = "complex_flows", 
                  type = "line", 
                  where = paste0("stream in ", prev_str)),
                ignore.stderr = T, intern = TRUE
      )

Which I tried to split it up like this:

prev_str_list<-split(prev_str,rep(1:ceiling(length(prev_str)/500),each=500))
      
      pr <- paste0("(", paste(sort(prev_str_list[[1]]), collapse = ","), ")", sep = "")
      
      # Create new vector map with all segments that form complex junctions
      execGRASS("v.extract", flags = c("overwrite", "quiet"), 
                parameters = list(
                  input = "streams_v", 
                  output = "complex_flows", 
                  type = "line", 
                  where = paste0("stream in ", pr)),
                ignore.stderr = T, intern = TRUE
      )
      
      for (pr in prev_str_list[-c(1)]) {
        pr <- paste0("(", paste(sort(pr), collapse = ","), ")", sep = "")
        
        # Create new vector map with all segments that form complex junctions
        execGRASS("v.extract", flags = c("overwrite", "quiet"), 
                  parameters = list(
                    input = "streams_v", 
                    output = "complex_flows_temp", 
                    type = "line", 
                    where = paste0("stream in ", pr)),
                  ignore.stderr = T, intern = TRUE
        )
        
        execGRASS("v.patch", flags = c("overwrite", "quiet"),
                  parameters = list(
                    input = c("complex_flows","complex_flows_temp"), 
                    output = "complex_flows2"),
                  ignore.stderr = T, intern = TRUE)
        
        execGRASS("g.rename", flags = c("overwrite", "quiet"),
                  parameters = list(
                    vector=c("complex_flows2","complex_flows")),
                  ignore.stderr = T, intern = TRUE)
        
      }

and the sql statement here also causes problems:

dt <- data.table(do.call(rbind, strsplit(
        execGRASS("db.select", 
                  parameters = list(
                    sql = paste0("select cat, stream, length, cum_length from streams_v where stream in", prev_str)
                  ), intern = T), 
        split = "\\|"))[-1, ])
      setattr(dt, "names", c("cat", "stream", "len", "cum_length"))
      dt[, `:=`(names(dt), lapply(.SD, as.numeric))]
      dt <- merge(dt, dt.junctions, by.y = "value", by.x = "stream")
      #setnames(dt, "value", "stream")
      dt[, newlen := -0.8 * cellsize]
      dt[len < (0.8 * cellsize), newlen:=-len]
      dt[, `:=`(pcat, seq(1, nrow(dt)))]
      write(paste(paste("P ", dt[, pcat], c(dt[, cat]), dt[, newlen], collapse = "\n")), 
            file = points)

which I tried to fix with:

pr <- paste0("(", paste(sort(prev_str_list[[1]]), collapse = ","), ")", sep = "")
      
      dt <- data.table(do.call(rbind, strsplit(
        execGRASS("db.select", 
                  parameters = list(
                    sql = paste0("select cat, stream, length, cum_length from streams_v where stream in", pr)
                  ), intern = T), 
        split = "\\|"))[-1, ])
      setattr(dt, "names", c("cat", "stream", "len", "cum_length"))
      dt[, `:=`(names(dt), lapply(.SD, as.numeric))]
      dt <- merge(dt, dt.junctions, by.y = "value", by.x = "stream")
      #setnames(dt, "value", "stream")
      dt[, newlen := -0.8 * cellsize]
      dt[len < (0.8 * cellsize), newlen:=-len]
      dt[, `:=`(pcat, seq(1, nrow(dt)))]
      write(paste(paste("P ", dt[, pcat], c(dt[, cat]), dt[, newlen], collapse = "\n")), 
            file = points)
      
      for (pr in prev_str_list[-c(1)]) {
        rw<-tail(1:length(unlist(prev_str_list[1:which.max(sapply(prev_str_list,function(x)all(x %in% pr)))])),
                 length(pr))
        
        pr <- paste0("(", paste(sort(pr), collapse = ","), ")", sep = "")
        
        dtt <- data.table(do.call(rbind, strsplit(
          execGRASS("db.select", 
                    parameters = list(
                      sql = paste0("select cat, stream, length, cum_length from streams_v where stream in", pr)
                    ), intern = T), 
          split = "\\|"))[-1, ])
        setattr(dtt, "names", c("cat", "stream", "len", "cum_length"))
        dtt[, `:=`(names(dtt), lapply(.SD, as.numeric))]
        dtt <- merge(dtt, dt.junctions, by.y = "value", by.x = "stream")
        #setnames(dt, "value", "stream")
        dtt[, newlen := -0.8 * cellsize]
        dtt[len < (0.8 * cellsize), newlen:=-len]
        dtt[, `:=`(pcat, rw)]
        write(paste(paste("P ", dtt[, pcat], c(dtt[, cat]), dtt[, newlen], collapse = "\n")), 
              file = points, append=T)
        dt<-rbind(dt,dtt)
      }

But it doesn't look like the "complex_flows" vector map is being generated correctly because this returns and empty data frame and the rest of the function fails after that

 dt.all_inflows <- data.frame(do.call(rbind, strsplit(
      execGRASS("v.to.db", flags = c("p", "quiet"), 
                parameters = list(
                  map = "complex_flows", 
                  type = "line", 
                  option = "end"
                ), intern = T), 
      split = "\\|"))[,-4], stringsAsFactors = FALSE)

any advice? I am dealing with a pretty large area, but can't set the stream threshold too low because there are some sampling sites on pretty small tributaries.

[MiKatt]

Thanks for your attempts for solving this issue. I am currently out of office and will have a look into it after my return at the beginning of August. Am 19.07.2021 um 00:17 schrieb p-schaefer:

|correct_compl_confluences()| seems to fail when there are too many complex confluences. Specifically, the |where| statement in this chunk fails: |prev_str <- paste0("(", paste(sort(prev_str), collapse = ","), ")", sep = "") execGRASS("v.extract", flags = c("overwrite", "quiet"), parameters = list( input = "streams_v", output = "complex_flows", type = "line", where = paste0("stream in ", prev_str)), ignore.stderr = T, intern = TRUE ) | Which I tried to split it up like this: |prev_str_list<-split(prev_str,rep(1:ceiling(length(prev_str)/500),each=500)) pr <- paste0("(", paste(sort(prev_str_list[[1]]), collapse = ","), ")", sep = "") # Create new vector map with all segments that form complex junctions execGRASS("v.extract", flags = c("overwrite", "quiet"), parameters = list( input = "streams_v", output = "complex_flows", type = "line", where = paste0("stream in ", pr)), ignore.stderr = T, intern = TRUE ) for (pr in prev_str_list[-c(1)]) { pr <- paste0("(", paste(sort(pr), collapse = ","), ")", sep = "") # Create new vector map with all segments that form complex junctions execGRASS("v.extract", flags = c("overwrite", "quiet"), parameters = list( input = "streams_v", output = "complex_flows_temp", type = "line", where = paste0("stream in ", pr)), ignore.stderr = T, intern = TRUE ) execGRASS("v.patch", flags = c("overwrite", "quiet"), parameters = list( input = c("complex_flows","complex_flows_temp"), output = "complex_flows2"), ignore.stderr = T, intern = TRUE) execGRASS("g.rename", flags = c("overwrite", "quiet"), parameters = list( vector=c("complex_flows2","complex_flows")), ignore.stderr = T, intern = TRUE) } | and the sql statement here also causes problems: |dt <- data.table(do.call(rbind, strsplit( execGRASS("db.select", parameters = list( sql = paste0("select cat, stream, length, cum_length from streams_v where stream in", prev_str) ), intern = T), split = "\\|"))[-1, ]) setattr(dt, "names", c("cat", "stream", "len", "cum_length")) dt[, `:=`(names(dt), lapply(.SD, as.numeric))] dt <- merge(dt, dt.junctions, by.y = "value", by.x = "stream") #setnames(dt, "value", "stream") dt[, newlen := -0.8 * cellsize] dt[len < (0.8 * cellsize), newlen:=-len] dt[, `:=`(pcat, seq(1, nrow(dt)))] write(paste(paste("P ", dt[, pcat], c(dt[, cat]), dt[, newlen], collapse = "\n")), file = points) | which I tried to fix with: |pr <- paste0("(", paste(sort(prev_str_list[[1]]), collapse = ","), ")", sep = "") dt <- data.table(do.call(rbind, strsplit( execGRASS("db.select", parameters = list( sql = paste0("select cat, stream, length, cum_length from streams_v where stream in", pr) ), intern = T), split = "\\|"))[-1, ]) setattr(dt, "names", c("cat", "stream", "len", "cum_length")) dt[, `:=`(names(dt), lapply(.SD, as.numeric))] dt <- merge(dt, dt.junctions, by.y = "value", by.x = "stream") #setnames(dt, "value", "stream") dt[, newlen := -0.8 * cellsize] dt[len < (0.8 * cellsize), newlen:=-len] dt[, `:=`(pcat, seq(1, nrow(dt)))] write(paste(paste("P ", dt[, pcat], c(dt[, cat]), dt[, newlen], collapse = "\n")), file = points) for (pr in prev_str_list[-c(1)]) { rw<-tail(1:length(unlist(prev_str_list[1:which.max(sapply(prev_str_list,function(x)all(x %in% pr)))])), length(pr)) pr <- paste0("(", paste(sort(pr), collapse = ","), ")", sep = "") dtt <- data.table(do.call(rbind, strsplit( execGRASS("db.select", parameters = list( sql = paste0("select cat, stream, length, cum_length from streams_v where stream in", pr) ), intern = T), split = "\\|"))[-1, ]) setattr(dtt, "names", c("cat", "stream", "len", "cum_length")) dtt[, `:=`(names(dtt), lapply(.SD, as.numeric))] dtt <- merge(dtt, dt.junctions, by.y = "value", by.x = "stream") #setnames(dt, "value", "stream") dtt[, newlen := -0.8 * cellsize] dtt[len < (0.8 * cellsize), newlen:=-len] dtt[, `:=`(pcat, rw)] write(paste(paste("P ", dtt[, pcat], c(dtt[, cat]), dtt[, newlen], collapse = "\n")), file = points, append=T) dt<-rbind(dt,dtt) } | But it doesn't look like the "complex_flows" vector map is being generated correctly because this returns and empty data frame and the rest of the function fails after that |dt.all_inflows <- data.frame(do.call(rbind, strsplit( execGRASS("v.to.db", flags = c("p", "quiet"), parameters = list( map = "complex_flows", type = "line", option = "end" ), intern = T), split = "\\|"))[,-4], stringsAsFactors = FALSE) | any advice? I am dealing with a pretty large area, but can't set the stream threshold too low because there are some sampling sites on pretty small tributaries. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#25>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEYLU6ZW5AII2SKJYAD2AFLTYNHIXANCNFSM5ASRXLWQ>.

-- Dr. Mira Kattwinkel Quantitative Landscape Ecology Institute for Environmental Sciences University of Koblenz-Landau Fortstraße 7 76829 Landau Germany Fon: + 49 6341 280-31553

[MiKatt]

I am now looking into this. However, can you please give me some more information about fails. What error messages do you get? Would it be possible to send me the stream network you get from derive streams (or the source dem and your settings)? What OS and what R and GRASS version are you using?
Thanks,
Mira

[MiKatt]

I was so far unable to reproduce the error with > 900 complex confluences. However, I updated the code as I also got an error further down in the v.db.update.

cut.str <- paste(c(dt.smallcut[stream != str_new_small, cat_small], 
                   dt.largecut[stream != str_new_large, cat_large]), collapse = ",")
cut.str<-paste0("(", cut.str, ")",sep="")`
execGRASS("v.db.update", flags = c("quiet"),
              parameters = list(
                map = "streams_v",
                column = "stream",
                where = paste0("cat_ in ",cut.str),
                query_column = "str_new"
              ))

How many complex confluences are you trying to fix?

[p-schaefer]

I am using R 4.0.4 on a Linux server running the RStudio Open-Source Server with Grass 7.8. I had an unfortunately large number of complex confluences (I think around 20,000). I can't get the exact error message right now because the server is busy calculating some other landscape attributes at the moment (but can retry a bit later in the week), but it was something along the lines of an SQL error with too many WHERE variables. My DEM is about 1.5GB, if you want me to share it. I was using a projected coordinate system (EPSG 3005), and had a burned stream network.

[MiKatt]

I updated the code so that is should now work with many complex confluences. The problem were too long strings in sql statements passed to GRASS.
Would you please check if it works now and give me some feedback? Please use the "dev" branch:
devtools::install_github("MiKatt/openSTARS", ref = "dev")

[p-schaefer]

Yes, it does seem to be running again. Thank you. I'll let you know if it is able to finish the operation.

[p-schaefer]

Everything is working now. Thank you.