JSON Error on qgis_configure #176

asitemade4u · 2023-09-18T22:25:23Z

Hi,

I am using Ubuntu 20.x
I installed the last version of QGis from the unstable branch. For now, I have left the installation as is, e.g. I have installed no QGis plugin
I installed the last version of R
I installed the last stable version of qgisprocess

When I execute qgisprocess::qgis_configure(), I get this message then the error:

GIS version is now set to: 3.26.3-Buenos
Using JSON for output serialization.
Using JSON for input serialization.
1 out of 4 available processing provider plugins are enabled.
parse error: premature EOF
                                       {   "gdal_version": "3.4.3",   
                     (right here) ------^

Please help,
S.

The text was updated successfully, but these errors were encountered:

florisvdh · 2023-09-20T09:05:10Z

Does this reproduce the error?

qgis_algorithms(query = TRUE, quiet = FALSE)

If so, where does it happen... try:

    result <- qgis_run(args = c("list", "--json"), encoding = "UTF-8")
    result_parsed <- jsonlite::fromJSON(result$stdout)

Also provide the output of:

qgis_version(debug = TRUE)
sessioninfo::session_info() # if needed, first do install.packages("sessioninfo")
system("apt policy qgis")

There's some chance it would pose no problem with the current QGIS release (3.32 + backports in LTR 3.28.x), which however is not built by the QGIS community for Ubuntu 20.04 (latest Ubuntu LTS is 22.04).

asitemade4u · 2023-09-20T13:47:28Z

Hi Floris,

Thanks for your prompt response.

Yes, the error reported was:

Error in if (qgis_using_json_output()) { : argument is of length zero

An error prevented to go further than the first command line:

! command is not a string (length 1 character)
Type .Last.error to see the more details.

and here is the outcome when I execute .Last.error:

1. qgisprocess::qgis_run(args = c("list", "--json"), encoding = "UTF-8")
2. withr::with_envvar(env, processx::run(path, args, ...), )
3. base::force(code)
4. processx::run(path, args, ...)
5. process$new(command, args, echo_cmd = echo_cmd, wd = wd, windows_verbatim_args = windows_verbatim_args, ...
6. local initialize(...)
7. processx:::process_initialize(self, private, command, args, stdin, stdout, ...
8. processx:::assert_that(is_string(command), is.character(args), is_std_conn(stdin), ...
9. processx:::throw(new_assert_error(attr(res, "msg")))

qgis_version(debug = TRUE) generates this error:

Error in strsplit(qgisprocess_cache$version, "-") :  non-character argument

I attach the outcome of session_info(): sessioninfo.txt
Here is the outcome of system("apt policy qgis"):

qgis:
  Installed: 1:3.26.3+32focal-ubuntugis
  Candidate: 1:3.26.3+32focal-ubuntugis
  Version table:
 *** 1:3.26.3+32focal-ubuntugis 500
        500 https://qgis.org/ubuntugis focal/main amd64 Packages
        100 /var/lib/dpkg/status
     3.22.7+dfsg-1~focal2 500
        500 http://ppa.launchpad.net/ubuntugis/ubuntugis-unstable/ubuntu focal/main amd64 Packages
     3.10.4+dfsg-1ubuntu2 500
        500 http://archive.ubuntu.com/ubuntu focal/universe amd64 Packages

florisvdh · 2023-09-20T14:13:33Z

Thank you!

qgis_version(debug = TRUE) generates this error:

Try to rerun as:

qgis_version(query = TRUE, debug = TRUE)

asitemade4u · 2023-09-20T14:19:36Z

I am afraid it yields the same:

Error: 
! command is not a string (length 1 character)
Type .Last.error to see the more details.

and the backtrace:

 1. qgisprocess::qgis_version(query = TRUE, debug = TRUE)
 2. qgisprocess:::qgis_query_version(quiet = quiet)
 3. base::tryCatch({ ...
 4. base::tryCatchList(expr, classes, parentenv, handlers)
 5. base::tryCatchOne(expr, names, parentenv, handlers[[1L]])
 6. value[[3L]](cond)
 7. qgisprocess::qgis_run(args = character(0))
 8. withr::with_envvar(env, processx::run(path, args, ...), )
 9. base::force(code)
10. processx::run(path, args, ...)
11. process$new(command, args, echo_cmd = echo_cmd, wd = wd, windows_verbatim_args = windows_verbatim_args, ...
12. local initialize(...)
13. processx:::process_initialize(self, private, command, args, stdin, stdout, ...
14. processx:::assert_that(is_string(command), is.character(args), is_std_conn(stdin), ...
15. processx:::throw(new_assert_error(attr(res, "msg")))

florisvdh · 2023-09-20T14:24:44Z

And this?

qgis_query_path(quiet = FALSE)

asitemade4u · 2023-09-20T14:27:33Z

This time the error is different:
I executed: qgisprocess::qgis_query_path(quiet = FALSE) (for some reason, I have to prefix all instructions by the name of the package) and got:

Error: 'qgis_query_path' is not an exported object from 'namespace:qgisprocess'

As though qgis_query_path did not belong to the package...

florisvdh · 2023-09-20T14:29:34Z

Sorry, you need to do qgis_path(query = TRUE, quiet = FALSE).

asitemade4u · 2023-09-20T14:31:26Z

I had checked it out ;o))
The error returned is:

Error in if (qgisprocess_cache$path == "qgis_process") { : 
  argument is of length zero

florisvdh · 2023-09-20T14:33:49Z

Also if you restart R?

asitemade4u · 2023-09-20T14:35:12Z

FYI, here is the environment I am using:

self-hosted Proxmox container
Ubuntu 20.0x (last version, updated)
code-server
connection from my machine (Debian Linux, last version) connected with code-server using SSH and Mutagen to keep the connection alive.

Everything works perfectly fine otherwise.

asitemade4u · 2023-09-20T14:38:03Z

So, as you directed, I rebooted the whole container.
Same errors....

florisvdh · 2023-09-20T14:40:07Z

OK thanks. Then let's go the terminal to check things with QGIS first. Can you run:

qgis --version
which qgis
which qgis_process
qgis_process --version

asitemade4u · 2023-09-20T14:42:22Z

QGIS 3.26.3-Buenos Aires 'Buenos Aires' (65e4edfdad)
/usr/bin/qgis
/usr/bin/qgis_process
Ah, this might be the reason (the container is running headless):

qt.qpa.xcb: could not connect to display 
qt.qpa.plugin: Could not load the Qt platform plugin "xcb" in "" even though it was found.
This application failed to start because no Qt platform plugin could be initialized. Reinstalling the application may fix this problem.

Available platform plugins are: eglfs, linuxfb, minimal, minimalegl, offscreen, vnc, xcb.

Aborted

asitemade4u · 2023-09-20T14:43:15Z

BTW, I have found NO indication on how to fix the issue above, which I have encountered before...
Any idea is welcome because one would then be able to use your plugin with a remote QGis, through SSH...

florisvdh · 2023-09-20T14:43:54Z

Try:

QT_QPA_PLATFORM = "offscreen" qgis_process --version

asitemade4u · 2023-09-20T14:44:28Z

I get: /bin/sh: 5: QT_QPA_PLATFORM: not found

florisvdh · 2023-09-20T14:44:54Z

QT_QPA_PLATFORM="offscreen" qgis_process --version

asitemade4u · 2023-09-20T14:46:10Z

Here you go:

QStandardPaths: XDG_RUNTIME_DIR not set, defaulting to '/tmp/runtime-stsinc'
QGIS 3.26.3-Buenos Aires 'Buenos Aires' (65e4edfdad)
QGIS code revision 65e4edfdad
Qt version 5.12.8
Python version 3.8.10
GDAL/OGR version 3.4.3
PROJ version 8.2.0
EPSG Registry database version v10.038 (2021-10-21)
GEOS version 3.10.2-CAPI-1.16.0
SQLite version 3.31.1
OS Ubuntu 20.04.6 LTS

florisvdh · 2023-09-20T14:50:06Z

Does this work too ?

QT_QPA_PLATFORM="offscreen" qgis_process list --json | wc -l

asitemade4u · 2023-09-20T14:52:09Z

Yup (correction):

QStandardPaths: XDG_RUNTIME_DIR not set, defaulting to '/tmp/runtime-stsinc'
6525

florisvdh · 2023-09-20T15:01:15Z

In a new R session, can you run library(qgisprocess), and after that run ls.str(envir = qgisprocess:::qgisprocess_cache). (mind the three colons) Please paste the entire output here. If you like you can collapse lengthy outputs inside a <details> HTML tag, like:

<details><summary>Title</summary>

```r
Some code
#> Some output
```

</details>

Which gives:

Title

Some code
#> Some output

florisvdh · 2023-09-20T15:06:21Z

BTW if you want to get rid of the XDG_RUNTIME_DIR warning, have a look at https://stackoverflow.com/questions/59790350

asitemade4u · 2023-09-20T15:06:30Z

So, after rebooting the Proxmox CT, I get far less than what you mentioned:

algorithms :  NULL
loaded_from :  NULL
path :  NULL
plugins :  NULL
use_json_output :  NULL
version :  NULL

florisvdh · 2023-09-20T15:08:39Z

Thanks. What did library(qgisprocess) output before that?

BTW a typical output would be (ignore the fact I'm using a locally built QGIS here):

> library(qgisprocess)
Attempting to load the package cache ... Success!
QGIS version: 3.33.0-Master, development state 4e83c82a
Having access to 1973 algorithms from 14 QGIS processing providers.
Run `qgis_configure(use_cached_data = TRUE)` to reload cache and get more details.
> ls.str(envir = qgisprocess:::qgisprocess_cache)
algorithms : tibble [1,973 × 24] (S3: tbl_df/tbl/data.frame)
loaded_from :  chr "/home/floris/.cache/R-qgisprocess/cache-0.1.0.rds"
path :  chr "/home/floris/git_repositories2/QGIS/build-4e83c82a/output/bin/qgis_process"
plugins : tibble [10 × 2] (S3: tbl_df/tbl/data.frame)
use_json_output :  logi TRUE
version :  chr "3.33.0-Master, development state 4e83c82a"

asitemade4u · 2023-09-20T15:11:46Z

You know, it is improving since I applied the fix to XDG_RUNTIME_DIR.
Now, when I execute library(qgisprocess) it does not say anything...

florisvdh · 2023-09-20T15:12:56Z

when I execute library(qgisprocess) it does not say anything...

Probably because it was loaded already. First restart R, then it should say something.

asitemade4u · 2023-09-20T15:14:03Z

(unfortunately) I spoke too fast....
After rebooting the container, I get:

Attempting to load the package cache ... No cache found.
Will try to reconfigure qgisprocess and build new cache ...

>>> PROBLEM encountered: couldn't build package cache! <<<
The 'qgis_process' command-line utility was either not found or
did not fulfil the needs to build the package cache.
Please run `qgis_configure()` to fix this and rebuild the cache.
See its documentation if you need to preset the path of qgis_process.
If the problem persists, make sure that you correctly installed QGIS
for your operating system using the instructions at
https://download.qgis.org.

asitemade4u · 2023-09-20T15:14:58Z

And when I tried to qgis_configure() I got the exact same output as before....

florisvdh · 2023-09-20T15:16:02Z

And when I tried to qgis_configure() I got the exact same output as before....

Please paste the output anyway, there have been several. I.e. does it report a QGIS version for example?

asitemade4u · 2023-09-20T15:16:52Z

getOption('qgisprocess.path') was not found.
Sys.getenv('R_QGISPROCESS_PATH') was not found.
Trying 'qgis_process' on PATH...
Success!
Now using 'qgis_process' in the system PATH.
>>> If you need another installed QGIS instance, run `qgis_configure()`;
    see `?qgis_configure` if you need to preset the path of 'qgis_process'.

QGIS version is now set to: 3.26.3-Buenos
Using JSON for output serialization.
Using JSON for input serialization.
1 out of 4 available processing provider plugins are enabled.
parse error: premature EOF
                                       {   "gdal_version": "3.4.3",

florisvdh · 2023-09-22T13:15:43Z

Thanks!

Maybe character encoding is the culprit?

Everything should be UTF-8 by default in processx::run(), I don't think that's the problem here. It still seems like the JSON string missed the final characters. However you may try by dropping options(encoding = "native.enc") in both Rprofile.* files.

So now my questions for the new state are:

can you load {qgisprocess}? Either 0.1.0 or 0.1.0.9177, shouldn't matter.
if not, are the double quote and curly brace still lacking if you write a newly made result object (made as in JSON Error on qgis_configure #176 (comment)) to a jsonstring.txt file? (cat(result$stdout, file = "jsonstring.txt")). Also check that result$stderr is still empty.
if double quote and curly brace are effectively included, then verify that result_parsed <- jsonlite::fromJSON(result$stdout) works. If it doesn't, it means that the missing final characters in the JSON string weren't the only problem.

florisvdh · 2023-09-22T13:20:24Z

If it doesn't, it means that the missing final characters in the JSON string weren't the only problem.

However when I used your first (malformed) jsonstring.txt file I could effectively reproduce your problem with jsonlite::fromJSON() and make it work after adding the final two characters (#176 (comment)), so I don't believe the encoding is at stake here.

So I do believe it's about those characters, and you already hit the solution I think.

florisvdh · 2023-09-22T13:22:21Z

BTW the 'NO carriage return after the last curly brace' (#176 (comment)) is normal and shouldn't be the problem.

asitemade4u · 2023-09-22T13:23:00Z

I tried commenting out the native.enc instructions and it created a lot of trouble . will reinstate them.

asitemade4u · 2023-09-22T13:26:54Z

As per your question above, I executed library("qgisprocess") andd got:

Attempting to load the package cache ... No cache found.
Will try to reconfigure qgisprocess and build new cache ...

>>> PROBLEM encountered: couldn't build package cache! <<<
The 'qgis_process' command-line utility was either not found or
did not fulfil the needs to build the package cache.
Please run `qgis_configure()` to fix this and rebuild the cache.
See its documentation if you need to preset the path of qgis_process.
If the problem persists, make sure that you correctly installed QGIS
for your operating system using the instructions at
https://download.qgis.org.

asitemade4u · 2023-09-22T13:28:59Z

Then I ran:

result <- withr::with_envvar(
  c(QT_QPA_PLATFORM="offscreen"), 
  processx::run("qgis_process", c("list", "--json"), encoding = "UTF-8")
)
 str(result)

And I got the XDG error again -- it had disappeared!

List of 4
 $ status : int 0
 $ stdout : chr "{\n  \"gdal_version\": \"3.4.3\",\n  \"geos_version\": \"3.10.2-CAPI-1.16.0\",\n  \"proj_version\": \"Rel. 8.2."| __truncated__
 $ stderr : chr "QStandardPaths: XDG_RUNTIME_DIR not set, defaulting to '/tmp/runtime-root'\n"
 $ timeout: logi FALSE

asitemade4u · 2023-09-22T13:30:21Z

And result_parsed <- jsonlite::fromJSON(result$stdout) gave me, as before:

Error: parse error: premature EOF
                                       {   "gdal_version": "3.4.3",   
                     (right here) ------^

asitemade4u · 2023-09-22T13:31:29Z

NB. I know why I got the XDG error: I executed R from root this time, not from my usual user, stsinc

No, I tried executing R from stsinc and I got the XDG error again...

florisvdh · 2023-09-22T13:35:12Z

Thanks for pursuing this.

Indeed the XDG error still needs more attention; from what we've seen it causes the trouble.

You can also set XDG_RUNTIME_DIR specifically for R if you like, in ~/.Renviron (text file with line XDG_RUNTIME_DIR="/home/...", read and set by R when it launches). However the .bashrc approach is better since it also applies when calling qgis_process directly in a shell.

asitemade4u · 2023-09-22T13:44:12Z

OK, following your advice, I added XDG_RUNTIME_DIR in Renviron and now it works all the time.

asitemade4u · 2023-09-22T13:45:44Z

Do you think I should add a bug report about qgis_process to QGis own bug reporting, referencing this thread?

florisvdh · 2023-09-22T13:57:31Z

now it works all the time

Including {qgisprocess} I assume?

Do you think I should add a bug report about qgis_process

No, because the XDG Base Directory Specification (XDG_RUNTIME_DIR being set) is not tied to QGIS; it is tied to your system as I understand it. So your solution just satisfies a need that qgis_process has, but also various other programs, I expect. It demonstrates that your Ubuntu 22.04 setup is different from most (I think). Normally this variable is just pre-set already (see also https://askubuntu.com/questions/872792/what-is-xdg-runtime-dir for more info). Here I get:

$ echo $XDG_RUNTIME_DIR
/run/user/1000

Anyway it is good to have this documented here, so that future users can find this too.

asitemade4u · 2023-09-22T14:08:45Z

Including {qgisprocess} I assume?

Nope, still behaving the same. And when executing qgis_configure() I get our old foe:

1 out of 3 available processing provider plugins are enabled.
parse error: premature EOF
                                       {   "gdal_version": "3.4.3",   
                     (right here) ------^

asitemade4u · 2023-09-22T14:09:23Z

I have to go to a meeting now. Thank you again for your tenacity. Hope we will overcome...

florisvdh · 2023-09-22T14:10:15Z

Do you think I should add a bug report about qgis_process

Well, on second thought, you could post an issue in the QGIS repo saying that JSON strings exported by qgis_process are corrupted when XDG_RUNTIME_DIR is not set. After all, qgis_process does work when that variable is not set, only not as expected and that can be regarded an issue. You can demonstrate it comparing following commands that generate bad vs good output:

QT_QPA_PLATFORM="offscreen" qgis_process list --json | tail -5

QT_QPA_PLATFORM="offscreen" XDG_RUNTIME_DIR="path/..." qgis_process list --json | tail -5

asitemade4u · 2023-09-22T14:12:47Z

It does not complain, so I assume it works ;o))

florisvdh · 2023-09-22T14:13:12Z

Including {qgisprocess} I assume?

Nope, still behaving the same. And when executing qgis_configure() I get our old foe:
1 out of 3 available processing provider plugins are enabled.
parse error: premature EOF
                                       {   "gdal_version": "3.4.3",   
                     (right here) ------^

So then you probably still have the XDG stderr too, as in your #176 (comment)?

florisvdh · 2023-09-22T14:15:05Z

It does not complain, so I assume it works ;o))

Well, library(qgisprocess) will only work if qgis_process predictably emits correct JSON strings, it seems that is the problem anyway. I advise you to make a QGIS issue indeed, as stated in above #176 (comment).

asitemade4u · 2023-09-22T14:15:33Z

To sum it up:

library(qgisprocess) works fine, does not complain
but qgisprocess::qgis_configure() still yields the error:

parse error: premature EOF
                                       {   "gdal_version": "3.4.3",   
                     (right here) ------^

asitemade4u · 2023-09-22T14:16:06Z

So a bit of an incoherence here.

florisvdh · 2023-09-22T14:16:13Z

library(qgisprocess) does not complain: also after restarting R?

asitemade4u · 2023-09-22T14:17:21Z

No, as soon as I restart R, the problem with the cache occurs again.

florisvdh · 2023-09-22T14:19:45Z

You can now also install qgisprocess 0.1.0.9177 as follows:

install.packages('qgisprocess', repos = c('https://r-spatial.r-universe.dev', 'https://cloud.r-project.org'))

It should give you more feedback and is less likely to end up in an unusable state. I.e., things like qgis_run() and qgis_version() etc should work. The complaint about the cache and about missing algorithms will remain though as long as qgis_process list --json gives a bad JSON string ...

florisvdh · 2023-09-22T14:22:24Z

Sidenote: rerunning library(qgisprocess) in the same R session after it was already loaded a first time, will never return anything, as its loading will be skipped.

asitemade4u · 2023-09-22T14:36:21Z

I have installed the last version (above) and got, as expected, the Error: parse error: premature EOF error.
Thank you again!

florisvdh · 2023-09-22T14:38:07Z

OK thanks.

In conclusion:

post the QGIS issue demonstrating the effect of XDG_RUNTIME_DIR along with your system setup (*)
try to figure out how to get rid of this 'XDG_RUNTIME_DIR not set' warning on your system, as it appears to be tied to the corrupt JSON. R package {qgisprocess} cannot solve this for you, so closing here, but you're welcome to add further updates on your problem.
there's the legacy no-JSON method in {qgisprocess} that you can use as fallback, explained in an earlier comment.

(*) however a missing XDG_RUNTIME_DIR does not harm the JSON string I get on my system; even though I do get the warning in such case:

$ QT_QPA_PLATFORM="offscreen" XDG_RUNTIME_DIR="" qgis_process list --json | tail -5
QStandardPaths: XDG_RUNTIME_DIR not set, defaulting to '/tmp/runtime-floris'
  "python_version": "3.10.12",
  "qgis_code_revision": "67d46100b5",
  "qgis_version": "3.32.3-Lima",
  "qt_version": "5.15.3"
}

But if you can reproduce the contrasting behaviours in the shell (see #176 (comment)), then that's a good starting point for an issue at QGIS!

asitemade4u · 2023-09-22T14:38:45Z

Thanks!

florisvdh closed this as completed Sep 22, 2023

florisvdh added the QGIS label Sep 22, 2023

florisvdh mentioned this issue Sep 4, 2024

locale problem in qgis_query_algorithms() #218

Closed

2 tasks

JSON Error on qgis_configure #176

JSON Error on qgis_configure #176

Comments

asitemade4u commented Sep 18, 2023

florisvdh commented Sep 20, 2023

asitemade4u commented Sep 20, 2023

florisvdh commented Sep 20, 2023

asitemade4u commented Sep 20, 2023

florisvdh commented Sep 20, 2023

asitemade4u commented Sep 20, 2023

florisvdh commented Sep 20, 2023

asitemade4u commented Sep 20, 2023

florisvdh commented Sep 20, 2023

asitemade4u commented Sep 20, 2023

asitemade4u commented Sep 20, 2023

florisvdh commented Sep 20, 2023

asitemade4u commented Sep 20, 2023

asitemade4u commented Sep 20, 2023 • edited Loading

florisvdh commented Sep 20, 2023

asitemade4u commented Sep 20, 2023

florisvdh commented Sep 20, 2023

asitemade4u commented Sep 20, 2023

florisvdh commented Sep 20, 2023

asitemade4u commented Sep 20, 2023

florisvdh commented Sep 20, 2023 • edited Loading

florisvdh commented Sep 20, 2023

asitemade4u commented Sep 20, 2023

florisvdh commented Sep 20, 2023

asitemade4u commented Sep 20, 2023

florisvdh commented Sep 20, 2023 • edited Loading

asitemade4u commented Sep 20, 2023 • edited Loading

asitemade4u commented Sep 20, 2023

florisvdh commented Sep 20, 2023

asitemade4u commented Sep 20, 2023

florisvdh commented Sep 22, 2023 • edited Loading

florisvdh commented Sep 22, 2023

florisvdh commented Sep 22, 2023

asitemade4u commented Sep 22, 2023

asitemade4u commented Sep 22, 2023

asitemade4u commented Sep 22, 2023

asitemade4u commented Sep 22, 2023

asitemade4u commented Sep 22, 2023 • edited Loading

florisvdh commented Sep 22, 2023

asitemade4u commented Sep 22, 2023

asitemade4u commented Sep 22, 2023

florisvdh commented Sep 22, 2023

asitemade4u commented Sep 22, 2023

asitemade4u commented Sep 22, 2023

florisvdh commented Sep 22, 2023

asitemade4u commented Sep 22, 2023

florisvdh commented Sep 22, 2023

florisvdh commented Sep 22, 2023 • edited Loading

asitemade4u commented Sep 22, 2023

asitemade4u commented Sep 22, 2023

florisvdh commented Sep 22, 2023

asitemade4u commented Sep 22, 2023

florisvdh commented Sep 22, 2023 • edited Loading

florisvdh commented Sep 22, 2023

asitemade4u commented Sep 22, 2023

florisvdh commented Sep 22, 2023 • edited Loading

asitemade4u commented Sep 22, 2023

asitemade4u commented Sep 20, 2023 •

edited

Loading

florisvdh commented Sep 20, 2023 •

edited

Loading

florisvdh commented Sep 20, 2023 •

edited

Loading

asitemade4u commented Sep 20, 2023 •

edited

Loading

florisvdh commented Sep 22, 2023 •

edited

Loading

asitemade4u commented Sep 22, 2023 •

edited

Loading

florisvdh commented Sep 22, 2023 •

edited

Loading

florisvdh commented Sep 22, 2023 •

edited

Loading

florisvdh commented Sep 22, 2023 •

edited

Loading