xhgui: document to insert contains invalid key: keys cannot contain "." #209

SimonDeconde · 2017-09-28T02:19:48Z

Hi there

I'm trying to install xhprof +xhgui, but something is not working with mongoDB. After a full setup, when I browse a page on my project, I'm getting errors in apache logs saying:

xhgui - document to insert contains invalid key: keys cannot contain ".": "main()==>load::myprojecttoprofile/index.php"

(I haven't been able to find any similar ticket issues for xhgui on this specific topic)

I'm using:

mongoDB 2.4.9
the latest xhgui git code (master branch, but I've also tried release 0.8.1 and 0.8.0)
xhprof 0.9.2
Ubuntu Ubuntu 14.04.5 LTS (virtual machine)

Is there something wrong with my setup?

Thanks!

felixkk · 2017-09-28T09:09:23Z

Same error on a debian server...
In the requirements, it says we should use "Mongodb 2.2.0 or later".
Maybe 2.4.9 is too late ?

markstory · 2017-09-28T17:25:37Z

Sounds like at some point mongo has changed what is valid in keys. I'm still using an old mongo (2.2.4). If . isn't valid in new mongo, we'll likely need to rebuild xhgui 😢

martins-olekss · 2017-09-30T21:27:56Z

Ran in same issue just now. Everywhere I search, only workaround is to modify keys before saving - in this specific case doesn't sound too possible.

Using mongo 2.6.10

As far I have found there is no setting to turn this key check off.

lauripiisang · 2017-10-02T09:53:16Z

This is odd. I'm using MongoDB 3.0.14 and have no such issue.

Do note that top-level keys probably cannot contain such a profile, this must be stored in a lower level key!

e.g. you cannot have

{
  "a.b" : {...}
}

but

{
  "a" : {
    "a.b" : { ... }
  }

is valid, at least from what I understand and what I see in my database.

I encountered a similar error and it turned out I was creating the profile object wrong.

lauripiisang · 2017-10-02T09:53:50Z

Can you please post the full document at the time of insert (use debugger?)

martins-olekss · 2017-10-02T12:48:29Z

@lauripiisang do I understand correctly, you wish to see $data content just before
$saver->save($data); in external/header.php ?

Just as side-note - noticed that this exception happens only when I use auto_append in server (nginx) config. If I disable auto_append and try to encompass code with xhprof_enable and xhprof_disable functions - exception doesn't happen anymore. Even more - new records show up in xhgui interface. But records are without any measurements and if I go inside each request - theres xhgui error

#0 /var/www/xhgui/src/Xhgui/Profile.php(359): Slim\Slim::handleErrors(2, 'Invalid argumen...', '/var/www/xhgui/...', 359, Array)
#1 /var/www/xhgui/src/Xhgui/Controller/Run.php(69): Xhgui_Profile->calculateSelf()
#2 /var/www/xhgui/src/routes.php(30): Xhgui_Controller_Run->view()
#3 [internal function]: {closure}()
#4 /var/www/xhgui/vendor/slim/slim/Slim/Route.php(468): call_user_func_array(Object(Closure), Array)
#5 /var/www/xhgui/vendor/slim/slim/Slim/Slim.php(1355): Slim\Route->dispatch()
#6 /var/www/xhgui/vendor/slim/slim/Slim/Middleware/Flash.php(85): Slim\Slim->call()
#7 /var/www/xhgui/vendor/slim/slim/Slim/Middleware/MethodOverride.php(92): Slim\Middleware\Flash->call()
#8 /var/www/xhgui/vendor/slim/slim/Slim/Middleware/SessionCookie.php(110): Slim\Middleware\MethodOverride->call()
#9 /var/www/xhgui/src/Xhgui/Middleware/Render.php(11): Slim\Middleware\SessionCookie->call()
#10 /var/www/xhgui/vendor/slim/slim/Slim/Slim.php(1300): Xhgui_Middleware_Render->call()
#11 /var/www/xhgui/webroot/index.php(10): Slim\Slim->run()
#12 {main}

lauripiisang · 2017-10-02T13:35:24Z

do I understand correctly, you wish to see $data content just before
$saver->save($data); in external/header.php ?

That would do, but I'd prefer the document submitted to mongo (deeper down the call stack).

Just as side-note - noticed that this exception happens only when I use auto_append in server (nginx) config.

With header.php, you're supposed to be using auto_prepend, not append!

Can I ask you to try out https://github.com/perftools/xhgui-collector in your project and see if you still encounter problems?

I use the register_shutdown_function like provided in header.php to register a shutdown function which does:

$data = array('profile' => xhprof_disable());
// some filtering to prevent leakage of user passwords via server_http_password etc 
// filling $data['meta'] like in header.php
// $saver is instance of Xhgui_Saver, constructed beforehand.
$saver->save($data);

I think what's different here is that you don't have the xhprof_disable call under profile key.

lauripiisang · 2017-10-02T13:38:41Z

I think I can get the profiling library I developed open-sourced so you can have an easier time at using this.

emilycraps · 2017-10-06T08:38:40Z

I have the same issue. This is the output for the $data variable that will be saved to Mongo:

Mongo version: v2.4.14
XHProf version: 0.9.4
PHP version: 5.6.16

samt2497 · 2017-10-10T21:57:53Z

Had the issue, fixed it by encoding the stored profile.
It might not be good for performance but helps anyone having this issue currently.

#210

Current environment
Php 7.1.8
mongo extension: master
mongodb 3.4.4

markstory · 2017-10-10T23:11:31Z

Thanks @samt2497 Closing as there is a pull request open now.

lauripiisang · 2017-10-11T08:15:39Z

Well, I found the reason why it worked for some and didn't work for others:

https://jira.mongodb.org/browse/PHPC-712 seems to have been a bug in the adapters causing the ignorance of said dot-check. That is to say, the documents could still be inserted, just not updated afaik.
The newer versions of the adapters fix this by not allowing keys with dots.

While @samt2497 -s provided unicode-based encoding-decoding solution is a band-aid for now, I think a long term solution should include solving the problem by restructuring the data.

I propose moving from

{
  "profiles": { 
    "main()==>load::shop/test.php" : {
      "ct" : 1
    }
  }

to

{
  "profiles": { 
    "some-uuid-here-1234" : {
      "stackName" : "main()==>load::shop/test.php",
      "ct" : 1
    }
  }

This preserves the possibility to view a sub-key on its own and avoids the dots-in-keys problem.

Suggestions welcome for stackName

lauripiisang · 2017-10-11T08:20:28Z

Reopening issue to get some opinions on the data restructure solution.

markstory · 2017-10-11T23:44:13Z

If we have to restructure the data should we also consider moving away from mongodb entirely? A few features like multi domain support and ad hoc queries are more complex because of mongo. Furthermore the awkward datetime handling and limited set of aggregation functions limit our ability to offer dashboards and summary data.

felixkk · 2017-10-14T06:50:10Z

@markstory
That would be good news !
Everything around me is using MySQL, having to install mongodb is a bit painful when you don't know.

The sysadmin finally managed to get around the bug and install xhgui, and it looks excellent.
If it could just have a more "friendlier" install phase, it would be a new must have for us.

michaelbutler · 2017-10-14T17:27:55Z

For what it's worth, I have a private fork of XHGUI that uses MySQL. I'm storing the data in MySQL in a separate project, and XHGUI is just reading off that table. To do this I tried to abstract the data storage away as much as possible, so one could "switch" to a different storage driver using the config file. Not everything is supported with this MySQL abstraction though, such as some of the aggregation, searching & filtering tools. But it works great to just load up individual profiles.

Overall, I think having XHGUI be built in a way that developers can implement their own storage drivers (memcached, redis, S3, etc.) would be great.

I also added an authentication system. That might be best kept as a separate PR though.

If I get some time I can try to clean up & push my changes back to XHGUI. I would love for the project to be converted to full PSR-4 and PSR-2 coding standard as well just to keep consistent and have better class names.

dhampik · 2017-10-25T10:07:35Z

For anyone experiencing this issue, I was able to make it work by adding the following snippet of code to xhgui/external/header.php right before ignore_user_abort(true); call. Not sure how correct is this and if it could affect anything else, but it did the trick for me.

        $profile = [];
        foreach($data['profile'] as $key => $value) {
            $profile[strtr($key, ['.' => '_'])] = $value;
        }
        $data['profile'] = $profile;

bogdan-dubyk · 2018-01-10T12:57:48Z

@dhampik is your fix require some more manipulations? It showing all available profilings list as well, but when i click on some of them I'm getting error:

#0 /var/www/xhgui/src/Xhgui/Profile.php(361): Slim\Slim::handleErrors(8, 'Undefined index...', '/var/www/xhgui/...', 361, Array)
#1 /var/www/xhgui/src/Xhgui/Controller/Run.php(69): Xhgui_Profile->calculateSelf()
#2 /var/www/xhgui/src/routes.php(30): Xhgui_Controller_Run->view()
#3 [internal function]: {closure}()
#4 /var/www/xhgui/vendor/slim/slim/Slim/Route.php(468): call_user_func_array(Object(Closure), Array)
#5 /var/www/xhgui/vendor/slim/slim/Slim/Slim.php(1355): Slim\Route->dispatch()
#6 /var/www/xhgui/vendor/slim/slim/Slim/Middleware/Flash.php(85): Slim\Slim->call()
#7 /var/www/xhgui/vendor/slim/slim/Slim/Middleware/MethodOverride.php(92): Slim\Middleware\Flash->call()
#8 /var/www/xhgui/vendor/slim/slim/Slim/Middleware/SessionCookie.php(110): Slim\Middleware\MethodOverride->call()
#9 /var/www/xhgui/src/Xhgui/Middleware/Render.php(11): Slim\Middleware\SessionCookie->call()
#10 /var/www/xhgui/vendor/slim/slim/Slim/Slim.php(1300): Xhgui_Middleware_Render->call()
#11 /var/www/xhgui/webroot/index.php(11): Slim\Slim->run()
#12 {main}

dhampik · 2018-01-10T14:03:19Z

@bogdan-dubyk no, only the change that I wrote.

tm1000 · 2018-01-24T02:38:33Z

@dhampik thank you. Worked for me.

apuppy · 2019-02-22T09:17:06Z

For anyone experiencing this issue, I was able to make it work by adding the following snippet of code to xhgui/external/header.php right before ignore_user_abort(true); call. Not sure how correct is this and if it could affect anything else, but it did the trick for me.
        $profile = [];
        foreach($data['profile'] as $key => $value) {
            $profile[strtr($key, ['.' => '_'])] = $value;
        }
        $data['profile'] = $profile;

It worked for me, thank you.

See perftools/xhgui#209 (comment) `invalid document for insert: keys cannot contain ".": "main()==>load::admin/index.php"`

matteo-gz · 2019-12-24T08:08:05Z

For anyone experiencing this issue, I was able to make it work by adding the following snippet of code to xhgui/external/header.php right before ignore_user_abort(true); call. Not sure how correct is this and if it could affect anything else, but it did the trick for me.
        $profile = [];
        foreach($data['profile'] as $key => $value) {
            $profile[strtr($key, ['.' => '_'])] = $value;
        }
        $data['profile'] = $profile;
It worked for me, thank you.

this way is so cool

glensc · 2020-10-21T18:05:30Z

Now Ive encountered this myself as well. with ZF1 the autoloader entries look like this:

    "Zend_Controller_Dispatcher_Standard::loadClass==>run_init::controllers\/ArticleController.php": {
      "ct": 1,
      "wt": 4230,
      "cpu": 6667,
      "mu": 843328,
      "pmu": 436912
    },

glensc · 2020-10-21T18:20:25Z

For everybody there seems to be "load::" in call stack, I also have "run_init::" prefixes.

I wonder it's xhprof quirk or something?

My data is from PHP 5.3 and xhprof 0.9.4

glensc · 2020-10-21T18:28:22Z

indeed, it's present in xhprof 0.9.4 extension.

these code paths add filenames to call tree, and filenames obviously can contain the dot:

later removed for 2.x (new maintainer):

longxinH/xhprof@d341e22

however "load::" is still present:

https://github.com/longxinH/xhprof/blob/v2.2.2-beta/extension/xhprof.c#L1213

glensc · 2020-10-21T21:21:09Z

Re: #209 (comment)

@michaelbutler still there?

have you considered sending PRs of your changes? over the time this project has got pdo support, but it's not complete either:

Equalize data store backends #320

michaelbutler · 2020-10-21T23:43:57Z

@glensc I haven't updated the project in quite a while but if I get a chance I'll try the latest version with PDO and see if I have anything to add

SimonDeconde changed the title ~~xhgui: document to insert contains invalid key: keys cannot contain~~ xhgui: document to insert contains invalid key: keys cannot contain "." Sep 28, 2017

samt2497 mentioned this issue Oct 10, 2017

ISSUE-209: Encode keys before storing on mongo. #210

Closed

markstory closed this as completed Oct 10, 2017

lauripiisang reopened this Oct 11, 2017

sfzhang88 mentioned this issue Jul 9, 2018

网站添加header.php监控的时候Nginx报502的错误 laynefyc/xhgui-branch#24

Closed

dv336699 pushed a commit to dv336699/xhgui-collector that referenced this issue Aug 19, 2019

invalid document for insert: keys cannot contain "."

c0465ed

See perftools/xhgui#209 (comment) `invalid document for insert: keys cannot contain ".": "main()==>load::admin/index.php"`

dv336699 mentioned this issue Aug 19, 2019

invalid document for insert: keys cannot contain "." perftools/xhgui-collector#29

Closed

glensc mentioned this issue Oct 21, 2020

Encode dot in MongoDB saver as underscore #352

Merged

glensc closed this as completed in #352 Oct 22, 2020

This was referenced Dec 15, 2020

Drio "."-hack ngyuki-sandbox/php-xhprof-xhgui#1

Closed

Use docker COPY directice edyan/docker-xhgui#10

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

xhgui: document to insert contains invalid key: keys cannot contain "." #209

xhgui: document to insert contains invalid key: keys cannot contain "." #209

SimonDeconde commented Sep 28, 2017

felixkk commented Sep 28, 2017 •

edited

Loading

markstory commented Sep 28, 2017

martins-olekss commented Sep 30, 2017

lauripiisang commented Oct 2, 2017

lauripiisang commented Oct 2, 2017

martins-olekss commented Oct 2, 2017

lauripiisang commented Oct 2, 2017 •

edited

Loading

lauripiisang commented Oct 2, 2017

emilycraps commented Oct 6, 2017

samt2497 commented Oct 10, 2017 •

edited

Loading

markstory commented Oct 10, 2017

lauripiisang commented Oct 11, 2017 •

edited

Loading

lauripiisang commented Oct 11, 2017

markstory commented Oct 11, 2017

felixkk commented Oct 14, 2017

michaelbutler commented Oct 14, 2017

dhampik commented Oct 25, 2017

bogdan-dubyk commented Jan 10, 2018

dhampik commented Jan 10, 2018

tm1000 commented Jan 24, 2018

apuppy commented Feb 22, 2019

matteo-gz commented Dec 24, 2019

glensc commented Oct 21, 2020

glensc commented Oct 21, 2020

glensc commented Oct 21, 2020 •

edited

Loading

glensc commented Oct 21, 2020

michaelbutler commented Oct 21, 2020

xhgui: document to insert contains invalid key: keys cannot contain "." #209

xhgui: document to insert contains invalid key: keys cannot contain "." #209

Comments

SimonDeconde commented Sep 28, 2017

felixkk commented Sep 28, 2017 • edited Loading

markstory commented Sep 28, 2017

martins-olekss commented Sep 30, 2017

lauripiisang commented Oct 2, 2017

lauripiisang commented Oct 2, 2017

martins-olekss commented Oct 2, 2017

lauripiisang commented Oct 2, 2017 • edited Loading

lauripiisang commented Oct 2, 2017

emilycraps commented Oct 6, 2017

samt2497 commented Oct 10, 2017 • edited Loading

markstory commented Oct 10, 2017

lauripiisang commented Oct 11, 2017 • edited Loading

lauripiisang commented Oct 11, 2017

markstory commented Oct 11, 2017

felixkk commented Oct 14, 2017

michaelbutler commented Oct 14, 2017

dhampik commented Oct 25, 2017

bogdan-dubyk commented Jan 10, 2018

dhampik commented Jan 10, 2018

tm1000 commented Jan 24, 2018

apuppy commented Feb 22, 2019

matteo-gz commented Dec 24, 2019

glensc commented Oct 21, 2020

glensc commented Oct 21, 2020

glensc commented Oct 21, 2020 • edited Loading

glensc commented Oct 21, 2020

michaelbutler commented Oct 21, 2020

felixkk commented Sep 28, 2017 •

edited

Loading

lauripiisang commented Oct 2, 2017 •

edited

Loading

samt2497 commented Oct 10, 2017 •

edited

Loading

lauripiisang commented Oct 11, 2017 •

edited

Loading

glensc commented Oct 21, 2020 •

edited

Loading