Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug]: streaming seems to be adding a lag to the raw data timestamps for my Prometheus charts #14318

Closed
andrewm4894 opened this issue Jan 24, 2023 · 17 comments
Labels
bug needs triage Issues which need to be manually labelled

Comments

@andrewm4894
Copy link
Contributor

andrewm4894 commented Jan 24, 2023

Bug description

related internal slack: https://netdata-cloud.slack.com/archives/CS3PB0VJ7/p1674554617334889

response from my child's nodes actual data:

# http://XXX:19999/api/v1/data?chart=prometheus_node_exporter_demo_cpu_1s.node_cpu_seconds_total-cpu_0-mode_idle&after=1674540987&before=1674542259

{
 "labels": ["time","node_cpu_seconds_total"],
    "data":
 [
 [ 1674542259,0.8247609],
 [ 1674542258,0.841978],
 [ 1674542257,0.7823306],
 [ 1674542256,0.7012704],
 [ 1674542255,0.772847],
 [ 1674542254,0.8417006],
 [ 1674542253,0.8514777],
 [ 1674542252,0.8294887],
 [ 1674542251,0.6607311],
 [ 1674542250,0.6516703],
 [ 1674542249,0.8421287],
 [ 1674542248,0.7650337],
 [ 1674542247,0.7959855],
 [ 1674542246,0.8121153],
 [ 1674542245,0.7914872],
 [ 1674542244,0.8561456],
 [ 1674542243,0.8704395],
 [ 1674542242,0.7997504],
 [ 1674542241,0.6393075],
 [ 1674542240,0.6592281],
 [ 1674542239,0.8264957],
 [ 1674542238,0.8504152],
 [ 1674542237,0.8015284],
 [ 1674542236,0.7860875],
 [ 1674542235,0.6922133],
 [ 1674542234,0.862977],
 [ 1674542233,0.8754637],
 [ 1674542232,0.833629],
 [ 1674542231,0.7401238],
 [ 1674542230,0.7575201],
 [ 1674542229,0.8452324],
 [ 1674542228,0.8203504],
 [ 1674542227,0.7620081],
 [ 1674542226,0.7088138],
 [ 1674542225,0.6739972],
 [ 1674542224,0.8097949],
 [ 1674542223,0.8582814],
 [ 1674542222,0.8307778],
 [ 1674542221,0.6965042],
 [ 1674542220,0.5741675],
 [ 1674542219,0.8485372],
 [ 1674542218,0.8375075],
 [ 1674542217,0.780788],
 [ 1674542216,0.8188419],
 [ 1674542215,0.7829097],
 [ 1674542214,0.8518301],
 [ 1674542213,0.8613155],
 [ 1674542212,0.8300742],
 [ 1674542211,0.639254],
 [ 1674542210,0.7559633],
 [ 1674542209,0.8383622],
 [ 1674542208,0.8109639],
 [ 1674542207,0.7271753],
 [ 1674542206,0.7749223],
 [ 1674542205,0.6401043],
 [ 1674542204,0.8531653],
 [ 1674542203,0.8704371],
 [ 1674542202,0.8553356],
 [ 1674542201,0.75343],
 [ 1674542200,0.6943985],
 [ 1674542199,0.8380617],
 [ 1674542198,0.860964],
 [ 1674542197,0.7529806],
 [ 1674542196,0.7154702],
 [ 1674542195,0.7656719],
 [ 1674542194,0.8320822],
 [ 1674542193,0.8063592],
 [ 1674542192,0.8239934],
 [ 1674542191,0.7224144],
 [ 1674542190,0.66387],
 [ 1674542189,0.8250219],
 [ 1674542188,0.7467034],
 [ 1674542187,0.7965528],
 [ 1674542186,0.818611],
 [ 1674542185,0.7744391],
 [ 1674542184,0.8505798],
 [ 1674542183,0.8559911],
 [ 1674542182,0.8362266],
 [ 1674542181,0.6737432],
 [ 1674542180,0.673452],
 [ 1674542179,0.842099],
 [ 1674542178,0.8000526],
 [ 1674542177,0.8017331],
 [ 1674542176,0.7853137],
 [ 1674542175,0.6827834],
 [ 1674542174,0.8629089],
 [ 1674542173,0.8767735],
 [ 1674542172,0.8423712],
 [ 1674542171,0.7436265],
 [ 1674542170,0.7294317],
 [ 1674542169,0.83122],
 [ 1674542168,0.8239452],
 [ 1674542167,0.7784078],
 [ 1674542166,0.6635432],
 [ 1674542165,0.7044015],
 [ 1674542164,0.8368092],
 [ 1674542163,0.8645096],
 [ 1674542162,0.8224526],
 [ 1674542161,0.7081878],
 [ 1674542160,0.5846497],
 [ 1674542159,0.8214957],
 [ 1674542158,0.8721365],
 [ 1674542157,0.7835096],
 [ 1674542156,0.7997286],
 [ 1674542155,0.7883361],
 [ 1674542154,0.7911354],
 [ 1674542153,0.8500986],
 [ 1674542152,0.8263107],
 [ 1674542151,0.6646219],
 [ 1674542150,0.7561364],
 [ 1674542149,0.8396608],
 [ 1674542148,0.8111249],
 [ 1674542147,0.7371583],
 [ 1674542146,0.7640112],
 [ 1674542145,0.5380662],
 [ 1674542144,0.8410598],
 [ 1674542143,0.843828],
 [ 1674542142,0.8533288],
 [ 1674542141,0.7442649],
 [ 1674542140,0.6974681],
 [ 1674542139,0.8444697],
 [ 1674542138,0.8411315],
 [ 1674542137,0.7461834],
 [ 1674542136,0.7221876],
 [ 1674542135,0.7605296],
 [ 1674542134,0.8572108],
 [ 1674542133,0.851503],
 [ 1674542132,0.8333948],
 [ 1674542131,0.7279283],
 [ 1674542130,0.6555531],
 [ 1674542129,0.8250183],
 [ 1674542128,0.7469035],
 [ 1674542127,0.7948057],
 [ 1674542126,0.8147142],
 [ 1674542125,0.7866024],
 [ 1674542124,0.8364493],
 [ 1674542123,0.865211],
 [ 1674542122,0.807876],
 [ 1674542121,0.6453051],
 [ 1674542120,0.699887],
 [ 1674542119,0.8485461],
 [ 1674542118,0.840429],
 [ 1674542117,0.7979274],
 [ 1674542116,0.8049238],
 [ 1674542115,0.6601197],
 [ 1674542114,0.8629263],
 [ 1674542113,0.8706846],
 [ 1674542112,0.8182539],
 [ 1674542111,0.7471268],
 [ 1674542110,0.7526492],
 [ 1674542109,0.8233232],
 [ 1674542108,0.8143946],
 [ 1674542107,0.6814894],
 [ 1674542106,0.6980408],
 [ 1674542105,0.7530896],
 [ 1674542104,0.8230331],
 [ 1674542103,0.8270848],
 [ 1674542102,0.8100587],
 [ 1674542101,0.711138],
 [ 1674542100,0.5647902],
 [ 1674542099,0.8452572],
 [ 1674542098,0.8204613],
 [ 1674542097,0.7780489],
 [ 1674542096,0.8038429],
 [ 1674542095,0.73921],
 [ 1674542094,0.8432393],
 [ 1674542093,0.860068],
 [ 1674542092,0.8263133],
 [ 1674542091,0.6633627],
 [ 1674542090,0.7520466],
 [ 1674542089,0.8600578],
 [ 1674542088,0.8503818],
 [ 1674542087,0.8077117],
 [ 1674542086,0.7973002],
 [ 1674542085,0.5810433],
 [ 1674542084,0.8420259],
 [ 1674542083,0.8596837],
 [ 1674542082,0.8355829],
 [ 1674542081,0.7524062],
 [ 1674542080,0.6857465],
 [ 1674542079,0.8280509],
 [ 1674542078,0.8488383],
 [ 1674542077,0.7900887],
 [ 1674542076,0.7346386],
 [ 1674542075,0.7684179],
 [ 1674542074,0.8264907],
 [ 1674542073,0.8481333],
 [ 1674542072,0.8174803],
 [ 1674542071,0.6957261],
 [ 1674542070,0.6571254],
 [ 1674542069,0.8285293],
 [ 1674542068,0.7171779],
 [ 1674542067,0.7692843],
 [ 1674542066,0.8065567],
 [ 1674542065,0.7981279],
 [ 1674542064,0.8471016],
 [ 1674542063,0.8360517],
 [ 1674542062,0.8264035],
 [ 1674542061,0.6616864],
 [ 1674542060,0.6972024],
 [ 1674542059,0.8419495],
 [ 1674542058,0.8366069],
 [ 1674542057,0.7698377],
 [ 1674542056,0.8029463],
 [ 1674542055,0.691794],
 [ 1674542054,0.8484224],
 [ 1674542053,0.8502913],
 [ 1674542052,0.8351237],
 [ 1674542051,0.742249],
 [ 1674542050,0.7262064],
 [ 1674542049,0.8209525],
 [ 1674542048,0.8347574],
 [ 1674542047,0.6630715],
 [ 1674542046,0.6912089],
 [ 1674542045,0.766448],
 [ 1674542044,0.8514156],
 [ 1674542043,0.8451209],
 [ 1674542042,0.8203499],
 [ 1674542041,0.7333508],
 [ 1674542040,0.5913339],
 [ 1674542039,0.8499487],
 [ 1674542038,0.8399893],
 [ 1674542037,0.7935099],
 [ 1674542036,0.8104859],
 [ 1674542035,0.7694812],
 [ 1674542034,0.8216333],
 [ 1674542033,0.8352306],
 [ 1674542032,0.8318645],
 [ 1674542031,0.6900679],
 [ 1674542030,0.7675276],
 [ 1674542029,0.8535671],
 [ 1674542028,0.8120638],
 [ 1674542027,0.781337],
 [ 1674542026,0.7888489],
 [ 1674542025,0.5710314],
 [ 1674542024,0.8432394],
 [ 1674542023,0.860075],
 [ 1674542022,0.8454749],
 [ 1674542021,0.7523755],
 [ 1674542020,0.6859108],
 [ 1674542019,0.824477],
 [ 1674542018,0.830647],
 [ 1674542017,0.7785042],
 [ 1674542016,0.7267224],
 [ 1674542015,0.7756658],
 [ 1674542014,0.8423717],
 [ 1674542013,0.8209684],
 [ 1674542012,0.8462078],
 [ 1674542011,0.7050985],
 [ 1674542010,0.6155699],
 [ 1674542009,0.8424903],
 [ 1674542008,0.7495342],
 [ 1674542007,0.7520887],
 [ 1674542006,0.8145459],
 [ 1674542005,0.7862318],
 [ 1674542004,0.8202065],
 [ 1674542003,0.8331457],
 [ 1674542002,0.8179107],
 [ 1674542001,0.6517048],
 [ 1674542000,0.6907361],
 [ 1674541999,0.8527991],
 [ 1674541998,0.8443131],
 [ 1674541997,0.7776545],
 [ 1674541996,0.7815196],
 [ 1674541995,0.6720218],
 [ 1674541994,0.8344798],
 [ 1674541993,0.8584916],
 [ 1674541992,0.8407897],
 [ 1674541991,0.7368648],
 [ 1674541990,0.732721],
 [ 1674541989,0.806586],
 [ 1674541988,0.816205],
 [ 1674541987,0.6927004],
 [ 1674541986,0.7113378],
 [ 1674541985,0.7765174],
 [ 1674541984,0.8147893],
 [ 1674541983,0.8337575],
 [ 1674541982,0.8013622],
 [ 1674541981,0.7086495],
 [ 1674541980,0.5532171],
 [ 1674541979,0.837688],
 [ 1674541978,0.8559291],
 [ 1674541977,0.7914248],
 [ 1674541976,0.7852866],
 [ 1674541975,0.7843987],
 [ 1674541974,0.8500336],
 [ 1674541973,0.8513394],
 [ 1674541972,0.8295537],
 [ 1674541971,0.6850137],
 [ 1674541970,0.7310035],
 [ 1674541969,0.8443601],
 [ 1674541968,0.8473369],
 [ 1674541967,0.7773559],
 [ 1674541966,0.6982229],
 [ 1674541965,0.6641106],
 [ 1674541964,0.8445911],
 [ 1674541963,0.8668808],
 [ 1674541962,0.8441091],
 [ 1674541961,0.7471042],
 [ 1674541960,0.7042875],
 [ 1674541959,0.8298496],
 [ 1674541958,0.8147821],
 [ 1674541957,0.7760495],
 [ 1674541956,0.7142649],
 [ 1674541955,0.7495495],
 [ 1674541954,0.858435],
 [ 1674541953,0.8486321],
 [ 1674541952,0.8266784],
 [ 1674541951,0.7214828],
 [ 1674541950,0.6200848],
 [ 1674541949,0.8186591],
 [ 1674541948,0.7648935],
 [ 1674541947,0.7949735],
 [ 1674541946,0.8134378],
 [ 1674541945,0.7863147],
 [ 1674541944,0.8250252],
 [ 1674541943,0.8483645],
 [ 1674541942,0.8079296],
 [ 1674541941,0.6447611],
 [ 1674541940,0.6955925],
 [ 1674541939,0.8331171],
 [ 1674541938,0.8355986],
 [ 1674541937,0.7599437],
 [ 1674541936,0.7359972],
 [ 1674541935,0.6502695],
 [ 1674541934,0.8615107],
 [ 1674541933,0.8685543],
 [ 1674541932,0.8435855],
 [ 1674541931,0.7371713],
 [ 1674541930,0.6926835],
 [ 1674541929,0.8097695],
 [ 1674541928,0.8004948],
 [ 1674541927,0.7504336],
 [ 1674541926,0.7080963],
 [ 1674541925,0.7614446],
 [ 1674541924,0.8199414],
 [ 1674541923,0.8144434],
 [ 1674541922,0.813068],
 [ 1674541921,0.6617556],
 [ 1674541920,0.6067984],
 [ 1674541919,0.8435666],
 [ 1674541918,0.81233],
 [ 1674541917,0.7792506],
 [ 1674541916,0.8123892],
 [ 1674541915,0.7876276],
 [ 1674541914,0.8284408],
 [ 1674541913,0.8765299],
 [ 1674541912,0.8294829],
 [ 1674541911,0.683007],
 [ 1674541910,0.7657783],
 [ 1674541909,0.8503987],
 [ 1674541908,0.8350061],
 [ 1674541907,0.7583173],
 [ 1674541906,0.7812574],
 [ 1674541905,0.6617818],
 [ 1674541904,0.8281647],
 [ 1674541903,0.8703323],
 [ 1674541902,0.800755],
 [ 1674541901,0.6952548],
 [ 1674541900,0.6859202],
 [ 1674541899,0.7812838],
 [ 1674541898,0.8306634],
 [ 1674541897,0.7683346],
 [ 1674541896,0.7067643],
 [ 1674541895,0.7658423],
 [ 1674541894,0.8500056],
 [ 1674541893,0.850079],
 [ 1674541892,0.8340207],
 [ 1674541891,0.7260467],
 [ 1674541890,0.6396966],
 [ 1674541889,0.8272037],
 [ 1674541888,0.7701867],
 [ 1674541887,0.7748268],
 [ 1674541886,0.7950854],
 [ 1674541885,0.7780741],
 [ 1674541884,0.8216489],
 [ 1674541883,0.8202881],
 [ 1674541882,0.7507496],
 [ 1674541881,0.6565326],
 [ 1674541880,0.7075789],
 [ 1674541879,0.8066634],
 [ 1674541878,0.8331724],
 [ 1674541877,0.7917422],
 [ 1674541876,0.779527],
 [ 1674541875,0.6523249],
 [ 1674541874,0.8248062],
 [ 1674541873,0.8448965],
 [ 1674541872,0.8088299],
 [ 1674541871,0.7470617],
 [ 1674541870,0.7342639],
 [ 1674541869,0.7600129],
 [ 1674541868,0.800586],
 [ 1674541867,0.7019915],
 [ 1674541866,0.71613],
 [ 1674541865,0.7546678],
 [ 1674541864,0.8153898],
 [ 1674541863,0.7526928],
 [ 1674541862,0.8072193],
 [ 1674541861,0.7115179],
 [ 1674541860,0.5629317],
 [ 1674541859,0.8387533],
 [ 1674541858,0.8249664],
 [ 1674541857,0.7897645],
 [ 1674541856,0.7839439],
 [ 1674541855,0.767818],
 [ 1674541854,0.8198407],
 [ 1674541853,0.8663177],
 [ 1674541852,0.8114558],
 [ 1674541851,0.6811924],
 [ 1674541850,0.7592113],
 [ 1674541849,0.8583606],
 [ 1674541848,0.8356226],
 [ 1674541847,0.7694881],
 [ 1674541846,0.7895133],
 [ 1674541845,0.6638031],
 [ 1674541844,0.8431629],
 [ 1674541843,0.8536205],
 [ 1674541842,0.8027474],
 [ 1674541841,0.7047756],
 [ 1674541840,0.7043132],
 [ 1674541839,0.8344417],
 [ 1674541838,0.8355855],
 [ 1674541837,0.7587372],
 [ 1674541836,0.7017193],
 [ 1674541835,0.730939],
 [ 1674541834,0.8432306],
 [ 1674541833,0.8533974],
 [ 1674541832,0.8056238],
 [ 1674541831,0.7077866],
 [ 1674541830,0.6315251],
 [ 1674541829,0.8158832],
 [ 1674541828,0.7114631],
 [ 1674541827,0.6931661],
 [ 1674541826,0.2429194],
 [ 1674541825,0],
 [ 1674541824,0],
 [ 1674541823,0],
 [ 1674541822,0],
 [ 1674541821,0],
 [ 1674541820,0],
 [ 1674541819,0],
 [ 1674541818,0],
 [ 1674541817,0],
 [ 1674541816,0],
 [ 1674541815,0],
 [ 1674541814,0],
 [ 1674541813,0],
 [ 1674541812,0],
 [ 1674541811,0],
 [ 1674541810,0],
 [ 1674541809,0],
 [ 1674541808,0],
 [ 1674541807,0],
 [ 1674541806,0],
 [ 1674541805,0],
 [ 1674541804,0],
 [ 1674541803,0],
 [ 1674541802,0],
 [ 1674541801,0],
 [ 1674541800,0],
 [ 1674541799,0],
 [ 1674541798,0],
 [ 1674541797,0],
 [ 1674541796,0],
 [ 1674541795,0],
 [ 1674541794,0],
 [ 1674541793,0],
 [ 1674541792,0],
 [ 1674541791,0],
 [ 1674541790,0],
 [ 1674541789,0],
 [ 1674541788,0],
 [ 1674541787,0],
 [ 1674541786,0],
 [ 1674541785,0],
 [ 1674541784,0],
 [ 1674541783,0],
 [ 1674541782,0],
 [ 1674541781,0],
 [ 1674541780,0],
 [ 1674541779,0],
 [ 1674541778,0],
 [ 1674541777,0],
 [ 1674541776,0],
 [ 1674541775,0],
 [ 1674541774,0],
 [ 1674541773,0],
 [ 1674541772,0],
 [ 1674541771,0],
 [ 1674541770,0],
 [ 1674541769,0],
 [ 1674541768,0],
 [ 1674541767,0],
 [ 1674541766,0],
 [ 1674541765,0],
 [ 1674541764,0],
 [ 1674541763,0],
 [ 1674541762,0],
 [ 1674541761,0],
 [ 1674541760,0],
 [ 1674541759,0],
 [ 1674541758,0],
 [ 1674541757,0],
 [ 1674541756,0],
 [ 1674541755,0],
 [ 1674541754,0],
 [ 1674541753,0.1365255],
 [ 1674541752,0.8325434],
 [ 1674541751,0.7363523],
 [ 1674541750,0.7263435],
 [ 1674541749,0.8216465],
 [ 1674541748,0.8202591],
 [ 1674541747,0.7442961],
 [ 1674541746,0.6324932],
 [ 1674541745,0.7631764],
 [ 1674541744,0.8313175],
 [ 1674541743,0.826817],
 [ 1674541742,0.8057776],
 [ 1674541741,0.7069347],
 [ 1674541740,0.6236345],
 [ 1674541739,0.8452087],
 [ 1674541738,0.8188552],
 [ 1674541737,0.769537],
 [ 1674541736,0.8055257],
 [ 1674541735,0.7447817],
 [ 1674541734,0.8277384],
 [ 1674541733,0.8571256],
 [ 1674541732,0.8203291],
 [ 1674541731,0.6733368],
 [ 1674541730,0.7630207],
 [ 1674541729,0.8344497],
 [ 1674541728,0.8388699],
 [ 1674541727,0.7755433],
 [ 1674541726,0.6855326],
 [ 1674541725,0.6457009],
 [ 1674541724,0.8359691],
 [ 1674541723,0.8584585],
 [ 1674541722,0.8439885],
 [ 1674541721,0.7471153],
 [ 1674541720,0.7056991],
 [ 1674541719,0.8384442],
 [ 1674541718,0.8252058],
 [ 1674541717,0.7810059],
 [ 1674541716,0.6956893],
 [ 1674541715,0.7796898],
 [ 1674541714,0.8400417],
 [ 1674541713,0.8433849],
 [ 1674541712,0.8420287],
 [ 1674541711,0.7310796],
 [ 1674541710,0.666868],
 [ 1674541709,0.8351069],
 [ 1674541708,0.755699],
 [ 1674541707,0.7572872],
 [ 1674541706,0.8287172],
 [ 1674541705,0.7851954],
 [ 1674541704,0.8193274],
 [ 1674541703,0.8535942],
 [ 1674541702,0.8043671],
 [ 1674541701,0.6717118],
 [ 1674541700,0.7054588],
 [ 1674541699,0.8406673],
 [ 1674541698,0.8334863],
 [ 1674541697,0.7502394],
 [ 1674541696,0.7716214],
 [ 1674541695,0.6559485],
 [ 1674541694,0.851575],
 [ 1674541693,0.8468004],
 [ 1674541692,0.8186108],
 [ 1674541691,0.7486865],
 [ 1674541690,0.7127012],
 [ 1674541689,0.808804],
 [ 1674541688,0.7081334],
 [ 1674541687,0.7416504],
 [ 1674541686,0.7073402],
 [ 1674541685,0.764101],
 [ 1674541684,0.8331939],
 [ 1674541683,0.8482321],
 [ 1674541682,0.8307731],
 [ 1674541681,0.7012756],
 [ 1674541680,0.5946103],
 [ 1674541679,0.8218058],
 [ 1674541678,0.8289351],
 [ 1674541677,0.7792979],
 [ 1674541676,0.8109533],
 [ 1674541675,0.7774966],
 [ 1674541674,0.8299812],
 [ 1674541673,0.8731395],
 [ 1674541672,0.7915777],
 [ 1674541671,0.6227767],
 [ 1674541670,0.7589341],
 [ 1674541669,0.8568141],
 [ 1674541668,0.8287716],
 [ 1674541667,0.7734559],
 [ 1674541666,0.7700203],
 [ 1674541665,0.6877295],
 [ 1674541664,0.8448598],
 [ 1674541663,0.8619715],
 [ 1674541662,0.8136139],
 [ 1674541661,0.7590856],
 [ 1674541660,0.678915],
 [ 1674541659,0.8348],
 [ 1674541658,0.8522032],
 [ 1674541657,0.7900735],
 [ 1674541656,0.6917105],
 [ 1674541655,0.7743727],
 [ 1674541654,0.8484628],
 [ 1674541653,0.8399422],
 [ 1674541652,0.8207495],
 [ 1674541651,0.7057146],
 [ 1674541650,0.6638906],
 [ 1674541649,0.8264966],
 [ 1674541648,0.7534873],
 [ 1674541647,0.7931992],
 [ 1674541646,0.8466355],
 [ 1674541645,0.7781824],
 [ 1674541644,0.8364763],
 [ 1674541643,0.866812],
 [ 1674541642,0.7985504],
 [ 1674541641,0.5543043],
 [ 1674541640,0.7055472],
 [ 1674541639,0.843212],
 [ 1674541638,0.8488116],
 [ 1674541637,0.7946141],
 [ 1674541636,0.7865076],
 [ 1674541635,0.6486806],
 [ 1674541634,0.8561824],
 [ 1674541633,0.8772699],
 [ 1674541632,0.7987368],
 [ 1674541631,0.7384362],
 [ 1674541630,0.7426705],
 [ 1674541629,0.8184356],
 [ 1674541628,0.8484024],
 [ 1674541627,0.7702249],
 [ 1674541626,0.7233471],
 [ 1674541625,0.7530271],
 [ 1674541624,0.8055047],
 [ 1674541623,0.7493391],
 [ 1674541622,0.8334686],
 [ 1674541621,0.7198082],
 [ 1674541620,0.6135515],
 [ 1674541619,0.8399755],
 [ 1674541618,0.8318225],
 [ 1674541617,0.8011646],
 [ 1674541616,0.8455377],
 [ 1674541615,0.7796661],
 [ 1674541614,0.8380949],
 [ 1674541613,0.8767448],
 [ 1674541612,0.8279513],
 [ 1674541611,0.676448],
 [ 1674541610,0.7738351],
 [ 1674541609,0.8402922],
 [ 1674541608,0.8402499],
 [ 1674541607,0.7897588],
 [ 1674541606,0.8072668],
 [ 1674541605,0.669453],
 [ 1674541604,0.8362578],
 [ 1674541603,0.8639126],
 [ 1674541602,0.8170785],
 [ 1674541601,0.7265804],
 [ 1674541600,0.6876785],
 [ 1674541599,0.8416241],
 [ 1674541598,0.8422937],
 [ 1674541597,0.7880969],
 [ 1674541596,0.728354],
 [ 1674541595,0.7915757],
 [ 1674541594,0.8528221],
 [ 1674541593,0.8569095],
 [ 1674541592,0.8339228],
 [ 1674541591,0.7295906],
 [ 1674541590,0.6617502],
 [ 1674541589,0.8353037],
 [ 1674541588,0.7262417],
 [ 1674541587,0.7683449],
 [ 1674541586,0.8103534],
 [ 1674541585,0.7744048],
 [ 1674541584,0.8535417],
 [ 1674541583,0.8667075],
 [ 1674541582,0.7974546],
 [ 1674541581,0.595201],
 [ 1674541580,0.7023305],
 [ 1674541579,0.8231821],
 [ 1674541578,0.8323235],
 [ 1674541577,0.7959611],
 [ 1674541576,0.784743],
 [ 1674541575,0.648661],
 [ 1674541574,0.8536229],
 [ 1674541573,0.8681],
 [ 1674541572,0.8318645],
 [ 1674541571,0.740389],
 [ 1674541570,0.7611889],
 [ 1674541569,0.8264457],
 [ 1674541568,0.8488993],
 [ 1674541567,0.7765704],
 [ 1674541566,0.7035659],
 [ 1674541565,0.7316352],
 [ 1674541564,0.751696],
 [ 1674541563,0.808061],
 [ 1674541562,0.7782957],
 [ 1674541561,0.6936902],
 [ 1674541560,0.6136047],
 [ 1674541559,0.8655458],
 [ 1674541558,0.850678],
 [ 1674541557,0.7615871],
 [ 1674541556,0.8150608],
 [ 1674541555,0.7912663],
 [ 1674541554,0.8482948],
 [ 1674541553,0.8514521],
 [ 1674541552,0.8262778],
 [ 1674541551,0.6662172],
 [ 1674541550,0.7642502],
 [ 1674541549,0.8452388],
 [ 1674541548,0.8565547],
 [ 1674541547,0.7898632],
 [ 1674541546,0.8130617],
 [ 1674541545,0.6556383],
 [ 1674541544,0.8444898],
 [ 1674541543,0.8585503],
 [ 1674541542,0.8440236],
 [ 1674541541,0.7454205],
 [ 1674541540,0.6988405],
 [ 1674541539,0.8483997],
 [ 1674541538,0.8322297],
 [ 1674541537,0.7764776],
 [ 1674541536,0.711661],
 [ 1674541535,0.7882434],
 [ 1674541534,0.834996],
 [ 1674541533,0.8564133],
 [ 1674541532,0.7977562],
 [ 1674541531,0.6304935],
 [ 1674541530,0.6218843],
 [ 1674541529,0.8227962],
 [ 1674541528,0.7390186],
 [ 1674541527,0.7762513],
 [ 1674541526,0.8137484],
 [ 1674541525,0.7950619],
 [ 1674541524,0.8293667],
 [ 1674541523,0.8622728],
 [ 1674541522,0.8021861],
 [ 1674541521,0.6547914],
 [ 1674541520,0.7011669],
 [ 1674541519,0.8267449],
 [ 1674541518,0.844928],
 [ 1674541517,0.7698793],
 [ 1674541516,0.799541],
 [ 1674541515,0.6705376],
 [ 1674541514,0.8348222],
 [ 1674541513,0.8586437],
 [ 1674541512,0.8320149],
 [ 1674541511,0.7267472],
 [ 1674541510,0.6723916],
 [ 1674541509,0.4018967],
 [ 1674541508,0],
 [ 1674541507,0],
 [ 1674541506,0],
 [ 1674541505,0],
 [ 1674541504,0.0647265],
 [ 1674541503,0.4735768],
 [ 1674541502,0.8225821],
 [ 1674541501,0.7153557],
 [ 1674541500,0.578859],
 [ 1674541499,0.8418559],
 [ 1674541498,0.8320332],
 [ 1674541497,0.7427257],
 [ 1674541496,0.8021339],
 [ 1674541495,0.7796051],
 [ 1674541494,0.8351739],
 [ 1674541493,0.8628793],
 [ 1674541492,0.8361303],
 [ 1674541491,0.6862406],
 [ 1674541490,0.7815812],
 [ 1674541489,0.8446028],
 [ 1674541488,0.8500143],
 [ 1674541487,0.8017757],
 [ 1674541486,0.7848162],
 [ 1674541485,0.6401746],
 [ 1674541484,0.8429838],
 [ 1674541483,0.8556279],
 [ 1674541482,0.830083],
 [ 1674541481,0.7554837],
 [ 1674541480,0.7108473],
 [ 1674541479,0.8268134],
 [ 1674541478,0.8501324],
 [ 1674541477,0.7603386],
 [ 1674541476,0.6314395],
 [ 1674541475,0.7676093],
 [ 1674541474,0.8584861],
 [ 1674541473,0.8536141],
 [ 1674541472,0.8526821],
 [ 1674541471,0.7369699],
 [ 1674541470,0.6567775],
 [ 1674541469,0.8306067],
 [ 1674541468,0.7381431],
 [ 1674541467,0.7863531],
 [ 1674541466,0.8151443],
 [ 1674541465,0.7964781],
 [ 1674541464,0.8352265],
 [ 1674541463,0.8612215],
 [ 1674541462,0.8245903],
 [ 1674541461,0.6519033],
 [ 1674541460,0.6917101],
 [ 1674541459,0.8487002],
 [ 1674541458,0.8393158],
 [ 1674541457,0.7905084],
 [ 1674541456,0.8114678],
 [ 1674541455,0.6488766],
 [ 1674541454,0.8528916],
 [ 1674541453,0.8701426],
 [ 1674541452,0.8540022],
 [ 1674541451,0.7586136],
 [ 1674541450,0.7224437],
 [ 1674541449,0.8433001],
 [ 1674541448,0.8456239],
 [ 1674541447,0.7453755],
 [ 1674541446,0.629577],
 [ 1674541445,0.7478545],
 [ 1674541444,0.8385847],
 [ 1674541443,0.8218851],
 [ 1674541442,0.7714578],
 [ 1674541441,0.6967398],
 [ 1674541440,0.5795329],
 [ 1674541439,0.8334863],
 [ 1674541438,0.8469587],
 [ 1674541437,0.7786262],
 [ 1674541436,0.76624],
 [ 1674541435,0.7531695],
 [ 1674541434,0.8352251],
 [ 1674541433,0.861404],
 [ 1674541432,0.8288411],
 [ 1674541431,0.6817803],
 [ 1674541430,0.7646084],
 [ 1674541429,0.8493361],
 [ 1674541428,0.873179],
 [ 1674541427,0.7949389],
 [ 1674541426,0.7558281],
 [ 1674541425,0.5915989],
 [ 1674541424,0.8196198],
 [ 1674541423,0.8701864],
 [ 1674541422,0.8427958],
 [ 1674541421,0.701424],
 [ 1674541420,0.730877],
 [ 1674541419,0.8433994],
 [ 1674541418,0.8533908],
 [ 1674541417,0.7944398],
 [ 1674541416,0.679144],
 [ 1674541415,0.7826592],
 [ 1674541414,0.8485188],
 [ 1674541413,0.8500345],
 [ 1674541412,0.837069],
 [ 1674541411,0.744362],
 [ 1674541410,0.6485219],
 [ 1674541409,0.833867],
 [ 1674541408,0.75514],
 [ 1674541407,0.7912199],
 [ 1674541406,0.8315691],
 [ 1674541405,0.7934542],
 [ 1674541404,0.828047],
 [ 1674541403,0.8698828],
 [ 1674541402,0.833086],
 [ 1674541401,0.6449705],
 [ 1674541400,0.6971572],
 [ 1674541399,0.841971],
 [ 1674541398,0.8383402],
 [ 1674541397,0.7764468],
 [ 1674541396,0.7945372],
 [ 1674541395,0.6919703],
 [ 1674541394,0.8531218],
 [ 1674541393,0.8723233],
 [ 1674541392,0.8150324],
 [ 1674541391,0.724212],
 [ 1674541390,0.7188765],
 [ 1674541389,0.8238247],
 [ 1674541388,0.8130432],
 [ 1674541387,0.6637907],
 [ 1674541386,0.7048732],
 [ 1674541385,0.7461068],
 [ 1674541384,0.8285563],
 [ 1674541383,0.8165645],
 [ 1674541382,0.7838878],
 [ 1674541381,0.6794304],
 [ 1674541380,0.6182075],
 [ 1674541379,0.8597471],
 [ 1674541378,0.84559],
 [ 1674541377,0.7831135],
 [ 1674541376,0.7936231],
 [ 1674541375,0.7728918],
 [ 1674541374,0.8433553],
 [ 1674541373,0.8581928],
 [ 1674541372,0.8213376],
 [ 1674541371,0.6826863],
 [ 1674541370,0.7610475],
 [ 1674541369,0.8667453],
 [ 1674541368,0.8400296],
 [ 1674541367,0.7742925],
 [ 1674541366,0.6896683],
 [ 1674541365,0.6610528],
 [ 1674541364,0.8334206],
 [ 1674541363,0.8513293],
 [ 1674541362,0.8309004],
 [ 1674541361,0.7286238],
 [ 1674541360,0.6989506],
 [ 1674541359,0.8502645],
 [ 1674541358,0.8415484],
 [ 1674541357,0.770766],
 [ 1674541356,0.6896895],
 [ 1674541355,0.7579654],
 [ 1674541354,0.8497704],
 [ 1674541353,0.8431529],
 [ 1674541352,0.8506473],
 [ 1674541351,0.7276432],
 [ 1674541350,0.6455886],
 [ 1674541349,0.8198672],
 [ 1674541348,0.7650714],
 [ 1674541347,0.803176],
 [ 1674541346,0.8154262],
 [ 1674541345,0.8008559],
 [ 1674541344,0.8470575],
 [ 1674541343,0.843095],
 [ 1674541342,0.8278883],
 [ 1674541341,0.6585121],
 [ 1674541340,0.6819097],
 [ 1674541339,0.8517054],
 [ 1674541338,0.8488901],
 [ 1674541337,0.7947228],
 [ 1674541336,0.7732561],
 [ 1674541335,0.5817887],
 [ 1674541334,0.8512019],
 [ 1674541333,0.8520324],
 [ 1674541332,0.8533746],
 [ 1674541331,0.7506441],
 [ 1674541330,0.729441],
 [ 1674541329,0.8312415],
 [ 1674541328,0.8188408],
 [ 1674541327,0.7574065],
 [ 1674541326,0.7024443],
 [ 1674541325,0.7715285],
 [ 1674541324,0.8371595],
 [ 1674541323,0.822769],
 [ 1674541322,0.8171563],
 [ 1674541321,0.7262435],
 [ 1674541320,0.5966372],
 [ 1674541319,0.8399719],
 [ 1674541318,0.8322154],
 [ 1674541317,0.7751262],
 [ 1674541316,0.7030891],
 [ 1674541315,0.7729556],
 [ 1674541314,0.7966345],
 [ 1674541313,0.8432046],
 [ 1674541312,0.8294203],
 [ 1674541311,0.6782106],
 [ 1674541310,0.7375707],
 [ 1674541309,0.8347696],
 [ 1674541308,0.847141],
 [ 1674541307,0.7816129],
 [ 1674541306,0.7644702],
 [ 1674541305,0.6666387],
 [ 1674541304,0.8501028],
 [ 1674541303,0.8554271],
 [ 1674541302,0.8124816],
 [ 1674541301,0.7144618],
 [ 1674541300,0.702811],
 [ 1674541299,0.8335664],
 [ 1674541298,0.8398239],
 [ 1674541297,0.7657145],
 [ 1674541296,0.7096942],
 [ 1674541295,0.7729402],
 [ 1674541294,0.8400989],
 [ 1674541293,0.8433617],
 [ 1674541292,0.8433093],
 [ 1674541291,0.7278586],
 [ 1674541290,0.6601528],
 [ 1674541289,0.8475001],
 [ 1674541288,0.7347304],
 [ 1674541287,0.7680713],
 [ 1674541286,0.8036092],
 [ 1674541285,0.7812026],
 [ 1674541284,0.8384655],
 [ 1674541283,0.838315],
 [ 1674541282,0.8441117],
 [ 1674541281,0.6353655],
 [ 1674541280,0.60195],
 [ 1674541279,0.823437],
 [ 1674541278,0.837263],
 [ 1674541277,0.781264],
 [ 1674541276,0.8062864],
 [ 1674541275,0.6651134],
 [ 1674541274,0.8401166],
 [ 1674541273,0.8438232],
 [ 1674541272,0.8031503],
 [ 1674541271,0.7490079],
 [ 1674541270,0.7159877],
 [ 1674541269,0.8047906],
 [ 1674541268,0.8219682],
 [ 1674541267,0.7657221],
 [ 1674541266,0.699489],
 [ 1674541265,0.7529419],
 [ 1674541264,0.7989961],
 [ 1674541263,0.7076692],
 [ 1674541262,0.7855574],
 [ 1674541261,0.6896649],
 [ 1674541260,0.5829546],
 [ 1674541259,0.848829],
 [ 1674541258,0.8381846],
 [ 1674541257,0.7832677],
 [ 1674541256,0.795126],
 [ 1674541255,0.7798246],
 [ 1674541254,0.8361627],
 [ 1674541253,0.8535775],
 [ 1674541252,0.8027078],
 [ 1674541251,0.6697221],
 [ 1674541250,0.7376058],
 [ 1674541249,0.8333928],
 [ 1674541248,0.8371475],
 [ 1674541247,0.7763613],
 [ 1674541246,0.7757944],
 [ 1674541245,0.6305077],
 [ 1674541244,0.8352102],
 [ 1674541243,0.8216283],
 [ 1674541242,0.81874],
 [ 1674541241,0.7457993],
 [ 1674541240,0.6923124],
 [ 1674541239,0.8066109],
 [ 1674541238,0.8301638],
 [ 1674541237,0.7588249],
 [ 1674541236,0.6664127],
 [ 1674541235,0.7646102],
 [ 1674541234,0.8395896],
 [ 1674541233,0.8303997],
 [ 1674541232,0.7805697],
 [ 1674541231,0.7003154],
 [ 1674541230,0.5824646],
 [ 1674541229,0.7941303],
 [ 1674541228,0.7175647],
 [ 1674541227,0.7617994],
 [ 1674541226,0.820122],
 [ 1674541225,0.7710659],
 [ 1674541224,0.8291873],
 [ 1674541223,0.8325922],
 [ 1674541222,0.7785057],
 [ 1674541221,0.5482492],
 [ 1674541220,0.668197],
 [ 1674541219,0.8020987],
 [ 1674541218,0.8229516],
 [ 1674541217,0.7866337],
 [ 1674541216,0.7964822],
 [ 1674541215,0.6599841],
 [ 1674541214,0.8571302],
 [ 1674541213,0.8499291],
 [ 1674541212,0.8266924],
 [ 1674541211,0.7490117],
 [ 1674541210,0.7178067],
 [ 1674541209,0.8161716],
 [ 1674541208,0.825909],
 [ 1674541207,0.749664],
 [ 1674541206,0.6872176],
 [ 1674541205,0.6751356],
 [ 1674541204,0.6727107],
 [ 1674541203,0.7849618],
 [ 1674541202,0.755472],
 [ 1674541201,0.695439],
 [ 1674541200,0.5604781],
 [ 1674541199,0.8267053],
 [ 1674541198,0.7972293],
 [ 1674541197,0.7463586],
 [ 1674541196,0.8213575],
 [ 1674541195,0.7785253],
 [ 1674541194,0.831209],
 [ 1674541193,0.8349338],
 [ 1674541192,0.8361765],
 [ 1674541191,0.6730004],
 [ 1674541190,0.7543042],
 [ 1674541189,0.8332556],
 [ 1674541188,0.840441],
 [ 1674541187,0.7934092],
 [ 1674541186,0.7791166],
 [ 1674541185,0.648496],
 [ 1674541184,0.8372072],
 [ 1674541183,0.8460984],
 [ 1674541182,0.8427299],
 [ 1674541181,0.7022514],
 [ 1674541180,0.6806016],
 [ 1674541179,0.7882892],
 [ 1674541178,0.8215851],
 [ 1674541177,0.7643923],
 [ 1674541176,0.6990288],
 [ 1674541175,0.7594173],
 [ 1674541174,0.8686322],
 [ 1674541173,0.8567961],
 [ 1674541172,0.8061565],
 [ 1674541171,0.6222301],
 [ 1674541170,0.6198469],
 [ 1674541169,0.8071843],
 [ 1674541168,0.7551472],
 [ 1674541167,0.786034],
 [ 1674541166,0.8007483],
 [ 1674541165,0.7707416],
 [ 1674541164,0.8154088],
 [ 1674541163,0.811272],
 [ 1674541162,0.8364634],
 [ 1674541161,0.6594124],
 [ 1674541160,0.6792941],
 [ 1674541159,0.8400154],
 [ 1674541158,0.8274104],
 [ 1674541157,0.7708267],
 [ 1674541156,0.7832139],
 [ 1674541155,0.6388792],
 [ 1674541154,0.8466449],
 [ 1674541153,0.8695777],
 [ 1674541152,0.8312354],
 [ 1674541151,0.7026402],
 [ 1674541150,0.7183184],
 [ 1674541149,0.7799998],
 [ 1674541148,0.8018091],
 [ 1674541147,0.6584753],
 [ 1674541146,0.7149477],
 [ 1674541145,0.7561694],
 [ 1674541144,0.8384949],
 [ 1674541143,0.8217784],
 [ 1674541142,0.7723441],
 [ 1674541141,0.6967413],
 [ 1674541140,0.5721426],
 [ 1674541139,0.8487182],
 [ 1674541138,0.8248229],
 [ 1674541137,0.7419558],
 [ 1674541136,0.7507147],
 [ 1674541135,0.7174823],
 [ 1674541134,0.8063173],
 [ 1674541133,0.8483827],
 [ 1674541132,0.8110375],
 [ 1674541131,0.6728966],
 [ 1674541130,0.7544887],
 [ 1674541129,0.8331571],
 [ 1674541128,0.8371644],
 [ 1674541127,0.7781637],
 [ 1674541126,0.7909091],
 [ 1674541125,0.6528764],
 [ 1674541124,0.7408791],
 [ 1674541123,0.8516837],
 [ 1674541122,0.8358002],
 [ 1674541121,0.7068862],
 [ 1674541120,0.7061795],
 [ 1674541119,0.7983021],
 [ 1674541118,0.8298956],
 [ 1674541117,0.7672005],
 [ 1674541116,0.7096057],
 [ 1674541115,0.7776482],
 [ 1674541114,0.8652804],
 [ 1674541113,0.8400709],
 [ 1674541112,0.8253505],
 [ 1674541111,0.7179184],
 [ 1674541110,0.5982518],
 [ 1674541109,0.8290551],
 [ 1674541108,0.7280644],
 [ 1674541107,0.7764406],
 [ 1674541106,0.8018966],
 [ 1674541105,0.7696183],
 [ 1674541104,0.82649],
 [ 1674541103,0.8567103],
 [ 1674541102,0.804913],
 [ 1674541101,0.6130578],
 [ 1674541100,0.6186727],
 [ 1674541099,0.8653764],
 [ 1674541098,0.8382263],
 [ 1674541097,0.7785831],
 [ 1674541096,0.8055866],
 [ 1674541095,0.697536],
 [ 1674541094,0.8364198],
 [ 1674541093,0.8768465],
 [ 1674541092,0.840584],
 [ 1674541091,0.7365715],
 [ 1674541090,0.7346062],
 [ 1674541089,0.8116794],
 [ 1674541088,0.7970918],
 [ 1674541087,0.6800169],
 [ 1674541086,0.6886213],
 [ 1674541085,0.7458986],
 [ 1674541084,0.8149502],
 [ 1674541083,0.8020435],
 [ 1674541082,0.8015778],
 [ 1674541081,0.6735862],
 [ 1674541080,0.5940096],
 [ 1674541079,0.8453714],
 [ 1674541078,0.8390316],
 [ 1674541077,0.7828326],
 [ 1674541076,0.7296891],
 [ 1674541075,0.351481],
 [ 1674541074,0],
 [ 1674541073,0],
 [ 1674541072,0],
 [ 1674541071,0],
 [ 1674541070,0],
 [ 1674541069,0],
 [ 1674541068,0],
 [ 1674541067,0],
 [ 1674541066,0],
 [ 1674541065,0],
 [ 1674541064,0],
 [ 1674541063,0],
 [ 1674541062,0],
 [ 1674541061,0],
 [ 1674541060,0],
 [ 1674541059,0],
 [ 1674541058,0],
 [ 1674541057,0],
 [ 1674541056,0],
 [ 1674541055,0],
 [ 1674541054,0],
 [ 1674541053,0],
 [ 1674541052,0],
 [ 1674541051,0],
 [ 1674541050,0],
 [ 1674541049,0],
 [ 1674541048,0],
 [ 1674541047,0],
 [ 1674541046,0],
 [ 1674541045,0],
 [ 1674541044,0],
 [ 1674541043,0],
 [ 1674541042,0],
 [ 1674541041,0],
 [ 1674541040,0],
 [ 1674541039,0],
 [ 1674541038,0],
 [ 1674541037,0],
 [ 1674541036,0],
 [ 1674541035,0],
 [ 1674541034,0],
 [ 1674541033,0],
 [ 1674541032,0],
 [ 1674541031,0],
 [ 1674541030,0],
 [ 1674541029,0],
 [ 1674541028,0],
 [ 1674541027,0],
 [ 1674541026,0],
 [ 1674541025,0],
 [ 1674541024,0],
 [ 1674541023,0],
 [ 1674541022,0],
 [ 1674541021,0],
 [ 1674541020,0],
 [ 1674541019,0],
 [ 1674541018,0],
 [ 1674541017,0],
 [ 1674541016,0],
 [ 1674541015,0],
 [ 1674541014,0],
 [ 1674541013,0],
 [ 1674541012,0],
 [ 1674541011,0],
 [ 1674541010,0],
 [ 1674541009,0],
 [ 1674541008,0],
 [ 1674541007,0],
 [ 1674541006,0],
 [ 1674541005,0],
 [ 1674541004,0],
 [ 1674541003,0],
 [ 1674541002,0],
 [ 1674541001,0],
 [ 1674541000,0],
 [ 1674540999,0],
 [ 1674540998,0],
 [ 1674540997,0],
 [ 1674540996,0],
 [ 1674540995,0.1386571],
 [ 1674540994,0.8514073],
 [ 1674540993,0.8607974],
 [ 1674540992,0.8545377],
 [ 1674540991,0.7433551],
 [ 1674540990,0.6537163],
 [ 1674540989,0.8415392],
 [ 1674540988,0.766482],
 [ 1674540987,0.8049369]
]
}

response from that node via the parent:

# http://XXX:19999/host/ml-demo-ml-enabled/api/v1/data?chart=prometheus_node_exporter_demo_cpu_1s.node_cpu_seconds_total-cpu_0-mode_idle&after=1674540987&before=1674542259

{
 "labels": ["time","node_cpu_seconds_total"],
    "data":
 [
 [ 1674542259,0.7380111],
 [ 1674542258,0.8374951],
 [ 1674542257,0.8322216],
 [ 1674542256,0.7788059],
 [ 1674542255,0.7391061],
 [ 1674542254,0.7415856],
 [ 1674542253,0.7905446],
 [ 1674542252,0.7493084],
 [ 1674542251,0.706842],
 [ 1674542250,0.7283548],
 [ 1674542249,0.7932807],
 [ 1674542248,0.7767592],
 [ 1674542247,0.7862085],
 [ 1674542246,0.778068],
 [ 1674542245,0.6085273],
 [ 1674542244,0.7044847],
 [ 1674542243,0.8351854],
 [ 1674542242,0.8070572],
 [ 1674542241,0.789161],
 [ 1674542240,0.7715845],
 [ 1674542239,0.7893227],
 [ 1674542238,0.8463274],
 [ 1674542237,0.8521208],
 [ 1674542236,0.7391557],
 [ 1674542235,0.7111975],
 [ 1674542234,0.8106582],
 [ 1674542233,0.8541635],
 [ 1674542232,0.7969032],
 [ 1674542231,0.7952874],
 [ 1674542230,0.7080905],
 [ 1674542229,0.7537789],
 [ 1674542228,0.8517211],
 [ 1674542227,0.8367995],
 [ 1674542226,0.7550967],
 [ 1674542225,0.6920824],
 [ 1674542224,0.7694142],
 [ 1674542223,0.8414558],
 [ 1674542222,0.8020603],
 [ 1674542221,0.7288073],
 [ 1674542220,0.7059713],
 [ 1674542219,0.7866791],
 [ 1674542218,0.851679],
 [ 1674542217,0.8363805],
 [ 1674542216,0.7670473],
 [ 1674542215,0.648291],
 [ 1674542214,0.737313],
 [ 1674542213,0.7566691],
 [ 1674542212,0.7421945],
 [ 1674542211,0.48641],
 [ 1674542210,0.1188677],
 [ 1674542209,0],
 [ 1674542208,0],
 [ 1674542207,0],
 [ 1674542206,0],
 [ 1674542205,0],
 [ 1674542204,0],
 [ 1674542203,0],
 [ 1674542202,0],
 [ 1674542201,0],
 [ 1674542200,0],
 [ 1674542199,0],
 [ 1674542198,0],
 [ 1674542197,0],
 [ 1674542196,0],
 [ 1674542195,0],
 [ 1674542194,0],
 [ 1674542193,0],
 [ 1674542192,0],
 [ 1674542191,0],
 [ 1674542190,0],
 [ 1674542189,0],
 [ 1674542188,0],
 [ 1674542187,0],
 [ 1674542186,0],
 [ 1674542185,0],
 [ 1674542184,0],
 [ 1674542183,0],
 [ 1674542182,0],
 [ 1674542181,0],
 [ 1674542180,0],
 [ 1674542179,0],
 [ 1674542178,0],
 [ 1674542177,0],
 [ 1674542176,0],
 [ 1674542175,0],
 [ 1674542174,0],
 [ 1674542173,0],
 [ 1674542172,0],
 [ 1674542171,0],
 [ 1674542170,0],
 [ 1674542169,0],
 [ 1674542168,0],
 [ 1674542167,0],
 [ 1674542166,0],
 [ 1674542165,0],
 [ 1674542164,0],
 [ 1674542163,0],
 [ 1674542162,0],
 [ 1674542161,0],
 [ 1674542160,0],
 [ 1674542159,0],
 [ 1674542158,0],
 [ 1674542157,0],
 [ 1674542156,0],
 [ 1674542155,0],
 [ 1674542154,0],
 [ 1674542153,0],
 [ 1674542152,0],
 [ 1674542151,0],
 [ 1674542150,0],
 [ 1674542149,0],
 [ 1674542148,0],
 [ 1674542147,0],
 [ 1674542146,0],
 [ 1674542145,0],
 [ 1674542144,0],
 [ 1674542143,0],
 [ 1674542142,0],
 [ 1674542141,0],
 [ 1674542140,0],
 [ 1674542139,0],
 [ 1674542138,0],
 [ 1674542137,0.5008842],
 [ 1674542136,0.7861987],
 [ 1674542135,0.7220369],
 [ 1674542134,0.7742169],
 [ 1674542133,0.8258514],
 [ 1674542132,0.7945842],
 [ 1674542131,0.6757097],
 [ 1674542130,0.6925173],
 [ 1674542129,0.7982968],
 [ 1674542128,0.8296842],
 [ 1674542127,0.8233186],
 [ 1674542126,0.7679122],
 [ 1674542125,0.6419102],
 [ 1674542124,0.7385802],
 [ 1674542123,0.8375195],
 [ 1674542122,0.7887305],
 [ 1674542121,0.7952954],
 [ 1674542120,0.7669708],
 [ 1674542119,0.7841307],
 [ 1674542118,0.8433144],
 [ 1674542117,0.8543843],
 [ 1674542116,0.7375525],
 [ 1674542115,0.7124314],
 [ 1674542114,0.7981347],
 [ 1674542113,0.8415007],
 [ 1674542112,0.8139132],
 [ 1674542111,0.7369417],
 [ 1674542110,0.647001],
 [ 1674542109,0.7406904],
 [ 1674542108,0.8473263],
 [ 1674542107,0.8592827],
 [ 1674542106,0.8010679],
 [ 1674542105,0.712835],
 [ 1674542104,0.7743189],
 [ 1674542103,0.8341422],
 [ 1674542102,0.8123568],
 [ 1674542101,0.7302135],
 [ 1674542100,0.7328048],
 [ 1674542099,0.8104921],
 [ 1674542098,0.840162],
 [ 1674542097,0.8516615],
 [ 1674542096,0.7951339],
 [ 1674542095,0.6800925],
 [ 1674542094,0.7606504],
 [ 1674542093,0.7958283],
 [ 1674542092,0.7484925],
 [ 1674542091,0.7986215],
 [ 1674542090,0.8038962],
 [ 1674542089,0.7987777],
 [ 1674542088,0.8391796],
 [ 1674542087,0.8415716],
 [ 1674542086,0.7358079],
 [ 1674542085,0.6759195],
 [ 1674542084,0.7734975],
 [ 1674542083,0.8460914],
 [ 1674542082,0.7859388],
 [ 1674542081,0.7764124],
 [ 1674542080,0.6931799],
 [ 1674542079,0.7560052],
 [ 1674542078,0.8508051],
 [ 1674542077,0.8381881],
 [ 1674542076,0.7886663],
 [ 1674542075,0.7185739],
 [ 1674542074,0.7727481],
 [ 1674542073,0.75309],
 [ 1674542072,0.7295499],
 [ 1674542071,0.7197655],
 [ 1674542070,0.7297677],
 [ 1674542069,0.7980148],
 [ 1674542068,0.8407184],
 [ 1674542067,0.8491686],
 [ 1674542066,0.7793274],
 [ 1674542065,0.6245488],
 [ 1674542064,0.70918],
 [ 1674542063,0.8310575],
 [ 1674542062,0.7996415],
 [ 1674542061,0.7996098],
 [ 1674542060,0.7896839],
 [ 1674542059,0.7992278],
 [ 1674542058,0.8564935],
 [ 1674542057,0.8501171],
 [ 1674542056,0.6938941],
 [ 1674542055,0.682478],
 [ 1674542054,0.8105679],
 [ 1674542053,0.8482421],
 [ 1674542052,0.7987334],
 [ 1674542051,0.7827699],
 [ 1674542050,0.7129534],
 [ 1674542049,0.765527],
 [ 1674542048,0.8576995],
 [ 1674542047,0.8405618],
 [ 1674542046,0.7964329],
 [ 1674542045,0.7033558],
 [ 1674542044,0.7560823],
 [ 1674542043,0.8476452],
 [ 1674542042,0.8315349],
 [ 1674542041,0.7330175],
 [ 1674542040,0.7265355],
 [ 1674542039,0.8130486],
 [ 1674542038,0.8441068],
 [ 1674542037,0.8399606],
 [ 1674542036,0.7694332],
 [ 1674542035,0.6669399],
 [ 1674542034,0.7547101],
 [ 1674542033,0.7865095],
 [ 1674542032,0.7670538],
 [ 1674542031,0.828283],
 [ 1674542030,0.8064969],
 [ 1674542029,0.8046727],
 [ 1674542028,0.8536265],
 [ 1674542027,0.8581669],
 [ 1674542026,0.6615054],
 [ 1674542025,0.618433],
 [ 1674542024,0.7742118],
 [ 1674542023,0.85186],
 [ 1674542022,0.8186117],
 [ 1674542021,0.8077447],
 [ 1674542020,0.6963811],
 [ 1674542019,0.7513897],
 [ 1674542018,0.8735417],
 [ 1674542017,0.8428892],
 [ 1674542016,0.7687818],
 [ 1674542015,0.7340027],
 [ 1674542014,0.7771324],
 [ 1674542013,0.8405419],
 [ 1674542012,0.8133576],
 [ 1674542011,0.7445677],
 [ 1674542010,0.7318491],
 [ 1674542009,0.7874628],
 [ 1674542008,0.7667018],
 [ 1674542007,0.8008844],
 [ 1674542006,0.7899558],
 [ 1674542005,0.6433272],
 [ 1674542004,0.7292598],
 [ 1674542003,0.8399146],
 [ 1674542002,0.8104059],
 [ 1674542001,0.8312159],
 [ 1674542000,0.8069606],
 [ 1674541999,0.8054169],
 [ 1674541998,0.859466],
 [ 1674541997,0.8682521],
 [ 1674541996,0.7422215],
 [ 1674541995,0.7198313],
 [ 1674541994,0.806724],
 [ 1674541993,0.8462585],
 [ 1674541992,0.8093829],
 [ 1674541991,0.8155206],
 [ 1674541990,0.7209438],
 [ 1674541989,0.7513754],
 [ 1674541988,0.8534878],
 [ 1674541987,0.8476866],
 [ 1674541986,0.7775049],
 [ 1674541985,0.6917152],
 [ 1674541984,0.7654872],
 [ 1674541983,0.8459451],
 [ 1674541982,0.8215284],
 [ 1674541981,0.752432],
 [ 1674541980,0.7547576],
 [ 1674541979,0.8223251],
 [ 1674541978,0.8555939],
 [ 1674541977,0.8533767],
 [ 1674541976,0.7910733],
 [ 1674541975,0.6750765],
 [ 1674541974,0.7624334],
 [ 1674541973,0.7762451],
 [ 1674541972,0.7427334],
 [ 1674541971,0.7947398],
 [ 1674541970,0.7845932],
 [ 1674541969,0.8130302],
 [ 1674541968,0.8629254],
 [ 1674541967,0.8530113],
 [ 1674541966,0.6855586],
 [ 1674541965,0.6384326],
 [ 1674541964,0.7624095],
 [ 1674541963,0.8318353],
 [ 1674541962,0.8115363],
 [ 1674541961,0.807346],
 [ 1674541960,0.6955122],
 [ 1674541959,0.7516015],
 [ 1674541958,0.8628177],
 [ 1674541957,0.8588392],
 [ 1674541956,0.7840833],
 [ 1674541955,0.7452543],
 [ 1674541954,0.7912559],
 [ 1674541953,0.8435048],
 [ 1674541952,0.8197146],
 [ 1674541951,0.7365902],
 [ 1674541950,0.7170664],
 [ 1674541949,0.7360102],
 [ 1674541948,0.7823062],
 [ 1674541947,0.7987505],
 [ 1674541946,0.7477883],
 [ 1674541945,0.6270581],
 [ 1674541944,0.7428197],
 [ 1674541943,0.8677959],
 [ 1674541942,0.7992718],
 [ 1674541941,0.7923732],
 [ 1674541940,0.7972015],
 [ 1674541939,0.8203365],
 [ 1674541938,0.8490723],
 [ 1674541937,0.8558153],
 [ 1674541936,0.7363338],
 [ 1674541935,0.7088874],
 [ 1674541934,0.8030187],
 [ 1674541933,0.8587925],
 [ 1674541932,0.8164997],
 [ 1674541931,0.8206068],
 [ 1674541930,0.7144286],
 [ 1674541929,0.7507404],
 [ 1674541928,0.8515316],
 [ 1674541927,0.8593779],
 [ 1674541926,0.8010395],
 [ 1674541925,0.7068279],
 [ 1674541924,0.7760604],
 [ 1674541923,0.8441114],
 [ 1674541922,0.8114953],
 [ 1674541921,0.7365577],
 [ 1674541920,0.7467023],
 [ 1674541919,0.8094925],
 [ 1674541918,0.8487764],
 [ 1674541917,0.8418114],
 [ 1674541916,0.7172674],
 [ 1674541915,0.6047228],
 [ 1674541914,0.7333122],
 [ 1674541913,0.7770517],
 [ 1674541912,0.7539407],
 [ 1674541911,0.7977898],
 [ 1674541910,0.8015848],
 [ 1674541909,0.8087321],
 [ 1674541908,0.8492131],
 [ 1674541907,0.8466848],
 [ 1674541906,0.7243463],
 [ 1674541905,0.6668824],
 [ 1674541904,0.7618886],
 [ 1674541903,0.8446414],
 [ 1674541902,0.8005852],
 [ 1674541901,0.8006545],
 [ 1674541900,0.7180207],
 [ 1674541899,0.7520063],
 [ 1674541898,0.8476542],
 [ 1674541897,0.8551873],
 [ 1674541896,0.7797629],
 [ 1674541895,0.7182014],
 [ 1674541894,0.5733431],
 [ 1674541893,0.1965346],
 [ 1674541892,0],
 [ 1674541891,0],
 [ 1674541890,0],
 [ 1674541889,0],
 [ 1674541888,0.2369473],
 [ 1674541887,0.6598281],
 [ 1674541886,0.7863734],
 [ 1674541885,0.6198017],
 [ 1674541884,0.7122358],
 [ 1674541883,0.8469932],
 [ 1674541882,0.7797505],
 [ 1674541881,0.7763134],
 [ 1674541880,0.7856868],
 [ 1674541879,0.8053748],
 [ 1674541878,0.8488297],
 [ 1674541877,0.8654438],
 [ 1674541876,0.7513677],
 [ 1674541875,0.7290683],
 [ 1674541874,0.8122435],
 [ 1674541873,0.8525305],
 [ 1674541872,0.8235222],
 [ 1674541871,0.8110731],
 [ 1674541870,0.6910607],
 [ 1674541869,0.7425243],
 [ 1674541868,0.8507775],
 [ 1674541867,0.8487687],
 [ 1674541866,0.798297],
 [ 1674541865,0.7219832],
 [ 1674541864,0.7666537],
 [ 1674541863,0.8447309],
 [ 1674541862,0.8192456],
 [ 1674541861,0.6826996],
 [ 1674541860,0.6919623],
 [ 1674541859,0.8148257],
 [ 1674541858,0.8543223],
 [ 1674541857,0.8623381],
 [ 1674541856,0.80502],
 [ 1674541855,0.6770578],
 [ 1674541854,0.7554179],
 [ 1674541853,0.7792851],
 [ 1674541852,0.7594852],
 [ 1674541851,0.8035176],
 [ 1674541850,0.8023173],
 [ 1674541849,0.8136642],
 [ 1674541848,0.848828],
 [ 1674541847,0.8595748],
 [ 1674541846,0.7354622],
 [ 1674541845,0.6566854],
 [ 1674541844,0.771882],
 [ 1674541843,0.8498463],
 [ 1674541842,0.8085217],
 [ 1674541841,0.8208972],
 [ 1674541840,0.7087051],
 [ 1674541839,0.7514634],
 [ 1674541838,0.8617025],
 [ 1674541837,0.8701493],
 [ 1674541836,0.8110106],
 [ 1674541835,0.7286133],
 [ 1674541834,0.7825677],
 [ 1674541833,0.8517172],
 [ 1674541832,0.8078694],
 [ 1674541831,0.6760696],
 [ 1674541830,0.6809539],
 [ 1674541829,0.7951638],
 [ 1674541828,0.8342331],
 [ 1674541827,0.8003336],
 [ 1674541826,0.7494634],
 [ 1674541825,0.6120952],
 [ 1674541824,0.7070138],
 [ 1674541823,0.846933],
 [ 1674541822,0.8127314],
 [ 1674541821,0.7750651],
 [ 1674541820,0.7519917],
 [ 1674541819,0.7924953],
 [ 1674541818,0.8487405],
 [ 1674541817,0.8606885],
 [ 1674541816,0.7468351],
 [ 1674541815,0.71655],
 [ 1674541814,0.8040283],
 [ 1674541813,0.8692375],
 [ 1674541812,0.8331566],
 [ 1674541811,0.7951914],
 [ 1674541810,0.6505106],
 [ 1674541809,0.7031376],
 [ 1674541808,0.8454398],
 [ 1674541807,0.8702357],
 [ 1674541806,0.7697059],
 [ 1674541805,0.7057637],
 [ 1674541804,0.7867508],
 [ 1674541803,0.8517855],
 [ 1674541802,0.8362884],
 [ 1674541801,0.7265116],
 [ 1674541800,0.7256307],
 [ 1674541799,0.8162361],
 [ 1674541798,0.8493076],
 [ 1674541797,0.8499587],
 [ 1674541796,0.8028814],
 [ 1674541795,0.6756792],
 [ 1674541794,0.751405],
 [ 1674541793,0.7910523],
 [ 1674541792,0.768721],
 [ 1674541791,0.8161942],
 [ 1674541790,0.8094386],
 [ 1674541789,0.8070319],
 [ 1674541788,0.8494065],
 [ 1674541787,0.8699622],
 [ 1674541786,0.7345544],
 [ 1674541785,0.657651],
 [ 1674541784,0.7701482],
 [ 1674541783,0.8471153],
 [ 1674541782,0.8023552],
 [ 1674541781,0.7986995],
 [ 1674541780,0.7266105],
 [ 1674541779,0.7718787],
 [ 1674541778,0.8668981],
 [ 1674541777,0.8516153],
 [ 1674541776,0.7709547],
 [ 1674541775,0.7115773],
 [ 1674541774,0.7705145],
 [ 1674541773,0.834151],
 [ 1674541772,0.7327903],
 [ 1674541771,0.6823185],
 [ 1674541770,0.7177005],
 [ 1674541769,0.7890917],
 [ 1674541768,0.8240592],
 [ 1674541767,0.808056],
 [ 1674541766,0.7411501],
 [ 1674541765,0.6240817],
 [ 1674541764,0.7420763],
 [ 1674541763,0.858953],
 [ 1674541762,0.8120084],
 [ 1674541761,0.7917757],
 [ 1674541760,0.7764399],
 [ 1674541759,0.8072817],
 [ 1674541758,0.8517347],
 [ 1674541757,0.8540348],
 [ 1674541756,0.7448081],
 [ 1674541755,0.7119668],
 [ 1674541754,0.8168394],
 [ 1674541753,0.8582314],
 [ 1674541752,0.8134976],
 [ 1674541751,0.7371302],
 [ 1674541750,0.6584255],
 [ 1674541749,0.7473119],
 [ 1674541748,0.8428623],
 [ 1674541747,0.8498202],
 [ 1674541746,0.7843794],
 [ 1674541745,0.6986674],
 [ 1674541744,0.7760954],
 [ 1674541743,0.8511305],
 [ 1674541742,0.8141914],
 [ 1674541741,0.7244262],
 [ 1674541740,0.7153609],
 [ 1674541739,0.8060341],
 [ 1674541738,0.8437558],
 [ 1674541737,0.8568041],
 [ 1674541736,0.7993633],
 [ 1674541735,0.6674671],
 [ 1674541734,0.7407471],
 [ 1674541733,0.788371],
 [ 1674541732,0.7828931],
 [ 1674541731,0.811769],
 [ 1674541730,0.8032512],
 [ 1674541729,0.8252265],
 [ 1674541728,0.8433716],
 [ 1674541727,0.8517568],
 [ 1674541726,0.7422142],
 [ 1674541725,0.6540049],
 [ 1674541724,0.7678143],
 [ 1674541723,0.8560555],
 [ 1674541722,0.8187419],
 [ 1674541721,0.8075733],
 [ 1674541720,0.6489959],
 [ 1674541719,0.7197282],
 [ 1674541718,0.8496805],
 [ 1674541717,0.8621609],
 [ 1674541716,0.8047675],
 [ 1674541715,0.7297872],
 [ 1674541714,0.7817471],
 [ 1674541713,0.8296801],
 [ 1674541712,0.7939048],
 [ 1674541711,0.7237613],
 [ 1674541710,0.7310935],
 [ 1674541709,0.8065746],
 [ 1674541708,0.8292742],
 [ 1674541707,0.825502],
 [ 1674541706,0.7879831],
 [ 1674541705,0.6361808],
 [ 1674541704,0.7209602],
 [ 1674541703,0.8399777],
 [ 1674541702,0.8115118],
 [ 1674541701,0.7315561],
 [ 1674541700,0.7373446],
 [ 1674541699,0.7808255],
 [ 1674541698,0.8187895],
 [ 1674541697,0.8517482],
 [ 1674541696,0.7480876],
 [ 1674541695,0.6994942],
 [ 1674541694,0.7847607],
 [ 1674541693,0.8476641],
 [ 1674541692,0.8128428],
 [ 1674541691,0.785895],
 [ 1674541690,0.6964408],
 [ 1674541689,0.7595982],
 [ 1674541688,0.855322],
 [ 1674541687,0.8424072],
 [ 1674541686,0.7659814],
 [ 1674541685,0.696003],
 [ 1674541684,0.7678213],
 [ 1674541683,0.8428845],
 [ 1674541682,0.8083113],
 [ 1674541681,0.7321642],
 [ 1674541680,0.7354205],
 [ 1674541679,0.80728],
 [ 1674541678,0.8399643],
 [ 1674541677,0.8527564],
 [ 1674541676,0.7947938],
 [ 1674541675,0.6726921],
 [ 1674541674,0.7674],
 [ 1674541673,0.787403],
 [ 1674541672,0.7476506],
 [ 1674541671,0.7895283],
 [ 1674541670,0.7863943],
 [ 1674541669,0.81129],
 [ 1674541668,0.8341181],
 [ 1674541667,0.8605871],
 [ 1674541666,0.7447304],
 [ 1674541665,0.596674],
 [ 1674541664,0.7127587],
 [ 1674541663,0.8370949],
 [ 1674541662,0.8029026],
 [ 1674541661,0.8111839],
 [ 1674541660,0.7169323],
 [ 1674541659,0.7537892],
 [ 1674541658,0.8451408],
 [ 1674541657,0.8274972],
 [ 1674541656,0.780247],
 [ 1674541655,0.7239009],
 [ 1674541654,0.7588887],
 [ 1674541653,0.817687],
 [ 1674541652,0.8004867],
 [ 1674541651,0.7278653],
 [ 1674541650,0.7201587],
 [ 1674541649,0.7874223],
 [ 1674541648,0.743158],
 [ 1674541647,0.7540103],
 [ 1674541646,0.7521177],
 [ 1674541645,0.6086428],
 [ 1674541644,0.7191182],
 [ 1674541643,0.849086],
 [ 1674541642,0.8084879],
 [ 1674541641,0.7918049],
 [ 1674541640,0.7822109],
 [ 1674541639,0.8064226],
 [ 1674541638,0.8473927],
 [ 1674541637,0.8415576],
 [ 1674541636,0.7298483],
 [ 1674541635,0.6953439],
 [ 1674541634,0.7848987],
 [ 1674541633,0.8419513],
 [ 1674541632,0.8025946],
 [ 1674541631,0.7945578],
 [ 1674541630,0.6806414],
 [ 1674541629,0.7368643],
 [ 1674541628,0.8274521],
 [ 1674541627,0.8258985],
 [ 1674541626,0.788777],
 [ 1674541625,0.7077402],
 [ 1674541624,0.7475282],
 [ 1674541623,0.8236829],
 [ 1674541622,0.8044712],
 [ 1674541621,0.7031401],
 [ 1674541620,0.7092062],
 [ 1674541619,0.8031649],
 [ 1674541618,0.838857],
 [ 1674541617,0.8097133],
 [ 1674541616,0.7556597],
 [ 1674541615,0.6175815],
 [ 1674541614,0.6987081],
 [ 1674541613,0.7519435],
 [ 1674541612,0.7333412],
 [ 1674541611,0.7975963],
 [ 1674541610,0.789279],
 [ 1674541609,0.8003778],
 [ 1674541608,0.8315779],
 [ 1674541607,0.8292957],
 [ 1674541606,0.6515698],
 [ 1674541605,0.5965568],
 [ 1674541604,0.7340726],
 [ 1674541603,0.8171948],
 [ 1674541602,0.8000892],
 [ 1674541601,0.808814],
 [ 1674541600,0.7070198],
 [ 1674541599,0.7614882],
 [ 1674541598,0.8544408],
 [ 1674541597,0.8448368],
 [ 1674541596,0.7917409],
 [ 1674541595,0.7238585],
 [ 1674541594,0.7658563],
 [ 1674541593,0.8274842],
 [ 1674541592,0.7930257],
 [ 1674541591,0.7183964],
 [ 1674541590,0.6834854],
 [ 1674541589,0.6624036],
 [ 1674541588,0.7325311],
 [ 1674541587,0.7722006],
 [ 1674541586,0.7434466],
 [ 1674541585,0.5997589],
 [ 1674541584,0.698149],
 [ 1674541583,0.8182445],
 [ 1674541582,0.7628587],
 [ 1674541581,0.7899683],
 [ 1674541580,0.7941832],
 [ 1674541579,0.8057396],
 [ 1674541578,0.8296814],
 [ 1674541577,0.8528013],
 [ 1674541576,0.7462634],
 [ 1674541575,0.7071346],
 [ 1674541574,0.7929838],
 [ 1674541573,0.8418828],
 [ 1674541572,0.8146064],
 [ 1674541571,0.8028117],
 [ 1674541570,0.6931616],
 [ 1674541569,0.7449597],
 [ 1674541568,0.8395561],
 [ 1674541567,0.8573921],
 [ 1674541566,0.7749837],
 [ 1674541565,0.6812893],
 [ 1674541564,0.7315887],
 [ 1674541563,0.8096558],
 [ 1674541562,0.7994655],
 [ 1674541561,0.7270912],
 [ 1674541560,0.7190946],
 [ 1674541559,0.816041],
 [ 1674541558,0.86424],
 [ 1674541557,0.8481683],
 [ 1674541556,0.7162115],
 [ 1674541555,0.6012624],
 [ 1674541554,0.7214234],
 [ 1674541553,0.7779985],
 [ 1674541552,0.768713],
 [ 1674541551,0.7974307],
 [ 1674541550,0.7806815],
 [ 1674541549,0.7951295],
 [ 1674541548,0.8075407],
 [ 1674541547,0.8412813],
 [ 1674541546,0.7470938],
 [ 1674541545,0.6538091],
 [ 1674541544,0.7614066],
 [ 1674541543,0.8400294],
 [ 1674541542,0.7937652],
 [ 1674541541,0.7951895],
 [ 1674541540,0.6895736],
 [ 1674541539,0.7424954],
 [ 1674541538,0.8597568],
 [ 1674541537,0.8626699],
 [ 1674541536,0.7649512],
 [ 1674541535,0.7055862],
 [ 1674541534,0.7447676],
 [ 1674541533,0.8063949],
 [ 1674541532,0.7233215],
 [ 1674541531,0.6848652],
 [ 1674541530,0.7277702],
 [ 1674541529,0.799098],
 [ 1674541528,0.8340273],
 [ 1674541527,0.8005368],
 [ 1674541526,0.751439],
 [ 1674541525,0.6055901],
 [ 1674541524,0.7141532],
 [ 1674541523,0.8451001],
 [ 1674541522,0.780595],
 [ 1674541521,0.7512345],
 [ 1674541520,0.7256919],
 [ 1674541519,0.7587196],
 [ 1674541518,0.8286317],
 [ 1674541517,0.8441072],
 [ 1674541516,0.7338703],
 [ 1674541515,0.7071907],
 [ 1674541514,0.7930351],
 [ 1674541513,0.8417727],
 [ 1674541512,0.8029246],
 [ 1674541511,0.7994755],
 [ 1674541510,0.7138032],
 [ 1674541509,0.6871409],
 [ 1674541508,0.7966856],
 [ 1674541507,0.8558832],
 [ 1674541506,0.7715902],
 [ 1674541505,0.6982617],
 [ 1674541504,0.7492063],
 [ 1674541503,0.8193391],
 [ 1674541502,0.8044595],
 [ 1674541501,0.7328349],
 [ 1674541500,0.7353151],
 [ 1674541499,0.8248864],
 [ 1674541498,0.852429],
 [ 1674541497,0.8399132],
 [ 1674541496,0.7869473],
 [ 1674541495,0.6320717],
 [ 1674541494,0.7268816],
 [ 1674541493,0.7733381],
 [ 1674541492,0.7494981],
 [ 1674541491,0.7935611],
 [ 1674541490,0.7804603],
 [ 1674541489,0.7954002],
 [ 1674541488,0.8435572],
 [ 1674541487,0.8482315],
 [ 1674541486,0.7111758],
 [ 1674541485,0.5909938],
 [ 1674541484,0.7464173],
 [ 1674541483,0.8584621],
 [ 1674541482,0.8024467],
 [ 1674541481,0.8057212],
 [ 1674541480,0.7369822],
 [ 1674541479,0.7645222],
 [ 1674541478,0.8587829],
 [ 1674541477,0.8683568],
 [ 1674541476,0.7889457],
 [ 1674541475,0.7280296],
 [ 1674541474,0.7732649],
 [ 1674541473,0.8158976],
 [ 1674541472,0.7370986],
 [ 1674541471,0.6801291],
 [ 1674541470,0.710623],
 [ 1674541469,0.7830459],
 [ 1674541468,0.8064051],
 [ 1674541467,0.8120727],
 [ 1674541466,0.7488018],
 [ 1674541465,0.6071329],
 [ 1674541464,0.7230113],
 [ 1674541463,0.8477911],
 [ 1674541462,0.809586],
 [ 1674541461,0.7865408],
 [ 1674541460,0.571023],
 [ 1674541459,0.1714354],
 [ 1674541458,0],
 [ 1674541457,0],
 [ 1674541456,0],
 [ 1674541455,0],
 [ 1674541454,0],
 [ 1674541453,0],
 [ 1674541452,0],
 [ 1674541451,0],
 [ 1674541450,0],
 [ 1674541449,0],
 [ 1674541448,0],
 [ 1674541447,0],
 [ 1674541446,0],
 [ 1674541445,0],
 [ 1674541444,0],
 [ 1674541443,0],
 [ 1674541442,0],
 [ 1674541441,0],
 [ 1674541440,0],
 [ 1674541439,0],
 [ 1674541438,0],
 [ 1674541437,0],
 [ 1674541436,0],
 [ 1674541435,0],
 [ 1674541434,0],
 [ 1674541433,0],
 [ 1674541432,0],
 [ 1674541431,0],
 [ 1674541430,0],
 [ 1674541429,0],
 [ 1674541428,0],
 [ 1674541427,0],
 [ 1674541426,0],
 [ 1674541425,0],
 [ 1674541424,0],
 [ 1674541423,0],
 [ 1674541422,0],
 [ 1674541421,0],
 [ 1674541420,0],
 [ 1674541419,0],
 [ 1674541418,0],
 [ 1674541417,0],
 [ 1674541416,0],
 [ 1674541415,0],
 [ 1674541414,0],
 [ 1674541413,0],
 [ 1674541412,0],
 [ 1674541411,0],
 [ 1674541410,0],
 [ 1674541409,0],
 [ 1674541408,0],
 [ 1674541407,0],
 [ 1674541406,0],
 [ 1674541405,0],
 [ 1674541404,0],
 [ 1674541403,0],
 [ 1674541402,0],
 [ 1674541401,0],
 [ 1674541400,0],
 [ 1674541399,0],
 [ 1674541398,0],
 [ 1674541397,0],
 [ 1674541396,0],
 [ 1674541395,0],
 [ 1674541394,0],
 [ 1674541393,0],
 [ 1674541392,0],
 [ 1674541391,0],
 [ 1674541390,0],
 [ 1674541389,0],
 [ 1674541388,0],
 [ 1674541387,0],
 [ 1674541386,0],
 [ 1674541385,0],
 [ 1674541384,0],
 [ 1674541383,0],
 [ 1674541382,0],
 [ 1674541381,0],
 [ 1674541380,0],
 [ 1674541379,0.5030493],
 [ 1674541378,0.8551755],
 [ 1674541377,0.8661914],
 [ 1674541376,0.8103866],
 [ 1674541375,0.6768019],
 [ 1674541374,0.7584642],
 [ 1674541373,0.7997346],
 [ 1674541372,0.7836795],
 [ 1674541371,0.817767],
 [ 1674541370,0.8058023],
 [ 1674541369,0.8139709],
 [ 1674541368,0.794831],
 [ 1674541367,0.8102359],
 [ 1674541366,0.7390676],
 [ 1674541365,0.664132],
 [ 1674541364,0.7483767],
 [ 1674541363,0.8277809],
 [ 1674541362,0.8104745],
 [ 1674541361,0.8135626],
 [ 1674541360,0.7061689],
 [ 1674541359,0.7556197],
 [ 1674541358,0.8617809],
 [ 1674541357,0.8699924],
 [ 1674541356,0.7993265],
 [ 1674541355,0.7323231],
 [ 1674541354,0.767124],
 [ 1674541353,0.830597],
 [ 1674541352,0.81524],
 [ 1674541351,0.7485437],
 [ 1674541350,0.6903267],
 [ 1674541349,0.7637092],
 [ 1674541348,0.8529325],
 [ 1674541347,0.8286196],
 [ 1674541346,0.7705538],
 [ 1674541345,0.6025647],
 [ 1674541344,0.712315],
 [ 1674541343,0.8599912],
 [ 1674541342,0.81283],
 [ 1674541341,0.7919243],
 [ 1674541340,0.7880867],
 [ 1674541339,0.8164849],
 [ 1674541338,0.8458739],
 [ 1674541337,0.837273],
 [ 1674541336,0.6824323],
 [ 1674541335,0.6743672],
 [ 1674541334,0.8166807],
 [ 1674541333,0.8640791],
 [ 1674541332,0.81276],
 [ 1674541331,0.8035906],
 [ 1674541330,0.7256497],
 [ 1674541329,0.7376907],
 [ 1674541328,0.8222601],
 [ 1674541327,0.8619999],
 [ 1674541326,0.778512],
 [ 1674541325,0.7037766],
 [ 1674541324,0.7760304],
 [ 1674541323,0.8488884],
 [ 1674541322,0.8182484],
 [ 1674541321,0.6831357],
 [ 1674541320,0.7112342],
 [ 1674541319,0.8279994],
 [ 1674541318,0.8589239],
 [ 1674541317,0.8533604],
 [ 1674541316,0.7861789],
 [ 1674541315,0.6629856],
 [ 1674541314,0.7510293],
 [ 1674541313,0.7918004],
 [ 1674541312,0.779227],
 [ 1674541311,0.8110564],
 [ 1674541310,0.8022655],
 [ 1674541309,0.8077088],
 [ 1674541308,0.861369],
 [ 1674541307,0.8674645],
 [ 1674541306,0.7316573],
 [ 1674541305,0.6611963],
 [ 1674541304,0.7466461],
 [ 1674541303,0.8353618],
 [ 1674541302,0.8188177],
 [ 1674541301,0.7958692],
 [ 1674541300,0.6354136],
 [ 1674541299,0.7028793],
 [ 1674541298,0.8528205],
 [ 1674541297,0.8582697],
 [ 1674541296,0.7970684],
 [ 1674541295,0.7363211],
 [ 1674541294,0.7670714],
 [ 1674541293,0.817856],
 [ 1674541292,0.7955527],
 [ 1674541291,0.7291748],
 [ 1674541290,0.6999461],
 [ 1674541289,0.7237806],
 [ 1674541288,0.8041104],
 [ 1674541287,0.8324729],
 [ 1674541286,0.7735118],
 [ 1674541285,0.6334764],
 [ 1674541284,0.7305013],
 [ 1674541283,0.8561741],
 [ 1674541282,0.7888362],
 [ 1674541281,0.7934202],
 [ 1674541280,0.788283],
 [ 1674541279,0.8135136],
 [ 1674541278,0.8329872],
 [ 1674541277,0.8308585],
 [ 1674541276,0.7272053],
 [ 1674541275,0.7030114],
 [ 1674541274,0.7940726],
 [ 1674541273,0.8474182],
 [ 1674541272,0.8179568],
 [ 1674541271,0.7840885],
 [ 1674541270,0.6273764],
 [ 1674541269,0.6982359],
 [ 1674541268,0.8484185],
 [ 1674541267,0.8396027],
 [ 1674541266,0.7847999],
 [ 1674541265,0.7127563],
 [ 1674541264,0.7812337],
 [ 1674541263,0.8437741],
 [ 1674541262,0.8097667],
 [ 1674541261,0.7546096],
 [ 1674541260,0.7309102],
 [ 1674541259,0.801641],
 [ 1674541258,0.8441227],
 [ 1674541257,0.8389427],
 [ 1674541256,0.793204],
 [ 1674541255,0.6533253],
 [ 1674541254,0.7386389],
 [ 1674541253,0.793017],
 [ 1674541252,0.7552858],
 [ 1674541251,0.7939725],
 [ 1674541250,0.802262],
 [ 1674541249,0.8146834],
 [ 1674541248,0.8485704],
 [ 1674541247,0.8470857],
 [ 1674541246,0.7213193],
 [ 1674541245,0.6348099],
 [ 1674541244,0.7015553],
 [ 1674541243,0.7963645],
 [ 1674541242,0.7939026],
 [ 1674541241,0.7876664],
 [ 1674541240,0.6719143],
 [ 1674541239,0.7452843],
 [ 1674541238,0.8652109],
 [ 1674541237,0.8755121],
 [ 1674541236,0.785838],
 [ 1674541235,0.6311444],
 [ 1674541234,0.7346261],
 [ 1674541233,0.814585],
 [ 1674541232,0.7368979],
 [ 1674541231,0.6882627],
 [ 1674541230,0.7221906],
 [ 1674541229,0.8158993],
 [ 1674541228,0.8352685],
 [ 1674541227,0.8320206],
 [ 1674541226,0.7925888],
 [ 1674541225,0.6294912],
 [ 1674541224,0.7345738],
 [ 1674541223,0.859445],
 [ 1674541222,0.8141506],
 [ 1674541221,0.7902038],
 [ 1674541220,0.7314528],
 [ 1674541219,0.7659848],
 [ 1674541218,0.8370067],
 [ 1674541217,0.8558078],
 [ 1674541216,0.718524],
 [ 1674541215,0.6730698],
 [ 1674541214,0.7640244],
 [ 1674541213,0.8126228],
 [ 1674541212,0.7667593],
 [ 1674541211,0.7595339],
 [ 1674541210,0.6076555],
 [ 1674541209,0.6783083],
 [ 1674541208,0.8464763],
 [ 1674541207,0.8649019],
 [ 1674541206,0.7704464],
 [ 1674541205,0.6798151],
 [ 1674541204,0.7661089],
 [ 1674541203,0.8517848],
 [ 1674541202,0.8305457],
 [ 1674541201,0.7687895],
 [ 1674541200,0.7576334],
 [ 1674541199,0.8181659],
 [ 1674541198,0.8438451],
 [ 1674541197,0.8451146],
 [ 1674541196,0.767418],
 [ 1674541195,0.6571219],
 [ 1674541194,0.7567043],
 [ 1674541193,0.790397],
 [ 1674541192,0.7671176],
 [ 1674541191,0.8036593],
 [ 1674541190,0.773711],
 [ 1674541189,0.7692654],
 [ 1674541188,0.8423786],
 [ 1674541187,0.8682112],
 [ 1674541186,0.7420435],
 [ 1674541185,0.6598928],
 [ 1674541184,0.766163],
 [ 1674541183,0.8518695],
 [ 1674541182,0.8246124],
 [ 1674541181,0.7234079],
 [ 1674541180,0.5525489],
 [ 1674541179,0.6942317],
 [ 1674541178,0.8680758],
 [ 1674541177,0.857082],
 [ 1674541176,0.7927028],
 [ 1674541175,0.7481249],
 [ 1674541174,0.7744599],
 [ 1674541173,0.8347347],
 [ 1674541172,0.6776661],
 [ 1674541171,0.6384157],
 [ 1674541170,0.7171637],
 [ 1674541169,0.8010265],
 [ 1674541168,0.8440814],
 [ 1674541167,0.8231546],
 [ 1674541166,0.7456499],
 [ 1674541165,0.5987219],
 [ 1674541164,0.7193116],
 [ 1674541163,0.8437442],
 [ 1674541162,0.7980489],
 [ 1674541161,0.7876995],
 [ 1674541160,0.7831745],
 [ 1674541159,0.8170382],
 [ 1674541158,0.8552116],
 [ 1674541157,0.8658495],
 [ 1674541156,0.7343958],
 [ 1674541155,0.6939481],
 [ 1674541154,0.7887499],
 [ 1674541153,0.8480758],
 [ 1674541152,0.7947701],
 [ 1674541151,0.70799],
 [ 1674541150,0.6269172],
 [ 1674541149,0.7326486],
 [ 1674541148,0.8473494],
 [ 1674541147,0.8417702],
 [ 1674541146,0.7895805],
 [ 1674541145,0.7004557],
 [ 1674541144,0.7658952],
 [ 1674541143,0.8401336],
 [ 1674541142,0.7983998],
 [ 1674541141,0.6873289],
 [ 1674541140,0.6890379],
 [ 1674541139,0.7948471],
 [ 1674541138,0.8351324],
 [ 1674541137,0.8414542],
 [ 1674541136,0.7903039],
 [ 1674541135,0.6740866],
 [ 1674541134,0.7674572],
 [ 1674541133,0.7993926],
 [ 1674541132,0.7733549],
 [ 1674541131,0.8128764],
 [ 1674541130,0.8006477],
 [ 1674541129,0.8154369],
 [ 1674541128,0.8421466],
 [ 1674541127,0.8629656],
 [ 1674541126,0.7370048],
 [ 1674541125,0.6636675],
 [ 1674541124,0.7685154],
 [ 1674541123,0.8485502],
 [ 1674541122,0.8240382],
 [ 1674541121,0.7463791],
 [ 1674541120,0.6452697],
 [ 1674541119,0.7414535],
 [ 1674541118,0.8578348],
 [ 1674541117,0.8406335],
 [ 1674541116,0.7784666],
 [ 1674541115,0.7203022],
 [ 1674541114,0.7769427],
 [ 1674541113,0.8252555],
 [ 1674541112,0.6901808],
 [ 1674541111,0.6650578],
 [ 1674541110,0.7219867],
 [ 1674541109,0.7884851],
 [ 1674541108,0.835282],
 [ 1674541107,0.8288818],
 [ 1674541106,0.7665174],
 [ 1674541105,0.6289766],
 [ 1674541104,0.7262287],
 [ 1674541103,0.8230994],
 [ 1674541102,0.7803068],
 [ 1674541101,0.7825505],
 [ 1674541100,0.7824893],
 [ 1674541099,0.7946002],
 [ 1674541098,0.838532],
 [ 1674541097,0.8659923],
 [ 1674541096,0.7279214],
 [ 1674541095,0.6943971],
 [ 1674541094,0.8048393],
 [ 1674541093,0.8499561],
 [ 1674541092,0.8156193],
 [ 1674541091,0.7247069],
 [ 1674541090,0.6204073],
 [ 1674541089,0.7304959],
 [ 1674541088,0.8479533],
 [ 1674541087,0.8634851],
 [ 1674541086,0.7748578],
 [ 1674541085,0.6989789],
 [ 1674541084,0.7783531],
 [ 1674541083,0.8462514],
 [ 1674541082,0.8259969],
 [ 1674541081,0.7385384],
 [ 1674541080,0.7381271],
 [ 1674541079,0.8179995],
 [ 1674541078,0.8548884],
 [ 1674541077,0.8210992],
 [ 1674541076,0.7001891],
 [ 1674541075,0.622768],
 [ 1674541074,0.7395345],
 [ 1674541073,0.780743],
 [ 1674541072,0.7687204],
 [ 1674541071,0.8043924],
 [ 1674541070,0.7953675],
 [ 1674541069,0.8039498],
 [ 1674541068,0.8443716],
 [ 1674541067,0.8647848],
 [ 1674541066,0.7285648],
 [ 1674541065,0.6356248],
 [ 1674541064,0.7584208],
 [ 1674541063,0.8349127],
 [ 1674541062,0.793434],
 [ 1674541061,0.7869848],
 [ 1674541060,0.6059362],
 [ 1674541059,0.6742019],
 [ 1674541058,0.8378342],
 [ 1674541057,0.8516356],
 [ 1674541056,0.7791971],
 [ 1674541055,0.724883],
 [ 1674541054,0.778788],
 [ 1674541053,0.8321644],
 [ 1674541052,0.7864916],
 [ 1674541051,0.7198812],
 [ 1674541050,0.7294846],
 [ 1674541049,0.7913756],
 [ 1674541048,0.7382644],
 [ 1674541047,0.7565151],
 [ 1674541046,0.7622458],
 [ 1674541045,0.6120406],
 [ 1674541044,0.7012374],
 [ 1674541043,0.8385686],
 [ 1674541042,0.796565],
 [ 1674541041,0.7957515],
 [ 1674541040,0.7724969],
 [ 1674541039,0.8102516],
 [ 1674541038,0.8362175],
 [ 1674541037,0.8447336],
 [ 1674541036,0.7186273],
 [ 1674541035,0.6902153],
 [ 1674541034,0.8041287],
 [ 1674541033,0.8439308],
 [ 1674541032,0.810464],
 [ 1674541031,0.7437179],
 [ 1674541030,0.6270174],
 [ 1674541029,0.7169404],
 [ 1674541028,0.8387556],
 [ 1674541027,0.8556641],
 [ 1674541026,0.7885146],
 [ 1674541025,0.6935711],
 [ 1674541024,0.7375634],
 [ 1674541023,0.819567],
 [ 1674541022,0.8104106],
 [ 1674541021,0.7299558],
 [ 1674541020,0.7425745],
 [ 1674541019,0.8153234],
 [ 1674541018,0.8281683],
 [ 1674541017,0.8092072],
 [ 1674541016,0.7652133],
 [ 1674541015,0.6563132],
 [ 1674541014,0.7480191],
 [ 1674541013,0.7964754],
 [ 1674541012,0.7593702],
 [ 1674541011,0.7894977],
 [ 1674541010,0.7922486],
 [ 1674541009,0.7809963],
 [ 1674541008,0.8210597],
 [ 1674541007,0.8266555],
 [ 1674541006,0.6561865],
 [ 1674541005,0.6085042],
 [ 1674541004,0.7620448],
 [ 1674541003,0.8518151],
 [ 1674541002,0.8138474],
 [ 1674541001,0.7973613],
 [ 1674541000,0.6794756],
 [ 1674540999,0.7444228],
 [ 1674540998,0.8614582],
 [ 1674540997,0.8633897],
 [ 1674540996,0.7843598],
 [ 1674540995,0.718697],
 [ 1674540994,0.7680412],
 [ 1674540993,0.8093442],
 [ 1674540992,0.7400953],
 [ 1674540991,0.6779696],
 [ 1674540990,0.6222168],
 [ 1674540989,0.7435458],
 [ 1674540988,0.826392],
 [ 1674540987,0.774463]
]
}

two problems:
A. these seems to be a lag on the childs data in the parent.
B. the actual values are all also slightly different for some reason.

Expected behavior

child data should be accurate on the parent

for example at 1674540996 the value drops from 0.1386571 to 0 but on the parent version of this data the drop does not happen - instead it happens about 6mins later.

image
image

1674541380 - 1674540995 = 385 seconds later

also you can see from above that the actual raw values themselves are all a little different for some reason.

Steps to reproduce

unsure how reprobucilbe this is tbh

  1. set up streaming
  2. enabled Prometheus collector
  3. compare some data
    ...

Installation method

kickstart.sh

System info

dsdas

Netdata build info

andrewmaguire@ml-demo-ml-enabled:~$ netdata -W buildinfo
Version: netdata v1.37.0-167-nightly
Configure options:  '--prefix=/usr' '--sysconfdir=/etc' '--localstatedir=/var' '--libexecdir=/usr/libexec' '--libdir=/usr/lib' '--with-zlib' '--with-math' '--with-user=netdata' '--with-bundled-protobuf' 'CFLAGS=-O2 -pipe' 'LDFLAGS='
Install type: custom
Features:
    dbengine:                   YES
    Native HTTPS:               YES
    Netdata Cloud:              YES 
    ACLK:                       YES
    TLS Host Verification:      YES
    Machine Learning:           YES
    Stream Compression:         YES
Libraries:
    protobuf:                YES (bundled)
    jemalloc:                NO
    JSON-C:                  YES
    libcap:                  NO
    libcrypto:               YES
    libm:                    YES
    tcalloc:                 NO
    zlib:                    YES
Plugins:
    apps:                    YES
    cgroup Network Tracking: YES
    CUPS:                    NO
    EBPF:                    YES
    IPMI:                    NO
    NFACCT:                  NO
    perf:                    YES
    slabinfo:                YES
    Xen:                     NO
    Xen VBD Error Tracking:  NO
Exporters:
    AWS Kinesis:             NO
    GCP PubSub:              NO
    MongoDB:                 NO
    Prometheus Remote Write: NO
Debug/Developer Features:
    Trace Allocations:       NO

Additional info

No response

@andrewm4894
Copy link
Contributor Author

andrewm4894 commented Jan 24, 2023

end result here is that if i look at the local dashboard of the child node itself i see the drops at ~06:16:

image

but if i look via the parent (which is what NC will do) i see its all shifted and the first drop now is at ~06.23 according to the parent which is wrong.

image

Note also: the values between the two nodes even after the shift is accounted for are also not identical - they are similar but not identical so for some reason its all been shifted on the parent and the actual values on the parent are not that of the child - they are somewhat close - hence which two shifted lines look similar - but not identical in actual values when you look at the api - so thats potentially a big issue here too.

@ilyam8
Copy link
Member

ilyam8 commented Jan 25, 2023

@andrewm4894 this issue may be fixed in #14319. Can you please re-test?

@andrewm4894
Copy link
Contributor Author

will do

@andrewm4894
Copy link
Contributor Author

andrewm4894 commented Jan 26, 2023

@ilyam8 this does not look to be fixed from what i can see:

left = looking via the parent
right = looking direct at the node

Same timeframes and charts but can see they look very different.

The size of the lag and difference in the numbers looks to be even more confusing now for some reason.

Will try explore more to see if can find anything else maybe useful.

image

andrewmaguire@ml-demo-parent:/netdata$ sudo git log
commit bf38a22f323be8fa2b35bf782068d903064530ed (HEAD -> master, origin/master, origin/HEAD)
Author: Emmanuel Vasilakis
Date:   Wed Jan 25 19:06:06 2023 +0200

    Revert "Misc SSL improvements" (#14327)
    
    Revert "Misc SSL improvements (#14317)"
    
    This reverts commit f4d017ccf7385df3e152eeb5ea1259702ab01043.
andrewmaguire@ml-demo-ml-enabled:/netdata$ sudo git log
commit bf38a22f323be8fa2b35bf782068d903064530ed (HEAD -> master, origin/master, origin/HEAD)
Author: Emmanuel Vasilakis
Date:   Wed Jan 25 19:06:06 2023 +0200

    Revert "Misc SSL improvements" (#14327)
    
    Revert "Misc SSL improvements (#14317)"
    
    This reverts commit f4d017ccf7385df3e152eeb5ea1259702ab01043.

@andrewm4894
Copy link
Contributor Author

Note: it just seem to be affecting the Prometheus charts i have. All the other typical charts seem in line. Even other collectors i have enabled like alarms, pandas, and zscores all also seem to line up so just looks like the prom stuff i have configured like this:

jobs:

  - name: node_exporter_demo_cpu_1s
    url: https://node.demo.do.prometheus.io/metrics
    update_every: 1 
    selector:
      allow:
        - node_cpu*  

In case that might narrow it down.

@ilyam8
Copy link
Member

ilyam8 commented Feb 1, 2023

@andrewm4894 it is been 3 dbengine2 improvements PRs since then, please re-test when u have time.

@andrewm4894
Copy link
Contributor Author

just recreated now - will let things run for a while and then test again.

@ilyam8
Copy link
Member

ilyam8 commented Feb 2, 2023

is the problem solved?

@andrewm4894
Copy link
Contributor Author

@ilyam8 i don't seem to even see the data via the parent now for some reason. going to try investigate more

image

@andrewm4894
Copy link
Contributor Author

best comparison i can get is this that still looks very different

image

@thefiredragon
Copy link

@andrewm4894 I've the same issue, streams to parent shows no metrics anymore

Bildschirm­foto 2023-02-13 um 16 26 30

@ilyam8
Copy link
Member

ilyam8 commented Feb 13, 2023

@thefiredragon doesn't look the same to me. What is your Netdata version? What is children nodes memory mode?

@andrewm4894
Copy link
Contributor Author

(i'll retest today on my end on latest versions)

@andrewm4894
Copy link
Contributor Author

looking better now - but will need to let it all run for a while to be sure:

image

@thefiredragon
Copy link

@ilyam8 sorry, my issue was not related to this issue, someone had disabled my firewall rules, so devices had only connected to the registry and stream goes into somewhere.

@ilyam8
Copy link
Member

ilyam8 commented Feb 15, 2023

Closing as it appears to be fixed (#14318 (comment)). @andrewm4894 re-open if there are any problems. Also, I don't see it being prometheus specific. The only thing that distinguishes prometheus from any other collector is (default) update_every: it is 10 vs 1.

@ilyam8 ilyam8 closed this as completed Feb 15, 2023
@andrewm4894
Copy link
Contributor Author

@ilyam8 cool - i'll do one last recheck on my end and re-open if still seeing it - was looking ok after a few hours yesterday so will just do one more check to be sure.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug needs triage Issues which need to be manually labelled
Projects
None yet
Development

No branches or pull requests

3 participants