Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support for scraping multiple mysqld hosts #651

Merged
merged 22 commits into from
Sep 1, 2022
Merged

Support for scraping multiple mysqld hosts #651

merged 22 commits into from
Sep 1, 2022

Conversation

MattiasAng
Copy link
Contributor

This is a continuation on the work done by samitpal in #504 to address PR comments and add module parameter support.

@SuperQ
Copy link
Member

SuperQ commented Jul 29, 2022

Nice, thanks for picking this up.

@SuperQ
Copy link
Member

SuperQ commented Jul 29, 2022

CC @sysadmind who is working on similar support for the postgres_exporter.

We're using an INI style file here, rather than yaml, since it's more MySQL style.

@MattiasAng

This comment was marked as resolved.

Copy link

@sysadmind sysadmind left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that this PR is functionally correct. I would prefer to see this align with the /probe pattern as outlined by the prometheus docs, but otherwise, I think things look good in general.

mysqld_exporter.go Outdated Show resolved Hide resolved
config_multi_host.go Outdated Show resolved Hide resolved
@MattiasAng

This comment was marked as resolved.

@MattiasAng
Copy link
Contributor Author

@sysadmind @SuperQ Probe endpoint is added now. Config could probably be handled a bit better as @sysadmind has done on postgres_exporter but due to time constraint I only refactored the parsing.

I have yet to update / clean up documentation so ignore that for now, would appreciate any feedback on the implementation.

@SuperQ
Copy link
Member

SuperQ commented Aug 18, 2022

Nice, thanks. I'll take a look at this soon.

@MattiasAng MattiasAng requested a review from sysadmind August 19, 2022 07:26
probe.go Outdated Show resolved Hide resolved
mysqld_exporter.go Outdated Show resolved Hide resolved
Signed-off-by: Mattias Ängehov <mattias.angehov@castoredc.com>
Signed-off-by: Mattias Ängehov <mattias.angehov@castoredc.com>
Signed-off-by: Mattias Ängehov <mattias.angehov@castoredc.com>
Signed-off-by: Mattias Ängehov <mattias.angehov@castoredc.com>
Signed-off-by: Mattias Ängehov <mattias.angehov@castoredc.com>
Signed-off-by: Mattias Ängehov <mattias.angehov@castoredc.com>
Signed-off-by: Mattias Ängehov <mattias.angehov@castoredc.com>
@MattiasAng MattiasAng requested a review from sysadmind August 23, 2022 09:05
@MattiasAng
Copy link
Contributor Author

MattiasAng commented Aug 23, 2022

Updated documentation and some overall cleanup from the previous implementation.

Do you want me to add anything to the changelog or do you handle that on release?
Since we are removing support for DATA_SOURCE_NAME this is a breaking change.

Copy link

@sysadmind sysadmind left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM @SuperQ should be able to comment on the Changelog. I'm not familiar with the release process for this project.

@SuperQ
Copy link
Member

SuperQ commented Aug 23, 2022

So I mentioned removing ENV var support in #306.

I'm wondering if we should include some basic command line / ENV replacement for this.

Maybe if we just had MYSQLD_EXPORTER_PASSWORD.

This could help issues like #458, we could properly escape passwords in the connection string to the MySQL driver.

I think we still want to support the sidecar use case easily, without a config file for credentials.

@MattiasAng
Copy link
Contributor Author

Maybe if we just had MYSQLD_EXPORTER_PASSWORD.

If we want to support sidecar I would say we would probably need to have both MYSQLD_EXPORTER_PASSWORD and MYSQLD_EXPORTER_USER, since I think (or hope?) that most users have a separate user for the exporter.

My proposal if we want to have this replacement for DATA_SOURCE_NAME is that we only implement it for the /metrics endpoint and that the environment variables takes precedence if they are set.

Signed-off-by: Mattias Ängehov <mattias.angehov@castoredc.com>
Signed-off-by: Mattias Ängehov <mattias.angehov@castoredc.com>
Signed-off-by: Mattias Ängehov <mattias.angehov@castoredc.com>
Signed-off-by: Mattias Ängehov <mattias.angehov@castoredc.com>
@SuperQ
Copy link
Member

SuperQ commented Aug 25, 2022

For configuration that's not in the file, we prefer command line arguments over env vars. The only exception needed is for secrets.

So rather than MYSQLD_EXPORTER_USER, we would want --mysqld.username.

Signed-off-by: Mattias Ängehov <mattias.angehov@castoredc.com>
README.md Outdated Show resolved Hide resolved
@MattiasAng
Copy link
Contributor Author

For configuration that's not in the file, we prefer command line arguments over env vars. The only exception needed is for secrets.

So rather than MYSQLD_EXPORTER_USER, we would want --mysqld.username.

Thanks for commenting so fast.

While I understand your line of thinking, I don't fully agree with going for that approach.

If a user still needs to set an environment variable (for the password) I feel that we are better off having all of these as environment variables. The only use-case where you don't need to set an environment variable in that case is for servers that do not require a password.

So in short, I would rather keep it as-is now, but if you insist it should be quite easy to fix by bringing in the arguments to ReloadConfig.

Signed-off-by: Mattias Ängehov <mattias.angehov@castoredc.com>
Signed-off-by: Mattias Ängehov <mattias.angehov@castoredc.com>
Signed-off-by: Mattias Ängehov <mattias.angehov@castoredc.com>
@MattiasAng MattiasAng requested review from roidelapluie and SuperQ and removed request for roidelapluie August 31, 2022 14:53
@roidelapluie
Copy link
Member

LGTM

@SuperQ
Copy link
Member

SuperQ commented Sep 1, 2022

Let's do this!

@SuperQ SuperQ merged commit 593b009 into prometheus:main Sep 1, 2022
@yinshurman
Copy link

yinshurman commented Oct 14, 2022

so how to configure multiple hosts and ports ?
when i set host and port in different section of mysql.cnf , just the one in [client] makes effect.
and when i only set user and password in client like this :


[client]
user = exporter
password = prometheus
# host = 10.203.4.249
# port = 10016
# host = 172.22.0.5
# port = 3306


# [client.server1]
# user = exporter 
# password = prometheus
# host = 10.203.4.249
# port = 10003

# host = 172.22.0.11
# port = 3307


# [client.server2]
# user = exporter 
# password = prometheus
# host = 10.203.4.249
# port = 10003

# host = 172.22.0.12
# port = 3308

and configure prometheus.yml file like this:

  - job_name: "mysql" # to get metrics  about mysql 
    params:
        # not required. default `client`
        auth_module: [client]
    static_configs:
      - targets: 
        - '172.22.0.5:3306'
        - '172.22.0.11:3307'
        - '172.22.0.12:3308'
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance 
      - target_label: __address__ 
        # the mysqld exporter host:port
        replacement: 172.22.0.6:9104

the mysqld exporter reports error information like this :
ts=2022-10-14T02:05:48.270Z caller=exporter.go:149 level=error msg="Error pinging mysqld" err="dial tcp 127.0.0.1:3306: connect: connection refused"

so , it seems must set host and port in some where other than prometheus.yml , if not ,the defautl 127.0.0.1:3306 is used .

any one can make it clear where to set the different hosts and ports ?
please help ! thanks very much !

@Suomea
Copy link

Suomea commented Oct 18, 2022

@yinshurman looks like not release yet?

@MattiasAng
Copy link
Contributor Author

The documentation is missing that metrics_path needs to be changed to /probe.

I.e.

  - job_name: "mysql" # to get metrics  about mysql 
    metrics_path: /probe

@Suomea
Copy link

Suomea commented Oct 18, 2022

The documentation is missing that metrics_path needs to be changed to /probe.

I.e.

  - job_name: "mysql" # to get metrics  about mysql 
    metrics_path: /probe

I want to use multi-target to avoid starting multiple processes on machine. my situation ->
mysqld_exporter and prometheus version

root@m1:~# ls -l
total 89184
-rw-r--r-- 1 root root  7450991 Oct 18 18:03 mysqld_exporter-0.14.0.linux-amd64.tar.gz
-rw-r--r-- 1 root root 83865783 Oct 18 18:06 prometheus-2.37.1.linux-amd64.tar.gz

mysqld_exporter config

# /srv/metrics/mysqld_exporter/my.cnf
[client]
user = root
password = 123456

prometheus config

scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: "prometheus"

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
      - targets: ["localhost:9090"]

  - job_name: "mysqld_exporter"
    metrics_path: /probe
    static_configs:
      - targets:
        - 172.27.145.40:3306
        - 172.27.145.41:3306
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        # The mysqld_exporter host:port
        replacement: localhost:9104

The prb is the targets is down
image

I don't know why this is,i think maybe the exporter verions is wrong? or something else?

@MattiasAng
Copy link
Contributor Author

You need to compile it yourself, there is no release for the commits in this PR.

@shubhamsetia05
Copy link

@MattiasAng Any plans to release this (support to monitor multiple mysql DB's) in helm? We also have a use case to monitor multiple AWS RDS instances.

SuperQ added a commit that referenced this pull request Mar 14, 2023
BREAKING CHANGES:

The exporter no longer supports the monolithic `DATA_SOURCE_NAME` enviornment variable.
To configure connections to MySQL you can either use a `my.cnf` style config file or command line arguments.

For example:

    export MYSQLD_EXPORTER_PASSWORD=secret
    mysqld_exporter --mysqld.address=localhost:3306 --mysqld.username=exporter

* [CHANGE] Allow tlsCfg.InsecureSkipVerify outside of mTLS #631
* [CHANGE] Update to exporter-toolkit v0.8.1 #677
* [FEATURE] Support for multi-target mysqld probes #651

Signed-off-by: SuperQ <superq@gmail.com>
@SuperQ SuperQ mentioned this pull request Mar 14, 2023
SuperQ added a commit that referenced this pull request Apr 14, 2023
BREAKING CHANGES:

The exporter no longer supports the monolithic `DATA_SOURCE_NAME` enviornment variable.
To configure connections to MySQL you can either use a `my.cnf` style config file or command line arguments.

For example:

    export MYSQLD_EXPORTER_PASSWORD=secret
    mysqld_exporter --mysqld.address=localhost:3306 --mysqld.username=exporter

* [CHANGE] Allow tlsCfg.InsecureSkipVerify outside of mTLS #631
* [CHANGE] Update to exporter-toolkit v0.8.1 #677
* [FEATURE] Support for multi-target mysqld probes #651

Signed-off-by: SuperQ <superq@gmail.com>
SuperQ added a commit that referenced this pull request Apr 14, 2023
BREAKING CHANGES:

The exporter no longer supports the monolithic `DATA_SOURCE_NAME` environment variable.
To configure connections to MySQL you can either use a `my.cnf` style config file or command line arguments.

For example:

    export MYSQLD_EXPORTER_PASSWORD=secret
    mysqld_exporter --mysqld.address=localhost:3306 --mysqld.username=exporter

We have also dropped some internal scrape metrics:
* `mysql_exporter_scrapes_total`
* `mysql_exporter_scrape_errors_total`
* `mysql_last_scrape_failed`

Changes:

* [CHANGE] Allow tlsCfg.InsecureSkipVerify outside of mTLS #631
* [CHANGE] Update to exporter-toolkit v0.8.1 #677
* [CHANGE] Fix shared metrics between requests #722
* [FEATURE] Add support for collecting metrics from sys.user_summary #628
* [FEATURE] Support for multi-target mysqld probes #651
* [FEATURE] Add MySQL TLS configurations #718
* [BUGFIX] Fix infoSchemaInnodbMetricsEnabledColumnQuery #687

Signed-off-by: SuperQ <superq@gmail.com>
SuperQ added a commit that referenced this pull request Apr 14, 2023
BREAKING CHANGES:

The exporter no longer supports the monolithic `DATA_SOURCE_NAME` environment variable.
To configure connections to MySQL you can either use a `my.cnf` style config file or command line arguments.

For example:

    export MYSQLD_EXPORTER_PASSWORD=secret
    mysqld_exporter --mysqld.address=localhost:3306 --mysqld.username=exporter

We have also dropped some internal scrape metrics:
* `mysql_exporter_scrapes_total`
* `mysql_exporter_scrape_errors_total`
* `mysql_last_scrape_failed`

Changes:

* [CHANGE] Allow tlsCfg.InsecureSkipVerify outside of mTLS #631
* [CHANGE] Update to exporter-toolkit v0.8.1 #677
* [CHANGE] Fix shared metrics between requests #722
* [FEATURE] Add support for collecting metrics from sys.user_summary #628
* [FEATURE] Support for multi-target mysqld probes #651
* [FEATURE] Add MySQL TLS configurations #718
* [BUGFIX] Fix infoSchemaInnodbMetricsEnabledColumnQuery #687

Signed-off-by: SuperQ <superq@gmail.com>
SuperQ added a commit that referenced this pull request Jun 19, 2023
* [CHANGE] Allow `tlsCfg.InsecureSkipVerify` outside of mTLS #631
* [CHANGE] Update to exporter-toolkit v0.8.1 #677
* [CHANGE] Fix shared metrics between requests #722
* [FEATURE] Add support for collecting metrics from `sys.user_summary` #628
* [FEATURE] Support for multi-target mysqld probes #651
* [FEATURE] Add MySQL TLS configurations #718
* [ENHANCEMENT] Add UNIX domain socket support for multi-target scraping #707
* [BUGFIX] Fix `infoSchemaInnodbMetricsEnabledColumnQuery` #687
* [BUGFIX] Allow empty passwords #742

Signed-off-by: SuperQ <superq@gmail.com>
@SuperQ SuperQ mentioned this pull request Jun 19, 2023
SuperQ added a commit that referenced this pull request Jun 22, 2023
* [CHANGE] Allow `tlsCfg.InsecureSkipVerify` outside of mTLS #631
* [CHANGE] Update to exporter-toolkit v0.8.1 #677
* [CHANGE] Fix shared metrics between requests #722
* [FEATURE] Add support for collecting metrics from `sys.user_summary` #628
* [FEATURE] Support for multi-target mysqld probes #651
* [FEATURE] Add MySQL TLS configurations #718
* [ENHANCEMENT] Add UNIX domain socket support for multi-target scraping #707
* [BUGFIX] Fix `infoSchemaInnodbMetricsEnabledColumnQuery` #687
* [BUGFIX] Allow empty passwords #742

Signed-off-by: SuperQ <superq@gmail.com>
SuperQ added a commit that referenced this pull request Jun 22, 2023
BREAKING CHANGES:

The exporter no longer supports the monolithic `DATA_SOURCE_NAME` environment variable.
To configure connections to MySQL you can either use a `my.cnf` style config file or command line arguments.

For example:

    export MYSQLD_EXPORTER_PASSWORD=secret
    mysqld_exporter --mysqld.address=localhost:3306 --mysqld.username=exporter

We have also dropped some internal scrape metrics:
- `mysql_exporter_scrapes_total`
- `mysql_exporter_scrape_errors_total`
- `mysql_last_scrape_failed`

The default client configuration file is now `.my.cnf` in the process working directory. Use `--config.my-cnf="$HOME/.my.cnf"` to retain the previous default.

Changes:

* [CHANGE] Allow `tlsCfg.InsecureSkipVerify` outside of mTLS #631
* [CHANGE] Update to exporter-toolkit v0.8.1 #677
* [CHANGE] Fix shared metrics between requests #722
* [CHANGE] Allow empty passwords #742
* [CHANGE] Don't use HOME env in the my-cnf config path. #745
* [FEATURE] Add support for collecting metrics from `sys.user_summary` #628
* [FEATURE] Support for multi-target mysqld probes #651
* [FEATURE] Add MySQL TLS configurations #718
* [FEATURE] Add config reload via /-/reload #734
* [ENHANCEMENT] Add UNIX domain socket support for multi-target scraping #707
* [ENHANCEMENT] Use `STRAIGHT_JOIN` in infoSchemaAutoIncrementQuery #726
* [BUGFIX] Fix `infoSchemaInnodbMetricsEnabledColumnQuery` #687
* [BUGFIX] Allow empty passwords #742

Signed-off-by: SuperQ <superq@gmail.com>
SuperQ added a commit that referenced this pull request Jun 24, 2023
BREAKING CHANGES:

The exporter no longer supports the monolithic `DATA_SOURCE_NAME` environment variable.
To configure connections to MySQL you can either use a `my.cnf` style config file or command line arguments.

For example:

    export MYSQLD_EXPORTER_PASSWORD=secret
    mysqld_exporter --mysqld.address=localhost:3306 --mysqld.username=exporter

We have also dropped some internal scrape metrics:
- `mysql_exporter_scrapes_total`
- `mysql_exporter_scrape_errors_total`
- `mysql_last_scrape_failed`

The default client configuration file is now `.my.cnf` in the process working directory. Use `--config.my-cnf="$HOME/.my.cnf"` to retain the previous default.

Changes:

* [CHANGE] Allow `tlsCfg.InsecureSkipVerify` outside of mTLS #631
* [CHANGE] Update to exporter-toolkit v0.8.1 #677
* [CHANGE] Fix shared metrics between requests #722
* [CHANGE] Allow empty passwords #742
* [CHANGE] Don't use HOME env in the my-cnf config path. #745
* [FEATURE] Add support for collecting metrics from `sys.user_summary` #628
* [FEATURE] Support for multi-target mysqld probes #651
* [FEATURE] Add MySQL TLS configurations #718
* [FEATURE] Add config reload via /-/reload #734
* [ENHANCEMENT] Add UNIX domain socket support for multi-target scraping #707
* [ENHANCEMENT] Use `STRAIGHT_JOIN` in infoSchemaAutoIncrementQuery #726
* [BUGFIX] Fix `infoSchemaInnodbMetricsEnabledColumnQuery` #687
* [BUGFIX] Allow empty passwords #742

Signed-off-by: SuperQ <superq@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants