Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[telemetry] Call sonic-cfggen Once #4901

Merged
merged 5 commits into from
Jul 8, 2020
Merged

[telemetry] Call sonic-cfggen Once #4901

merged 5 commits into from
Jul 8, 2020

Conversation

tahmed-dev
Copy link
Contributor

@tahmed-dev tahmed-dev commented Jul 6, 2020 •
edited
Loading

sonic-cfggen call is slow and this is taking place in the SONiC
boot up process. The change uses templates to assemble all required
vars into single template file. With this change, telemetry now callse
once into sonic-cfggen.

signed-off-by: Tamer Ahmed tamer.ahmed@microsoft.com

- Why I did it
Reduce CPU time consumed during SONiC boot

- How I did it
New code that uses templates and jq. The new code calls into SONiC cfggen once

- How to verify it
- 1. script timing

root@str-s6000-acs-14:/# time /usr/bin/telemetry_old.sh
/usr/sbin/telemetry  -logtostderr --server_crt /etc/sonic/telemetry/streamingtelemetryserver.cer --server_key /etc/sonic/telemetry/streamingtelemetryserver.key  --ca_crt /etc/sonic/telemetry/dsmsroot.cer --port 8080 -v=2

real	0m12.995s
user	0m9.384s
sys	0m2.362s
root@str-s6000-acs-14:/# time /usr/bin/telemetry_new.sh
/usr/sbin/telemetry  -logtostderr --server_crt /etc/sonic/telemetry/streamingtelemetryserver.cer --server_key /etc/sonic/telemetry/streamingtelemetryserver.key  --ca_crt /etc/sonic/telemetry/dsmsroot.cer --port 8080 -v=2

real	0m1.885s
user	0m1.408s
sys	0m0.397s

- 2. tested on target and telemetry docker was loaded and the binary is running.

- Description for the changelog

- A picture of a cute animal (not mandatory but encouraged)

@tahmed-dev tahmed-dev marked this pull request as ready for review July 6, 2020 17:23
@tahmed-dev
[telemetry] Call sonic-cfggen Once
099f693 
sonic-cfggen call is slow and this is taking place in the SONiC
boot up process. The change uses templates to assemble all required
vars into single template file. With this change, telemetry now callse
once into sonic-cfggen.

signed-off-by: Tamer Ahmed <tamer.ahmed@microsoft.com>
@tahmed-dev tahmed-dev force-pushed the taahme/telemetry-calls-sonic-cfggen-once branch from 93c67f5 to 099f693 Compare July 6, 2020 18:17
pavel-shirshov
pavel-shirshov previously approved these changes Jul 6, 2020
View reviewed changes
Copy link
Contributor

@pavel-shirshov pavel-shirshov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Look good. Thank you!

dockers/docker-sonic-telemetry/telemetry.sh Outdated Show resolved Hide resolved
lguohan
lguohan previously approved these changes Jul 6, 2020
View reviewed changes
@qiluo-msft qiluo-msft requested a review from hui-ma July 6, 2020 19:56
@hui-ma hui-ma requested a review from pra-moh July 6, 2020 20:04
qiluo-msft
qiluo-msft previously approved these changes Jul 6, 2020
View reviewed changes
yxieca
yxieca previously approved these changes Jul 6, 2020
View reviewed changes
jleveque
jleveque previously approved these changes Jul 6, 2020
View reviewed changes
@jleveque
Copy link
Contributor

jleveque commented Jul 6, 2020

Retest mellanox please

pavel-shirshov
pavel-shirshov previously approved these changes Jul 6, 2020
View reviewed changes
@tahmed-dev tahmed-dev force-pushed the taahme/telemetry-calls-sonic-cfggen-once branch from 96a607c to 970c445 Compare July 6, 2020 23:16
@tahmed-dev tahmed-dev requested a review from hui-ma July 6, 2020 23:23
lguohan
lguohan previously approved these changes Jul 7, 2020
View reviewed changes
hui-ma
hui-ma previously approved these changes Jul 7, 2020
View reviewed changes
pra-moh
pra-moh previously approved these changes Jul 7, 2020
View reviewed changes
@tahmed-dev tahmed-dev dismissed stale reviews from pra-moh and hui-ma via 0723fee July 7, 2020 21:55
@tahmed-dev tahmed-dev requested a review from jleveque July 7, 2020 22:19
@tahmed-dev tahmed-dev merged commit 9ad368b into sonic-net:master Jul 8, 2020
abdosi pushed a commit that referenced this pull request Jul 11, 2020
@tahmed-dev @abdosi
[telemetry] Call sonic-cfggen Once (#4901)
1bb3918 
sonic-cfggen call is slow and this is taking place in the SONiC
boot up process. The change uses templates to assemble all required
vars into single template file. With this change, telemetry now calls
once into sonic-cfggen.

signed-off-by: Tamer Ahmed <tamer.ahmed@microsoft.com>
bbinxie added a commit to SONIC-DEV/sonic-buildimage that referenced this pull request Jul 12, 2020
@bbinxie @abdosi
@gracelicd @arlakshm @stephenxs @tahmed-dev @yxieca @jleveque @prsunny
201911 (#199)
a1243c3 
* [brcmsai]: Updated BRCM SAI Debina package to 3.7.5.1-2 (sonic-net#4916)

Fix for Copp Rules not having Policer Rate-Limit applied.

Signed-off-by: Abhishek Dosi <abdosi@microsoft.com>

* [nephos]: upgrade Nephos SAI version to c749df (sonic-net#4814)

Verified with Nephos nps8365 based platform Accton AS7116-54x.

* "[config]: Multi ASIC loopback changes (sonic-net#4895)

Resubmitting the changes for (sonic-net#4825) with fixes for sonic-bgpcdgd test failures
Signed-off-by: Arvindsrinivasan Lakshmi Narasimhan <arlakshm@microsoft.com>

* [mellanox]: Support warm reboot on MSN4700 (sonic-net#4910)

* [Submodule Update] Sonic-platform-common
[sfp_base] Update return value documentation of channel-specific methods
(#98)
[SfpBase] Fix key name typo in docstring (#99)
[sfp] Tweak key names of some transceiver info fields (#97)
[sfputil] Make SfpUtilHelper.get_physical_to_logical noexcept as
in SfpUtilBase (#96)

* [Submodule update] sonic-platform-daemons
[xcvrd] Update key names in 'get_media_settings_value()' (#63)
[xcvrd] Tweak some transceiver info key names (#62)

* [Submodule update] sonic-utilities
[sfpshow][mock_state_db] Tweak key names of some transceiver info fields
(sonic-net#958)
[config] Fix syntax error (sonic-net#966)
[config] Fix indentation level in _get_disabled_services_list() (sonic-net#965)

* [Submodule Update] sonic-swss
[aclorch] Use IPv6 Next Header internally for protocol number on MLNX
platform (sonic-net#1343)

* [Submodule Update]
 Add support for attribute capability query in lua script (sonic-net#362)

* Cherry-pick was not clean. Fixing it.
Signed-off-by: Abhishek Dosi <abdosi@microsoft.com>

* [telemetry] Call sonic-cfggen Once (sonic-net#4901)

sonic-cfggen call is slow and this is taking place in the SONiC
boot up process. The change uses templates to assemble all required
vars into single template file. With this change, telemetry now calls
once into sonic-cfggen.

signed-off-by: Tamer Ahmed <tamer.ahmed@microsoft.com>

* [mgmt docker] move pycryptodome installation to the end of the docker building (sonic-net#4917)

* [mgmt docker] move pycryptodome installation to the end of the docker building

Signed-off-by: Ying Xie <ying.xie@microsoft.com>

* pin down the version to current: 3.9.8

* comment

* Add support for bcmsh and bcmcmd utlitites in multi ASIC devices (sonic-net#4926)

Signed-off-by: Arvindsrinivasan Lakshmi Narasimhan <arlakshm@microsoft.com>
This PR has changes to support accessing the bcmsh and bcmcmd utilities on multi ASIC devices
Changes done
- move the link of /var/run/sswsyncd from docker-syncd-brcm.mk to docker_image_ctl.j2
- update the bcmsh and bcmcmd scripts to take -n [ASIC_ID] as an argument on multi ASIC platforms

* [caclmgrd] Improve code reuse (sonic-net#4931)

Improve code reuse in `generate_block_ip2me_traffic_iptables_commands()` function.

* [Submodule Update] sonic-utilities
Intf table migration for APP_DB entries during warmboot (sonic-net#980)
[Multi NPU] Time Improvements to the config reload/load_minigraph
commands  (sonic-net#917)

* [Submodule Update] sonic-py-swssdk
 [MultiDB]: use python class composition to avoid confusion in base
class (#74)

* [Submodule update] sonic-snmpagent. Movent to 201911 Branch with with
following PR's :
Implement cbgpPeer2State in CiscoBgp4MIB (#119)
Fix index nodes in LLDP tables whose access right is not-accessible.
(#112)
 Fix quagga/FRR parser on IPv6 BGP sessions (#122)
 [lint] Fix some syntax errors or warnings (#127)
  Update README.md: Add lgtm badges (#128)
  [Multi-asic]: Support multi-asic platform (#126)
  Simplify test code (#132)
  [Multi-asic]: Namespace support for LLDP and Sensor tables (#131)
  Fix undefined variable and warning message (#134)
  Fix SNMP AgentX socket connection timeout when using
  Namespace.get_all() (#140)
  [Namespace] Fix interfaces counters in InterfacesMIB RFC 2863 (#141)
   Fix LGTM reported alert of PR#141 (#142)

* [bgpcfgd] - Fix a key error during delete (sonic-net#4946)

Co-authored-by: abdosi <58047199+abdosi@users.noreply.github.com>
Co-authored-by: gracelicd <39251567+gracelicd@users.noreply.github.com>
Co-authored-by: arlakshm <55814491+arlakshm@users.noreply.github.com>
Co-authored-by: Stephen Sun <5379172+stephenxs@users.noreply.github.com>
Co-authored-by: Abhishek Dosi <abdosi@microsoft.com>
Co-authored-by: Tamer Ahmed <tamer.ahmed@microsoft.com>
Co-authored-by: Ying Xie <yxieca@users.noreply.github.com>
Co-authored-by: Joe LeVeque <jleveque@users.noreply.github.com>
Co-authored-by: Prince Sunny <prince.sunny@microsoft.com>
qiluo-msft pushed a commit to qiluo-msft/sonic-buildimage that referenced this pull request Jul 12, 2020
@tahmed-dev @qiluo-msft
[telemetry] Call sonic-cfggen Once (sonic-net#4901)
f4eae5d 
sonic-cfggen call is slow and this is taking place in the SONiC
boot up process. The change uses templates to assemble all required
vars into single template file. With this change, telemetry now calls
once into sonic-cfggen.

signed-off-by: Tamer Ahmed <tamer.ahmed@microsoft.com>
bbinxie added a commit to SONIC-DEV/sonic-buildimage that referenced this pull request Jul 22, 2020
@bbinxie @abdosi
@gracelicd @arlakshm @stephenxs @tahmed-dev @yxieca @jleveque @prsunny @daall @Staphylo
update from azure201911 (#203)
c978449 
* [brcmsai]: Updated BRCM SAI Debina package to 3.7.5.1-2 (sonic-net#4916)

Fix for Copp Rules not having Policer Rate-Limit applied.

Signed-off-by: Abhishek Dosi <abdosi@microsoft.com>

* [nephos]: upgrade Nephos SAI version to c749df (sonic-net#4814)

Verified with Nephos nps8365 based platform Accton AS7116-54x.

* "[config]: Multi ASIC loopback changes (sonic-net#4895)

Resubmitting the changes for (sonic-net#4825) with fixes for sonic-bgpcdgd test failures
Signed-off-by: Arvindsrinivasan Lakshmi Narasimhan <arlakshm@microsoft.com>

* [mellanox]: Support warm reboot on MSN4700 (sonic-net#4910)

* [Submodule Update] Sonic-platform-common
[sfp_base] Update return value documentation of channel-specific methods
(#98)
[SfpBase] Fix key name typo in docstring (#99)
[sfp] Tweak key names of some transceiver info fields (#97)
[sfputil] Make SfpUtilHelper.get_physical_to_logical noexcept as
in SfpUtilBase (#96)

* [Submodule update] sonic-platform-daemons
[xcvrd] Update key names in 'get_media_settings_value()' (#63)
[xcvrd] Tweak some transceiver info key names (#62)

* [Submodule update] sonic-utilities
[sfpshow][mock_state_db] Tweak key names of some transceiver info fields
(sonic-net#958)
[config] Fix syntax error (sonic-net#966)
[config] Fix indentation level in _get_disabled_services_list() (sonic-net#965)

* [Submodule Update] sonic-swss
[aclorch] Use IPv6 Next Header internally for protocol number on MLNX
platform (sonic-net#1343)

* [Submodule Update]
 Add support for attribute capability query in lua script (sonic-net#362)

* Cherry-pick was not clean. Fixing it.
Signed-off-by: Abhishek Dosi <abdosi@microsoft.com>

* [telemetry] Call sonic-cfggen Once (sonic-net#4901)

sonic-cfggen call is slow and this is taking place in the SONiC
boot up process. The change uses templates to assemble all required
vars into single template file. With this change, telemetry now calls
once into sonic-cfggen.

signed-off-by: Tamer Ahmed <tamer.ahmed@microsoft.com>

* [mgmt docker] move pycryptodome installation to the end of the docker building (sonic-net#4917)

* [mgmt docker] move pycryptodome installation to the end of the docker building

Signed-off-by: Ying Xie <ying.xie@microsoft.com>

* pin down the version to current: 3.9.8

* comment

* Add support for bcmsh and bcmcmd utlitites in multi ASIC devices (sonic-net#4926)

Signed-off-by: Arvindsrinivasan Lakshmi Narasimhan <arlakshm@microsoft.com>
This PR has changes to support accessing the bcmsh and bcmcmd utilities on multi ASIC devices
Changes done
- move the link of /var/run/sswsyncd from docker-syncd-brcm.mk to docker_image_ctl.j2
- update the bcmsh and bcmcmd scripts to take -n [ASIC_ID] as an argument on multi ASIC platforms

* [caclmgrd] Improve code reuse (sonic-net#4931)

Improve code reuse in `generate_block_ip2me_traffic_iptables_commands()` function.

* [Submodule Update] sonic-utilities
Intf table migration for APP_DB entries during warmboot (sonic-net#980)
[Multi NPU] Time Improvements to the config reload/load_minigraph
commands  (sonic-net#917)

* [Submodule Update] sonic-py-swssdk
 [MultiDB]: use python class composition to avoid confusion in base
class (#74)

* [Submodule update] sonic-snmpagent. Movent to 201911 Branch with with
following PR's :
Implement cbgpPeer2State in CiscoBgp4MIB (#119)
Fix index nodes in LLDP tables whose access right is not-accessible.
(#112)
 Fix quagga/FRR parser on IPv6 BGP sessions (#122)
 [lint] Fix some syntax errors or warnings (#127)
  Update README.md: Add lgtm badges (#128)
  [Multi-asic]: Support multi-asic platform (#126)
  Simplify test code (#132)
  [Multi-asic]: Namespace support for LLDP and Sensor tables (#131)
  Fix undefined variable and warning message (#134)
  Fix SNMP AgentX socket connection timeout when using
  Namespace.get_all() (#140)
  [Namespace] Fix interfaces counters in InterfacesMIB RFC 2863 (#141)
   Fix LGTM reported alert of PR#141 (#142)

* [bgpcfgd] - Fix a key error during delete (sonic-net#4946)

* [Submodule Update] sonic-utilities
 Fix the None Type Exception when Interface Table does not exist (cold
boot) as part of db migration (sonic-net#986)

* Fix the below frr start.sh jija2 exception in 201911 image syslog: (sonic-net#4958)

File "/usr/local/bin/sonic-cfggen", line 380, in <module>
     main()
   File "/usr/local/bin/sonic-cfggen", line 354, in main
     print(template.render(data))
   File "/usr/local/lib/python2.7/dist-packages/jinja2/environment.py", line 1090, in render
     self.environment.handle_exception()
   File "/usr/local/lib/python2.7/dist-packages/jinja2/environment.py", line 832, in handle_exception
     reraise(*rewrite_traceback_stack(source=source))
   File "<template>", line 1, in top-level template code
   File "/usr/local/lib/python2.7/dist-packages/jinja2/environment.py", line 471, in getattr
     return getattr(obj, attribute)
 jinja2.exceptions.UndefinedError: 'WARM_RESTART' is undefined

Signed-off-by: Abhishek Dosi <abdosi@microsoft.com>

* [Submodule update] sonic-snmpagent
 [201911] Fix interface counters in RFC1213 (#144)

* [docker-ptf] Add support for spytest to ptf container (sonic-net#4410)

- Install apt and pip dependencies
- Define traffic generator service

Signed-off-by: Danny Allen <daall@microsoft.com>

* [arista] update Arista drivers submodules (sonic-net#4967)

Merge most of the changes that recently made it to master.
This will be the last such merge operation and future commits will only cherry-pick fixes and targeted features.

Major fixes and features,
- reboot cause enhancement with more hardware reboot cause reporting
- fix reboot cause parsing issue with 201811 release
- fix get_change_event logic
- fix error message on missing sysfs entry by our plugins
- final piece of the platform refactors for fan and sensor reporting through the platform API

* [201911][devices] Update SFP keys to align with new standard (sonic-net#4976)

Align SFP key names with new standard defined in sonic-net/sonic-platform-common#97

- hardwarerev -> hardware_rev
- serialnum -> serial
- manufacturename -> manufacturer
- modelname -> model
- Connector -> connector

* [201911][sonic-telemetry] Update submodule (sonic-net#4987)

Point submodule to new 201911 branch of sonic-telemetry and update pointer to the current HEAD of the 201911 branch

* src/sonic-telemetry aaa9188...01b5365 (1):
  > [testdata] Update SFP keys to align with new standard (#39)

* [201911][sudoers] Add `sonic_installer list` to read-only commands (sonic-net#4997)

`sonic_installer list` is a read-only command. Specify it as such in the sudoers file.

This will also ensure the new `show boot` command, which calls `sudo sonic_installer list` under the hood doesn't fail due to permissions.

* [caclmgrd] Filter DHCP packets based on dest port only (sonic-net#4995)

Co-authored-by: abdosi <58047199+abdosi@users.noreply.github.com>
Co-authored-by: gracelicd <39251567+gracelicd@users.noreply.github.com>
Co-authored-by: arlakshm <55814491+arlakshm@users.noreply.github.com>
Co-authored-by: Stephen Sun <5379172+stephenxs@users.noreply.github.com>
Co-authored-by: Abhishek Dosi <abdosi@microsoft.com>
Co-authored-by: Tamer Ahmed <tamer.ahmed@microsoft.com>
Co-authored-by: Ying Xie <yxieca@users.noreply.github.com>
Co-authored-by: Joe LeVeque <jleveque@users.noreply.github.com>
Co-authored-by: Prince Sunny <prince.sunny@microsoft.com>
Co-authored-by: Danny Allen <daall@microsoft.com>
Co-authored-by: Samuel Angebault <staphylo@arista.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@pra-moh pra-moh pra-moh approved these changes

@hui-ma hui-ma hui-ma approved these changes

@jleveque jleveque jleveque approved these changes

@lguohan lguohan lguohan approved these changes

@yxieca yxieca Awaiting requested review from yxieca

@pavel-shirshov pavel-shirshov Awaiting requested review from pavel-shirshov

@qiluo-msft qiluo-msft Awaiting requested review from qiluo-msft qiluo-msft is a code owner

Assignees
No one assigned
Labels
Included in 201911 Branch Request for 201911 Branch :shipit: Request for 202006 Branch
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
9 participants
@tahmed-dev @jleveque @lguohan @qiluo-msft @pavel-shirshov @yxieca @hui-ma @pra-moh @abdosi