Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

App-local statistics counters #931

Merged
merged 143 commits into from
Jul 13, 2016
Merged

Conversation

eugeneia
Copy link
Member

@eugeneia eugeneia commented Jun 7, 2016

This branch implements per-app statistics counters and uses this feature to implement statistics for Intel_app (per pciaddress actually) and vhost_user.

The basic mechanism is that each app now has a shmpath (automatically set based on its name) to which core.shm.path is set to when its methods are executed. core.counter is updated to use core.shm.resolve, so counters opened in app:new are created under the app's shmpath. Another convention in this PR is that apps store their counters in their counters field.

doc/statistics.md describes the schema used for the counters defined by this PR. It deviates from related standards in the way it counts packets. E.g. it defines broadcast packet counts ⊆ multicast packet counts ⊆ total packet counts. I think this makes the most sense because it allows app authors to choose how fine grained their stats are. I also don't think we should buy into the ifType madness.

The SNMP code formerly in intel10g.lisp was extracted to lib.ipc.shmem.iftable_mib using the schema / API described above.

Anything in this PR is open for discussion change (obviously), but I would like to get it merged and then develop these features incrementally to get things rolling. E.g. @alexandergall has pointed out that we need more data types than counters, etc. My rationale is that with this PR merged we can start recording important statistics in other apps.

App counters (as well as link counters) can be read using snabb top -c, e.g.:

bash-4.3# program/snabbnfv/selftest.sh > nfv.log &
bash-4.3# ./snabb top -c A_Virtio
 dtime                          1,465,310,436                 
 rxbytes                        18,186,581,184                
 rxdrop                         0                             
 rxmcast                        9                             
 rxpackets                      12,012,296                    
 txbytes                        85,150,962                    
 txmcast                        0                             
 txpackets                      990,127                       
 type                           4,097                         
bash-4.3# ./snabb top -c 'A_NIC.tx -> A_Virtio.rx'
 dtime                          1,465,310,436                 
 rxbytes                        174,939,590                   
 rxpackets                      2,032,431                     
 txbytes                        174,939,590                   
 txdrop                         0                             
 txpackets                      2,032,431                     
bash-4.3# ./snabb top -c 0000:01:00.0
 dtime                          1,465,310,436                 
 macaddr                        0                             
 mtu                            9,014                         
 promisc                        1                             
 rxbcast                        0                             
 rxbytes                        0                             
 rxdrop                         0                             
 rxerrors                       0                             
 rxmcast                        0                             
 rxpackets                      0                             
 speed                          10,000,000                    
 status                         1                             
 txbcast                        0                             
 txbytes                        1,518                         
 txdrop                         0                             
 txerrors                       0                             
 txmcast                        17                            
 txpackets                      17                            
 type                           4,096                         

lukego and others added 30 commits February 22, 2016 13:35
Set the shared memory path (shm.path) to a private namespace for each
app with prefix "app/$name".

This means that apps can create shm objects such as counters and by
default these will appear in a local namespace for that app.
I rewrote the git-workflow explanation with the goal of briefly but
clearly explaining both how to submit a change to Snabb Switch and
also how to be a subsystem maintainer.

This is intended to be a chapter in the manual.
Contains "XXX rewrite" where old sections have been removed but
replacements not written yet.
This file now contains a diagram so it needs to have separate .src.md
and .md versions. This can be cleaned up when these changes eventually
merge with the on-demand markdown (snabbco#829).
also stopped capitalizing "Pull Request" because it seemed awkward.
 - Use "apps/" instead of "app/" for uniformity
 - Set shm path to "apps/$name" when calling `app:stop' too
 - Unlink "apps/$name" after `app:stop' using `shm.unlink'
 - Add a test case to core.app selftest
Resolved conflicts with incoming changes from master branch:

*.src.md replaced with simply *.md

"Snabb Switch" being renamed to "Snabb"
Ditaa images are not kept in tree anymore...
option and support injecting a function to determine the
current time.
lukego and others added 5 commits July 1, 2016 09:15
Bug was to set _G.developer_debug to nil when debugging is disabled. In
Lua you cannot use nil as a table value and should use false instead.

Further discussion at:
snabbco#956 (comment)
Print debug messages only when enabled by editing the source file, not
when _G.developer_debug is enabled.

The debug mode prints a message every time a timer is executed. This
could easily produce thousands of messages per second in reasonable
applications. This seems appropriate when specifically debugging the
timer module but not when enabling a more general debug mode.
lukego and others added 11 commits July 1, 2016 16:50
MRG_RXBUF is enabled by default in the beginning, and QEMU will initially
negotiate a feature set with Snabb NFV that includes MRG_RXBUF.  This
adds a field onto the virtio header, in legacy mode.  However if we
later negotiate to not have MRG_RXBUF, we need to re-set this value to
not have the extra fields.

(cherry picked from commit df8e83c)
…dicate."

Reason: its actually slower than the initial naive version.

This reverts commit c186591.

# Conflicts:
#	src/apps/vhost/vhost_user.lua
# Conflicts:
#	src/apps/vhost/vhost_user.lua
#	src/lib/protocol/README.md
Fixed packet documentation merge conflict
@kbara
Copy link
Contributor

kbara commented Jul 4, 2016

@eugeneia Shall I merge your newest change onto kbara-next as well?

@alexandergall
Copy link
Contributor

Can you guys please also consider eugeneia#14? Sorry, I'm losing track of who merges what from whom :)

@eugeneia
Copy link
Member Author

eugeneia commented Jul 5, 2016

@kbara No need to merge new commits from this branch, they are related to the performance regression in next.

@alexandergall Generally everything happens on the snabbco repository, e.g. opening PRs on my fork will be invisible to everyone but me. Best to close/reopen your PR on the source repository, then it will be picked up by the regular maintenance flow.

@eugeneia eugeneia merged commit f891072 into snabbco:master Jul 13, 2016
dpino added a commit to dpino/snabb that referenced this pull request Sep 14, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

8 participants