Add pseudo-nodes for known services #1863

2opremio · 2016-09-14T07:09:16Z

2opremio · 2016-09-14T09:14:20Z

We already show pseudo nodes (with a cloud shape for now), for Dynamo and S3: the two AWS services for which we know reverse DNS lookups lead to an observable pattern.

Next step is adding a DNS packet inspector in the probes to extend it to IPs without sane revese lookups.

AWS S3 and Dynamo to start with

prog/main.go

+	// DNS Snooper
+	flag.BoolVar(&flags.probe.dnsSnooperEnabled, "probe.dns-snooper.enabled", true, "Enable DNS Snooper (to identify external services being contacted)")
+	flag.StringVar(&flags.probe.dnsSnooperInterface, "probe.dns-snooper.interface", "any", "Networking interface on which to snoop")
+


2opremio · 2016-09-19T14:51:57Z

This is ready to review. Here's an example of the precision obtained with the snooper:

Note that certain external services are not properly identified if Scope is started after persistent connections without reverse DNS lookups are made (i.e. the DNS Snooper cannot gather those queries since it wasn't running at the time they happened).

probe/endpoint/reporter.go

+	if resolvedNames, err := r.reverseResolver.get(addr); err == nil {
+		names = append(names, resolvedNames...)
+	}
+	if len(names) > 0 {


render/process.go

-		// internet node
-		node = theInternetNode(n)
+		// external pseudoNode
+		node = toInternetNode(n)


render/theinternet.go


 	"github.com/weaveworks/scope/probe/host"
 	"github.com/weaveworks/scope/report"
 )

+var (
+	// ServiceNodeIDPrefix is how the ID all service pseudo nodes begin


render/theinternet.go

+		// and having separate nodes for them makes visualizations worse
+		regexp.MustCompile(`^ec2.*\.amazonaws\.com$`),
+	}
+)


render/theinternet.go

+		// and having separate nodes for them makes visualizations worse
+		regexp.MustCompile(`^ec2.*\.amazonaws\.com$`),
+	}
+)


rade

blah

probe/endpoint/dns_snooper_linux_amd64.go

+		return nil, err
+	}
+	defer inactive.CleanUp()
+	if err = inactive.SetPromisc(true); err != nil {


2opremio · 2016-09-20T09:04:45Z

@rade PTAL

rade · 2016-09-20T16:01:04Z

needs some squashing. rest LGTM

...if available. Follow-up to #1863.

2opremio force-pushed the 1082-nodes-for-known-services branch 3 times, most recently from 0fe7157 to 31497c4 Compare September 14, 2016 09:10

2opremio force-pushed the 1082-nodes-for-known-services branch 10 times, most recently from 97e0ac7 to 268651b Compare September 16, 2016 14:12

rade mentioned this pull request Sep 19, 2016

show more details of a node's internet connections #1875

Merged

Alfonso Acosta added 4 commits September 19, 2016 14:38

Add pseudo-nodes for known services

5c080ec

AWS S3 and Dynamo to start with

Implement DNS snooper

0223b0b

Simplify service node matching

028ed32

Minor fixes/refinements

f5ad6a8

2opremio force-pushed the 1082-nodes-for-known-services branch from f512a03 to d47db56 Compare September 19, 2016 14:39

rade reviewed Sep 19, 2016

View reviewed changes

Exclude ec2 since it's too generic

193bfec

2opremio force-pushed the 1082-nodes-for-known-services branch from d47db56 to b351db7 Compare September 19, 2016 14:45

2opremio changed the title ~~[WIP] Add pseudo-nodes for known services~~ Add pseudo-nodes for known services Sep 19, 2016

2opremio assigned rade Sep 19, 2016

rade reviewed Sep 19, 2016

View reviewed changes

rade suggested changes Sep 19, 2016

View reviewed changes

rade reviewed Sep 19, 2016

View reviewed changes

probe/endpoint/dns_snooper_linux_amd64.go

return nil, err

}

defer inactive.CleanUp()

if err = inactive.SetPromisc(true); err != nil {

This comment was marked as abuse.

Sign in to view

This comment was marked as abuse.

Sign in to view

This comment was marked as abuse.

Sign in to view

2opremio force-pushed the 1082-nodes-for-known-services branch 2 times, most recently from 21520e4 to c24f502 Compare September 20, 2016 09:04

2opremio force-pushed the 1082-nodes-for-known-services branch from c24f502 to 94dc9cb Compare September 20, 2016 09:07

2opremio mentioned this pull request Sep 20, 2016

Customizable external services #1876

Open

2opremio force-pushed the 1082-nodes-for-known-services branch 2 times, most recently from 62442d1 to b15cce0 Compare September 20, 2016 10:14

rade approved these changes Sep 20, 2016

View reviewed changes

rade assigned 2opremio and unassigned rade Sep 20, 2016

Review feedback

c5ac315

2opremio force-pushed the 1082-nodes-for-known-services branch from b15cce0 to c5ac315 Compare September 20, 2016 16:50

2opremio merged commit c64dfa6 into master Sep 20, 2016

2opremio deleted the 1082-nodes-for-known-services branch September 20, 2016 21:00

rade added a commit that referenced this pull request Sep 22, 2016

show forward DNS name for internet addresses

270799a

...if available. Follow-up to #1863.

rade added a commit that referenced this pull request Sep 22, 2016

show forward DNS name for internet addresses

2da6dee

...if available. Follow-up to #1863.

rade mentioned this pull request Sep 22, 2016

show forward DNS name for internet addresses #1881

Merged

rade added a commit that referenced this pull request Sep 22, 2016

show forward DNS name for internet addresses

5ce5d54

...if available. Follow-up to #1863.

2opremio mentioned this pull request Sep 26, 2016

Per user/organization persistent configuration weaveworks/service#882

Closed

rade mentioned this pull request Oct 18, 2016

scope-probe appears to have connection to everything #1929

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add pseudo-nodes for known services #1863

Add pseudo-nodes for known services #1863

2opremio commented Sep 14, 2016 •

edited by rade

Loading

2opremio commented Sep 14, 2016 •

edited

Loading

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.

2opremio commented Sep 19, 2016 •

edited

Loading

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.

rade left a comment

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.

2opremio commented Sep 20, 2016

rade commented Sep 20, 2016

Add pseudo-nodes for known services #1863

Add pseudo-nodes for known services #1863

Conversation

2opremio commented Sep 14, 2016 • edited by rade Loading

2opremio commented Sep 14, 2016 • edited Loading

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.

2opremio commented Sep 19, 2016 • edited Loading

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.

rade left a comment

Choose a reason for hiding this comment

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.

2opremio commented Sep 20, 2016

rade commented Sep 20, 2016

2opremio commented Sep 14, 2016 •

edited by rade

Loading

2opremio commented Sep 14, 2016 •

edited

Loading

2opremio commented Sep 19, 2016 •

edited

Loading