lookup inside an alert not working

[ayashjorden]

Hi,
The scenario is as follows:

1. in Rule Editor page, add lookup object and alert using the lookup.
2. test the newly added alert.

result: empty result in 'Result' view

[gbrayut]

Try using a hard coded LookupAll to see if it is the tag matching that is failing:

warn = $srv_doc_count < LookupAll(“min_doc_per_host”, “warn_min”, “host=mail-work-test”)

[kylebrandt]

Lookupall is only a template func
On Jun 4, 2015 12:32 PM, "Greg Bray" notifications@github.com wrote:

Try using a hard coded LookupAll to see if it is the tag matching that is
failing:

warn = $srv_doc_count < LookupAll(“min_doc_per_host”, “warn_min”,
“host=mail-work-test”)

—
Reply to this email directly or view it on GitHub
#1035 (comment)
.

[gbrayut]

Differences between expression functions and template functions makes me a sad panda.

[yinchuan]

Get same result with @ayashjorden.
with env: CentOS 6.6 x64, bosun-linux-amd64 v0.3.0, chage to graphite backend, use config as below:

graphiteHost = http://192.168.2.53/render/
stateFile = /data/bosun.state

template test {
    subject = test subject
    body = test body
}

lookup test {
    entry host=* {
        crit = 10
        warn = 8
    }
}

alert test {
    template = test
    warn = avg(graphite("server1.load", "1h", "", "host.metric")) > lookup("test", "warn")
    crit = avg(graphite("server1.load", "1h", "", "host.metric")) > 10
}

when test alert test at Rule Editor, in History the crit is right, but get "Warn": null.

[yinchuan]

by the way, use LookupAll in template can get write value.

[yinchuan]

@ayashjorden Hello, any idea about this?

[ayashjorden]

Hi @yinchuan,
It will take me few more days to get back to this, sorry that I don't have more helpful answer.

Yarden

[yinchuan]

@mjibson Hello, any idea about this?

[cheribral]

I ran into this as well. Working with the forecasting example for disk space, I changed it to use some static numbers, and put in an extra variable to see what was happening.

$warn_percent = $percent_free <  80
$crit_percent = $percent_free <  90
$blah = 0 < lookup("disk_space", "crit_percent_free")

I set the notification to print, and put all three in the subject line. It is printing $blah as 0 < lookup("disk_space", "crit_percent_free") literally as if it were a string rather than evaluating it. All the other variables are printed correctly as the result of the expression.

[kylebrandt]

I'm think that this is because we are using search, so unless you are on OpenTSDB and you are sending datapoint to /index (or relayig through bosun) search would have no results:

bosun/cmd/bosun/conf/lookup.go

Line 28 in 97df011

matches, err := search.Match(av, []string{tag[ak]})

[kylebrandt]

That func I pointed at doesn't use search data, this one does though: https://github.com/bosun-monitor/bosun/blob/master/cmd/bosun/conf/conf.go#L1209

[ayashjorden]

@kylebrandt
I understand that Lookups works only when using OpenTSDB?

I though I can use the Lookup structure to write generic configuration. meaning using a lookup in alerts.

[captncraig]

Correct, the lookup function as is relies on search to find tag keys and values. The problem is lookup is really a standalone expression. It is given no context of what tags exist or that to do. I can think of two real solutions to this:

1. Create a lookupC() function that accepts an expression to use as the context for it's lookup.

   I think this would lead to a lot of things like

   $foo = q(....)
   crit = $foo > lookupC($foo, "lookup_name", "field")
   

   which feels a littel cumbersome, but would be explicit.

2. Attempt to infer the context from the expression. If we assume/enforce that a lookup is only valid inside a binary operation, we can get the tags from the non-lookup side and pass them into the lookup as context.

   Example:

   $foo = q(...)
   crit = $foo > lookup("lookup_name","field")
   

   We can evaluate the query, pull out the unique tag sets and pass that as the context to the lookup. The lookup can then create one result per source tagset and the results will union perfectly. This should also be nice as it won't need to query anything additional outside of the current expression. That logic would all need to be somehow done in walkBinary though, so lookups may not be valid elsewhere.

[kylebrandt]

I think this is correct. I can't think of lookups having any meaning outside of this case, except maybe if you wanted to pass the value of a lookup to a function. I don't think we do that anywhere currently.

[kylebrandt]

Digging into this, I found out there is already a function that works around this that wasn't documented:

        "lookupSeries": {
            Args:   []models.FuncType{models.TypeSeriesSet, models.TypeString, models.TypeString},
            Return: models.TypeNumberSet,
            Tags:   lookupSeriesTags,
            F:      lookupSeries,
        },

So all you have to do is pass your series into the function:

lookup test {
    entry host=ny-bosun01 {
        value = 1
    }
    entry host=* {
        value = 2
    }
}

template test {
    subject = {{.Last.Status}}: {{.Alert.Name}} on {{.Group.host}}
    body = `<p>Name: {{.Alert.Name}}
    <p>Tags:
    <table>
        {{range $k, $v := .Group}}
            <tr><td>{{$k}}</td><td>{{$v}}</td></tr>
        {{end}}
    </table>`
}

alert test {
    template = test
    $q = q("avg:rate:os.cpu{host=*bosun*}", "5m", "")
    crit = avg($q) > lookupSeries($q, "test", "value")
}

[angadsingh]

would be great to have this added to the documentation. took us a while to reach this thread.