3.0.0.dev3 Bolt.spec(config={}) is not passed to storm_conf when using sparse run

[Darkless012]

class WordCount(Topology):
    word_spout = WordSpout.spec(
        config={"foo":"bar"}
    )

class WordSpout(Bolt):
    def initialize(self, storm_conf, context):
        foo_var = storm_conf.get('foo')
        ...

foo_var is None.

How did you pass config to storm in your previous code (2.x.x) ShellBolt?

[dan-blanchard]

Hmm... weird. I swear we do this in production at Parse.ly, but maybe we're only reading config options passed via the --option foo=bar arguments to sparse submit.

[Darkless012]

I've made additional change into FluxStormBolt/Spout so that there is a function, that appends configuration to storm config. But so far it accepts only strings or list of string. It has trouble with dictionaries, since Flux can't convert yaml dict to Java whatever automatically.

The custom java classes, can be also included in project and set in project.clj, which mean I was able to "override" Flux default bolt with my behaviour. Which could be handy, even for Streamparse. Consider having custom Wrapper Bolt/Spout, so you can actually depend on Flux only on converting YAML to topology, and not on having proper "ShellBolt".

[macheins]

Is there any progress on this issue? Are there alternative ways to instantiate Spouts/Bolts within a Topology definition with custom (non-global) parameters?

@Darkless012 is this change published somewhere?

[Darkless012]

I've made issue at Apache Storm https://issues.apache.org/jira/browse/STORM-1870
But it is not finished, since it doesn't support complete functionality (Passing dictionaries) + it throws WARNINGS.

I've also made pullrequest #266 whenever the flux will be ready.

---

However if you really need this, I was able to include custom java classes inside streamparse project, with this functionality. If you want to know more, please write me and email and we can discuss it further, but it includes custom version of streamparse code + java libraries inside project, which is a lot of fuss for "standard" user. However it is doable, since we need it in production.

[dan-blanchard]

@Darkless012, this issue only affects sparse run, correct? That must be why I haven't encountered it before, because I do most of my testing using this Storm VM.

[Darkless012]

Well it affects also sparse submit because it creates yaml and from that yaml topology is created.
And the yaml is not passing a thing. So the bolt (spout) doesn't know anything about bolt(spout)-specific config.

[therachelbentley]

Any updates on this?

[Darkless012]

Well, still waiting for upstream/Flux changes in Apache Storm.
See https://issues.apache.org/jira/browse/STORM-1870.

[therachelbentley]

Could this also be related to setting the parallelism? If I try setting it to 4, it still stays set at 1.

[Darkless012]

Do you mean parallelism of bolts and spouts? It is set in topologies/<name_of_topology>.py file, like here: http://streamparse.readthedocs.io/en/master/topologies.html#simple-python-example

And it should work.
If not, please describe your problem more as a new issue ;)

[dan-blanchard]

One thing I should note is that you can now specify a Topology.config attribute with settings you want to apply to your whole topology, and that appears to work fine with sparse run. Component-specific settings are still waiting on upstream.

[Darkless012]

https://issues.apache.org/jira/browse/STORM-1870 - Patch is ready for reviewing.
It will probably be in next release of Storm :) whoo!

[HeartSaVioR]

Sorry to be late. STORM-1870 is merged and will be released to 1.1.0.

[goldiiman]

hi

I have the config passing issue, to get the fix do I need to update Storm or stremparse or both?

I have storm 1.0.2 and sparse 3.1.1.

thanks
martin

[dan-blanchard]

Once storm 1.1.0 is out, you'll just need to upgrade to that. Streamparse should not need to be upgraded.

[dan-blanchard]

Will be fixed in 3.5.0 release coming out later today.