EzDevInfo.com

Including Precompiled libraries with Storm .jar files

Is there a way to include precompiled libraries like SciPi and NumPy with Apache Storm using Pyleus for .jar files so that it does not try to compile those Python libraries each time you compile the jar?

How to run pyleus on Storm

I'm a learning on Storm, I have installed zookeeper, storm, python and pyleus. the first step, I copy python script from pyleus web as a sample (https://github.com/Yelp/pyleus/tree/aaa423864f953332202832b8fd8404e03d3d74e3 ) and try to run it in storm server, the sample include below 3 files: pyleus_topology.yaml, dummy_spout.py and dummy_bolt.py the 2 py file has been put into a folder namely "my_first_topology" but when I run the pyleus build command in my VMware server(CentOS64-bit), the command can run 20 second and then, I got below error:

[root@localhost bin]# pyleus build /root/Desktop/CRM_ETL-Project-Storm/my_first_topology/pyleus_topology.yaml pyleus build: error: [VirtualenvError] Failed to execute Python module: my_first_topology.dummy_spout. Error: /tmp/tmpZMIXa3/resources/pyleus_venv/bin/python: No module named my_first_topology

what I can do for it? any steps I missed?

the script for reference 1> pyleus_topology.yaml

name: my_first_topology

topology:

- spout:
    name: my-first-spout
    module: my_first_topology.dummy_spout
- bolt:
    name: my-first-bolt
    module: my_first_topology.dummy_bolt
    groupings:
        - shuffle_grouping: my-first-spout

2> dummy_spout.py

from pyleus.storm import Spout

class DummySpout(Spout):

OUTPUT_FIELDS = ['sentence', 'name']
def next_tuple(self):
    self.emit(("This is a sentence.", "spout",))

if name == 'main': DummySpout().run()

3> dummy_bolt.py

from pyleus.storm import SimpleBolt

class DummyBolt(SimpleBolt):

OUTPUT_FIELDS = ['sentence']
def process_tuple(self, tup):
    sentence, name = tup.values
    new_sentence = "{0} says, \"{1}\"".format(name, sentence)
    self.emit((new_sentence,), anchors=[tup])

if name == 'main': DummyBolt().run()