EzDevInfo.com

zookeeper lock stayed locked

I am using celery and zookeeper (kazoo lock) to lock my workers. I have a problem when I kill (-9) one of the workers before releasing the lock then that lock stays locked forever.

So my question is: Does killing the process release locks in that process or is this some bug in zookeeper?

How to use Python kazoo library?

I am planning to use Python kazoo library for Zookeeper. It's all about Python question here not zookeeper at all I guess meaning how to use Python kazoo properly..

I am totally new to python so I have no idea how to get going and how to use kazoo to connect with zookeeper.

This is the document I was reading to start using kazoo for Zookeeper.

http://kazoo.readthedocs.org/en/latest/install.html

In that wiki, they have asked to install kazoo. And they are using some pip command for that?

What does pip do here? And I am currently using windows so I have cygwin installed and python installed as well. I am using Python 2.7.3

host@D-SJC-00542612 ~
$ python
Python 2.7.3 (default, Dec 18 2012, 13:50:09)
[GCC 4.5.3] on cygwin

Now what I did is - I copied this command exactly as it is from the above website - pip install kazoo and ran it on my cygwin command prompt.

host@D-SJC-00542612 ~
$ pip install kazoo
Downloading/unpacking kazoo
  Running setup.py egg_info for package kazoo

    warning: no previously-included files found matching '.gitignore'
    warning: no previously-included files found matching '.travis.yml'
    warning: no previously-included files found matching 'Makefile'
    warning: no previously-included files found matching 'run_failure.py'
    warning: no previously-included files matching '*' found under directory 'sw'
    warning: no previously-included files matching '*pyc' found anywhere in distribution
    warning: no previously-included files matching '*pyo' found anywhere in distribution
Downloading/unpacking zope.interface>=3.8.0 (from kazoo)
  Running setup.py egg_info for package zope.interface

    warning: no previously-included files matching '*.dll' found anywhere in distribution
    warning: no previously-included files matching '*.pyc' found anywhere in distribution
    warning: no previously-included files matching '*.pyo' found anywhere in distribution
    warning: no previously-included files matching '*.so' found anywhere in distribution
Requirement already satisfied (use --upgrade to upgrade): distribute in c:\python27\lib\site-packages (from zope.interface>=3.8.0->kazoo)
Installing collected packages: kazoo, zope.interface
  Running setup.py install for kazoo

    warning: no previously-included files found matching '.gitignore'
    warning: no previously-included files found matching '.travis.yml'
    warning: no previously-included files found matching 'Makefile'
    warning: no previously-included files found matching 'run_failure.py'
    warning: no previously-included files matching '*' found under directory 'sw'
    warning: no previously-included files matching '*pyc' found anywhere in distribution
    warning: no previously-included files matching '*pyo' found anywhere in distribution
  Running setup.py install for zope.interface

    warning: no previously-included files matching '*.dll' found anywhere in distribution
    warning: no previously-included files matching '*.pyc' found anywhere in distribution
    warning: no previously-included files matching '*.pyo' found anywhere in distribution
    warning: no previously-included files matching '*.so' found anywhere in distribution
    building 'zope.interface._zope_interface_coptimizations' extension
    ********************************************************************************
    WARNING:

            An optional code optimization (C extension) could not be compiled.

            Optimizations for this package will not be available!
    ()
    Unable to find vcvarsall.bat
    ********************************************************************************
    Skipping installation of C:\Python27\Lib\site-packages\zope\__init__.py (namespace package)
    Installing C:\Python27\Lib\site-packages\zope.interface-4.0.5-py2.7-nspkg.pth
Successfully installed kazoo zope.interface
Cleaning up...

Does it got installed properly? Now I can start writing code in python to connect with zookeeper?

Sorry for asking all these dumb questions as I don't have any background with python so learning little bit here..

It's all about Python question here not zookeeper at all I guess..

Using kazoo (zookeeper client) in django

I was wondering if someone could point me to an example of using kazoo within a Django application. My use case is to use kazoo to retrieve a list of values, and process them, within a request cycle.

Any pointers would be much appreciated.

How to watch on descendant child nodes in Python using kazoo?

I recently started working with Python for Zookeeper. I am using kazoo library for Zookeeper.

I have a very simple use case using kazoo for Zookeeper. I have a root node which is - /root. Now I need to keep a watch on the root node /root and if the new node which gets added to root node /root is /root/testing then I will only keep a watch on the /root/testing node. I don't want to keep a watch on any other node apart from the testing node. And then if any new children get added up to the /root/testing node, then I will keep a watch on them as well.

Let's say the child below that gets added up -

`/root/testing/test1`

Then I will keep a watch on test1 node as well.

Is this possible to do using Kazoo in Zookeeper? I am only able to keep a watch on one Zookeeper node (/root) with the code below:

#!/usr/bin/python
import time

from kazoo.client import KazooClient

zk = KazooClient(hosts='127.0.0.1:2181')
zk.start()

########################################
@zk.ChildrenWatch("/root/testing")
def watch_children(children):
   print(children)

########################################

while True:
    time.sleep(5)

Can anyone help me with this in making multiple watches on the child node?

Handle badarg in Erlang

I am very new to the Erlang and I am getting badarg error when I try to convert binary to string as shown below.

Prefix = binary:bin_to_list(wh_json:get_ne_value(<<"prefix">>, Patterns)),

where Patterns are:

Pattern1--> {[{<<"prefix">>,<<>>},{<<"callerId">>,<<"1001">>},{<<"cid_regex">>,<<"^\\+?1001">>}]}

Pattern2--> {[{<<"prefix">>,<<"12">>},{<<"callerId">>,<<"1001">>},{<<"cid_regex">>,<<"^\\+?1001">>}]}

for Pattern2 it works fine but for Pattern1 I am getting this error because prefix does not have any value in Pattern1.

So, can any one tell me how I can handle this situation where prefix value can be null or any value, it should work for both the conditions.

How to add children node data updation watch in Python using kazoo package

I want to add watch on all children nodes of a node in Python using kazoo client but ChildrenWatch only watches on child add or remove not on data updation of any child node. I am searching for a simple recipe which does this task.

Sample code will be like this

from kazoo.client import KazooClient

zk = KazooClient(hosts='127.0.0.1:2181')
zk.start()

@zk.SomeRecipe("/root")
def watch_children_update(event):
    print("Updated child at path:%s data:%s stat:%s" % event.path, event.data, event.stat)

How to merge two json string in Python?

I recently started working with Python and I am trying to concatenate one of my JSON String with existing JSON String. I am also working with Zookeeper so I get the existing json string from zookeeper node as I am using Python kazoo library.

# gets the data from zookeeper
data, stat = zk.get(some_znode_path)
jsonStringA = data.decode("utf-8")

if I print jsonStringA it gives me like this -

{"error_1395946244342":"valueA","error_1395952003":"valueB"}

But if I do print json.loads(jsonString) then it prints out like this -

{u'error_1395946244342': u'valueA', u'error_1395952003': u'valueB'}

Here jsonStringA will have my existing JSON String. Now I have another key-value pair which I need to add in the exiting jsonStringA -

Below is my Python code -

# gets the data from zookeeper
data, stat = zk.get(some_znode_path)
jsonStringA = data.decode("utf-8")

timestamp_in_ms = "error_"+str(int(round(time.time() * 1000)))
node = "/pp/tf/test/v1"
a,b,c,d = node.split("/")[1:]
host_info = "h1"
local_dc = "dc3"
step = "step2"

My existing jsonStringA will be like this after extracting from zookeeper -

{"error_1395946244342":"valueA","error_1395952003":"valueB"}

Now I need to append this key-value pair in the jsonStringA -

"timestamp_in_ms":"Error Occured on machine "+host_info+" in datacenter "+ local_dc +" on the "+ step +" of process "+ c +"

So in short I need to merge below key-value pair -

"error_1395952167":"Error Occured on machine h1 in datacenter dc3 on the step2 of process test"

So final JSON String will look like this -

{"error_1395946244342":"valueA","error_1395952003":"valueB","error_1395952167":"Error Occured on machine h1 in datacenter dc3 on the step2 of process test"}

Is this possible to do?

Bug in kazoo 2.0 library

I am using kazoo in project I am working on. But after some time I get an error message that new connection to zookeper cannot be opened.

So I wrote a little test program:

for i in range(0, 1000):
    print i

    zk = kazoo.client.KazooClient()
    zk.start()
    zk.stop()

And after around 500 iteration I've got an error: "Too many open files".

Am I using kazoo somehow wrong?

How to get watch count of ZooKeeper

Now I want to get the watch count of the specified znode in ZooKeeper. I have read the official tutorial and known the mntr command. But it's for the whole QuorumServer.

So does ZooKeeper provide the API to get the number of watches of the specified znode? I'm also using kazoo and how can we get this data?

Unable to start kazoo client. Fails with 'An operation was attempted on something that is not a socket'

I am just starting out with Kazoo and I can't get a very simple program to run:

from kazoo.client import KazooClient
import logging
logging.basicConfig(level=logging.DEBUG)
from kazoo.retry import KazooRetry
_retry = KazooRetry(max_tries=1000, delay=0.5, backoff=2)
zk = KazooClient(hosts='164.55.92.8:2181', logger=logging, timeout=10, connection_retry=_retry, read_only=True)

zk.start()
import time
print('sleeping 5!')
time.sleep(5)
zk.stop()

The output is shown below:

ERROR:root:Unhandled exception in connection loop  
Traceback (most recent call last):  
  File "C:\Python27\lib\site-packages\kazoo-2.0-py2.7.egg\kazoo\protocol\connection.py", line 522, in _connect_attempt  
    [], [], timeout)[0]  
  File "C:\Python27\lib\site-packages\kazoo-2.0-py2.7.egg\kazoo\handlers\threading.py", line 250, in select  
    return select.select(*args, **kwargs)  
error: (10038, 'An operation was attempted on something that is not a socket')  
INFO:root:Zookeeper session lost, state: CLOSED  
Exception in thread Thread-3:  
Traceback (most recent call last):  
  File "C:\Python27\lib\threading.py", line 551, in __bootstrap_inner  
    self.run()  
  File "C:\Python27\lib\threading.py", line 504, in run  
    self.__target(*self.__args, **self.__kwargs)  
  File "C:\Python27\lib\site-packages\kazoo-2.0-py2.7.egg\kazoo\protocol\connection.py", line 466, in zk_loop  
    if retry(self._connect_loop, retry) is STOP_CONNECTING:  
  File "C:\Python27\lib\site-packages\kazoo-2.0-py2.7.egg\kazoo\retry.py", line 123, in __call__  
    return func(*args, **kwargs)  
  File "C:\Python27\lib\site-packages\kazoo-2.0-py2.7.egg\kazoo\protocol\connection.py", line 483, in _connect_loop  
    status = self._connect_attempt(host, port, retry)  
  File "C:\Python27\lib\site-packages\kazoo-2.0-py2.7.egg\kazoo\protocol\connection.py", line 522, in _connect_attempt  
    [], [], timeout)[0]  
  File "C:\Python27\lib\site-packages\kazoo-2.0-py2.7.egg\kazoo\handlers\threading.py", line 250, in select  
    return select.select(*args, **kwargs)  
error: (10038, 'An operation was attempted on something that is not a socket')  


Traceback (most recent call last):  
  File "C:\Users\klow\Google Drive\mycode\mykazoo\kazooo.py", line 8, in <module>  
    zk.start()  
  File "C:\Python27\lib\site-packages\kazoo-2.0-py2.7.egg\kazoo\client.py", line 537, in start  
    raise self.handler.timeout_exception("Connection time-out")  
TimeoutError: Connection time-out  
>>>   

I am running this on a Window 7 laptop and the Zookeeper server is running on a Linux box. It looks like a TCP connection has been made up something quickly screws up after that. I looked into the code a bit. I can see that the TCP connection has been established at connection.py:510 (self._connect(host, port)) and self._socket has been assigned to object returned by create_tcp_connection(socket, *args, **kwargs). However, self.handle.select() at connectiom.py:521 doesn't like the socket. Any idea? Thank you in advance!

How to watch for events on the descendant nodes in ZooKeeper using kazoo?

I recently started working with Python for Zookeeper. I am using kazoo library for Zookeeper. I need to keep a watch on my root node which is -

/my/example

couple of other nodes which might get added to my above root node will be like this -

/my/example/workflow
/my/example/workflow/v1
/my/example/workflow/v1/step1
/my/example/workflow/v1/step2

Now I need to check whether the children which got added in root node /my/example is /my/example/workflow or not. If workflow node gets added up in /my/example then I will keep a watch on /my/example/workflow node only and if any new children gets added up in /my/example/workflow node then I need to keep a watch on that node as well.

Let's say, children of /my/example/workflow is /my/example/workflow/v1, so now I need to keep a watch on /my/example/workflow/v1 and then if any new nodes gets added up on this node /my/example/workflow/v1 such as /my/example/workflow/v1/step1 and /my/example/workflow/v1/step2 then I need to print the children of /my/example/workflow/v1 node and I won't make any new watches now.

Now I am not sure how to keep on calling watches on my children until certain point, here in this case till /my/example/workflow/v1 I need to keep on watching and as soon as all the steps nodes gets added up, I need to print the children of /my/example/workflow/v1. Below is my code which works fine for watching on only one root node and now I am not sure how to do my above problem?

#!/usr/bin/python
from kazoo.client import KazooClient

zk = KazooClient(hosts='127.0.0.1:2181')
zk.start()

@zk.ChildrenWatch("/my/example")
def watch_children(children):
    print("Children are now: %s" % children)

Any help is really appreciated on this. I was following the documentation by reading the kazoo tutorial from here

Unable to connect to whoistle_app Kazoo

Possible fixes: * Ensure the whistle service you are trying to connect to is running on the host * Ensure that you are using the same cookie as the whistle node, "./sup -c " * Verify that the hostname being used is a whistle node

No handlers could be found for logger "kazoo.client"

While giving the hardcoded value for the following code , its working fine.

#!/usr/bin/env python3
import time
from kazoo.client import KazooClient
import os

nodes = ["172.22.105.53"]

zk = KazooClient(hosts='172.22.105.53:2181')

output : no error

But the following lines gives error like No handlers could be found for logger "kazoo.client"

#!/usr/bin/env python3
import time
from kazoo.client import KazooClient
import os

nodes = ["172.22.105.53"]

lead = "172.22.105.53"
zk = KazooClient(hosts='lead:2181')

any help on this regard is quietly appreciable.

what is this amplify.module = function(whapp, module, config, construct, methods) {

i was checking on code on kazoo UI platform so i found this code in one of its js file and when i search online regarding amplifyjs module then their is not link or content related this topic. does anyone have idea about what exactly is this?

/*!
 * Amplify Module
 *
 * Version: @VERSION
 * Released: @DATE
 * Source: http://amplifyjs.com/module
 *
 * Copyright 2010 appendTo LLC. (http://appendto.com/team)
 * Dual licensed under the MIT or GPL licenses.
 * http://appendto.com/open-source-licenses
 */
(function( amplify, $, undefined ) {
var modules = {};
amplify.module = function(whapp, module, config, construct, methods) {

Kazoo installation from source code

I need help to install and verify kazoo from its source code.

I have cloned source code of kazoo, kazoo-ui and freeswitch from this page

I have also executed make command on kazoo source code. After that, I don't know how to proceed.

I was not able to run ./configure and make install on its source code.