EzDevInfo.com

json type in Cassandra data model?

I am wondering if I could have json data type in the column family:

My table will have a unique row key, and column name of "tweets_json" and column value of json content.

How would I create such a table in CQL/cassandra or using python Driver or CQLengine?

tweet_json = json.encode({
                "tweet_id" : tweet_id,
                "body" : tweet_body,
                "user_name" : this_user,
                "timestamp" : timestamp
            })

Basicall

DataStax Enterprise 4.6.1 / C* Pagination in Python using cqlengine 0.21.0

I am currently trying to paginate query results from DSE 4.6.1 (Cassandra 2.0.12.200) with Python and cqlengine 0.21.0.

My table being queried is:

CREATE TABLE tags_for_search (
  village_id int,
  tag_prefix text,
  tag text,
  time timeuuid,
  author_id int,
  tag_id uuid,
  type text,
  PRIMARY KEY ((village_id, tag_prefix), tag, time)
) WITH CLUSTERING ORDER BY (time DESC) AND
  bloom_filter_fp_chance=0.010000 AND
  caching='KEYS_ONLY' AND
  comment='' AND
  dclocal_read_repair_chance=0.100000 AND
  gc_grace_seconds=864000 AND
  index_interval=128 AND
  read_repair_chance=0.000000 AND
  populate_io_cache_on_flush='false' AND
  default_time_to_live=0 AND
  speculative_retry='99.0PERCENTILE' AND
  memtable_flush_period_in_ms=0 AND
  compaction={'class': 'SizeTieredCompactionStrategy'} AND
  compression={'sstable_compression': 'LZ4Compressor'};

Is there an alternative for paging/pagination (DataStax Enterprise / DSE 4.6.1) of results in Python (using cqlengine 0.21.0)? The documented solution (http://cqlengine.readthedocs.org/en/latest/topics/queryset.html#token-function) appears to be broken due to #7016.

My initial query for data is:

SELECT * FROM that_keyspace.tags_for_search WHERE "tag_prefix" = 'der' AND "tag_text" = '#derpy1' AND "village_id" = 1 LIMIT 10000;

Or in Python via cqlengine:

village_tags = VillageSearch.objects.filter(village_id=1, tag_prefix='der', tag_text='#derpy1')

tags_data = []
for village_tag in village_tags:
    tags_data.append(village_tag.to_dict())

first_page = village_tags
last = first_page[-1]
next_page = list(village_tags.filter(pk__token__gt=cqlengine.Token(last.pk)))

It is throwing the error:

code=2200 [Invalid query] message="Column "village_id" cannot be restricted by both an equality and an inequality relation"

Is there an alternative that I can use to avoid this bug for immediate usage?

Thank you for any assistance you are able to provide!

Cassandra query making - Cannot execute this query as it might involve data filtering and thus may have unpredictable performance

I'd the following Cassandra Model:-

class Automobile(Model):
    manufacturer = columns.Text(primary_key=True)
    year = columns.Integer(index=True)
    model = columns.Text(index=True)
    price = columns.Decimal(index=True)

I needed the following queries:-

q = Automobile.objects.filter(manufacturer='Tesla')
q = Automobile.objects.filter(year='something')
q = Automobile.objects.filter(model='something')
q = Automobile.objects.filter(price='something')

These all were working fine, until i wanted multiple column filtering, ie when I tried

q = Automobile.objects.filter(manufacturer='Tesla',year='2013')

it throws an error saying Cannot execute this query as it might involve data filtering and thus may have unpredictable performance.

I rewrote the query with allowed_filtering, but this is not an optimal solution.

Then upon reading more, I edited my model as follow:-

class Automobile(Model):
    manufacturer = columns.Text(primary_key=True)
    year = columns.Integer(primary_key=True)
    model = columns.Text(primary_key=True)
    price = columns.Decimal()

With this I was able to filter multiple coulms as well, without any warning.

When I did DESCRIBE TABLE automobile, it shows this creates composite key PRIMARY KEY ((manufacturer), year, model).

So, my question is what if I declare every attribute as primary key? Is there any problem with this, since I'll be able to filter multiple columns as well.

This is just a small model. What if I had a model such as:-

class UserProfile(Model):
    id = columns.UUID(primary_key=True, default=uuid.uuid4)
    model = columns.Text()
    msisdn = columns.Text(index=True)
    gender = columns.Text(index=True)
    imei1 = columns.Set(columns.Text)
    circle = columns.Text(index=True)
    epoch = columns.DateTime(index=True)
    cellid = columns.Text(index=True)
    lacid = columns.Text(index=True)
    mcc = columns.Text(index=True)
    mnc = columns.Text(index=True)
    installed_apps = columns.Set(columns.Text)
    otp = columns.Text(index=True)
    regtype = columns.Text(index=True)
    ctype = columns.Text(index=True)
    operator = columns.Text(index=True)
    dob = columns.DateTime(index=True)
    jsonver = columns.Text(index=True)

and if I declare every attribute as PK, is there any problem with this?

How do I design this cassandra model properly?

I know the title is a bit broad, but I have tried a bunch of things with different errors, so I was going to use this space to describe each of them. I have a model in datastax's python-driver below:

class DIDSummary(Model):
    __keyspace__ = 'processor_api'

    did = columns.Text(required=True, primary_key=True, partition_key=True)
    month = columns.DateTime(required=True, primary_key=True, partition_key=True)
    direction = columns.Text(required=True)
    duration = columns.Counter(required=True)
    cost = columns.Counter(required=True)

but then when I try this:

didsummaries = DIDSummary.filter(did__in=dids, month__in=datetime_range)

I get this error:

code=2200 [Invalid query] message="Partition KEY part did cannot be restricted by IN relation (only the last part of the partition key can)"

I then tried this:

class DIDSummary(Model):
    # ...
    month = columns.DateTime(required=True, primary_key=True)
    # ...

Which brought up the error:

code=2200 [Invalid query] message="Clustering column "month" cannot be restricted by an IN relation"

I also tried this:

class DIDSummary(Model):
    # ...
    month = columns.DateTime(required=True, primary_key=True, index=True)
    # ...

Which gave me this error:

InvalidRequest: code=2200 [Invalid query] message="Secondary indexes are not supported on counter tables"

When I tried to sync_table(DIDSummary)

Finally, I tried to do this:

class DIDSummary(Model):
    # ...
    month = columns.DateTime(required=True, primary_key=True)
    direction = columns.Text(required=True)
    # ...

Which wouldn't even let me run the server, ending up with this error in the log:

cassandra.cqlengine.models.ModelDefinitionException: counter models may not have data columns

Any help on how I should better design this so I can run the above command would be greatly appreciated. Thanks!

EDIT: I also have to do this:

didsummaries = DIDSummary.filter(did__in=dids, month=month)

which means that actually did has to be the last part of my partition_key. Can I just assign a random uuid to each element and just lookup the elements based on primary_keys and indexes? Would that work? I'm super lost.

Cqlengine queries: how to return dictionary?

I'm using cqlengine in Django application, where Cassandra is the secondary database.

In some cases I need to join manually the results of SQL and NoSQL database requests.

For SQL I'm using:

model_name.objects.all().values() 

to return dictionary, rather than model-instance objects.

But I can't find the appropriate methods in cqlengine.

As a beginner in python, I do not know how best to implement this functionality within cqlengine library.

Maybe you have some examples of program code, providing this?

cqlengine model default values and uuid

Working with cqlengine models i have found an unexpected behaviour with default_values and uuid.

I am using python 3.4, and cqlengine from cassandra-driver 2.5.0.

with following code:

from cassandra.cqlengine import columns, connection, management
from cassandra.cqlengine.models import Model

import uuid

class Person(Model):
    id = columns.UUID(primary_key=True, default=uuid.uuid4())
    first_name  = columns.Text()
    last_name = columns.Text()

connection.setup(['127.0.0.1'], 'test_keyspace', 1, protocol_version=2)
management.sync_table(Person)

print("default value")
for i in range(10):
    p = Person().save()
    print(p.id)


print("\nparameter value")
for i in range(10):
    p = Person(id=uuid.uuid4()).save()
    print(p.id)

get output:

default value
703ddd0a-5a36-4c07-975b-638a9f7d2404
703ddd0a-5a36-4c07-975b-638a9f7d2404
703ddd0a-5a36-4c07-975b-638a9f7d2404
703ddd0a-5a36-4c07-975b-638a9f7d2404
703ddd0a-5a36-4c07-975b-638a9f7d2404
703ddd0a-5a36-4c07-975b-638a9f7d2404
703ddd0a-5a36-4c07-975b-638a9f7d2404
703ddd0a-5a36-4c07-975b-638a9f7d2404
703ddd0a-5a36-4c07-975b-638a9f7d2404
703ddd0a-5a36-4c07-975b-638a9f7d2404

parameter value
1366251d-36fc-480e-abae-bbbd957035af
e0f6936d-08d9-4308-bb68-79ba8d8db162
2cbcf896-96ec-4d29-993f-84336a13a3b5
2b0c765e-6cd8-4da0-86e5-14f49f5fa8fc
42ea40ed-8b12-4da3-aea6-ca7192d35290
13db1568-addf-4c60-be00-7c538c366245
b889c159-cf21-4193-99df-f2fe0275bce9
cd3bcf63-cf4b-4729-a1fb-bde413f9273d
99692a23-42a1-40d3-a41f-d5fb77bff691
008f496a-4c74-4cd4-9675-3f6a6e3a393c

why default argument in model always produce same uuid?

thanks

Cassandra 2 - CQLEngine : Store Images error

I'm trying to store images in database.This is my code for get an Image :

image = Image.open(...a resource on web...)
imageData = StringIO.StringIO()
image.save(imageData, image.format)
myImage = imageData.getvalue()

But when trying to store in database by this:

myTable.create(...some fields , image=myImage)

I catch an exception with this message:

Bad Request: Invalid STRING constant(ffd8ffe0.. and so on...adss4das) for image of type blob

I previously store images by these codes using Cassandra1.2.9!

But when I installed Cassandra2.0 , this problem happened!

I check my code line by line,and I'm sure that error in the way of storing images in C2.0 or getting image.

Python - Redis : Best practice serializing objects for storage in Redis

I am considering to serialize a big set of database records for cache in Redis, using python and Cassandra. I have either to serialize each record and persist a string in redis or to create a dictionary for each record and persist in redis as a list of dictionaries.

Which way is faster? pickle each record? or create a dictionary for each record?

And second : Is there any method to fetch from database as list of dic's? (instead of a list of model obj's)

Updates in CQL-engine

I use Cassandra with Django via cqlengine. Let's assume I have a model like this:

class ExampleModel(Model):
    example_id      = columns.UUID(primary_key=True, default=uuid.uuid4)
    example_types   = columns.List(columns.Integer)
    example_other   = columns.Integer()

I create an object using the following code:

ex = ExampleModel.create(example_types=[1,2,3])

Somewhere else I want to update this row (eg. add some value to example_other) knowing it's example_id:

ex2 = ExampleModel.create(example_id=<this id I know>, example_other=5)

In the old version of cqlengine (0.4.6) it used to work, that is example_types stored in the database used to contain [1,2,3]. After an upgrade to 0.8.5 and after creating ex2, the same object stored in database has example_types set to an empty list. I understand that this is probably cleaner, but I want to be able to update row (alter particular column without altering rest of them) without having to fetch it first. How to achieve this? Thanks in advance!

Cassandra - Cqlengine - TTL Support

I need to use TTL via cqlengine; But there is no documentation about it. Can someone help me about it. Thanks

How to query rows by blob primary key from Cassandra using Python's cqlengine

My Cassandra table looks like this:

uid                     | flwuid                     | tuuid
------------------------+----------------------------+--------------------------------------
0x50f893bf620d628f0500 | 0x527ce3c3c5ebc4dd7e8b4567 | 1e970c24-488d-11e3-b19a-7054d219d35c
0x505a31a507b8d04b0005 | 0x526a66cbc5ebc4d3708b456c | 6081f230-4797-11e3-b9af-7054d219d35c
0x505a31a507b845000005 | 0x527ce3c3c5ebc4dd7e8b4567 | 9da514bc-4878-11e3-bcb1-7054d219d35c
0x51ee6cef6cbf7679f7a0 | 0x524bfd0dc5ebc486738b4567 | 6bdf89c6-4b93-11e3-a8a0-7054d219d35c
0x51ee160a6cbf7679f7a0 | 0x526a7149c5ebc494708b456c | 8e8a1c4c-45fe-11e3-9b35-7054d219d35c

I have a cqlengine model:

class Followers(Model):
    uid = cqlengine.Bytes(primary_key=True)
    flwuid = cqlengine.Bytes(primary_key=True)
    tuuid = cqlengine.TimeUUID()

To get all rows with uid = 0x50f893bf620d628f0500 I am trying to do this:

Followers.objects(uid='0x50f893bf620d628f0500')

but getting an exception:

CQLEngineException: Bad Request: Invalid STRING constant (3078353066383933626636323064363238663035303030303061) for uid of type blob

How can specify correct blob value for cqlengine' query?

Cassandra 2.0.1, cqlengine 0.9.1.

Thanks in advance.

cqlengine: way to create nested map

In what ways we can create a nested map in cqlengine models, For e.g

class Test(Model):
     id = columns.UUID(primary_key=True, default=uuid.uuid4)
     address = columns.Map(key_type, value_type)

I want "value_type"(in address column) to be a dictionary again, how should we achieve this using cqlengine data modelling.

AttributeError: 'NoneType' object has no attribute '_consistency' when trying to show a list from database

I want to show a list from Cassandra database and show it in a html template usgin django. when I'm trying to show data it says: AttributeError: 'NoneType' object has no attribute '_consistency'

Views.py:

def music_list(request):
    files = Music.objects.all()
    return render_to_response('music_list.html',{'files':files})

Model.py:

class Music(Model):
    id = columns.UUID(primary_key=True)
    title = columns.Text()
    description = columns.Text()
    filename = columns.Text()

Music_list.html

{% extends "master.html" %}

{% block title %}This is the title from main page{% endblock %}

{% block contents %}
    <ul>
        {% for f in files %}
            {{ f.id }}
        {% endfor %}
        <li><a rel='nofollow' href="#">Item1</a></li>
        <li><a rel='nofollow' href="#">Item2</a></li>
        <li><a rel='nofollow' href="#">Item3</a></li>
    </ul>
{% endblock %}

This is error page:

AttributeError at /music-list
'NoneType' object has no attribute '_consistency'
Request Method: GET
Request URL:    http://127.0.0.1:8000/music-list
Django Version: 1.5.1
Exception Type: AttributeError
Exception Value:    
'NoneType' object has no attribute '_consistency'
Exception Location: C:\Python27\lib\site-packages\cqlengine\connection.py in execute, line 239
Python Executable:  C:\Python27\python.exe
Python Version: 2.7.4
Python Path:    
['D:\\Developer Center\\PyCharm\\MusicStore',
 'C:\\Python27\\lib\\site-packages\\pip-1.3.1-py2.7.egg',
 'C:\\Python27\\lib\\site-packages\\mysql_python-1.2.4-py2.7-win32.egg',
 'D:\\Developer Center\\PyCharm\\MusicStore',
 'C:\\Windows\\SYSTEM32\\python27.zip',
 'C:\\Python27\\DLLs',
 'C:\\Python27\\lib',
 'C:\\Python27\\lib\\plat-win',
 'C:\\Python27\\lib\\lib-tk',
 'C:\\Python27',
 'C:\\Python27\\lib\\site-packages',
 'C:\\Python27\\lib\\site-packages\\PIL']
Server time:    Mon, 13 Jan 2014 11:24:09 +0330
Error during template rendering

In template D:\Developer Center\PyCharm\MusicStore\templates\music_list.html, error at line 7
'NoneType' object has no attribute '_consistency'
1   {% extends "master.html" %}
2   
3   {% block title %}This is the title from main page{% endblock %}
4   
5   {% block contents %}
6       <ul>
7           {% for f in files %}
8               {{ f.id }}
9           {% endfor %}
10          <li><a rel='nofollow' href="#">Item1</a></li>
11          <li><a rel='nofollow' href="#">Item2</a></li>
12          <li><a rel='nofollow' href="#">Item3</a></li>
13      </ul>
14  {% endblock %}
15  
Traceback Switch to copy-and-paste view

C:\Python27\lib\site-packages\django\core\handlers\base.py in get_response
                        response = callback(request, *callback_args, **callback_kwargs) ...
▶ Local vars
D:\Developer Center\PyCharm\MusicStore\MusicStoreApp\views.py in music_list
    return render_to_response('music_list.html',{'files':files}) ...
▶ Local vars
C:\Python27\lib\site-packages\django\shortcuts\__init__.py in render_to_response
    return HttpResponse(loader.render_to_string(*args, **kwargs), **httpresponse_kwargs) ...
▶ Local vars
C:\Python27\lib\site-packages\django\template\loader.py in render_to_string
        return t.render(Context(dictionary)) ...
▶ Local vars
C:\Python27\lib\site-packages\django\template\base.py in render
            return self._render(context) ...
▶ Local vars
C:\Python27\lib\site-packages\django\template\base.py in _render
        return self.nodelist.render(context) ...
▶ Local vars
C:\Python27\lib\site-packages\django\template\base.py in render
                bit = self.render_node(node, context) ...
▶ Local vars
C:\Python27\lib\site-packages\django\template\debug.py in render_node
            return node.render(context) ...
▶ Local vars
C:\Python27\lib\site-packages\django\template\loader_tags.py in render
        return compiled_parent._render(context) ...
▶ Local vars
C:\Python27\lib\site-packages\django\template\base.py in _render
        return self.nodelist.render(context) ...
▶ Local vars
C:\Python27\lib\site-packages\django\template\base.py in render
                bit = self.render_node(node, context) ...
▶ Local vars
C:\Python27\lib\site-packages\django\template\debug.py in render_node
            return node.render(context) ...
▶ Local vars
C:\Python27\lib\site-packages\django\template\loader_tags.py in render
            result = block.nodelist.render(context) ...
▶ Local vars
C:\Python27\lib\site-packages\django\template\base.py in render
                bit = self.render_node(node, context) ...
▶ Local vars
C:\Python27\lib\site-packages\django\template\debug.py in render_node
            return node.render(context) ...
▶ Local vars
C:\Python27\lib\site-packages\django\template\defaulttags.py in render
        len_values = len(values) ...
▶ Local vars
C:\Python27\lib\site-packages\cqlengine\query.py in __len__
        self._execute_query() ...
▶ Local vars
C:\Python27\lib\site-packages\cqlengine\query.py in _execute_query
            columns, self._result_cache = self._execute(self._select_query()) ...
▶ Local vars
C:\Python27\lib\site-packages\cqlengine\query.py in _execute
            return execute(q, consistency_level=self._consistency) ...
▶ Local vars
C:\Python27\lib\site-packages\cqlengine\connection.py in execute
        consistency_level = connection_pool._consistency ...
▶ Local vars

Python cassandra-driver OperationTimeOut on every query in Celery task

I have a problem with every insert query (little query) which is executed in celery tasks asynchronously. In sync mode when i do insert all done great, but when it executed in apply_async() i get this:

OperationTimedOut('errors=errors=errors={}, last_host=***.***.*.***, last_host=None, last_host=None',)

Traceback:

Traceback (most recent call last):
  File "/var/nfs_www/***/env_v0/local/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/var/nfs_www/***/env_v0/local/lib/python2.7/site-packages/celery/app/trace.py", line 437, in __protected_call__
    return self.run(*args, **kwargs)
  File "/var/nfs_www/***/www_v1/app/mods/news_feed/tasks.py", line 26, in send_new_comment_reply_notifications
    send_new_comment_reply_notifications_method(comment_id)
  File "/var/nfs_www/***www_v1/app/mods/news_feed/methods.py", line 83, in send_new_comment_reply_notifications
    comment_type='comment_reply'
  File "/var/nfs_www/***/www_v1/app/mods/news_feed/models/storage.py", line 129, in add
    CommentsFeed(**kwargs).save()
  File "/var/nfs_www/***/env_v0/local/lib/python2.7/site-packages/cqlengine/models.py", line 531, in save
    consistency=self.__consistency__).save()
  File "/var/nfs_www/***/env_v0/local/lib/python2.7/site-packages/cqlengine/query.py", line 907, in save
    self._execute(insert)
  File "/var/nfs_www/***/env_v0/local/lib/python2.7/site-packages/cqlengine/query.py", line 786, in _execute
    tmp = execute(q, consistency_level=self._consistency)
  File "/var/nfs_www/***/env_v0/local/lib/python2.7/site-packages/cqlengine/connection.py", line 95, in execute
    result = session.execute(query, params)
  File "/var/nfs_www/***/env_v0/local/lib/python2.7/site-packages/cassandra/cluster.py", line 1103, in execute
    result = future.result(timeout)
  File "/var/nfs_www/***/env_v0/local/lib/python2.7/site-packages/cassandra/cluster.py", line 2475, in result
    raise OperationTimedOut(errors=self._errors, last_host=self._current_host)
OperationTimedOut: errors={}, last_host=***.***.*.***

Does anyone have ideas about problem?

I found this When cassandra-driver was executing the query, cassandra-driver returned error OperationTimedOut, but my query is very little and problem only in celery tasks.

UPDATE:

I made a test task and it raises this error too.

@celery.task()
def test_task_with_cassandra():
    from app import cassandra_session
    cassandra_session.execute('use news_feed')
    return 'Done'

UPDATE 2: Made this:

@celery.task()
def test_task_with_cassandra():
    from cqlengine import connection
    connection.setup(app.config['CASSANDRA_SERVERS'], port=app.config['CASSANDRA_PORT'],
                     default_keyspace='test_keyspace')
    from .models import Feed
    Feed.objects.count()
    return 'Done'

Got this:

NoHostAvailable('Unable to connect to any servers', {'***.***.*.***': OperationTimedOut('errors=errors=Timed out creating connection, last_host=None, last_host=None',)})

From shell i can connect to it

UPDATE 3: From deleted thread on github issue (found this in my emails): (this worked for me too) Here's how, in substance, I plug CQLengine to Celery:

from celery import Celery
from celery.signals import worker_process_init, beat_init
from cqlengine import connection
from cqlengine.connection import (
    cluster as cql_cluster, session as cql_session)

def cassandra_init():
    """ Initialize a clean Cassandra connection. """
    if cql_cluster is not None:
        cql_cluster.shutdown()
    if cql_session is not None:
        cql_session.shutdown()
    connection.setup()

# Initialize worker context for both standard and periodic tasks.
worker_process_init.connect(cassandra_init)
beat_init.connect(cassandra_init)

app = Celery()

This is crude, but works. Should we add this snippet in the FAQ ?

Cassandra cqlengine remove from map

Documentation of cqlengine have only update method for columns.Map. This method does not allow to remove an element from the map column(only add or update). Is there a simple way to remove an element from the map column using clqengine?