Django with Pluggable MongoDB Storage troubles

I'm trying to use django, and mongoengine to provide the storage backend only with GridFS. I still have a MySQL database.

I'm running into a strange (to me) error when I'm deleting from the django admin and am wondering if I am doing something incorrectly.

my code looks like this:

# settings.py
from mongoengine import connect
connect("mongo_storage")

# models.py
from mongoengine.django.storage import GridFSStorage
class MyFile(models.Model):
    name = models.CharField(max_length=50)
    content = models.FileField(upload_to="appsfiles", storage=GridFSStorage())
    creation_time = models.DateTimeField(auto_now_add=True)
    last_update_time = models.DateTimeField(auto_now=True)

I am able to upload files just fine, but when I delete them, something seems to break and the mongo database seems to get in an unworkable state until I manually delete all FileDocument.objects. When this happens I can't upload files or delete them from the django interface.

From the stack trace I have:

/home/projects/vector/src/mongoengine/django/storage.py in _get_doc_with_name
        doc = [d for d in docs if getattr(d, self.field).name == name] ...
▼ Local vars
Variable    Value
_[1]    
[]
d   

docs    
Error in formatting: cannot set options after executing query
name    
u'testfile.pdf'
self    

/home/projects/vector/src/mongoengine/fields.py in __getattr__
        raise AttributeError 

Am I using this feature incorrectly?

UPDATE:

thanks to @zeekay's answer I was able to get a working gridfs storage plugin to work. I ended up not using mongoengine at all. I put my adapted solution on github. There is a clear sample project showing how to use it. I also uploaded the project to pypi.

Another Update:

I'd highly recommend the django-storages project. It has lots of storage backed options and is used by many more people than my original proposed solution.

Flask throwing 'working outside of request context' when starting sub thread

I am trying to start a new thread in Python inside of a Flask application. I am doing background work that gets triggered by the request, but I don't need to wait for the work to be done to respond to the request.

Is it possible to set the flask request in this sub-threat to the request that came in? Reason being, our ACL on our queries to our DB (mongoengine in front of mongoDB) relies on the request's user (it grabs it from flask's request object) to see if they have access to the objects, and its blowing up because the request is not available in the sub-thread.

Any thoughts would be much appreciated.

Here's pseudo code of how I am handling it now, but it is not working.

@app.route('/my_endpoint', methods=['POST'])
def my_endpoint_handler():
    #do tracking in sub-thread so we don't hold up the page
    def handle_sub_view(req):
        from flask import request
        request = req
        # Do Expensive work
    thread.start_new_thread(handle_sub_view, (request))
    return "Thanks"

Flask and Mongo [closed]

Thinking of a web service entirely built on top of MongoDB, while I am pretty confortable with PyMongo, I would like to know if you guys have any positive or negative experiences/stories about either of these ODMs: MongoKit, MongoEngine and MongoAlchemy, the latter having a Flask specific package "Flask-mongoalchemy".

Use MongoEngine and PyMongo together

I want to use MongoEngine for my next project. Now I'm wondering whether I could also use PyMongo directly in the same project. Just for the case that I need something very special that is not supported directly via mongoengine.

Are there any doubts that this would work, or that I should not do that!?

Sort using MongoEngine?

How do I sort the query objects in MongoEngine, like I would in a regular mongodb query?

http://www.mongodb.org/display/DOCS/Sorting+and+Natural+Order

PyMongo vs MongoEngine for Django

For one of my projects I prefered using Django+Mongo.

Why should I use MongoEngine, but not just PyMongo? What are advantages? Querying with PyMongo gives results that are allready objects, aren't they? So what is the purpose of MongoEngine?

find() and findOne() in mongoengine

How can I do a quick find() or findOne() with mongoengine, I already have this but it does not seems to be the right way:

Cars.objects()._collection.find_one({'model':2013})

Mongoengine creation_time attribute in Document

I am trying to add a creation_time attribute to my documents. The following would be an example:

import datetime

class MyModel(mongoengine.Document):
    creation_date = mongo.DateTimeField()
    modified_date = mongo.DateTimeField(default=datetime.datetime.now)

Django models have built in parameter for their DateTimeField objects like add_now, etc., but MongoEngine does not support this.

I am wondering if best way to do this is the following:

m,created = MyModel.objects.get_or_create()
if created:
    m.creation_date = datetime.datetime.now()

or if there is a better, nicer way.

Using MongoEngine Document class methods for custom validation and pre-save hooks

I am currently exploring the possibilities of the MongoEngine "object document mapper". What is currently not clear to me is to what extent I can move my validation and object creation logic to the Document objects themselves.

I have the impression that it should not be a problem, but I'm not finding a lot of examples/caveats/best practices regarding issues as

• Custom validation functions that are automatically called on save() to evaluate if field contents are valid;
• Automatic generation of the identifier on save(), based on the hash of the contents of a field;

I think I need to override the save() method, so that I can call my custom logic, but the lack of examples leads me to believe that that may be a wrong approach...

Any examples, or references to high-quality codebases using mongoEngine, are welcome.

Implementing Bi-Directional relationships in MongoEngine

I'm building a Django application that uses MongoDB and MongoEngine to store data. To present a simplified version of my problem, say I want to have two classes: User and Page. Each page should associate itself with a user and each user a page.

from mongoengine import *

class Page(Document):
    pass

class User(Document):
    name = StringField()
    page = ReferenceField(Page)

class Page(Document):
    content = StringField()
    user = ReferenceField(User)

(Note that Page must be defined before User. If I am missing a Pythonic way to handle circular dependencies, let me know.) Each document can be created and saved just fine, but assigning a Page to a User throws an error.

u = User(name='Jeff')
u.save()
p = Page(content="I'm a page!")
p.save()
p.user = u
p.save()
u.page = p
u.save()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "build\bdist.win32\egg\mongoengine\document.py", line 71, in save
  File "build\bdist.win32\egg\mongoengine\base.py", line 303, in validate
mongoengine.base.ValidationError: Invalid value for field of type "ReferenceField"

Can anyone explain why this exception is being thrown, what I am doing wrong, and how I can avoid it?

MongoDB using an OR clause in mongoengine

I'm using python's mongoengine to query MongoDB, and have loved it for the most part, but I'm having an issue with an advanced query.

Here's my model

class ContentItem(Document):
    account = ReferenceField(Account)
    creator = ReferenceField(User)
    public = BooleanField(default=False) 
    last_used = DateTimeField(default=datetime.now)

I would like to make a query for all ContentItem's that are of a particular account, and are either created by the logged in user or are public. Here's the query I wrote

query = ContentItem.objects.filter( (Q(account=account) & Q(public=True)) |  (Q(account=account) & Q(creator=logged_in_user)) ).order_by('-last_used')

or:

query = ContentItem.objects.filter( Q(account=account) & ( Q(public=True) |  Q(creator=logged_in_user) ) ).order_by('-last_used')

But these seem to be XOR where if either public, or the creator but not both. Is this expected?

Am I overlooking something? Should I do this directly with mongodb instead of mongoengine?

My current workaround is to make two different queries and combine the results, but as the # of Content Items gets larger the result is taking a long time to come back because I need to get all items before I can order them, thereby losing all the benefit of (django) paginated results.

Mongo connections never released - Django and Mongoengine running on gunicorn with gevent

I have a django application using mongoengine running on gunicorn with gevent workers. Under load, mongo connection count climbs up to about 3 thousand and never goes back down. Even after the load test is completed, the number of mongo connections stays constant. A restart of gunicorn releases the connections.

Package Versions

gunicorn==0.17.4
mongoengine==0.8.7
pymongo==2.7

mongodb 2.6.0

I have my mongoengine connection settings in an environment specific django settings file:

MONGO_DATABASES = {
    'default': {
        'DB': '****',
        'HOST': ***********:27017',
        'PORT': 27017
    }
}

from gevent import monkey
monkey.patch_all()
from mongoengine import connect
connect(MONGO_DATABASES['default']['DB'], host=MONGO_DATABASES['default']['HOST'],       port=MONGO_DATABASES['default']['PORT'], max_pool_size=100)

Is there something I need to do to make sure that unused connections eventually get released?

Thanks,

Doug

MongoEngine ListField within a EmbeddedDocument throws TypeError on validation

I am not sure if it is a bug within MongoEngine or if I miss something. I have the following Models set up:

class Features(EmbeddedDocument):
    version = FloatField()
    data = ListField(StringField)

class Article(Document):
    vendor = ReferenceField(Vendor)
    url = URLField()
    author = StringField()
    clean_content = StringField()
    features = EmbeddedDocumentField(Features)

When I test my models like this:

#add vendor
vendor = Vendor(name="techcrunch", config="vendor config")
vendor.save()

#create features
features = Features(version = 1.0)
features.data = ["5", "89"]

#add article
article = Article(vendor = vendor, url ="http://www.techcrunch.com", 
                  author ="MG Siegler", clean_content = "Apple rocks!")
article.features = features
article.save()

I get the following error:

TypeError: unbound method _validate() must be called with StringField instance as first argument (got str instance instead)

Can someone explain that?

EDIT:

Nevermind. I found my error.

It has to be:

class Features(EmbeddedDocument):
    version = FloatField()
    data = ListField(StringField())

MongoEngine User authentication (django)

I am trying to use MongoEngine in a django project I am writing. I am having difficulty getting (or understanding how) the authentication backend works.

The user object as far as I can tell is not stored in the request.

I have it working but I am not sure if I am doing it in the right/safe way. If someone could look at my code I would be much appreciated.

def login(request):
    user = authenticate(request.POST['username'],request.POST['password'])
    if user is not None:
        request.session['user'] = user
        if user.is_authenticated:
            return HttpResponse(user)
    else:
        return HttpResponse('login failed')

def new_page(request):
    try:
        user = request.session['user']
        if user.is_authenticated:
            return HttpResponse('welcome')
    except:
        return HttpResponse('need be logged in')

in my settings.py I have added at the top of the file:

AUTHENTICATION_BACKENDS = (
    'mongoengine.django.auth.MongoEngineBackend',
)

SESSION_ENGINE = 'mongoengine.django.sessions'

import mongoengine
mongoengine.connect('project')

MongoEngine -- how to custom User model / custom backend for authenticate()

SUMMARY

How do I use a custom User model and a custom authentication backend (to allow for email / password authentication) with Django + MongoEngine? (Is a custom backend even necessary for that? ...i.e., to use an email for username when authenticating with MongoEngine.)

Is there any documentation with a straight-forward (and complete!) example of using a custom user object while using Mongo as the primary datastore when authenticating in Django? (Postgres has such clearer and more comprehensive docs...)

DETAIL

MongoEngine seems to give you just two flavors of authentication--the "Classic" (aka 'mongoengine.django.auth.MongoEngineBackend') way...OR...the "Custom User Model" (aka 'django.contrib.auth.backends.ModelBackend') way--both of which more or less succinctly outlined in Nicolas Cortot's answer to a different question here:

Python-Social-Auth fails with mongoEngine (Django)

Both of these authentication techniques give you access to an authenticate() method similar to Django's AbstractBaseUser class--a method which relies on a *check_password* function. However, the minute you use the so-called "Custom User Model" flavor of authentication (as outlined in the above link)...and then pair that with a custom backend (in order to use emails for usernames)...you run into trouble due to the absence of access to the typical authenticate() function.

For example, like so...

accounts.models.py

# ...with postgres, I'd subclass AbstractBaseUser...but with Mongo...(?)

from django.conf import settings
from mongoengine.fields import EmailField, BooleanField 

from mongoengine.django.auth import User

class MyUser(User):

    email = EmailField(max_length=254, unique=True)
    is_active = BooleanField(default=True)
    is_admin = BooleanField(default=False)

    USERNAME_FIELD = 'email'
    REQUIRED_FIELDS = ''

    ...

my_custom_backend.py

# ...is a custom backend even necessary to use email for authentication instead of username?

from django.conf import settings
from django.contrib.auth.models import check_password
#from mongoengine.django.auth import check_password
#from django.contrib.auth.hashers import check_password
from models import MyUser

    class EmailAuthBackend(object):

        def authenticate(self, email=None, password=None):

# ...uh oh, since I'm NOT using one of the usual backends with a pre-existing authenticate()
# method, there ain't a native check_password() function available. Means I have to hash the
# password, etc.

So, seemingly, I'm obliged to write my own check_password function. To get all of the goodness inherent with the AbstractBaseUser class typically found with a PostgreSQL authentication, I'd have to totally inflate my custom User model, which seems hacky and can't be very DRY.

Am I getting totally confused here? ...i.e., is it actually totally unnecessary to use a custom backend if I want to use emails instead of usernames for authentication when using MongoEngine?

I feel like I may have a fundamental misunderstanding of how Django works with MongoEngine in regard to authentication, and with regard to how I've modeled and called upon custom user object / my particular subclassing of MongoEngine's user object during that process...

Because--as it is right now--I'm getting an "'AnonymousUser' object has no attribute 'backend'" error message in the browser. I also noted that this problem sometimes exists for unexpected reasons--namely: perhaps, the authenticate() method expects a hashed password, or because the login (email) is too long...? For more instances where this latter circumstance might be the case, see:

Django Register Form 'AnonymousUser' object has no attribute 'backend'

settings.py

INSTALLED_APPS = (
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.sites',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'django.contrib.admin',
    'mongoengine.django.mongo_auth',
    'accounts',
)

AUTHENTICATION_BACKENDS = (
    'mongoengine.django.auth.MongoEngineBackend',
    #'accounts.my_custom_backend.EmailAuthBackend',
    #'django.contrib.auth.backends.ModelBackend',
)

AUTH_USER_MODEL = 'mongo_auth.MongoUser'
MONGOENGINE_USER_DOCUMENT = 'accounts.models.User'

accounts.views.py

from django.contrib.auth import login as django_login
from my_custom_backend import EmailAuthBackend
from forms import AuthenticationForm

def login(request):

    form = AuthenticationForm(data=request.POST)
    if form.is_valid():
        try:
            backend = EmailAuthBackend()
            user = backend.authenticate(email=request.POST['email'], password=request.POST['password'])
            django_login(request, user)
            return redirect('/')
        except DoesNotExist:
            return HttpResponse('user does not exist')
    else:
        form = AuthenticationForm()

    return render_to_response('accounts/login.html',
       { 'form': form },
       context_instance=RequestContext(request))