EzDevInfo.com

Using Google Cloud Datastore with NDB API?

There is a lot of info on using NDB API with Google App Engine Datastore but I can't find any info on how to use NDB with Google Cloud Datastore. The only module I found is googledatastore which is very primitive library.

How is App Engine Datastore different from the Cloud Datastore? Is NDB available for the Cloud Datastore?

Google Cloud Storage Signed URLs with Google App Engine

It's frustrating to deal with the regular Signed URLs (Query String Authentication) for Google Cloud Storage.

Google Cloud Storage Signed URLs Example -> Is this really the only code available in the whole internet for generating Signed URLs for Google Cloud Storage? Should I read it all and adapt it manually for Pure Python GAE if needed?

It's ridiculous when you compare it with AWS S3 getAuthenticatedURL(), already included in any SDK...

Am I missing something obvious or does everyone face the same problem? What's the deal?

Datastore: Is there plan to add GQLQuery support?

I am using gcloud-python library for a project which needs to serve following use case:

• Get a batch of entities with a subset of its properties (projection)
• gcloud.datastore.api.get_multi() provides me batch get but not projection
• and gcloud.datastore.api.Query() provides me projection but not batch get (like a IN query)

AFAIK, GQLQuery provides both IN query(batch get) and projections. Is there a plan to support GQLQueries in gcloud-python library? OR, is there another way to get batching and projection in single request?

Connecting to Google Datastore using gcloud

I am trying to connect to my Google Datastore instance but I can't find any sample code for how to initialize a connection!

Gcloud's main documentation immediately starts with how to assign an Entity or a Query but skips the crucial connection bit. The demo section seems specific to the demo in that it uses test environment variables which is quite confusing.

Most importantly, I am interested in what environment variables I need to define and if it is possible to define the service email and path to key from within the python program.

Highly appreciate it if anyone can provide a sample code of how to commit something to a datastore instance.

Thanks!

Try to init a Repo using the G cloud, for an existing project

There is an existing project I am a developer on with ID: placingliterature

I am attempting to run the command gcloud init placingliterature

I ran g auth and logged in properly.

I tried with or without the gitRepo already existing

I get this error:

C:\Program Files\Google\Cloud SDK>gcloud init placingliterature Initialized gcloud directory in [C:\Program Files\Google\Cloud SDK\placingliterature.gcloud]. Cloning [https://source.developers.google.com/p/placingliterature/r/default] into [default]. Cloning into 'C:\Program Files\Google\Cloud SDK\placingliterature\default'... Failed to erase credential: Element not found fatal: Authentication failed for 'https://source.developers.google.com/p/placingliterature/r/default/' ERROR: Command '['git', 'clone', 'https://source.developers.google.com/p/placingliterature/r/default', 'C:\Program File s\Google\Cloud SDK\placingliterature\default', '--config', 'credential.helper=gcloud.cmd']' returned non-zero exit s tatus 128 ERROR: Unable to initialize project [placingliterature], cleaning up [C:\Program Files\Google\Cloud SDK\placingliteratur e]. ERROR: (gcloud.init) Unable to initialize project [placingliterature].

Anyone know why this is happening?

Fetching 100 Results at a Time in Google App Engine

I was hoping someone could explain to me how to use offsets or cursors in App Engine. I'm using gcloud to remote access entities for a huge data migration, and would like to grab data in batches of 100.

I'm guessing there is a very simple way to do this, but the documentation doesn't dive into cursors all too much. Here is what I have so far:

client = datastore.Client(dataset_id=projectID)

# retrieve 100 Articles
query = client.query(kind='Article').fetch(100)

for article in query:
  print article

How could I mark the end of that batch of 100 and then move into the next one? Thanks so much!

Edit:

I should mention that I do not have access to the app engine environment, which is why I'm a bit lost at the moment... :(

Using Datastore outside of App Engine (python)?

I am used to Google Cloud SQL, where you can connect to a database outside of GAE. Is something like this possible for the GAE datastore, using the Python NDB interface ideally ?

Basically, my use-case is I want to run acceptance tests that pre-populate and clean a datastore.

It looks like the current options are a JSON API or protocol buffers -- in beta. If so, it's kind of a pain then I can't use my NDB models to populate the data, but have to reimplement them for the tests, and worry that they haven't been saved to the datastore in the exact same way as if through the application.

Just checking I'm not missing something....

PS. yes I know about remote_api_shell, I don't want a shell though. I guess piping commands into it is one way, but ugghh ...

Programmatically get Google Datastore record count and used storage

I need to get Google Datastore usage statistics - storage usage, and record count for my application.

At the moment I only can count fetch all items and count them. But I have no idea how I can get storage amount used.

Is there any reports API I can use? (Like Google Apps report api)

PS. I use gcloud-python library to access Datastore.

Edited:

Thank you Andrei - Angels Like Reb

That's how I did it:

from gcloud import datastore
stat = list(datastore.Query('__Stat_Total__').fetch())[0]
print "bytes", stat['bytes']
print "count", stat['count']

Using gcloud-python in GAE

I've got a bunch of little Raspberry Pis running some python code which saves directly to the Datastore (skips GAE) using the gcloud-python datastore package. This works great. I now want to present the data via web and mobile clients using Google App Engine. On my MacBook I installed GAE using the installer and gcloud via pip. I can write a simple python script and execute it directly from the terminal which is able to write and read from the datastore via gcloud - that also works just fine.

However, when I try to incorporate that same code into GAE it fails. Based on my research, I expect that it is a PATH issue but after several hours of various attempts I am unable to resolve this. Suggestions would be appreciated.

I believe this post is similar to my issue: Google App Engine, Change which python version

Here are some pieces of information which may be relevant:

Python version

$ which python
/usr/bin/python

Per the referenced Stack Overflow issue I set the preferences of GAE to have a Python Path of /usr/bin/python. I have tried it

$PATH

$ echo $PATH
/Users/sheridangray/Projects/adt-bundle-mac-x86_64-20140702/sdk/google-cloud-sdk/bin:/Users/sheridangray/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/opt/X11/bin

os.path from Python interpreter

$ python
Python 2.7.6 (default, Sep  9 2014, 15:04:36)
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.39)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> os.path
<module 'posixpath' from '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/posixpath.pyc'>

gcloud installation

$ pip install --upgrade gcloud
Requirement already up-to-date: gcloud in /Library/Python/2.7/site-packages
...

Python 2.7 references

$ sudo find / -name python2.7
Password:
/Applications/Dropbox.app/Contents/Frameworks/Python.framework/Versions/2.7/include/python2.7
/Applications/Dropbox.app/Contents/Frameworks/Python.framework/Versions/2.7/lib/python2.7
/Applications/Dropbox.app/Contents/Resources/include/python2.7
/Applications/Dropbox.app/Contents/Resources/lib/python2.7
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.10.sdk/System/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.10.sdk/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.10.sdk/usr/include/python2.7
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.10.sdk/usr/lib/python2.7
find: /dev/fd/3: Not a directory
find: /dev/fd/4: Not a directory
/System/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7
/System/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7
/usr/bin/python2.7
/usr/lib/python2.7

Relevant Code

from gcloud import datastore

class MainHandler(webapp.RequestHandler):
  def get(self):
     dataset = datastore.get_dataset(dataset_id, email_address, private_key_file)

GAE Log File

*** Running dev_appserver with the following flags:
--skip_sdk_update_check=yes --port=8080 --admin_port=8000
Python command: /usr/bin/python
INFO     2014-11-21 09:02:03,276 devappserver2.py:745] Skipping SDK update check.
INFO     2014-11-21 09:02:03,288 api_server.py:172] Starting API server at: http://localhost:49183
INFO     2014-11-21 09:02:03,292 dispatcher.py:185] Starting module "default" running at: http://localhost:8080
INFO     2014-11-21 09:02:03,294 admin_server.py:118] Starting admin server at: http://localhost:8000
INFO     2014-11-21 09:02:34,319 module.py:709] default: "GET / HTTP/1.1" 200 2
INFO     2014-11-21 09:02:34,455 module.py:709] default: "GET /favicon.ico HTTP/1.1" 200 8348
INFO     2014-11-21 09:02:44,359 module.py:387] Detected file changes:
  /Users/sheridangray/Projects/city-pulse-web/main.py
ERROR    2014-11-21 09:02:46,443 webapp2.py:1552] gcloud
Traceback (most recent call last):
      File "/Users/sheridangray/Projects/adt-bundle-mac-x86_64-20140702/sdk/google-cloud-sdk/platform/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/webapp2-2.5.2/webapp2.py", line 1535, in __call__
rv = self.handle_exception(request, response, e)
  File "/Users/sheridangray/Projects/adt-bundle-mac-x86_64-20140702/sdk/google-cloud-sdk/platform/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/webapp2-2.5.2/webapp2.py", line 1529, in __call__
rv = self.router.dispatch(request, response)
  File "/Users/sheridangray/Projects/adt-bundle-mac-x86_64-20140702/sdk/google-cloud-sdk/platform/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/webapp2-2.5.2/webapp2.py", line 1278, in default_dispatcher
return route.handler_adapter(request, response)
  File "/Users/sheridangray/Projects/adt-bundle-mac-x86_64-20140702/sdk/google-cloud-sdk/platform/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/webapp2-2.5.2/webapp2.py", line 1102, in __call__
return handler.dispatch()
  File "/Users/sheridangray/Projects/adt-bundle-mac-x86_64-20140702/sdk/google-cloud-sdk/platform/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/webapp2-2.5.2/webapp2.py", line 572, in dispatch
return self.handle_exception(e, self.app.debug)
  File "/Users/sheridangray/Projects/adt-bundle-mac-x86_64-20140702/sdk/google-cloud-sdk/platform/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/webapp2-2.5.2/webapp2.py", line 570, in dispatch
return method(*args, **kwargs)
  File "/Users/sheridangray/Projects/city-pulse-web/main.py", line 63, in get
dataset = datastore.get_dataset(dataset_id, email_address, private_key_file)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/gcloud/datastore/__init__.py", line 103, in get_dataset
connection = get_connection(client_email, private_key_path)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/gcloud/datastore/__init__.py", line 65, in get_connection
    from gcloud.datastore.connection import Connection
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/gcloud/datastore/connection.py", line 3, in <module>
    from gcloud import connection
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/gcloud/connection.py", line 8, in <module>
class Connection(object):
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/gcloud/connection.py", line 24, in Connection
USER_AGENT = "gcloud-python/{0}".format(get_distribution('gcloud').version)
  File "/Users/sheridangray/Projects/adt-bundle-mac-x86_64-20140702/sdk/google-cloud-sdk/platform/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/setuptools-0.6c11/pkg_resources.py", line 311, in get_distribution
if isinstance(dist,Requirement): dist = get_provider(dist)
  File "/Users/sheridangray/Projects/adt-bundle-mac-x86_64-20140702/sdk/google-cloud-sdk/platform/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/setuptools-0.6c11/pkg_resources.py", line 197, in get_provider
return working_set.find(moduleOrReq) or require(str(moduleOrReq))[0]
  File "/Users/sheridangray/Projects/adt-bundle-mac-x86_64-20140702/sdk/google-cloud-sdk/platform/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/setuptools-0.6c11/pkg_resources.py", line 666, in require
needed = self.resolve(parse_requirements(requirements))
  File "/Users/sheridangray/Projects/adt-bundle-mac-x86_64-20140702/sdk/google-cloud-sdk/platform/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/setuptools-0.6c11/pkg_resources.py", line 565, in resolve
    raise DistributionNotFound(req)  # XXX put more info here
DistributionNotFound: gcloud
INFO     2014-11-21 09:02:46,456 module.py:709] default: "GET / HTTP/1.1" 500 5010
INFO     2014-11-21 09:02:46,514 module.py:709] default: "GET /favicon.ico HTTP/1.1" 304 -

Google Cloud Storage + Python : Any way to list obj in certain folder in GCS?

I'm going to write a Python program to check if a file is in certain folder of my Google Cloud Storage, the basic idea is to get the list of all objects in a folder, a file name list, then check if the file abc.txt is in the file name list.

Now the problem is, it looks Google only provide the one way to get obj list, which is uri.get_bucket(), see below code which is from https://developers.google.com/storage/docs/gspythonlibrary#listing-objects

uri = boto.storage_uri(DOGS_BUCKET, GOOGLE_STORAGE)
for obj in uri.get_bucket():
    print '%s://%s/%s' % (uri.scheme, uri.bucket_name, obj.name)
    print '  "%s"' % obj.get_contents_as_string()

The defect of uri.get_bucket() is, it looks it is getting all of the object first, this is what I don't want, I just need get the obj name list of particular folder(e.g gs//mybucket/abc/myfolder) , which should be much quickly.

Could someone help answer? Appreciate every answer!

Internal Kinds Returned When Retrieving All Entities Belonging to a Particular Namespace

I am trying to retrieve all entities belonging to a particular namespace. The query is quite simple

query = datastore.Query(namespace=<namespace>)

Running this however returns keys belonging to internal kinds which aren’t part of the data I am storing, for example I get entities belonging to this kind:

__Stat_Ns_Kind_IsRootEntity__

Do you know how I can prevent this? Can I refine my query to exclude these?

How to use gcloud commands programmatically via Python

The Google documentation is a little generic on this topic and I find it hard to get around the different APIs and terms they're using, so I'm wondering if someone could point me to the right direction.

I'm looking for a way to call the gcloud command directly from Python. I've installed gcloud in my Python environment and as an example to follow, I'd like to know how to do the following from Python:

gcloud compute copy-files [Source directory or file name] [destination directory of file name]