EzDevInfo.com

Attacking Python's pickle

I'm writing a web app that stores user input in an object. This object will be pickled.

Is it possible for a user to craft malicious input that could do something egregious when the object is unpickled?

Here's a really basic code example that ignores wonderful principles such as encapsulation but epitomizes what I'm looking at:

import pickle

class X(object):
    some_attribute = None

x = X()
x.some_attribute = 'insert some user input that could possibly be bad'

p = pickle.dumps(x)

# Can bad things happen here if the object, before being picked, contained
# potentially bad data in some_attribute?
x = pickle.loads(p)

Can I pickle a python dictionary into a sqlite3 text field?

Any gotchas I should be aware of? Can I store it in a text field, or do I need to use a blob? (I'm not overly familiar with either pickle or sqlite, so I wanted to make sure I'm barking up the right tree with some of my high-level design ideas.)

Python ValueError: insecure string pickle

PWhen I am trying to load something I dumped using cPickle, I get the error message:

ValueError: insecure string pickle

Both the dumping and loading work are done on the same computer, thus same OS: Ubuntu 8.04.

How could I solve this problem?

What is the difference between pickle and shelve?

I am learning about object serialization for the first time. I tried reading and 'googling' for differences in the modules pickle and shelve but I am not sure I understand it. When to use which one? Pickle can turn every python object into stream of bytes which can be persisted into a file. Then why do we need the module shelve? Isn't pickle faster?

Pickle or json?

I need to save to disk a little dict object which keys are strings and values are ints and then recover it. Something like this:

{'juanjo': 2, 'pedro':99, 'other': 333}

Which and why is the best option? Serialize it with pickle or with simplejson?

I'm using Python 2.6

How can I use pickle to save a dict?

I have looked through the information that the Python docs give, but I'm still a little confused. Could somebody post sample code that would write a new file then use pickle to dump a dictionary into it?

Can't pickle when using python's multiprocessing Pool.map()

I'm trying to use multiprocessing's Pool.map() function to divide out work simultaneously. When I use the following code, it works fine:

import multiprocessing

def f(x):
    return x*x

def go():
    pool = multiprocessing.Pool(processes=4)        
    print pool.map(f, range(10))


if __name__== '__main__' :
    go()

However, when I use it in a more object-oriented approach, it doesn't work. The error message it gives is:

PicklingError: Can't pickle <type 'instancemethod'>: attribute lookup
__builtin__.instancemethod failed

This occurs when the following is my main program:

import someClass

if __name__== '__main__' :
    sc = someClass.someClass()
    sc.go()

and the following is my someClass class:

import multiprocessing

class someClass(object):
    def __init__(self):
        pass

    def f(self, x):
        return x*x

    def go(self):
        pool = multiprocessing.Pool(processes=4)       
        print pool.map(self.f, range(10))

Anyone know what the problem could be, or an easy way around it?

Multiprocessing: using Pool.map on a function defined in a class

when i run something like

from multiprocessing import Pool
p = Pool(5)
def f(x):
     return x*x
p.map(f, [1,2,3])

it works fine. However, putting this as a function of a class

class calculate(object):
 def run(self):
  def f(x):
   return x*x

  p = Pool() 
  return p.map(f, [1,2,3])

cl = calculate()
print cl.run()

gives me the following error.

Exception in thread Thread-1:
Traceback (most recent call last):
  File "/sw/lib/python2.6/threading.py", line 532, in __bootstrap_inner
    self.run()
  File "/sw/lib/python2.6/threading.py", line 484, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/sw/lib/python2.6/multiprocessing/pool.py", line 225, in _handle_tasks
    put(task)
PicklingError: Can't pickle <type 'function'>: attribute lookup __builtin__.function failed

I've seen a post from Alex Martelli dealing with the same kind of problem, but it wasn't explicit enough.

Common use-cases for pickle in Python

I've looked at the pickle documentation, but I don't understand where pickle is useful.

What are some common use-cases for pickle?

Is there an easy way to pickle a python function (or otherwise serialize its code)?

I'm trying to transfer a transfer a function across a network connection (using asyncore). Is there an easy way to serialize a python function (one that, in this case at least, will have no side affects) for transfer like this?

I would ideally like to have a pair of functions similar to these:

def transmit(func):
    obj = pickle.dumps(func)
    [send obj across the network]

def receive():
    [receive obj from the network]
    func = pickle.loads(s)
    func()

Using pickle.dump - TypeError: must be str, not bytes

I'm using python3.3 and I'm having a cryptic error when trying to pickle a simple dictionary.

Here is the code:

import os
import pickle
from pickle import *
os.chdir('c:/Python26/progfiles/')

def storvars(vdict):      
    f = open('varstor.txt','w')
    pickle.dump(vdict,f,)
    f.close()
    return

mydict = {'name':'john','gender':'male','age':'45'}
storvars(mydict)

and I get:

Traceback (most recent call last):
  File "C:/Python26/test18.py", line 31, in <module>
    storvars(mydict)
  File "C:/Python26/test18.py", line 14, in storvars
    pickle.dump(vdict,f,)
TypeError: must be str, not bytes

Python multiprocessing pickling error

I am sorry that I can't reproduce the error with a simpler example, and my code is too complicated to post. If I run the program in IPython shell instead of the regular python, things work out well.

I looked up some previous notes on this problem. They were all caused by using pool to call function defined within a class function. But this is not the case for me.

Exception in thread Thread-3:
Traceback (most recent call last):
  File "/usr/lib64/python2.7/threading.py", line 552, in __bootstrap_inner
    self.run()
  File "/usr/lib64/python2.7/threading.py", line 505, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/lib64/python2.7/multiprocessing/pool.py", line 313, in _handle_tasks
    put(task)
PicklingError: Can't pickle <type 'function'>: attribute lookup __builtin__.function failed

I would appreciate any help.

UPDATE: The function I pickle is defined at the top level of the module. Though it calls a function that contains a nested function. i.e, f() calls g() calls h() which has a nested function i(), and I am calling pool.apply_async(f). f(), g(), h() are all defined at the top level. I tried simpler example with this pattern and it works though.

Python serialization - Why pickle?

I understood that Python pickling is a way to 'store' a Python Object in a way that does respect Object programming - different from an output written in txt file or DB.

Do you have more details or references on the following points:

• where are pickled objects 'stored'?
• why is pickling preserving object representation more than, say, storing in DB?
• can I retrieve pickled objects from one Python shell session to another?
• do you have significant examples when serialization is useful?
• does serialization with pickle imply data 'compression'?

In other words, I am looking for a doc on pickling - Python.doc explains how to implement pickle but seems not dive into details about use and necessity of serialization.

How to hash a large object (dataset) in Python?

I would like to calculate a hash of a Python class containing a dataset for Machine Learning. The hash is meant to be used for caching, so I was thinking of md5 or sha1. The problem is that most of the data is stored in NumPy arrays; these do not provide a __hash__() member. Currently I do a pickle.dumps() for each member and calculate a hash based on these strings. However, I found the following links indicating that the same object could lead to different serialization strings:

• Hash of None varies per machine
• Pickle.dumps not suitable for hashing

What would be the best method to calculate a hash for a Python class containing Numpy arrays?

How to pickle a namedtuple instance correctly

I'm learning how to use pickle. I've created a namedtuple object, appended it to a list, and tried to pickle that list. However, I get the following error:

pickle.PicklingError: Can't pickle <class '__main__.P'>: it's not found as __main__.P

I found that if I ran the code without wrapping it inside a function, it works perfectly. Is there an extra step required to pickle an object when wrapped inside a function?

Here is my code:

from collections import namedtuple
import pickle

def pickle_test():
    P = namedtuple("P", "one two three four")
    my_list = []
    abe = P("abraham", "lincoln", "vampire", "hunter")
    my_list.append(abe)
    f = open('abe.pickle', 'w')
    pickle.dump(abe, f)
    f.close()

pickle_test()