*JJ

A decorator decorator for Django views

2014-06-26T02:06:06-04:00

Hitting a new level of meta, I just wrote a decorator decorator for Django views.

The context here is that I needed to write some custom view decorators that gave me access to the request object, since the normal built-in decorators like user_passes_test only offer access to the request.user object. While doing that, I wanted them to be able to apply to both function- and class-based views without having to use something like the method_decorator decorator, but still having it know to wrap dispatch on a class-based view unless otherwise specified.

Enter the decorator decorator:

from functools import wraps
from types import FunctionType

def class_or_function_view_decorator(class_method_to_wrap='dispatch'):
    '''
    A decorator that enables other decorators to decorate both function-
    and class-based views.  Can also be used on class-based views' methods.

    Usage:
        @class_or_function_view_decorator()
        def sayhello(f):
           """
           A decorator that says hello before the function is called.
           When used on a class, it will wrap the class's `dispatch` method.

           Usage:
                @sayhello
                def saybye():
                    print 'bye!'

                @sayhello
                def MyClass(object):
                    def dispatch(self):
                        print 'bye!'
           """
           @functools.wraps(f)
           def wrapped(*args, **kwargs):
                print 'hello!'
                return f(*args, **kwargs)
            return wrapped

        @class_or_function_view_decorator('__init__')
        def saybye(f):
            """
            When this decorator is used on a class, it will wrap the class's `__init__` method,
            unlike the previous example which wrapped `dispatch`.
            """
            @functools.wraps(f)
            def wrapped(*args, **kwargs):
                x = f(*args, **kwargs)
                print 'bye!'
                return x
            return wrapped

    This decorator is used to decorate a view decorator to enable it to be used on
    function-based views or on class-based views.  If the resulting decorator is used on class-based views, you can
    choose which method of the class it will decorate, defaulting to 'dispatch' if not provided.
    If used on function-based views, the resulting decorator works as normal.

    '''
    def outside_wrapper(f):
        @wraps(f)
        def wrapped(to_be_wrapped):
            if type(to_be_wrapped) is FunctionType:
                return f(to_be_wrapped)
            else:
                setattr(to_be_wrapped, class_method_to_wrap, f(getattr(to_be_wrapped, class_method_to_wrap)))
                return to_be_wrapped
        return wrapped
    return outside_wrapper

In short, this allows me to create a single decorator that can be used anywhere, like so:

@class_or_function_view_decorator()
def request_passes_test(test_func, login_url=None):
    """
    Decorator for views that checks that the session passes the given test,
    redirecting to the log-in page if necessary. The test should be a callable
    that takes the request object and returns True if the test passes.
    """

    def decorator(view_func):
        @wraps(view_func, assigned=available_attrs(view_func))
        def _wrapped_view(request, *args, **kwargs):
            if test_func(request):
                return view_func(request, *args, **kwargs)
            else:
                if login_url:
                    return redirect(login_url)
                else:
                    raise PermissionDenied()
        return _wrapped_view
    return decorator

How to destroy your Python program

2014-06-06T22:38:13-04:00

I discovered today that Python’s True and False are actually constants—which means that their values can be changed. While this has been disallowed in Python 3, it still allows you to wreak some havoc in your—or someone else’s—Python 2 programs.

Definitely don’t do this

I wrote a tiny module that, when imported, will reverse your boolean values for the entirety of that Python instance—not just that namespace.

# destroybooleans.py

# For some reason, in Python 2.x True and False are both constants--meaning that
#   they can have new values assigned to them.
# This malicious script exploits that.  Upon import, will reverse
#   True and False for all modules in the current Python environment.
# This obviously can have serious consequences.

import sys

# If we reverse them by still using the `bool` type, checking if type(False)==bool
#   won't help us catch this
sys.modules['__builtin__'].True = bool(0)
sys.modules['__builtin__'].False = bool(1)

def __modules():
    return sys.modules

def faux_method():
    print "I did some nasty stuff on my way into your runtime >:D"

faux_method()

This obviously could be maliciously hidden in a larger package meant to have other functionality in order to mess up your code. I imagine that it would be difficult to notice in part due to Python’s otherwise very convenient truthiness and falsiness behavior. This module does not affect [] and {}, for example, being false-y, nor does it prevent most objects from being truthy. This would only affect explicit calls to True and False.

This leads to some rather odd behavior:

>>> import destroybooleans
I did some nasty stuff on my way into your runtime >:D
>>> True
False
>>> False
True
>>> bool(1)
True
>>> bool([])
False

Text-to-Columns in Google Docs Spreadsheets (with Python)

2014-06-01T19:11:00-04:00

Despite being an Excel power user, I’m currently without Excel on my computer for the time being so for some data analysis I had to do I had to make do with Google Docs’ spreadsheets. While decent, and impressive for a web application, unfortunately it’s not great for dealing with large amounts of data. Additionally, it lacks some glaring functionality, particularly a “Text to Columns” utility, which is useful for converting in-line a csv-like delimited string of data into individual cells. While there are a couple of third-party-written add-ins for this, they don’t work very well.

My Fix

The best way I found to deal with this involved a couple of manual steps, but saved me a lot of time overall. The process was effectively the following:

Copy cell with data from spreadsheet
Paste into Python
Process with Python, returning string ready to be copy/pasted back into Google Docs
Paste into Google docs in appropriate row

The biggest challenge here was preparing a string to return that Google Docs would automatically paste into the corresponding columns as appropriate rather than a single cell. I figured out the right formatting by analyzing exactly what the contents of my clipboard were when I copied multiple columns from the spreadsheet. It turned out that the big secret was that the data was simply tab-delimited, so all I had to do was generate tab-delimited strings to paste back into Google Docs.

I used Python specifically in order to take advantage of the excellent csv built-in module, which is great at parsing all kinds of oddly formatted and delimited files.

Code you can use would be somewhat as follows:

import csv, StringIO

def text_to_columns(text):
    '''
    A function that takes in a single row of delimited data from a spreadsheet, and returns a tab-delimited string response meant to be copied and pasted back into the spreadsheet to be automatically resolved into the appropriate columns.  Note that you will likely have to `print` the return string in order to copy the tabs correctly.
    '''

    # We need to use StringIO here because csv.reader only reads buffers
    reader = csv.reader(StringIO.StringIO(text))
    print '\t'.join(reader.next())

I copied the appropriate cell from the spreadsheet, pasted into the console like so:

>>> t = '''<paste>'''<enter>

Or for multiple lines, for convenience, like so:

>>> t = ['''<paste>''',
         '''<paste>''',
         ...]

And then I ran them all through the function:

>>> for item in t:
        text_to_columns(item)

And then I copied the printed result.

Note that I used print rather than simply returning the result because I needed output that correctly rendered the tabs rather than rendering \t, which is Python’s in-language way of displaying tabs. Google Docs would only correctly place the data into separate columns if I had copied actual tabs.

Extending this solution

The true best way of extending this would be to find something as good as Python’s csv.reader in Javascript and building a Google Apps Script with it.

Otherwise, there is likely a way to have Python pull the contents of the clipboard and then place the results of the function in the clipboard. If you know of how to do this, I’d be interested to hear how.

Update

I ran across this excellent blog post that describes some useful strategies and libraries for trying to access the clipboard from Python. I haven’t tried it yet, but it’s worth looking into. Note that if you use pyobjc it takes a very long time to compile if you install using pip.

Ruby-like string interpolation in Python

2014-05-29T18:18:19-04:00

I’ve been experimenting with Rails a bit for the past couple weeks. This has been pretty exciting for me, as I’ve generally been much more of a Python developer and I’d never touched Ruby before, so it was very interesting to see a completely different approach to programming and language philosophy.

While the lack of parentheses and many conventions that you “just have to learn” sometimes frustrate me, at least from my first couple of weeks with the language, I’ve seen quite a few features that I like and somewhat wish I had available to me in Python. One of these is string interpolation.

String interpolation: To replace a variable with its value within a string.

Ruby

Ruby implements string interpolation like this:

2.1.1 :001 > stringvar = "I'm a string"
 => "I'm a string"
2.1.1 :002 > numvar = 3
 => 3
2.1.1 :003 > "This is stringvar: #{stringvar}, and this is numvar: #{numvar}"
=> "This is stringvar: I'm a string, and this is numvar: 3"

Note that it only works with double-quotes (") which are “weaker.” If you try it with single-quotes, you’ll get the following:

2.1.1 :004 > puts 'This is stringvar: #{stringvar}, and this is numvar: #{numvar}'
This is stringvar: #{stringvar}, and this is numvar: #{numvar}

My understanding is that you can do this with anything accessible within the local scope, so that would include class or global variables (e.g., ones beginning with @). The flexibility of string interpolation goes even farther, though, as you can actually call methods and perform other arbitrary operations within the interpolation:

2.1.1 :010 > "Upper case: #{stringvar.upcase}"
 => "Upper case: I'M A STRING"
2.1.1 :011 > "Addition: #{numvar + 4}"
 => "Addition: 7"
2.1.1 :012 > def add10(num)
2.1.1 :013?>   num + 10
2.1.1 :014?> end
 => :add10
2.1.1 :015 > "Method call: #{add10 numvar}"
 => "Method call: 13"

Clearly powerful, but could also be somewhat dangerous.

Python

Python, with the philosophy “explicit is better than implicit”, does not have this string interpolation in quite the same way—you need to pass in what you’d like to interpolate explicitly.

The old syntax for this (and still used by many today) is the following:

>>> "This is a string %s, or a number: %d, or a number as a string: %s" % \
...    ('arbitrary string', 3, 39)
'This is a string arbitrary string, or a number: 3, or a number as a string: 39
>>> "Example with formatting: %.4f" % 3
'Example with formatting: 3.0000'
>>> "Named arguments: %(hey)s, %(name)s" % {'hey': 'Hey', 'name': 'Jason'}
'Named arguments: Hey, Jason'

The new style is a .format method called directly on the string:

>>> "Lots of {} to {verb} here, like binary conversion: {num:b}".format(
        'examples', verb='show', num=3
    )
'Lots of examples to show here, like binary conversion: 11'

And you are also able to access objects’ attributes or items, like so:

>>> "Item 1: {0[1]} | Dictionary 'key': {1[key]}".format(['a', 'b'], {'key': 'value'})
"Item 1: b | Dictionary 'key': value"

Note that above, we’re calling the different arguments positionally, so {0} refers to the first positional argument to format and {1} to the second (Python is 0-indexed!), and then we’re accessing internals via brackets.

You can’t call methods:

>>> def add1(num):
...   return num + 1
...
>>> "Test: {add1(test)}".format(test=3)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
KeyError: 'add1(test)'

There are lots of other good examples that can be found in the Python docs here.

Accessing variables in the immediate scope

Unfortunately, you can’t do what Ruby does in terms of accessing any variable or value in the string’s scope. This makes sense from an explicitness perspective (we know exactly what we’re passing to the string at all times), but sometimes it is convenient to not have to worry about that and to be able to type quickly.

The best way I found to do this in Python is to pass in the locals() dictionary, which is a dictionary of the local scope.

>>> a = 1
>>> b = "hey"
>>> "This is a: {a} | This is b: {b}".format(**locals())
'This is a: 1 | This is b: hey'
>>> locals()
{'a': 1, 'b': 'hey', '__builtins__': <module '__builtin__' (built-in)>, '__package__': None, '__name__': '__main__', '__doc__': None}

It’s a bit less “Pythonic,” but ultimately it still gets the job done.

Within classes

This doesn’t really work quite so well in classes and starts to become unwieldy. As you’ll see below, accessing the class instance’s attributes don’t work with locals() because they’re all actually attached to self within the methods.

>>> class Test(object):
...     a = 3
...     def __init__(self):
...         self.b = 'hey'
...         print locals()

>>> t = Test()
{'self': <__main__.Test at 0x10e978e50>}

Ultimately, if we really wanted to, we could still access them within the string this way:

>>> class Test(object):
>>>     a = 3
>>>     b = 'hey'
>>>     def stringinterp(self):
>>>         return "a: {0.a}, b: {0.b}".format(self)

>>> Test().stringintepr()
'a: 3, b: hey'

But at that point, I don’t think it would be very convenient, and would effectively be defeating the purpose of string interpolation anyway.

Killing a Python WAT: Forcing immutable default arguments

2014-04-26T00:00:00-04:00

Fellow Hacker Schooler Amy had a great post today on mutable default arguments in Python, a very typical Python “WAT”.

(If you haven’t seen the CodeMash 2012 WAT video, you should.)

In this case, Amy wrote about how unintuitively in Python, when creating a function with default arguments, those arguments are actually mutable and preserved, meaning any mutations made to the default arguments remain that way for all future calls to that function. We just often don’t see this behavior because most of the time numbers and strings are used as default arguments, which by implementation are immutable and so aren’t given a chance to exhibit this behavior.

The example she used:

def foo(l=[]):
    l.append('cat')
    return l

What happens when you call foo repeatedly? You’d expect l to be reinstantiated every time with a new empty list, but instead, you get the following:

>>> foo()
['cat']
>>> foo()
['cat', 'cat']
>>> foo()
['cat', 'cat', 'cat']

Ultimately, this is because the default argumetns for the foo function are stored in foo.func_defaults, which contains our empty list in question. The list is only instantiated once, and since lists are mutable, every time we call l.append, we’re mutating the actual default for future calls to the function.

The only reason this doesn’t occur when we use number or string defaults is because they’re always immutable, so even if we make a change that looks like a mutation, we’re really getting a new object.

def new_foo(a=3):
    a += 1
    # we're actually getting a new object 4
    return a

Read the rest of her post for more on why this is the case.

This got me thinking that it must be possible to write a decorator that would preserve the original default arguments, even in mutable cases. Basically, what we’d be doing is intercepting Python’s reference to <function>.func_defaults with our own.

So what would this look like?

Make the decorator

First, let’s get our basic decorator structure out:

def enforce_defaults(function):

    def wrapper(*args, **kwargs):
        return function(*args, **kwargs)

    wrapper.func_name = function.func_name

    return wrapper

Getting our default arguments

Next, we’d like to grab the default arguments and variable names from our to-be-decorated function. We know already we can get the defaults from function.func_defaults, and we can grab the variable names from function.func_code.co_varnames. We also will grab the number of arguments for the function from function.func_code.co_argcount.

def enforce_defaults(function):
    varnames = function.func_code.co_varnames
    defaults = function.func_defaults
    argcount = function.func_code.co_argcount

    def wrapper(*args, **kwargs):
        return function(*args, **kwargs)

    wrapper.func_name = function.func_name

    return wrapper

The strategy we’re going to use here is to build up a dictionary of all the arguments that we’re eventually going to pass to the decorated function. We can’t know ahead of time which arguments will and won’t be passed, so we’ll start with a dictionary of the default arguments, and replace them with the updated, user-supplied args as we go along.

function.func_defaults only returns the defaults to us, though, not a full array we can use to match to the full list of varnames. For example, for function bas:

def baz(a, b, c=[], d=4):
    pass

We get the following:

>>> baz.func_code.co_varnames
('a', 'b', 'c', 'd')
>>> baz.func_defaults
([], 4)

So we’ll have to match our func_defaults tuple to the end of our varnames list, so that we match up our defaults with the right args. We can do that like this:

>>> varnames = baz.func_code.co_varnames
>>> defaults = baz.func_defaults
>>> varnames[-len(defaults):]
('c', 'd')
>>> matched = zip(varnames[-len(defaults):], defaults)
>>> matched
[('c', []), ('d', 4)]
>>> dict(matched)
{'c': [], 'd': 4}

So there we have our initial args dictionary with our default arguments:

def enforce_defaults(function):
    varnames = function.func_code.co_varnames
    defaults = function.func_defaults
    argcount = function.func_code.co_argcount

    def wrapper(*args, **kwargs):
        # "holder" is where we're storing our args
        holder = dict(zip(varnames[-len(defaults):], defaults))

        return function(*args, **kwargs)

    wrapper.func_name = function.func_name

    return wrapper

Other variables in the function?

Unfortunately, Python functions are going to store all of the function variables in func_code.co_varnames, including ones declared in-function. For example:

def test(a, b=3):
    c = 4

>>> test.func_code.co_varnames
('a', 'b', 'c')
>>> test.func_defaults
(3,)
>>> test.func_code.co_argcount
2

To deal with that scenario, because the internally declared variables are stuck on at the end, we’ll have to make a slight modification to our holder variable. Since we know from argcount how many arguments they are, we can go back to varnames and simply just chop off any extra appended variables before doing our trick with [-len(defaults):].

holder = dict(zip(varnames[:argcount][-len(defaults):], defaults))

Oh no!

Looks like what we’ve done isn’t helping at all! If we stick a print statement in there to debug, you can see what’s happening:

def enforce_defaults(function):
    varnames = function.func_code.co_varnames
    defaults = function.func_defaults
    argcount = function.func_code.co_argcount

    def wrapper(*args, **kwargs):
        # "holder" is where we're storing our args
        holder = dict(zip(varnames[:argcount][-len(defaults):], defaults))

        print holder

        return function(*args, **kwargs)

    wrapper.func_name = function.func_name

    return wrapper

@enforce_defaults
def foo(l=[]):
    l.append('cat')
    return l

# >>> foo()
# {'l': []}
# ['cat']
# >>> foo()
# {'l': ['cat']}
# ['cat', 'cat']

And after all that work! Why is this happening to us?

Right now, we’re accessing our defaults for holder from the defaults variable, but that’s actually just a direct reference to the original function.func_defaults, and so we’re just mutating the originals like before. In short, we haven’t actually changed anything.

We can make sure we’re always getting new args rather than our old mutating ones by using copy.deepcopy.

def enforce_defaults(function):
  import copy
  varnames = function.func_code.co_varnames
  defaults = function.func_defaults
  argcount = function.func_code.co_argcount

  def wrapper(*args, **kwargs):
    inner_defaults = copy.deepcopy(defaults)
    holder = dict(zip(varnames[:argcount][-len(inner_defaults):], inner_defaults))

    print holder # for debugging

    return function(**holder)

  wrapper.func_name = function.func_name

  return wrapper

Now, we should be good to go:

# >>> foo()
# {'l': []}
# ['cat']
# >>> foo()
# {'l': []}
# ['cat']

Excellent! We’ve solved our mutating default arguments problem. Now, we just have to finish the decorator.

Accounting for user-supplied args

We’ll need to read in both *args and **kwargs and update our holder dictionary accordingly, to make sure if the user has supplied any updated arguments over our defaults, we don’t continue passing the defaults.

For **kwargs, it’s pretty easy; we simply update the dictionary:

holder.update(kwargs)

Since *args is ordered, we can simply use zip with the function’s argument names, similarly to how we did above:

holder.update(dict(zip(varnames, args)))

And then, we just have to update our calling of function, since we’re only using our specially curated dictionary:

return function(**holder)

…and voilà! We’re done! You can now use mutable objects in functions with our decorator without fear!

As final proof:

@enforce_defaults
def foo(l=[]):
    l.append('cat')
    return l

>>> foo()
['cat']
>>> foo()
['cat']
>>> foo()
['cat']

…and you could just do that all day.

Final code

def enforce_defaults(function):
  import copy
  varnames = function.func_code.co_varnames
  defaults = function.func_defaults
  argcount = function.func_code.co_argcount

  def wrapper(*args, **kwargs):
    inner_defaults = copy.deepcopy(defaults)
    holder = dict(zip(varnames[:argcount][-len(inner_defaults):], inner_defaults))
    holder.update(dict(zip(varnames, args)))
    holder.update(kwargs)

    return function(**holder)

  wrapper.func_name = function.func_name

  return wrapper

Re-implementing Clojure’s comp

2014-04-22T00:00:00-04:00

While working through Clojure for the Brave and True, I came across an explanation of Clojure’s comp function and some basic implementations of versions of comp that take only two arguments or three arguments. The tutorial challenged me to try to completely re-implement comp to accept an arbitrary number of function arguments, and that got me thinking that it would be a very good exercise to try in Clojure.

The book had an implentation of the two-argument comp that looked something like this;

(defn two-comp [f g]
  (fn [& args]
    (f (apply g args))))

I worked off of that as a base for a while, but kept running into trouble. I kept running into rather complicated functions-of-functions that seemed to be returning functions that operated on the input functions, rather than the actual arguments, which was not what we were looking for. It’s a bit hard to wrap your head around this, since it’s very meta and we’re talking about a function that has functions as arguments and returns a function of functions as its result.

I decided to take a stab at this in Python, my go-to language.

What I came up with ended up looking like this:

def clojure_comp(*args):
    def wrap_function(to_wrap, wrapper):
        def return_function(*inner_args):
            # Here is where we actually call the function
            return wrapper(to_wrap(*inner_args))
        return return_function

    return reduce(wrap_function, reversed(args))

This ended up working out very nicely, using string maniuplation as a test:

>>> clojure_comp(lambda x: x.upper(), lambda x: x.strip(), lambda x: x.replace('a', 'b'))(" abcdefgh   ")
'BBCDEFG'

So I gave this another shot in Clojure.

(defn mycomp [& args]
  "A function that takes an arbitrary number of functions and returns a function that 
  applies each of those functions, last first, to the input arguments."
  (defn wrapper-function [wrapped wrapper]
    (fn [& inner-args] (wrapper (apply wrapped inner-args)))
    )

  (reduce wrapper-function (reverse args))

  )

And it does work—

user=> ((mycomp clojure.string/trim clojure.string/lower-case) "   this IS a test   ")
"this is a test"

But ultimately, I think it might be a bit too Pythonic rather than right for Clojure. I don’t tend to see too many functions that declare another named function within themselves, and I also feel like this is the kind of function that might better use recursion or loop than reduce, which I feel like I’m falling back on as a replacement for Python’s convenient for iteration.

That being said, it could still be a valuable way of doing it. I recently got complimented by a Javascript developer for my use of reduce in some Javascript I was writing for the Sefaria project. He said reduce and map were functional elements of JS that even most seasoned Javascript developers don’t use, so he was surprised that I was—but to me, languages like Clojure make it seem quite natural to use, and much more elegant in many cases than iteration and mutation.