Since the release of Python 2.2 in 2001, all Python functions have closed over bindings in outer scopes. Before this release variables referenced from outer scopes had to be passed in as arguments. A common workaround to this limiting behavior was to set these at function definition time with keyword arguments:
#!/usr/bin/env python2.0
def find(self, name):
return filter(lambda x, name=name: x == name, self.list_attribute)
Since 2001 this workaround is no longer necessary and the references
to name
in the body of the lambda function are resolved using the
current value in the outer find
function:
#!/usr/bin/env python2.2
def find(self, name):
return filter(lambda x: x == name, self.list_attribute)
However Python functions could not modify these bindings: although an object referred to by a variable from an outer scope can be modified, like appending an item to a list,
#!/usr/bin/env python2
def get_odds(candidates):
odds = []
def add_if_odd(x):
if x % 2 == 1:
l.append(x) # This type of modification is fine.
for x in candidates:
add_if_odd(x)
return odds
a function could not change to which object a variable from an outer scope referred.
#!/usr/bin/env python2
def largest_odd(candidates):
highest = 1
def replace_if_odd(x):
if x % 2 == 1:
highest = x # Reassigning highest does not work!
# This creates a local variable instead.
for x in sorted(candidates):
save_if_odd(x)
return highest
One of my favorite changes in Python 3 is the addition of the nonlocal keyword. Described by PEP 31041 as “access to names in outer scopes” and in other places as “rebinding closures” and “read/write closures,” this allows functions to rebind variables in outer scopes. By marking a variable name nonlocal we specify that when that variable name is assigned to in this function, we mean the already existing binding from outside this function. The nonlocal keyword works analogously to the global keyword which allows reassignment of global variables in Python 2 and 3.
#!/usr/bin/env python3
def largest_odd(candidates):
highest = 1
def replace_if_odd(x):
nonlocal highest # Nonlocal only exists in python 3.
if x % 2 == 1: # Because of the previous line
highest = x # this rebinds the outer highest.
for x in sorted(candidates):
save_if_odd(x)
return highest
I think this is a great addition that helps me write code the way I would in JavaScript or a Lisp. But despite this lack of rebinding closures in Python 2 frustrating some programmers coming from these and other languages, it doesn’t seem to be a big deal to Python programmers. The feature doesn’t make the top 11 of Aaron Meurer’s top 10 awesome features of Python that you can’t use because you refuse to upgrade to Python 3.
Folks who write Python code, what do you think of the new nonlocal keyword in Python 3?
— Thomas Ballinger (@ballingt) January 5, 2016
Here are some thoughts on why this situation doesn't come up
very much in Python and how people worked around not having
it in Python 2.
Global scope isn’t
Python 2 functions can rebind some variables:
the module-level variables known as globals.
global
, available in Python 2, marks a variable name
in a function as referring to the global variable in that module.
This provides some fraction of the power of full closures
since every Python function closes over the “global” variables
of the module in which it was defined.
I also find I write less local functions in Python because of
the built-in module system than I do in JavaScript, and these
top-level functions have no enclosing scope besides the global
scope to use nonlocal
on.
Community norm: no data hiding
An emblematic use of closures in JavaScript is the protection of private data, but in Python data privacy isn’t enforced.
While using a closure to hide data is a common pattern in JavaScript,
function createPerson(name){
var age = 10;
return {
birthday : function(){
age = age + 1;
},
greet : function(){
console.log("Hi, I'm", name, "and I'm", age, "years old");
}
}
}
var me = createPerson('Tom')
me.greet()
that same pattern in Python 3 would be considered pretty weird code:
#!/usr/bin/env python3
class Person(object):
pass
@staticmethod
def create(name):
age = 10
p = Person()
def birthday():
nonlocal age
age = age + 1
def greet():
print("Hi, I'm", name, "and I'm", age, "years old")
p.birthday = birthday
p.greet = greet
return p
me = Person.create('Tom')
me.greet()
Mutable object hack
Writing to a closed over variable is useful for producing a callback which records somewhere that it was called. Here’s a signal handing example in Python 3:
#!/usr/bin/env python3
from signal import signal, SIGWINCH
def notice_window():
window_size_changed = False
def handler(signum, frame):
nonlocal window_size_changed
window_size_changed = True
signal(SIGWINCH, handler)
while True:
if window_size_changed:
window_size_changed = False
print('window size has changed')
The most direct translation of the above is to use
the simplest possible mutable object to store
the boolean window_size_changed
value.
Read-only closures with mutable objects seem
to provide an (ugly) equivalent to full closures:
#!/usr/bin/env python2
from signal import signal, SIGWINCH
def notice_window_with_mutable_object():
window_size_changed = [False]
def handler(signum, frame):
window_size_changed[0] = True
signal(SIGWINCH, handler)
while True:
if window_size_changed[0]:
window_size_changed[0] = False
print('window size has changed')
This works in every case I can think of, but is ugly because the list only obfuscates what’s going on. When I see this pattern in Python 2 code I wish it were Python 3 so it could be fixed, but it seems pretty rare.
Mutability and method binding:
It’s much more common to find a function that could rebind an outer variable, but instead mutates an object more descriptive than a one-element list.
When passing in a callback to a function a Python method is particularly convenient because that method will be bound to that object at attribute lookup time. This more clearly communicates what state might be modified because methods of objects often change the state of the objects to which the belong.
#!/usr/bin/env python2
class Noticer:
def __init__(self):
self.window_size_changed = False
def handler(self, signum, frame):
self.window_size_changed = True
def notice_window_with_method():
noticer = Noticer()
signal(SIGWINCH, noticer.handler)
while True:
if noticer.window_size_changed:
noticer.window_size_changed = False
print('window size has changed')
Less callback-oriented interfaces
Python codebases seem to prefer passing objects that conform to informal interfaces to passing functions directly, using attributes of that object for the state being changed.
Further reading:
- PEP 3104: Access to Names in Outer Scopes
- PEP 227: Statically Nested Scopes
- Stack Overflow posts about workarounds for Python 2: 1 2 3 4 5
Thanks Julia Evans and Lindsey Kuper for comments and corrections.
-
Python enhancement proposals are great reading! They summarize mailing list discussion by a lot of smart people with Python experience considering every angle of a new feature. I recently listened to a
Podcast.__init__
episode about PEPs that was fun and informative too. ↩︎