February 24, 2018

Python with statement and context managers

By Mohammed Abualrob Code Snippets 0 Comments

Introduction

Welcome to a new code snippets post. Today, we are going to summarize the use of with statement and context managers in Python. In coding, a typical scenario is to acquire a resource, use it and finally release the resource. If an exception occurs, the program may not have a chance to free the allocated resource (i.e. a leak). In most programming languages, the solution is to use a try catch clause. In Python, we can use the same technique (i.e. try catch clause) however, using with statement is a better alternative. It eliminates extra lines of code and makes sure the resource is freed automatically even in the case of exceptions. So, how does Python implement that? The answer is context managers…

Context managers

Behind the scenes, Python with statement uses a context manager. The keyword context refers to a resource (ex. file, socket, etc) context. Context manager is simply a class that implements the context manager protocol or a function that returns a context manager object. To implement the context manager protocol, define a class which implements the enter and exit methods such that…

Implement __enter__ method. This method should return the resource that we are tying to acquire
Implement __exit__ method. This method should release the acquired resource

We will demonstrate how to implement our own context manager later. Let us now describe the with statement syntax…

With statement syntax

The with statement syntax can be described as follows…

The keyword with at the beginning followed by
class or function that returns a context manager object followed by
The keyword as followed by
An alias for the returned resource followed by
The with statement body inside which we do something with the resource
The moment the body is out of scope, the cleanup part is executed automatically

Here is how the with statement syntax looks like…

with context_manager_object as resource_object:
   resource_object.do_something()

1 2	with context_manager_object as resource_object: resource_object.do_something()

For example, writing to a file…

with open('data.txt', 'w') as F
   F.write('data')

1 2	with open('data.txt', 'w') as F F.write('data')

Which is equivalent to…

# Try to write to a file
try:
    # Open a file for writing
    F = open('data.txt','w')
    # Write some text
    F.write('data')

# Notify the user if something went 
# wrong while writing to the file
except IOError:
    print('An error occured while trying to write to the file')

# If any other error occurs    
except:
    print('An error occured'):

# Close the file    
finally:
    F.close()

# Try to write to a file

try:

# Open a file for writing

F = open('data.txt','w')

# Write some text

F.write('data')

# Notify the user if something went

# wrong while writing to the file

except IOError:

print('An error occured while trying to write to the file')

# If any other error occurs

except:

print('An error occured'):

# Close the file

finally:

F.close()

Context manager example

In this example, we are going to implement a dummy context manger and use the with statement. Take a look…

# An example context manager class
class ContextManager():
    # Class constructor
    # Our resource is a simple string
    def __init__(self, message):
        self.message = message
        
    # Setup stage, it should return the resource
    # that we want to aqcuire
    def __enter__(self):
        print('Step 1: preparing...')
        return self.message

    # Teardown stage. Typically, you handle exceptions
    # here
    def __exit__(self, exception, value, traceback):
        print('Step 3: done!')

# Test our context manager
with ContextManager('Step 2: work in progress...') as resource:
    print(resource)

# An example context manager class

class ContextManager():

# Class constructor

# Our resource is a simple string

def __init__(self, message):

self.message = message

# Setup stage, it should return the resource

# that we want to aqcuire

def __enter__(self):

print('Step 1: preparing...')

return self.message

# Teardown stage. Typically, you handle exceptions

# here

def __exit__(self, exception, value, traceback):

print('Step 3: done!')

# Test our context manager

with ContextManager('Step 2: work in progress...') as resource:

print(resource)

If you run the code snippet above, you should get the following output…

Step 1: preparing...
Step 2: work in progress...
Step 3: done!

Step 1: preparing...

Step 2: work in progress...

Step 3: done!

Sounds like a good idea! Let us mention few candidate scenarios to use the with statement…

When to use with statement?

Any pair of operations that can be done together such as preparation & cleanup, setup & teardown, etc. are good candidates for using a with statement. A typical example is opening a file, writing to then closing the file. There are countless other examples that are good fit for the with statement. To mention a few…

Reading from and writing to files
Networking and sockets
Database access
Locks in multithreaded applications

with statement example

The typical example of opening a file was demonstrated earlier, let us take one more example…

Database connection

Database connection is a good example on using context managers and with statement as it involves a setup stage (i.e. connection) and clean stage (i.e. commit and close). Take a look…

# Import sqlite module
import sqlite3

# DB connection context manager
class database():
    
    # Initialize with path to database
    def __init__(self, path):
        self.path = path
 
    # On enter connect to database
    # Return cursor
    def __enter__(self):
        self.conn = sqlite3.connect(self.path)
        return self.conn.cursor()

    # On exit commit work then close
    def __exit__(self, exc_class, exc, traceback):
self.conn.commit()
        self.conn.close()

# Test database context manager
with database('example.db') as csr:
    csr.execute("CREATE TABLE students (id text, name text, grade int)")
    csr.execute("INSERT INTO students VALUES ('ID1', 'Mark', 90)")
    csr.execute("INSERT INTO students VALUES ('ID2', 'John', 95)")
    csr.execute("SELECT * FROM students")

    for row in csr.fetchall():
        print(row)

# Import sqlite module

import sqlite3

# DB connection context manager

class database():

# Initialize with path to database

def __init__(self, path):

self.path = path

# On enter connect to database

# Return cursor

def __enter__(self):

self.conn = sqlite3.connect(self.path)

return self.conn.cursor()

# On exit commit work then close

def __exit__(self, exc_class, exc, traceback):

self.conn.commit()

self.conn.close()

# Test database context manager

with database('example.db') as csr:

csr.execute("CREATE TABLE students (id text, name text, grade int)")

csr.execute("INSERT INTO students VALUES ('ID1', 'Mark', 90)")

csr.execute("INSERT INTO students VALUES ('ID2', 'John', 95)")

csr.execute("SELECT * FROM students")

for row in csr.fetchall():

print(row)

If you run the code snippet above, you should get the following output…

(u'ID1', u'Mark', 90)
(u'ID2', u'John', 95)

1 2	(u'ID1', u'Mark', 90) (u'ID2', u'John', 95)

Decorator based context manager

Context managers can also be implemented using decorators. For more information about decorators, refer to the references section. The @contextmanager decorator defined in the contextlib library can be used to decorate a generator function (refer to the references section for more information about generators) that calls yield exactly once. Everything before the yield statement is considered the code for enter. Everything after the yield statement is the code for exit. Here is an example on opening a file…

# Import context manager
from contextlib import contextmanager

# Decorate open file function 
# using context manager decorator
@contextmanager
def openFile(path, mode):
    # Open file as usual
    F = open(path, mode)
    # Return file handle as a resource
    yield F

    # Close the file
    F.close()

# Let us test it
with openFile('data.txt', 'w') as F:
    F.write('data’)

# Import context manager

from contextlib import contextmanager

# Decorate open file function

# using context manager decorator

@contextmanager

def openFile(path, mode):

# Open file as usual

F = open(path, mode)

# Return file handle as a resource

yield F

# Close the file

F.close()

# Let us test it

with openFile('data.txt', 'w') as F:

F.write('data’)

Nesting context managers

Context managers can be composed. Take a look at the following example…

# Define a context manager
class ContextManager():
    # Initialize with name
    def __init__(self, name):
        self.name = name

    # Print entering name
    def __enter__(self):
        print('Entering {}'.format(self.name))
        return self.name

    # Print exiting name
    def __exit__(self, exception, value, traceback):
        print('Exiting {}'.format(self.name))

# Let us test nesting
with ContextManager('CM1') as A:
    with ContextManager('CM2') as B:
        print(A)
        print(B)

# Define a context manager

class ContextManager():

# Initialize with name

def __init__(self, name):

self.name = name

# Print entering name

def __enter__(self):

print('Entering {}'.format(self.name))

return self.name

# Print exiting name

def __exit__(self, exception, value, traceback):

print('Exiting {}'.format(self.name))

# Let us test nesting

with ContextManager('CM1') as A:

with ContextManager('CM2') as B:

print(A)

print(B)

If you run the code snippet above, you should get the following output…

Entering CM1
Entering CM2
CM1
CM2
Exiting CM2
Exiting CM1

Entering CM1

Entering CM2

CM1

CM2

Exiting CM2

Exiting CM1

You can also use the following syntax…

with ContextManager('CM1') as A, ContextManager('CM2') as B:
        print(A)
        print(B)

with ContextManager('CM1') as A, ContextManager('CM2') as B:

print(A)

print(B)

and this…

with contextlib.nested(ContextManager('CM1'), ContextManager('CM2')) as (A, B):
    print(A)
    print(B)

with contextlib.nested(ContextManager('CM1'), ContextManager('CM2')) as (A, B):

print(A)

print(B)

That is it for today, let us summarize…

Summary

Python with statement can be used to acquire a resource, consume it and finally release the resource while eliminating extra lines of code and making sure the resource is freed automatically even in the case of exceptions
Python implements the with statement using a context manager. Context manager is simply a class that implements the context manager protocol or a function that returns a context manager object
The context manager protocol has an enter and exit method where setup and tear down operations take place
The syntax for the with statement in Python is as follows

with context_manager_object as resource_object:
   resource_object.do_something()

1 2	with context_manager_object as resource_object: resource_object.do_something()

Any pair of operations that can be done together such as preparation & cleanup, setup & teardown, etc. are good candidates for using a with statement. The typical example is opening a file, writing to then closing the file
The with statement in Python can be implemented using either a class or a decorator
The with statement in Python can be nested as well

References

Thanks for reading. Please comeback…

8 BIT AVENUE

Python with statement and context managers

More from my site

About Author

Mohammed Abualrob

Add a Comment