Context managers allow you to allocate and release resources precisely when you want to. The most widely used example of context managers is the statement.

Suppose you have two related operations which you’d like to execute as a pair, with a block of code in between. Context managers allow you to do specifically that.

When to use Context Manager

The main purpose of context managers is, as you might’ve guessed, resource management. What does this mean in practice? The most common example is opening files.

Opening a file consumes a limited resource called a file descriptor. If you try to open too many files at once, depending on your operating system, you may get an error or completely crash your program.

n_files = 1000000
files = []

for i in range(n_files):
    files.append(open('test.txt'))


# OSError: [Errno 24] Too many open files

To avoid file descriptor leakage (as presented above), we need to close the files after we’re done with them. Closing the files is done with the close() method.

n_files = 1000000
files = []

for i in range(n_files):
    f = open('test.txt')
    files.append(f)
    f.close()

# no errors, all correct!

This works totally fine if we have generally simple programs. Although, as our programs or our file manipulations get more complicated, determining when and how to close the files may get tricky.

In other programming languages, a common method to manage this is a try … except … finally block. In Python, we can utilize a context manager.

Essentially, the context manager guarantees that all necessary operations will happen at the right time. In the example with opening files, the context manager will close the file and release the file descriptor when we are finished working with the file.

with … as

Now that we know why we need to use context managers, let’s learn how to do that. A context manager is introduced by a with keyword followed by the context manager itself and the name of the variable. The basic syntax is the following:

# invoking a context manager
with statement as variable_name:
    ...

statement here is anything that acts like a context manager (meaning, it supports specific context manager methods). It can be either a custom made context manager or Python’s internal one. The following objects are some of the context managers in Python:

• File objects (and other streams like io.StringIO or io.BytesIO);
• Sockets;
• Locks and semaphores in the threading module;
• Database connections;
• Mock objects.

We can also nest this construction:

# nested context manager
with statement1 as var1:
    with statement2 as var2:
        # and so on
        ...

Most commonly, with … as statement is used when working with files. Let’s see how we can do that.

Working with files

A file object that we get when we use the open() function acts as a context manager, so we can use it as the statement part of the code. This is how it can be done:

with open('test.txt') as f:
    # work with the file
    ...

As you can see, it is very simple! It also allows us to shorten our code a little since we don’t need to explicitly close the file at the end.

Note that you CAN explicitly close the file within the with … as statement, it won’t be an error.
Now the situation with a million files, this is how it looks if we use context manager:

n_files = 1000000
files = []

for i in range(n_files):
    with open('test.txt') as f:
        files.append(f)

# OK!

Now, let’s look at a more realistic example. Suppose, we have a file with the movies directed by Quentin Tarantino named tarantino.txt. We want to read this file and print the titles:

with open('tarantino.txt', 'r', encoding='utf-8') as f:
    for line in f:
        
        print(line.strip())   #rid of newline symbols

We’ll get the following output:

#Reservoir Dogs
#Pulp Fiction
#Jackie Brown
#Kill Bill: Volume 1
#Kill Bill: Volume 2
#Grindhouse: Death Proof
#Inglorious Basterds
#Django Unchained
#The Hateful Eight
#Once Upon a Time in Hollywood

Now, imagine that we want to process these titles, say, make them all lowercase, and have it saved to a file. Here’s how it can be done:

with open('tarantino.txt', 'r', encoding='utf-8') as in_file, \
     open('tarantino_lowercase.txt', 'w', encoding='utf-8') as out_file:
    for line in in_file:
        out_file.write(line.lower())

The file tarantino_lowercase.txt that we’ve created in the process, will contain the titles of Tarantino movies written in lowercase.

Summary

In this topic, we’ve learned about context managers, special structures used for effective managing of resources. Basically, context managers help make sure that all necessary operations have been carried out. Context managers are usually introduced by a with … as statement.

In practice, you’ll most commonly encounter this in opening files. However, you can also create custom context managers for your own purpose.

Let’s try to solve the problem:

Create 10 files, named file1.txt, file2.txt and so on till file10.txt. The files should contain the number corresponding to their name. So, file1.txt should contain one line with number 1, file2.txt — one line with number 2 and so on.

for i in range(1, 11):
    with open(f'file{i}.txt', 'w') as f:
        f.write(str(i))