08 Exception Handling to Improve Program Stability

08 Exception Handling to Improve Program Stability #

Hello, I’m Jingxiao.

In today’s lesson, I’d like to talk to you about exception handling in Python. Like in other languages, exception handling is a common and important mechanism and coding practice in Python.

In my work, I have seen many cases where an engineer submitted code but forgot to handle exceptions in certain areas. Unfortunately, these exceptions occurred quite frequently, so shortly after pushing the code to the production environment, an urgent notification would be received – the server crashed.

If the situation is serious and has a significant impact on users, the engineer would have to attend a special meeting for self-reflection. It can be said that this is quite miserable. Such incidents are frequent and they teach us that understanding and handling exceptions in a program is particularly crucial.

Errors and Exceptions #

First of all, let’s understand what errors and exceptions are in Python. What is the relationship and difference between the two?

Generally speaking, there are at least two types of errors in a program, one is syntax error, and the other is exception.

Syntax error, as you may know, refers to the code you write that does not comply with the programming syntax and cannot be recognized or executed. For example, in the following example:

if name is not None
    print(name)

The colon is missing in the if statement, which does not comply with Python’s syntax, so the program will raise an error invalid syntax.

On the other hand, exceptions refer to cases where the program’s syntax is correct, can be executed, but encounters an error during execution and throws an exception. For example, in the following three cases:

10 / 0
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ZeroDivisionError: integer division or modulo by zero

order * 2
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'order' is not defined

1 + [1, 2]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: unsupported operand type(s) for +: 'int' and 'list'

These expressions are syntactically correct, but obviously, we cannot divide a number by zero; we cannot perform calculations with an undefined variable; and it is not acceptable to add an integer and a list.

Therefore, when the program reaches these points, it raises an exception and terminates execution. In the examples, ZeroDivisionError, NameError, and TypeError are three common types of exceptions.

Of course, there are many other exception types in Python, such as KeyError which indicates that a key is not found in a dictionary; FileNotFoundError which indicates that a request to read a file was made but the corresponding file does not exist, and so on. I will not go into detail on each one here, but you can refer to the relevant documentation for more information.

Handling Exceptions #

As mentioned earlier, if an exception is raised at some point in the program, the program will be terminated and exited. You may wonder if there is a way to keep the program running without terminating it. The answer is yes, and this is what we call exception handling. We usually use try and except to handle exceptions. For example:

try:
    s = input('please enter two numbers separated by comma: ')
    num1 = int(s.split(',')[0].strip())
    num2 = int(s.split(',')[1].strip())
    ... 
except ValueError as err:
    print('Value Error: {}'.format(err))

print('continue')
...

Here, the user is expected to input two integer numbers separated by a comma. After extracting the numbers, the program will perform further operations (note that the input function will convert the input to a string). If we input 'a,b', the program will raise the invalid literal for int() with base 10: 'a' exception and jump out of the try block.

Since the exception type raised by the program is ValueError, and it matches the exception type caught by the except block, the except block will be executed. It will output Value Error: invalid literal for int() with base 10: 'a' and print continue.

please enter two numbers separated by comma: a,b
Value Error: invalid literal for int() with base 10: 'a'
continue

We know that the except block only accepts exception types that match it and executes accordingly. If the exception raised by the program does not match, the program will still be terminated and exited.

So, using the previous example, if we only input '1', the exception raised by the program will be IndexError: list index out of range, which does not match ValueError, so the except block will not be executed, and the program terminates and exits (the continue is not printed).

please enter two numbers separated by comma: 1
IndexError Traceback (most recent call last)
IndexError: list index out of range

However, it is obvious that emphasizing only one type has its limitations. So, how do we solve this?

One solution is to include multiple exception types in the except block, like in the following examples:

try:
    s = input('please enter two numbers separated by comma: ')
    num1 = int(s.split(',')[0].strip())
    num2 = int(s.split(',')[1].strip())
    ...
except (ValueError, IndexError) as err:
    print('Error: {}'.format(err))

print('continue')
...

Or the second way:

try:
    s = input('please enter two numbers separated by comma: ')
    num1 = int(s.split(',')[0].strip())
    num2 = int(s.split(',')[1].strip())
    ...
except ValueError as err:
    print('Value Error: {}'.format(err))
except IndexError as err:
    print('Index Error: {}'.format(err))

print('continue')
...

This way, as long as one of the exception types in the except block matches the actual exception, the block will be executed.

However, in many cases, it is difficult to ensure that the program covers all possible exception types. Therefore, a more general approach is to declare the caught exception type in the last except block as Exception. Exception is the base class for all non-system exceptions and can match any non-system exception. The code can be written as follows:

try:
    s = input('please enter two numbers separated by comma: ')
    num1 = int(s.split(',')[0].strip())
    num2 = int(s.split(',')[1].strip())
    ...
except ValueError as err:
    print('Value Error: {}'.format(err))
except IndexError as err:
    print('Index Error: {}'.format(err))
except Exception as err:
    print('Other error: {}'.format(err))

print('continue')
...

Or, you can omit the exception type after except, which means it matches any exception (including system exceptions):

try:
    s = input('please enter two numbers separated by comma: ')
    num1 = int(s.split(',')[0].strip())
    num2 = int(s.split(',')[1].strip())
    ...
except ValueError as err:
    print('Value Error: {}'.format(err))
except IndexError as err:
    print('Index Error: {}'.format(err))
except:
    print('Other error')

print('continue')
...

It should be noted that when there are multiple except blocks in the program, only one of them will be executed at most. In other words, if multiple except blocks declare exception types that match the actual exceptions, only the first except block will be executed, and the others will be ignored.

In exception handling, a commonly used practice is to use finally, often together with try and except. Regardless of what happens, the statements in the finally block will be executed, even if return is used in the preceding try and except blocks.

One common application scenario is file reading:

import sys
try:
    f = open('file.txt', 'r')
    .... # some data processing
except OSError as err:
    print('OS error: {}'.format(err))
except:
    print('Unexpected error:', sys.exc_info()[0])
finally:
    f.close()

In this code, the try block attempts to read the file.txt file and perform a series of data processing operations. In the end, whether the reading is successful or fails, the statements in the finally block will be executed to close the file and ensure data integrity. Therefore, in the finally block, we usually put statements that need to be executed no matter what.

It is worth mentioning that for file reading, we often use with open as well, which you may have seen in the previous example. with open automatically closes the file at the end, making the statement more concise.

User-defined Exceptions #

In the previous examples, we saw many built-in exception types in Python. You may wonder if you can create your own exception types.

The answer is yes, Python allows us to do that. In the following example, we create a custom exception type called MyInputError and define and implement the __init__() and __str__() functions (called when the exception is printed directly):

class MyInputError(Exception):
    """Exception raised when there're errors in input"""
    def __init__(self, value): # initialization of the custom exception type
        self.value = value
    def __str__(self): # string representation of the custom exception type
        return ("{} is invalid input".format(repr(self.value)))

try:
    raise MyInputError(1) # raise the MyInputError exception
except MyInputError as err:
    print('error: {}'.format(err))

If you execute the above code block and print the output, you will get the following result:

error: 1 is invalid input

In practice, if the built-in exception types cannot meet our requirements, or if we want to add additional functionality to the exception to make it more detailed and readable, we can define our own custom exception types. However, in most cases, the built-in exception types in Python are sufficient.

Scenarios and Considerations for Using Exceptions #

After learning the basic knowledge, let’s focus on the scenarios and considerations for using exceptions.

Generally speaking, in a program, if we are not sure whether a piece of code can be executed successfully, we often need to use exception handling. In addition to the file reading example mentioned above, I can give another example to illustrate.

In the backend of a large social networking site, it is necessary to return corresponding records for user requests. User records are often stored in a key-value structured database. After receiving a request, we take the user’s ID and use it to query the records for this person in the database, and then we can return the corresponding result.

The raw data returned by the database is often in the form of a JSON string, so we first need to decode the JSON string. You may easily think of the following method:

import json
raw_data = queryDB(uid) # Return corresponding information based on the user's ID
data = json.loads(raw_data)

Is this code enough?

You should know that if the input string in the json.loads() function does not conform to its specifications, it will not be able to decode it and will throw an exception. Therefore, it is necessary to add exception handling.

try:
    data = json.loads(raw_data)
    ....
except JSONDecodeError as err:
    print('JSONDecodeError: {}'.format(err))

However, there is one thing to remember: we should not go to the other extreme and abuse exception handling.

For example, when you want to find the value corresponding to a key in a dictionary, you should never write it in the following way:

d = {'name': 'jason', 'age': 20}
try:
    value = d['dob']
    ...
except KeyError as err:
    print('KeyError: {}'.format(err))

Admittedly, this code does not have any bugs, but it is confusing and redundant. If your code is full of this kind of writing, it will undoubtedly be an obstacle for reading and collaboration. Therefore, for flow-control code logic, we generally do not use exception handling.

For the dictionary example, it is better to write it like this:

if 'dob' in d:
    value = d['dob']
    ...

Summary #

In this lesson, we learned about Python’s exception handling and its use cases. You should focus on the following points:

  • An exception typically occurs when there is an error during program execution, causing it to terminate and exit. We commonly use the try-except statement to handle exceptions, so the program doesn’t get terminated and can continue running.

  • When handling exceptions, if there are statements that must be executed, such as closing a file after opening it, they can be placed in the finally block.

  • Exception handling is usually used when you’re unsure if a certain piece of code will execute successfully or can’t easily determine the outcome, such as database connections or reads. For normal flow control logic, do not use exception handling; use conditional statements instead.

Thought Question #

Finally, let me leave you with a thought question. In exception handling, if there are multiple exceptions thrown in the try block, do we need to use multiple try-except blocks? Taking the example of connecting to and reading from a database, which of the following two approaches do you think is better?

First approach:

try:
    db = DB.connect('<db path>') # may raise an exception
    raw_data = DB.queryData('<viewer_id>') # may raise an exception
except (DBConnectionError, DBQueryDataError) err:
    print('Error: {}'.format(err))

Second approach:

try:
    db = DB.connect('<db path>') # may raise an exception
    try:
        raw_data = DB.queryData('<viewer_id>') # may raise an exception
    except DBQueryDataError as err:
        print('DB query data error: {}'.format(err))
except DBConnectionError as err:
    print('DB connection error: {}'.format(err))

Feel free to write your answer in the comments, along with any insights or doubts you had from today’s learning. You are also welcome to share this article with your colleagues and friends.