An Exceptional Rise

🚨 Exceptional Python 🐍

Understanding Errors in context, from simple scripts to complex logic.

What are exceptions?

Exceptions in Python––often referred to simply as Errors–– are among the first things encountered when working with the language.

Errors happen when things go wrong: when something exceptional happens

Think of it this way:

The ordinary expectation of a Python program is that it simply runs… except in the case of Exceptions.

From the very first Python script you write…

greeting = "Hello, world!"
print(greetig)

…errors are part of your life.

NameError, for example, tells us about typos.

Traceback (most recent call last):
  File "main.py", line 2, in <module>
    print(greetig)
NameError: name 'greetig' is not defined

Errors guide you towards functioning code

numbers = [1,2,3]
total = sum(numbers))

Typos can be easy to miss…

… so very specific errors like SyntaxError tell you exactly what’s going on.

  File "main.py", line 2
    total = sum(numbers))
                        ^
SyntaxError: unmatched ')'

Let’s say you have a script from getting last names from company emails:

email_addresses = [
    "emperor_palpatine@sith.com",
    "kylo_ren@sith.com",
]

for email in email_addresses:
    after_underscore = email.split("_")[1]  # ["kylo", "ren@sith.com"][1]
    last_name = after_underscore.split("@")[0]  # ["ren", "sith.com"][0]
    print(last_name)

When you run it, it works as expected:

$ python script.py

palpatine
ren

…Until you add another email address

email_addresses = [
    "emperor_palpatine@sith.com",
    "kylo_ren@sith.com",
    "empire@sith.com",  # uh oh
]

$ python script.py

palpatine
ren
Traceback (most recent call last):
  File "main.py", line 8, in <module>
    after_underscore = email.split("_")[1]
IndexError: list index out of range

(IndexError means you tried to get an element from a list that isn’t there.)

If you take a look back at the script…

for email in email_addresses:
    after_underscore = email.split("_")[1]      # !!!
    last_name = after_underscore.split("@")[0]
    print(last_name)

You realize that empire@sith.com does not have an _ in it.

email.split("_")[1] breaks, because there’s nothing to split on:

"empire@sith.com".split("_")[1] results in ["empire@sith.com"][1]. Ouch!

Errors can be dealt with.

So far, all the errors we’ve seen halt our program.

Since errors are often understandable and expected, it follows that Python would give us tools to let the program carry on.

Handling errors is built in to the language

The try and except keywords allow us to “make exceptions” in the flow of our programs.

Hey Python! Go ahead and try this next block of code, but feel free to make an exception and keep going if something happens.

try:
    do_stuff()
except:
    print("An error happened when doing stuff.")

Fixing the script

All we need to do is add a try:except

for email in email_addresses:
    try:
        after_underscore = email.split("_")[1]
        last_name = after_underscore.split("@")[0]
        print(last_name)
    except:
        # we understand what's wrong, so make note of it and carry on!
        print("Found invalid email: ", email)
        continue

Adding another email after the invalid one proves that things are ok now:

email_addresses = [
    "emperor_palpatine@sith.com",
    "kylo_ren@sith.com",
    "empire@sith.com",  # this is fine now!
    "darth_maul@sith.com"
]

Run the script and:

$ python script.py

palpatine
ren
Found invalid email:  empire@sith.com
maul

More complexity, more errors

All of a sudden, the data is like this:

email_addresses = [
    {"email": "emperor_palpatine@sith.com", "hair_color": "white"},
    {"email": "kylo_ren@sith.com", "hair_color": "black"},
    {"email": "empire@sith.com"},
    {"email": "darth_maul@sith.com"}
]

And the script must now print, for example, Good ol 'white-haired' palpatine instead of just saying his last name.

The change is easy enough.

for email in email_addresses:
    try:
        after_underscore = email["email"].split("_")[1]
        hair_color = email["hair_color"]
    except:
        print("Found invalid email: ", email)
        continue
    last_name = after_underscore.split("@")[0]
    print(f"Good ol '{hair_color}-haired' {last_name}")

But the output…

Good ol 'white-haired' palpatine
Good ol 'black-haired' ren
Found invalid email:  {'email': 'empire@sith.com'}
Found invalid email:  {'email': 'darth_maul@sith.com'}

darth_maul@sith.com didn’t cause an error before…but now it does?

Specificity is important.

I’ve been keeping something from you: Our use of except so far has been :flushed: unacceptable.

The except keyword, as you may know, is usually followed by an exception class.

In practice when you want to catch any error, you say:

try:
    something()
except Exception:  # all exceptions are subclasses of Exception
    something_exceptional()

You rarely want to catch every error.

In most cases, it’s a code smell. :nose:

If there’s a known exception-causing behavior in some code, the exception in question should be known as well.

Let’s refactor that script a bit

and say except IndexError, since we know for sure that the code can run into that.

email_addresses = [
    {"email": "emperor_palpatine@sith.com", "hair_color": "white"},
    {"email": "kylo_ren@sith.com", "hair_color": "black"},
    {"email": "empire@sith.com"},
    {"email": "darth_maul@sith.com"}
]

for email in email_addresses:
    try:
        after_underscore = email["email"].split("_")[1]
        hair_color = email["hair_color"]
    except IndexError:                            # 😎 pro.
        print("Found invalid email: ", email)
        continue
    last_name = after_underscore.split("@")[0]
    print(f"Good ol '{hair_color}-haired' {last_name}")

Run it once more…

$ python script.py

Good ol 'white-haired' palpatine
Good ol 'black-haired' ren
Found invalid email:  {'email': 'empire@sith.com'}
Traceback (most recent call last):
  File "main.py", line 11, in <module>
    hair_color = email["hair_color"]
KeyError: 'hair_color'

The output has changed!

By being more specific with error handling, a new and distinct unhappy path has been revealed.

The KeyError is pointing out that {'email': 'empire@sith.com'}["hair_color"] can’t run. It’s not an “invalid email” issue. It’s something else entirely.

🎉 We just gained precise knowledge of a new error condition!

The two conidions are:

  1. When an email address does not contain a _, an IndexError appears.
  2. :new: When a dictionary in email_addresses lacks a "hair_color" key, a KeyError appears.

Now that’s what I call software development!

There are several plausible approaches to fixing this.

1️⃣: except multiple errors

except can reference a tuple of many exception types.

    try:
        after_underscore = email["email"].split("_")[1]
        hair_color = email["hair_color"]
    except (IndexError, KeyError):
        print("Found invalid email or missing hair color: ", email)
        continue

:no_entry_sign: Not a good solution. Handling 2 distinct errors the same way is flimsy.

$ python script.py

Good ol 'white-haired' palpatine
Good ol 'black-haired' ren
Found invalid email or missing hair color:  {'email': 'empire@sith.com'}
Found invalid email or missing hair color:  {'email': 'darth_maul@sith.com'}

The output doesn’t really tell us the exact issue, so debugging will require some research.

2️⃣: Add another except

    try:
        after_underscore = email["email"].split("_")[1]
        hair_color = email["hair_color"]
    except IndexError:
        print("Found invalid email: ", email)
        continue
    except KeyError:
        print("Found missing hair color for sith lord: ", email)
        continue

Now the output is clearer, :no_entry_sign: but this is still not great.

$ python script.py

Good ol 'white-haired' palpatine
Good ol 'black-haired' ren
Found invalid email:  {'email': 'empire@sith.com'}
Found missing hair color for sith lord:  {'email': 'darth_maul@sith.com'}

2️⃣: Add another except – Why not??

While the output is clearer, the code is not.

    try:
        after_underscore = email["email"].split("_")[1]
        hair_color = email["hair_color"]
    except IndexError:
        print("Found invalid email: ", email)
        continue
    except KeyError:
        print("Found missing hair color for sith lord: ", email)
        continue

The try has 2 expressions in it. In the 1st except, we know first expression can IndexError. The 2nd except deals with the second expression, which a can KeyError.

We know exactly what’s happening, and this solution does not reflect our explicit & precise knowledge of the error conditions.

3️⃣: Make it boring and ugly with two try:excepts

    try:
        after_underscore = email["email"].split("_")[1]
    except IndexError:
        print("Found invalid email: ", email)
        continue
    try:
        hair_color = email["hair_color"]
    except KeyError:
        print("Found record with missing hair color: ", email)
        continue

:white_check_mark: The un-slick (no bare except), un-fancy (no tuple of exception types), un-flat (look at all those indents!) solution is The True Way.

The only expression contained in any try should be the exact expression that may cause the exception type in the except.

In short 💡

Specificity, Specificity, Specificity

Now what?

We shall now stray from the runnable snippets of code seen so far and look at more illustrative, contrived examples.

Capture 🕳️

The try:except keyword-pair have another friend: as.

When you except ... as, you can hold on to an exception for further inspection.

In [1]: try:
   ...:     {}["uh oh"]
   ...: except Exception as error:
   ...:     print(error)
   ...:
'uh oh'

You’ll notice, however, that the output of print(error) is not very helpful or clear. The error getting raised is for sure a KeyError, but unfortunately the string representation of KeyError is just… the key.

This issue is not isolated to KeyError.

In [2]: try:
   ...:     {}[object()]
   ...: except Exception as error:
   ...:     print(error)
   ...:
<object object at 0x107637340>

And it’s not so bad in many cases. TypeError has a nice message on it:

In [3]: try:
    ...:     int({})
    ...: except Exception as error:
    ...:     print(error)
    ...:
int() argument must be a string, a bytes-like object or a real number, not 'dict'

Still, some specifics would be nice, no? Nothing there says TypeError.

Dealing with ambiguity like a pro

I know I said

The only expression contained in any try should be the exact expression that may cause the exception type in the except.

earlier.

And I stand by it in most cases. But it’s pure armchair-pedantry to suggest that there is no reason to ever except Exception.

When vagueness is guaranteed, respond with handspun specificity.

Re-raising & Bespoke Errors

A common scenario where you may want to except Exception is with third-party SDKs. Maybe they’re written in C, or it’s just not clear at all what kind of errors could bubble out.

def functionality(sdk: BlackBoxSdk) -> None:
    try:
        sdk.do_stuff()
    except Exception as error:
        print("The SDK raised a mystery error :(")
        # now what??

The Zen of Python says:

In the face of ambiguity, resist the tempation to guess.

So how can we resist and be certain?

In the interest of not guessing…

You could just “re-raise” the error.

def functionality(sdk: BlackBoxSdk) -> None:
    try:
        sdk.do_stuff()
    except Exception as error:
        print("The SDK raised a mystery error :(")
        raise error  # 💥

This halts the application in the event of the SDK messing up, and provides a modicum of evidence with a print().

The print() is really the only improvement over just calling do_sutff() and watching the application explode.

Do not obey: Command.

The True Way is to make your very own exception.

class BlackBoxError(Exception):
    """Raise me when the BlackBoxSdk acts up"""


def functionality(sdk: BlackBox) -> None:
    try:
        sdk.do_stuff()
    except Exception as error:
        print("The SDK raised a mystery error :(")
        raise BlackBoxError from error  # 😮

Why…. is that better?

class BlackBoxError(Exception):
    """Raise me when the BlackBoxSdk acts up"""


def functionality(sdk: BlackBox) -> None:
    try:
        sdk.do_stuff()
    except Exception as error:
        print("The SDK raised a mystery error :(")
        raise BlackBoxError from error  # 😮

Go forth and raise.

Understanding exceptions is fine. Handling them is well and good. Taking control and raising your own is key to long-lasting and nice-to-maintain programs.