In my last article, I started looking at "pytest", a framework for testing Python programs that's really changed the way I look at testing. For the first time, I really feel like testing is something I can and should do on a regular basis; pytest makes things so easy and straightforward.
One of the main topics I didn't cover in my last article is user input and output. How can you test programs that expect to get input from files or from the user? And, how can you test programs that are supposed to display something on the screen?
So in this article, I describe how to test input and output in a variety of ways, allowing you to test programs that interact with the outside world. I try not only to explain what you can do, but also show how it fits into the larger context of testing in general and pytest in particular.
Say you have a function that asks the user to enter an integer and then returns the value of that integer, doubled. You can imagine that the function would look like this:
def double():
x = input("Enter an integer: ")
return int(x) * 2
How can you test that function with pytest? If the function were to take an argument, the answer would be easy. But in this case, the function is asking for interactive input from the user. That's a bit harder to deal with. After all, how can you, in your tests, pretend to ask the user for input?
In most programming languages, user input comes from a source known as
standard input (or stdin). In Python, sys.stdin is a read-only file object from
which you can grab the user's input.
So, if you want to test the "double" function from above, you can (should) replace
sys.stdin with another file. There are two problems with this, however. First, you
don't really want to start opening files on disk. And second, do you really want to
replace the value of sys.stdin in your tests? That'll affect more than just one
test.
The solution comes in two parts. First, you can use the pytest "monkey patching"
facility to assign a value to a system object temporarily for the duration of the
test. This facility requires that you define your test function with a parameter named
monkeypatch. The pytest system notices that you've defined it with that parameter,
and then not only sets the monkeypatch local variable, but also sets it up to let you
temporarily set attribute names.
In theory, then, you could define your test like this:
def test_double(monkeypatch):
monkeypatch.setattr('sys.stdin', open('/etc/passwd'))
print(double())
In other words, this tells pytest that you want to open /etc/passwd and feed its
contents to pytest. This has numerous problems, starting with the fact that
/etc/passwd contains multiple lines, and that each of its lines is non-numeric.
The function thus chokes and exits with an error before it even gets to the (useless)
call to print.
But there's another problem here, one that I mentioned above. You don't really want to
be opening files during testing, if you can avoid it. Thus, one of the great
tools in my testing toolbox is Python's StringIO class. The beauty of
StringIO is
its simplicity. It implements the API of a "file" object, but exists only in memory
and is effectively a string. If you can create a StringIO instance, you can pass it
to the call to monkeypatch.setattr, and thus make your tests precisely the way you
want.
Here's how to do that:
from io import StringIO
from double import double
number_inputs = StringIO('1234\n')
def test_double(monkeypatch):
monkeypatch.setattr('sys.stdin', number_inputs)
assert double() == 2468
You first create a StringIO object containing the input you want to simulate from the
user. Note that it must contain a newline (\n) to ensure that you'll see the end of
the user's input and not hang.
You assign that to a global variable, which means you'll be able to access it from within your test function. You then add the assertion to your test function, saying that the result should be 2468. And sure enough, it works.
I've used this technique to simulate much longer files, and I've been quite satisfied
by the speed and flexibility. Just remember that each line in the input "file"
should end with a newline character. I've found that creating a StringIO with a
triple-quoted string, which lets me include newlines and write in a more natural
file-like way, works well.
You can use monkeypatch to simulate calls to a variety of other objects as well. I
haven't had much occasion to do that, but you can imagine all sorts of
network-related objects that you don't actually want to use when testing. By monkey-patching those objects during testing, you can pretend to connect to a remote server,
when in fact you're just getting pre-programmed text back.
What happens if you call the test_double function with a string? You probably
should test that too:
str_inputs = StringIO('abcd\n')
def test_double_str(monkeypatch):
monkeypatch.setattr('sys.stdin', str_inputs)
assert double() == 'abcdabcd'
It looks great, right? Actually, not so much:
E ValueError: invalid literal for int() with base 10: 'abcd'
The test failed, because the function exited with an exception. And that's okay; after all, the function should raise an exception if the user gives input that isn't numeric. But, wouldn't it be nice to specify and test it?
The thing is, how can you test for an exception? You can't exactly use a usual assert
statement, much as you might like to. After all, you somehow need to trap the exception,
and you can't simply say:
assert double() == ValueError
That's because exceptions aren't values that are returned. They are raised through a different mechanism.
Fortunately, pytest offers a good solution to this, albeit with slightly different
syntax than you've seen before. It uses a with statement, which many Python
developers recognize from its common use in ensuring that files are flushed and
closed when you write to them. The with statement opens a block, and if an
exception occurs during that block, then the "context manager"—that is, the object
that the with runs on—has an opportunity to handle the exception. pytest takes
advantage of this with the pytest.raises context manager, which you can use in the
following way:
def test_double_str(monkeypatch):
with pytest.raises(ValueError):
monkeypatch.setattr('sys.stdin', str_inputs)
result = double()
Notice that you don't need an assert statement here, because the
pytest.raises is,
effectively, the assert statement. And, you do have to indicate the type of error
(ValueError) that you're trying to trap, meaning what you expect to receive.
If you want to capture (or assert) something having to do with the exception that was
raised, you can use the as part of the with statement. For example:
def test_double_str(monkeypatch):
with pytest.raises(ValueError) as e:
monkeypatch.setattr('sys.stdin', str_inputs)
results = double()
assert str(e.value) == "invalid literal for int()
↪with base 10: 'abcd'"
Now you can be sure that not only was a ValueError exception raised, but also what
message was raised.
I generally advise people not to use print in their functions. After all, I'd like
to get some value back from a function; I don't really want to display something on
the screen. But at some point, you really do actually need to display things to the
user. How can you test for that?
The pytest solution is via the capsys plugin. Similar to monkeypatch, you declare
capsys as a parameter to your test function. You then run your function, allowing
it to produce its output. Then you invoke the readouterr function on
capsys,
which returns a tuple of two strings containing the output to stdout and its
error-message counterpart, stderr. You then can run assertions on those strings.
For example, let's assume this function:
def hello(name):
print(f"Hello, {name}!")
You can test it in the following way:
def test_hello(capsys):
hello('world')
captured_stdout, captured_stderr = capsys.readouterr()
assert captured_stdout == 'Hello, world!'
But wait! This test actually fails:
E AssertionError: assert 'Hello, world!\n' == 'Hello, world!'
E - Hello, world!
E ? -
E + Hello, world!
Do you see the problem? The output, as written by print, includes a trailing
newline (\n) character. But the test didn't check for that. Thus, you can check for
the trailing newline, or you can use str.strip on stdout:
def test_hello(capsys):
hello('world')
captured_stdout, captured_stderr = capsys.readouterr()
assert captured_stdout.strip() == 'Hello, world!'
pytest continues to impress me as a testing framework, in no small part because it combines a lot of small, simple ideas in ways that feel natural to me as a developer. It has gone a long way toward increasing my use of tests, both in general development and in my teaching. My "Weekly Python Exercise" subscription service now includes tests, and I feel like it has improved a great deal as a result.
In my next article, I plan to take a third and final look at pytest, exploring some of the other ways it can interact with (and help) write robust and useful programs.
Reuven Lerner teaches Python, data science and Git to companies around the world. You can subscribe to his free, weekly "better developers" e-mail list, and learn from his books and courses at http://lerner.co.il. Reuven lives with his wife and children in Modi'in, Israel.
