With any actively developed (python) coding project, you’re and your team are going to be running the same set of tests sometimes hundreds of times per week. If there’s an HTTP request to any 3rd-party source in there, this can cause problems. API calls can be expense, excessive scraping of the same source can cause IP blacklisting and the calls could just slow down your whole test process, adding extra baggage to the code deployment process.

To fix this, we can use Python’s Mock library. Mock is really useful for creating fake Function calls, fake Classes and other fake objects which can return fake values. In most cases when testing, you are really just testing how the application parses data rather than the reliability of the 3rd party service. The API’s response is generally the same. Mock can let you simulate the API’s response and parse its data rather than actually have to make the call each time.

It’s quite tricky to set up so I thought I would write a tutorial. The situation set up has a few components but I’ll try and explain it as well as possible. Let’s say there is a service that provides some useful API response. There’s a site, HTTPBin, set up by Kenneth Reitz to test HTTP libraries, which we will use here. Check out: https://httpbin.org/ip. The content is as follows:

{
  "origin": "123.123.123.123", 
}

Let’s say our program wants to grab the IP address in the origin field. Yup – a fairly pointless program but this will be analogous to many situations you’ll encounter.

Here’s a totally over-engineered class to get data from this service. When the class is initialized in __init__, it creates a base_url variable pointing to HTTPBin. The main handler function is the get_ip function, which simply grabs that field’s content. This first makes a call to api_call which uses requests.get to grab that HTTP data.

from requests import get

class MyClass:
    def __init__(self):
        self.base_url = "https://httpbin.org/ip"

    def api_call(self):
        """Makes an API call"""

        result = get(self.base_url)
        if result.ok:
            return result.json()

        return False

    def get_ip(self):
        """Gets the language from the API response"""

        try:
            data = self.api_call()
            return data['origin']
        except (TypeError, KeyError, IndexError):
            return "HTTP request to {0} failed".format(self.base_url)

To run this code its simply (in a Python Shell):

>>> import my_module
>>> test = my_module.MyClass()
>>> test.get_ip()
u'123.123.123.123'

What if we want to mock out requests.get? The Mock module documentation is quite unclear on how to target a specific function deep within a class. It turns out the easiest way to do this is not MagicMock or return_value but instead to use the counter-intuitively named “side_effect” feature. This is the testing module pre-mocking:

import unittest

from my_module import MyClass

class TestClassifier(unittest.TestCase):
    def setUp(self):
        self.ip_grabber = MyClass()
    
    def test_ip_grabber(self):
        ip = self.ip_grabber.get_ip()
        self.assertIsInstance(ip, basestring, "IP should be a string")
        self.assertEqual(ip.count('.'), 3, "IP should have 3 dots")

if __name__ == '__main__':
    unittest.main()

As you can see, this is a standard set of tests to check that the ip_grabber function returns a valid-ish IP address. It is run as follows:

mruttley$ python test_my_module.py 
.
----------------------------------------------------------------------
Ran 1 test in 0.293s

OK

However, the problem here is that it is going to call the actual API each time you run the tests. To stop this, let’s integrate the mock module:

import unittest
import mock

from my_module import MyClass

class fake_get():
    def __init__(self, url):
        self.ok = True
    def json(self):
        return {'origin': '123'}

class TestClassifier(unittest.TestCase):
    def setUp(self):
        self.ip_grabber = MyClass()
    
    @mock.patch('my_module.get', side_effect=fake_get)
    def test_ip_grabber(self, fake_get):
        
        ip = self.ip_grabber.get_ip()
        
        self.assertIsInstance(ip, basestring, "IP should be a string")
        self.assertEqual(ip.count('.'), 3, "IP should have 3 dots")

if __name__ == '__main__':
    unittest.main()

Here we’ve:

1. Imported the mock module. Note: if you get an error about “wraps” in the “six” module then it is almost certainly because you have more than one installation of six or mock and one needs to be deleted.
2. Create a fake function fake_get to replace requests.get with. This actually returns just “123” for now so you can see how it makes the test fail below.
3. Added the mock.patch wrapper around the test_ip_grabber function. Very important here is specifying the function name as it is imported in my_module NOT as it appears in the Python standard library; i.e. we are doing “my_module.get” rather than “requests.get”. The side_effect= then says to replace that with whatever function we want.
4. The fake function specified by side effect must now be added as an argument to the function.

Running this, we get:

mruttley$ python test_my_module.py
F
======================================================================
FAIL: test_ip_grabber (__main__.TestClassifier)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Library/Python/2.7/site-packages/mock/mock.py", line 1305, in patched
    return func(*args, **keywargs)
  File "test_my_module.py", line 19, in test_ip_grabber
    self.assertEqual(ip.count('.'), 3, "IP should have 3 dots")
AssertionError: IP should have 3 dots

----------------------------------------------------------------------
Ran 1 test in 0.002s

FAILED (failures=1)

Mock’s side_effect has replaced requests.get  To make this pass, just replace 

return {'origin': '123'}

return {'origin': '123.123.123.123'}

mruttley$ python test_my_module.py
.
----------------------------------------------------------------------
Ran 1 test in 0.000s

OK

Tests pass and zero HTTP traffic!