Flask Series: Testing

New code, everything that might break or does not work correctly should be tested. 100% code coverage does not guarantee a working system, but still you have to try to implement as much or more test code as production one.

There are different types of testing, but I will concentrate on unit testing.

Unit Testing
Unit test is a piece of code that tests a unit of work, logical unit in the tested system. Your unit tests should be:
– automated;
– independent;
– consistent and repeatable;
– maintainable;

Python Testing Frameworks
– others;

unittest is the built-in testing module for implementing unit tests in Python, it is an xUnit framework and shares some important components:
– test runner – executes the tests and provides the test results to the user;
– test case – smallest unit of testing;
– test fixture – preparation needed for test cases execution;
– test suite – set of tests cases based on some criteria;
– test execution;
– assertions – function that verifies some state or behavior;

Test Discovery 
You could use the unittest support for test discovery to run your unit tests:

cd project_directory
python -m unittest discover

More information on the unittest discovery functionality can be found here.

To verify if your code is working correcly you have to use the assert methods, provided by the TestClass:
– assertEqual;
– assertNotEqual;
– assertTrue;
– etc.;

Complete list with assert statements.

nose extends the unittest framework. It is not built-in like unittest, that’s why you should install it:

pip install nose

Together with the nose package and nosetests script will be installed on your machine, used to discover and run tests.

cd project_directory

More information on nose can be found here.

How to Test a Flask Application?

Flask exposes the Werkzeug test client and handles the context locals. It is important to set TESTING configuration setting to true, to propagate the exceptions to the test client.

Helpful information how to test systems that requires logins can be found in these tests.

Session can be asserted for given keys and values in the following way:

Further tips and tricks on testing Flask applications can found here.

This help article section provides additional information on Flask session testing.

Flask-Testing Extension
Flask-Testing is very helpful Flask extension, that make Flask developers testing efforts easier.


pip install Flask-Testing

Example Tests:

Test Doubles
This is a concept used to describe a scenario where you replace your production object with a testing one, that can be easily configured, manipulated and asserted (if needed).
– Fake;
– Stubs;
– Mocks;
– etc.

I can suggest you the term explanation provided by Martin Fowler in his bliki article about Test Doubles.

Mocking is very important in testing scenarios, in order to avoid external dependencies in your code, like service calls, I/O operations, etc.

In the flask-bookshelf I will use mock framework.

Code Coverage
Code Coverage in Wikipedia: code coverage is a measure used to describe the degree to which the source code of a program is tested by a particular test suite.

To measure the code coverage in the Flask application I will use coverage.

pip install coverage

Now that you have it installed, to get the needed measurement data you have to run the following command:

coverage run –source=app_folder_path

To print the prepared report execute:

coverage report 

The complete demo application, described in this blog post, can be found here.

In the next blog post I will describe how to work with views and web forms in your Flask application.

Serverspec new matchers: Part 2

In my previous blog post I have described the new matchers implemented for the iis_app_pool resource type. In this one I will show examples how to use the new ones for the iis_website and service, together with the two resource types: event_log and perf_counter.


IIS Website resource type


In order to test the IIS website site bindings, you should use the have_site_bindings matcher.

You can give a protocol and chain them like this.

You could specify an IP Address and chain them like this.

or specify a host header

Note: All of the methods can be used together.


In order to test the IIS website virtual directories, you should use the have_virtual_dir matcher.

You can give a path to the virtual directory name, like this.


In order to test the IIS website applications, you should use the have_site_application matcher.

Additionaly you can specify the application pool name, used by the application, and/or the physical path to it.


Service resource type.


In order to test the path to the executable, you should use the have_path_name matcher.


Windows eveng log resource type.

In order to test whether an event log exists, you should use exist matcher.


Performance counter resource type.

In order to test whether a performance counter category exists, you should use exist matcher.

You can specify counter name, like this.


Serverspec new matchers: Part 1

As I wrote in my previous post, Serverspec allows users to describe the machine setup using many different resource types. There are lots of resource types already implemented in the framework but based on the setup, which should be described, they can be insufficient.

The good thing about Serverspec is that it is easily extensible. Recently I have added several new matchers and resource types, which I will describe in the next few blog posts.

Lets start with iis_app_pool resource type and its new matchers.


IIS Application Pool resource type


In order to test the IIS Application pool, whether the Enable 32-Bit Applications option is enabled, you should use the have_32bit_enabled matcher.


In order to test the IIS Application pool Idle Time-out setting, you should use the have_idle_timeout matcher(in minutes).


In order to test the IIS Application pool Identity type(which configures how the app pool runs), you should use the have_identity_type matcher.


In order to test the IIS Application pool Regular Time Interval, you should use the have_periodic_restart matcher(in minutes).


In order to test the IIS Application pool Load User Profile setting, you should use the have_user_profile_enabled matcher.


In order to test the IIS Application pool username specified in case of Specific User used, you should use the have_username matcher.


In order to test the IIS Application pool Managed Pipeline Mode, you should use the have_managed_pipeline_mode matcher.


In order to test whether the IIS Application pool is started, you should use the started matcher.

In my next blog post I will have a look at the matchers for the iis_website and service resource types together with the two new resource types event_log and perf_counter types.

Note: The matchers described above can found in the Serverspec gem 1.14 and above.

Getting started with Serverspec

What is Serverspec?
Rspec tests for verifying the configuration of a given machine. It is used together with configuration management tools like Salt, Ansible, Puppet, Chef, etc. as a integration testing framework.

It is a Ruby-based framework, which uses internally Shell(Linux environments), Powershell(Windows environment) scripts for checking the target machines if the configuration management tool has configured them correctly.


The easiest way to install is to use the serverspec gem:

gem install serverspec


I will concentrate on testing Windows machines using the Serverspec framework, but the approach is the same for Linux ones.

To get started with Serverspec, the serverspec-init tool can be used, it creates the needed setup for a testing suite:

Select OS type: 

  1) UN*X
  2) Windows 

Select number: 2 

Select a backend type: 

  1) SSH
  2) Exec (local) 

Select number: 2 

Note: The setup is done for windows local environment.

The command will create the following setup on the local machine:


The httpd_spec.rb contains example Rspec tests.

Executing the rake spec command(rake gem is needed) will run the Rspec tests.

rake spec

Resource Types

Serverspec has many different resource types implemented, which allow users to describe the machine setup in their Rspec tests.

Helpful information about these resource types can be found here.