Of course I can run coverage for tests locally, but I think it should be accessible on coveralls. You cannot blindly rely on a test report until you are sure that they are actually working, otherwise it could give you a false sense of security.

I have faced a problem where I cannot be sure that pytest runs all the tests. Moreover I have got a test failures with pytest on tests that nose does not perform (and I do not why then they are not deleted). Here is the list:

• test_sankey not deprecated, runs without issues
• test_skew not deprecated, has outdated baseline images or tolerance problems (..FF.F: RMS 0.014, 0.264, 0.065)
• test_ttconv not deprecated, runs without issues
• test_usetex not deprecated, has outdated baseline images or tolerance problems (.F: RMS 0.261)

What benefits can we have by including test files coverage in the report? I collected dark spots in the test suite:

Important (test not running, uncovered cases):

• test_animation:test_no_length_frames
• test_axes:test_pcolorargs
• test_backend_pdf:test_use14corefonts does not run on travis (TRAVIS env variable was removed?)
• test_contour:test_contour_manual_labels
• test_image:test_minimized_rasterized
• test_mlab:Test_detrend.test_detrend_mean_2D_none
• test_mlab:Test_detrend.test_detrend_mean_2D_none_T
• test_mlab:Test_detrend.test_detrend_mean_2D_axis0
• test_mlab:Test_detrend.test_detrend_mean_2D_axis1
• test_mlab:Test_detrend.test_detrend_mean_2D_axism1
• test_mlab:Test_detrend.test_detrend_str_linear_2d_slope_off_axis0
• test_mlab:Test_detrend.test_detrend_detrend_linear_1d_slope_off_axis1
• test_mlab:test_griddata_nn
• test_subplots:test_shared
• test_patheffects:test_PathEffect_points_to_pixels

Less important (partial coverage):

• test_backend_svg _test_determinism
• test_cbook:test_results_whiskers_range
• test_cbook:test_results_whiskers_percentiles
• test_dates:test_date2num_dst
• test_delaunay
• test_mlab:Test_window
• test_mlab:Test_spectral_nosig_real_onesided
• test_units:test_numpy_facade

Not important (requires tests for the tests, and other cases):

• test_collections check_allprop unused function?
• test_collections check_segments
• test_transforms:test_non_affine_caching transform_path_non_affine&transform_non_affine
• test_triplot_return

Someone can say that we need tests for the tests to have full coverage -- I would say no, some special places can be marked with #pragma: no cover, but other can be fixed to have full coverage.

Currently this can be done with nose-cov plugin and after migrating to pytest we will have exactly same coverage reports with pytest-cov.