Allure report doesn't take in correct number of tests when using parametrized tests where tests parameters are duplicated #864

Activity

1. delatrie commented on Jun 9, 2025

   

   delatrie

   on Jun 9, 2025

   Contributor

   More actions

   Hi, @qwertzy-antonio-godinho, thanks for opening the issue.

   Could you please clarify the use case behind having two identical sets of parameter values?
2. qwertzy-antonio-godinho commented on Jun 11, 2025

   

   qwertzy-antonio-godinho

   on Jun 11, 2025

   Last edited by qwertzy-antonio-godinho

   Author

   More actions

   Hi, @delatrie, hope you're well. The use case is that at work we use maps for different "products" (which at a later stage diverge in characteristics), but to the point, the issue that I'm reporting is that newer allure versions do not report on the correct number of tests ran by pytest like in previous version 2.9.44.

   A better example of my use case would be something like:

   from enum import Enum
   import pytest
   
   class Products(Enum):
       PRODUCT_1 = 'PROD_1'
       PRODUCT_2 = 'PROD_2'
       PRODUCT_3 = 'PROD_1'
   
   @pytest.mark.parametrize(
       'a, b', 
       [    
           (Products.PRODUCT_1.value, 'PROD_1'), 
           (Products.PRODUCT_2.value, 'PROD_2'),
           (Products.PRODUCT_3.value, 'PROD_1'),
       ]
   )
   
   def test_one(a, b):
       assert a == b
   

   There are 3 tests, but unfortunately Allure only shows 2 in the report (in our test suite to misses close to 200 tests).

   Please see bellow:
   

   When there are failures, Allure also does not report the correct number of failed tests:

   

   I think the issue could probably be related to this one: #778
3. sharovd commented on Jun 15, 2025

   

   sharovd

   on Jun 15, 2025

   Contributor

   More actions

   @delatrie, hi!
   I have investigated this issue and tried a few solutions, but none of them work in every known case 🙁
   Could you please take a look? Perhaps I’ve overlooked something, or maybe we could agree on a compromise.

   • To fix issue using pytest.mark.parametrize will miss some testcase in report #778, we could include the parameter name and value used to calculate the historyId. Currently, we only use the value. However, this change would not solve the problem described here.
   • Another option would be to use item.callspec.id to calculate the historyId. However, this would break the fix for Changing an id of a pytest parameter set breaks test's history #744, where only the parameter id changes. Perhaps we can consider stopping support for that scenario?
   • I also tried using the parameter index (item.callspec.indices), but if the order of the test data changes, the historyId will be different and we won't be able to match the same test in the history, which doesn't feel right.

   I’d be glad to hear any other ideas on how to solve this. I’m ready to finalize this work, prepare a PR, and update the tests. Thanks!
4. HenriqueMagnagoTyphoon commented on Oct 22, 2025

   

   HenriqueMagnagoTyphoon

   on Oct 22, 2025

   More actions

   Is this a new behaviour? I am using different ids for different tests, and I do not recall having missing tests in the allure report. I have been using them for quite a while. This is a simplified version of the test I was running:

   import pytest
   from random import random as rd
   
   n = 100
   
   @pytest.mark.parametrize("val", [round(rd(), 1) for _ in range(n)], ids=[f'Case{x}' for x in range(n)])
   def test_foo(val):
       assert val < 0.5