Tests were skipped if package was invalid by Pazus · Pull Request #197 · utPLSQL/utPLSQL

Pazus · 2017-02-04T10:22:47Z

If a package is invalid and contains suite level hooks then tests were skipped with no failure. No all tests mark as failed.
This might fix #192

What's done:

refactored us_suite_item child types, made procedure call to do_execute final and defined at the ut_suite_item_level
changed the order of calling event and setting start_time of an item to be the same across every child. Start and finish events are called even if the item is ignored, For ignored items end_time = start_time.
If you have a package with "before"/"after" suite-level hooks and the spec is invalid, then ut_executable of first "before" hook leads to the exeption. This should mark all the tests in the suite marked as failed. To do this if the suite is not "valid" the loop is added to iterate through ut_test items as they are also going to fail on is_valid procedure.

Pazus · 2017-02-04T11:45:18Z

also address #137

Pazus · 2017-02-04T11:46:25Z

also #139

jgebal · 2017-02-04T19:12:53Z

Can you explain how is this change addressing all the mentioned issues?

How will the executor behave now when we have:

A test package without body
A test package with invalid body
A test package with invalid package spec
A test package that is raising an exception in beforeall
A test package that is raising an exception in afterall
A test package that is raising an exception in beforeeach
A test package that is raising an exception in aftereach
A test package that is raising an exception in test

Also, what will happen if before, after or test procedure is issuing a commit?

Pazus · 2017-02-05T06:45:14Z

I should have written better description of the change. Edited the initial post.
What we corrently have:

A test package without body - tests are skipped. Need to be fixed
A test package with invalid body - tests are marked as failed
A test package with invalid package spec - tests are marked as failed
A test package that is raising an exception in beforeall - tests are skipped. need to be fixed
A test package that is raising an exception in afterall - single test is marked as failed (not the last one... unstable)
A test package that is raising an exception in beforeeach - tests are skipped. need to be fixed
A test package that is raising an exception in aftereach - all tests marked as failed
A test package that is raising an exception in test - the test is marke as failed

Pazus · 2017-02-05T06:58:54Z

@jgebal We need to agree on the right behavior for beforeall, afterall, beforeeach and when there is no body

Pazus · 2017-02-08T18:25:29Z

New behavior:
A test package without body - each test is reported as failed
A test package with invalid body - each test is reported as failed
A test package with invalid package spec - tests are skipped. Only valid specifications are parsed for annotations.
A test package that is raising an exception in beforeall - each test is reported as failed
A test package that is raising an exception in afterall - each test is reported ok but then each is marked as failed

Remove rooms by name
  Removes a room without content in it
  Does not remove room when it has content
  Raises exception when null room name given
  Removes a room without content in it (FAILED - 1)
  Does not remove room when it has content (FAILED - 2)
  Raises exception when null room name given (FAILED - 3)
 
Failures:
 
  1) remove_empty_room
        
        error: Afterall procedure failed: 
               ORA-20001: asd
               ORA-06512: at "UT3.TEST_REMOVE_ROOMS_BY_NAME", line 29
               ORA-06512: at line 6
       
  2) room_with_content
        
        error: Afterall procedure failed: 
               ORA-20001: asd
               ORA-06512: at "UT3.TEST_REMOVE_ROOMS_BY_NAME", line 29
               ORA-06512: at line 6
       
  3) null_room_name
        
        error: Afterall procedure failed: 
               ORA-20001: asd
               ORA-06512: at "UT3.TEST_REMOVE_ROOMS_BY_NAME", line 29
               ORA-06512: at line 6
       
Finished in ,030293 seconds
3 tests, 0 failed, 3 errored, 0 ignored

A test package that is raising an exception in beforeeach - each test is reported as failed
A test package that is raising an exception in aftereach - each test is reported ok but then each is marked as failed

Remove rooms by name
  Removes a room without content in it
  Removes a room without content in it (FAILED - 1)
  Does not remove room when it has content
  Does not remove room when it has content (FAILED - 2)
  Raises exception when null room name given
  Raises exception when null room name given (FAILED - 3)
 
Failures:
 
  1) remove_empty_room
        
        error: Aftereach procedure failed:
               ORA-20001: asd
               ORA-06512: at "UT3.TEST_REMOVE_ROOMS_BY_NAME", line 40
               ORA-06512: at line 6
       
  2) room_with_content
        
        error: Aftereach procedure failed:
               ORA-20001: asd
               ORA-06512: at "UT3.TEST_REMOVE_ROOMS_BY_NAME", line 40
               ORA-06512: at line 6
       
  3) null_room_name
        
        error: Aftereach procedure failed:
               ORA-20001: asd
               ORA-06512: at "UT3.TEST_REMOVE_ROOMS_BY_NAME", line 40
               ORA-06512: at line 6
       
Finished in ,052271 seconds
3 tests, 0 failed, 3 errored, 0 ignored

A test package that is raising an exception in test - the test is marked as failed

…o child items

…invalid-package-v2

Pazus · 2017-02-17T06:08:07Z

New behavior:
A test package without body - each test is reported as failed
A test package with invalid body - each test is reported as failed
A test package with invalid package spec - tests are skipped. Only valid specifications are parsed for annotations.
A test package that is raising an exception in beforeall - each test is reported as failed
A test package that is raising an exception in afterall - each test is reported ok but then each is marked as failed

Remove rooms by name
  Removes a room without content in it
  Does not remove room when it has content
  Raises exception when null room name given
 
Warnings:
 
Afterall procedure failed: 
ORA-20001: EXCEPTION TEST
ORA-06512: at "UT3.TEST_REMOVE_ROOMS_BY_NAME", line 34
ORA-06512: at line 6
 
Finished in ,03872 seconds
3 tests, 0 failed, 0 errored, 0 ignored. 1 warning(s)

A test package that is raising an exception in beforeeach - each test is reported as failed
A test package that is raising an exception in aftereach - each test is reported ok but then each is marked as failed

Remove rooms by name
  Removes a room without content in it
  Does not remove room when it has content
  Raises exception when null room name given
 
Warnings:
 
Aftereach procedure failed:
ORA-20001: EXCEPTION TEST
ORA-06512: at "UT3.TEST_REMOVE_ROOMS_BY_NAME", line 31
ORA-06512: at line 6
 
Aftereach procedure failed:
ORA-20001: EXCEPTION TEST
ORA-06512: at "UT3.TEST_REMOVE_ROOMS_BY_NAME", line 31
ORA-06512: at line 6
 
Aftereach procedure failed:
ORA-20001: EXCEPTION TEST
ORA-06512: at "UT3.TEST_REMOVE_ROOMS_BY_NAME", line 31
ORA-06512: at line 6
 
Finished in ,046337 seconds
3 tests, 0 failed, 0 errored, 0 ignored. 3 warning(s)

A test package that is raising an exception in test - the test is marked as failed

Remove rooms by name
  Removes a room without content in it (FAILED - 1)
  Does not remove room when it has content
  Raises exception when null room name given
 
Failures:
 
  1) remove_empty_room
        
        error: ORA-20001: EXCEPTION TEST
               ORA-06512: at "UT3.TEST_REMOVE_ROOMS_BY_NAME", line 47
               ORA-06512: at line 6
       
Finished in ,035998 seconds
3 tests, 0 failed, 1 errored, 0 ignored.

A test package that is raising an exception in beforetest - each test is reported as failed

Remove rooms by name
  Removes a room without content in it (FAILED - 1)
  Does not remove room when it has content
  Raises exception when null room name given
 
Failures:
 
  1) remove_empty_room
        
        error: ORA-20001: EXCEPTION TEST
               ORA-06512: at "UT3.TEST_REMOVE_ROOMS_BY_NAME", line 37
               ORA-06512: at line 6
       
Finished in ,020943 seconds
3 tests, 0 failed, 1 errored, 0 ignored.

A test package that is raising an exception in aftertest - each test is reported ok but then each is marked as failed

Remove rooms by name
  Removes a room without content in it (FAILED - 1)
  Does not remove room when it has content
  Raises exception when null room name given
 
Failures:
 
  1) remove_empty_room
        
        error: ORA-20001: EXCEPTION TEST
               ORA-06512: at "UT3.TEST_REMOVE_ROOMS_BY_NAME", line 40
               ORA-06512: at line 6
       
Finished in ,025337 seconds
3 tests, 0 failed, 1 errored, 0 ignored.

Pazus · 2017-02-17T06:21:24Z

What currently I don't like is that warnings are not connected to the place were error occured. Should be improved.

jgebal · 2017-02-22T20:09:05Z

I think that if beforeall/beforeeach fails, we should fail the tests that depend not hose blocks.
For aftereach, we should also fail test.To do that, we need however to report on the test execution after the afteraeach block was executed for that test.

If the afterall block fails, we should not fail the tests that already have executed successfully.
Instead, we could report exceptions raised in before/after blocks on test/suite level.

To do that:

test would need to have placeholder for exceptions
suite would need to have placeholder for exceptions
each dynamic block (before/exec/after) invoked in test would populate exceptions list in the ut_test object
each dynamic block (before/after) invoked in suite, would populate exceptions list in the ut_suite object
Reporters should include exceptions outputs from both suite level and test level.
If you look at the XSD for JUnit you will notce that it is supporting stderr and stdout on suite level.

Teamcity reporter should indicate progress using the after_test hook.
I think we should remove some of the reporter hooks. They give lots of flexibility, but also bring confusion on when to use which hook use them.
Maybe hooks before/after run/suite/tests (6 hooks) would be sufficient?

jgebal · 2017-02-22T20:15:31Z

I like the new behaviour though not sure about the way it's implemented. Need to think about ways to make it cleaner.

…as failed. Now the after_test gets executed regardless of before_test status

…/utPLSQL into fix-for-invalid-package-v2 # Conflicts: # source/core/types/ut_test.tpb

Pazus · 2017-02-23T09:05:31Z

Behavior after recent changes:
New behavior:
A test package without body - each test is reported as failed
A test package with invalid body - each test is reported as failed
A test package with invalid package spec - tests are skipped. Only valid specifications are parsed for annotations.
A test package that is raising an exception in beforeall - each test is reported as failed
A test package that is raising an exception in afterall - each test is reported ok, warnings are displayed

Remove rooms by name
  Removes a room without content in it
  Does not remove room when it has content
  Raises exception when null room name given
 
Warnings:
 
  1) test_remove_rooms_by_name - Afterall procedure failed: 
       ORA-20001: Test exception
       ORA-06512: at "UT3.TEST_REMOVE_ROOMS_BY_NAME", line 35
       ORA-06512: at line 6
 
Finished in ,044902 seconds
3 tests, 0 failed, 0 errored, 0 ignored. 1 warning(s)

A test package that is raising an exception in beforeeach - each test is reported as failed
A test package that is raising an exception in aftereach - each test is reported ok but then each is marked as failed

Remove rooms by name
  Removes a room without content in it
  Does not remove room when it has content
  Raises exception when null room name given
 
Warnings:
 
  1) test_remove_rooms_by_name - Aftereach procedure failed:
       ORA-20001: Test exception
       ORA-06512: at "UT3.TEST_REMOVE_ROOMS_BY_NAME", line 31
       ORA-06512: at line 6
 
  2) test_remove_rooms_by_name - Aftereach procedure failed:
       ORA-20001: Test exception
       ORA-06512: at "UT3.TEST_REMOVE_ROOMS_BY_NAME", line 31
       ORA-06512: at line 6
 
  3) test_remove_rooms_by_name - Aftereach procedure failed:
       ORA-20001: Test exception
       ORA-06512: at "UT3.TEST_REMOVE_ROOMS_BY_NAME", line 31
       ORA-06512: at line 6
 
Finished in ,05071 seconds
3 tests, 0 failed, 0 errored, 0 ignored. 3 warning(s)

A test package that is raising an exception in test - the test is marked as failed

Remove rooms by name
  Removes a room without content in it (FAILED - 1)
  Does not remove room when it has content
  Raises exception when null room name given
 
Failures:
 
  1) remove_empty_room
        
        error: ORA-20001: Test exception
               ORA-06512: at "UT3.TEST_REMOVE_ROOMS_BY_NAME", line 48
               ORA-06512: at line 6
       
Finished in ,035726 seconds
3 tests, 0 failed, 1 errored, 0 ignored.

A test package that is raising an exception in beforetest - each test is reported as failed

Remove rooms by name
  Removes a room without content in it (FAILED - 1)
  Does not remove room when it has content
  Raises exception when null room name given
 
Failures:
 
  1) remove_empty_room
        
        error: ORA-20001: Test exception
               ORA-06512: at "UT3.TEST_REMOVE_ROOMS_BY_NAME", line 39
               ORA-06512: at line 6
       
Finished in ,039346 seconds
3 tests, 0 failed, 1 errored, 0 ignored.

A test package that is raising an exception in aftertest - each test is reported ok but then each is marked as failed

Remove rooms by name
  Removes a room without content in it (FAILED - 1)
  Does not remove room when it has content
  Raises exception when null room name given
 
Failures:
 
  1) remove_empty_room
        
        error: ORA-20001: Test exception
               ORA-06512: at "UT3.TEST_REMOVE_ROOMS_BY_NAME", line 42
               ORA-06512: at line 6
       
Finished in ,045523 seconds
3 tests, 0 failed, 1 errored, 0 ignored.

…invalid-package-v2

jgebal · 2017-02-23T09:36:18Z

A small update to the description (in bold)

A test package that is raising an exception in aftertest - each test is reported as failed

Remove rooms by name
  Removes a room without content in it (FAILED - 1)
  Does not remove room when it has content
  Raises exception when null room name given
 
Failures:
 
  1) remove_empty_room
        
        error: ORA-20001: Test exception
               ORA-06512: at "UT3.TEST_REMOVE_ROOMS_BY_NAME", line 42
               ORA-06512: at line 6
       
Finished in ,045523 seconds
3 tests, 0 failed, 1 errored, 0 ignored.

…code that is raising exceptions in different blocks. Changed error message raised when suite is not found to be more adequate to the reasons for the error.

jgebal · 2017-02-24T00:55:09Z

I think the code is in a good shape to be merged.
Before merging however, it would be good to update the documentation. The comments on this PR are of a high value and I'm pretty sure, we will get many questions around how thins work in those cases. Having that documented seems to be the best option as we can refer to the doc at any time.

…code that is raising exceptions in different blocks. Changed error message raised when suite is not found to be more adequate to the reasons for the error.

Pazus · 2017-02-24T07:51:35Z

Documented the behavior

fix for invalid package v2

e926425

Pazus added the bugfix label Feb 4, 2017

Pazus added this to the version3 milestone Feb 4, 2017

Pazus self-assigned this Feb 4, 2017

Pazus added the in progress label Feb 4, 2017

Merge branch 'develop' into fix-for-invalid-package-v2

88da3e2

Merge branch 'develop' into fix-for-invalid-package-v2

8c615dd

teststep

7840258

Pazus added 3 commits February 8, 2017 21:26

implemented fail procedure at ut_test_item level to push failures t…

40a9b44

…o child items

Merge remote-tracking branch 'remotes/upstream/develop' into fix-for-…

227fe70

…invalid-package-v2

made buffer clear autonomous

ff26e36

introduced warning concept

9caa870

Pazus requested a review from jgebal February 17, 2017 06:21

Pazus added the enhancement label Feb 17, 2017

jgebal and others added 3 commits February 22, 2017 21:35

Changed logic, where the after_test was not executed if before_test h…

84d6dd4

…as failed. Now the after_test gets executed regardless of before_test status

refactored

fa52a09

Merge branch 'fix-for-invalid-package-v2' of https://github.com/Pazus…

a16f130

…/utPLSQL into fix-for-invalid-package-v2 # Conflicts: # source/core/types/ut_test.tpb

Pazus added 2 commits February 23, 2017 12:06

refactor reporting

7fa55d0

reverted test staff

2c9835b

Pazus added 2 commits February 23, 2017 12:08

Merge remote-tracking branch 'remotes/upstream/develop' into fix-for-…

565340f

…invalid-package-v2

last minute change

26273ad

Added tests for behavior when execution invalid objects or unit test …

4d2789c

…code that is raising exceptions in different blocks. Changed error message raised when suite is not found to be more adequate to the reasons for the error.

jgebal approved these changes Feb 24, 2017

View reviewed changes

jgebal and others added 4 commits February 24, 2017 00:55

Added tests for behavior when execution invalid objects or unit test …

1be8914

…code that is raising exceptions in different blocks. Changed error message raised when suite is not found to be more adequate to the reasons for the error.

Update annotations.md

c8ead98

Update annotations.md

4695d62

Create exception-reporting.md

96d3d25

Updated exception-reporting.

ee89c42

jgebal merged commit e023d9e into utPLSQL:develop Feb 24, 2017

jgebal removed the in progress label Feb 24, 2017

Pazus deleted the fix-for-invalid-package-v2 branch February 24, 2017 11:49

Conversation

Pazus commented Feb 4, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Pazus commented Feb 4, 2017

Uh oh!

Pazus commented Feb 4, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jgebal commented Feb 4, 2017

Uh oh!

Pazus commented Feb 5, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Pazus commented Feb 5, 2017

Uh oh!

Pazus commented Feb 8, 2017

Uh oh!

Pazus commented Feb 17, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Pazus commented Feb 17, 2017

Uh oh!

jgebal commented Feb 22, 2017

Uh oh!

jgebal commented Feb 22, 2017

Uh oh!

Pazus commented Feb 23, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jgebal commented Feb 23, 2017

A small update to the description (in bold)

Uh oh!

jgebal commented Feb 24, 2017

Uh oh!

Pazus commented Feb 24, 2017

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Pazus commented Feb 4, 2017 •

edited

Loading

Pazus commented Feb 4, 2017 •

edited

Loading

Pazus commented Feb 5, 2017 •

edited

Loading

Pazus commented Feb 17, 2017 •

edited

Loading

Pazus commented Feb 23, 2017 •

edited

Loading