Q7

Parameterized tests

Details

  • Test Mode:
    Manual

Description

Why

Each conference we are being asked on whether it is possible to have parameterized tests in Q7. Here is a couple of use cases:

  • Forms testing. For instance, we want to test Java's new class wizard. Instead of having several test cases entering different values to 'class' and 'package' fields we can have a single parameterized test case.
  • UI Settings. Suppose we have added a dark theme to Eclipse and want to execute all tests on both light and dark themes. We can have two preference contexts.

What

The idea is to have a new artifact type – Supercontext (name is subject to change). Supercontext is a context which can refer to multiple contexts of given type. For instance, Workspace supercontext refers to 1..* workspace contexts.

When supercontext is added to a test, then test should be executed with each of referred contexts. Thus, if test refers to more than one supercontext, all possible combinations should be covered. So if 1st supercontext refers 10 contexts and 2nd supercontext refers 3 contexts, we should execute a test case 30 times.

Execution view (and therefore an html report) should display these executions as separate test cases with a name of test case suffixed by all contexs, like this: TestNewClassDialog (darkTheme, invalidClassName). Ideally, if user selects such a test case in execution view, we need to be able to replay only selected variant of test.

How

When user creates a new Supercontext, he selects a context type (any of existing context type, including Group and Parameters). Editor contents should be pretty much the same as in Group context, except that there should be no 'Apply' and 'Capture' buttons. Drag'n'drop from Q7 Explorer should work as well.

Further we are going to make an editor more complex in case it refers to multiple group contexts. Rougly speaking, when user edits Group supercontext, editor contents looks like this:

  Contexts Parameter1 Parameter2
variant1 Workspace1, Worksbench1 value1.1 value1.2
variant2 Workspace2, Workbench2 value2.1 value2.2

Basically what we see here is that each row represents a group context, 1st column displays context name, 2nd column shows referenced contexts and further columns just a special representation of parameter context, where all values are displayed as separate columns (assuming that there's exactly one parameter context inside a group context), so it is better to see overall picture and makes us possible to add new parameter to all contexts at once.

And then we can go further – in a table above group and parameter contexts don't have any value outside of this supercontext. And if there's 10 test variants, we dont' need to store 20 extra files (1 group context and 1 parameter context per each variant), so we can store them 'embedded' inside a supercontext.

Preliminary plan

  1. Simple UI and persistence (in no particular order)
    • EMF model
    • New supercontext wizard
    • Icon decoration – so that I can see that this file is Preferences supercontext.
    • Simple Editor
  2. Execution tweaking
    • Recognize that this test case spawns N parameterized executions
    • Ability to instantiate and execute a particular parameterized execution (if one of my 100 variants fails, I don't want to wait for other 99 to be executed when launching it from Q7 IDE)

Execution modification is the most dangerous and challenging part here. I think it should be possible to 'expand' a supertest into a bunch of parameterized tests during execution preparation, so that existing launch delegate won't even know about all this stuff, and execution view with execution reports will work correctly automatically. The only potential trap here is when user adds new context to supercontext (and thus increases variants count), but we use cached launch configuration. Also this will make things simple with Q7 Server – all supertests will be expanded on the client and server won't even know about this.

Activity

Hide
Andrey Sobolev added a comment - 18/Mar/13 2:40 PM

Initial version are commited to master.

+ New SuperContext wizard
+ execution
+ HTML, Multiple HTML reports are updated.
+ selective execution of only required configurations from ExecutionView
+ SuperContext editors
+ Initial label decoration for super contexts.

No server support at this moment.

Show
Andrey Sobolev added a comment - 18/Mar/13 2:40 PM Initial version are commited to master. + New SuperContext wizard + execution + HTML, Multiple HTML reports are updated. + selective execution of only required configurations from ExecutionView + SuperContext editors + Initial label decoration for super contexts. No server support at this moment.
Hide
Andrey Sobolev added a comment - 10/May/13 5:27 PM

Cloud support for default and super contexts are added.

Show
Andrey Sobolev added a comment - 10/May/13 5:27 PM Cloud support for default and super contexts are added.

People

  • Assignee:
    komaz
    Reporter:
    komaz
Vote (0)
Watch (0)

Dates

  • Created:
    11/Mar/13 3:10 PM
    Updated:
    28/Jun/13 7:15 PM
    Resolved:
    28/Jun/13 7:15 PM