Model Based Testing
Is just another testing concept. it's mainly built around the idea of using the models which describe a system to produce test cases to test the system. in UML we can look at a system's statecharts diagram to generate test cases that will guarantee state coverage for example. or even other coverage criteria. another example is use case based testing, which uses the use case model to generate high level user acceptance test cases.
Test Modeling
Test modeling describes an approach to model the overall testing activity of a system. specifically the testing system could be considered a seperate system of the testable system. and then it can be modeled through structural, behaviour and implementaion diagrams. in a nutshell test modeling is all about descibing testing concepts of a system as models that go hand in hand with the system under test models.
Showing posts with label model based testing. Show all posts
Showing posts with label model based testing. Show all posts
Tuesday, July 22, 2008
Wednesday, May 21, 2008
Model based testing
1. An overview of model based testing:
Model based testing approach basically attempts to automate the test design rather than the execution. this is done b generating tests automatically of the SUT. in general, we usually go from requirements into test cases. but here the approach is to model different aspects of the system and then generate test cases to test that model. MBT is a black box testing with some functional and up to system testing.
The France telecoms example: a service let you enter a phone number and the system will produce a name and an address that is associated with that number. A finite state machine model could be written to model this use case.
To generate tests for this use case, we could for example take a random walk or a path through the different states in the state automaton, by defining the actions at each stage. A smarter algorithm to archive coverage exists, for example the Chinese post man algorithm. or trying all the transitions in a state, one by one, like testing all the interactions.
other algorithms are testing all variants of a given use case, which is sort of user directed.
there is two approaches to execute these tests:
off-line : generate test cases and store them in files, execution are separate .
online : generation and execution are tightly coupled. this is very important when SUT is indeterministic. so essentially you generate a test case, and then execute it. waiting for the result before generating the next.
Benefits of MBT:
the real cost saving is in test design: modeling time is arguable less than manual test design time. other benefits include enhancing requirements from the raised questions throughout modeling. it helps to automate traceability. the only draw back is modeling, since it can be costly and needs expertise.
2. Black box Vs White box models:
Test generation algorithm:
black box model:
in the black box model we need to generate tests from a model inside a black box, which means we don't see the internal of the model. it however, always tells us the current state, and provides a list of possible inputs. we then explore the model and find out about the behavior from the different input sequences we put in, and the corresponding state.
in white box model:
it tells us upfront what kind of behavior it has. by looking at the internals of this model, one can perform full analysis on all possible paths it can take. and thus try them all.
To model complex systems we use extended finite state machines(like a programming language with variables, guards). which could give us much more expressiveness.
Pros and Cons:
black:
uses executable actions, which means we can use any language that is executable( java ), but harder to understand since you cant see the model but rather have to explore it.
white:
uses unfamiliar notation such as UML/OCL, but easy to analyze .
you can have some sophisticated reasoning. and plan the paths ahead.
3.Black Box model example
modelJUnit : MBT with black box. allows you to write models in java (familiar) with full power of the language ( more expressivity) integrate well with JUnit ( focuses on unit testing)
provide simple test generation algorithms (random walk, greedy walk ...etc)
Imagine we wanna test the set element of the java elements . simplify to use only two elements. then we can model this FSA in java, by modeling the state as in having elements in the two slots by booleans. and model the transitions as methods that are annotated as actions.
so then a test user will choose the test generation algorithm, and the model to generate the tests.
so then a 150 action sequences get produced, which are the test cases . now these tests can be used to do offline or even online testing, but we need to have an oracle to tell us if the tests are passing . like for every change the test model performs, the SUT performs a change. we need to check that these are matching.
now to really mesure how good these tests that were generated, is through a tool. this is done throught the concept of mutation. so this tool will take on your junit tests and run them on mutated code, and if the test doesnt cover the mutation then the test is not really that good. its realted to code coverage but not the same.
4.white box model example
model in UML, then export the tests from that and going the test designer to generate some tests, and export them into some executable language to the to execute them . imagine we have a library system, we model it in UML, then we model an object diagram to generate test data. we also describe the methods in OCL. now to generate tests we can use the constraints in OCL to guide us thought the right path in state machine.
checkout this awesome Google talk :
http://video.google.com/videoplay?docid=5521890509476590796
Model based testing approach basically attempts to automate the test design rather than the execution. this is done b generating tests automatically of the SUT. in general, we usually go from requirements into test cases. but here the approach is to model different aspects of the system and then generate test cases to test that model. MBT is a black box testing with some functional and up to system testing.
The France telecoms example: a service let you enter a phone number and the system will produce a name and an address that is associated with that number. A finite state machine model could be written to model this use case.
To generate tests for this use case, we could for example take a random walk or a path through the different states in the state automaton, by defining the actions at each stage. A smarter algorithm to archive coverage exists, for example the Chinese post man algorithm. or trying all the transitions in a state, one by one, like testing all the interactions.
other algorithms are testing all variants of a given use case, which is sort of user directed.
there is two approaches to execute these tests:
off-line : generate test cases and store them in files, execution are separate .
online : generation and execution are tightly coupled. this is very important when SUT is indeterministic. so essentially you generate a test case, and then execute it. waiting for the result before generating the next.
Benefits of MBT:
the real cost saving is in test design: modeling time is arguable less than manual test design time. other benefits include enhancing requirements from the raised questions throughout modeling. it helps to automate traceability. the only draw back is modeling, since it can be costly and needs expertise.
2. Black box Vs White box models:
Test generation algorithm:
black box model:
in the black box model we need to generate tests from a model inside a black box, which means we don't see the internal of the model. it however, always tells us the current state, and provides a list of possible inputs. we then explore the model and find out about the behavior from the different input sequences we put in, and the corresponding state.
in white box model:
it tells us upfront what kind of behavior it has. by looking at the internals of this model, one can perform full analysis on all possible paths it can take. and thus try them all.
To model complex systems we use extended finite state machines(like a programming language with variables, guards). which could give us much more expressiveness.
Pros and Cons:
black:
uses executable actions, which means we can use any language that is executable( java ), but harder to understand since you cant see the model but rather have to explore it.
white:
uses unfamiliar notation such as UML/OCL, but easy to analyze .
you can have some sophisticated reasoning. and plan the paths ahead.
3.Black Box model example
modelJUnit : MBT with black box. allows you to write models in java (familiar) with full power of the language ( more expressivity) integrate well with JUnit ( focuses on unit testing)
provide simple test generation algorithms (random walk, greedy walk ...etc)
Imagine we wanna test the set element of the java elements . simplify to use only two elements. then we can model this FSA in java, by modeling the state as in having elements in the two slots by booleans. and model the transitions as methods that are annotated as actions.
so then a test user will choose the test generation algorithm, and the model to generate the tests.
so then a 150 action sequences get produced, which are the test cases . now these tests can be used to do offline or even online testing, but we need to have an oracle to tell us if the tests are passing . like for every change the test model performs, the SUT performs a change. we need to check that these are matching.
now to really mesure how good these tests that were generated, is through a tool. this is done throught the concept of mutation. so this tool will take on your junit tests and run them on mutated code, and if the test doesnt cover the mutation then the test is not really that good. its realted to code coverage but not the same.
4.white box model example
model in UML, then export the tests from that and going the test designer to generate some tests, and export them into some executable language to the to execute them . imagine we have a library system, we model it in UML, then we model an object diagram to generate test data. we also describe the methods in OCL. now to generate tests we can use the constraints in OCL to guide us thought the right path in state machine.
checkout this awesome Google talk :
http://video.google.com/videoplay?docid=5521890509476590796
Wednesday, May 7, 2008
A Model based, Model transformation testing framework
I did a presentation on model transformation testing last week ...and I am posting it here
Subscribe to:
Posts (Atom)