There are not many conferences on Software Testing that are held here in Canada, well this year a big one is. For the first time ever CAST 2008 conference is being held right here in Toronto. The show theme is Beyond the Boundaries: Interdisciplinary Approaches to Software Testing.
Mark you calendar for July 14-16th.
If your interested in speaking the Call for Papers deadline is February 4th, 2008.
To see more information on the conference click this link CAST 2008
Tuesday Jan. 29th, 2008 at the Toronto Association of Systems and Software Quality (TASSQ) meeting join myself and Barry Gervin for an evening presentation on VSTS Edition for Testers. Click the link for details about this presentation including a link for registration.
Hope to see you there ... Testa
Microsoft has launched a new community for software testers to share and learn about everything testing. There are and will be videos, articles, and blogs all dealing with software testing.
Join the Software Testing Discussion Forum and participate through out conversations.
Check it out -> Microsoft Tester Center
Sean Lumlev from Microsoft has done an excellent job explaining Web Test Correlation and how VS Team System 2008 (Orcas) new helper feature works.
Check out Sean’s blog by clicking this link Web Test Correlation and read all about it.
VS Team System 2008 (Orcas) has improved validation rules.
Test-level validation rules allow you to add the same rule to every request by simply clicking the Web test node and selecting Add Validation Rule. A dialog box opens select the validation rule, in the properties you can edit or add options and parameters. Click Ok and you’re done.
Reasons for using a test-level rule:
· Errors in the server application can cause incorrect flow causing the playback of the Web test to vary from the recorded sequence
· Validate content that must appear on every page
· Validate that all pages conform to a specific criterion
VS Team System 2008 (Orcas) has improved validation rules.
Listed below are new validation rules:
· Stop test on error
· Search request and response
· Add validation rule for title
· Redirect validation
· Provide test level validation rules
· Expected HTTP code
· Warning level for errors on dependents
AJAX Requests & JavaScript Pop-ups
Currently VSTS 2005 requires an add-in tool called Fiddler to record AJAX requests. If you are not sure what AJAX is watch this video Measuring the Business Value of AJAX . As shown in the video using Fiddler we have the ability to capture AJAX requests, then using VSTS we are able to manage the Fiddler recordings in VSTS, see the requests in results and add the tests to our Load tests. If you watch the video the value AJAX adds is pretty impressive.
VSTS 2008 (ORCAS) now gives you a new feature, it captures AJAX requests with the web test recorder.
Along with AJAX the web test recorder also is able to record JavaScript pop-ups.
Fiddler is no longer required – VSTS 2008 Web test recorder does the job.
Orcas - Visual Studio Team System 2008 - new features coming for Web and Load testing and a feature of interest to QA in Unit testing. Stay tuned I will be blogging on the list of new features below:
Web Testing
· Recorder now records AJAX and popups
· Test-level validation rules
· Auto-validate response URL
· Composition/decomposition
· Easily bind to CVS or XML
· UI improvements in playback
· Support for multiple web test plug ins and parameterized plug ins
· API improvements
· Expected HTTP code
· Request-level plug ins
· Correlation helper
Load Testing
· Load test result manager
· Summary report
· Support for multiple graphs, new performance graph, vertical zoom, synch zoom
· New load model for user pacing, users running
· User init/terminate functions
· Support for iteration count, cool down
· Unit tests under load performance improvements
· Profiler Integration
Unit Testing
· Data binding improvements
o Easily bind to CSV, XML
o Auto-deploy local files and fix up connect strings
o Iterate correctly in a load test
Surfing the internet can be mind boggling when searching industry terminology. Try searching terminology for testing that a solution is able to handle its data obligations while conducting itself in a suitable fashion. How do you like those words?
Terminology I found and have experienced:
Stress Testing
Load Testing
Performance Testing
Volume Testing
Concurrency Testing
Endurance Testing
Scalability Testing
Soak Testing
What does all this mean and why has Microsoft in Team System only used two of these?
I believe Microsoft in Team System has done the right thing. How many terms do we need to identify testing a solution can handle real life usage with peak times and large amounts of data?
VS Team System Performance testing is designed for developers to measure, evaluate, and target performance issues in their code. Profiling is used in performance testing; it is the process of observing and recording metrics about the behavior of solution code. Issues normally stem from code that performs slowly, or uses too much system memory.
Load testing is used to simulate many users accessing a server at the same time. In load testing, the web tests simulate multiple users opening simultaneous connections to a server and making multiple HTTP requests. Unit tests added to a load test can be used to test data access.
With all the features of the Load tool you are definitely stressing the solution, adding volume, testing concurrency, endurance, scalability and soak so why all these terms. With a combined performance test of developer code and load testing of the solution, we are testing the “Performance” of the solution in my opinion. Let’s keep it to the two terms so everyone understands what’s being tested and why.
Quote for today.
The ability to simplify means to eliminate the unnecessary so that the necessary may speak. (Hans Hofmann)
Have a great weekend … Testa J
I have blogged about testing early, getting your test team involved at the start of the project one item I neglected to include was Load testing. So let’s catch up and talk about when load testing should be done. A lot of projects wait till either UAT or just prior to implementation to perform Load testing, does that make sense?
What is Load Testing? -- It is the process of creating demand on a system or devise and measuring its response. It is modeling the expected usage of a software program by simulating multiple users accessing the program’s services concurrently.
With VS Team System Load Tester you are grouping and running your existing unit and web tests. When a build is ready the unit tests are run against that build. We are verifying the build quality and regression testing that any fixes made did not break something else. The QA team gets a quality build and run their web and manual tests and report any bugs. If the bug’s reported are not high severities then let’s put the unit and web tests into a load test and run it. You’ll have to do some planning ahead like a clean environment and data set up for the load testing. An important factor during load testing is having an appropriate data size to find issues like missing indexes, to many indexes, locks, bad query plans, the list goes on and on.
Load testing early will find performance issues sooner which in most instances are due to architectural issues in the application. Finding these just before implementation can turn into a No Go decision. Finding them early allows lots of time to analyze and make the necessary changes, and you can retest the fix without delaying other testing, development or implementation.
This is a best practice that with VS Team System allows us to do so easily it’s crazy not too.
Quote by Andy Warhol (1928-1987): “They always say time changes things, but you actually have to change them yourself.”
Lets change Load Testing start early don’t leave it till the end.
Enjoying a sunny summer … Testa J
Every test you create whether it is Unit, Manual, Web or Load can have work items associated to them. If you’re working with the MSF Agile process the work item Scenario is a detailed requirement, a bug is a work item, risk is documented in a work item. You have the ability to associate work item(s) to your tests. Remember that work items are stored in the TFS as is test results, you can run queries against anything stored in the TFS, you therefore have the ability to report on the testing of requirements, risk, bugs. Think about knowing that a bug has been retested and that it’s passed. I know prior to VSTS we used a lot of excel spreadsheets and human intervention to document this information, now it’s a click of your mouse, select the work item to associate and your done. Run your queries and the information displays.
How to associate a work item to a test …
To access a tests property open the window Test View which gives you a view of all your tests created. Right click on a test and select properties, the Properties window will open under “Misc” you will find the property Associate Work Item. Note that you can move the Properties window to the right of your screen under Team Explorer/ Solution/Project and Test View. You can also do this using Test Manager.
Next Steps:
1. Click the button icon in the Associate Work Item property field
2. Window opens with an Add button enabled
3. Click the Add button
4. The Work Items Picker window opens
5. Select the Team Project your working on
There are three ways to get the work item(s) you want to associate you can select a query, key in the work item identifier number, use the work item title and type. The easiest way would be to find work items by query.
6. By default the option “Saved query:” is selected click the dropdown and select query named “All Work Items”
7. Click the Find button
The Select items to add to the work item list displays all work items in your project and by default they are all selected.
8. Click the Unselect All button
9. Pick a scenario or bug or any work item you please
10. Click OK
Your returned to the previous window with the work items identifier’s displayed. If you’ve selected incorrectly you can click on an id and remove or click Details to see the actual work item.
11. Click OK again
The test property Associated Work Item now populates with the work items identifier number you selected.
Picture of the windows talked about above:
Manual Test Property:
Work Item Picker:
Run queries and see what scenarios your tests are covering, if your real good query to see if what scenarios don’t have test associated.
Try it out … Testa
We've heard of code coverage which is normally used by the developer's to verify how much of the code they've written was tested by their Unit tests. Microsoft took it a step beyond developers and gave the QA the ability to run code coverage against our tests. You can with any test type including Manual Tests have code coverage running during execution and see how much of the code is being tested. It's a great way to know whether or not the QA test suites created are covering all the code. With both developers and testers running code coverage against a Build we get a real good idea of how good that Build really is.
Check out Eric Sink's blog on Code Coverage at -> http://www.ericsink.com/articles/Code_Coverage.html
A fellow by the name of Paul Stovell did some creative thinking and came up with a tool called Scenario Coverage Analyser for TFS, which basically is an MSBuild task. Scenario Coverage Analyser allows you to summarize your code coverage statistics in terms of the Scenarios they cover. We are assuming you’re using the work item Scenario to document your requirements in VSTS.
A quick explanation of how this works, you modify the Team Builds to call Scenario Coverage Analyser. You run your Team Build and Unit Tests with code coverage Scenario Coverage Analyser does three things;
- Looks at the code coverage results generated
- Maps the individual code coverage results to the work items that the code was written for
- Produces an HTML report called Traceability Matrix
This add-in tool only runs with Builds so it does not work with the QA Test execution, but it gives the QA Team a report on how much of each Scenario (requirement) was tested by the Unit Tests.
Thoughts to comment on:
If a Scenario shows that the unit test tested 90% of its related code how much more testing of that Scenario would QA really need to do?
What % of Scenario Coverage would be acceptable?
Would this be a report you'd share with your customer?
Is there a like tool that could do the same analysis of code coverage run against the QA Tests?
Wouldn't that be a nice add-in for the QA Team? 
To see more on this great add-in tool here is a link to Paul Stovell's blog ->
http://www.paulstovell.net/blog/index.php/scenario-coverage-analyser-for-tfs/
Paul’s how to set up Scenario Coverage Analyser ->
http://www.paulstovell.net/blog/index.php/scenario-coverage-analyser-for-tfs-setting-it-up/
To download the tool -> http://readify.net/Default.aspx?tabid=269
Try it out and let us know how it works for you - add a comment to my blog with your thoughts.
Thanks to Paul for this great addition to VSTS.
Another sunny day ... Testa 
ObjectSharp is now offering a Team System Quality Assurance Best Practices course of which I blogged about a few blogs ago.
The course and labs cover the following topics:
- QA Role in the Software Development Life Cycle
- Overview of Visual Studio Team System
- Team Communications
- Working with Visual Studio
- Version Control
- Manual Testing
- Web Testing
- Other Testing
- Load Testing
- Builds
- Defect Tracking
For more details on the course go to: http://www.objectsharp.com/training/CourseDetail.aspx?id=5030
Link to the public course schedule: http://www.objectsharp.com/training/ClassSchedule.aspx
We offer a public course in Toronto or we can host your team’s private, customized course at our fully equipped facility. We also offer onsite, customized training with our portable laptops.
I want to also share with you a great place for a weekend away. I was at a family wedding this past weekend and stayed at the “Kingbridge Centre” which is a conference centre that is open for anyone to stay. I highly recommend it, great place with lots to do and it’s just 45 minutes from downtown TO. Check it out at www.kingbridgecentre.com . The wedding was great too!
On the phone in our room at Kingbridge was a quote by Alexander Graham Bell it says alot about what I believe is the key to success in IT projects.
“Before anything else, preparation is the key to success”
Talk to you later … Testa
Best Practice in any project using any tool is involving the QA Team, the project community at the beginning of the project planning stage. I've blogged about this previously but now I'd like to share how the QA Team can help and what they would be doing.
During project planning the QA Mgr and/or Lead should be involved so that during planning a comprehensive understanding of what is involved in the overall project can be obtained. The QA representative can help in planning from knowledge of previous projects for instance test timelines, test needs, resource needs, and has the opportunity to share ideas of how QA can assist other teams.
During preliminary project planning the QA Mgr/Lead can then start planning out the QA Team so that they have the right people sourced and available to start as soon as deemed required. This eliminates running around trying to hire QA resources a week prior to testing, which normally tends to be a disaster.
Now you have a QA team with a manager or lead, what do we do, get involved with requirements. I attended a web cast recently called Elicit Effective Requirements While Building Trust. It seems a lot of people are talking about "Requirements" these days the same points keep being repeated, see the list below:
Requirements development to include:
· plan to involve the stakeholders
· creation of a product vision
· establish a common glossary
· use multiple elicitation methods
· involve the entire project community
· use multiple model techniques both text and diagrams
· identify scope early - drive modelling from scope
· verify as you iterate
· prioritize continually
The next step is to involve the entire project community including stakeholders (the business) to create a vision, document the scope then analyze and verify requirements at the same time identifying priorities and risk. A great way to do this is by conducting workshops where requirement development is conducted in an organized, well planned atmosphere. I'll share tips I got from the web cast for a successful workshop. From experience this does work great in fact during a consultant gig I did where planning was started without the stakeholder’s I suggested to the project manager we do some workshops with the stakeholders. We basically followed this same pattern and what came out of it was the needs of the stakeholders had not been fully understood and what has being planned in development was about 70% incorrect. The workshops with the stakeholder early in the project saved the day in that situation.
Workshop Tips:
Before
· Use a planning team.
· Mind you’re "P’s": purpose, participants, principles, products, place, process.
· Uncover hidden agendas.
· Expose decisions.
During
· Establish and use ground rules.
· Make decision-making transparent.
· Openly acknowledge conflict.
· Build in serious play and prototypes.
· Enrich the work with multi-models and wall work.
After
· Conduct a sponsor show-and-tell.
· Facilitate a workshop retrospective.
· Share outcomes with the entire community.
· Use feedback to improve the process.
How does VSTS help in all this?
You can use templates that come with VSTS or upload your own for gathering and documenting high level requirements. With the ability to share using VSTS Team Explorer or the website Project Portal or for those that don’t have VSTS use Team Plain free (with a TFS license). I’ll blog about Team Plain another day but it gives your project community full edit/update access to work items with out having to have VSTS installed. During the workshop you can use the VSTS templates, Vision, the Persona, Requirements List, Issues, Project Checklist, work items Scenario and Quality of Service Requirements to translate the work into documents. Use work item Task to document additional work even assign it, work item Risk to identify areas that have been identified as a risk to the project. All this is stored in TFS and accessible to the project community. An easy format in workshops to get the documents written and updated is to assign them to specific people then have others proof read and a signoff process. Keep in mind one persons interpretation may not be another’s so it has to be a team effort. During the workshop there maybe times when you have to agree to disagree this is a good time to use the Triage List to track these instances and go back to them later with additional input from the community for clarification.
Once all requirements have been reviewed, analyzed and fully documented present them to the project sponsor’s, the entire community, get feedback, make revisions and finally get signoff.
The last step before the individual teams break out to start the next cycles have another workshop meeting where you plan out Builds, what will be in each build, estimate timelines, identify testing needs for each build, arrange tester/developer unit test design, regression testing of the builds, what happens if a build does not pass validation, logging of bugs found in builds, development load testing any work effort that can be identified and planned involving builds. Attendee’s at this workshop should include a Release Manager, Development, QA and Project Management builds affect all these areas’s and therefore needs to be a collaborative effort. All the build information should be documented, published and stored in VSTS TFS.
To conclude planning, requirement development workshops and build planning workshops help all teams in the project community to enable better team planning creates a well informed project community, the left hand knows what the right hand is doing, and it’s a gigantic step towards a successful project. VSTS aids the project community in organizing the documentation that comes from planning and workshops and allows easy communication to everyone. The requirements become work items that will be tracked as the project progresses, tasks identifies other work to be done, risk identifies stuff to watch and may even require further analysis, this all is the initial setup of your VSTS project.
As for the QA Team they now have a good understanding of what will need tested and can plan out their testing, make sure they have the right resources, they can work with development in creating better more comprehensive unit tests, plan for early load testing with unit tests, start getting tests cases/scenario’s written for the planned builds, they can determine what they will need to test thereby eliminating test duplication within the project community. Bring it on we’re ready!
Barry Gervin an ObjectSharp partner commented once; If QA is responsible for the quality of the product how come they aren’t involved from the start of the project. How come they aren’t involved in the unit testing, build verification, requirement verification? It is a good question and one that all Software Projects should be asking themselves.