The Test Team at DSI have been using Microsoft Visual Studio Team System (VSTS) 2008 Test Edition for over a year now. We first began using the beta version of the 2008 product late last year and have been consistently relying on the released product for both Web testing and Load testing since then. So now’s as good a time as any to do our ‘Year in the Life’ review of this product and see what we know now, that wasn’t so clear, when we first began using the product.
First up I took Mark, one of our application testers aside, and asked him some questions about the product to see what new nuggets of information he could share, then I spoke to Ciara and Eoin the other application testers on my team, here is how those conversations went.
Peter>> “Hi Mark how’s it going?”
Mark>> “Hi Peter I’m fine, I’m just running some automated web tests.”
Peter>> “Good timing so, I just wanted to see how your getting on with the Microsoft Visual Studio 2008 Team System, Test product?”
Mark>> “Really well actually. I have now automated most of our application testing. We are in maintenance mode now and VS application has now really come into its own, we have written web tests for the majority of the functionality and it’s really cut down on the regression time when new builds are being released.”
Peter>> “That’s great and Mark I was wondering does it take much time to keep the web tests up to date, surely they break from time to time?”
Mark>> “That’s right Peter, but I have found that the effort involved isn’t that great really. Having said that I have been using the product on a consistent basis and I have really come to understand the product. Often what tends to happen is that I’ll get a new build and in running the web tests I’ll see some tests are failing, when I look closer it becomes evident that something has changed in the application I am testing and the test no longer applies in the way that it did before. This is good really?”
Peter>> “Why is that good surely you’d hope all your tests would pass?”
Mark>> “Steady on Peter I’m about to get to that. It’s good because the failing tests often point out where changes have occurred in the application and I focus testing on these areas. More often than not the changes that are needed to get the web test to pass again are minimal though and I can complete my changes in under an hour. I then re-run the web tests and hey presto they pass. Obviously if an area has been re-written there’s less ‘Hey presto’ and more ‘Gusto’, in that it takes longer for me to fix up those tests and I may even have to re-record parts of the web test scenario.”
Peter>> “Right so, looks like the maintenance effort required is minimal so, but as you said it depends what changes in the application being tested.”
Mark>> “You got it in one.”
Mark>> “Here’s something that might interest you. I actually came across a situation recently where I could not use this product for load testing. We’ll I was asked by a Project Manager to see if we could use Microsoft Visual Studio Team System (VSTS) 2008 Test Edition to load test an oracle forms application that runs in a java applet but we couldn’t use it here because VSTS Web tests are designed to test HTTP web servers rather than the client side code that runs in the browser.”
Peter>> “Oh right, so what did you do?”
Peter>> “Sounds interesting, perhaps we can do a blog article on these tools another day.”
Mark>> “Sure, I’m just trialling these products at the moment but expect to know more about these soon.”
Peter>> “So back to MSVS 2008 Team System Test edition, have you learned anything else recently?”
Mark>> “Sure have, I have been browsing the MSDN forum quite a bit and its very helpful, I have learned quite a bit by using it. The forum offers help and advice on web tests and load tests.”
Peter>> “That’s great Mark, thanks for all that.”
Ciara>> “Hi Guys, I just over heard ye talking about Microsoft Visual Studio Team System, recently I was having an issue whereby I needed to capture a session id from a request and pass it into all subsequent requests. One of the requests was a dependent request that was only available at run-time. What I had to do was copy the dependent request at run time and then pasted the request into the web test, replaced the session id with the dynamic one and now at run time the web test runs perfectly.”
Peter>> “That’s handy to know.”
Ciara>> “Actually another handy thing that I learned recently was in the replacement of parameters, basically at the beginning I used to manually replace parameters in each of the requests, which to be honest was becoming a bit tedious. If you extracted a variable from one of the requests and this had to be replaced in all subsequent requests then it took a bit of time to do all this. What I am now doing is opening the web tests as an XML file and then doing a find and replace, this blanket find and replace works a lot better and is a lot faster than the way I was working previously.”
Peter>> “That’s a definite time saver and probably less tedious and more accurate.Thanks for that Ciara.”
Peter>> “I’ll just pop over to Eoin and see how he’s getting on with the tool.”
Peter>> “Eoin, how it going, I’m just doing a tour around the test department to discuss how were all getting on with Microsoft Visual Studio Team System, Test Edition.”
Eoin>> “Sure, I’m getting on well with the tool. I was experiencing an issue though whereby I was encountering some ‘404 Not Found’ errors for some of the URLs because when recording, the browser ignores dependent requests (css or gif) that it can not find. The playback engine will then look at the html response and attempt to retrieve each dependent link from the response. To fix this issue we chose to set the parsing of dependent links to false for those requests (css or gif) that had dependent links that failed. This works for me because I don’t care about those errors. Once I ignore them the test passed fine.”
Peter>> “I heard that you had identified some good training material on MSVSTS Test Edition.”
Eoin>> “Sure did, there are some links here that I certainly found helpful, I found information on Web Test Creation , information on adding an XML file as a data source , I also found some helpful video demo’s on how to work with agents and controllers and a range of other videos which would help a user understand Visual Studio.”
Peter>> “Thanks for that, Eoin, Mark and Ciara. I have one thing that I want to share with you, I had a situation whereby the application I was testing closes the Web app main page and gets loaded in a new window that closes the previous window. This was a problem for me because it closed the recorder as well. Here’s how I got around that:-“
When you start recording a test and load the URL of the web site, the login screen closes the window you were recording in and waits for input. Unfortunately, the recording is stopped when the window closes and you can’t go any further. Is there some way to start recording in an existing IE window instead of the new IE window that typically opens up?
Try the following.
1) Create a new web test
2) When IE appears navigate to http://www.microsoft.com or any web site. This link will get recorded, but you can delete it later in the editor.
3) Shift click on a link on this page.
4) A new browser should appear with the page that you click on loaded. In this new browser enter the address for you website.
5) Login and start browsing. This should close the second instance of IE and continue recording because the first instance is still active.
6) When you are done, click the stop button.
7) In the editor remove the links that were recorded in the first 3 steps.
Alternatively access the following post.
Peter>> “Right so, to sum up”, The Visual Studio Team System, Test Edition 2008 software has added value to our test team. The tool has cut down on our regression test times and has enabled us to release more stable, less bug laden software. Overall, It has allowed us testers to be more confident that the software we are releasing is of higher quality. We give the tool the mark of approval.”