From John.Wiersba at thomsonreuters.com Wed Jan 21 15:59:32 2009 From: John.Wiersba at thomsonreuters.com (John.Wiersba at thomsonreuters.com) Date: Wed, 21 Jan 2009 18:59:32 -0500 Subject: [Test Automation] Mocking real world to test a wrapper Message-ID: Gabor, Thanks for your post in reply to my email. I thought of several other questions based on your proposed solution. Here are a few of them: 1) Your example is very perl-centric. What if we make the script to be tested a shell script in a system comprising a large number of shell scripts. Each one of those shell scripts makes numerous calls to external utilities. It would be very painful to wrap each external utilitity used by a shell script the way adduser is wrapped. 2) In addition to the external utilities, the shell script itself could have many side-effects (changes to the file system, etc). How best to test those without splitting them off into their own separate scripts which could be wrapped? 3) How does one prevent the complexity of the wrappers and the testing framework from overwhelming the complexity of the system being tested? It's no fun finding bugs in the testing code itself. 4) How best to retrofit testing into a system not built from the ground up for testing? -- John Wiersba -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.szabgab.com/pipermail/test-automation/attachments/20090121/fec2b168/attachment.htm From szabgab at gmail.com Wed Jan 21 23:34:21 2009 From: szabgab at gmail.com (Gabor Szabo) Date: Thu, 22 Jan 2009 09:34:21 +0200 Subject: [Test Automation] Mocking real world to test a wrapper In-Reply-To: References: Message-ID: Hi John, 2009/1/22 : > Gabor, > > Thanks for your post in reply to my email. I thought of several other > questions based on your proposed solution. Here are a few of them: That's good material for discussion, I hope others will also give some ideas. > 1) Your example is very perl-centric. What if we make the script to be > tested a shell script in a system comprising a large number of shell > scripts. Each one of those shell scripts makes numerous calls to external > utilities. It would be very painful to wrap each external utilitity used by > a shell script the way adduser is wrapped. The example was given in Perl but in any high-level language that would be more or less the same. Writing clean Unix Shell scripts and testing them is difficult. Probably a lot more difficult than writing and testing Perl or Python or Ruby scripts. On the long run I'd recommend to switch to one of the higher-level languages mentioned above. For a beginning I'd write my tests in one of those languages. For testing those shell scripts probably the cleanest way is to setup a virtual computer e.g. using VirtualBox, deply my scripts there and check their effects there. In this case you don't even need to replace most of the external tools. > 2) In addition to the external utilities, the shell script itself could have > many side-effects (changes to the file system, etc). How best to test those > without splitting them off into their own separate scripts which could be > wrapped? A Virtual Computer can help in this too. > > 3) How does one prevent the complexity of the wrappers and the testing > framework from overwhelming the complexity of the system being tested? It's > no fun finding bugs in the testing code itself. I don't really have an answer to this one. > > 4) How best to retrofit testing into a system not built from the ground up > for testing? Well this is maybe the hardest :-) Both technically and how to "sell" it. Technically it is difficult as you need to implement a lot of the parts of the test before it can be used without side-effects - all those questions you have are mostly due to this. When the tests are built during the development of those wrapper scripts both the wrapper scripts and the tests are more simple. "Sales" - wise it is difficult as many times the real benefit of automated tests come when you run them as a regression test. In order to reach the point where the tests have good enough coverage to have a real chance to catch a bug you'll have to spend a lot of time writing those tests. Start by educating everyone around you that this is a long process and they will see the revenues only after a lot of work. Having said that you still need to find an area where you might be able to get value out of the system soon. You have to ask yourself why do you want to write tests ? Probably either because there are bugs in the system you'd like to fix or because you'd like to add new features. You might have other reasons but let me address these two. In both of these cases you are about to change the code. So try focusing on the part that is about to change. If you have a bug try writing a test that reveals the bug. Other than that, find an area that is the easiest to separate from the rest. Make sure your testing does not have side effects in the real world. Write some tests. Get your feet wet. And most importantly, enjoy it ! regards Gabor -- Gabor Szabo http://szabgab.com/blog.html Test Automation Tips http://szabgab.com/test_automation_tips.html