W3C home > Mailing lists > Public > www-qa@w3.org > July 2002

Summary of findings - Outsourcing QA and Testing

From: Vijay Sikka <vsikka@nirixa.com>
Date: Sun, 21 Jul 2002 11:59:57 -0700
Message-ID: <3D3B04AD.7000504@nirixa.com>
To: www-qa@w3.org

Dear Friends,
In April, I had sent out a request for summary from friends and
colleagues on this list for Outsourcing QA and Testing.

The response was overwhelming and I really want to thank all
of you for sending back your thoughts and comments.

I have summarized responses below along with the original questions
we had asked.


Vijay Sikka
Nirixa, Inc.
303 Almaden Boulevard 6th Floor
San Jose, CA - 95110

a. Has outsourcing QA and Testing been a productive experience?

Most of the colleagues believed that pure outsourcing models didn't work 
as intended. They had no proximity with the QA team and closed loop 
interaction. They felt that outsourced QA work resulted in the following 
common feedback:
- Outsourcer didn't understand our product
- Outsourcer didn't find a lot of bugs they should have found
- Outsourcer didn't understand what they were supposed to do
- We didn't get good reporting
Several of them had more pleasant experiences if they had internal QA 
managers interacting and controlling the outsourcing team.

b. Did you save costs while maintaining the timeliness of your product 

Responses on this ranged from some very successful and well-managed 
projects and product deliverables resulting in cost savings to unhappy 
campers who shipped products that didn't work and spent more than 
planned. Another key aspect that emerged was that cost should never be 
the only consideration when outsourcing QA. Cost of not doing QA or 
doing poor QA would be increased exponentially if a product were 
released with bugs that prevent users from using it.

c. Which companies you found did a good job for you?

Most of those surveyed preferred to keep the names of the companies they 
worked with anonymous. All of the colleagues believed that asking the 
companies for current customer references and checking them for 
satisfaction worked well. Several outsourced QA and test companies 
maintain a satisfaction percentage metric of their customers through 
established interviews and questionnaires. It is a good idea to ask the 
outsourcing provider for a report on this metric.

d. Which country did you outsource to? Ireland, Canada, Israel, India, 
China or other?

Most respondents had worked with Indian outsourcing vendors. Most cited 
proficiency in English and quality of work from Indian companies as 
better than other outsourcing countries. Some cited experience working 
with Indian companies that maintained strong management presence in 
America as well as QA labs in India as better suited to their working 
style. A few didn't enjoy working with outsourcers from any other country.

e. Was the time difference a hurdle for team communication?

All agreed that the time difference in team communication could become a 
hurdle in productivity if the outsourcing company with a good 
methodology and process didn't address it. It was observed also that the 
addressing time difference issue with good process led to better 
handling of tighter build/regression schedules as release dates came 
closer. Most of the colleagues observed the following cycle to be very 
effective in proactively handling the time difference to an advantage.
1. Developers deposited latest build and release in the source control 
at night before leaving for the day.
2. The QA and test experts in the outsourcing company would run the test 
cases and generate reports on the release and upload the bug tracking 
system for the developers
3. Developers would review the bugs and test result reports and work on 
fixing the bugs.

This cycle reportedly worked very effectively with outsourcing QA and 
test companies in India that were 12.5 hours ahead in time.

Some outsourcing companies reportedly worked in multiple shifts to 
accommodate overlaps with the developers in USA for conferencing and one 
on one communication.

f. Was language an issue in communication?

People reported issues with accented English from outsourcing companies 
and the fact that Americans are not trained to talk at ESL (English as 
Second Language) speeds. Both of these issues were addressed by 
companies that used outsourcing through Internet Relay Chat and 
teleconferencing that involved people from the outsourcing companies who 
were in United States. It helped to work with outsourcing companies, 
which maintained offices and key executives and management in United States.

g. What type of testing did you successfully outsource? Blackbox? Unit? 
Functional? Load? Regression?

Most of the companies that have used outsourcing reported trying 
functional, unit, white-box and black box testing.

h. Did you completely replace your QA department or did you use 
outsourcing as an extension?

All of the companies that have worked with outsourcing QA and test 
providers reported working with them as extensions of their existing QA 
departments. That was the model that worked best with them. Some 
companies who once had QA departments reportedly replaced them with an 
outsourced company. Often the outsourcing company in the beginning acted 
as overflow for QA departments who were overloaded. Sometimes 
outsourcing QA and test experts were brought in where the customers 
never had QA and used the outsourcing company to help them transition 
into having their own QA team.

i. Did you require the QA and testing people to have computer science or 
software background?

Most companies and colleagues we talked to wanted QA and testing people 
with computer science and software background. However, they sometimes 
lamented that QA outsourcing companies gave them people with limited or 
no HTML, JavaScript, DHTML, C++/Java backgrounds. Companies also warned 
against accepting at the face value claims by the outsourcing company 
that they had "Unix experts" or "Windows experts". Most recommended 
finding out if the QA and test people had certifications by Sun or 
Microsoft or some other US based training and certification organization.

An important factor in each successful outsourcing partnership was 
having a QA manager who is already familiar with the product and 
company-testing practices should be leading the offshore team -- onsite 
at the remote location.

j. Which standards bodies did you find most relevant to your QA and 
testing? ISO? ASQC?

All companies and colleagues reported that standards bodies had no 
relevancy. They didn't recommend ISO 9000 or any standards 
certifications of the outsourcing companies.
Received on Sunday, 21 July 2002 14:54:48 UTC

This archive was generated by hypermail 2.3.1 : Tuesday, 6 January 2015 20:40:29 UTC