Identifying the Problems

When I joined CCM, the marketing and software development departments both reported to the President and were constantly at odds.  Both departments had equal "ownership" of the products, but they had differing motives and interests.

Our salespeople indicated that some potential clients had heard of the benefits accruing to "relational databases".  It was agreed that the next major release would use a relational database - this pleased marketers because now they could use a new buzzword and it pleased the developers because they could learn a new technology.  Unfortunately, no one was looking out for the clients...

I was hired after 18 man-months of effort had been spent porting the products to a relational database.  The yearly release promised to CCM clients under their maintenance contracts was already late.   I was hired to help finish this effort.

I decided to learn the relational database by verifying that adequate performance was possible (the reports they had produced so far seemed slow but they assumed they could optimize later).  I produced a version of an existing CCM product that skipped everything but  the data retrieval.  I produced a similar version using the relational database.  I proved that the relational database was three times slower than the previous approach - which meant that the new product, even optimized, could not produce any more than 1,500 records in the time that the old product processed 2,000.  Sales/Marketing confirmed that speed was much more important than the relational label.  In a collaborative meeting, they made it clear that improving performance and adding analytic capabilities were key to increasing customer satisfaction and making sales.

I was asked whether I could produce new, substantially improved versions of their products within six months.  I agreed subject to a variety of conditions - it was obvious to me that I needed to be able to solve several key problems:

1. The existing code was virtually impossible to maintain.  Even modules that were used in several products had different versions for each product.  I needed code and procedures that I could leverage.
   
2. The organizational structure was counter-productive.  Decision-makers lacked information, programmers lacked discipline and a business orientation.
   
3. Morale was low.  People were finger-pointing and making excuses.

Solutions: Re-Organization

Re-organization was key to our efforts.  I requested and received sole responsibility for the design, support and marketing (but not sales) of our products.  I set up two teams - a Development Group and a Marketing Group.  The Marketing Group performed functions typically labeled as product management - helping with product design, testing, developing manuals and marketing materials and acting as a liaison between developers and salespeople and clients.  

This helped address the morale issue - I made it clear that infighting was not productive nor did I care who was responsible for past problems.  Each team member was responsible to make sure that quality products were delivered on time and was expected to help out whenever bottlenecks in one area threatened our success.

I also emphasized that our job was to produce products that could be sold - all decisions would be made based on business factors. 

Solutions: Code Re-Write

The most controversial decision (and the most important) was my decision to scrap all of the existing code.  I doubt a single person at CCM believed I could pull it off until we got one month into the work.  I refused to manage the project unless allowed to make this choice and they gave in.  It must have taken a great deal of courage - if I failed, they would be further behind with serious issues ahead.

My decision was not that hard.  CCM's six products were really two similar products that each came in three flavors - a great deal of the code should have been common code and maintaining the code separately was silly.  Maintenance and testing of the existing code would have been a nightmare. Furthermore, we would have made no progress in terms of human or software capital.

I had six months to re-write six products - the same amount of time that had been spent doing a limited port of one of the products.  The key factors to success were:

1. Waste no time: I could not have programmers standing by idly while a new architecture was built.
   
2. Leverage, leverage, leverage: I had to dramatically reduce the total amount of code needed.
   
3. Use resources efficiently: avoid critical path problems, identify bottlenecks early.
   
4. Win people over: to succeed, people had to start believing early.

The best thing about the existing code was the database code - it was a fast, clever approach.  Unfortunately, it was scattered throughout the entire product line.  The best programmer under me was assigned to build a database engine that could be used independently - useable by any of our report-writing modules to retrieve data.  This made him productive early and helped get him comfortable with what we were trying to do.  At the same time, I added an options file that allowed us to customize the product without having to release different versions of the code.  This helped win over the Sales people - who liked being able to customize their demonstrations on the fly.

Ninety percent of the remaining modules were a variety of reports.  Each report allowed users to select one or more "row variable" - aggregation levels ( Sex, AgeGroup, DiseaseCategory, ...) and generated a report.  Trivial stuff really, but each report routine had slowly gotten unreadable through the addition of a variety of special cases.  I solved this by hiring a person to do nothing but quality assurance and getting together with the product manager and a senior programmer to develop a standard report template.  We identified a basic structure that all reports followed.  We then started reviewing the special cases - attempting to classify them.  We defined new terminology that broke all special cases into two or three categories.  We agreed on a report specification form that the Marketing Group was responsible for developing for each report.  I developed a code template that made it easy to translate a specification form into code.  The QA person was responsible for comparing the final code with the specification and then for testing it.  This created a process that we could time - we soon knew that we could produce x reports in y weeks.  That gave the credibility that we needed - it was obvious we could finish in time.  It also helped build a sense of teamwork - when the inevitable errors happened, it was obvious and understandable what went wrong and the errors were easily fixed.

It also ensured that virtually everyone in my  group was fully employed.  The Marketing Group produced specifications that went to programmers who produced code that went back to the Marketing Group for sign-off.  This freed me up to work on the non-standard elements.  For example, each product had a "Custom Table" that allowed the user to specify calculations.  The crude parser used before could not be used - I didn't even know which of the six copies was closest to being reliable.  I worked on the portions of the product that required significant re-design and on enhancements while my  team cranked out the standard modules.