Let’s just dive in by recapping the operatic development process using software terms:

1. The product manager writes functional specs (the libretto)
2. The architect (composer) writes technical specs (the sheet music) based on the functional ones
3. Management chooses a reasonable release date
4. About a year before the release date, development begins on the infrastructure and wireframes (stage, sets, costumes), congruent to the functional and technical specs.
   • The infrastructure may well be sourced from elsewhere, or built in-house — whatever makes the most financial and artistic sense
5. The main components (principal singers and stage management) are identified
6. With roughly six months to go, the technical lead (conductor) learns the technical specs (sheet music) backwards and forwards and begins work on the main components (the principal singers)
   • The same as how, in software, the architect and the lead developer could be the same person so in opera, the composer and conductor could be the same person: Verdi conducted the first runs of all of his operas, and so have most composers. But after that first run, their operas are conducted just as well by other conductors.
7. The UX designer (director) begins fleshing out the UI (choreography)
8. Development begins on smaller components (auditions for smaller parts, chorus)
9. Support infrastructure is stood up (props, rehearsal rooms, living arrangements)
10. Management meticulously schedules acceptance testing (rehearsals), including bug fixing time.
11. A couple of months before release, major development is done (the choreography, principal parts, sets, and costumes are all largely in order), the infrastructure is in place, and developer testing begins (all cast members rehearse at home)
12. A month before release, developer testing is complete, and development is largely frozen: the infrastructure is ready, the entire cast is on site, and know their parts and music. Developer testing on the main components has been especially rigorous.
13. Acceptance testing begins:
    1. The testing phase is the climax of the process. All hands are on deck, for very full days, for the duration.
    2. It takes place in a lightweight mock environment (the off-stage rehearsal room) with mock props and costumes
    3. The mock environment is designed to be easily reset and changed from one test to another, which is not the case for the production environment (the stage with sets)
    4. The architect, UX designer, developers, and management are all present
    5. Each workflow (scene) is thoroughly tested
    6. The testing harness supports starting from any point in a workflow, stopping at any point, and resetting to any point, so that the entire workflow doesn’t have to be run every time, making it possible to easily test every individual feature
    7. The testing harness supports various levels of integration testing, from interaction of only main components with mocked minor components, to full end-to-end testing
    8. Project management keeps track of each test’s outcome against the specs and design
    9. Project management files and tracks enhancement requests and bug reports (changes and corrections to choreography and singing) made by the developers and designers, who update specs and designs as needed
    10. The tech lead and UX designer make tweaks as needed and file bug reports, which are worked on with high urgency
    11. Once the tests are going acceptably well, testing moves from the mock environment and into a real one: the actual stage
    12. More enhancements and fixes are made until the UX designer and developer are happy with the outcome
    13. Beta users (coworkers, friends, early adopters) are brought in for final testing. There are very few, if any, surprises at this point.
14. The product is released on opening night
15. A point release or two may quickly follow, with minor changes (in the first couple of performances)

That should all be very familiar, but how does that differ from the process in most software shops? From my experience, it’s the beginning and the end: the design and the testing. Most shops focus on the middle: the development itself, with testing being a necessary evil, and design often being a quick drawing with some boxes and arrows, maybe just in someone’s head.

Don’t Focus on the Coding

The above process takes probably about 18 months, and the first half of it is all design. Main feature development takes about six months, and testing takes half of that again. So a full two-thirds of the process is design and testing — not development. On top of that, at the end of that six months of development, everything is already well-tested by the developers, with unit tests and rudimentary integration and functional tests. Which means that even less of that six months is actually spent on developing features; maybe as much as half that time. Counting that developer testing, we may be looking at nine months of design, three months of development, and six months of testing.

Design

At this point, you may be aghast at the idea of spending less than 20% of a release cycle coding, and 50% of it designing. The important thing to remember is that designing is programming. The difference is that it’s at a higher level, and it consumes almost all of the research time. It’s a more efficient, and more disciplined, approach than starting to code with only a vague plan, and changing direction as problems arise, pausing to learn new technologies, finding the right widget library, and then realizing that the protocol you started with doesn’t support one of your main features as you wrote it, which means going back to the drawing board.

That going back and re-coding, that’s what wastes a lot of time. That’s why the opera has so much design up-front and why a release date isn’t even chosen until the detailed design is complete, and thoroughly understood. And the opera is not alone, because every disciplined engineering profession does the same. I’ve never worked in civil engineering, but I’m pretty sure that every iron nail in a highway overpass is designed and placed before one worker arrives on site. A big design up front — not coming down a waterfall, but rather in a process that is flexible and agile — documented in great functional specs, is the hallmark of the timely and efficient building of anything.

When done well and thoroughly, it front-loads the vast majority of the variability. If you spend the first half of the cycle coming up with a great design, the last half should be an exercise in typing. The great screenwriter Aaron Sorkin said “You have to think about what you’re going to write before you write it”. Software works exactly the same way. The more specific and detailed your thinking, and the design document that comes out of that, the less surprises there will be in the development and testing phases.

This means the design should address gotchas head on: every technology used should be well-understood, it should be vetted to make sure it works like the architect thinks, and the design document should be as detailed as possible, with APIs, flowcharts, state machines and anything else that can reduce risk and variability in the second half of the cycle. Plus, that document should be very readable, and even entertaining, so that the variety of audiences that read it (devs, managers, QA, tech writers, etc) don’t fall asleep doing so.

If I’ve taken one thing away from my opera experience, is that it’s the important of a thorough design up front, in which literally every note is written before a singer is hired.

Testing

If there are two things I can take away from the opera, the second is definitely the outsized scope of testing. At the opera, testing is not only built into every part of the process — from design to development to release — but everyone is also laser-focused on it. Conductors play their music on the piano to make sure it doesn’t just sound good in their head. Singers practice for hours a day with coaches, who you can think of as the developers of singing. They essentially run unit tests all day long, making changes until the output (the voice) matches the spec that is the sheet music.

And once rehearsals start, everything falls in place, for the most part, with ease. There are always changes because a singer turned out to be taller than the director imagined, so he has to be placed further downstage; and three people in the chorus constantly put the emphasis on the wrong syllable of some Italian word they’ve never heard of, so the maestro has to explain Italian pronunciation yet again.

Enhancements and bugs are built into the acceptance testing phase, but they’re assumed to be minor. Easily changed and corrected, with relatively little effort from the developer. You don’t find out that the soprano can’t hit the high note at this stage, because she auditioned long before. You don’t find out that the set design is all wrong, because an intricate miniature replica was built first. If the main singer gets the flu, there’s a cover waiting to step in, having shadowed the singer at every rehearsal. Everything has been vetted, pre-tested, and every contingency addressed so that there are no show stoppers. Even without any changes at this phase, the opera would be good. The changes here are the little details that make it great: a flawless, amazing performance.

To get there, the testing has to happen in design and during development first. The acceptance testing has to involve all the roles in the production. And it has to be properly planned, with ample time scheduled for it, so that its success is the number one priority for everyone during that time.

Conclusion

By now, I’ve been to dozens of operas, many on opening nights, many in the audience and many backstage or on-stage, and I can count on one finger the number of major blunders I’ve experienced. It was about the fifth performance of a production of Turandot, when the king repeated the same line twice. People that weren’t paying attention to the Italian wouldn’t have even noticed, but someone else played the part the rest of the run.

I’ve seen many more small issues, like people dropping prop coins or papers, walking out the wrong exit, or forgetting to take a jacket off. But the lack of issues in general is astounding, to say nothing of the ability to release on time to the day, and on budget, year after year after year. All of that comes solely from the company’s ability to front-load risk, which then lets them plan the release with certainty, and devote ample time to testing and refactoring, so that the final product is an absolute gem.