Synthstuff - music, photography and more...

Obamacare - the surge to fix the website

Great long article at The New York Times:

Inside the Race to Rescue a Health Care Site, and Obama
As a small coterie of grim-faced advisers shuffled into the Oval Office on the evening of Oct. 15, President Obama�s chief domestic accomplishment was falling apart 24 miles away, at a bustling high-tech data center in suburban Virginia.

HealthCare.gov, the $630 million online insurance marketplace, was a disaster after it went live on Oct. 1, with a roster of engineering repairs that would eventually swell to more than 600 items. The private contractors who built it were pointing fingers at one another. And inside the White House, after initially saying too much traffic was to blame, Mr. Obama�s closest confidants had few good answers.

The political dangers were clear to everyone in the room: Vice President Joseph R. Biden Jr.; Kathleen Sebelius, the health secretary; Marilyn Tavenner, the Medicare chief; Denis McDonough, the chief of staff; Todd Park, the chief technology officer; and others. For 90 excruciating minutes, a furious and frustrated president peppered his team with questions, drilling into the arcane minutiae of web design as he struggled to understand the scope of a crisis that suddenly threatened his presidency.

�We created this problem we didn�t need to create,� Mr. Obama said, according to one adviser who, like several interviewed, insisted on anonymity to share details of the private session. �And it�s of our own doing, and it�s our most important initiative.�

Got that right Barry -- you could have hired some professionals -- SAP, Google or Amazon -- at the outset and could have come in for about a million and the website would have been ready in about a year. More:

The website had barely been tested before it went live, so a large number of software and hardware defects had not been uncovered. Fixing the account creation software simply exposed other problems; people still could not register to buy insurance. A system intended to handle 50,000 simultaneous users was fundamentally unstable, unable to handle even a tiny fraction of that. As few as 500 users crippled it, according to people involved.

Bad move -- they could have hired out the testing. The last lab I worked for at MSFT did just this -- we could simulate huge loads against a server. We had over 2,500 client machines, each of which could simulate several hundred random users. Had a lot of fun figuring out how to 'push' a disk drive image out to any number of machines. Got pretty good at it. Just for fun I dug up a photo -- here is one row of 420 machines. There were five more rows. Back to the story -- the number of people working to fix the website is amazing:

But while the contractors were grateful to Mr. Zients for helping to create order, they saw the administration�s �tech surge� � announced by Mr. Obama in the Rose Garden a few days before QSSI took over � as mostly an exercise in public relations.

The announcement conjured images of an army of software engineers descending on the project. In fact, the surge centered on about a half-dozen people who had taken leave from various technology companies to join the effort. They included Michael Dickerson, a site reliability engineer at Google who had also worked on Mr. Obama�s campaign and now draws praise from contractors as someone who is �actually making a difference,� one said.

Even so, one person working on the project said, �Surge was probably an overstatement.�

Six people? A good team would need about fifteen. Manager, manager's admin, a person to keep track of the design specs, integrate change orders and distribute work to the individual programmers, two people to do the compiling and distribution of the new code and maintain the version control system and ten programmers. Have daily short meetings of top-level people (client included if they wish) and weekly thrash-it-out meetings with all hands (but no clients ever). You would have a large website up and running in no time. It is a long article (seven pages) but worth reading just for an insight in how not to build an enterprise-level website. And this great quote:

�There�s so much wrong, you just don�t know what�s broken until you get a lot more of it fixed,� Mark Bertolini, the chief executive of Aetna

Lulu and my insurance premiums went up $100 each and our co-pay is now $40. The sooner this thing gets repealed and free-market capitalism is allowed to work, the better for everyone. Posted by DaveH at December 2, 2013 8:49 PM