The Ten Worst Federal IT Programs, According to the Federal IT Dashboard

Today was the deadline for agencies to conduct their own TechStat review of their IT project portfolio.  And based on the data available from the Federal IT Dashboard, there’s some ‘splaining to be done.

On the whole, things don’t look so bad. While IT Dashboard tracks 805 portfolio programs in the Federal government, only 40  of those programs fall into the Dashboard’s red zone with a rating of 3.o (out of 10) or below.  And those programs amount to 5% of the federal government’s 2011 continuing resolution IT spending.

But that 5% is 2 billion dollars. And when Admiral Mullen is saying he’s reluctant to deploy ships because money is too tight, that’s $2 billion that might be useful someplace else.

The bottom 10 have the distinction of having a rating of less than 2.5 — and seven of them actually have a rating of 0.0.  That’s right–these programs are in such bad shape they don’t even push the needle off the pin at the end of the scale.  And here, in descending order, are the ignoble 10:

Department Portfolio Investment Name Spending in FY 2011 (in millions) Rating
Department of Veterans Affairs Benefits Legacy VETSNET-2012 $26.98 2.300803
Department of Transportation DOTXX127: Delphi Data Management Center (originally part of OSTXX001: Delphi) $7.91 2.26256
Department of Transportation DOTXX129: Delphi Version Two (originally part of OSTXX001: Delphi) $8.34 1.846669
Department of Veterans Affairs Medical 21st Century  Revenue Improvement and Systems Enhancements – 2012 (RISE) $0.00 0
Nuclear Regulatory Commission Time and Labor Modernization (TLM) $0.88 0
Department of Veterans Affairs Medical 21st Century CAPRI-2012 $6.26 0
Department of Veterans Affairs Medical 21st Century Registries-2012 $11.82 0
Department of Homeland Security FEMA – NFIP Information Technology Systems & Services $23.41 0
Department of Veterans Affairs Corporate 21st Century SAM (former FLITE)-2012 $36.18 0
Department of Veterans Affairs Medical 21st Century Development Core-2012 $76.82 0
Department of Transportation FAAXX504: En Route Automation Modernization (ERAM) $220.01 0


One of these programs, the Dept. of Veterans Affairs’ Medical 21st Century  Revenue Improvement and Systems Enhancements – 2012 (RISE), isn’t out of the starting blocks yet–it’s still in pre-procurement phase, but has already been singled out by Vivek Kundra and VA CIO Roger Baker.  The worst of the lot, however, is the FAA’s En Route Automation Modernization (ERAM) program, the next-generation flight routing system that has fallen far behind its deployment schedule.  In the words of the Department of Transportation’s own assessment:

“Due to operational issues ERAM deployment is behind schedule and the ERAM Improvement Plan has been developed. This Plan describes what FAA has completed to date to achieve sustained operations at the two key sites and what approach will be taken to resume the schedule for deployment of the remaining 18 sites. The plan is to achieve Initial Operational Capability Operations (IOC) at 7 sites during FY2011, 6 sites during FY2012 and the remaining 7 sites by the end of FY2013. The investment will go to the Joint Resources Council in June 2011 to rebaseline to extend the current program segment from 2011-2014 and establish the next useful segment.”

Under the continuing resolutions, FAA has spent over $200 million thus far on the program, which is projected to extend out now to until 2020.  Sure, it’s a high-value program, and high value programs carry with them a certain amount of additional allowances for overcoming risks and overruns. But just where ERAM is en route to right now is anybody’s guess.


Electronic Health Records Mean 500 percent Storage Growth for Midsize Hospital

The infrastructure challenges created by the drive to achieve meaningful use of electronic medical records can be substantial, even for a relatively small facility. Take, for example, York Hospital , a 79-bed hospital in southeastern Maine, where moving toward meaningful use has resulted in a more-than-meaningful jump in infrastructure.

Kevin Foster, the network administrator for York Hospital’s patient information systems department, told me recently that the hospital has seen a spike in its in-house data center hardware. In 2009, the hospital standardized on McKesson’s Paragon software as its primary patient information system. “Over the course of the last year, we’ve been in the process of upgrading those systems to the latest versions to accommodate meaningful use requirements,” Foster said. “As far as the infrastructure goes, those applications have required a significant hardware increase.”

Foster explained that the hospital has replaced close to 60 percent of its servers since 2009. “We only have a handful of legacy servers that are over 5 or 6 years old running,” he told me. “That’s been a pretty positive move.” But the number of new servers required has also grown quickly, as new software has been added and the rollout has moved forward. To manage the expanding fleet, Foster said that York Hospital has begun to deploy VMware virtualization to consolidate servers.

Currently, the hospital’s servers are about 15 percent virtualized, but Foster said the goal is to reach 60 to 70 percent virtualization of server workloads. “Obviously, we look at things like extremely intense SQL Server sols, fax servers, and anything else with modems as not going to virtualization,” he said.

Foster also said that there are a number of legacy applications that won’t work in a virtual environment. He hopes that in the next two to three years, those applications will be phased out, as their data is migrated to virtualization-friendly software.

One barrier to pushing virtualization forward is software vendor support. But Foster told me that’s quickly changing. “[We] still have some applications from McKesson that aren’t officially supported on virtual servers,” he said. That means the vendor will only help troubleshoot a problem if it occurs on physical servers. “That’s been an ongoing battle with them,” he said. “But luckily, over the past year, they’ve started to approve virtualization (on more apps) and see the light, as it were. They’re running this in their own labs and seeing it work, so there’s no reason that their customers can’t run the applications on virtual servers.”

When it comes to storage, the impact is even more significant: Foster said that since the implementation of York’s electronic health records system, storage requirements are up nearly 500 percent. “Right now, we’re running about 96 terabytes of tier-one storage, whereas this time two years ago, we were probably at 20 terabytes,” he said. “And we’re just looking at adding more. For the next release of the Paragon software (to meet meaningful use requirements), they’re telling us that the audit log we’re going to have to maintain is upwards of 1terabyte a month of data change, and that we’re going to have to potentially retain for upwards of 7 years. It has us freaked out a little bit right now.”

originally posted by me on Virtual Integrated System Blog.


Back in the saddle again

I wasn’t sure that I’d ever willfully return to a full-time journalism gig after I parted ways with 1105 Government Information Group.  I had been lured away from what at the time had been a burgeoning freelance career to run Defense Systems magazine, only to be cast out as part of a staff purge at 1105 13 months later, and I was pretty sure that the right editorial job wouldn’t ever come along.  So, I went back to the precarious life, and aside from some rather insane hours and juggling a host of projects (some, that in retrospect, were quixotic at best), things have been okay.

Then I got a call out of the blue: was I interested in running a Federal government technology newsletter?

So, after a few weeks of discussion, I’ve made the decision to return to a full-time job covering the Federal market. But this time, it’s different.  Mostly, that’s because the company that came calling is Mark Bisnow’s outfit, his eponymous almost-always-B2B newsletter company. Also, I’ll be putting out a daily, hopefully with the assistance of a reporter to be named later.

The approach of Bisnow’s FedTech newsletter is to cram a lot of information into a small package, leavening it with a look at the people who make up the Fed technology community.  That means reaching out–a lot–to the community.   As a former Navy officer and former government contractor,  and someone who’s covered the government sector on and off for over 20 years now as a journalist, it’s a great deal for me–I get to talk to people I’ve always had a great deal of admiration and respect for (even if it didn’t always seem that way while I was writing The Packet Rat column for GCN).

So, once more into the breach, and all that.  I’m looking forward to a long, wild ride.

cloud computing, Enterprise IT, NASA, sticky, tech

Chris Kemp Quits, as Fed Budget and Inertia Beat Govtrepeneurs Down

Chris Kemp, who had a few short weeks ago been greeted with rockstar fervor at the Cloud/Gov conference in Washington, DC, has stepped down from his role as NASA’s Chief Technology Officer for Information Technology.  Kemp was the champion of NASA’s Nebula program, the agency’s private cloud effort , and helped with the General Services Administration’s launch of the Apps.gov cloud service program. But in the face of budget cuts and continued institutional resistance to his agenda for changing government IT, Kemp submitted his resignation in March.

“Whereas I thought I had the best of both worlds being a Headquarters employee stationed in Silicon Valley,” Kemp said in a blog post announcing his move, “I actually had the worst of both worlds… no influence when I can’t be in all of those meetings at NASA HQ, with no mandate to manage projects at Ames. As budgets kept getting cut and continuing resolutions from Congress continued to make funding unavailable, I saw my vision for the future slowly slip further from my grasp.”

Kemp’s dillema, while certainly higher profile than that of many state and local CIOs and CTOs, is hardly unique.  With revenues at historic lows, and budgets tight, it’s perhaps harder than ever to try to achieve meaningful change in the way agencies run their information technology, even at tech-focused agencies like NASA.  At the federal level, the budget standoff threatens to put major initiatives that could actually save the government more money on hold.

But perhaps more dangerous, the uncertainties around IT budgets and programs at all levels of government can be demoralizing, particularly to the most talented and valuable members of IT organizations who have options elsewhere.  As other employment opportunities emerge, government IT organizations could see an exodus of talent, making it even more difficult to do more with less.


Enterprise IT, State & Local, tech

Why You Should Buy Storage Stocks: Electronic Health Records

I was talking the other day to the network administrator of a 70-bed hospital with a dozen internal physicians’ practices about the trials and tribulations of achieving the first phase of the “meaningful use” standards set by the American Recovery and Reinvestment Act’s embedded Health Information Technology for Economic and Clinical Health (HITECH) Act.  Yes, I sometimes talk to people about such things.  And maybe more people should.

Back when they passed the “bailout” bill (ARRA) last year, Congress embedded the HITECH Act, which basically offers a bounty to hospitals if they can not just put electronic health records in place, but use them in a meaningful way across their organization to improve care, reduce errors, and reduce paperwork.  There’s a big cash reward for meeting Phase I of these standards before the end of 2011–$2 million, plus $200 for every patient discharged past the 1150th patient and up to the 23,000th patient. In other words, high-volume hospitals could see as much as $6.37 million in incentives in the first year.  That amount goes down by a quarter for each succeeding year.  So over 4 years, hospitals that comply with meaningful use could see between $5 million and $10.9 million.

Not enough of an incentive? There’s a stick with that carrot–hospitals that don’t get their IT systems in compliance with the Health Insurance Portability and Accountability Act’s security and privacy standards are exposed to potential government civil suits and penalties. And the Department of Health and Human Services has finally started to get serious about HIPAA–Prince Georges’ County, MD based Cignet can tell you about that.

So, hospitals are paying out a big chunk of their capital budgets this year, if they haven’t in previous years, to upgrade their patient information systems.  And they’re discovering that electronic health record systems are, to put it bluntly, storage pigs.

The network administrator I was talking to said that in the two years they’ve had their EHR system in place, their tier 1 storage requirements — that’s their mission-critical online data storage–has grown by almost 500 percent.   Mind you, this is a relatively small hospital, and that 500 percent increase came from going from 20 terabytes to almost 100 terabytes.

But those storage requirements grow daily. And as they bring their systems into meaningful use compliance,  their audit trail data will amount to about another terabyte of data a month–which they’ll have to retain for seven years or so.  So, add another 84 terabytes of audit trails over 7 years, plus whatever natural growth in records they have from new patients, emergency room visits, and visits to the dozen or so physicians practices they own.

In other words, the big winner from ARRA HITECH is the storage industry.  The storage dillema of that small hospital is being writ small, medium, and large across the country at every clinic, doctor’s office, and hospital.  That means petabytes of new storage sitting in someone’s data center somewhere.


Department of Labor offers grants to help homeless vets, but where’s the Gov 2.0?

From the Department of Labor (tip of the tail to @craignewmark) :

The U.S. Department of Labor’s Veterans’ Employment and Training Service today announced the availability of $600,000 in “Stand Down” grants that will provide an estimated 10,000 homeless veterans with opportunities to reintegrate into society. The grants are being awarded under the department’s Homeless Veterans’ Reintegration Program.

“Stand Down events across the country will reach thousands of homeless veterans — including homeless female veterans — with opportunities to re-enter America’s workforce,” said Secretary of Labor Hilda L. Solis, who is also chair of the Interagency Council on Homelessness. “Through these grants, the federal government will engage states and local communities as active partners in readying those who have served our nation for civilian jobs.”

via VETS News Release: US Department of Labor announces ‘Stand Down’ grants to assist about 10,000 homeless veterans [03/09/2011].

DoL is going to offer ip to $10,000 in funding to each applicant.  One-day events get a maximum of $7,000.

It’s too bad they’re not taking the grant applications online, or creating a way to connect organizations seeking grants to maximize their effectiveness in staging events (and help magnify the value of the grant money spent).


Cyberdefense and Information Assurance

Worst Practices: Learning the Wrong Lessons from WikiLeaks « Unisys Security Blog

This post can be read in its entirety over at the  Unisys Security Blog, where it was contributed as a guest blog.


The dark cloud of the WikiLeaks debacle should have a bright silver lining. The exposure of classified Department of Defense and State Department data by WikiLeaks gives us a teachable moment on information security — not just for government agencies, but for any organization that stores, handles, and processes sensitive information.

The vast amount of classified data — over 75,000 Defense Department incident reports and more than 115,000 classified diplomatic cables — and the damage caused by their exposure reveals common flaws in how organizations typically handle sensitive information. But as with past data breaches, many organizations will learn the wrong lessons. And the actions they take as a result will make their organizations less productive and, perhaps, even less secure.

Read the rest here.