The Ten Worst Federal IT Programs, According to the Federal IT Dashboard

Today was the deadline for agencies to conduct their own TechStat review of their IT project portfolio.  And based on the data available from the Federal IT Dashboard, there’s some ‘splaining to be done.

On the whole, things don’t look so bad. While IT Dashboard tracks 805 portfolio programs in the Federal government, only 40  of those programs fall into the Dashboard’s red zone with a rating of 3.o (out of 10) or below.  And those programs amount to 5% of the federal government’s 2011 continuing resolution IT spending.

But that 5% is 2 billion dollars. And when Admiral Mullen is saying he’s reluctant to deploy ships because money is too tight, that’s $2 billion that might be useful someplace else.

The bottom 10 have the distinction of having a rating of less than 2.5 — and seven of them actually have a rating of 0.0.  That’s right–these programs are in such bad shape they don’t even push the needle off the pin at the end of the scale.  And here, in descending order, are the ignoble 10:

Department Portfolio Investment Name Spending in FY 2011 (in millions) Rating
Department of Veterans Affairs Benefits Legacy VETSNET-2012 $26.98 2.300803
Department of Transportation DOTXX127: Delphi Data Management Center (originally part of OSTXX001: Delphi) $7.91 2.26256
Department of Transportation DOTXX129: Delphi Version Two (originally part of OSTXX001: Delphi) $8.34 1.846669
Department of Veterans Affairs Medical 21st Century  Revenue Improvement and Systems Enhancements – 2012 (RISE) $0.00 0
Nuclear Regulatory Commission Time and Labor Modernization (TLM) $0.88 0
Department of Veterans Affairs Medical 21st Century CAPRI-2012 $6.26 0
Department of Veterans Affairs Medical 21st Century Registries-2012 $11.82 0
Department of Homeland Security FEMA – NFIP Information Technology Systems & Services $23.41 0
Department of Veterans Affairs Corporate 21st Century SAM (former FLITE)-2012 $36.18 0
Department of Veterans Affairs Medical 21st Century Development Core-2012 $76.82 0
Department of Transportation FAAXX504: En Route Automation Modernization (ERAM) $220.01 0


One of these programs, the Dept. of Veterans Affairs’ Medical 21st Century  Revenue Improvement and Systems Enhancements – 2012 (RISE), isn’t out of the starting blocks yet–it’s still in pre-procurement phase, but has already been singled out by Vivek Kundra and VA CIO Roger Baker.  The worst of the lot, however, is the FAA’s En Route Automation Modernization (ERAM) program, the next-generation flight routing system that has fallen far behind its deployment schedule.  In the words of the Department of Transportation’s own assessment:

“Due to operational issues ERAM deployment is behind schedule and the ERAM Improvement Plan has been developed. This Plan describes what FAA has completed to date to achieve sustained operations at the two key sites and what approach will be taken to resume the schedule for deployment of the remaining 18 sites. The plan is to achieve Initial Operational Capability Operations (IOC) at 7 sites during FY2011, 6 sites during FY2012 and the remaining 7 sites by the end of FY2013. The investment will go to the Joint Resources Council in June 2011 to rebaseline to extend the current program segment from 2011-2014 and establish the next useful segment.”

Under the continuing resolutions, FAA has spent over $200 million thus far on the program, which is projected to extend out now to until 2020.  Sure, it’s a high-value program, and high value programs carry with them a certain amount of additional allowances for overcoming risks and overruns. But just where ERAM is en route to right now is anybody’s guess.


Electronic Health Records Mean 500 percent Storage Growth for Midsize Hospital

The infrastructure challenges created by the drive to achieve meaningful use of electronic medical records can be substantial, even for a relatively small facility. Take, for example, York Hospital , a 79-bed hospital in southeastern Maine, where moving toward meaningful use has resulted in a more-than-meaningful jump in infrastructure.

Kevin Foster, the network administrator for York Hospital’s patient information systems department, told me recently that the hospital has seen a spike in its in-house data center hardware. In 2009, the hospital standardized on McKesson’s Paragon software as its primary patient information system. “Over the course of the last year, we’ve been in the process of upgrading those systems to the latest versions to accommodate meaningful use requirements,” Foster said. “As far as the infrastructure goes, those applications have required a significant hardware increase.”

Foster explained that the hospital has replaced close to 60 percent of its servers since 2009. “We only have a handful of legacy servers that are over 5 or 6 years old running,” he told me. “That’s been a pretty positive move.” But the number of new servers required has also grown quickly, as new software has been added and the rollout has moved forward. To manage the expanding fleet, Foster said that York Hospital has begun to deploy VMware virtualization to consolidate servers.

Currently, the hospital’s servers are about 15 percent virtualized, but Foster said the goal is to reach 60 to 70 percent virtualization of server workloads. “Obviously, we look at things like extremely intense SQL Server sols, fax servers, and anything else with modems as not going to virtualization,” he said.

Foster also said that there are a number of legacy applications that won’t work in a virtual environment. He hopes that in the next two to three years, those applications will be phased out, as their data is migrated to virtualization-friendly software.

One barrier to pushing virtualization forward is software vendor support. But Foster told me that’s quickly changing. “[We] still have some applications from McKesson that aren’t officially supported on virtual servers,” he said. That means the vendor will only help troubleshoot a problem if it occurs on physical servers. “That’s been an ongoing battle with them,” he said. “But luckily, over the past year, they’ve started to approve virtualization (on more apps) and see the light, as it were. They’re running this in their own labs and seeing it work, so there’s no reason that their customers can’t run the applications on virtual servers.”

When it comes to storage, the impact is even more significant: Foster said that since the implementation of York’s electronic health records system, storage requirements are up nearly 500 percent. “Right now, we’re running about 96 terabytes of tier-one storage, whereas this time two years ago, we were probably at 20 terabytes,” he said. “And we’re just looking at adding more. For the next release of the Paragon software (to meet meaningful use requirements), they’re telling us that the audit log we’re going to have to maintain is upwards of 1terabyte a month of data change, and that we’re going to have to potentially retain for upwards of 7 years. It has us freaked out a little bit right now.”

originally posted by me on Virtual Integrated System Blog.