“While you were racing…”

9/2011

by Lee Gruenfeld

The greatest compliment we get is when someone

tells us they didn’t even know we existed.

                                                              -- John Bertsch, Director

                                                                 Ironman Race Operations Center

Up until a few weeks ago, I’d always assumed that the Ironman World Championship was run by elves. I could see the athletes and the volunteers but how did a fresh load of ice get to the aid station at Mile 15? How did the trash that was building up at the turnaround disappear? How did sag wagons get to the bike crash near the airport so fast and how did two new Port-a-Potties suddenly materialize on the Queen K near Waikoloa Village Road?

The only logical explanation was elves, and for the last twenty years that was good enough for me.

As it turns out, the real explanation is quite a bit more interesting than that. Have you ever seen footage of flight operations on an aircraft carrier? Aside from the awe-inspiring spectacle of dozens of noisy jets being violently flung off a floating airstrip, there’s the more subtly fascinating sight of 5,000 people acting in intricate concert to make it possible.

What you might not know is that those carrier procedures are modeled after Ironman race operations. (If you’re wondering how it’s possible that a naval operation that has been going on for seventy years could be modeled after one that has only been in existence for four, you don’t read enough science fiction.) The Race Operations Center, or ROC, is one of those things you hope you never need but are mighty glad are there if you do. In order to explain it, a little history is in order.

The original “race operations center” consisted of event director Diana Bertsch, a cell phone and a trailer full of radios. The people handling registration, aid stations, medical assistance, traffic control and a host of other functions knew their jobs and knew them well, but any unforeseen circumstance requiring coordination among two or more of them inevitably made its way to Diana’s earpiece, through her brain and back out as a series of directives, but only to a small number of recipients. Others who might have been indirectly affected learned of the situation by happenstance, or not at all. The system worked, and worked well, but its limitations nagged at the event director.

“I’d always thought there was a better way,” Diana says, “but since things were working so well, there wasn’t a lot of incentive to change. So while we made improvements every year, we didn’t undertake a major overhaul.”

That all changed on the morning of October 15, 2006. “The earthquake was a nasty wake-up call,” Diana remembers. “I wondered what would have happened had it occurred on race day. How would we have marshaled and directed and communicated efficiently with all the resources that would have been required to deal with that situation?”

She went on to explain that they’ve always had critical incident plans, such as procedures to quickly move people to higher ground in the event of a tsunami. “But what we were missing was tight coordination with municipal authorities.”

Fortunately, the answer was close at hand. Very close, as it happened. John Bertsch, ex-commissioner of both the fire and police departments on the Big Island, was a credentialed expert in public safety and was also intimately familiar with the inner workings of the World Championship.

“Diana gave us a clear mandate,” he said of the team he eventually put together. “We had to come up with a single, centralized nerve center capable of coordinating and communicating with every facet of race-day operations, and do it in a way that made it easier for people to do their jobs rather than layer on one more thing for them to worry about.”

John knew that all the elements were already in place but also knew they were factionalized. “We had the medical team, aid station coordination, bike tech support, police, fire…but there was no good way of hooking people together and sharing information.”

The first thing he did was involve every external agency that could play a part in the execution of the race, whether it was during the normal course of events or only during an emergency. This included not only police and fire officials, but the TSA, FBI, the Departments of Defense and Transportation, hotel security, SWAT, the Director of Civil Defense. “We even deal with the US Navy and the Coast Guard,” John said, “although the Coast Guard’s role pretty much ends when the last swimmer is out of the water.”

Focus groups were conducted to gather ideas on developing a plan for how best to serve the athletes, volunteers and the community. As the plan started coming together, John brought in Stacey Aguiar. “We’re going to need a room,” he told her, one big enough to accommodate everybody who could conceivably be drawn in to deal with any of the myriad scenarios they were considering. “Why don’t you figure out how to do that?”

What Stacey came up with was a configuration that provided a physical seat for each of the twelve race directors, every representatives of the municipal agencies, three admins and three “dispatch” teams covering the main operational functions. That comes to over three dozen seats.

“The dispatch teams are the primary users of the ROC during normal operations,” she says, “normal” meaning non-emergency. “Even when everything is going perfectly, it’s still pretty frenetic, but it never gets chaotic. Those three teams and the procedures they use are what keep it that way.”

The three teams are medical, athlete (primarily bike) support and race operations. Each team consists of a dispatcher, who does the communicating, and a data logger, known as a “CAD operator,” who is responsible for entering pertinent information into a computer system. The medical team also includes an ER physician.

While John is the director of the ROC, managing the room is Stacey’s responsibility; she determines schedules and shifts based on anticipated workloads and is also responsible for training all of the participants in procedures and the use of the ROC's technology. While not all of the seats are occupied at one time —for one thing, most of the race directors are out on the course throughout the day— John and Diana have the ability to page any and all to the center should the need arise, as it might during a major emergency. (More about that later.)

At the heart of the operation is a computer system designed by Pat Riley, whose regular job is doing much the same for the Gig Harbor Fire Department in Washington state. A quintessential computer and electronics geek, Pat’s first love is studying systems. “Complex interactions that form a smooth, well-integrated whole have always fascinated me,” he says, “and creating one of those is an immensely gratifying undertaking.”

Several years ago when the ROC was still being laid out, one of the chief’s from the Gig Harbor FD volunteered in the medical tent in Kona. John Bertsch walked by just as the chief opened his laptop during a break to check in on the situation back home. “I glanced at the screen and asked him what it was,” John recalls. “It’s everything, the guy answers, and he points to fire trucks, hydrants, incident reports, all overlaid onto a map. And the fire trucks were moving!”

It took John about two seconds to see the situational awareness possibilities for Ironman. “I got in touch with Pat and said, I gotta have one of those.”

Pat: “At first I thought, sure, an easy adaption.” But it wasn’t that simple. For one thing, the event didn’t own its own computers. And since all the vehicles were short-term rentals, nothing could be permanently mounted in them. “On top of that,” he says, “the whole thing is over seventeen hours after it starts. You can’t gradually phase it in; the entire enchilada has to be in place from Minute One and there are no second chances.”

Intrigued with the possibility of creating a totally new system from scratch, Pat threw away the fire department template and started thinking about a fresh approach. “The first thing I decided is that we had to build this system in the cloud instead of housing it on a local platform. That meant it had to be entirely Web-based, accessible from anywhere on anything that could host a browser.”

And since there were going to be over five hundred separate touch points, using a computer for each one was financially and logistically impractical. “Ten years ago we would have been dead in the water on that one, but since the advent of the smart phone, there were all kinds of possibilities.” He settled on Nextel phones as the primary device, then set about creating an app that could be downloaded into each of them. “What we essentially did was turn these five hundred rented phones into five hundred handheld aircraft transponders. Each unit has its own unique code that identifies it in the central system.”

The next step was to create a means of graphical presentation. A tidal wave of data is useless unless there’s a way to filter and display it in a manner that doesn’t overwhelm the user. “Anyone who’s familiar with the ‘Big Board’ from movies like Dr. Strangelove and Fail Safe will recognize our display,” Pat says, adding  proudly, “except ours is light years ahead of those clunky, black-and-white visuals.”

Color-coded icons dot a satellite map of the Kona course. “Depending on who needs to know what at any point in time,” Pat says, “we can show aid stations, highway markers, medical vans, sag wagons, you name it. And when they move, the icons move.” In addition, right-clicking on any icon calls up incident data entered by the CAD operators.

 

The ROC Board graphic display uses color-coded icons to provide a real-time snapshot of what’s happening out on the race course.

 

The ROC was rolled out in beta in 2008 and became fully operational in 2010. By that time it had gotten so sophisticated that not everyone was fully aware of all its capabilities.

“We were looking at the Board last year,” John recalls, “and all of a sudden one of the medical van icons turned black. I thought maybe the system had a glitch and turned to ask Pat about it. He said no, he’d just forgotten to tell me about something new he’d added.”

“There were occasional reports about drivers going too fast on the way to incidents,” Pat continues. “So I added this feature that turns the icon black if the speed goes above sixty, and you can also see the exact speed.”

“We want them to get there fast,” John says, “but first of all we want them to get there. So we all smiled at each other and then I picked up a radio and called the driver and I said, Hey, slow down! You’re doing seventy-eight! There was this big silence and up on the Board we watched the speed plummet and could practically feel him slamming on the brakes. About ten seconds later the guy gets on the radio and says, How the bleep did you know that!

Back in the day, if there was a medical problem out on the course, someone would put out a general call until one of the medical vans picked it up and headed for the scene. The problem was that nobody had an overall picture so there was no way to tell if that was really the best van to send. Now, a quick glance at the Board along with the medical dispatcher’s knowledge of what each of his teams is doing at the moment allows him to call up the closest available team and get them on the case.

Spotting anomalies is a particular strength of the system. “We can do a lot,” Pat explains. “If there are an unusual number of flats at Mile 50, we’re going to know it right away from the data that’s been logged and get a crew out there to check the road. We’ve also got direct feeds from the Department of Transportation’s street cameras and police surveillance cameras and can selectively put the images on the Board.”

They can also log data on individual athletes. If a runner has a medical problem but gets going again, the information can be used to see whether he showed up at the next timing point. Nextel “transponders” are carried by every ice truck, every NBC crew, the motorcyclists accompanying the male and female race leaders, even the helicopters. All of their locations are tracked in real time and any of them can be in two-way communication within seconds. There are other feeds into the center as well, including information on earthquake activity from the US Geological Survey, readings form the Pacific Tsunami Warning Center and detailed weather information from Doppler radar. Ten-minute briefings are held every 90 minutes to review overall status and incident reports.

“Our job isn’t to tell the race directors how to do their jobs,” John emphasizes. “What we do is give them the information and tools to do their jobs better. If you wanted to sum up the purpose of the center in a couple of sentences, you could say it’s twofold: First, we gather information, cast it into useful form, and disseminate it back out to the people who need it. Second, we provide a central command facility in which relevant personnel can gather in the event of an emergency.”

As an old-school systems pro myself, one of the things I find most impressive about the ROC is how it has reduced the amount of time wasted in deciding how to decide. The center conforms to the federal NIMS (National Incident Management System) protocol, as do many of the agencies it deals with, and so the procedures of gathering, interpreting and disseminating information are familiar to all the players. Equally important, there is a well-defined chain of command in place, with no need to spend a lot of time debating about who has the authority to do what. When incidents arise that need resolving, the individuals responsible for that resolution are able to focus on making the best decisions possible without getting distracted by the mechanics of decision-making.

The ultimate authority rests with Diana. “I used to get bombarded with things better handled by other people,” she says. “Now, I’m able to concentrate on more critical matters while still keeping my finger on the overall pulse.”

More correctly, the ultimate authority rests with Diana so long as the ROC stays the ROC. Should any of a number of pre-defined situations arise, the ROC turns into the EOC (Emergency Operations Center), and then life inside the room changes dramatically.

“Authority shifts depending on the nature of the emergency,” John says. “This is an island, and our resources are strictly limited. If what we need isn’t within fifteen minutes of us, it’s going to be hours or maybe not at all. So if there’s an earthquake and parts of the course become impassable, or if there’s a brushfire that interferes with visibility to the point where safety is compromised, we’re on our own, and if one of the agencies asks us to turn the center over to them, we’ll do it. We’re firm believers in continuity of operations and a unified command structure.” In other words, only one person at a time can be the Big Kahuna, and who it is depends on what’s going on.

And that, friends, is how fresh ice gets to the aid station at Mile 15 and why no athlete in distress is alone for long and why you’re in good hands if an earthquake in Japan threatens to swamp the finish line and why we’ll probably have a lot of warning should a sixty-ton spacecraft with a badly decaying orbit decide to have its homecoming party in our vicinity.

At least that’s what they tell me. Personally, I still think it’s elves.