Collaboration Gets the MBTA Moving Into the Security Fast Lane
The history of mass transportation around the Boston and eastern Massachusetts area goes back more than four centuries. They lay claim to the first working subway system in the North America and one of the earliest motorized bus transit authorities in the nation. But it has been less than five years that what is the Massachusetts Bay Transportation Authority, commonly known as the “T”, has had a centralized command and control strategy for its security operations.
That was problematic when you consider the scope of the MBTA’s operations. The MBTA is the nation’s fifth largest mass transit system, serving almost five million patrons in 176 cities and towns with an area of 3,249 square miles. The authority maintains 183 bus routes, two of which are Bus Rapid Transit lines, three rapid rail transit lines, five light rail routes, four trackless trolley lines and 13 commuter rail routes that encompass more than 145 individual commuter stations. The average weekday ridership for the entire system is approximately 1.3 million passenger trips.
It is also crucial to protect a fleet of other equipment consisting of more than 1,000 diesel and CNG buses, 32 dual mode buses, 28 ETB′s (electric trolley buses), 410 heavy rail vehicles, 200 light rail vehicles, 10 streetcars, 90 commuter rail locomotives, 410 commuter rail coaches,464 MBTA-owned specially equipped vans and sedans, and an additional 182 contractor-supplied specially equipped vans and sedans.
Prior to the establishment of the MBTA’s security and emergency management department, security issues were handled in large part by the MBTA transit police, which has primary jurisdiction on MBTA property and vehicles in each of the districts. The department has an authorized strength of more than 260 officers and 10 civilians and focuses most of their efforts patrolling Boston and the surrounding communities.
Job one for MBTA’s new security and emergency management department was bridge building with MBTA transit police, operations and IT. For a multi-faceted organization with a traditional business model, it could have been a tough sell. But not so says Randy Clarke, senior director of security and emergency management and Bradford Baker, deputy director of security & emergency management for MassDOT/MBTA.
“The biggest challenge was overcoming the organizational culture that was used to doing things as they had always done them in the past. There was some silo issues, but never really any push back,” says Baker. “In fact we had extremely quick buy-in from both the police force and upper management.”
Both Clarke and Baker had been working with the MBTA as security and technology consultants, so they were the logical choice to head the new department. When they assumed their new roles, the only cameras in the entire system were analog and were mostly used for fare evasion purposes. There was no technical or security systems staff. In 2009 the new security department finalized security assessments and created a five-year security plan.
Once the MBTA created the security and emergency management positions, the two former consultants found themselves tasked with building the foundation of a new department for one of the nation’s most dynamic transit systems. Clarke played to his strengths of organizational management and team-building, while Baker was brought in to further solidify the technology roadmap.
“I’m focused on technology,” chuckles Baker, “but I like to call Randy the director of accountability.”
For Clarke, there was the luxury of having significant DHS funding filtering into the new department. Still, the challenge and his main goal were to get this funding under control and directed into the proper projects. There was also the challenge of establishing a corporate-like security program and culture inside an extremely complex institutional business model.
“What makes this even more of a unique operation is that we not only have the 6,000 people employed at the T, our department actually is DOT-wide and responsible for another 4,000 employees, so there are personnel challenges,” says Clarke... “But when you consider that the T’s infrastructure dates back to 1898, it’s old. We sort of joke that nobody was thinking about installing fiber when they were digging those tunnels back then.”
Over the past four years plus, Clarke credits a vibrant spirit of team building among all the 25 different departments – including MBTA transit police, maintenance, operations, communications, engineering and the control center -- that interact with his security and emergency management group to create a cohesive organizational structure.
“It is all about collaboration and building a clear mission statement from the beginning,” adds Clarke, who has seen his department grow from two to 18 employees. “We had to earn trust from everyone we initially approached. But now we are a lot more independent since we have proved our value and all parties involved understand where we want and need to be in five to ten years.”
The MBTA had a lot of disparate projects being implemented by several groups within the organization. Clarke and Baker decided the best approach was to finish up a couple of key ongoing projects and halt all others so the team could reassess its priorities. The ultimate vision was to ensure that all designs that were in the pipeline reflected the long-term plan.
“Putting a five-year plan together was very important. We approached it like a risk mitigation session, assessing what we had based on the NIPP philosophy of detect, defend, deter and devalue. ,” Clarke stresses, referring to the important role the implementation of the Department of Homeland Security’s National Preparedness System (NPS) plays in supporting the building of a sustainable and core-competent plan.
When the roadmap was formalized, Clarke undertook the rigorous task of coordinating funding. He relied on the DHS’ Transit Security Grant Program (TSGP), which authorized Congress to divert more than $84 billion to strengthen the nation’s critical infrastructure and its transit systems against potential terrorist threats.
One of Clarke’s first steps was hiring a grants and budget analyst specifically dedicated to getting their security priorities in order. The MBTA has been awarded with more than $150 million since the TSGP started sometime in 2004.
“Dealing with the constant and changing landscape of funding issues is a major component of what we do. We have been very fortunate to get significant support from DHS and TSGP, and I can’t say enough about the positive collaboration and resource support that has been made available through both DHS and FEMA,” emphasizes Clarke. “There is a lot of paperwork involved in securing that $150 million. There are auditing aspects, accounting principles, grant writing and management, not to mention then having to manage all the budget implications of the various projects. It’s a lot of work managing the money and setting technology priorities that fit your vision.”
Because Clarke and Baker stepped into their respective security roles having a working history with the MBTA, they absolutely knew what they wanted – and what they didn’t. The most daunting problem they faced when it came time to initiate their video migration plan was more than 700 analog cameras populating the system and no IP technology devices or infrastructure.
The first call went out to an array of IP camera vendors who would need to match a very strict set of performance criteria as the migration to IP began. The second call addressed the void of a video management software solution. At the same time as the video hardware and software issues were being addressed, the team tackled what they called the linchpin to their entire migration process. The implementation of a scalable PSIM solution ranked as the team’s key technology mission which would allow them to tie the old to the new.
MBTA is currently transitioning from 700 analog cameras that were on a NICE Vision platform to 6,000-7,000 cameras using VidSys to control and manage the Geutebruck and Genetec VMS, Lenel’s access control and monitoring, along with Stentofon Communications’ emergency phone and intercom systems.
Baker says that cameras like those from Samsung and others were chosen based on picture quality, price point and their tight integration with the Geutebruck VMS platform. These cameras with the selected Geutebruck VMS allow them to tightly manage and throttle camera output during low interest scenes, aiding in efficient network utilization and storage, along with motion detection alarms.
“Our biggest problem when we got here was when we found out we were tied to a single technology provider for our video system – and there was no access control. If we simply wanted to add a string of DVRs, it took a lot of development money, plus we had to get on their roadmap to fit their development schedule. We had absolutely no leverage,” says Clarke, realizing he and his team never wanted to be held hostage by technology vendors again. “Obviously our plan was to move to an IP platform using Samsung and Axis cameras from the start. We also wanted to make sure it was us and not the technology providers or integrators that had leverage when it came to project management. But the most important step for us was to get a PSIM in place.”
The team realized that if they were to follow the migration plan of transitioning from existing video technology to IP, an integrated PSIM and VMS solution that helped users ramp up their technology knowledge was mandatory.
“Putting a PSIM overlay platform onto the system was really the only way to make this happen. Having been a manager and a user of systems, you know that you just don’t put up a solution as complicated as a PSIM and expect it to keep running all the time. We needed to determine how to get resources behind it and keep the system realtime. Once we become more reliant on this system, the more critical it will be.” Baker says. “This has come to a head now as we go virtual with all our servers. We built out an independent security network from scratch with our own private fiber optic infrastructure to run all these signals back to the command center.”
Baker adds this had to be done since they knew as they expanded to thousands of new IP-based cameras; the current network wouldn’t have been able to handle the capacity. He also maintains that the system needed to be sustainable and couldn’t rely on old technology.
Now as the MBTA security and emergency management team begin to future-proof their system, there are some key elements being addressed. They plan to move heavily into analytics, which is currently in a pilot program with BRS Labs. It will handle everything from fare evasion to tunnel security.
“Resiliency and redundancy are the keys for ensuring our success. We have to maintain up time and availability of all systems. These systems must continue to function through power outages and network events,” continues Baker, who adds that as they moved forward with the initial systems, the manufacturer-recommended redundancy platforms lacked mature enough solutions to meet his standards. ”It was actually a bit of a failure point, so we worked towards virtualizing our servers using standard virtualization platforms. This offered us the greatest redundancy and resiliency possible for all of our systems and also provided centralized systems diagnostics to give us the ability to better monitor the health of our systems. It might be the biggest step we will take in the next five years.”
Clarke says his goal is to receive more granular and real-time information from all their edge devices and provide a more holistic way to monitor the overall system.
Clarke and Baker demonstrate that a true collaborative effort among users, technology vendors and systems integrators can allow for a fast-tracked solution that produces successful results and unlimited potential.
