Optimistic dissatisfaction with the status quo of security
This article is a condensed version of a keynote speech Parisa gave at Black Hat Conference on July 8, 2018.
As I kid, I used to spend hours at the arcade playing whack-a-mole. With a toy mallet in hand, I’d smash as many plastic moles as possible. But the more moles I whacked, the faster they popped up out of their holes.
I haven’t played this arcade game in years, but there have been times when my career in computer security felt like a reality version of whack-a-mole. Computer security issues are emerging at a quickening pace, and everyone’s energy is spent knocking out the same problems over and over and over.
We have to stop taking a whack-a-mole approach to security. Instead, we need to focus our energy on tackling the root causes of bad security, strategically investing in long-arc defense projects, and building out our coalitions beyond security experts.
Tackle the root cause
As the world becomes more dependent on safe and reliable technology, we can no longer be satisfied with isolated security fixes. Instead, we need to identify and tackle the underlying causes of bad security—whether they’re structural, organizational or technical.
Project Zero, a team that formed at Google in 2014, aims to advance the understanding of offensive security and improve defensive strategies. Over the past four years, the team has reported more than 1,400 vulnerabilities in a variety of targets, including operating systems, browsers, antivirus software, password managers, hardware and other popular software. But what's more impressive than that number is the impact we’re seeing across industry in terms of tackling the root causes of bad security.
In the case of Project Zero, the team recognized that vendor response times for fixing critical security reports varied hugely, and it often didn’t tip in favor of the people using the technology. Unfortunately, software vendors don’t always have incentives aligned that prioritize security. To address that underlying problem, Project Zero introduced a consistent 90-day disclosure policy that removed the historical, time-consuming negotiation between security researchers and vendors.
Initially, this deadline-driven approach was controversial. It caused short-term pain for organizations that needed to make structural changes. But sticking to this approach resulted in vendors investing more in solving root problems that, for whatever reason, weren’t previously addressed. Since the introduction of the deadline-driven disclosure policy, one large vendor doubled the number of security updates released each year, and another vendor improved response time by 40 percent. When it came to the controversial deadline, 98 percent of the security issues Project Zero reported have been fixed within 90 days, up from 25 percent.
Through all of this, Project Zero worked in the open to advance the public’s understanding of exploitation techniques. Ultimately, the team recognized that one individual security researcher isn’t likely to change the behavior of a large vendor, but a larger public response can. The team sought out opportunities for collaboration with other vendors, and people came together, both inside and outside the walls of Google, to analyze and build defenses against exploits discovered in the wild.
Solving the root problems—especially in today’s distraction-driven environments—isn’t always the fastest or easiest route to take, but it builds a foundation for a more secure future.
Celebrate milestones to make progress on strategic projects
To make real security change, we need to commit to long-arc defense efforts, no matter how complex they may be or how long they take to complete. Maintaining momentum for these projects requires strategically picking milestones, communicating them repeatedly and celebrating progress along the way.
In 2014, the Chrome team set out on a mission to drive the adoption of HTTPS on the open web. We wanted the web to be secure by default, instead of opt-in secure. We also wanted to address confusion in our existing network security indicators; users weren’t perceiving the risk of HTTP connections given our lack of a warning. We knew this project would take many years to complete because of the complexity of the web ecosystem and the associated risk of making big changes to browser security warnings.
It's important to remember that nobody owns the web. It’s an open ecosystem of multiple players, each with different incentives and constraints—so projects of this magnitude require wrangling a lot of moving parts. To avoid creating warning fatigue and confusion about the web, we set strategic milestones over a long period and share them publicly.
My job as a manager was to make sure my team believed change was possible and that they stayed optimistic over the entire course of the project. We shared a comprehensive step-by-step strategy and published the plan on our developer wiki for feedback. Our milestone-based plan started out simple and increasingly upped the pressure over time. Internally, we found fun and inexpensive ways to keep team morale high. We kicked off a brainstorming day with a poetry slam—finger snapping included! We made celebratory HTTPS cakes, pies and cookies. We also had a team chat to share updates, challenges and a lot of GIFs.
Building momentum externally was equally important. When sites made the switch to the more secure HTTPS, we celebrated with the broader community—usually via Twitter. And we published a transparency report that shed light on top sites and their HTTPS status. Hooray for openness!
Since our official announcement of these changes, HTTPS usage has made incredible progress. The web is ultimately more secure today because of a loose coalition of people who were able to stay committed to seeing a long, ambitious project all the way through. Which brings me to my third point...
Build a coalition
As we proactively invest in ambitious defense projects where the benefits aren’t immediately clear, we need to build a strong coalition of champions and supporters.
In 2012, the Chrome team started its Site Isolation effort, a project that mitigated the risk of cross-site data theft on the web. The project turned out to be the largest architecture change and code refactor in the history of Chrome! This was no small task considering Chrome is 10 years old, has more than 10 million lines of C++ code and has hundreds of engineers committing hundreds of changes each day from around the world. The core Site Isolation team was made up of only around 10 people, so building a strong coalition of support for the project outside of the team was critical for its success.
Originally, we thought this project would take a year to complete. Turns out we were off by more than a factor of five! Estimation mistakes like this tend to put a bullseye on a project’s back from upper management—and with good reason. Luckily, the team regularly articulated progress to me and the reasons why it was more work than first anticipated. They also demonstrated positive impact in terms of overall Chrome code health, which benefited other parts of Chrome. That gave me additional cover to defend the project and communicate its value to senior stakeholders over the years.
Aside from management, the team needed allies from partner teams. If other Chrome team members weren’t motivated to help or didn’t respond quickly to questions, emails and code reviews, then this 10-person project could have dragged on forever. The team kept a positive attitude and went out of their way to help others, even if it didn't relate directly to their own project. Ultimately, they conducted themselves as good citizens to build a community of support—a good lesson for all of us. We might be able to find the problems and technical solutions on our own, but we rely on everyone working on technology to help clear the path to a safer future.
We’ll keep finding complex problems to solve as technology evolves, but I’m optimistic that we can continue to keep people safe. It just requires a little bit of change. We need to take a different approach to computer security that doesn’t feel like playing whack-o-mole. So let’s band together—inside and outside of our organizations—and commit to ambitious projects that solve the root problems. And let’s not forget to celebrate our wins along the way! 🎉
