Trail of Bits

It Depends

Episode Summary

Most people imagine software engineers tapping keyboards in a kombucha-keg filled room. But modern software isn’t written... It’s assembled. Developers write code, but they don't start from scratch - They use open-source code and libraries, developed by a community. Those building blocks are themselves dependent on other pieces of open-source software, which are built atop yet others, and so on. The dependencies of this software supply chain are therefore recursive - ‘nested,’ like a Russian Matryoshka doll. So you ask whether your software is safe, the answer is, "It Depends."

Episode Notes


Clint Bruce

Clint Bruce is a former Navy Special Warfare Officer, a graduate of the US Naval Academy, decorated athlete, and seasoned entrepreneur. A 4-year letter winner at Navy playing middle linebacker, captain and MVP of the ’96 Aloha Bowl Championship team, he was named to multiple all-star teams his senior year. He enjoyed opportunities with both the Baltimore Ravens and New Orleans Saints and was inducted into the Navy/Marine Corps Stadium Hall of Fame in 2009. Clint’s desire to serve was deep and firmly rooted. He left the NFL to pursue becoming a Navy SEAL and successfully completed BUDS (Basic Underwater Demolition SEAL Training) in 1998 with Class 217. Joining SEAL Team FIVE, Clint completed multiple deployments pre and post-911 directly involved in counter-terrorism and national security missions globally. He is a co-founder of Carry the Load, which was founded to restore true meaning to Memorial Day and celebrate the service and sacrifice of Police, Fire, and Rescue personnel and their families during the month of May. Clint lives in Dallas with his college sweetheart and three daughters who are not impressed that he played football or was a Navy SEAL.

Patrick Gray

Patrick Gray is the producer and presenter of the Risky Business weekly information security podcast, a weekly podcast that launched in 2007. He formerly was a journalist for publications including, ZDNet Australia, The Sydney Morning Herald, The Age, The Bulletin (magazine) and Men's Style Australia.

Eric Olson

Eric Olson is the Director of Threat Intelligence for Jet Blue Airways. A threat intelligence professional for more than 20 years, Eric has had executive roles including Senior Vice President of Product Management and Vice President, Intellugence Operations, at LookingGlass Cyber Solutions, and was VP of Product Strategy at Cyveillance.

Allan Friedman

Allan Friedman is Senior Advisor and Strategist at the United States Cybersecurity and Infrastructure Security Agency, and one of the nation's leading experts on Software Bill of Materials. Allan leads CISA's efforts to coordinate SBOM initiatives inside and outside the US government, and around the world. He is known for applying technical and policy expertise to help audiences understand the pathways to change in an engaging fashion, and is frequently invited to speak or keynote to industry, academic, and public audiences. Wearing the hats of both a technologist and a policy maker, Allan has over 15 years of experience in international cybersecurity and technology policy. His experience and research focuses on economic and market analyses of information security. On the practical side, he has designed, convened, and facilitated national and international multistakeholder processes that have produced real results, helping diverse organizations finding common ground on contentious, cutting edge issues.

Evan Sultanik, PhD

Evan Sultanik is a Principal Computer Security Researcher at Trail of Bits. A computer scientist with extensive experience both in industry (as a software engineer) and academia, Evan is an active contributor to open source software. He is author of more than two dozen peer-reviewed academic papers, and is particularly interested in intelligent, distributed/peer-to-peer systems. Evan is editor of and frequent contributor to the International Journal of PoC||GTFO

William Woodruff

William Woodruff is a senior security engineer at Trail of Bits, contributing to the engineering and research practices in work for corporate and governmental clients. He has developed several of our open-source projects (e.g., twa, winchecksec, KRF, and mishegos). His work focuses on fuzzing, program analysis, and automated vulnerability reasoning. Outside of Trail of Bits, William helps to maintain the Homebrew project, the dominant macOS package manager. Before joining Trail of Bits, he was a software engineering intern at Cipher Tech Solutions, a small defense subcontractor. He has participated in the Google Summer of Code for four years (two as a student, two as a mentor) and taught a class in ethical hacking as a college senior. William holds a BA in philosophy from the University of Maryland (2018).

HOST: Nick Selby

An accomplished information and physical security professional, Nick leads the Software Assurance Practice at Trail of Bits, giving customers at some of the world's most targeted companies a comprehensive understanding of their security landscape. He is the creator of the Trail of Bits podcast, and does everything from writing scripts to conducting interviews to audio engineering to Foley (e.g. biting into pickles). Prior to Trail of Bits, Nick was Director of Cyber Intelligence and Investigations at the NYPD; the CSO of a blockchain startup; and VP of Operations at an industry analysis firm.


Story Editor: Chris Julin
Associate Editor: Emily Haavik
Executive Producer: Nick Selby
Executive Producer: Dan Guido


Recorded at Rocky Hill Studios, Ghent, NY - Nick Selby, Engineer;
22Springroad Tonstudio, Übersee, Germany - Volker Lesch, Engineer

Remote recordings were conducted at Whistler, BC, Canada (Nick Selby); Clint Bruce was recorded in a Google Meet session; Patrick Gray provided recordings of himself from Australia, courtesy of the Risky Business podcast. Eric Olson recorded himself on an iPhone. Washington, DC (tape sync of Allan Friedman by George Mocharko). Trail of Bits supports and adheres to the Tape Syncers United Fair Rates Card.

Edited by Emily Haavik and Chris Julin
Mastered by Chris Julin


You can watch a video of this episode.


Dispatches From Technology's Future, the Trail of Bits theme, Chris Julin
EVERYBODY GET UP - No Vocals & FX - Ian Post
JD SCAVENGER by Randy Sharp
RIPPLES by Tamuz Dekel
FUTURE PERFECT, Evgeny Bardyuzha
THE SWINDLER, The Original Orchestra]
LITTLE EDGY, Chris Julin
SCAPES: Gray North


With the exception of any Copyrighted music herein, Trail of Bits Season 1 Episode 3; It Depends © 2022 by Trail of Bits is licensed under Attribution-NonCommercial-NoDerivatives 4.0 International. This license allows reuse: reusers may copy and distribute the material in any medium or format in unadapted form and for noncommercial purposes only (noncommercial means not primarily intended for or directed towards commercial advantage or monetary compensation), provided that reusers give credit to Trail of Bits as the creator. No derivatives or adaptations of this work are permitted. To view a copy of this license, visit

Referenced in this Episode:

The original blog post announcing the availability of It Depends describes the history you just heard with more technical specificity, and also of course links to the GitHub repository where you can download It Depends and try it for yourself. 

That blog post also links to the repository where you can download pip-audit, and give that a whirl.

In the 2021 Executive Order on Improving the Nation’s Cybersecurity, the Biden Administration announced that it would require SBOMs for all software vendors selling to the federal government.

Dependabot is a tool available to GitHub users. 

If you’re interested in the catalog of open source projects Trail of Bits participates in and contributes to, please read the blog post 

Celebrating our 2021 Open Source Contributions. There, you can read about our work contributing for example to LLVM - the compiler and toolchain technologies we discuss in the Podcast episode Future - to Pwndbg, a GDB plug-in that makes debugging with GDB “suck less.” The post includes links to contributions our engineer consultants have made to a huge range of open source projects from assert-rs to ZenGo-X.

Meet the Team:


Chris Julin has spent years telling audio stories and helping other people tell theirs. These days he works as a story editor and producer for news outlets like APM Reports, West Virginia Public Broadcasting, and Marketplace. He has also taught and mentored hundreds of young journalists as a professor. For the Trail of Bits podcast, he serves as story and music editor, sound designer, and mixing and mastering engineer.


For the past 10 years Emily Haavik has worked as a broadcast journalist in radio, television, and digital media. She’s spent time writing, reporting, covering courts, producing investigative podcasts, and serving as an editorial manager. She now works as an audio producer for several production shops including Us & Them from West Virginia Public Broadcasting and PRX, and APM Reports. For the Trail of Bits podcast, she helps with scripting, interviews, story concepts, and audio production.

Episode Transcription

NARRATOR (NICK SELBY): We’ve been thinking a lot about supply chains lately. Because they’ve been breaking. Coffee. Surgical masks. Computer chips. Condoms. Lunchables. They’ve all fallen victim to supply chain problems. 

And that got me thinking about my friend Clint Bruce.

CLINT BRUCE  [FROM VIDEO CALL]: - Hey buddy, how are you? 

NICK SELBY [FROM VIDEO CALL]:  - I’m good, man, it’s really good to see you. Hey, Clint, thanks so much for doing this

NARRATOR: Clint’s a former Lieutenant Commander in Naval Special Warfare - a Navy SEAL-  and I was remembering something he told me a while back, so I got him on a video call. 

NICK SELBY [FROM VIDEO CALL]:  - Tell me why supply chains matter to Navy SEALs

CLINT BRUCE [FROM VIDEO CALL]: Basically, you put my team in front of a door anywhere in the world and our job is to get through it.  People know about that from the news and movies, which is kinda cool, but what they may not think about is how we get to the door in the first place. Because some Marines drove us in a truck, after they pick us up from the point where the Army dropped us off after meeting us on shore, because the Navy brought us to the shore from the aircraft carrier, that the Air Force transport brought us to from the base, and each of those things has to work from the people being healthy to the vehicles starting when they’re supposed to, so basically, if we don’t have that entire supply chain up and running, my guys don’t kick in a door and be heroes and get to do shaving commercials. 

NARRATOR: So yeah, the SEALs have a supply chain. 

Anybody who makes or assembles or, really, does anything maintains what’s known in the industry as a “BOM” - a Bill of Materials, like the one Clint was describing. At an industrial chemical plant, or television factory, or tire maker, wherever: every thing you buy has a BOM, and each component of every BOM in turn has its own BOM - and so on. String all these together? You have a supply chain. 

MUSIC - Everybody Get Up - No Vocals & FX - Ian Post

NARRATOR: Modern software has a supply chain. 

Most people have a quaint vision of well-paid software engineers tapping keyboards in a giant, fern, foosball-table- and- kombucha-keg filled room, modern software actually isn’t written. 

It’s assembled. 

Sure, developers still write code -- but for the core functions of most software, they don't start from scratch. They use open-source code and libraries, developed by a community, and available to everyone. And those open-source building blocks are themselves dependent on OTHER pieces of open-source software, which are built atop yet others, and so on. So the dependencies are recursive, or ‘nested’ like a Russian Matryoshka doll. 

And, as it’s become clear to anyone who’s tried to buy a car recently, anything that depends on a supply chain can suffer a break in that chain. 

So when you ask whether your software is safe, well…It depends.


NARRATOR: In 2021, talk about the software supply chain moved from the pocket-protector crowd to the mainstream after the high-profile hack of Solar Winds - software used by tens of thousands of companies around the world, ironically, to improve their security. 

PATRICK GRAY: If you had to point to one single event that made people care about supply chain security, it was the SolarWinds thing. No question.

NARRATOR: Patrick Gray is a journalist covering the world of information security, and he’s run the Risky Business podcast since 2007.  

PATRICK GRAY: it does make sense because as soon as you start [00:00:45] scratching the surface of what software supply chains actually look like just under [00:00:50] that surface, it is a horror show

NARRATOR: Even though companies and security experts were sounding the alarm for years, until Solar Winds, the damage wasn’t scary enough or clear and present enough to get noticed in the boardroom. Here's one reason for that. The software supply chain is highly complex. So some business people simplify it to make it easier to talk about. 

ERIC OLSON: On my tombstone it’s going to read, ‘Nuance matters.”

NARRATOR: Eric Olson is the Director of Threat Intelligence and Incident Response for Jet Blue Airways.

ERIC OLSON: So-called supply chain attacks are like ‘cyber security:’ people talk about it like it’s a monolithic thing, and it’s just not. Defenders work for business people – where the nuance is (properly) lost.  Attack details don’t really matter from their perspective. the attackers understand that nuance matters, because they care about the one, minute opening that lets them do what they do.

MUSIC JD Scavenger by Randy Sharp

NARRATOR: The nuance Eric mentions is based on the components that are in your software - the kind that Solar Winds highlighted for us: what is in your software supply chain? The only way to find that out is to get a read on all of your dependencies. 

To do this, you can generate something called a software bill of materials - an "SBOM."

ALLAN FRIEDMAN: So if you go to the store today and buy a Twinkie, it's going to come with a list of ingredients. Why don't we expect that same level of transparency in software, right? 

NARRATOR: That’s Allan Friedman, senior advisor and strategist at CISA - the Cybersecurity and Infrastructure Security Agency. Allan’s been talking about SBOM for years now - a lot longer than most of us have been. 

ALLAN FRIEDMAN: Software isn't hewn out of alabaster marble by tonsured monks on Greek islands. It's something that's assembled of known components. An SBOM is the enumeration of those components. This product uses this component which in turn uses these other components …

NARRATOR: But SBOMs remained the realm of frustrated policy wonks, gobsmacked that everyone kept not listening - until one day in May 2021, just months after the SolarWinds debacle was first reported, the White House put them center stage: in the new Executive Order on Improving the Nation’s Cybersecurity. The Biden Administration announced that it would require SBOMs for all software vendors used by the federal government.

ALLAN FRIEDMAN: This is the highest levels of government trying to figure out how do we react and make legitimate progress on both the security of our software, the security of the software supply chain… 

NARRATOR: With that one Executive Order, SBOM went from a nice-to-have, to something you need if you want to do business with the federal government. 

And in the end, the work at Trail of Bits to develop It Depends was based upon work supported by DARPA - the Defense Advanced Research Projects Agency. 


NARRATOR: Now, there are already commercial tools that will deliver an SBOM.

But none of these tools went a step further, and collected the nested dependencies. So, even though all software is dependent upon something - that’s dependent upon something else - that “something else” wouldn’t show up. That’s a big problem. 

And it’s a problem that Trail of Bits wanted to solve.

EVAN SULTANIK: Hi, I'm Evan Sultanik. I'm a principal security engineer at trail of bits…The original problem we were trying to solve was to create a tool that could automatically analyze and compare open source code bases to see how similar they were to each other with respect to the upstream dependencies that they used. We wanted to be able to see what's the probability that if an upstream library were vulnerable or compromised, how would that trickle down into the software we actually care about?

NARRATOR: To start off, they did what most software engineers do: 

EVAN SULTANIK: Initially, we wanted to leverage as much existing tooling as we could to solve this problem.  

MUSIC: Ripples by Tamuz Dekel

NARRATOR: The team initially homed in on Dependabot - a tool that was available in GitHub, one of the most popular code repository applications in the world, Dependabot is a tool that can automatically scan your codebase, highlight out-of-date dependency libraries, then insert suggested updates to them within the development workflow. 

EVAN SULTANIK: Dependabot seemed like a great choice because it's open source, and it's… integrated with GitHub. It seemed like a great idea. It can already scan code bases, and… it can enumerate all of the libraries and dependencies in a codebase.

EVAN SULTANIK: my colleague, Alex Remie, quickly made the first proof of concept of a tool that could extract SBOMs using dependabot. 


NARRATOR: But Alex was super busy on another project, so …

EVAN SULTANIK: He transferred that to Eric Kilmer, who was going to polish it… and he got a fully functioning proof of concept. However, from that exercise, we quickly realized that one, it would be very cumbersome to extend dependabot to create the superset of dependencies that we were really interested in. Also dependabot, it was really slow. And we thought that it shouldn't have to be that slow.

NARRATOR: Then the showstopper:

EVAN SULTANIK:  another problem is dependabot doesn't even support common languages like C and C++.

MUSIC: Future Perfect,  Evgeny Bardyuzha

NARRATOR: Dependabot just wasn’t working. We had to  get that full universe of nested dependencies the team was looking for, and we had to make it applicable to a range of common languages and package managers. Otherwise the project was just another SBOM generator. 

To make matters more complex, to deploy software, you have to deal with package managers. You’ve seen their ilk when you download a Windows exe file or a Mac installer. The problem with package managers is, they - might … just … grab any one of several qualifying versions to satisfy the requirement, based on what else is running on that box. 

EVAN SULTANIK: There's almost never just one dependency resolution for a piece of software. The dependencies that the package manager will choose to satisfy its requirements are going to depend on the environment in which it's being deployed. It might have drastically different libraries that get installed on staging than it will in production.


NARRATOR: That's big, Evan says. And it makes it clear what we need to do. 

EVAN SULTANIK: ,we want to enumerate all possible dependency resolutions… not just the single one on the system on which Dependabot is run.

NARRATOR: So the unique proposition of It Depends is that it recursively builds a project’s dependency graph starting from either a source code repository or a package specification, enumerating every dependency it could possibly have, not just a single resolution. 

If Trail of Bits could create a tool that could capture those dependency nuances, and could support common languages like C and C++, this could be a really big advance in dependency management. And of course, like all our tools, this needed to be free:

EVAN SULTANIK: From day one, we thought that if a tool like this existed, it would be incredibly beneficial to the open source community. So it was always our intention to release it for free to the public as an open source tool.

NARRATOR: The path to get there wasn’t easy. But with all the risks that come with not knowing what’s actually running in your software, this achievement begged the question: why had no one made a tool like this before?

WILLIAM WOODRUFF: My name is William Woodruff, and I'm a senior security engineer at Trail of Bits.

NARRATOR: Will has a guess at why.

WILLIAM WOODRUFF:  I think the single reason why we haven't seen a lot of other tools attempt to create a sort of whole universe software bill of material is because it is just on its face, it's a very difficult problem to generalize…

NARRATOR: Actually, it’s a mess; a dog’s dinner of possibilities. This flies in the face of why it feels so easy to click that install wizard and watch an installation get magically handled for you. So we’d like to give you an idea of what is happening behind the scenes when you click that Go button - it’s actually…Well… It’s crazy. 

WILLIAM WOODRUFF: there are three major operating systems in current use. Two of which are commonly used for server deployments. On those you can have any number of system level dependencies, versions of operating system vendor dependencies, depending on how up to date the system is, you can have hot fixes, patches that are applied by the company, at runtime, because you have specific requirements. 

NARRATOR: Will has spent a lot of time in his career thinking about dependencies - for six years, he’s been a maintainer of Homebrew - a free and open-source software package management system that makes it easier to install software on Apple machines - and that’s where he got his first taste of dependency management. More recently, he’s been a professional contributor to the packaging ecosystem for the Python language, adding features to Py P I  - the Python Package Index, and adjacent tooling. 

Sometimes, rather than just depending on an open source library and loading it when they need to, software developers will actually copy the files of that library directly into their code, and begin to maintain it separately from the open source project. That’s called Vendoring, and it throws another monkey into this wrench:   

WILLIAM WOODRUFF: You can have versions of Python packages that are spoofed because the Python package has been vendored in by the operating systems package manager instead of the Python package index installer. So all of these things are very frustrating exceptions to the general process of dependency resolution and dependency analysis. It’s one of these things where the 90% case is extremely easy. And then the 10% case requires an extraordinary amount of long term detailed effort to model everything correctly.

MUSIC: The Swindler by The Original Orchestra

NARRATOR: One of the things that sets It-Depends apart is the fact that it doesn’t assume you did everything right. 

WILLIAM WOODRUFF: a lot of security tools assume that people are going to take the happy path. That means that they assume that all the dependencies and sub dependencies… are fully resolved locally. And that you know exactly, if you haven't already fetched those dependencies and sub dependencies, which exact versions… you're going to fetch. This is a very conservative assumption to make when you're doing analyses

NARRATOR: Just to be clear, the assumption that a user took the Happy Path? It’s almost never correct. 


NARRATOR: That leaves the Sad Path. 


WILLIAM WOODRUFF:  The sad path is basically that degree of uncertainty. When you install Python packages to the Python packaging ecosystem, each package can run arbitrary code, which at runtime can make dependency decisions for you, depending on the kind of operating system you're running on, for example. 

NARRATOR: This means that package managers and the packages you are installing will actually make different decisions about which dependencies to give you when you click that Go button, depending on, say, whether you are on a Linux machine or a Windows machine. They don’t ask.

WILLIAM WOODRUFF: Because of this, there is all kinds of sadness.

NARRATOR: The sad path involves a lot of uncertainty. 

WILLIAM WOODRUFF:  when you're on the sad path… you have to work from the outside in, is the way that I think about it. So your view of the system is what's happened after you've finished setting up all your dependencies and setting up your runtime, you know, you only know at the end, what's what it's going to look like, you don't have a sort of determinate view from from the from the outset. as a result, you have to do a lot more work if you want to approach or approximate that end view, without actually having to do the entire setup. 

NARRATOR: It-Depends doesn’t work for everything, so with support from Google, Trail of Bits developed another tool called pip-audit. Pip-audit looks through Python dependency trees for known security vulnerabilities - things that are publicly known to the open source community to be a weakness in a popular package. It works symbiotically with It-Depends: 

WILLIAM WOODRUFF: The way that I think of pip audit and it-depends is that they cover two halves of the same whole. Pip audit is a developer-focused tool that helps individual Python application developers identify the vulnerable dependencies in their stack, patch them, get them updated, and move on with their lives. It-depends is a sort of swiss army knife that security engineers like myself and others might use to generate an entire manifest of a project, determine perhaps what the more structural issues within that package might be, or project might be. 


NARRATOR: A tool like it-depends can be used in many different ways. One of the specific goals at Trail of Bits - was to support the company’s assurance practice. So when a Trail of Bits security engineer begins to assess a customer’s code base, they can run it-depends to get a better idea of what the dependencies are, and what vulnerabilities might exist.

EVAN SULTANIK:  software development teams can integrate it-depends into their CI pipeline, and be emailed anytime that the SBOM of their software tool changes. Companies undergoing due diligence or auditing can also run it-depends to generate an SBOM. And in general, companies who are concerned about supply chain issues can set their minds at ease by at least understanding the current requirements of their software.

NARRATOR: A  former Facebook engineer, Charity Majors, once said that trying to fix software without seeing what it’s doing is like trying to repair an engine by placing your ear against the car’s fender.

At its core, the goal of it-depends was to get underneath the surface. To give all developers a tool to check their work, look under the hood, and make sure what they’re building is safe. Meanwhile, Evan and his team at Trail of Bits continue to work on it-depends, to make it a stronger, more efficient tool. 

EVAN SULTANIK:  Development of it-depends isn't over, there's still lots of work to do. More languages to support and more package managers to support. 

NARRATOR: For example, one problem is that despite an effectively global standardization of known security bugs into a library called Common Vulnerabilities and Exposures - CVE - the names used by people who discover vulnerabilities are often marketed differently from the official CVE name.

EVAN SULTANIK: It's actually a really difficult problem to go from those names to the CVEs that affect it. Because it's effectively a string matching problem. For many pieces of software. Some package managers do already have a vulnerability database that we can leverage…. But for C and C++ code, it's actually really hard, in many cases, to map CVEs to the libraries that they're using.

NARRATOR: And Trail of Bits is now getting around to detection of that vendored code: 

EVAN SULTANIK:  It-depends currently doesn't detect that. It-depends is only looking at external dependencies of a code base. But if you copy-paste vulnerable code, you are going to have that vulnerability in your code base. So that's still an open problem.


NARRATOR: Trail of Bits is strategically committed to open-source work - with an equally strong commitment to both development and continuous updates of the work done by ourselves and others. In 2021, Trail of Bits employees submitted more than 190 contributions to non-Trail of Bits repositories.

EVAN SULTANIK:  We've already had some excellent open source contributions. It looks like we're getting a lot of traction and usage in the community and we're really excited about the future of it-depends. 

NARRATOR: There are links in the show notes to these articles and more: 

The original blog post announcing the availability of It Depends describes the history you just heard with more technical specificity, and also of course links to the GitHub repository where you can download It Depends and try it for yourself. 

That blog post also links to the repository where you can download pip-audit, and give that a whirl.

The show notes also link to the 2021 Executive Order on Improving the Nation’s Cybersecurity

If you’re interested in the catalog of open source projects Trail of Bits participates in and contributes to, please read the blog post Celebrating our 2021 Open Source Contributions. The post links to contributions our engineer consultants have made to a huge range of open source projects from assert-rs to ZenGo-X. 

Season One of Trail of Bits is available for download now, wherever you get your podcasts. 


MUSIC IN Scapes - Gray North

NARRATOR: The people who worked on this podcast are Emily Haavik, Chris Julin, Dan Guido, Clint Bruce, Patrick Gray, Eric Olson, Allan Friedman, Evan Sultanik, Will Woodruff, Trent Brunson, and hi, I’m Nick Selby, I’m the Director of the Software Assurance Practice here at Trail of Bits.  

Chris Julin made our theme music. 

Trail of Bits helps secure some of the world's most targeted organizations and devices. We combine high-end security research with a real-world attacker mentality to reduce risk and fortify code. We believe the most meaningful security gains hide at the intersection of human intellect and computational power. Learn more at trailofbits dot com; on Twitter we are AT trailofbits; Dan Guido’s Twitter is AT Dguido, and I’m AT fuzztech.