Safeguarding the World from the SolarWinds Hack and Future Supply Chain Attacks

by Cole Kennedy, Andrés Vega | Monday, Dec 14, 2020 | DevSecOps

Over the weekend, FireEye discovered an active threat (SOLARBURST) from a trojanized software supply chain. CISA has issued an emergency directive to remove SolarWinds from United States Department of Defense IT systems. Supply chain compromises are becoming a frequent occurrence, and this does not come as a surprise to us; as leading practitioners at BoxBoat and VMware, we understand the threat vector and the widespread damage a release compromise can have, and for that reason, we urge the need for Secure Software Supply Chain (SSSC)

What Happened?

During the release process, attackers compromised the SolarWinds build process. Alterations were made to the source code, an artifact used during the build process, or the release process. In spite of the compromise, the malware was signed with SolarWind’s own certificates, indicating mismanagement of certificate keys and lack of hardware-based signing mechanisms, or worse, an oversight on authenticating their build process practices.

These compromised binaries were downloaded and installed in production systems. The malware has been running undetected since at least March 2020. The vulnerability allows attackers to forge single sign-on tokens that impersonate any of the organization’s existing users and accounts. It is unclear how many systems are currently affected. CISA’s emergency guidance is to shut down all systems running SolarWinds. Removing SolarWinds will leave many network operators without a network monitoring solution amplifying the risk.

Build Time Mitigation Steps

Secure Software Supply Chains require signed code commits and trusted build artifacts. Build engineers should avoid “BlackBox” libraries. Code review and sign-off of commits merged into production builds must undergo an organizational change management process. This process is the most essential part of delivering secure software. No vendor solution will fix a broken change management process. However, as one example, GitLab with a GitFlow release process does come close.

Each software build must-have steps that verify the integrity of the software. We work closely with ecosystem partners such as GitLab and Aqua Security whose solutions ensure our customer’s software is not released with known CVEs. Our services teams, particularly BoxBoat’s, have years of experience turning organizational change policies into pipeline automation.

No single person should ever be responsible for a software release, and the process must be as automated as possible. A Secure Software Supply chain must go through the following key stages:

• Source Code Management
• Code Building
• Artifact Packaging
• Scanning
• Attestation/Control
• Distribution/Storage
• Verification
• Test
• Provision/Deploy
• Connect/Wiring

Build step attestors should sign release and artifact meta-data. In BoxBoat’s world, our artifacts are generally container images deployed on Kubernetes. As a best practice, we implement TUF using Notary for our security-conscious customers. However, TUF can be applied to ANY release process.

We have recently started working with the open-source project in-toto. The In-toto project aims to provide a framework to attest each step of the build process cryptographically. This process’s result is a cryptographically attestable digital bill of materials (i.e., a cryptographically-verifiable paper trail of all the actions performed in the supply chain); this BOM can automate downstream change management processes. For example, DataDog’s implementation of in-toto in their pipelines would likely have stopped the SolarWinds attack dead in its tracks. Affected systems trusted the SolarWinds CA based on no evidence of release process or verification. Administrators had no access to the SolarWinds BOM. An In-toto based supply attestation system would provide this verification to enterprise administrators.

Run Time Mitigation Steps

It is extremely difficult or impossible to detect a zero-day exploit. Exploits such as the SolarBurst in addition to the OWASP Top 10 Web Application attacks are most effective and plausible in perimeter-based security architecture. Implementation of a zero-trust security model, or “trust-zero” network means that applications are only granted access to specific network resources strictly based on their cryptographically provable identity.

In the military, soldiers are issued a common access card (CAC). This ID card, however, does not grant any privileges on base. When entering a building, the ID card is scanned, and a backend system verifies access to that building. The CAC card only verifies identity, not access. Application services have much more access to sensitive material, yet many organizations do not verify their identity as part of a standardized process.

Just as soldiers are required to prove their identity before entering a secure facility, we should also verify every application’s identity before they are granted privileges to sensitive data and system resources.

For software, this identity verification comes in the form of attestation of attributes such as the SHASUM. A SHA256 hash proves, without a doubt, the integrity of software. It is like a DNA match but leveraged for software service identity verification.

We have started implementing the Secure Production Identity Framework for Everyone (SPIFFE/SPIRE) project for service identity in all engagements going forward. The SPIRE software implementation provides the framework for a universal service identity. SPIRE does this by issuing an x.509 certificate, with a service identity based on selectors defined by administrators. For example, an identity certificate will only be issued if the SHASUM of a container image and the specific hardware it is running on (from something like TPM or cloud meta-data services) can be verified. This certificate can then be used to form and filter mTLS connections to authorized services with an application proxy such as Envoy Proxy.

Implementation of this open source zero-trust architecture reduces the blast radius even if malicious code makes its way into production systems. It allows network operators fine-grained traceability of services within IT infrastructure and permits the immediate revocation of credentials across the enterprise upon discovering the threat.

Key Takeaways

Securing a software supply chain is extremely important. Enterprise software customers must demand insight into the build and release process from their vendors. In an open-source dominated ecosystem, only a small part of closed source application’s code comes from the vendor. A signed image or binary is no longer enough to validate the integrity of the software. An enterprise must require a signed bill of materials with each release so they can determine the appropriate threat level and mitigations.

Frameworks such as in-toto and SPIFFE/SPIRE provide the way forward to a future without supply-chain compromises. BoxBoat is a cloud native integrator with experience implementing these patterns for security-conscious commercial enterprises and DoD programs at all impact levels.

Andrés Vega is a product line manager at VMWare where he works with the Tanzu team. Tanzu provides the framework to build, run, and manage modern applications by providing a way for an application to run and, from its environment, get all the information necessary to communicate securely over the network.

References:

1. https://www.fireeye.com/blog/threat-research/2020/12/evasive-attacker-leverages-solarwinds-supply-chain-compromises-with-sunburst-backdoor.html
2. https://msrc-blog.microsoft.com/2020/12/13/customer-guidance-on-recent-nation-state-cyber-attacks/
3. https://www.zdnet.com/article/microsoft-fireeye-confirm-solarwinds-supply-chain-attack/
4. https://krebsonsecurity.com/2020/12/u-s-treasury-commerce-depts-hacked-through-solarwinds-compromise/
5. https://github.com/cncf/sig-security/pull/486