Developers gonna develop. That’s why we’re developers. We want to set some implementation goal and then make that a reality. We like to stay heads down and focus on the immediate task at hand. Unfortunately, this can sometimes cause collateral damage. Secondary objectives can get ignored or even trampled in the race to meet the primary target. It’s also likely that other promising developments will get missed as they fall off the main path. Dealing with these issues is one of the many functions of compliance regulations.
SDLC, SOC 2, GDPR, CMMI, FIPS, PCI, & HIPAA. What comes to mind when you see these? While they might not all be four-letter-words, I suspect some choice four-letter-words come out when some of you think about these acronyms. While I cannot give you a silver bullet to deal with them, I can offer suggestions and pointers toward another four letter word: Help.
What gives me the authority to offer such help? Honestly, not much. I am not a build and release engineer, nor do I play one on TV. Nevertheless, I have to deal with these things as a developer, and I thought other developers might benefit from my perspective. I have been under the influence of one Paul Reed for the last few years. We contract with Paul to act as the build and release engineer for my team, so his experience and perspective have shaped our SDLC journey.
My experience is that most developers (myself included) do not like to think about the full SDLC. Developers want to solve hard and interesting technical challenges. We pour our sweat and tears into “making it work,” and then want to move on to the next challenge after we succeed. Planning and testing are sometimes viewed as necessary evils, and deployment can be thought about at the end like gift wrapping.
Then reality hits. Bugs happen, and customers start having issues. We have to know what version of the code they run. Modern software development uses third-party libraries extensively, and those can have vulnerabilities. We have to track the versions of that software and whether or not it has vulnerabilities. With multiple versions of all the code floating around, we have to make sure the right ones land in the right hands, be that QA, security, or customers.
It’s probably fine.
We can now return to those acronyms I mentioned at the beginning of this post. I can sum them up with one word: Compliance. That word can leave very different tastes in your mouth depending on your experience. Ideally, successful compliance is third-party confirmation of behavior. It’s a prerequisite to play in certain spaces. We can all acknowledge that it’s not a silver bullet. It does not cover everything, and it cannot keep up with the rapid change of many tech industries.
Let’s dive a bit more into Service Organizational Control, or SOC. That compliance framework deals with security, availability, integrity, confidentiality, and privacy. All the “ilities,” basically. SOC 1 focuses on financial aspects, while SOC 2 focuses on the handling of user data. Getting SOC compliant involves auditors and lots of auditing. Type 1 compliance involves a single audit, and means that the compliant organization passed at a given point of time. Type 2 involves ongoing auditing, and success represents a compliant lifestyle.
Threat Stack achieved Type 2 of SOC 2 with zero exceptions on its first try. We have done and will continue to do blog posts and webinars to share how we did that from a DevSecOps perspective. From my point of view as a developer, we did three things as an organization to pave the way for success:
- Universal buy in: Everybody at the company, from new hires all the way up to board members, were 100% committed to making this happen.
- Start where we were: Security did not pull arbitrary “best practices” out of a hat and then implement them, but instead documented existing practices and built from there.
- Automated referential integrity: Our Ops team built a tool called Sockembot to help us follow process. It acted like a real-time “Clippy” for SOC 2 tasks.
I should also mention a common issue here: letter of the law vs. spirit of the law. That comparison gets used in a number of ways. Shakespeare used it as a plot device for antagonists to procure mercy from villains. The Christian Bible highlights the tension and paradox involved in satisfying the letter but missing the spirit. The idea of “gaming the system” continues in that vein.
I wish to stand the perspective a little on its head. Instead of using the letter of the law to get around the spirit of compliance, use it to interpolate the spirit behind the compliance and aim for that. Do not justify failures. Seek to do more.
Here’s a configuration management horror story. Our static analysis tool indicated that I had a couple of numeric type issues in a third-party library: libevent. The tool even gave me the file name and line number. So I pulled up the location in our viewer to find the flaw:
The problem is obviously not a blank line, but having some experience with static analysis I realized it might be an artifact of the post processing. Maybe the issue happens in the macro of the line before? I unrolled the macro and found nothing. I discussed with my security group and they did not see the referenced issue either. I even went back to the static analysis vendor, and they could not see the issue either.
Then I realized the problem: Our application used a different branch:
Of course, this led to more questions. Why did we not use master? There were commits from a previous engineer (no longer at the company) on master and on the branch. Were they important? Could we upgrade the library? Did commits make it upstream? Did we need to pull over local changes to the upgrade? All of these needed answers, and all those answers required spelunking.
We have an open source tool, that Paul Reed developed, to help with this going forward: git-vendor-mirror. git-vendor-mirror replicates CVS’s vendor branches in Git, providing isolation and standardization. Specifically, it takes into account the requirements of non-developers (release, QA, managers, legal) in a way that arbitrary forking does not solve. You can get more details about the design from the documentation in the repo: https://github.com/preed/git-vendor-mirror.
Here’s the result when we updated node:
We have clear traceability to which version we are using, what we have changed, and how to move forward for the next upgrade.
The Lesson Learned . . .
The lesson learned here is this: While the shortest path between two points may be a straight line, you might trample on a bunch of other positive things along the way if you take that shortest path. It also means you will miss a bunch of other points that might be useful. That’s why it’s crucial to consider other stakeholders, take the blinders off as developers, and consider more than just local maximums.
There’s more to the SDLC than just pushing bits. Compliance requirements can help remind us of that, but we have to leverage them effectively. You can choose to make compliance a hurdle to jump or a springboard to use as a vault.