How do you know your configuration management plan is doing what you think it's doing if you're getting different output from the same source? It's hard to tell if the sources of variation in the output are harmless or meaningful.
Well, if the target is to have the same binary from the same set of sources every time you do a build, then the configuration management plan is not working. A good configuration management plan is to ensure that things that can change are managed. It is not working, it needs to be revised.
In some of the plans I have seen, some deviations are tolerated but those deviations are spelled out in excruciating detail.
I've dealt with it by wishing I had reproducible builds.
If I'm debugging a production issue and want to create a debug build from the same source to step through things, it would be nice if I could just build both debug and release, check the hash on the release to confirm it matches production, then start debugging, without reference to any external records or other systems.
Reproducible builds reduce the number of links in the chain needed to verify what you're really doing.
I don't get it. Are you saying it's too hard or it doesn't solve the problem? It's worked in my experience.
I mean, my response isn't entirely a rhetorical question. If there's a reason it wouldn't work I'd be curious what makes his situation different from mine.
So I should have said, "Is there a reason you can't just build debug binaries at the same time as you build release"? I mean, I guess, but it's a little disappointing that this is the nugget that you've been dancing around.
The world "just" is my pet peeve. Before you say "why don't you just xxx" make sure you actually understand the problem thoroughly. Otherwise it's impolite to make a suggestion to "just do ...". If there is one thing I have learned it's that good devs make sure they really understand the situation before making suggestions.
I think this is related to "the grass is greener on the other side". Other people's problems seem eminently more solvable than our own. Sometimes that's even true but often it isn't.
That's pretty paranoid if you're already trusting the source or binary. Have you previously ensured that same source acts non-maliciously on all malicious inputs and runs on secure OS? (Sources of most attacks.)
In medical devices you validate the binary. The problem is if you rebuild the code and now it's different then what? Consider that compilers sometimes produce the wrong code. I've hand compilers do this to me. Build, run tests. Test fails. Investigate, assembly looks 'weird'. Rebuild and now it's correct. GAH!
That makes sense. It could help there depending on what validation requirements are.
High-assurance systems is my thing btw. There's a few ways we approach this issue. One is a certifying compiler (eg CompCert C) that's verified and tested to basically never screw up. CompCert got SIL qualification recently with Solid Sands partnering to validate them, too.
Another I like is equivalence testing across multiple binaries from multiple compilers with a large, automatically-generated test suit and fuzzers. Such test generators are already great for bug hunting. You just rerun them through the same app compiled with other compilers. Helps catch app and compiler bugs. If you have CompCert, I have another trick: re-compile with CompCert, re-test each case that failed, and any that suddenly pass were likely compiler bugs. The multiple compilers approach should catch them, though.
Now, I should point out that what reproducible builds is addressing are often changes in hashes that aren't changes in functionality. Just stuff like timestamps or symbols. You should be able to show the regulators that the only thing that changed was that kind of stuff. Maybe we need tools that automate that, too, showing diffs annotated by what does or doesn't affect execution.
Anyway, what exactly do they require of you for source-to-object correspondence? And is this U.S. F.D.A. or a non-U.S. regulator?
“Anyway, what exactly do they require of you for source-to-object correspondence?”
In my case this is not really clear. You have quite a bit of leeway but you never know when you will get rejected. We had things that are approved a few years ago being rejected now because some rules have been tightened.
With reproducible builds you could save a lot of time validating build chains. It would be a huge timesaver and probably also allow us to be more creative with the build chain because if the binary is the same you know it worked.
