I accidentally found a security issue while benchmarking postgres changes.
If you run debian testing, unstable or some other more "bleeding edge" distribution, I strongly recommend upgrading ASAP.
I accidentally found a security issue while benchmarking postgres changes.
If you run debian testing, unstable or some other more "bleeding edge" distribution, I strongly recommend upgrading ASAP.
I was doing some micro-benchmarking at the time, needed to quiesce the system to reduce noise. Saw sshd processes were using a surprising amount of CPU, despite immediately failing because of wrong usernames etc. Profiled sshd, showing lots of cpu time in liblzma, with perf unable to attribute it to a symbol. Got suspicious. Recalled that I had seen an odd valgrind complaint in automated testing of postgres, a few weeks earlier, after package updates.
Really required a lot of coincidences.
One more aspect that I think emphasizes the number of coincidences that had to come together to find this:
I run a number "buildfarm" instances for automatic testing of postgres. Among them with valgrind. For some other test instance I had used -fno-omit-frame-pointer for some reason I do not remember. A year or so ago I moved all the test instances to a common base configuration, instead of duplicate configurations. I chose to make all of them use -fno-omit-frame-pointer.
Afaict valgrind would not have complained about the payload without -fno-omit-frame-pointer. It was because _get_cpuid() expected the stack frame to look a certain way.
Additionally, I chose to use debian unstable to find possible portability problems earlier. Without that valgrind would have had nothing to complain.
Without having seen the odd complaints in valgrind, I don't think I would have looked deeply enough when seeing the high cpu in sshd below _get_cpuid().
@AndresFreundTec I recognize -fno-omit-frame-pointer as being important for ASAN https://github.com/google/sanitizers/wiki/AddressSanitizerFlags which has some overlap in functionality with valgrind (but obviously a fundamentally different approach).
There are more coincidences that are even less interesting. But even the above should make it clear how unlikely it was that I found this thing.
@AndresFreundTec lmfaooooooo the fact that this was found because of -fno-omit-frame-pointer confirms this is the final FOSS Discourse Event, literally every single thing that people have ever talked about in FOSS for the past year makes an appearance here
@AndresFreundTec now that @fedora, @ubuntu and @archlinux have frame pointers, maybe this #xzorcist issue will encourage #Debian to follow suit
@AndresFreundTec but still, you did. 🙇👏
Nonetheless, amazing work. Hope you get a Gold Star at the office...
@AndresFreundTec outstanding...luck , knowledge and experience have met the right person at the right point in time.
Just to be clear: I didn't mean that I didn't do good - I did. I mean that we got unreasonably lucky here, and that we can't just bank on that going forward.
@AndresFreundTec dont for a minute think that you are ever having a holiday again! We need you monitoring those timings! The world needs you! 🏅
@AndresFreundTec but there are thousands of devs running all sorts of tests. The question is not "what were the odds that you missed it", but "what were the odds that everybody missed it"...
@AndresFreundTec That was more than just good - you probably stopped a devastating attack on the whole industry. The open source community owes you a huge debt of gratitude. Had that exploit gotten into the wild it would have been awful. Catching it early was an immense win.
I hope Microsoft recognizes how much of a contribution you made to the entire industry.
@AndresFreundTec Hey dude, you really save Millions, thanks a lot.
(Luck or not you did a fantastic job).
@AndresFreundTec Yep! Important to remember that one of the coincidences in the chain was that it landed in your lap, and you had built a set of skills over your career that were necessary to dig deeper. 🍻