The two most interesting insights I got from this article are that:
(1) SSDs are good for privacy for average users since
they are cleaning up dirty blocks in the background.
However, IMO, privacy-conscious users who are running a daily
free-space wipe, a conventional hard disk is superior
because it guarantees that all dirty blocks are erased.
A free-space wipe on an SSD can't guarantee that reserved or
remapped blocks get erased.
(2) He says, "Somewhat counter-intuitively, information deleted
from certain types of encrypted volumes (some configurations of
BitLocker, TrueCrypt, PGP and other containers) may be easier to
recover ... if the investigator knows either the original
password or binary decryption keys for the volume".
If you delete a file in your encrypted volume (but don't do a
free-space wipe inside your encrypted volume), then someone who
knows your key could potentially recover that file.
But that's always been true -- it's true for both SSD and
conventional drives.
What I think the author is saying is that someone who use an encrypted
volume doesn't benefit from the SSD's cleaning of dirty blocks
in the background because the entire encrypted volume looks like
it's in use to the SSD controller.
But I don't see how he concludes that it's "easier".
You lose the benefit of the SSD's garbage collection, but to recover a
deleted file from inside an encrypted volume (assuming you have the
user's key) is neither easier nor more difficult with an SSD vs. a
conventional disk.
> When Bitlocker is first configured on a partition, the entire partition is read, encrypted and written back out. As this is done, the NTFS file system will issue Trim commands to help the SSD optimize its behavior.
On Linux systems, you can configure LUKS encrypted volume to use trim[1], but there are additional consequences you need to consider before doing that[2].
Regarding point #2, I think "may be easier to recover" means easier than if it weren't encrypted for the reason you stated (the blocks are in use and won't be trimmed), not easier than on a conventional drive.
> A free-space wipe on an SSD can't guarantee that reserved or remapped blocks get erased.
I think most controllers implement a secure erase feature that guarantees the data have been erased from NAND.
> the entire encrypted volume looks like it's in use to the SSD controller
I have always wondered how encrypted volume worked on an SSD. It seems this will lead to serious performance issues due to ineffective garbage collection.
Yes, a secure erase of the entire disk. That does not help with erasing the free space (a free-space wipe). Erasing the free space can't be done purely at the controller level since the controller can't tell which blocks are free and which used.
But you can image the logical filesystem on the SSD, do the free-space wipe, and then restore the image to the wiped SSD. If you could find a way to automate on a Mac with FileVault then you'd be popular.
Wait, what? The whole point of wiping free space is to prevent file recovery tools from doing a low level scan of the drive and retrieving deleted data, since a delete doesn't actually zero out the space, it just marks it as available for reuse.
But you don't know whether the blocks are actually being over written or if they've been marked as bad or if the information is in slack space or etc.
"Pointless" was too strong. "Not secure" would have been better.
EDIT--
More specifically, most utils don't tell you that they don't touch slack space unless you search the forums. See, for example, CCleaner which lists a few limitations of CCleaner's free space wipe, but which makes no mention of slack space.
Thing is, that depends on the filesystem. I know that the usual shred commands don't work all that great on Ext4 due to journalling and such, but Microsoft actually provides[1] a first party tool to do it on NTFS (at the file level) to handle those edge cases.
Apple provides[2] a built-in free space nuking tool as well.
That leaves sector reallocation by the disk controller, but doesn't that only happen if data can't be properly written in the first place anyways?
> > A free-space wipe on an SSD can't guarantee that reserved or remapped blocks get erased.
>
> I think most controllers implement a secure erase feature that guarantees the data have been erased from NAND.
yes, ata secure erase, which guarantees erasure of the entire drive (excluding vendor specific areas)
Newer technology has no inherrent responsibility to live by old forensic standards of past generations. A Solid State Drive (not, Solid State Drive Drive) does not "destroy" court evidence. Firstly, show me the court record where the data was first introduced. Secondly, lookup the legal terms for destroying court records/evidence then explain to me how this scenario applies.
>A Solid State Drive (not, Solid State Drive Drive) does not "destroy" court evidence.
This struck me also. The author was writing as if the newer devices should be the same as older HDs. IMHO destroying potential court evidence is a good thing for the user. Sort of like a 5th amendment drive.
I would think that feature is an added benefit and nothing should be done about it except ensure TRIM is enabled and active. We should not be running our private lives with a goal of assisting lawsuits and prosecution, particularly actions against ourselves.
Spoliation of evidence is one of those things that courts really, really dislike. If you're found to have intentionally destroyed evidence, the court can give instructions to the jury essentially saying to assume any destroyed evidence was showing guilt.
Many businesses and professional individuals operate under a legal requirement to keep records that can't be satisfied if they are too reliant on SSD storage. Also, deleting information after the commencement of legal proceedings can be a criminal offense.
Undeleting data was never guaranteed on hard disks or SSDs and thus doesn't satisfy data retention requirements anyway. RAID isn't backup and forensic tools aren't archives.
I'd like to take advantage of the hardware AES encryption present in many SSD implementations. But I have difficulty believing it's not back-doored.
I'm not doing anything nefarious, but these days law enforcement seems ever more determined to gin up evidence and argument to suit their purpose. Also, if they can use the back door, how long until a malicious third party can?
But they could never use such a technique against a savvy tech user who knows that the erase command deletes the internal encryption key on an SSD. Using the technique in any court case would be tantamount to publicizing it and destroying the reputation of the hard drive manufacturer in question, as everyone would know a back door exists in their devices.
There would always be doubt, and some kind of parallel construction would surely be used to protect the secret technique.
I could see it being used in hundreds of cases per year for a decade before anyone could prove anything. Just like ubiquitous internet surveillance. And even when irrefutable proof surfaces, the very same people who staunchly said "that could never happen" will say "of course, how could you even be surprised?"
It's probably something like an FCC compliance requirement.
Does your hard drive provide pointless-something-random functionality that RSA/NSA thought up? No? Well it's not allowed to be sold legally inside the United States, then. Please return with a properly functioning product.
To clarify, do you mean that the OP (the Belkasoft article) is a terrible write-up of your work, or that the link you provided (of which you're a co-author?) is a terrible write-up?
Interesting that this is essentially a fight between two arms of the government: spooks, who want to delete information forever, and cops, who never want any information deleted at all.
Actually, most computer forensics has nothing to do with the government.
My girlfriend does it for a large company, and almost all of it is relatively boring stuff, like recovering email and documents when they get sued and loading it into Clearwell or EnCase for the attorneys to review.
You mean you've not rifled through her bank statements!
/sarcastic
Thanks for telling me, appreciate it. I know salaries vary so much between countries and universities, previous experience and what not, but its nice to get a ballpark figure.
I used to think this too, I was quite open about how well I was doing and how much I made.
Now not so much. I've seen what it does to people. They turn against you.
Some of my friends make half of what I do but they hang out with me and think of me as their equal. So why is it that I make so much more??? This is the question that they cannot answer for themselves and it makes them bitter. Shit, I've even had it from my own parents.
The human mind has a hard time putting itself in the position of others. Do yourself a favour, tell people but observe how their attitudes change. You aren't going follow my advice until you've seen it for yourself anyway.
"This is the question that they cannot answer for themselves and it makes them bitter."
They can answer it, they just don't want to because they know it would make them feel uncomfortable. Facing one's own flaws and self-perceived failings is a difficult task.
maybe in happy happy fun fun land where ideals are reality.
Like it or not, economic power is power in our society, and a higher salary is more inclined to having a higher economic power. It matters to you as well.
Seems to me that spooks are in the difficult position of wanting to delete their own information forever, but have nobody else's information ever deleted.
I'm the author of the first reference cited by this article, and the coiner of the term 'self-corrosion' for this phenomenon. First of all, thank you to the author of the headline article for their interesting article and for citing our research.
I'd say our main findings were a little bit different to what is described in the article, though I'd agree with most of what was written there.
We discovered that SSD drives can wipe themselves (with their own GC) even in the absence of TRIM commands and despite the use of forensic write-blockers that block both writes and trims being sent on the ATA/SATA bus. To my mind, that's what is really shocking - you get this phenomenon even when the very best forensic tools are used and even on OS's that aren't using TRIM. (My coauthor was a professional forensic investigator armed with professional equipment).
For example, imagine if you had some data on your disk that was fragmented all over the disk. If the disk has a garbage collector that wants to consolidate flash sectors so it can erase the leftover space after consolidation (e.g. to improve performance), then you're going to get deleted data being purged without any TRIM command being involved after the consolidate/erase operation.
If I remember right (it's been a few years), some firmwares also detect fast-formatting operations in OS's that don't support TRIM and use that as a clue to trigger automatic GC. That was the really stunning one for us. A fast format by the user led to the disk wiping itself just minutes later under forensic conditions.
Of course this sounds great for privacy, self-wiping and so on, but the problem is that it could look like this accidental wiping was an intentional attempt to destroy evidence (e.g. manual wipe, logic bomb or something). That's where things get tricky.
It looks like the link isn't working, here's a working link:
That paper was written for any educated person to understand, not just forensic experts, so I hope you enjoy it if you do take a look. We talk about both the technical and legal side of things in the paper.
Thanks for reading, and I'll check in on this comment later in case anyone has questions.
One other thing, if you want to try this out at home, all the code we wrote for our experiment was given at the end of the PDF paper as open source scripts, feel welcome to try it on more modern disks/OS and tell us what happens :-)
I strongly suspect what you saw in this paper was Samsung's[1] auto-trim (people often use inaccurate terms like "idle garbage collection") that reads the NTFS allocation bitmap and trims free space, a feature that was only included on a few SSDs because it is a potentially unsafe rampant layering violation. In the history of SSDs, almost none have auto-trim, so the results from this paper are highly non-representative of SSDs in general.
[1] Note that the Corsair SSD used in this paper is rebranded from Samsung.
"I strongly suspect what you saw in this paper was Samsung's[1] auto-trim "
Yes, that was the reason for picking that drive (the research budget for the project was a mere $500!), since I was pretty confident the effect might show up from it.
However, it was a lucky guess that it would show up in the presence of a write blocker and in the time frame of a forensic investigation (normally the first thing that happens is they quickly copy the disk), since those are unusual constraints.
I think I prefer to call the technology the same as the manufacturer's name: idle garbage collection - since TRIM has a clear meaning in this context, and the drive is not automatically generating TRIM commands and it doesn't behave exactly as though the O/S had issued them either (e.g. most but not all gets wiped). Hope we can agree to disagree on that one!
SSDs are very different from spinning platters. Instead of creating complicated devices that try to mimic spinning platters, why not have a different storage model entirely?
They're very different in terms of implementation, but actually have somewhat similar bottlenecks.
RAM, SSD and HDD, all have penalty for random seeking of data and benefit from pre-fetching sequential data into faster memory.
The differences we do have in SSDs have been solved by the SSDs internally remapping blocks to facilitate even wear, and by "trim", which now all major operating systems support.
If you will push forward a new FS, you'll need a better reason than the switch to SSD.
RAM, SSD and HDD, all have penalty for random seeking of data and benefit from pre-fetching sequential data into faster memory.
But the latencies are different, especially between the 1st two and the 3rd. Cost per MB are also very different between the three. Are you saying that a few orders of magnitude doesn't make a difference to determining optimal caching strategies?
The differences we do have in SSDs have been solved by the SSDs internally remapping blocks to facilitate even wear, and by "trim", which now all major operating systems support.
No, that's like saying you need a completely different set of algorithms because you changed the overall speed of your computer components up or down.
Algorithms are not designed with absolute speeds in mind. It's relative. The seek to read ratio of RAM is very roughly comparable to that of SSD and to that of HDD. It's just that SSD is faster than HDD and RAM is way faster than SSD.
You may need to tweak a few numbers, such as how aggressive the read-ahead is, and how big the buffers are, but none of this necessitates a new file system or a brand new way of seeing things.
"They're different so they need different file systems" is honestly a bit of magical thinking, hoping for discovering untapped potential, just because of some fuzzy feeling of novelty SSD invokes.
What would you change specifically in a file system optimized for SSD? Be specific, and then we'll have something to talk about.
> No, that's like saying you need a completely different set of algorithms because you changed the overall speed of your computer components up or down.
yes? Why are you presenting that as a ridiculous scenario?
The way you optimize an algorithm is going to be completely different when you can assume the HDD is as fast to access as RAM. It would also completely change the way OS's do paging.
The post you're responding to isn't saying it's impossible to get work done, they're saying it's an opportunity to optimize, which it is. And since a FS has such a dramatic effect on the performance of a system (one of the biggest draws of SSD is it's speed), the idea of designing one with SSD in mind specifically isn't that outlandish.
Now whether or not what we have is "good enough" is a different discussion, but you presenting that as something that should be ridiculed is just short sighted and ignorant.
The seek to read ratio of RAM is very roughly comparable to that of SSD and to that of HDD.
Totally untrue. SSDs seek faster than HDDs, but HDDs have higher bandwidth than SSDs. This is true almost across the board. I don't know where you got the idea that this isn't true.
And with RAM (DDR specifically) you can generally pipeline accesses, thus negating the bandwidth penalty due to bank open latency. Not true of a single HDD.
What would you change specifically in a file system optimized for SSD?
Drop the notion that you gain any benefit from spatial locality. You're then freed from any data layout constraint and can do great things like content-based addressing.
Don't believe me? These guys are building an empire on this idea: http://www.xtremio.com/ (Disclaimer: I work there.)
Algorithms are not designed with absolute speeds in mind.
Entirely incorrect in certain instances, especially with regard to implementation, and specifically the ones we are talking about. Caching schemes are specifically designed with speeds and hardware costs in mind. Also, you seem to be laboring under the misconception that I don't know what an algorithm is.
Didn't mention read disturbance. MLC and TLC flash (esp. the latter) have semi-destructive reads, so that you need to re-write a block after several thousand reads as well as on any write.
So you can't treat a drive as a ROM, even if you disabled physical writes somehow. Of course, you probably have enough read cycles available to do quite a few full scans of a drive...
"Modern SSD drives employ smart wear leveling techniques [3] that, instead of re-using existing blocks of memory, will write to a different block when data stored in a certain block is being modified."
Can this behavior be exploited to enable a hardware based file versioning system? For example, SSD explicitly exposes to OS, where new blocks are written and which blocks they are overwriting. This would allow FS to cheaply track multiple versions of files. When a portion of a file is overwritten with some new change, this version is discarded and SSD is instructed that rest of blocks that were storing changes for that version of the file are expendable as well. Depending on SSD capacity and usage, a simple algorithm of overwriting oldest block first, would provide several versions for each changed file virtually for free.
What you're describing is basically copy-on-write: https://en.wikipedia.org/wiki/Copy_on_write, which is a part of most modern filesystems (ZFS, BTRFS, etc). That makes it software-based, but the idea is the same: you can easily make snapshots, roll back to previous versions, etc.
I'm confused why they went to the trouble of building a custom FPGA setup, you can just buy a Universal chip programmer that can read the contents of flash for around $1,000. The article also doesn't address the fact that many SSD drives encrypt data before writing it to the flash which makes this approach impossible.
(1) SSDs are good for privacy for average users since they are cleaning up dirty blocks in the background.
However, IMO, privacy-conscious users who are running a daily free-space wipe, a conventional hard disk is superior because it guarantees that all dirty blocks are erased. A free-space wipe on an SSD can't guarantee that reserved or remapped blocks get erased.
(2) He says, "Somewhat counter-intuitively, information deleted from certain types of encrypted volumes (some configurations of BitLocker, TrueCrypt, PGP and other containers) may be easier to recover ... if the investigator knows either the original password or binary decryption keys for the volume".
If you delete a file in your encrypted volume (but don't do a free-space wipe inside your encrypted volume), then someone who knows your key could potentially recover that file. But that's always been true -- it's true for both SSD and conventional drives.
What I think the author is saying is that someone who use an encrypted volume doesn't benefit from the SSD's cleaning of dirty blocks in the background because the entire encrypted volume looks like it's in use to the SSD controller.
But I don't see how he concludes that it's "easier". You lose the benefit of the SSD's garbage collection, but to recover a deleted file from inside an encrypted volume (assuming you have the user's key) is neither easier nor more difficult with an SSD vs. a conventional disk.