F3 - an alternative to h2testw

I started this project when I bought a 32GB microSDHC card for my Android back in 2010, and found out that this card always fails when one fills it up. Googling about this issue, I arrived at the blogs Fight Flash Fraud and SOSFakeFlash, which recomend the software H2testw (see here or here) to test flash memories.

I downloaded H2testw and found two issues with it: (1) it is for Windows only, and (2) it is not open source. However, its author, Harald Bögeholz, was kind enough to include a text file that explains what it does, and provided the pseudo random number generator used in H2testw.

This page is about my GPLv3 implementation of H2testw, f3probe, and f3fix. My implementation of H2testw, which I've broken into two applications named f3write and f3read, runs on Linux, Macs, Windows/Cygwin, and FreeBSD. f3probe is the fastest way to identify fake drives and their real sizes. f3fix enables users to use the real capacity of fake drives without losing data as long as the memory chips don't lose the data. f3probe and f3fix currently runs only on Linux.

F3 stands for Fight Flash Fraud, or Fight Fake Flash.

Help wanted

I maintain this project in my spare time, and I no longer have been able to answer all questions and address all feedback that I receive. Nevertheless, I still think that F3's users can stop flash counterfeiters, but I need a little bit of help to keep moving forward. How can you help?

I've originally implemented F3 to address a personal need, but you have turned it into a great tool. Let's work closer to bring it to the next level!

Download and Compile

The files of the stable version of F3 are here. The command below uncompresses the files. Follow instructions in file README to compile it on your computer:

$ unzip f3-5.0.zip
  

Starting at version 2.0, F3 supports the platform Mac. Mac users may want to check out Thijs Kuipers' page for help.

Starting at version 3.0, F3 supports the platform Windows/Cygwin, and adopts H2testw's file format. People interested in exchanging files between F3 and H2testw should read the section about it to understand the caveats.

Starting at version 4.0, F3 supports the platform FreeBSD. Mac users: Version 4.0 does not compile on Macs. The issue has been fixed on version 5.0.

Starting at version 5.0, F3 includes f3probe and f3fix as experimental, and for Linux only.

How to use f3write and f3read

If you prefer watching a video than reading the text below, check out Spatry's Cup of Linux's video demo of F3 on YouTube here.

My implementation of H2testw is composed of two applications: f3write, and f3read. f3write fills a file system up with 1GB files named N.h2w, where N is a number. Whereas, f3read validates those files. If the content of all N.h2w files is valid, the drive is fine. The last file may be less than 1GB since f3write takes all available space for data. Below the result on my fake card:

$ ./f3write /media/michel/5EBD-5C80/
Free space: 28.83 GB
Creating file 1.h2w ... OK!
Creating file 2.h2w ... OK!
Creating file 3.h2w ... OK!
Creating file 4.h2w ... OK!
Creating file 5.h2w ... OK!
Creating file 6.h2w ... OK!
Creating file 7.h2w ... OK!
Creating file 8.h2w ... OK!
Creating file 9.h2w ... OK!
Creating file 10.h2w ... OK!
Creating file 11.h2w ... OK!
Creating file 12.h2w ... OK!
Creating file 13.h2w ... OK!
Creating file 14.h2w ... OK!
Creating file 15.h2w ... OK!
Creating file 16.h2w ... OK!
Creating file 17.h2w ... OK!
Creating file 18.h2w ... OK!
Creating file 19.h2w ... OK!
Creating file 20.h2w ... OK!
Creating file 21.h2w ... OK!
Creating file 22.h2w ... OK!
Creating file 23.h2w ... OK!
Creating file 24.h2w ... OK!
Creating file 25.h2w ... OK!
Creating file 26.h2w ... OK!
Creating file 27.h2w ... OK!
Creating file 28.h2w ... OK!
Creating file 29.h2w ... OK!
Free space: 0.00 Byte
Average Writing speed: 2.60 MB/s

$ ./f3read /media/michel/5EBD-5C80/
                  SECTORS      ok/corrupted/changed/overwritten
Validating file 1.h2w ...       0/  2097152/      0/      0
Validating file 2.h2w ...       0/  2097152/      0/      0
Validating file 3.h2w ...       0/  2097152/      0/      0
Validating file 4.h2w ...       0/  2097152/      0/      0
Validating file 5.h2w ...       0/  2097152/      0/      0
Validating file 6.h2w ...       0/  2097152/      0/      0
Validating file 7.h2w ...       0/  2097152/      0/      0
Validating file 8.h2w ...       0/  2097152/      0/      0
Validating file 9.h2w ...       0/  2097152/      0/      0
Validating file 10.h2w ...       0/  2097152/      0/      0
Validating file 11.h2w ...       0/  2097152/      0/      0
Validating file 12.h2w ...       0/  2097152/      0/      0
Validating file 13.h2w ...       0/  2097152/      0/      0
Validating file 14.h2w ...       0/  2097152/      0/      0
Validating file 15.h2w ...       0/  2097152/      0/      0
Validating file 16.h2w ...       0/  2097152/      0/      0
Validating file 17.h2w ...       0/  2097152/      0/      0
Validating file 18.h2w ...       0/  2097152/      0/      0
Validating file 19.h2w ...       0/  2097152/      0/      0
Validating file 20.h2w ...       0/  2097152/      0/      0
Validating file 21.h2w ...       0/  2097152/      0/      0
Validating file 22.h2w ...       0/  2097152/      0/      0
Validating file 23.h2w ...       0/  2097152/      0/      0
Validating file 24.h2w ... 1916384/   180768/      0/      0
Validating file 25.h2w ...  186816/  1910336/      0/      0
Validating file 26.h2w ...       0/  2097152/      0/      0
Validating file 27.h2w ...       0/  2097152/      0/      0
Validating file 28.h2w ...       0/  2097152/      0/      0
Validating file 29.h2w ...   28224/  1705280/      0/      0

  Data OK: 1.02 GB (2131424 sectors)
Data LOST: 27.81 GB (58322336 sectors)
	       Corrupted: 27.81 GB (58322336 sectors)
	Slightly changed: 0.00 Byte (0 sectors)
	     Overwritten: 0.00 Byte (0 sectors)
Average Reading speed: 9.54 MB/s
  

This report shows that my flash card is pretty much garbage since it can only hold 1.02GB. f3write only writes to free space, and will not overwrite existing files as long as they aren't named N.h2w. However, as the previous report also shows, files from 1.h2w to 23.h2w were written before 24.h2w and yet had all their content destroyed. Therefore, it is not wise to test nonempty cards because if the card has a problem, it may erase the old files.

When f3read reads a sector (i.e. 512 bytes, the unit of communication with the card), f3read can check if the sector was correctly written by f3write, and figure out in which file the sector should be and in which position in that file the sector should be. Thus, if a sector is well formed, or with a few bits flipped, but read in an unexpected position, f3read counts it as overwritten. Slightly changed sectors, are sectors at right position with a fews bits flipped.

Notice that f3write doesn't overwrite sectors by itself, it's done by the memory card's controller as a way to difficult a user to uncover its fault. So, the way sectors are overwritten is totally dependent on the controller; from the point of view of a file system, what f3read sees, the way the controller wraps around seems often contrived, but from the memory chip's view, what the controller sees, it is just a truncation of addresses.

The last lines of the output of f3write and f3read provide good estimates of the writing and reading speeds of the tested card. This information can be used to check if the claimed class of the card is correct. Check this link out for more information about classes. Note that the speeds provided by F3 are estimates, don't take them as perfect since they suffer influence even from other processes in your machine! Also, be aware that your card reader and USB port can limit the throughput of the drive.

Later I bought a second card that works just fine; I got the following output running F3 on it:

$ ./f3write /media/michel/6135-3363/
Free space: 29.71 GB
Creating file 1.h2w ... OK!
Creating file 2.h2w ... OK!
Creating file 3.h2w ... OK!
Creating file 4.h2w ... OK!
Creating file 5.h2w ... OK!
Creating file 6.h2w ... OK!
Creating file 7.h2w ... OK!
Creating file 8.h2w ... OK!
Creating file 9.h2w ... OK!
Creating file 10.h2w ... OK!
Creating file 11.h2w ... OK!
Creating file 12.h2w ... OK!
Creating file 13.h2w ... OK!
Creating file 14.h2w ... OK!
Creating file 15.h2w ... OK!
Creating file 16.h2w ... OK!
Creating file 17.h2w ... OK!
Creating file 18.h2w ... OK!
Creating file 19.h2w ... OK!
Creating file 20.h2w ... OK!
Creating file 21.h2w ... OK!
Creating file 22.h2w ... OK!
Creating file 23.h2w ... OK!
Creating file 24.h2w ... OK!
Creating file 25.h2w ... OK!
Creating file 26.h2w ... OK!
Creating file 27.h2w ... OK!
Creating file 28.h2w ... OK!
Creating file 29.h2w ... OK!
Creating file 30.h2w ... OK!
Free space: 0.00 Byte
Average Writing speed: 4.90 MB/s

$ ./f3read /media/michel/6135-3363/
                  SECTORS      ok/corrupted/changed/overwritten
Validating file 1.h2w ... 2097152/        0/      0/      0
Validating file 2.h2w ... 2097152/        0/      0/      0
Validating file 3.h2w ... 2097152/        0/      0/      0
Validating file 4.h2w ... 2097152/        0/      0/      0
Validating file 5.h2w ... 2097152/        0/      0/      0
Validating file 6.h2w ... 2097152/        0/      0/      0
Validating file 7.h2w ... 2097152/        0/      0/      0
Validating file 8.h2w ... 2097152/        0/      0/      0
Validating file 9.h2w ... 2097152/        0/      0/      0
Validating file 10.h2w ... 2097152/        0/      0/      0
Validating file 11.h2w ... 2097152/        0/      0/      0
Validating file 12.h2w ... 2097152/        0/      0/      0
Validating file 13.h2w ... 2097152/        0/      0/      0
Validating file 14.h2w ... 2097152/        0/      0/      0
Validating file 15.h2w ... 2097152/        0/      0/      0
Validating file 16.h2w ... 2097152/        0/      0/      0
Validating file 17.h2w ... 2097152/        0/      0/      0
Validating file 18.h2w ... 2097152/        0/      0/      0
Validating file 19.h2w ... 2097152/        0/      0/      0
Validating file 20.h2w ... 2097152/        0/      0/      0
Validating file 21.h2w ... 2097152/        0/      0/      0
Validating file 22.h2w ... 2097152/        0/      0/      0
Validating file 23.h2w ... 2097152/        0/      0/      0
Validating file 24.h2w ... 2097152/        0/      0/      0
Validating file 25.h2w ... 2097152/        0/      0/      0
Validating file 26.h2w ... 2097152/        0/      0/      0
Validating file 27.h2w ... 2097152/        0/      0/      0
Validating file 28.h2w ... 2097152/        0/      0/      0
Validating file 29.h2w ... 2097152/        0/      0/      0
Validating file 30.h2w ... 1491904/        0/      0/      0

  Data OK: 29.71 GB (62309312 sectors)
Data LOST: 0.00 Byte (0 sectors)
	       Corrupted: 0.00 Byte (0 sectors)
	Slightly changed: 0.00 Byte (0 sectors)
	     Overwritten: 0.00 Byte (0 sectors)
Average Reading speed: 9.42 MB/s
  

Since f3write and f3read are independent, f3read can be used as many times as one wants, although f3write is needed only once. This allows one to easily repeat a test of a card as long as the N.h2w files are still available.

As a final remark, if you want to run f3write and f3read with a single command, check out the shell script log-f3wr. This script runs f3write and f3read, and records their output into a log file. Use example: log-f3wr log-filename /media/michel/5EBD-5C80/

Users' notes

Randy Champoux has brought to my attention that f3read could eventually read data from the system cache instead of from the flash card. Since version 2.0, F3 eliminates this possibility as long as the kernel honors the system call posix_fadvise(2) with advice POSIX_FADV_DONTNEED. Linux has and honor posix_fadvise(2)/POSIX_FADV_DONTNEED, the Mac does not have the system call, and I don't know if Windows/Cygwin, or FreeBSD honors it. In doubt about this issue, just disconnect and connect back the device after f3write runs and before calling f3read.

Notice that the issue pointed by Randy Champoux is entirely an OS matter, that is, it doesn't change if the drive being tested is fake or not. In 2014, I've run into a "smarter" fake card that tries hard to behave like a good one using its internal cache to fool the test. In practice, these newer cards can only mislead f3read with a limited number of blocks, but those looking for a precise, repeatable report should follow the advice of disconnecting and connecting back the device before f3read runs. Consider the real example of a fake drive that presents this behavior. The drive announces a size of 128GB but its real capacity is less than 8GB:

$ ./f3write --end-at=16 /media/michel/DISK_IMG/ && ./f3read /media/michel/DISK_IMG/
Free space: 124.97 GB
Creating file 1.h2w ... OK!                               
Creating file 2.h2w ... OK!                           
Creating file 3.h2w ... OK!                           
Creating file 4.h2w ... OK!                           
Creating file 5.h2w ... OK!                           
Creating file 6.h2w ... OK!                           
Creating file 7.h2w ... OK!                         
Creating file 8.h2w ... OK!                         
Creating file 9.h2w ... OK!                         
Creating file 10.h2w ... OK!                         
Creating file 11.h2w ... OK!                         
Creating file 12.h2w ... OK!                         
Creating file 13.h2w ... OK!                         
Creating file 14.h2w ... OK!                         
Creating file 15.h2w ... OK!                         
Creating file 16.h2w ... OK!                        
Free space: 108.97 GB
Average writing speed: 2.87 MB/s
                  SECTORS      ok/corrupted/changed/overwritten
Validating file 1.h2w ... 2097152/        0/      0/      0
Validating file 2.h2w ... 2097152/        0/      0/      0
Validating file 3.h2w ... 2097152/        0/      0/      0
Validating file 4.h2w ... 2097152/        0/      0/      0
Validating file 5.h2w ... 2097152/        0/      0/      0
Validating file 6.h2w ... 2097152/        0/      0/      0
Validating file 7.h2w ... 2097152/        0/      0/      0
Validating file 8.h2w ...  266176/  1830976/      0/      0
Validating file 9.h2w ...       0/  2097152/      0/      0
Validating file 10.h2w ...       0/  2097152/      0/      0
Validating file 11.h2w ...       0/  2097152/      0/      0
Validating file 12.h2w ...       0/  2097152/      0/      0
Validating file 13.h2w ...       0/  2097152/      0/      0
Validating file 14.h2w ...       0/  2097152/      0/      0
Validating file 15.h2w ...       0/  2097152/      0/      0
Validating file 16.h2w ...  523920/  1573232/      0/      0

  Data OK: 7.38 GB (15470160 sectors)
Data LOST: 8.62 GB (18084272 sectors)
	       Corrupted: 8.62 GB (18084272 sectors)
	Slightly changed: 0.00 Byte (0 sectors)
	     Overwritten: 0.00 Byte (0 sectors)
Average reading speed: 12.73 MB/s
  

After disconnecting the drive and connecting it back, f3read produced the following output:

$ ./f3read /media/michel/DISK_IMG/
                  SECTORS      ok/corrupted/changed/overwritten
Validating file 1.h2w ... 2097152/        0/      0/      0
Validating file 2.h2w ... 2097152/        0/      0/      0
Validating file 3.h2w ... 2097152/        0/      0/      0
Validating file 4.h2w ... 2097152/        0/      0/      0
Validating file 5.h2w ... 2097152/        0/      0/      0
Validating file 6.h2w ... 2097152/        0/      0/      0
Validating file 7.h2w ... 2097152/        0/      0/      0
Validating file 8.h2w ...  266176/  1830976/      0/      0
Validating file 9.h2w ...       0/  2097152/      0/      0
Validating file 10.h2w ...       0/  2097152/      0/      0
Validating file 11.h2w ...       0/  2097152/      0/      0
Validating file 12.h2w ...       0/  2097152/      0/      0
Validating file 13.h2w ...       0/  2097152/      0/      0
Validating file 14.h2w ...       0/  2097152/      0/      0
Validating file 15.h2w ...       0/  2097152/      0/      0
Validating file 16.h2w ...       0/  2097152/      0/      0

  Data OK: 7.13 GB (14946240 sectors)
Data LOST: 8.87 GB (18608192 sectors)
	       Corrupted: 8.87 GB (18608192 sectors)
	Slightly changed: 0.00 Byte (0 sectors)
	     Overwritten: 0.00 Byte (0 sectors)
Average reading speed: 12.50 MB/s
  

Notice that file 16.h2w, that last file f3write wrote, has no longer good sectors. What happened is that the last sectors of 16.h2w were in the internal cache of the drive when f3read ran right after f3write, but were not there after the forced reset. The internal cache will fool any test that doesn't write beyond the real capacity of the drive plus the size of the internal cache, and does not hard reset the drive. One can estimate the size of this cache as follows: 523920 * 512B ~ 256MB.

Tom Metro once ran f3write on a 16GB flash drive formated with ext2 file system, and obtained puzzling free space at the end of f3write's output:

% ./f3write /media/Kodi/
Free space: 14.50 GB
Creating file 1.h2w ... OK!
Creating file 2.h2w ... OK!
Creating file 3.h2w ... OK!
Creating file 4.h2w ... OK!
Creating file 5.h2w ... OK!
Creating file 6.h2w ... OK!
Creating file 7.h2w ... OK!
Creating file 8.h2w ... OK!
Creating file 9.h2w ... OK!
Creating file 10.h2w ... OK!
Creating file 11.h2w ... OK!
Creating file 12.h2w ... OK!
Creating file 13.h2w ... OK!
Creating file 14.h2w ... OK!
Free space: 755.80 MB
Average writing speed: 13.77 MB/s
  

This happened because ext2 and some other file systems reserve space for special purposes. So they don't allow f3write to use that reserved space. It's mostly safe to ignore that free space. If one wants to use all space possible, there're two options: (1) using a file system that doesn't reserve space (e.g FAT), or (2) reducing the reserved space. How to go for the second option depends on the used file system. The page explains how to reduce the reserved space on ext2, ext3, and ext4 file systems.

Elliot Macneille has ran into an application that reports the size of one of its good flash cards as 15.71GB, whereas f3read only finds 14.63GB. Details on how space is accounted varies with operating system, applications, file system used to format the drive, etc. However, there is a common source for this problem that often explains most of the difference: part of the computer industry (including F3) takes 1GB as being 2^30 bytes, whereas the rest of the industry assumes that 1GB is equal to 10^9 bytes. Some people use GiB for the first definition, but its use is not universal, and some users even get confused when they see this unit. With this information in mind, the mistery is easily solved: 14.63GiB * 2^30 / 10^9 = 15.71GB.

When Art Gibbens tested a flash card hosted in a camera connected to his Linux box, at some point F3 didn't show progress, and could not be killed. After a reboot, the card was read only. Using an adapter to connect his card directly to his machine, he recreated the partition of the card adapting the steps described in section "How to 'fix" a fake card', and successfully ran F3 with the card in the adpater. Thus, Art's experience is a good warning if you're testing your card in a device other than an adapter. Please, don't take it as a bug of F3. I'm aware of only two things that can make a process "survive" a kill signal: hardware failures, and/or bugs in the kernel. F3 doesn't run in kernel mode, so Art's camera is likely the root of the problem.

Darrell Johnson has reported that a flash card he got stopped working after he filled it up. This could be that the only memory chip the card had died, but it is just speculation since Darrell was not able to obtain more information. The important message here is that if you test your card with F3, or just copy files into it, and it stops working, it's not your fault because writing files to a card shouldn't damage it if it is a legitimate card!

Username Kris, here, asked what's the difference between "dosfsck -vtr /dev/sda1" and F3. dosfsck(8) makes two assumptions that F3 does not: (1) one needs write access to the device being tested, not the file system in it; (2) hardware may fail, but it won't lie. The first assumption implies that one likely needs root's rights to run dosfsck, what is just a small incovenience for simple uses. The second assumption is troublesome because a fake card may be able to persuade dosfsck(8) to report it's fine, or not report the whole problem, or give users the illusion the memory card was fixed when it wasn't. I singled dosfsck(8) out because of the question about it, but those two assumptions are true for fsck software for other file systems and badblocks(8) as well.

Mac user Athanasios Tourtouras noticed that Spotlight of OS X, which runs in the background, also indexes the content of removable drives. Although Spotlight does not interferes with f3write/f3read, its indexing takes away around 2MB/s of bandwidth, so f3write/f3read will run slower as well as their speed measurements will underestimate the real capacity of the drive. Not to mention that you likely don't want to index test files. You can disable the indexing of removable drives including the flash drive to Spotlight's exclude list by going to System Preferences / Spotlight / Privacy.

On the compatibility with H2testw's file format

Starting at version 3.0, F3 generates files following H2testw's file format. This feature should help users that use both applications and/or interact with users that use the other application. However, there are two caveats explained in this section that users should be aware.

Verifying files created by H2testw with F3read. The caveat here is that H2testw only writes files whose size is a multiple of 1MB, whereas F3write fills up the memory card. Thus, verifying files created by H2testw with F3read works, but, likely, will not test approximately 1MB of space that H2testw does not write.

Verifying files created by f3write with H2testw. The caveat here is that H2testw requires that all files have sizes that are multiple of 1MB. When it is not the case for a single file, H2testw rejects all files, and issues the message "Please delete files *.h2w." This problem often comes up with the last file f3write generates to fill up the space available in the memory card being tested. The solution is to truncate the size of the last file f3write generates to the closest multiple of 1MB. Assuming the last file is 30.h2w, the following command does exactly that:

$ truncate --size=/1M /media/michel/6135-3363/30.h2w
  

If you want to exchange files with H2testw users often, check out the shell script f3write.h2w. This script calls truncate after f3write runs successfully.

f3probe - the fastest flash test

H2testw's algorithm has been the gold standard for detecting fake flash (see here and here) because it is robust against all counterfeits. However, as drives' capacity grows, the time to test these newer drives becomes so painful that one rarely runs H2testw's algorithm on a whole drive, but only a fraction of it. See question "Why test only 25% or 32GB?" on this FAQ for a defense of this approach.

The problem with this approach is that drives are still getting bigger, and conterfeiters may, in the future, be able to profit with fake drives whose real capacity are large enough to fool these partial tests. This problem is not new. For example, Steve Si implemented FakeFlashTest.exe, which has successfully reduced the amount of time to test drives, and is still able to give a good estimate of the real size of fake drives. Yet, FakeFlashTest.exe's algorithm is not a definitive answer to the problem because FakeFlashTest.exe's algorithm still needs to write to at least all good memory of tested drives.

f3probe writes at most ceiling(log2(announced_blocks)) * 2^10 / 10 + 1 blocks. The derivation of this formula is available in the comments of function probe_device_max_blocks in file libprobe.c. In order to put this result in perspective, consider a 128GB fake drive tested below with f3probe whose announced_blocks is 32768000, and real size 1876307 blocks: ceiling(log2(32768000)) * 2^10 / 10 + 1 = 2561 blocks. That is, f3probe writes no more than 2561 / 1876307 ~ 0.14% of all that FakeFlashTest.exe writes. Even if FakeFlashTest.exe wrote only 1% of the real size of the drive, f3probe would still write only 14% of what FakeFlashTest.exe would write under this hypothetical, two-order improvement! Moreover, f3probe provides the exact geometry of the drive, what allows one to "fix" the drive using the highest capacity possible.

This breakthrough against counterfeiters was only possible because f3probe's algorithm assumes a tight model of how fake drives work. In spite of the fact that I have not found a drive, fake or not, that confuses f3probe, I've marked f3probe as experimental for now because this model has not been battle proven. Although there is a chance of finding out that the model is incomplete, there is also a chance that the model can be simplified if one can be sure that not all types of fake flash the model predicts really exist; the latter chance holds a promise of even higher testing speeds. Of course, efficient flash tests like the one implemented in f3probe may slow down as fake chips become "smarter". For now, though, f3probe gives us the upper hand over counterfeiters.

Finally, thanks to f3probe being free software, and once f3probe is battle proven, f3probe could be embedded on smartphones, cameras, MP3 players, and other devices to stop once and for all the proliferation of fake flash.

How to use f3probe

Different from f3write/f3read that works on the file system of the drive, f3probe works directly over the block device that controls the drive. In practice, this means three requirements. First, one has to have root access (i.e. superuser account) on the machine that will run the test. Second, the user must know how to find the block device of the drive. Third, you must be careful on the previous requirement to avoid messing your machine up. If you don't have root access, you can't use f3probe; use f3write/f3read in this case. The use example below helps with the second requirement, but don't forget that you are the one responsable for doing it right!

The command lsblk(8) is handy to find the block device of the drive. In the example below, which I got running lsblk on my laptop, an experient user can quickly identify that my flash drive that is mounted at "/media/michel/A902-D705" is block device "/dev/sdb". If you don't have much experience, you may want to run lsblk before connecting the drive to your computer, and to run lsblk again after connecting the drive to easily identify what was added to the output of lsblk. Checking out the content of folder "/media/michel/A902-D705" to confirm that it's the correct drive is a good idea as well. The block device "sdb" is the disk (see column "TYPE"), and "sdb1" is the first and only partition of my flash drive; your drive may have none or more partitions. You want to choose the drive, not a partition.

$ lsblk 
NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda      8:0    0 232.9G  0 disk 
+-sda1   8:1    0   218G  0 part /
+-sda2   8:2    0     1K  0 part 
+-sda5   8:5    0    15G  0 part [SWAP]
sdb      8:16   1   3.8G  0 disk 
+-sdb1   8:17   1   3.8G  0 part /media/michel/A902-D705
sr0     11:0    1  1024M  0 rom  

If you get confused between "sdb" and "sdb1", don't worry, f3probe will report the mistake and point out the proper one. However, I cannot emphasize it enough, you MUST identify the correct drive. If I had chosen "sda", f3probe may have messied my computer. Once the device is chosen, just prefix it with "/dev/" to obtain its full name.

Once you have carefully identified the device, you run f3probe like in the example below (please use the correct device!):

$ sudo ./f3probe --time-ops /dev/sdb
F3 probe 5.0
Copyright (C) 2010 Digirati Internet LTDA.
This is free software; see the source for copying conditions.

Please unplug and plug back the USB drive. Waiting... Thanks

Please unplug and plug back the USB drive. Waiting... Thanks

Please unplug and plug back the USB drive. Waiting... Thanks

Please unplug and plug back the USB drive. Waiting... Thanks

Please unplug and plug back the USB drive. Waiting... Thanks

Please unplug and plug back the USB drive. Waiting... Thanks

Please unplug and plug back the USB drive. Waiting... Thanks

Please unplug and plug back the USB drive. Waiting... Thanks

Please unplug and plug back the USB drive. Waiting... Thanks

Please unplug and plug back the USB drive. Waiting... Thanks

CAUTION		CAUTION		CAUTION
No more resets are needed, so do not unplug the drive
Probe finished, recovering blocks... Done

Bad news: The device `/dev/sdb' is a counterfeit of type limbo

You can "fix" this device using the following command:
f3fix --last-sec=15010455 /dev/sdb

Device geometry:
	     *Real* size: 7.16 GB (1876307 blocks)
	  Announced size: 125.00 GB (32768000 blocks)
	          Module: 128.00 GB (2^37 Bytes)
	      Block size: 4.00 KB (2^12 Bytes)
	Last good sector: 15010455

Probe time: 71.80 seconds
Probe read op: count=156, total time=2.57s, avg op time=16.51ms
Probe write op: count=107, total time=20.57s, avg op time=192.26ms
Probe reset op: count=10, total time=48.65s, avg op time=4865.23ms

There is a lot in the previous example. First, it took only 71.80s for f3probe to identify that this 128GB drive had only 7.16GB. Second, I used command sudo(8) to run f3probe as root. Third, I used option "--time-ops" to add the last three lines of the output; these lines show the time taken to read, write, and reset the drive during the test. The rest of this section covers other aspects of this output in greater detail.

A significant part of the previous output is filled with lines such "Please unplug and plug back the USB drive. Waiting... Thanks", and, in fact, this is an essential topic to understand f3probe. As discussed in section Users' notes of f3write/f3read, fake drives use their cache to hide themselves. Resetting the drive flushes this cache, and allows f3probe to really read from the effective storage the drive has. The number of resets f3probe issues varies not only with the announced size of the drive, but also with the operation times that f3probe measures during the process; the goal is to tune the algorithm on the fly to minimize probe time. Thus, don't worry if you see a different number of resets for your drive, or for f3probe executions on the same drive.

The default reset method of f3probe is to ask the user to manually unplug and plug back the USB drive. This method is the default because it is the only method that works 100% of the time in my tests. Even if one doesn't have to test a large number of drives, one can still appreciate having an automated reset method since it takes a significant portion of the probe time. In the previous example, it took 48.65s/71.80s ~ 68% of the probe time. f3probe supports an automated reset method that is evoked with option "--reset-type=1". This method is the same one present in usbreset.c available on the this page. Unfortunately, this automated method does not work with all fake drives, what means f3probe may report a fake drive as the real thing when this reset method is used. If f3probe reports a card is fake using this reset method, or any other reset method, the card is fake. I expect that new reset methods will be found, but these two are the only methods currently available.

If you're looking for an interesting, small hardware project, you can implement a gadget that connects to a USB port of your computer, has another connector to receive a USB drive, and resets the drive turning the power off and on with a command issued in software. All USB frames, but the special reset command, are transparently exchanged between the drive and the computer. If you create this device, please send me an e-mail; I want to buy one. Also, consider crowdfunding it.

Notice the "CAUTION"s after the resets. They are there to let you know that no more resets will be issued, so you can leave the drive alone and connected to your computer. I added this warning because I accidentally unplugged the drive after the last reset, and this messes f3probe.

The line "Probe finished, recovering blocks... Done" in the previous output lets the user know that f3probe has recovered all blocks in the drive to their original state. While this feature is very convenient, don't rely too much on it; if f3probe crashes, this feature won't work. In case you are running f3probe on a memory-constrainted computer (e.g. an old Raspberry Pi board), you can keep this feature, and still reduce the amount of memory needed using option "--min-memory". If you don't have memory to test a large drive even using option "--min-memory", disable this feature completely with option "--destructive". When option "--destructive" is used, f3probe will not save the content of the modified blocks, so you may also want to use this option to save time when you don't care about the content on the drive.

The line "Bad news: The device `/dev/sdb' is a counterfeit of type limbo" summarizes the results presented below this line. All of my flash drives are either of the type good, bad (seriously failing), or limbo (a type of fake drives). But the model f3probe uses has two other fake types. If you ever find other type, please let me know, and consider donating the drive to my collection. If nobody reports other types up to one year after the public release of f3probe, I'll drop those other types from the model to speed up f3probe's algorithm.

The probe time of 71.80 seconds includes the time to run the probe algorithm, take measurements, and the time to perform all operations on the drive. But it doesn't include the time to recover the saved blocks (if this feature is enabled). Therefore, the test above took rougthly another 20.57s (i.e. total write time) to write all blocks back to the drive. As some will notice, the time to perform all operations on the drive is what dominates the probe time: 2.57s + 20.57s + 48.65s = 71.79s.

The next example gives you the chance to practice reading f3probe's outputs:

$ sudo ./f3probe --time-ops /dev/sdb
F3 probe 5.0
Copyright (C) 2010 Digirati Internet LTDA.
This is free software; see the source for copying conditions.

Please unplug and plug back the USB drive. Waiting... Thanks

Please unplug and plug back the USB drive. Waiting... Thanks

Please unplug and plug back the USB drive. Waiting... Thanks

Please unplug and plug back the USB drive. Waiting... Thanks

Please unplug and plug back the USB drive. Waiting... Thanks

Please unplug and plug back the USB drive. Waiting... Thanks

Please unplug and plug back the USB drive. Waiting... Thanks

Please unplug and plug back the USB drive. Waiting... Thanks

Please unplug and plug back the USB drive. Waiting... Thanks

Please unplug and plug back the USB drive. Waiting... Thanks

CAUTION		CAUTION		CAUTION
No more resets are needed, so do not unplug the drive
Probe finished, recovering blocks... Done

Good news: The device `/dev/sdb' is the real thing

Device geometry:
	     *Real* size: 3.77 GB (989184 blocks)
	  Announced size: 3.77 GB (989184 blocks)
	          Module: 4.00 GB (2^32 Bytes)
	      Block size: 4.00 KB (2^12 Bytes)
	Last good sector: 7913471

Probe time: 68.15 seconds
Probe read op: count=58, total time=0.21s, avg op time=3.64ms
Probe write op: count=36, total time=14.39s, avg op time=399.66ms
Probe reset op: count=10, total time=53.55s, avg op time=5355.45ms

This second drive is a good one; it has all blocks necessary to hold its announced size of 3.77GB, what is roughly 4GB.

The next section shows how to fix the 128GB drive using f3fix as suggested by f3probe.

Users' notes

Dingyuan Wang has noticed that drives powered by batteries will not reset their caches when they are unconnected from a USB port since the batteries maintains their cache; see discussion here. If you find yourself dealing with this case, you are left with only two options. First, you can try the reset parameter "--reset-type=1". If it works, problem solved. If not, your second option is to fall back to f3write/f3read.

How to "fix" a fake card

You should not easily settle down for a fake card, fight back and get your money back! Doing so will help you and others. If you are still reading this section, you already realized that you own a fake card, and would like to be able to use it without losing data.

As shown in the previous section, my 128GB fake card can only hold 7.16GB. Moreover, f3probe suggested how I can use f3fix to fix my drive. The execution of f3fix went as follows:

$ sudo ./f3fix --last-sec=15010455 /dev/sdb
F3 fix 5.0
Copyright (C) 2010 Digirati Internet LTDA.
This is free software; see the source for copying conditions.

Drive `/dev/sdb' was successfully fixed

If f3fix reports that you need to force the kernel to reload the new partition table, just unplug and plug the drive back. Once the new partition is available, format it:

$ sudo mkfs.vfat /dev/sdb1
mkfs.fat 3.0.26 (2014-03-07)

At this point your card should be working fine, just mount the new partition to access it. However, before using the drive, test all its blocks with f3write/f3read. The test of my card went as follows:

$ ./f3write /media/michel/4A51-B7F5
Free space: 7.14 GB
Creating file 1.h2w ... OK!                         
Creating file 2.h2w ... OK!                         
Creating file 3.h2w ... OK!                         
Creating file 4.h2w ... OK!                         
Creating file 5.h2w ... OK!                         
Creating file 6.h2w ... OK!                         
Creating file 7.h2w ... OK!                        
Creating file 8.h2w ... OK!                       
Free space: 0.00 Byte
Average writing speed: 2.90 MB/s

$ ./f3read /media/michel/4A51-B7F5
                  SECTORS      ok/corrupted/changed/overwritten
Validating file 1.h2w ... 2097152/        0/      0/      0
Validating file 2.h2w ... 2097152/        0/      0/      0
Validating file 3.h2w ... 2097152/        0/      0/      0
Validating file 4.h2w ... 2097152/        0/      0/      0
Validating file 5.h2w ... 2097152/        0/      0/      0
Validating file 6.h2w ... 2097152/        0/      0/      0
Validating file 7.h2w ... 2097152/        0/      0/      0
Validating file 8.h2w ...  299040/        0/      0/      0

  Data OK: 7.14 GB (14979104 sectors)
Data LOST: 0.00 Byte (0 sectors)
	       Corrupted: 0.00 Byte (0 sectors)
	Slightly changed: 0.00 Byte (0 sectors)
	     Overwritten: 0.00 Byte (0 sectors)
Average reading speed: 16.73 MB/s

As reported by f3write/f3read above, the real memory of my fake drive is in good shape. But it may not be the case for yours. For example, the following is f3read's output for a 16GB drive with real size of 7GB fixed as described in this section:

                  SECTORS      ok/corrupted/changed/overwritten
Validating file 1.h2w ... 2097152/        0/      0/      0
Validating file 2.h2w ... 2097152/        0/      0/      0
Validating file 3.h2w ... 2097088/       64/      0/      0
Validating file 4.h2w ... 2097152/        0/      0/      0
Validating file 5.h2w ... 2088960/     8192/      0/      0
Validating file 6.h2w ... 2097152/        0/      0/      0
Validating file 7.h2w ... 2037632/        0/      0/      0

  Data OK: 6.97 GB (14612288 sectors)
Data LOST: 4.03 MB (8256 sectors)
	       Corrupted: 4.03 MB (8256 sectors)
	Slightly changed: 0.00 Byte (0 sectors)
	     Overwritten: 0.00 Byte (0 sectors)
Average reading speed: 946.46 KB/s

If you get some sectors corrupted, repeat the f3write/f3read test. Some drives recover from these failures on a second full write cycle. However, if the corrupeted sectors persist, the drive is junk because not only is it a fake drive, but its real memory is already failing.

Good luck!

Repository

The Git repository is kept here. An archive of this page until version 4.0 is available here.

Author

Michel Machado. E-mail me at michel at digirati dot com dot br.

Please try to figure out the solution for your question on your own, or ask for help from a nearby friend before you e-mail me. As a new parent, my spare time has sensibly shrunk. For reporting bugs, requesting features, and making suggestions, open an issue on GitHub here to allow other users to help me.

Copyright and License

F3 is licensed under the GNU General Public License (GPL) version 3.

Copyright (c) 2010 Digirati.