F3 - an alternative to h2testw

This project started when I bought a 32GB microSDHC card for my Android back in 2010, and found out that this card always fails when one fills it up. Googling about this issue, I arrived at the blogs Fight Flash Fraud and SOSFakeFlash, which recomend the software H2testw (see here or here) to test flash memories.

I downloaded H2testw and found two issues with it: (1) it is for Windows only, and (2) it is not open source. However, its author, Harald Bögeholz, was kind enough to include a text file that explains what it does, and provided the pseudo random number generator used in H2testw. This page is about my GPLv3 implementation of H2testw, which run on Linux, Macs, and Windows/Cygwin. I call my implementation F3, what is short for Fight Flash Fraud, or Fight Fake Flash.

Help wanted

I maintain this project in my spare time, and, as a new parent, I haven't been able to answer all questions and address all feedback that I receive. Nevertheless, I still think that F3's users can stop flash counterfeiters, but I need a little bit of help to keep moving forward. How can you help?

I've originally implemented F3 to address a personal need, but you have turned it into a great tool. Let's work closer to bring it to the next level!

Download and Compile

The files of the stable version of F3 are here. The command below uncompresses the files. Follow instructions in file README to compile it on your computer:

$ unzip f3-3.0.zip
  

Starting at version 2.0, F3 supports the platform Mac. Mac users may want to check out Thijs Kuipers' page for help.

Starting at version 3.0, F3 supports the platform Windows/Cygwin, and adopts H2testw's file format. People interested in exchanging files between F3 and H2testw should read the section about it to understand the caveats.

How to use F3

If you prefer watching a video than reading the text below, check out Spatry's Cup of Linux's video demo of F3 on YouTube here.

F3 is composed of two binaries: f3write, and f3read. f3write fills a file system up with 1GB files named N.h2w, where N is a number (i.e. /[0-9]+/). Whereas, f3read validates those files. If the content of all N.h2w files is valid, the memory is fine. The last file may be less than 1GB since f3write takes all available space for data. Below the result on my fake card:

$ ./f3write /media/5EBD-5C80/
Free space: 28.83 GB
Creating file 1.h2w ... OK!
Creating file 2.h2w ... OK!
Creating file 3.h2w ... OK!
Creating file 4.h2w ... OK!
Creating file 5.h2w ... OK!
Creating file 6.h2w ... OK!
Creating file 7.h2w ... OK!
Creating file 8.h2w ... OK!
Creating file 9.h2w ... OK!
Creating file 10.h2w ... OK!
Creating file 11.h2w ... OK!
Creating file 12.h2w ... OK!
Creating file 13.h2w ... OK!
Creating file 14.h2w ... OK!
Creating file 15.h2w ... OK!
Creating file 16.h2w ... OK!
Creating file 17.h2w ... OK!
Creating file 18.h2w ... OK!
Creating file 19.h2w ... OK!
Creating file 20.h2w ... OK!
Creating file 21.h2w ... OK!
Creating file 22.h2w ... OK!
Creating file 23.h2w ... OK!
Creating file 24.h2w ... OK!
Creating file 25.h2w ... OK!
Creating file 26.h2w ... OK!
Creating file 27.h2w ... OK!
Creating file 28.h2w ... OK!
Creating file 29.h2w ... OK!
Free space: 0.00 Byte
Average Writing speed: 2.60 MB/s

$ ./f3read /media/5EBD-5C80/
                  SECTORS      ok/corrupted/changed/overwritten
Validating file 1.h2w ...       0/  2097152/      0/      0
Validating file 2.h2w ...       0/  2097152/      0/      0
Validating file 3.h2w ...       0/  2097152/      0/      0
Validating file 4.h2w ...       0/  2097152/      0/      0
Validating file 5.h2w ...       0/  2097152/      0/      0
Validating file 6.h2w ...       0/  2097152/      0/      0
Validating file 7.h2w ...       0/  2097152/      0/      0
Validating file 8.h2w ...       0/  2097152/      0/      0
Validating file 9.h2w ...       0/  2097152/      0/      0
Validating file 10.h2w ...       0/  2097152/      0/      0
Validating file 11.h2w ...       0/  2097152/      0/      0
Validating file 12.h2w ...       0/  2097152/      0/      0
Validating file 13.h2w ...       0/  2097152/      0/      0
Validating file 14.h2w ...       0/  2097152/      0/      0
Validating file 15.h2w ...       0/  2097152/      0/      0
Validating file 16.h2w ...       0/  2097152/      0/      0
Validating file 17.h2w ...       0/  2097152/      0/      0
Validating file 18.h2w ...       0/  2097152/      0/      0
Validating file 19.h2w ...       0/  2097152/      0/      0
Validating file 20.h2w ...       0/  2097152/      0/      0
Validating file 21.h2w ...       0/  2097152/      0/      0
Validating file 22.h2w ...       0/  2097152/      0/      0
Validating file 23.h2w ...       0/  2097152/      0/      0
Validating file 24.h2w ... 1916384/   180768/      0/      0
Validating file 25.h2w ...  186816/  1910336/      0/      0
Validating file 26.h2w ...       0/  2097152/      0/      0
Validating file 27.h2w ...       0/  2097152/      0/      0
Validating file 28.h2w ...       0/  2097152/      0/      0
Validating file 29.h2w ...   28224/  1705280/      0/      0

  Data OK: 1.02 GB (2131424 sectors)
Data LOST: 27.81 GB (58322336 sectors)
	       Corrupted: 27.81 GB (58322336 sectors)
	Slightly changed: 0.00 Byte (0 sectors)
	     Overwritten: 0.00 Byte (0 sectors)
Average Reading speed: 9.54 MB/s
  

This report shows that my flash card is pretty much garbage since it can only hold 1.02GB. f3write only writes to free space, and will not overwrite existing files as long as they aren't named N.h2w. However, as the previous report also shows, files from 1.h2w to 23.h2w were written before 24.h2w and yet had all their content destroyed. Therefore, it is not wise to test nonempty cards because if the card has a problem, it may erase the old files.

When f3read reads a sector (i.e. 512 bytes, the unit of communication with the card), f3read can check if the sector was correctly written by f3write, and figure out in which file the sector should be and in which position in that file the sector should be. Thus, if a sector is well formed, or with a few bits flipped, but read in an unexpected position, f3read counts it as overwritten. Slightly changed sectors, are sectors at right position with a fews bits flipped.

Notice that f3write doesn't overwrite sectors by itself, it's done by the memory card's controller as a way to difficult a user to uncover its fault. So, the way sectors are overwritten is totally dependent on the controller; from the point of view of a file system, what f3read sees, the way the controller wraps around seems often contrived, but I think that from the memory chip's view, what the controller sees, it is just a truncation of addresses.

The last lines of the output of f3write and f3read provide good estimates of the writing and reading speeds of the tested card. This information can be used to check if the claimed class of the card is correct. Check this link out for more information about classes. Note that the speeds provided by F3 are estimates, don't take them as perfect since they suffer influence even from other processes in your machine!

Later I bought a second card that works just fine; I got the following output running F3 on it:

$ ./f3write /media/6135-3363/
Free space: 29.71 GB
Creating file 1.h2w ... OK!
Creating file 2.h2w ... OK!
Creating file 3.h2w ... OK!
Creating file 4.h2w ... OK!
Creating file 5.h2w ... OK!
Creating file 6.h2w ... OK!
Creating file 7.h2w ... OK!
Creating file 8.h2w ... OK!
Creating file 9.h2w ... OK!
Creating file 10.h2w ... OK!
Creating file 11.h2w ... OK!
Creating file 12.h2w ... OK!
Creating file 13.h2w ... OK!
Creating file 14.h2w ... OK!
Creating file 15.h2w ... OK!
Creating file 16.h2w ... OK!
Creating file 17.h2w ... OK!
Creating file 18.h2w ... OK!
Creating file 19.h2w ... OK!
Creating file 20.h2w ... OK!
Creating file 21.h2w ... OK!
Creating file 22.h2w ... OK!
Creating file 23.h2w ... OK!
Creating file 24.h2w ... OK!
Creating file 25.h2w ... OK!
Creating file 26.h2w ... OK!
Creating file 27.h2w ... OK!
Creating file 28.h2w ... OK!
Creating file 29.h2w ... OK!
Creating file 30.h2w ... OK!
Free space: 0.00 Byte
Average Writing speed: 4.90 MB/s

$ ./f3read /media/6135-3363/
                  SECTORS      ok/corrupted/changed/overwritten
Validating file 1.h2w ... 2097152/        0/      0/      0
Validating file 2.h2w ... 2097152/        0/      0/      0
Validating file 3.h2w ... 2097152/        0/      0/      0
Validating file 4.h2w ... 2097152/        0/      0/      0
Validating file 5.h2w ... 2097152/        0/      0/      0
Validating file 6.h2w ... 2097152/        0/      0/      0
Validating file 7.h2w ... 2097152/        0/      0/      0
Validating file 8.h2w ... 2097152/        0/      0/      0
Validating file 9.h2w ... 2097152/        0/      0/      0
Validating file 10.h2w ... 2097152/        0/      0/      0
Validating file 11.h2w ... 2097152/        0/      0/      0
Validating file 12.h2w ... 2097152/        0/      0/      0
Validating file 13.h2w ... 2097152/        0/      0/      0
Validating file 14.h2w ... 2097152/        0/      0/      0
Validating file 15.h2w ... 2097152/        0/      0/      0
Validating file 16.h2w ... 2097152/        0/      0/      0
Validating file 17.h2w ... 2097152/        0/      0/      0
Validating file 18.h2w ... 2097152/        0/      0/      0
Validating file 19.h2w ... 2097152/        0/      0/      0
Validating file 20.h2w ... 2097152/        0/      0/      0
Validating file 21.h2w ... 2097152/        0/      0/      0
Validating file 22.h2w ... 2097152/        0/      0/      0
Validating file 23.h2w ... 2097152/        0/      0/      0
Validating file 24.h2w ... 2097152/        0/      0/      0
Validating file 25.h2w ... 2097152/        0/      0/      0
Validating file 26.h2w ... 2097152/        0/      0/      0
Validating file 27.h2w ... 2097152/        0/      0/      0
Validating file 28.h2w ... 2097152/        0/      0/      0
Validating file 29.h2w ... 2097152/        0/      0/      0
Validating file 30.h2w ... 1491904/        0/      0/      0

  Data OK: 29.71 GB (62309312 sectors)
Data LOST: 0.00 Byte (0 sectors)
	       Corrupted: 0.00 Byte (0 sectors)
	Slightly changed: 0.00 Byte (0 sectors)
	     Overwritten: 0.00 Byte (0 sectors)
Average Reading speed: 9.42 MB/s
  

As a final remark, since f3write and f3read are independent, f3read can be used as many times as one wants, although f3write is needed only once. This allows one to easily repeat a test of a card as long as the N.h2w files are still available.

Users' notes

Randy Champoux has brought to my attention that, on small fake flash cards, there could be an issue with f3read reading some data from the system cache instead of from the flash card. Since version 2.0, F3 eliminates this possibility as long as the kernel honors the system call posix_fadvise(2) with advice POSIX_FADV_DONTNEED. Linux has and honor posix_fadvise(2)/POSIX_FADV_DONTNEED, whereas Mac does not. Although it's still a real possibility for older vesions of F3, or on kernels that don't follow advices, it should be very rare because the cache works on the announced size of the flash card not on its real size, and f3read starts from the beginning of the data written, that is, there is a rotation of what's on the buffers. In my experience, with exception of servers, which easily have buffers larger than 1GB, the issue shouldn't come out on desktops for any flash card that announces a size of 4GB or more. Finally, unmounting and mounting the device again before calling f3read should eliminate this issue.

When Art Gibbens tested a flash card hosted in a camera connected to his Linux box, at some point F3 didn't show progress, and could not be killed. After a reboot, the card was read only. Using an adapter to connect his card directly to his machine, he recreated the partition of the card adapting the steps described in section "How to 'fix" a fake card', and successfully run F3 with the card in the adpater. Thus, Art's experience is a good warning if you're testing your card in a device other than an adapter. Please, don't take it as a bug of F3. I'm aware of only two things that can make a process "survive" a kill signal: hardware failure, and/or bug in the kernel. F3 doesn't run in kernel mode, so Art's camera is likely the root of the problem.

Darrell Johnson has reported that a flash card he got stopped working after he filled it up. This could be that the only memory chip the card had died, but it is just speculation since Darrell was not able to obtain more information. The important message here is that if you test your card with F3, or just copy files into it, and it stops working, it's not your fault because writing files to a card shouldn't damage it if it is a legitimate card!

Username Kris, here, asked what's the difference between "dosfsck -vtr /dev/sda1" and F3. dosfsck(8) makes two assumptions that F3 does not: (1) one needs right access to the device being tested, not the file system in it; (2) hardware may fail, but it won't lie. The first assumption implies that one likely needs root's rights to run dosfsck, what is just a small incovenience for simple uses. The second assumption is troublesome because a fake card may be able to persuade dosfsck(8) to report it's fine, or not report the whole problem, or give users the illusion the memory card was fixed when it wasn't. I singled dosfsck(8) out because of the question about it, but those two assumptions are true for fsck softwares for other file system formats and badblocks(8) as well.

On the compatibility with H2testw's file format

Starting at version 3.0, F3 generates files following H2testw's file format. This feature should help users that use both applications and/or interact with users that use the other application. However, there are two caveats explained in this section that users should be aware.

Verifying files created by H2testw with F3read. The caveat here is that H2testw only writes files whose size is a multiple of 1MB, whereas F3write fills up the memory card. Thus, verifying files created by H2testw with F3read works, but, likely, will not test approximately 1MB of space that H2testw does not write.

Verifying files created by f3write with H2testw. The caveat here is that H2testw requires that all files have sizes that are multiple of 1MB. When it is not the case for a single file, H2testw rejects all files, and issues the message "Please delete files *.h2w." This problem often comes up with the last file f3write generates to fill up the space available in the memory card being tested. The solution is to truncate the size of the last file f3write generates to the closest multiple of 1MB. Assuming the last file is 30.h2w, the following command does exactly that:

$ truncate --size=/1M /media/6135-3363/30.h2w
  

If you want to exchange files with H2testw users often, check out the shell script f3write.h2w. This script calls truncate after f3write runs successfully.

How to "fix" a fake card

IMPORTANT: this section is only intended for advanced users, if you don't understand it, don't follow the instructions described here! If you make a mistake with fdisk, you may erase all your data on your computer. I am not able to help you with these steps, please look for a friend that can follow them for you.

You should not easily settle down for a fake card, fight back and get your money back! Doing so will help you and others. If you are still reading this section, you already realized that you own a fake card, and would like to be able to use it without losing data. I haven't extensively tested the process listed below, but it worked for me, and MAY work for you as well. Notice that I used a Linux box, so you may have to make adjustments to these steps according to your platform.

As shown in the previous section, my fake card can only hold 1GB. So, using fdisk I have removed the old partition and created a new one with only 1GB. My card was recognized at /dev/sdh, change this parameter according to your case. My fdisk session was as follows:

# fdisk -l /dev/sdh

Disk /dev/sdh: 32.5 GB, 32505331712 bytes
1 heads, 32 sectors/track, 1983968 cylinders
Units = cylinders of 32 * 512 = 16384 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

   Device Boot      Start         End      Blocks   Id  System
/dev/sdh1   *           5     1983968    31743423+   c  W95 FAT32 (LBA)
Partition 1 does not end on cylinder boundary.

# fdisk /dev/sdh

WARNING: DOS-compatible mode is deprecated. It's strongly recommended to
         switch off the mode (command 'c') and change display units to
         sectors (command 'u').

Command (m for help): d
Selected partition 1

Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (2-1983968, default 2): 
Using default value 2
Last cylinder, +cylinders or +size{K,M,G} (2-1983968, default 1983968): +1G

Command (m for help): t
Selected partition 1
Hex code (type L to list codes): c
Changed system type of partition 1 to c (W95 FAT32 (LBA))

Command (m for help): a
Partition number (1-4): 1

Command (m for help): p

Disk /dev/sdh: 32.5 GB, 32505331712 bytes
1 heads, 32 sectors/track, 1983968 cylinders
Units = cylinders of 32 * 512 = 16384 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

   Device Boot      Start         End      Blocks   Id  System
/dev/sdh1   *           2       65538     1048592    c  W95 FAT32 (LBA)

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.

WARNING: If you have created or modified any DOS 6.x
partitions, please see the fdisk manual page for additional
information.
Syncing disks.
  

Once the new partition is created, just format it:

# mkfs.vfat /dev/sdh1
mkfs.vfat 3.0.7 (24 Dec 2009)
  

At this point your card should be working just fine, but before using it, test it again with F3! The test of my card went as follows:

$ ./f3write /media/45B9-B05C/
Free space: 1022.00 MB
Creating file 1.h2w ... OK!
Free space: 0.00 Byte
Average Writing speed: 1.88 MB/s

$ ./f3read /media/45B9-B05C/
                  SECTORS      ok/corrupted/changed/overwritten
Validating file 1.h2w ... 2093048/        0/      0/      0

  Data OK: 1022.00 MB (2093048 sectors)
Data LOST: 0.00 Byte (0 sectors)
	       Corrupted: 0.00 Byte (0 sectors)
	Slightly changed: 0.00 Byte (0 sectors)
	     Overwritten: 0.00 Byte (0 sectors)
Average Reading speed: 8.45 MB/s
  

Repository

The Git repository is kept here. The old page with the history is here; it is no longer maintained.

Author

Michel Machado. E-mail me at michel at digirati dot com dot br.
Please try to figure out the solution for your question on your own, or ask for help from a nearby friend before you e-mail me. As a new parent, my spare time has sensibly shunk.

Copyright and License

F3 is licensed under the GNU General Public License (GPL) version 3.

Copyright (c) 2010 Digirati.