UBIFS - UBI File-System

Original Website:http://www.linux-mtd.infradead.org/doc/ubifs.html

UBIFS - UBI File-System

Table of contents

1. Big red note
2. Overview
3. Power-cuts tolerance
4. UBIFS and MLC NAND flash
5. The unstable bits issue
6. Source code
7. Mailing list
8. User-space tools
9. Scalability
10. Write-back support
11. Write-back knobs in Linux
12. UBIFS write-buffer
13. UBIFS in synchronous mode vs JFFS2
14. Synchronization exceptions for buggy applications
15. Compression
16. Checksumming
17. Read-ahead
18. Space for superuser
19. Extended attributes
20. Mount options
21. Flash space accounting issues
22. Documentation
23. Raw flash vs. FTL devices

Big red note

One thing people have to understand when dealing with UBIFS is that UBIFS is very different to any traditional file system - it does not work on top of block devices (like hard drives, MMC/SD cards, USB flash drives, SSDs, etc). UBIFS was designed to work on top of raw flash, which has nothing to do with block devices. This is why UBIFS does not work on MMC cards and the like - they look like block devices to the outside world because they implement FTL(Flash Translation Layer) support in hardware, which simply speaking emulates a block device on top of the built-in raw flash. Please, make sure you understand the differences between raw flash and, say, MMC flash before dealing with UBIFS. This section should help.

Overview

UBIFS is a new flash file system developed by Nokia engineers with help of the University of Szeged. In a way, UBIFS may be considered as the next generation of the JFFS2 file-system.

JFFS2 file system works on top of MTD devices, but UBIFS works on top of UBI volumes and cannot operate on top of MTD devices. In other words, there are 3 subsystems involved:

• MTD subsystem, which provides uniform interface to access flash chips. MTD provides an notion of MTD devices (e.g., /dev/mtd0) which basically represents raw flash;
• UBI subsystem, which is a wear-leveling and volume management system for flash devices; UBI works on top of MTD devices and provides a notion of UBI volumes; UBI volumes are higher level entities than MTD devices and they are devoid of many unpleasant issues MTD devices have (e.g., wearing and bad blocks); see here for more information;
• UBIFS file system, which works on top of UBI volumes.

Here is a list of some of UBIFS features:

• scalability - UBIFS scales well with respect to flash size; namely, mount time, memory consumption and I/O speed does not depend on flash size (currently it is not 100% true for memory consumption, but the dependency is very weak, and this may be fixed); UBIFS (not UBI!) should work fine for hundreds of GiB flashes; however, UBIFS depends on UBI which has scalability limitations (see here); nonetheless, the UBI/UBIFS stack scales much better than JFFS2, and if UBI becomes a bottleneck, it is always possible to implement UBI2 without changing UBIFS;
• fast mount - unlike JFFS2, UBIFS does not have to scan whole media when mounting, it takes milliseconds for UBIFS to mount the media, and this does not depend on flash size; however, UBI initialization time depends on flash size and has to be taken into account (see here for more details);
• write-back support - this dramatically improves the throughput of the file system in many workloads, comparing to JFFS2, which is write-through; see here for more details;
• tolerance to unclean reboots - UBIFS is a journaling file system and it tolerates sudden crashes and unclean reboots; UBIFS just replays the journal and recovers from the unclean reboot; mount time is a little bit slower in this case, because of the need to replay the journal, but UBIFS does not need to scan whole media, so it anyway takes fractions of a second to mount UBIFS; note, authors payed special attention to this UBIFS aspect, seehere;
• fast I/O - even with write-back disabled (e.g., if UBIFS is mounted with the "-o sync" mount option) UBIFS shows good performance which is close to JFFS2 performance; bear in mind, it is extremely difficult to compete with JFFS2 in synchronous I/O, because JFFS2 does not maintain indexing data structures on flash, so it does not have the maintenance overhead, while UBIFS does have it; but UBIFS is still fast because of the way UBIFS commits the journal - it does not move the data physically from one place to another but instead, it just adds corresponding information to the file system index and picks different eraseblocks for the new journal (i.e., UBIFS has sort of "wandering" journal which constantly changes the position); there are other tricks like multi-headed journal which make UBIFS perform well;
• on-the-flight compression - the data are stored in compressed form on the flash media, which makes it possible to put considerably more data to the flash than if the data were not compressed; this is very similar to what JFFS2 has; UBIFS also allows to switch the compression on/off on per-inode basis, which is very flexible; for example, one may switch the compression off by default and enable it only for certain files which are supposed to compress well; or one may switch compression on by default but disable it for supposedly uncompressible data like multimedia files; at the moment UBIFS supports only zlib and LZO compressors and it is not difficult to add more; see this section for more information.
• recoverability - UBIFS may be fully recovered if the indexing information gets corrupted; each piece of information in UBIFS has a header which describes this piece of information, and it is possible to fully reconstruct the file system index by scanning the flash media; this is very similar to JFFS2; to make it more clear, imaging you have wiped out the FAT table on your FAT file system; for FAT FS this would be fatal; but if you similarly wipe out UBIFS index, you still may re-construct it, although a special user-space tool would be required to do this (this utility is not implemented at the moment, though);
• integrity - UBIFS (as well as UBI) checksums everything it writes to the flash media to guarantee data integrity, UBIFS does not leave data or meta-data corruptions unnoticed (JFFS2 is doing the same); By default, UBIFS checks only meta-data CRC when it reads from the media, but not the data CRC; however, you may force CRC checking for data using one of the UBIFS mount options - see here.

Power-cuts tolerance

Both UBI (see here) and UBIFS are tolerant to power-cuts, and they were designed with this property in mind.

Year 2011 note: however, there is an unsolved unstable bits issue which makes UBI/UBIFS fail to recover after a power cut on modern SLC and MLC flashes. This issue has not been observed on older SLC NANDs back at the time UBI/UBIFS was being developed. Note, the below text is quite old and has been written before the unstable bits issue has been first discovered.

UBIFS has internal debugging infrastructure to emulate power failures and the authors used it for extensive testing. It was tested for long time with power-fail emulation. The advantage of the emulation is that it emulates power failures even at the situations which happen not very often. For example, when the master node is updated, or the log is changed. The probability to interrupt the system at those moments is very low in real-life.

There is also a powerful user-space test program called integck which performs a lot of random I/O operations and checks the integrity of the FS after remount. This test can also handle emulated power-cuts and check the FS integrity.

Real power-cut tests have also been done on OneNAND flash. We used Power Node devices which are controlled via serial line and can switch the power of the connected device on and off. UBIFS survived more than 100000 power-cuts while running stress tests.

We've also done real power-cut tests on Spansion NOR flash. Some problems were found, but they were fixed and the board survived 10000 power-cuts after this. Please, see related discussions at the MTD mailing list. The thread has "UBIFS Corrupt during power failure" subject. The beginning of the thread may be found here, then it continues here, and then continues here, and here, and here.

UBIFS and MLC NAND flash

UBIFS authors never tested UBI/UBIFS on MLC flash devices. Let's consider some specific aspects of MLC NAND flashes:

• [OK] MLC NAND flashes are more "faulty" than SLC, so they use stronger ECC codes; these ECC codes often occupy whole OOB area (as do the ECC codes on some newer SLC flashes, which are more error-prone than previous generations of flash); this is not a problem for UBI/UBIFS, because neither UBIFS nor UBI use OOB area;
• [OK] when the data are written to an eraseblock, they have to be written sequentially, from the beginning of the eraseblock to the end of it; this is also not a problem because it is exactly what UBI and UBIFS do (see also thissection);
• [OK] MLC flashes have rather short eraseblock life-cycles of just a few thousand erase cycles; this is not a problem because UBI uses a deterministic wear-leveling algorithm. However, the default 4096 erases wear-levelling threshold may need to be lessened for MLC.
• [NEED WORK] MLC flashes exhibit bit-flips as a result of "program disturb" and "read disturb" errors (see here). These errors are sometimes referred to as "reversible" errors in NAND datasheets, meaning that they disappear once the block in which they are located is erased; as opposed to "irreversible" errors which are due to cell wear and cause permanent bit failures. Note that SLC flashes have these same errors, but they are much more common on MLC:
  • NAND flashes have a so called "read-disturb" property, which means that a NAND page read operation may introduce a persistent bit change, not necessarily located in the page being read; the ECC code would fix it, but more read operations may introduce more bit changes and correctable ECC errors may turn into uncorrectable ECC errors; however, when these errors occur on the same page that is being read, this should not be a problem because UBI is doing scrubbing; in other words, once UBI notices that there is a correctable bit-flip in an eraseblock, it moves the contents of this physical eraseblock to a different physical eraseblock, and re-maps the corresponding logical eraseblock to the new physical eraseblock; so UBI refreshes the data and gets rid of bit-flips, thus improving data integrity.
  • "Read-disturb" errors can also occur on a page other that the one being read, but which is within the same eraseblock. This is not a problem if page read operations are spread around somewhat evenly within the eraseblock, since the bit-flip will soon be detected and corrected through the "scrubbing" process described above. However, if a particular page within a block is rarely read, scrubbing will not have a chance to fix errors, and they may accumulate over time until they become unfixable. This is very similar the next problem.
  • NAND flashes also have a "program-disturb" property, which means that if you program a NAND page, you may introduce a bit-flip in a different NAND page. The bit change can be fixed by ECC, but with time the changes may accumulate and become unfixable. Current UBI bit-flip handling only partially helps here, because it is passive, which means that UBI notices bit-flips only when performing users' read requests. So if you never read the NAND page which accumulates bit-flips, UBI will never notice this.

  The read and program disturb issues should be possible to handle by implementing a kind of "flash crawler" which would read all of the NAND pages in the background from time to time (at UBI level) making UBI notice and fix bit-flips. This is not implemented though, and this can probably be done from user-space.

• [NEED WORK] There is another aspect of MLC flashes which may need closer attention: the "paired pages" problem (e.g., see this Power Point presentation). Namely, MLC NAND pages are coupled in a sense that if you cut power while writing to a page, you corrupt not only this page, but also one of the previous pages which is paired with the current one. For example, pages 0 and 3, 1 and 4, 2 and 5, 3 and 6 in and so on (in the same eraseblock) may be paired (page distance is 4, but there may be other distances). So if you write data to, say, page 3 and cut the power, you may end up with corrupted data in page 0. UBIFS is not ready to handle this problem at the moment and this needs some work.

  UBIFS can handle this problem by avoiding using the rest of free space in LEBs after a sync or commit operation. E.g., if start writing to a new journal LEB, and then have a sync or commit, we should "waste" some amount of free space in this LEB to make sure that the previous paired page does not contain synced data. This way we guarantee that a power cut will not corrupt the synced or committed data. And the "wasted" free space can be re-used after that LEB has been garbage-collected. Similar to all the other LEBs we write to (LPT, log, orphan, etc). This would require some work and would make UBIFS slower, so this should probably be optional. The way to attack this issue is to improve UBIFS power cut emulation and implement "paired-pages" emulation, then use the integck test for testing. After all the issues are fixed, real power-cut tests could be carried out.

• [NEED WORK] The "unstable bits issue", which is not MLC-specific, described here.

The unstable bits issue

In the MTD community the "unstable bits" term is used to describe data instabilities caused by power cuts while writing or erasing. The unstable bits issue is still not resolved in UBI and UBIFS, and it was reported several times in the MTD mailing list. In theory, this issue should be visible in any flash, but for some reason back at the times when we developed UBI/UBIFS and extensively tested them on a robust SLC NAND, we did not observe it. No one reported about this issue for NOR flash yet. However, on modern SLC and MLC flashes this problem is reproducible.

The unstable bits are the result of a power cut during a program or erase operation. Depending on when the power cut has happened, they can corrupt the data or the free space. Consider the following 4 situations:

1. The power cut happens just before the NAND page program operation finishes. After reboot the page may be read correctly and without a single bit-flip say, 2 times, and the 3rd time you may get an ECC error. This happens because the page contains a number of unstable bits which are sometimes read correctly and sometimes not.
2. The power cut happens just after the NAND page program operation starts. After reboot, the page may be read correctly (return all 0xFFs) most of the time, but sometimes you may get some bits set to zero. Moreover, if you then program this page, it also may be sometimes read correctly, but sometimes return an ECC error. The reason is again the unstable bits in the NAND page.
3. The power cut happens just before the eraseblock erase operation finishes. After reboot, the eraseblock may contain unstable bits and data in this eraseblock may suddenly become corrupted.
4. The power cut happens just after the eraseblock erase operation starts. After reboot, the eraseblock may contain unstable bits and sometimes return zero bits on read, or corrupted data if you program it.

The number of unstable bits resulting from a power-cut may be greater than what the ECC algorithm is able to correct. This is why a previously readable page may suddenly become unreadable, or conversely a previously unreadable page may suddenly become readable.

Here is an example scenario how UBIFS may fail. UBIFS writes data node A to the journal LEB, and a power cut of type 1 happens. After the reboot, UBIFS recovery code reads that LEB, no bit-flips are reported by MTD, all the CRCs match, everything looks fine. UBIFS just assume that this LEB is all-right and the free space at the end of this LEB can be used for writing more data. UBIFS performs the commit operations, writes more user data, and everything works fine until the user reads node A by reading the corresponding file: an ECC error happens and the user gets the EIO error.

The EIO may be what the user gets instead of his/her data also if a type 2 power cut happens, and UBIFS re-uses the corrupted free space for writing new nodes, and then these nodes are read.

The solution is to teach UBIFS to erase-cycle any LEB which could potentially be written to when the power cut happened. This is not only about the journal LEBs, but also LPT, log, master and orphan LEBs. This means that the valid data from this LEB has to be read (and only once!) and then it should be written back to this LEB using the atomic LEB change UBI operation. This has to be done even if the LEB looks all-right - no corruptions, all 0xFFs at the end.

Similarly, UBI has to erase-cycle every eraseblock which could potentially be erased when the power cut happened.

The other requirement is that during the recovery UBI/UBIFS should read data from the media only once. This is easy to demonstrate on the delayed recovery example. The delayed recovery happens when after a power cut the file-system is mounted R/O, in which case UBIFS must not write anything to the flash, and the real recovery is delayed until the FS is re-mounted R/W. Currently UBIFS just scans the journal during mounting R/O, drops (or "remembers") corrupted nodes, and "does not let" users read them. But there is no guarantee that UBIFS spots all the corrupted nodes during the first scanning, so users may get EIO while reading data from the R/O-mounted FS.

When UBIFS is then remounted R/W, it actually drops the corrupted nodes from the flash media by erase-cycling the corresponding LEBs. And UBIFS re-reads all the LEB data again. And there is no guarantee that UBIFS will get the same corruptions again.

So it is important to make sure that the corrupted LEBs are read only once. E.g., we can cache the results of the first scanning, and then use that data when running the delayed recovery, instead of re-reading the data. Probably we may remember only the last NAND page containing valid nodes, not whole LEB, since for the journal only unstable bits of type 1 and 2 are relevant.

There are similar double-read issues in UBI scanning - when it finds 2 PEBs belonging to the same LEB and it has to find out which one is newer. The volume table has to be erase-cycled as well in UBI.

There are more issues related to unstable bits of type 2 and 3 in UBI, I think. This all needs a very careful look, and this is not trivial to fix because of the complexity: UBIFS as any file-system has many interfaces and a lot of states. The best strategy to attack this problem would be:

1. Improve the existing power cut emulation infrastructure in UBIFS and start emulating unstable bits. Start with emulating only one type of unstable bits, e.g., type 1.
2. Use the integck test to stress the file-system with power cut emulation enabled - the test can re-start when an emulated power cut happens. This will allow you to very quickly emulate hundreds of power cuts in interesting places. Fix all the bugs. Make sure it is rock solid. Of course, if you have various independent issues, you may temporary hack the power cut emulation code to emulate unstable bits only at certain places, to temporarily limit the amount of problems you have to simultaneously deal with.
3. Start emulating other types of unstable bits, and fix all the issues one-by-one.
4. Go down to UBI and add a similar power cut emulation infrastructure. But emulate unstable bits only in UBI-specific on-flash data structures - the EC/VID headers and the volume table. Improve the integck test to support that infrastructure and fix all the issues.
5. Run real power cut tests on real hardware.

Source code

The UBIFS git tree is

git://git.infradead.org/ubifs-2.6.git

Here is the corresponding Git-web view.

The git tree has master and linux-next branches. The master branch contains the most recent stuff which is often incomplete, buggy, or not tested very well. This branch may be re-based from time to time. The linux-next branch contains stable UBIFS updates and fixes. This branch is included to the linux-next git tree and goes to main-line. This branch contains patched which will be merged upstream during the merge window.

There are also 3.0 - 3.6 kernel back-ports which may be found here:

git://git.infradead.org/~dedekind/ubifs-v3.10.git
git://git.infradead.org/~dedekind/ubifs-v3.9.git
git://git.infradead.org/~dedekind/ubifs-v3.8.git
git://git.infradead.org/~dedekind/ubifs-v3.7.git
git://git.infradead.org/~dedekind/ubifs-v3.6.git
git://git.infradead.org/~dedekind/ubifs-v3.5.git
git://git.infradead.org/~dedekind/ubifs-v3.4.git
git://git.infradead.org/~dedekind/ubifs-v3.3.git
git://git.infradead.org/~dedekind/ubifs-v3.2.git
git://git.infradead.org/~dedekind/ubifs-v3.1.git
git://git.infradead.org/~dedekind/ubifs-v3.0.git

Mailing list

You are welcome to send feed-back, bug-reports, patches, etc to the MTD mailing list. Feel free to ask questions, but please, check the UBIFS FAQ before doing this.

User-space tools

There is only one UBIFS user-space tool at the moment - mkfs.ubifs, which creates UBIFS images. The tool may be found in the MTD utils repository (the mkfs.ubifs sub-directory):

git://git.infradead.org/mtd-utils.git

The images produced by mkfs.ubifs may be written to UBI volumes using ubiupdatevol or may be further fed to the ubinize tool to create an UBI image which may be put to the raw flash.

Scalability

All the data structures UBIFS is using are trees, so it scales logarithmically in terms of flash size. However, UBI scales linearly (see here) which makes overall UBI/UBIFS stack scalability to be linear. But UBIFS authors believe it is possible to create logarithmically scalable UBI2 and improve the situation. Current UBI should be OK for 2-16GiB raw flashes, depending on the I/O speed and the requirements.

Note, although UBI scalability is linear, it anyway scales much better than JFFS2, which was originally designed for small ~32MiB NOR flashes. JFFS2 has scalability issues on the "file system level", while UBI/UBIFS stack has scalability issues only on lower "raw flash level". The following table describes the issues in more details.

Scalability issue	JFFS2	UBIFS
Mount time linearly depends on the flash size	True, the dependency is linear, because JFFS2 has to scan whole flash media when mounting.	UBIFS mount time does not depend on the flash size. But UBI needs to scan the flash media, which is actually quicker than JFFS2 scanning. So overall, UBI/UBIFS has this linear dependency.
Memory consumption linearly depends on the flash size	True, the dependency is linear.	UBIFS memory does depend on the flash size in the current implementation, because the LPT shrinker is not implemented. But it is not difficult to implement the LPT shrinker and get rid of the dependency. It is not implemented only because the memory consumption is too small to make the coding work worth it. UBI memory consumption linearly depends on flash size. Thus, overall UBI/UBIFS stack has the linear dependency.
Mount time linearly depends on the file system contents	True, the more data are stored on the file system, the longer it takes to mount it, because JFFS2 has to do more scanning work.	False, mount time does not depend on the file system contents. At the worst case (if there was an unclean reboot), UBIFS has to scan and replay the journal which has fixed and configurable size.
Full file system checking is required after each mount	True. JFFS2 has to check whole file system just after it has been mounted in case of NAND flash. The checking involves reading all nodes for each inode and checking their CRC checksums, which consumes a lot of CPU. For example, this may be observed by runningtop just after JFFS2 has been mounted. This slows down overall system boot-up time. Fundamentally, this is needed because JFFS2 does not store space accounting information (i.e., free/dirty space) on the flash media but instead, gathers this information by scanning the flash media.	False. UBIFS does not scan/check whole file system because it stores the space accounting information on the flash media in the so-called LPT (Logical eraseblock Properties Tree) tree.
Memory consumption linearly depends on file system contents	True. JFFS2 keeps a small data structure in RAM for each node on flash, so the more data are stored on the flash media, the more memory JFFS2 consumes.	False. UBIFS memory consumption does not depend on how much data are stored on the flash media.
File access time linearly depends on its size	True. JFFS2 has to keep in RAM so-called "fragment tree" for each inode corresponding to an opened file. The fragment tree is an in-memory RB-tree which is indexed by file offset and refers on-flash nodes corresponding to this offset. The fragment tree is not stored on the flash media. Instead, it is built on-the-flight when the file is opened for the first time. To build the fragment tree, JFFS2 has to read each data node corresponding to this inode from the flash. This means, the larger is the file, the longer it takes to open it for the first time. And the larger is the file the more memory it takes when it is opened. Depending on the system, JFFS2 becomes nearly unusable starting from certain file size.	False. UBIFS stores all the indexing information on the media in the indexing B-tree. Whenever a piece of data has to be read from the file system, the B-tree is looked-up and the corresponding flash address to read is found. There is a TNC cache which caches the B-tree nodes when the B-tree is looked-up, and the cache is shrinkable, which means it can be shrunk when the kernel needs more memory.
File-system performance depends on I/O history	True. Since JFFS2 is fully synchronous, it writes data to the flash media as soon as the data arrives. If one changes few bytes in the middle of a file, JFFS2 writes a data node which contains those bytes to the flash. If there are many random small writes all over the place, the file system becomes fragmented. JFFS2 merges small fragments to 4KiB chunks, which involves re-compression and re-writing the data. But this "defragmentation" is happening during garbage collection and at random time, because JFFS2 wear-leveling algorithm is based on random eraseblock selection. So if there were a lot of small writes, JFFS2 becomes slower some time later - the performance just goes down out of the blue which makes the system less predictable.	False. UBIFS always writes in 4KiB chunks. This does not hurt the performance much because of the write-back support: the data changes do not go to the flash straight away - they are instead deferred and are done later, when (hopefully) more data are changed at the same data page. And write-back usually happens in background.

Write-back support

UBIFS supports write-back, which means that file changes do not go to the flash media straight away, but they are cached and go to the flash later, when it is absolutely necessary. This helps to greatly reduce the amount of I/O which results in better performance. Write-back caching is a standard technique which is used by most file systems like ext3 or XFS.

In contrast, JFFS2 does not have write-back support and all the JFFS2 file system changes go the flash synchronously. Well, this is not completely true and JFFS2 does have a small buffer of a NAND page size (if the underlying flash is NAND). This buffer contains last written data and is flushed once it is full. However, because the amount of cached data are very small, JFFS2 is very close to a synchronous file system.

Write-back support requires the application programmers to take extra care about synchronizing important files in time. Otherwise the files may corrupt or disappear in case of power-cuts, which happens very often in many embedded devices. Let's take a glimpse at Linux manual pages:

$ man 2 write
....
NOTES
       A  successful return from write() does not make any guarantee that data
       has been committed to disk.  In fact, on some buggy implementations, it
       does  not  even guarantee that space has successfully been reserved for
       the data.  The only way to be sure is to call fsync(2)  after  you  are
       done writing all your data.
...

This is true for UBIFS (except of the "some buggy implementations" part, because UBIFS does reserve space for cached dirty data). This is also true for JFFS2, as well as for any other Linux file system.

However, some (perhaps not very good) user-space programmers do not take write-back into account. They do not read manual pages carefully. When such applications are used in embedded systems which run JFFS2 - they work fine, because JFFS2 is almost synchronous. Of course, the applications are buggy, but they appear to work well enough with JFFS2. But the bugs show up when UBIFS is used instead. Please, be careful and check/test your applications with respect to power cut tolerance if you switch from JFFS2 to UBIFS. The following is a list of useful hints and advices.

• If you want to switch into synchronous mode, use the -o sync option when mounting UBIFS; however, the file system performance will drop - be careful. Also remember that UBIFS mounted in synchronous mode provides less guarantees than JFFS2 - refer this section for details.
• Always keep in mind the above statement from the manual pages and run fsync() for all important files you change; of course, there is no need to synchronize "throw-away" temporary files; Just think how important the file data is and decide; do not use fsync() unnecessarily, because this will hit the performance.
• If you want to be more accurate, you may use fdatasync(), in which cases only data changes will be flushed, but not inode meta-data changes (e.g., "mtime" or permissions); this might be more optimal than using fsync() if the synchronization is done often, e.g., in a loop; otherwise just stick with fsync().
• In shell, the sync command may be used, but it synchronizes the whole file system which might not be optimal; and there is a similar libc sync() function.
• You may use the O_SYNC flag of the open() call; this will make sure all the data (but not meta-data) changes go to the media before the write() operation returns; but in general, it is better to use fsync(), because O_SYNCmakes each write to be synchronous, while fsync() allows to accumulate many writes and synchronize them at once.
• It is possible to make certain inodes to be synchronous by default by setting the "sync" inode flag; in a shell, the chattr +S command may be used; in C programs, use the FS_IOC_SETFLAGS ioctl command; Note, themkfs.ubifs tool checks for the "sync" flag in the original FS tree, so the synchronous files in the original FS tree will be synchronous in the resulting UBIFS image.

Let us stress that the above items are true for any Linux file system, including JFFS2.

fsync() may be called for directories - it synchronizes the directory inode meta-data. The "sync" flag may also be set for directories to make the directory inode synchronous. But the flag is inherited, which means all new children of this directory will also have this flag. New files and sub-directories of this directory will also be synchronous, and their children, and so forth. This feature is very useful if one needs to create a whole sub-tree of synchronous files and directories, or to make all new children of some directory to be synchronous by default (e.g., /etc).

The fdatasync() call for directories is "no-op" in UBIFS and all UBIFS operations which change directory entries are synchronous. However, you should not assume this for portability (e.g., this is not true for ext2). Similarly, the "dirsync" inode flag has no effect in UBIFS.

The functions mentioned above work on file-descriptors, not on streams (FILE *). To synchronize a stream, you should first get its file descriptor using the fileno() libc function, then flush the stream using fflush(), and then synchronize the file using fsync() or fdatasync(). You may use other synchronization methods, but remember to flush the stream before synchronizing the file. The fflush() function flushes the libc-level buffers, while sync(),fsync(), etc flush kernel-level buffers.

Please, refer this FAQ entry for information about how to atomically update the contents of a file. Also, the Theodore Tso's article is a good reading.

Write-back knobs in Linux

Linux has several knobs in "/proc/sys/vm" which you may use to tune write-back. The knobs are global, so they affect all file-systems. Please, refer the "Documentation/sysctl/vm.txt" file fore more information. The file may be found in the Linux kernel source tree. Below are interesting knobs described in UBIFS context and in a simplified form.

• dirty_writeback_centisecs - how often the Linux periodic write-back thread wakes up and writes out dirty data. This is a mechanism which makes sure all dirty data hits the media at some point.
• dirty_expire_centisecs - dirty data expire period. This is maximum time data may stay dirty. After this period of time it will be written back by the Linux periodic write-back thread. IOW, the periodic write-back thread wakes up every "dirty_writeback_centisecs" centi-seconds and synchronizes data which was dirtied "dirty_expire_centisecs" centi-seconds ago.
• dirty_background_ratio - maximum amount of dirty data in percent of total memory. When the amount of dirty data becomes larger, the periodic write-back thread starts synchronizing it until it becomes smaller. Even non-expired data will be synchronized. This may be used to set a "soft" limit for the amount of dirty data in the system.
• dirty_ratio - maximum amount of dirty data at which writers will first synchronize the existing dirty data before adding more. IOW, this is a "hard" limit of the amount of dirty data in the system.

Note, UBIFS additionally has small write-buffers which are synchronized every 3-5 seconds. This means that most of the dirty data are delayed by dirty_expire_centisecs centi-seconds, but the last few KiB are additionally delayed by 3-5 seconds.

UBIFS write-buffer

UBIFS is asynchronous file-system (read this section for more information). As other Linux file-system, it utilizes the page cache. The page cache is a generic Linux memory-management mechanism. It may be very large and cache a lot of data. When you write to a file, the data are written to the page cache, marked as dirty, and the write returns (unless the file is synchronous). Later the data are written-back.

Write-buffer is an additional UBIFS buffer, which is implemented inside UBIFS, and it sits between the page cache and the flash. This means that write-back actually writes to the write-buffer, not directly to the flash.

The write-buffer is designated to speed-up UBIFS on NAND flashes. NAND flashes consist of NAND pages, which are usually 512, 2KiB or 4KiB in size. NAND page is the minimal read/write unit of NAND flash (see this section).

Write-buffer size is equivalent to NAND page size (so it is tiny comparing to the page cache). It's purpose is to accumulate small writes, and write full NAND pages instead of partially filled. Indeed, imagine we have to write 4 512-byte nodes with half a second interval, and NAND page size is 2KiB. Without write-buffer we would have to write 4 NAND pages and waste 6KiB of flash space, while write-buffer allows us to write only once and waste nothing. This means we write less, we create less dirty space so UBIFS garbage collector will have to do less work, we save power.

Well, the example shows an ideal situation, and even with the write-buffer we may waste space, for example in case of synchronous I/O, or if the data arrives with long time intervals. This is because the write-buffer has an associated timer, which flushes it every 3-5 seconds, even if it isn't full. We do this for data integrity reasons.

Of course, when UBIFS has to write a lot of data, it does not use write buffer. Only the last part of the data which is smaller than the NAND page ends up in the write-buffer and waits more for data, until it is flushed by the timer.

The write-buffer implementation is a little more complex, and we actually have several of them - one for each journal head. But this does not change the basic idea behind the write-buffer.

Few notes with regards to synchronization:

• "sync()" also synchronizes all write-buffers;
• "fsync(fd)" also synchronizes all write-buffers which contain pieces of "fd";
• synchronous files, as well as files opened with "O_SYNC", bypass write-buffers, so the I/O is indeed synchronous for this files;
• write-buffers are also bypassed if the file-system is mounted with the "-o sync" mount option.

Take into account that write-buffers delay the data synchronization timeout defined by "dirty_expire_centisecs" (see here) by 3-5 seconds. However, since write-buffers are small, only few data are delayed.

UBIFS in synchronous mode vs JFFS2

When UBIFS is mounted in synchronous mode (-o sync mount options) - all file system operations become synchronous. This means that all data are written to flash before the file-system operations return.

For example, if you write 10MiB of data to a file f.dat using the write() call, and UBIFS is in synchronous mode, then UBIFS guarantees that all 10MiB of data and the meta-data (file size and date changes) will reach the flash media before write() returns. And if a power cut happens after the write() call returns, the file will contain the written data.

The same is true for situations when f.dat has was opened with O_SYNC or has the sync flag (see man 2 chattr).

It is well-known that the JFFS2 file-system is synchronous (except a small write-buffer). However, UBIFS in synchronous mode is not the same as JFFS2 and provides somewhat less guarantees that JFFS2 does with respect to sudden power cuts.

In JFFS2 all the meta-data (like inode atime/mtime/ctime, inode size, UID/GID, etc) are stored in the data node headers. Data nodes carry 4KiB of (compressed) data. This means that the meta-data information is duplicated in many places, but this also means that every time JFFS2 writes a data node to the flash media, it updates inode size as well. So when JFFS2 mounts it scans the flash media, finds the latest data node, and fetches the inode size from there.

In practice this means that JFFS2 will write these 10MiB of data sequentially, from the beginning to the end. And if you have a power cut, you will just lose some amount of data at the end of the inode. For example, if JFFS2 starts writing those 10MiB of data, write 5MiB, and a power cut happens, you will end up with a 5MiB f.dat file. You lose only the last 5MiB.

Things are a little bit more complex in case of UBIFS, where data are stored in data nodes and meta-data are stored in (separate) inode nodes. The meta-data are not duplicated in each data node, like in JFFS2. UBIFS never writes data nodes beyond the on-flash inode size. If it has to write a data node and the data node is beyond the on-flash inode size (the in-memory inode has up-to-data size, but it is dirty and was not flushed yet), then UBIFS first writes the inode to the media, and then it starts writing the data. And if you have an interrupt, you lose data nodes and you have holes (or old data nodes, if you are overwriting). Lets consider an example.

• User creates an empty file f.dat. The file is synchronous, or UBIFS is mounted in synchronous mode. User calls the write() function with a 10MiB buffer.
• The kernel first copies all 10MiB of the data to the page cache. Inode size is changed to 10MiB as well and the inode is marked as dirty. Nothing has been written to the flash media so far. If a power cut happens at this point, the user will end up with an empty f.dat file.
• UBIFS sees that the I/O has to be synchronous, and starts synchronizing the inode. First of all, it writes the inode node to the flash media. If a power cut happens at this moment, the user will end up with a 10MiB file which contains no data (hole), and if he read this file, he will get 10MiB of zeroes.
• UBIFS starts writing the data. If a power cut happens at this point, the user will end up with a 10MiB file containing a hole at the end.

Note, if the I/O was not synchronous, UBIFS would skip the last step and would just return. And the actual write-back would then happen in back-ground. But power cuts during write-back could anyway lead to files with holes at the end.

Thus, synchronous I/O in UBIFS provides less guarantees than JFFS2 I/O - UBIFS has an effect of holes at the end of files. In ideal world applications should not assume anything about the contents of files which were not synchronized before a power-cut has happened. And "mainstream" file-systems like ext3 do not provide JFSS2-like guarantees.

However, UBIFS is sometimes used as a JFFS2 replacement and people may want it to behave the same way as JFFS2 if it is mounted synchronously. This is doable, but needs some non-trivial development, so this was not implemented so far. On the other hand, there was no strong demand. You may implement this as an exercise, or you may try to convince UBIFS authors to do this.

Synchronization exceptions for buggy applications

As this section describes, UBIFS is an asynchronous file-system, and applications should synchronize their files whenever it is required. The same applies to most Linux file-systems, e.g. XFS.

However, many applications ignore this and do not synchronize files properly. And there was a huge war between user-space and kernel developers related to ext4 delayed allocation feature. Please, see the Theodore Tso's blog post. More information may be found in this LWN article.

In short, the flame war was about 2 cases. The first case was about the atomic re-name, where many user-space programs did not synchronize the copy before re-naming it. The second case was about applications which truncate files, then change them. There was no final agreement, but the "we cannot ignore the real world" argument found ext4 developers' understanding, and there were 2 ext4 changes which help both problems.

Roughly speaking, the first change made ext4 synchronize files on close if they were previously truncated. This was a hack from file-system point of view, but it "fixed" applications which truncate files, write new contents, and close the files without synchronizing them.

The second change made ext4 synchronize the renamed file.

Well, this is not exactly correct description, because ext4 does not write the files synchronously, but actually initiates asynchronous write-out of the files, so the performance hit is not very high. For the truncation case this means that the file is synchronized soon after it is closed. For the re-name case this means that ext4 writes data before it writes the re-name meta-data.

However, the application writers should never rely on these things, because this is not portable. Instead, they should properly synchronize files. The ext4 fixes were because there were many broken user-space applications in the wild already.

We have plans to implement these features in UBIFS, but this has not been done yet. The problem is that UBI/MTD are fully synchronous and we cannot initiate asynchronous write-out, so we'd have to synchronously write files on close/rename, which is slow. So implementing these features would require implementing asynchronous I/O in UBI, which is a big job. But feel free to do this :-).

Compression

UBIFS supports on-the-fly compression, which means it compresses data before writing them to the flash media, and decompresses before reading them, and this is absolutely transparent to the users. UBIFS compresses only regular files data. Directories, device nodes and so on are not compressed. Meta-data and the indexing information are not compressed as well.

At the moment UBIFS supports LZO and zlib compressors. Zlib provides better compression ratio, but LZO is faster in both compression and decompression. LZO is the default compressor for UBIFS and for the mkfs.ubifs utility. And of course you may disable UBIFS compression altogether using the "-x none" mkfs.ubifs option.

UBIFS splits all data on 4KiB chunks and compresses each chunk independently. This is not optimal, because larger chunks of data would compress better, but this still provides noticeable flash space economy. For example, real-life root file-system image for an ARM platform becomes ~40% smaller with LZO compression and ~50% smaller with zlib compression. This means that you may fit a 300MiB rootfs image into a 256MiB UBI volume and still have about 100MiB of free space. However, the figures may be different depending on the contents of the file-system. For example, if your file-system mostly contains mp3 files, UBIFS will be unable to efficiently compress them, just because mp3 files are already compressed.

In UBIFS it is possible to enable or disable compression individually for each inode by setting or cleaning their compression flag. Note, the compression flag of directories is inherited, which means that when files and sub-directories are created, they inherit the compression flag of the parent directory. Please, refer this section for instruction about how the compression flag may be toggled.

It is also possible to somewhat combine LZO and zlib compressors, see this FAQ section.

It's also worth noting that JFFS2 LZO compression is a little bit different to UBIFS zlib compression. UBIFS uses the crypto-API deflate method, while JFFS2 uses zlib library directly. As a result, UBIFS and JFFS2 use different zlib compression options. Namely, JFFS2 uses deflate level 3 and window bits 15, while UBIFS uses deflate level 6 and window bits -11 (minus makes zlib avoid putting a header to the output data stream). Experiments with compressing ARM code showed that JFFS2 compression ratio is slightly smaller, decompression speed is also slightly slower, but compression speed is a bit faster.

Checksumming

Every piece of information UBIFS writes to the media has a CRC-32 checksum. UBIFS protects both data and meta-data with CRC. Every time the meta-data is read, the CRC checksum is verified. CRC-32 is quite strong function and any data corruption will most probably be noticed. The same is true for UBI, by the way, it verifies every piece of meta-data.

The data CRC is not verified by default. We do this to improve the default file-system read speed. But UBIFS allows to switch the data verification on using the chk_data_crc mount option. This decreases UBIFS read speed a little, but provides better integrity protection. With this option on you may be sure that any piece of information UBIFS reads from the media will be verified and any corruption will most probably be noticed.

Note, currently UBIFS cannot disable CRC-32 calculations on write, because UBIFS recovery process depends on in. When recovering from an unclean reboot and re-playing the journal, UBIFS has to be able to detect broken and half-written UBIFS nodes and drop them, and UBIFS depends on the CRC-32 checksum here.

In other words, if you use UBIFS with data CRC-32 checking disabled, you still have the CRC-32 checksum attached to each piece of data, and you may mount UBIFS with the chk_data_crc option to enable CRC-32 checking at any time (e.g., when you suspect the file-system might be corrupted because you visited the Large Hadron Collider and exposed your flash to proton beams).

NOTE!: before 2.6.39 the default UBIFS behavior was the oppsite - it did check data CRC-32 by default and the no_chk_data_crc had to be used.

Read-ahead

Read-ahead is an optimization technique which makes the file system read a little bit more data than users actually ask. The idea is that files are often read sequentially from the beginning to the end, so the file system tries to make next data available before the user actually asks for them.

Linux VFS is capable of doing read-ahead and this does not require any support from the file system. This probably works well for traditional block-based file systems, however this does not work well for UBIFS. UBIFS works with UBI API, which works with MTD API, which is synchronous. MTD API is pretty trivial and does not have any request queues. This means that VFS blocks UBIFS readers and makes them wait for read-ahead process. In opposite, block-device API is asynchronous and readers do not wait for read-ahead.

VFS read-ahead was designed for hard drives, and it was benchmarked with hard-drives. But the nature of raw flash devices is very different to the nature of Hard Drives Raw flash devices do not heave such a huge seek time as hard drives do, so the techniques which work for HDDs do not necessarily work well for flash.

That said, VFS read-ahead only slows UBIFS down instead of improving it, so UBIFS disables VFS read-ahead. But UBIFS has its own internal read-ahead, which we call "bulk-read". You may enable bulk-read using the "bulk_read" UBIFS mount option.

Some flashes may read faster if the data are read at one go, rather than at several read requests. For example, OneNAND can do "read-while-load" if it reads more than one NAND page. So UBIFS may benefit from reading large data chunks at one go, and this is exactly what bulk-read does.

If UBIFS notices that a file is being read sequentially (at least 3 sequential 4KiB blocks has been read), and if UBIFS sees that the further file data resides sequentially at the same eraseblock, it starts reading data ahead using large read requests, which makes it possible to read at higher rates. So UBIFS reads more than it is asked to, and it pushes the read-ahead data to the file caches, so the data become instantly available for the further user read requests.

Here is an example. Suppose the user is reading a file sequentially. We are lucky and the file is not fragmented on the media. Suppose LEB 25 contains data nodes belonging to this file, and the data nodes are logically (in terms of logical file offset) and physically (in terms of LEB/offset addresses) sequential. Suppose user requests to read data node at LEB 25 offset 0. In this case UBIFS will actually read whole LEB 25 at one go, then populate the file cache with all the read data. And when the user asks the next piece of data, it will already be in the cache.

Obviously, the bulk-read feature may slow UBIFS down in some work-loads, so you should be careful. It is also worth noting that bulk-read feature cannot help on highly fragmented file-systems. Although UBIFS does not fragment file-systems (e.g., the Garbage-Collector does not re-order data nodes), but UBIFS does not try to defragment them. For example, if you write a file sequentially, it won't be fragmented. But if you write more than one file at a time, they may become fragmented (well, this also depends on how write-back flushes the changes), and UBIFS won't automatically defragment them. However, it is possible to implement a background defragmenter. It is also possible to have per-inode journal head and avoid mixing data nodes belonging to different inodes in the same LEB. So there is room for improvements.

Space for superuser

UBIFS reserves some space for the superuser (root), which means that when the file-system is full for normal users, there is still little space for the super-user. File-systems like ext2 have a similar feature. mkfs.ubifs has -R option which may be used to specify the reserved space size.

By default only root may use the reserved space. But it is possible to extend the list of power users who are able to utilize the reserved space. UBIFS may record several user and group IDs at the superblock and allow them to utilize the reserved space as well. However, current mkfs.ubifs utility does not have corresponding command line options, but it should be trivial to implement them.

Note, UBIFS prints the amount of reserved space when mounts the file-system. See UBIFS messages in the system log.

Extended attributes

UBIFS supports extended attributes if the corresponding configuration option is enabled (no additional mount options are required). It supports the user, trusted, and security name-spaces. However, access control lists (ACL) support is not implemented.

Note, currently mkfs.ubifs ignores extended attributes and does not write them to the target file-system image.

Mount options

The following are UBIFS-specific mount options.

• chk_data_crc (default) - check data CRC-32 checksums;
• no_chk_data_crc - do not check data CRC-32 checksums, see this section for more details;
• bulk_read - enable bulk-read, see here;
• no_bulk_read (default) - do not bulk-read.

Example:

$ mount -o no_chk_data_crc /dev/ubi0_0 /mnt/ubifs

mounts UBIFS file-system to /mnt/ubifs and disables data CRC checking.

Besides, UBIFS supports the standard sync mount option which may be used to disable UBIFS write-back and write-buffer caching and make it fully synchronous. Note, UBIFS does not support "atime", so the atime mount option has no effect.

Flash space accounting issues

Traditional file systems like ext2 can easily calculate amount of free space. The calculation is usually quite precise and users are accustomed to this. However, the situation is very different in UBIFS - it cannot really report precise amount of free space which confuses users. Instead, it reports minimum amount of free space, which usually less than the real amount. Sometimes the mistake may be very high. For example, UBIFS may report (via the statfs()system call) that there is no free space, but one would still be able to write quite a lot.

To put it differently, UBIFS is often lying about the amount of free space it has. As a rule, it may fit considerably more bytes than it reports. However, it never reports more free space than it has. It reports less, and very rarely it may report the exact amount. And this is not because UBIFS authors are jerks, there are fundamental reasons for this, which are discussed below.

Effect of compression

The first factor is UBIFS on-flight compression. Users usually seem to expect that if file system reports N bytes of free space, than it is possible to create an N-byte file. And because of the compression, this is not quite true for UBIFS. Depending on how well the file data compresses, UBIFS may fit several times more than it reports.

When UBIFS calculates free space, it does not a-priori know anything about the data which is going to be written, so it cannot take into account the compression, so it always assumes the worst-case scenario when the data does not compress.

Well, this does not sound as a big issue. However, compression becomes an issue for free space reporting when compression is combined with write-back. Namely, UBIFS cannot know how well the cached dirty data would compress, and the only way to find this out is to actually compress it. See below.

Effect of write-back

Suppose there are X bytes of dirty file data in the page cache. They will be flushed to the flash media later, but they are in RAM so far. UBIFS (namely, the budgeting sub-system) has reserved X + O bytes on the flash media for this data, where O is file system overhead (e.g., the data has to be indexed, each data node has a header, etc).

The problem is that UBIFS cannot accurately calculate X and O, and it uses pessimistic worst-case calculations, so that when the cached data are flushed, they may take considerably less flash space than the reserved X + O. For example, this may lead to the following situations.

$ df
Filesystem           1K-blocks      Used Available Use% Mounted on
ubi0:ubifs               49568     49568         0 100% /mnt/ubifs
$ sync
$ df
Filesystem           1K-blocks      Used Available Use% Mounted on
ubi0:ubifs               49568     39164      7428  85% /mnt/ubifs

First time df reported zero free space, but after the sync it reported 15% free space. This is because there were a lot of cached dirty data, and UBIFS reserved all flash space for them. But once the data has reached the flash media, they took considerably less flash space.

Here are the reasons why UBIFS reserves more space than it is needed.

• One of the reasons is again related to the compression. The data are stored in the uncompressed form in the cache, and UBIFS does know how well it would compress, so it assumes the data wouldn't compress at all. However, real-life data usually compresses quite well (unless it already compressed, e.g. it belongs to a .tgz or .mp3 file). This leads to major over-estimation of the X component.
• Due to the design, UBIFS nodes never cross logical eraseblock (or LEB, see here) boundaries, so there are small spots of wasted space at the end of eraseblocks. The amount of this wasted flash space depends on the data and in which order this data has been written or changed. And traditionally UBIFS pessimistically assumes maximum possible amount of wasted space, which leads to over-estimation of the O component. See the next sub-section.

Thus, UBIFS reports more accurate free space value if it is synchronized.

Wastage

As it was mentioned above, UBIFS nodes do not cross LEB boundaries. Consider the following numbers:

• maximum UBIFS node size (non-compressed data node) is 4256 bytes;
• smallest UBIFS node size (a data node with 8 bytes of data, corresponding to a file tail) is 56 bytes;
• depending on name length, directory entry nodes take 56-304 bytes;
• typical LEB size in case of NAND flash with 128KiB physical eraseblocks and 2048 bytes NAND page is 126KiB (or 124KiB if the NAND chip does not support sub-pages, see here).

Thus, if the vast majority of nodes on the flash were non-compressed data nodes, UBIFS would waste 1344 bytes at the ends of 126KiB LEBs. But real-life data are often compressible, so data node sizes vary, and the amount of wasted space at the ends of eraseblocks varies from 0 to 4255.

UBIFS is doing some job to put small nodes like directory entries to the ends of LEBs to lessen the amount of wasted space, but it is not ideal and UBIFS still may waste unnecessarily large chunks of flash space at the ends of eraseblocks.

When reporting free space, UBIFS does not know which kind of data are going to be written to the flash media, and in which sequence. Thus, it assumes the maximum possible wastage of 4255 bytes per LEB. This calculation is too pessimistic for most real-life situations and the average real-life wastage is considerably less than 4255 bytes per LEB. However, UBIFS reports the absolute minimum amount of free space user-space applications may count on.

The above means that the larger is LEB size, the better is UBIFS free space prediction. E.g., UBIFS is better in this respect on NANDs with 128KiB eraseblock size, comparing to NANDs with 16KiB eraseblock size.

Dirty space

Dirty space is the flash space occupied by UBIFS nodes which were invalidated because they were changed or removed. For example, if the contents of a file is re-written, than corresponding data nodes are invalidated and new data nodes are written to the flash media. The invalidated nodes comprise dirty space. There are other mechanisms how dirty space appears as well.

UBIFS cannot re-use dirty space straight away, because corresponding flash areas do not contain all 0xFF bytes. Before dirty space can be re-used, UBIFS has to garbage-collect corresponding LEBs. The idea of Garbage collector which reclaims dirty space is the same as in JFFS2. Please, refer the JFFS2 design document for more information.

Roughly, UBIFS garbage collector picks a victim LEB which has some dirty space and moves valid UBIFS nodes from the victim LEB to the LEB which was reserved for GC. This produces some amount of free space at the end of the reserved LEB. Then GC pick new victim LEB, and moves the data to the reserved LEB. When the reserved LEB is full, UBIFS picks another empty LEB (e.g., the old victim which had been made free a step ago), and continues moving nodes from the victim LEB to the new reserved LEB. The process continues until a full empty LEB is produced.

UBIFS has a notion of minimum I/O unit size, which characterizes minimum amount of data which may be written to the flash (see here for more information). Typically, UBIFS works on large-page NAND flashes and min. I/O size is 2KiB.

Consider a situation when GC picks eraseblocks with less than min. I/O unit size dirty space. When all nodes from the victim LEB have been moved to the reserved LEB, the last min. I/O unit of the reserved LEB has to be written to the flash media, which means no space would be reclaimed. The reason why the last min. I/O unit of the reserved LEB has to me written immediately is because the victim LEB cannot be erased before all the moved nodes have reached the media. Indeed, otherwise an unclean reboot would result in lost data.

Well, things are actually not that simple and UBIFS GC actually tries not to waste space, but it is not always possible and UBIFS GC is far from being ideal. Anyway, what matters is that UBIFS cannot always reclaim dirty space if the amount of it is less than min. I/O unit size.

When UBIFS reports free space to the users, it treats dirty space as available for new data, because after garbage-collection dirty space becomes free space. But we have just showed, UBIFS cannot reclaim all dirty space and turn it into free space. Worse, UBIFS does not precisely know how much dirty space it can reclaim. So it again uses pessimistic calculations.

Thus, the less dirty space the FS has, and the smaller is dirty space fragmentation, the more precise is UBIFS free space reporting. In practice this means that a file system which is close to be full has less accurate free space reporting comparing to a less full file system, because this file system presumably has more dirty space.

Note, to fix this issue, UBIFS would need to run GC in statfs(), which would turn as much dirty space as possible into free space, which would result in more precise free space reporting. However, this would make statfs() very slow. Another possibility would be to implement background GC in UBIFS (just like in JFFS2), which would lessen effect of dirty space with time.

Precise index size is not known

As you probably know, UBIFS maintain the FS index on flash. The index takes some flash space. There also UBIFS journal, which contain FS data. The FS data in the journal is not indexed, which means that on-flash index does not refer it. Instead, UBIFS keeps indexing information for the journal in RAM. When the file system is mounted, UBIFS has to scan the journal and build this part of the index in RAM. So the journal is like a small JFFS2 file system inside UBIFS.

The journal becomes indexed as the result of the commit operation. During the commit UBIFS updates the on-flash index and makes it refer information in the journal. Then UBIFS picks other LEBs for the new journal, so the journal changes is position after the commit.

UBIFS maintains precise accounting of the index size. That is, UBIFS always knows how many bytes the on-flash index takes. However, UBIFS does not know precisely how much will the index grow (or shrink) after the commit. This means, it does not know whether how much will the index size change after the journal data references will be included into the on-flash index. And UBIFS again makes pessimistic calculations here and assumes worst-case scenario.

However, after the commit operation, UBIFS knows exact index size again. The sync() system not only flushes all dirty data, but it also call makes UBIFS commit the journal. This means that file system synchronization the makes free space prediction mistake lower.

It is worth noting that this is not a fundamental thing. It is just an UBIFS implementation detail. UBIFS could calculate precise index size without actually running the commit operation, but the UBIFS authors found it difficult to implement. And the effect of the index size uncertainty should be low.

Documentation

If flash file systems is a completely new area for you, it is recommended to start from learning JFFS2, because many basic ideas are the same in UBIFS. Read the JFFS2 design document.

You may find the description of main JFFS2 issues, as well as very basic UBIFS ideas in the JFFS3 design document. Remember, the document in general is old and out-of-date. We do not use the "JFFS3" name anymore, and JFFS3 was re-named to UBIFS. The document was written when UBI did not exist and the document assumes that JFFS3 is talking directly to the MTD device, just like JFFS2. However, the JFFS2 overview, JFFS3 Requirements, and Introduction to JFFS3 chapters are still mostly valid and give a good introduction into basic UBIFS ideas like wandering tree and the journal. Although please note, that the superblock description is irrelevant for UBIFS. UBIFS is based on UBI and does not need that trick. However, the superblock location idea may be used to create new scalable UBI2 layer.

This web-page as well as the UBIFS FAQ contains a plenty of UBIFS information. And you have to study UBI as well, because UBIFS depends on the services provided by the UBI layer. See the UBI documentation and UBI FAQsections.

Look at UBIFS presentation slides (ubifs.odp) which give another UBI/UBIFS overview. The slides were prepared in OpenOffice.org Impress 2.4, so you need OpenOffice to see them. The slides contain animation, so you have to watch them in "slide show" mode (use F5 key). And if you do not have any possibility to get OpenOffice, here is a pdf version, but it is very ugly because it does not store the animation and draws all animation steps at once.

There is an UBIFS white-paper document available as well. However, it might be rather difficult for newbies, so we recommend to start with the JFFS3 design document. The UBIFS white-paper gives a complete UBIFS design picture and describes the UBIFS internals. The white-paper does not contain some details which you may find at this web-page or in the UBIFS FAQ, and vice-versa.

And finally, there is UBIFS source code. The code has a great deal of comments, so we recommend to look there if you need all the details. And of course, you are welcome to ask questions at the UBIFS mailing list.

Raw flash vs. FTL devices

FTL stands for "Flash Translation Layer" and it is software which emulates a block device on top of flash hardware. At early days FTL ran on the host computer. For example, old PCMCIA flash devices were essentially raw flash devices, and the PCMCIA standard defined the media storage format for them. So the host computer had to run the FTL software driver which implemented PCMCIA FTL. However, nowadays FTL is usually firmware, and it is run by the controller which is built into the storage device. For example, if you look inside an USB flash drive, you'll find there a NAND chip (or several of them), and a micro-controller, which runs FTL firmware. Some USB flash drives are known to have quite powerful ARM processors inside. Similarly, MMC, eMMC, SD, SSD, and other FTL devices have a built-in controller which runs FTL firmware.

All FTL devices have an interface which provides block I/O access. Well, the interfaces are different and they are defined by different specifications, e.g., MMC, eMMC, SD, USB mass storage, ATA, and so on. But all of them provide block-based access to the device. By block-based access we mean that whole device is represented as an linear array of (usually 512-byte) blocks. Each block may be read or written.

Linux has an abstraction of a block device. For example hard drives are block devices. Linux has many file systems and the block I/O subsystem, which includes elevators and so on which have been created to work with block devices (historically - hard drives). So the idea is that the same software may be used with FTL devices. For example, you may use FAT file system on your MMC card, or ext3 file system or your SSD.

Although most flashes on the commodity hardware have FTL, there are systems which have bare flashes and do not use FTL. Those are mostly various handheld devices and embedded systems. Raw flash devices are very different to block devices. They have different work model, they have tighter constraints and more issues than block devices. In case of FTL devices these constraints and issues are hidden, but in case of raw flash the software has to deal with them. Please refer to this table for some more details about the difference between block devices and raw flashes.

UBIFS file system has been designed for raw flash. It doesn't work with block devices and it assumes the raw flash device model. In other words, it assumes the device has eraseblocks, which may be written to, read from, or erased. UBIFS takes care of writing all data out-of-place, doing garbage-collection and so on. UBIFS utilizes UBI, which is doing stuff like wear-leveling and bad eraseblock handling. All these things are not normally needed for block devices.

Very often people ask questions like "why would one need to use raw flash and why not just use eMMC, or something like this?". Well, there is no simple answer, and the following is what UBIFS developers think. Please, take into account the date of this writing (3 May 2009). The answer is given in form of a list of non-structured items, and the reader should structure it in a way which is appropriate for his system. And because mass storage systems mostly use NAND flash (modern FTL devices also have NAND flash arrays inside), we talk specifically about NAND flashes. Also, we'd like to emphasize that we do not give general recommendations and everything depends on system requirements.

• Bare NAND chips are cheaper and simpler, which is very important for small system. However, it seems like the industry pushes FTL devices forward and the situation is not that simple and obvious anymore. Indeed, an FTL device is more complex than a raw NAND of similar size, because an FTL device has an additional controller inside, and so on. But since the industry tends to produce a lot of FTL devices, and actually sell a lot of them, the price is going down.
• If you need a flash storage where you are going to use FAT file system, then in most cases you should stick with an FTL device (eMMC, MMC, SD or whatever). Just make sure the FTL device is doing proper wear-leveling.
• The other situation is when you are going to use your FTL device for system storage (e.g. for rootfs) and use a more robust file system like ext3. In this situation you should take into account various system requirements like tolerance to sudden power cuts. The following items are mostly related to system storage situations.
• FTL devices are "black boxes". FTL algorithms are normally vendor secrets. But we know that NAND flash has issues like wear-leveling, bad blocks handling, read-disturb and so on. And it is important to get them right, especially in case of MLC NAND flash, which may have very short eraseblock life-time (e.g., only 1000 erase-cycles). But because FTL algorithms are closed, it is difficult to be sure whether a specific FTL device gets everything right or not.
• If you start thinking about how FTL could be implemented, you realize that it must do things like garbage collection (sometimes referred to as "reclaim process"). And flash hardware pretty much requires most writes to be out-of-place. But how does FTL behave in case of sudden power-cuts? What if a power-cut happens while it is in the middle of doing garbage collection? Does the FTL device guarantee that the data which was on the flash media before the power cut happens will not disappear or become corrupted?
• The power-cut tolerance may be tested, while it is quite difficult to test stuff like wear-leveling or read-disturb handling, because it may require too much time.
• We have heard reports that some USB flash drives wear out very quickly, i.e., they start reporting I/O errors after few weeks of intensive use. This means that FTL does not do proper wear-leveling. This does not mean that all USB flash drives are bad - just that you should be careful.
• We have heard reports that MMC, eMMC, and SD cards corrupt and lose data if power is cut during writing. Even the data which were there long time before may corrupt or disappear. This means that they have bad FTL which does not do things properly. But again, this does not have to be true for all MMCs/eMMCs and SDs - there are many different vendors. Be careful, though.
• In general, if you glance back into the history, many FTL devices were mostly used with FAT file system for storing stuff like photo and video. FAT file system is not reliable by definition, which suggests that FTL devices may also be not very reliable, just because historically this was not really required. Indeed, it is not a big deal to lose a couple of photos. However, it is crucial to make sure that system libraries do not corrupt because of power-cuts.
• Good FTL must be a rather complex piece of software, especially if it deals with MLC NAND (which is used in most modern mass storage devices). Implementing it in firmware might be a difficult task, and running it might require a powerful controller. Instead, we suspect that vendors may user various tricks or compromises to keep their devices "good enough" and cheap. For example, it is known that some vendors optimize their FTL devices for FAT, and if you start using ext3 on top of one, you might face some unexpected problems or the device may become worse than you would imagine. Of course, it is often difficult to verify this with closed FTL.
• SSD drives are probably very different to eMMC, MMC/SD etc. We have not worked with SSD drives. They are expensive and probably have powerful CPUs that run complex firmware which probably gets things right.
• FTL devices are becoming more popular and better, although it is not easy to distinguish between good and bad FTL devices (of course vendors would assure you their device is perfect). Generally, there is nothing wrong in using an FTL device as long as you trust it, you have tested it, or it simply fits your system requirements.
• In case of raw flash we know exactly what we are doing. UBI/UBIFS handles all aspects of NAND flash like bad erase-blocks and wear-leveling. It guarantees power-cut tolerance. It is open and available, so you may always validate, test, and fix it. There is no lie about what it can and cannot do. On the other hand, with FTL devices you do not have much visibility to what is going on inside, vendors may lie about how good their FTL device is. If you find a bug in the firmware, vendors do not usually provide you a fast and easy way to update it, and so on.
• Theoretically, UBIFS may do a better job because it knows much more information about the files than FTL. For example, UBIFS knows about deleted files, while FTL does not, so FTL may do unneeded work trying to preserve the sectors belonging to deleted files. However, some FTL devices support "discard" requests and may benefit from the file system hints about unused sectors. Nevertheless, in general, UBIFS should do a better job on a bare NAND than a traditional FS on an FTL device with a similar NAND chip. On the other hand, FTL devices may include multiple NAND chips, highly parallelize things and provide fast I/O. SSDs are probably good examples.
• Obviously, the advantage of FTL devices is that you use old and trusted software on top of them. Be careful, though, as sometimes this may not be 100% true. UBIFS authors once tested a good brand eMMC with respect to the power cut tolerance. Some severe problems were found. Also, it was found that ext3 was not really usable with that eMMC either. What happened was that power cuts sometimes left some eMMC sectors not readable - the read operation returned ECC errors. But for ext3 read errors are fatal - it is not designed to handle them. The fsck.ext3 tool also refused to repair a file system which had unreadable sectors.

So it is indeed difficult to give an answer. Just think about cons and pros, take into account your system requirements and decide. Nonetheless, raw flashes are used, mostly in the embedded world, and this is why UBIFS has been developed.