ioutils_migration.md (20713B)

IOUtils Migration Guide

Improving performance through a new file API

What is IOUtils?

IOUtils is a privileged JavaScript API for performing file I/O in the Firefox frontend. It was developed as a replacement for OS.File, addressing bug 1231711. It is not to be confused with the unprivileged DOM File API.

IOUtils provides a minimal API surface to perform common I/O tasks via a collection of static methods inspired from OS.File. It is implemented in C++, and exposed to JavaScript via WebIDL bindings.

The most up-to-date API can always be found in IOUtils.webidl.

Differences from `OS.File`

IOUtils has a similar API to OS.File, but one should keep in mind some key differences.

No `File` instances (except `SyncReadFile` in workers)

Most of the IOUtils methods only operate on absolute path strings, and don't expose a file handle to the caller. The exception to this rule is the openFileForSyncReading API, which is only available in workers.

Furthermore, OS.File was exposing platform-specific file descriptors through the `fd` attribute. IOUtils does not expose file descriptors.

WebIDL has no `Date` type

IOUtils is written in C++ and exposed to JavaScript through WebIDL. Many uses of OS.File concern themselves with obtaining or manipulating file metadata, like the last modified time, however the Date type does not exist in WebIDL. Using IOUtils, these values are returned to the caller as the number of milliseconds since 1970-01-01T00:00:00Z. Dates can be safely constructed from these values if needed.

For example, to obtain the last modification time of a file and update it to the current time:

let { lastModified } = await IOUtils.stat(path);

let lastModifiedDate = new Date(lastModified);

let now = new Date();

await IOUtils.touch(path, now.valueOf());

Some methods are not implemented

For various reasons (complexity, safety, availability of underlying system calls, usefulness, etc.) the following OS.File methods have no analogue in IOUtils. They also will not be implemented.

void unixSymlink(in string targetPath, in string createPath)
string getCurrentDirectory(void)
void setCurrentDirectory(in string path)
object open(in string path)
object openUnique(in string path)

Errors are reported as `DOMException`s

When an OS.File method runs into an error, it will throw/reject with a custom `OS.File.Error`. These objects have custom attributes that can be checked for common error cases.

IOUtils has similar behaviour, however its methods consistently reject with a `DOMException` whose name depends on the failure:

| Exception Name | Reason for exception | | -------------- | -------------------- | | NotFoundError | A file at the specified path could not be found on disk. | | NotAllowedError | Access to a file at the specified path was denied by the operating system. | | NotReadableError | A file at the specified path could not be read for some reason. It may have been too big to read, or it was corrupt, or some other reason. The exception message should have more details. | | ReadOnlyError | A file at the specified path is read only and could not be modified. | | NoModificationAllowedError | A file already exists at the specified path and could not be overwritten according to the specified options. The exception message should have more details. | | OperationError | Something went wrong during the I/O operation. E.g. failed to allocate a buffer. The exception message should have more details. | | UnknownError | An unknown error occurred in the implementation. An nsresult error code should be included in the exception message to assist with debugging and improving IOUtils internal error handling. |

`IOUtils` is mostly async-only

OS.File provided an asynchronous front-end for main-thread consumers, and a synchronous front-end for workers. IOUtils only provides an asynchronous API for the vast majority of its API surface. These asynchronous methods can be called from both the main thread and from chrome-privileged worker threads.

The one exception to this rule is openFileForSyncReading, which allows synchronous file reading in workers.

`OS.File` vs `IOUtils`

Some methods and options of OS.File keep the same name and underlying behaviour in IOUtils, but others have been renamed. The following is a detailed comparison with examples of the methods and options in each API.

Reading a file

IOUtils provides the following methods to read data from a file. Like OS.File, they accept an options dictionary.

Note: The maximum file size that can be read is UINT32_MAX bytes. Attempting to read a file larger will result in a NotReadableError.

Promise<Uint8Array> read(DOMString path, ...);

Promise<DOMString> readUTF8(DOMString path, ...);

Promise<any> readJSON(DOMString path, ...);

// Workers only:
SyncReadFile openFileForSyncReading(DOMString path);

Options

| OS.File option | IOUtils option | Description | | ------------------ | ---------------------------- | ----------------------------------------------------------------------------------------------------------------------------- | | bytes: number? | maxBytes: number? | If specified, read only up to this number of bytes. Otherwise, read the entire file. Default is null. | | compression: 'lz4' | decompress: boolean | If true, read the file and return the decompressed LZ4 stream. Otherwise, just read the file byte-for-byte. Default is false. | | encoding: 'utf-8' | N/A; use readUTF8 instead. | Interprets the file as UTF-8 encoded text, and returns a string to the caller. |

Examples

Read raw (unsigned) byte values

`OS.File`

let bytes = await OS.File.read(path); // Uint8Array

`IOUtils`

let bytes = await IOUtils.read(path); // Uint8Array

Read UTF-8 encoded text

`OS.File`

let utf8 = await OS.File.read(path, { encoding: 'utf-8' }); // string

`IOUtils`

let utf8 = await IOUtils.readUTF8(path); // string

Read JSON file

`IOUtils`

let obj = await IOUtils.readJSON(path); // object

Read LZ4 compressed file contents

`OS.File`

// Uint8Array
let bytes = await OS.File.read(path, { compression: 'lz4' });
// string
let utf8 = await OS.File.read(path, {
    encoding: 'utf-8',
    compression: 'lz4',
});

`IOUtils`

let bytes = await IOUtils.read(path, { decompress: true }); // Uint8Array
let utf8 = await IOUtils.readUTF8(path, { decompress: true }); // string

Synchronously read a fragment of a file into a buffer, from a worker

`OS.File`

// Read 64 bytes at offset 128, workers only:
let file = OS.File.open(path, { read: true });
file.setPosition(128);
let bytes = file.read({ bytes: 64 }); // Uint8Array
file.close();

`IOUtils`

// Read 64 bytes at offset 128, workers only:
let file = IOUtils.openFileForSyncReading(path);
let bytes = new Uint8Array(64);
file.readBytesInto(bytes, 128);
file.close();

Writing to a file

IOUtils provides the following methods to write data to a file. Like OS.File, they accept an options dictionary.

Promise<unsigned long long> write(DOMString path, Uint8Array data, ...);

Promise<unsigned long long> writeUTF8(DOMString path, DOMString string, ...);

Promise<unsigned long long> writeJSON(DOMString path, any value, ...);

Options

| OS.File option | IOUtils option | Description | | -------------------- | ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | backupTo: string? | backupFile: string? | Identifies the path to backup the target file to before performing the write operation. If unspecified, no backup will be performed. Default is null. | | tmpPath: string? | tmpPath: string? | Identifies a path to write to first, before performing a move to overwrite the target file. If unspecified, the target file will be written to directly. Default is null. | | noOverwrite: boolean | mode: 'overwrite' or 'create' | 'create' mode will refuse to overwrite an existing file. Default is 'overwrite'. | | flush: boolean | flush: boolean | If true, force the OS to flush its internal buffers to disk. Default is false. | | encoding: 'utf-8' | N/A; use writeUTF8 instead. | Allows the caller to supply a string to be encoded as utf-8 text on disk. |

Examples

Write raw (unsigned) byte values

`OS.File`

let bytes = new Uint8Array();
await OS.File.writeAtomic(path, bytes);

`IOUtils`

let bytes = new Uint8Array();
await IOUtils.write(path, bytes);

Write UTF-8 encoded text

`OS.File`

let str = "";
await OS.File.writeAtomic(path, str, { encoding: 'utf-8' });

`IOUtils`

let str = "";
await IOUtils.writeUTF8(path, str);

Write A JSON object

`IOUtils`

let obj = {};
await IOUtils.writeJSON(path, obj);

Write with LZ4 compression

`OS.File`

let bytes = new Uint8Array();
await OS.File.writeAtomic(path, bytes, { compression: 'lz4' });
let str = "";
await OS.File.writeAtomic(path, str, {
    compression: 'lz4',
});

`IOUtils`

let bytes = new Uint8Array();
await IOUtils.write(path, bytes, { compress: true });
let str = "";
await IOUtils.writeUTF8(path, str, { compress: true });

Move a file

IOUtils provides the following method to move files on disk. Like OS.File, it accepts an options dictionary.

Promise<void> move(DOMString sourcePath, DOMString destPath, ...);

Options

| OS.File option | IOUtils option | Description | | -------------------- | ----------------------------------- | ---------------------------------------------------------------------------- | | noOverwrite: boolean | noOverwrite: boolean | If true, fail if the destination already exists. Default is false. | | noCopy: boolean | N/A; will not be implemented | This option is not implemented in IOUtils, and will be ignored if provided |

Example

`OS.File`

await OS.File.move(srcPath, destPath);

`IOUtils`

await IOUtils.move(srcPath, destPath);

Remove a file

IOUtils provides one method to remove files from disk. OS.File provides several methods.

Promise<void> remove(DOMString path, ...);

Options

| OS.File option | IOUtils option | Description | | ---------------------------------------------------------- | --------------------- | ---------------------------------------------------------------------------------------------------------------- | | ignoreAbsent: boolean | ignoreAbsent: boolean | If true, and the destination does not exist, then do not raise an error. Default is true. | | N/A; OS.File has dedicated methods for directory removal | recursive: boolean | If true, and the target is a directory, recursively remove the directory and all its children. Default is false. |

Examples

Remove a file

`OS.File`

await OS.File.remove(path, { ignoreAbsent: true });

`IOUtils`

await IOUtils.remove(path);

Remove a directory and all its contents

`OS.File`

await OS.File.removeDir(path, { ignoreAbsent: true });

`IOUtils`

await IOUtils.remove(path, { recursive: true });

Remove an empty directory

`OS.File`

await OS.File.removeEmptyDir(path); // Will throw an exception if `path` is not empty.

`IOUtils`

await IOUtils.remove(path); // Will throw an exception if `path` is not empty.

Make a directory

IOUtils provides the following method to create directories on disk. Like OS.File, it accepts an options dictionary.

Promise<void> makeDirectory(DOMString path, ...);

Options

| OS.File option | IOUtils option | Description | | ----------------------- | -------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | ignoreExisting: boolean | ignoreExisting: boolean | If true, succeed even if the target directory already exists. Default is true. | | from: string | createAncestors: boolean | If true, IOUtils will create all missing ancestors in a path. Default is true. This option differs from OS.File, which requires the caller to specify a root path from which to create missing directories. | | unixMode: number | permissions: unsigned long | The file mode to create the directory with. Ignored on Windows. Default is 0755. | | winSecurity | N/A | IOUtils does not support setting custom directory security settings on Windows. |

Example

`OS.File`

await OS.File.makeDir(srcPath, destPath);

`IOUtils`

await IOUtils.makeDirectory(srcPath, destPath);

Update a file's modification time

IOUtils provides the following method to update a file's modification time.

Promise<void> setModificationTime(DOMString path, optional long long modification);

Example

`OS.File`

await OS.File.setDates(path, new Date(), new Date());

`IOUtils`

await IOUtils.setModificationTime(path, new Date().valueOf());

Get file metadata

IOUtils provides the following method to query file metadata.

Promise<void> stat(DOMString path);

Example

`OS.File`

let fileInfo = await OS.File.stat(path);

`IOUtils`

let fileInfo = await IOUtils.stat(path);

Copy a file

IOUtils provides the following method to copy a file on disk. Like OS.File, it accepts an options dictionary.

Promise<void> copy(DOMString path, ...);

Options

| OS.File option | IOUtils option | Description | | ------------------------------------------------------------------- | -------------------- | -------------------------------------------------------------------| | noOverwrite: boolean | noOverwrite: boolean | If true, fail if the destination already exists. Default is false. | | N/A; OS.File does not appear to support recursively copying files | recursive: boolean | If true, copy the source recursively. |

Examples

Copy a file

`OS.File`

await OS.File.copy(srcPath, destPath);

`IOUtils`

await IOUtils.copy(srcPath, destPath);

Copy a directory recursively

`OS.File`

// Not easy to do.

`IOUtils`

await IOUtils.copy(srcPath, destPath, { recursive: true });

Iterate a directory

At the moment, IOUtils does not have a way to expose an iterator for directories. This is blocked by bug 1577383. As a stop-gap for this functionality, one can get all the children of a directory and iterate through the returned path array using the following method.

Promise<sequence<DOMString>> getChildren(DOMString path);

Example

`OS.File`

for await (const { path } of new OS.FileDirectoryIterator(dirName)) {</p>
  ...
}

`IOUtils`

for (const path of await IOUtils.getChildren(dirName)) {
  ...
}

Check if a file exists

IOUtils provides the following method analogous to the OS.File method of the same name.

Promise<boolean> exists(DOMString path);

Example

`OS.File`

if (await OS.File.exists(path)) {
  ...
}

`IOUtils`

if (await IOUtils.exists(path)) {
  ...
}

Set the permissions of a file

IOUtils provides the following method analogous to the OS.File method of the same name.

Promise<void> setPermissions(DOMString path, unsigned long permissions, optional boolean honorUmask = true);

Options

Example

`OS.File`

await OS.File.setPermissions(path, { unixMode: 0o600 });

`IOUtils`

await IOUtils.setPermissions(path, 0o600);

FAQs

Why should I use `IOUtils` instead of `OS.File`?

Bug 1231711 provides some good context, but some reasons include:

reduced cache-contention,
faster startup, and
less memory usage.

Additionally, IOUtils benefits from a native implementation, which assists in performance-related work for Project Fission.

We are actively working to migrate old code usages of OS.File to analogous IOUtils calls, so new usages of OS.File should not be introduced at this time.

Do I need to import anything to use this API?

Nope! It's available via the IOUtils global in JavaScript (ChromeOnly context).

Can I use this API from C++ or Rust?

Currently usage is geared exclusively towards JavaScript callers, and all C++ methods are private except for the Web IDL bindings. However given sufficient interest, it should be easy to expose ergonomic public methods for C++ and/or Rust.

Why isn't `IOUtils` written in Rust?

At the time of writing, support for Web IDL bindings was more mature for C++ oriented tooling than it was for Rust.

Is `IOUtils` feature complete? When will it be available?

IOUtils is considered feature complete as of Firefox 83.

	tor-browser The Tor Browser
	git clone https://git.dasho.dev/tor-browser.git
	Log \| Files \| Refs \| README \| LICENSE

tor-browser

IOUtils Migration Guide

What is IOUtils?

Differences from `OS.File`

No `File` instances (except `SyncReadFile` in workers)

WebIDL has no `Date` type

Some methods are not implemented

Errors are reported as `DOMException`s

`IOUtils` is mostly async-only

`OS.File` vs `IOUtils`

Reading a file

Options

Examples

Read raw (unsigned) byte values

Read UTF-8 encoded text

Read JSON file

Read LZ4 compressed file contents

Synchronously read a fragment of a file into a buffer, from a worker

Writing to a file

Options

Examples

Write raw (unsigned) byte values

Write UTF-8 encoded text

Write A JSON object

Write with LZ4 compression

Move a file

Options

Example

Remove a file

Options

Examples

Remove a file

Remove a directory and all its contents

Remove an empty directory

Make a directory

Options

Example

Update a file's modification time

Example

Get file metadata

Example

Copy a file

Options

Examples

Copy a file

Copy a directory recursively

Iterate a directory

Example

Check if a file exists

Example

Set the permissions of a file

Options

Example

FAQs