UnixFS

3.1.1 File type

A File is a container over an arbitrary sized amount of bytes. Files are either single block or multi-block. A multi-block file is a concatenation of multiple child files.

{
  "Links": [{"Hash":"Qmfoo"}, {"Hash":"Qmbar"}],
  "Data": {
    "Type": "File",
    "blocksizes": [20, 30]
  }
}

3.1.2 Directory Type

A Directory, also known as folder, is a named collection of child Nodes:

• Every link in PBNode.Links is an entry (child) of the directory, and PBNode.Links[].Name gives you the name of that child.
• Duplicate names are not allowed. Therefore, two elements of PBNode.Link CANNOT have the same Name. If two identical names are present in a directory, the decoder MUST fail.

The minimum valid PBNode.Data field for a directory is as follows:

{
  "Type": "Directory"
}

The remaining relevant values are covered in Metadata.

3.1.3 Symlink type

A Symlink represents a POSIX symbolic link. A symlink MUST NOT have children.

The PBNode.Data.Data field is a POSIX path that MAY be inserted in front of the currently remaining path component stack.

3.1.4 HAMTDirectory

A HAMT Directory is a Hashed-Array-Mapped-Trie data structure representing a Directory. It is generally used to represent directories that cannot fit inside a single block. These are also known as "sharded directories:, since they allow you to split large directories into multiple blocks, known as "shards".

• decode(PBNode.Data).hashType indicates the multihash function to use to digest the path components used for sharding. It MUST be murmur3-x64-64 (0x22).
• decode(PBNode.Data).Data.Data is a bit field, which indicates whether or not links are part of this HAMT, or its leaves. The usage of this field is unknown, given that you can deduce the same information from the link names.
• decode(PBNode.Data).Data.fanout MUST be a power of two. This encodes the number of hash permutations that will be used on each resolution step. The log base 2 of the fanout indicate how wide the bitmask will be on the hash at for that step. fanout MUST be between 8 and probably 65536. .

The field Name of an element of PBNode.Links for a HAMT starts with an uppercase hex-encoded prefix, which is log2(fanout) bits wide.

3.3.1 mode

The mode is for persisting the file permissions in numeric notation [spec].

• If unspecified, this defaults to
  • 0755 for directories/HAMT shards
  • 0644 for all other types where applicable
• The nine least significant bits represent ugo-rwx
• The next three least significant bits represent setuid, setgid and the sticky bit
• The remaining 20 bits are reserved for future use, and are subject to change. Spec implementations MUST handle bits they do not expect as follows:
  • For future-proofing, the (de)serialization layer must preserve the entire uint32 value during clone/copy operations, modifying only bit values that have a well defined meaning: clonedValue = ( modifiedBits & 07777 ) | ( originalValue & 0xFFFFF000 )
  • Implementations of this spec must proactively mask off bits without a defined meaning in the implemented version of the spec: interpretedValue = originalValue & 07777

3.3.2 mtime

A two-element structure ( Seconds, FractionalNanoseconds ) representing the modification time in seconds relative to the unix epoch 1970-01-01T00:00:00Z. The two fields are:

1. Seconds ( always present, signed 64bit integer ): represents the amount of seconds after or before the epoch.
2. FractionalNanoseconds ( optional, 32bit unsigned integer ): when specified, represents the fractional part of the mtime as the amount of nanoseconds. The valid range for this value are the integers [1, 999999999].

Implementations encoding or decoding wire-representations MUST observe the following:

• An mtime structure with FractionalNanoseconds outside of the on-wire range [1, 999999999] is not valid. This includes a fractional value of 0. Implementations encountering such values should consider the entire enclosing metadata block malformed and abort the processing of the corresponding DAG.
• The mtime structure is optional. Its absence implies unspecified rather than 0.
• For ergonomic reasons, a surface API of an encoder MUST allow fractional 0 as input, while at the same time MUST ensure it is stripped from the final structure before encoding, satisfying the above constraints.

Implementations interpreting the mtime metadata in order to apply it within a non-IPFS target MUST observe the following:

• If the target supports a distinction between unspecified and 0/1970-01-01T00:00:00Z, the distinction must be preserved within the target. For example, if no mtime structure is available, a web gateway must not render a Last-Modified: header.
• If the target requires an mtime ( e.g. a FUSE interface ) and no mtime is supplied OR the supplied mtime falls outside of the targets accepted range:
  • When no mtime is specified or the resulting UnixTime is negative: implementations must assume 0/1970-01-01T00:00:00Z (note that such values are not merely academic: e.g. the OpenVMS epoch is 1858-11-17T00:00:00Z)
  • When the resulting UnixTime is larger than the targets range ( e.g. 32bit vs 64bit mismatch), implementations must assume the highest possible value in the targets range. In most cases, this would be 2038-01-19T03:14:07Z.

5.1.1 Separate Metadata Node

In this scheme, the existing Metadata message is expanded to include additional metadata types (mtime, mode, etc). It contains links to the actual file data, but never the file data itself.

This was ultimately rejected for a number of reasons:

1. You would always need to retrieve an additional node to access file data, which limits the kind of optimizations that are possible. For example, many files are under the 256 KiB block size limit, so we tend to inline them into the describing UnixFS File node. This would not be possible with an intermediate Metadata node.
2. The File node already contains some metadata (e.g. the file size), so metadata would be stored in multiple places. This complicates forwards compatibility with UnixFSv2, as mapping between metadata formats potentially requires multiple fetch operations.

5.1.2 Metadata in the Directory

Repeated Metadata messages are added to UnixFS Directory and HAMTShard nodes, the index of which indicates which entry they are to be applied to. Where entries are HAMTShards, an empty message is added.

One advantage of this method is that, if we expand stored metadata to include entry types and sizes, we can perform directory listings without needing to fetch further entry nodes (excepting HAMTShard nodes). However, without removing the storage of these datums elsewhere in the spec, we run the risk of having non-canonical data locations and perhaps conflicting data as we traverse through trees containing both UnixFS v1 and v1.5 nodes.

This was rejected for the following reasons:

1. When creating a UnixFS node, there's no way to record metadata without wrapping it in a directory.
2. If you access any UnixFS node directly by its CID, there is no way of recreating the metadata which limits flexibility.
3. In order to list the contents of a directory including entry types and sizes, you have to fetch the root node of each entry, so the performance benefit of including some metadata in the containing directory is negligible in this use case.

5.1.3 Metadata in the File

This adds new fields to the UnixFS Data message to represent the various metadata fields.

It has the advantage of being simple to implement. Metadata is maintained whether the file is accessed directly via its CID or via an IPFS path that includes a containing directory. In addition, metadata is kept small enough that we can inline root UnixFS nodes into their CIDs so that we can end up fetching the same number of nodes if we decide to keep file data in a leaf node for deduplication reasons.

Downsides to this approach are:

1. Two users adding the same file to IPFS at different times will have different CIDs due to the mtimes being different. If the content is stored in another node, its CID will be constant between the two users, but you can't navigate to it unless you have the parent node, which will be less available due to the proliferation of CIDs.
2. Metadata is also impossible to remove without changing the CID, so metadata becomes part of the content.
3. Performance may be impacted as well as if we don't inline UnixFS root nodes into CIDs, so additional fetches will be required to load a given UnixFS entry.

5.1.4 Side Trees

With this approach, we would maintain a separate data structure outside of the UnixFS tree to hold metadata.

This was rejected due to concerns about added complexity, recovery after system crashes while writing, and having to make extra requests to fetch metadata nodes when resolving CIDs from peers.

5.1.5 Side Database

This scheme would see metadata stored in an external database.

The downsides to this are that metadata would not be transferred from one node to another when syncing, as Bitswap is not aware of the database and in-tree metadata.

5.2.1 Seconds

The integer portion of UnixTime is represented on the wire using a varint encoding. While this is inefficient for negative values, it avoids introducing zig-zag encoding. Values before the year 1970 are exceedingly rare, and it would be handy having such cases stand out, while ensuring that the "usual" positive values are easily readable. The varint representing the time of writing this text is 5 bytes long. It will remain so until October 26, 3058 (34,359,738,367).

5.2.2 FractionalNanoseconds

Fractional values are effectively a random number in the range 1 to 999,999,999. In most cases, such values will exceed 2^28 (268,435,456) nanoseconds. Therefore, the fractional part is represented as a 4-byte fixed32, as per Google's recommendation.