You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
manual: More on stores, building and mounting in the file system
Expand the manual on
1. store paths (be more explicit about store path base
names)
2. How store objects are exposed on the file system (new page and new
section of page)
3. Building derivations (lots of stuff on that page)
Copy file name to clipboardExpand all lines: doc/manual/source/glossary.md
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -35,7 +35,7 @@
35
35
36
36
A derivation can be thought of as a [pure function](https://en.wikipedia.org/wiki/Pure_function) that produces new [store objects][store object] from existing store objects.
37
37
38
-
Derivations are implemented as [operating system processes that run in a sandbox](@docroot@/store/building.md#builder-execution).
38
+
Derivations are implemented as [operating system processes that run in a sandbox](@docroot@/store/building.md).
39
39
This sandbox by default only allows reading from store objects specified as inputs, and only allows writing to designated [outputs][output] to be [captured as store objects](@docroot@/store/building.md#processing-outputs).
40
40
41
41
A derivation is typically specified as a [derivation expression] in the [Nix language], and [instantiated][instantiate] to a [store derivation].
Nix's [file system object] data model is minimal and abstract.
4
+
But to actually be used by software, file system objects need to be made available through the operating system's file system.
5
+
This is sometimes called "mounting" or "exposing" the file system object, though do note it may or may not be implemented with what the operating system calls "mounting".
6
+
7
+
[file system object]: ../file-system-object.md
8
+
9
+
## Metadata normalization
10
+
11
+
File systems typically contain other metadata that is outside Nix's data model.
12
+
To avoid this other metadata being a side channel and source of nondeterminism, Nix is careful to normalize to fixed values.
13
+
For example, on Unix, the following metadata normalization occurs:
14
+
15
+
- The creation and last modification timestamps on all files are set to Unix Epoch 1s (00:00:01 1/1/1970 UTC)
16
+
17
+
- The group is set to the default group
18
+
19
+
- The Unix mode of the file to 0444 or 0555 (i.e., read-only, with execute permission enabled if the file was originally executable).
20
+
21
+
- Any possible `setuid` and `setgid` bits are cleared.
22
+
23
+
> **Note**
24
+
>
25
+
> Setuid and setgid programs are not currently supported by Nix.
26
+
> This is because the Nix archives used in deployment have no concept of ownership information,
27
+
> and because it makes the build result dependent on the user performing the build.
28
+
29
+
> **Explanation**
30
+
>
31
+
> As discussesed before, Nix essentially shares its file system object data model with other tools like Git.
32
+
> But those tools tend to ignore this metadata in both directions --- when reading files, like Nix, but when writing files, timestamps are set organically, and the user is free to set other special perms (setuid, setgid, sticky, etc.) however they like, with the proviso that since they are ignored by Git, git will silently loose that information.
33
+
> This metadata normalization for determinism is therefore what distinguishes Nix from other tools more than the data model itself.
34
+
>
35
+
> Nix's approach is motivated by determinstic building. Whereas git can assume that humans running commands will simply ignore timestamps etc. as appropriate, understanding they are local and ephemeral, Nix aims to run software that was not necessarily designed with Nix in mind, and is unaware of whatever sandboxing/virtualization is in place.
Copy file name to clipboardExpand all lines: doc/manual/source/store/index.md
+30-1Lines changed: 30 additions & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -2,4 +2,33 @@
2
2
3
3
The *Nix store* is an abstraction to store immutable file system data (such as software packages) that can have dependencies on other such data.
4
4
5
-
There are [multiple types of Nix stores](./types/index.md) with different capabilities, such as the default one on the [local filesystem](./types/local-store.md) (`/nix/store`) or [binary caches](./types/http-binary-cache-store.md).
5
+
Concretely, albeit using concepts that are only defined in the rest of the chapter, a store consists of:
6
+
7
+
- A set of [store objects][store object], the immutable file system data.
8
+
9
+
This can also be looked at as a map from [store paths][store path] to store objects.
10
+
11
+
- A set of [derivations][derivation], instructions for building store objects.
12
+
13
+
This can also be looked at as a map from [store paths][store path] to derivations.
14
+
Since store paths to derivations always end in `.drv`, and store paths to other store objects never do, the two maps can also be combined into one.
15
+
Derivations can also be encoded as store objects too.
16
+
17
+
- A [build trace], a record of which derivations have been built and what they produced.
There are [multiple types of Nix stores][store type] with different capabilities, such as the default one on the [local file system][local store] (`/nix/store`) or [binary caches][binary cache].
Nix's [store object] and [file system object] data models are minimal and abstract.
4
+
But to actually be used by software, store objects need to be made available through the operating system's file system.
5
+
6
+
This is done by exposing all the store objects in a single *[store directory]*.
7
+
Every entry in that directory has a *[store path base name]* as the name, and the file system object of one of the store objects in the store as the child file.
8
+
Store objects exposed in this way can then be referenced by *[store paths][store path]*.
9
+
10
+
[store object]: ./store-object.md
11
+
[file system object]: ./file-system-object.md
12
+
[store path]: #store-path
13
+
[store path base name]: #store-path-base-name
14
+
[store directory]: #store-directory-path
8
15
9
-
Nix implements references to [store objects](./store-object.md) as *store paths*.
16
+
## Store Path Base Name
10
17
11
-
Think of a store path as an [opaque], [unique identifier]:
12
-
The only way to obtain store path is by adding or building store objects.
13
-
A store path will always reference exactly one store object.
18
+
Nix implements references to store objects as *store path base names*.
19
+
20
+
Think of a store path base name as an [opaque], [unique identifier]:
21
+
The only way to obtain a store path base name is by adding or building store objects.
22
+
A store path base name will always reference exactly one store object.
Exactly how the digest is calculated depends on the type of store path.
51
+
[Base32]: https://en.wikipedia.org/wiki/Base32
52
+
53
+
Exactly how the digest is calculated depends on the type of store object being referenced.
47
54
Store path digests are *supposed* to be opaque, and so for most operations, it is not necessary to know the details.
48
55
That said, the manual has a full [specification of store path digests](@docroot@/protocols/store-path.md).
49
56
50
-
## Store Directory
51
-
52
-
Every [Nix store](./index.md) has a store directory.
53
-
54
-
Not every store can be accessed through the file system.
55
-
But if the store has a file system representation, the store directory contains the store’s [file system objects], which can be addressed by [store paths](#store-path).
57
+
## Store Directory Path
56
58
57
-
[file system objects]: ./file-system-object.md
59
+
Every [Nix store] has a store directory path.
60
+
This is an absolute, lexically canonical (not containing any `..`, `.`, or similar) path which points to the directory where all store objects are to be found.
58
61
59
-
This means a store path is not just derived from the referenced store object itself, but depends on the store that the store object is in.
62
+
[Nix store]: ./index.md
60
63
61
64
> **Note**
62
65
>
63
66
> The store directory defaults to `/nix/store`, but is in principle arbitrary.
64
67
65
-
It is important which store a given store object belongs to:
68
+
## Store Path
69
+
70
+
A store path is the pair of a store directory path and a [store path base name].
71
+
It is rendered to a file system path as the concatenation of
When we have fixed a given store, or given store directory path (that all the stores in use share), the abstract syntax for store paths and the abstract syntax for store path base names coincide: the store directory path is known from context, so only the other two fields vary from one store path to the next.
86
+
87
+
## Exposing Store Objects in OS File Systems {#exposing}
88
+
89
+
Not every store can be accessed through the file system.
90
+
But if the store has a file system representation, the following should be true:
91
+
92
+
- The store directory path is canonical: no prefix of the path (i.e. path of the first *n* path segments) points to a symlink.
93
+
(This is a separate condition in addition to "lexical canonicity", which is a property of just the path itself, whereas regular "canonicity" is an additional property about the path and the filesystem it navigates jointly.)
94
+
95
+
- The store directory path in fact points to a directory.
96
+
97
+
- The store directory contains, for every store object in the store, the [file system object] of that store object at the (rendered) [store path base name].
98
+
The permissions and other metadata for these files in the store directory is in the normal form described in [Exposing in OS file systems](./file-system-object/os-file-system.md).
99
+
100
+
The above properties mean that the following file accesses will work.
101
+
Suppose we have a store available on the file system per the above rules, and `b6gvzjyb2pg0kjfwrjmg1vfhh54ad73z-firefox-33.1` is the store path base name of a store object in that store.
102
+
103
+
- Suppose that the store directory (path) is `/foo/bar`.
104
+
Then, `/foo/bar/b6gvzjyb2pg0kjfwrjmg1vfhh54ad73z-firefox-33.1` exists and is the file system object of that store object.
105
+
106
+
- Suppose that we don't know what the store directory path of the store is, but we do have a capability `storeDir` to the store directory on the file system.
107
+
(This would be a "file descriptor" on Unix, or a "file handle" on Windows.)
will succeed (so long as the file system object is not a symlink), and the yielded capability will point to the file system object of that store object.
113
+
114
+
(The behavior for symlinks is harder to specify because of limitations in POSIX.)
115
+
116
+
## Relocating store objects
117
+
118
+
The inclusion of the store directory path in the full rendered store path means that it is not just derived from the referenced store object itself, but depends on the store that the store object is in.
119
+
(And actually, many of the ways of computing the digest also depend on the store directory path.
120
+
So this is also true even just for store path base names, in general.)
121
+
122
+
It is therefore important to consider which store a given store object belongs to:
66
123
Files in the store object can contain store paths, and processes may read these paths.
67
124
Nix can only guarantee referential integrity if store paths do not cross store boundaries.
68
125
69
-
Therefore one can only copy store objects to a different store if
126
+
One can only copy store objects to a different store if
Copy file name to clipboardExpand all lines: src/libstore/include/nix/store/globals.hh
+1-1Lines changed: 1 addition & 1 deletion
Original file line number
Diff line number
Diff line change
@@ -189,7 +189,7 @@ public:
189
189
0,
190
190
"cores",
191
191
R"(
192
-
Sets the value of the `NIX_BUILD_CORES` environment variable in the [invocation of the `builder` executable](@docroot@/store/building.md#builder-execution) of a derivation.
192
+
Sets the value of the `NIX_BUILD_CORES` environment variable in the [invocation of the `builder` executable](@docroot@/store/building.md#env-vars) of a derivation.
193
193
The `builder` executable can use this variable to control its own maximum amount of parallelism.
0 commit comments