MacOS and nix

[PedroRegisPOAR]

Notes about nix and MacOS and AMD and ARM

TODO: only neested virtualization on macOS did not word by default. Need to document this.
https://nixcademy.com/posts/nix-on-macos/

Some background about AMD vs ARM

Take a look at:

• God-Tier Developer Roadmap, start=883&end=901
• Is Linux an OS, a kernel or both? (Linux vs GNU/Linux) + https://github.com/apple/darwin-xnu
• How a CPU Works in 100 Seconds // Apple Silicon M1 vs Intel i9
• I use Arch on an M1 MacBook, btw
• Emulation - Computerphile, start=68&end=201
• https://opensource.apple.com/releases/
• 24C3: Inside the Mac OS X Kernel
• GentooBer #048- Darwin, o SO livre da Apple
• Apple won't like this... - Run MacOS on ANY PC
• Run macOS on Linux with 1 COMMAND
• "AArch64" and "ARM64" refer to the same thing.
• https://unix.stackexchange.com/a/755326
• It only supports Linux, not macOS (some people use brew on Linux)., about dtruss and https://ianthehenry.com/posts/how-to-learn-nix/more-flakes/
  -It only supports Linux, not macOS (some people use brew on Linux)., about dtrace
• https://stackoverflow.com/a/54134303 and https://stackoverflow.com/a/67179752 , about llvm-otool, was it renamed? nix shell nixpkgs#llvm it has llvm-objdump

UTM

It worked, but still missing a lot of documentation!

• Is there a way to ssh into the VM? utmapp/UTM#2465 (comment), by Liz Rice
• [MacOS] Start VM Headless/In Background utmapp/UTM#2280 (comment)
• MacOS could't connect with ssh to VM ubuntu utmapp/UTM#3143 (comment)
• Is there a way to ssh into the VM? utmapp/UTM#2465 (comment)
• UTM Networking Mac M1, start=874&end=1006
• [MacOS] Start VM Headless/In Background utmapp/UTM#2280 (comment)
• Linux VMs on an M1-based Mac with VScode and UTM, by Liz Rice
• Unclear when to use port forwarding and what the network modes mean utmapp/UTM#3342 (comment)
• https://docs.getutm.app/settings-qemu/qemu/#use-local-time-for-base-clock
• https://docs.getutm.app/settings-apple/virtualization/#macos-13-clipboard-sharing
• https://discourse.nixos.org/t/change-my-mind-i-still-want-single-user-on-macos/16278

Why?

I don't think this is an issue that should be fixed on our end, its more of a homebrew install issue, and homebrew specifically states that some packages may break if not installed in their default location
containers/podman#12161 (comment)

I try to install as many packages as possible through Nix, since it’s the only way to synchronize packages across all of NixOS, Arch Linux and MacOS.
From: https://www.fbrs.io/nix-overlays/

The main issue we found with using Docker locally is that docker filesharing is extremely slow, especially on MacOS. The interwebs have plenty of resources to address this problem but these approaches simply mitigate rather than resolve the underlying performance issue.

Docker performance is pretty bad for Rails development but it’s even worse for front-end apps that require a gazillion files to be loaded and written (cough cough webpack). Poor Docker performance usually leads developers to give up on Docker for their frontend - and return to a painfully slow backend development process.
https://tech.precisionnutrition.com/dev-tooling/2020/05/03/speedy-dev-envs-with-nix.html

https://www.slant.co/topics/511/~best-mac-package-managers
https://news.ycombinator.com/item?id=27825420

Setting up nix-darwin home-manager and flakes on M1 Macbook Pro OSX Monterey 12.0.1
Discontinuing support for macOS Monterey

Mitchell is a Co-founder and CTO of HashiCorp, creators of popular tools such as Terraform, Vault, Vagrant, Packer, and more.

List:

• NixOS VM on Mac Setup
• https://twitter.com/mitchellh/status/1396219104172724225
• Dev Tool Time with Mitchell Hashimoto
• https://twitter.com/CgCnu/status/1463864049007947780

TODO: watch it
this Did someone say virtual is slow?, start=76&end=181
+
Walkthrough of Nix Install and Setup on MacOS
+
This Linux PC Runs macOS Faster Than a Real Mac

TODO:

• Nix: What Even is it Though
• macOS to NixOS the Purely Functional Linux Distribution by Daniel Britten
• NixOS, nix-darwin, and home-manager
• Installing Nixos and Home manager on a Dell Xps 15 and Macbook air
• Setting up nix-darwin home-manager and flakes on M1 Macbook Pro OSX Monterey 12.0.1
• Running a NixOS VM on macOS, by tweag

The sandbox

Really watch this:

• Nix on Darwin – History, challenges, and where it's going by Dan Peebles (NixCon 2017), start=260&end=269&version=3
• Nix on Darwin – History, challenges, and where it's going by Dan Peebles (NixCon 2017), start=327&end=340&version=3
• Nix on Darwin – History, challenges, and where it's going by Dan Peebles (NixCon 2017), start=1460&end=1470&version=3

See https://discourse.nixos.org/t/nixos-virtual-machine-on-apple-m1-arm64/15167 and https://mac.getutm.app/

NixOS/nixpkgs#95903 (comment)

Bugs, many bugs

> # This works on my computer (host: Ubuntu 14.04.1, guest: MINIX 3.2.1). It's
> # not my fault if it doesn't work on yours.
https://gist.github.com/ashaindlin/ebf807cd14b82066058e#file-vm-sh-L19-L20

I tried 10 code editors, start=0&end=19

In my case after copying of the executable the bug that I wanted to debug stopped happening ..(
https://stackoverflow.com/questions/31045575/how-to-trace-system-calls-of-a-program-in-mac-os-x#comment80637364_46799054

how a silent computer bug ended 6 lives

Bugs and more bugs:

• https://github.com/NixOS/nix/pull/9860
• emacsclient does not work in Nix shells due to changed TMPDIR NixOS/nixpkgs#55201 (comment)
• Do you trust the compiler?, start=20&end=186
• NixOS: Everything Everywhere All At Once, start=120&end=164
• O caso dos 40 ms extras - Life of a Netflix Engineer (Partner), start=1008&end=1046
• Is NixOS Overrated?, start=120&end=158
• Is NixOS Overrated?, start=159&end=194
• Is NixOS Overrated?, start=416&end=431
• TODO: The White House Endorses Rust https://www.youtube.com/watch?v=hhmvZwQ9bJU&t=54s
• Por Que Estou Desistindo do Slackware, start=259&end=337

Take an look at this:

If you remove the first word from the string "hello world", what should the result be? This is the story of how we discovered that the answer could be your root password!
https://lock.cmpxchg8b.com/zenbleed.html

List:

• https://1password.community/discussion/133105/error-connecting-to-agent-permission-denied-when-forwarding-1password-ssh-agent-to-docker
• https://forums.docker.com/t/ssh-agent-use-in-dockerfile-breaks-subsequent-commands-during-build/134559/8
• https://tinyapps.org/docs/arm-vms it is at least impressive amount of carefully collected bugs
• nix-env --upgrade silently converts profile to new experimental format (on Ubuntu) NixOS/nix#6032 (comment)

Domen Kožar - Nix Is Going Mainstream (SoN2022 - public lecture series), start=480&end=516

The NixOS "shebang" and the GNU/Linux kernel:

• Thinking about that time NixOS was accidentally creating multi-megabyte shebangs: https://lwn.net/Articles/779997/ In one case I saw, the shebang was 35 megabytes due to some funny quadratic behavior.
  https://twitter.com/grhmc/status/1669157194682580993

• The case of the supersized shebang, by Jonathan Corbet and again in https://bugs.ghostscript.com/show_bug.cgi?id=699964 from https://github.com/NixOS/nixpkgs/issues/24485#issuecomment-429636677

https://www.youtube.com/embed/QkcjCRE8Vvs?start=673&end=807&version=3
+
https://discussion.fedoraproject.org/t/disabling-selinux-breaks-docker-overlayfs/20232

MWEs + how to never write bug, start=264&end=294

• mwe – Packages and image files for MWEs
• Minimum Working Example (MWE) [duplicate]
• I've just been asked to write a minimal working example (MWE), what is that?

Unfortunately there are really hard/impossible to reproduce bugs:
how to never write bug, start=294&end=319

I too faced similar issues of stuttering. This was because of the input overflowed error which came up during the streaming. If the chunk size of every communication is reduced to 512bytes the communication is coming out well as per my experience.
Ref.: https://stackoverflow.com/a/36214862/9577149

If nothing works:

• how to never write bug, start=406&end=437
• Interview with Sr. C Dev | Prime Reacts,start=230&end=263

Specific to javascript, but valid for many other situations:

• how to never write bug, start=211&end=294
• how to never write bug, start=371&end=404

This kind of bug is scary:

• GCC fails because argument list created by nix is too long NixOS/nixpkgs#41340
• https://twitter.com/grhmc/status/1669157194682580993?s=20
• macOS: unexpected EOF reading a line NixOS/nix#3605 (comment)
• macOS: unexpected EOF reading a line NixOS/nix#3605 (comment)
• https://github.com/NixOS/nixpkgs/blob/nixos-22.05/pkgs/applications/virtualization/qemu/default.nix#L263-L266
• https://www.johnbcoughlin.com/posts/nix-dynamic-linking/
• https://discourse.nixos.org/t/imagemagicks-convert-command-fails-due-to-fontconfig-error/20518/5
• podman 4.0.2 Homebrew/homebrew-core#95318 (comment)
• qemu: add 9p virtfs support on Darwin NixOS/nixpkgs#122420 (comment)
• Allowing NixOS VM's to be run on macOS NixOS/nixpkgs#108984 (comment)
• Allowing NixOS VM's to be run on macOS NixOS/nixpkgs#108984 (comment)
• macOS support? Mic92/nixos-shell#16 (comment)
• Allow installing multi-user as root on macOS and Linux NixOS/nix#6882
• https://cardano.stackexchange.com/questions/3878/sandbox-error-when-running-nix-build-on-macos
• https://youtu.be/WuWZqSSoLxY?t=480

TODO: test is

nix \
--extra-experimental-features 'nix-command flakes ca-derivations' \
realisation \
info \
github:NixOS/nixpkgs/58c85835512b0db938600b6fe13cc3e3dc4b364e#hello

this path will be fetched (0.02 MiB download, 0.11 MiB unpacked):
  /nix/store/c4n3i1nfhvcim30s4ij502cw84yl4vdn-hello-2.12.1
error: unexpected end-of-file

nix shell nixpkgs#nix-info --command nix-info --markdown

• system: "aarch64-darwin"
• host os: Darwin 22.3.0, macOS 13.2.1
• multi-user?: yes
• sandbox: no
• version: nix-env (Nix) 2.10.2
• channels(root): "nixpkgs"
• nixpkgs: /nix/var/nix/profiles/per-user/root/channels/nixpkgs

{"description":"A collection of packages for the Nix package manager","lastModified":1688392541,"locked":{"lastModified":1688392541,"narHash":"sha256-lHrKvEkCPTUO+7tPfjIcb7Trk6k31rz18vkyqmkeJfY=","owner":"NixOS","repo":"nixpkgs","rev":"ea4c80b39be4c09702b0cb3b42eab59e2ba4f24b","type":"github"},"locks":{"nodes":{"root":{}},"root":"root","version":7},"original":{"id":"nixpkgs","type":"indirect"},"originalUrl":"flake:nixpkgs","path":"/nix/store/gmdhl9qfaic5765lxw2wj6hb2ifjmhjd-source","resolved":{"lastModified":1688392541,"narHash":"sha256-lHrKvEkCPTUO+7tPfjIcb7Trk6k31rz18vkyqmkeJfY=","owner":"NixOS","repo":"nixpkgs","rev":"ea4c80b39be4c09702b0cb3b42eab59e2ba4f24b","type":"github"},"resolvedUrl":"github:NixOS/nixpkgs/ea4c80b39be4c09702b0cb3b42eab59e2ba4f24b","revision":"ea4c80b39be4c09702b0cb3b42eab59e2ba4f24b","url":"github:NixOS/nixpkgs/ea4c80b39be4c09702b0cb3b42eab59e2ba4f24b"}

nix --extra-experimental-features 'nix-command flakes ca-derivations' realisation info github:NixOS/nixpkgs/nixpkgs-unstable#python3

these 5 paths will be fetched (13.08 MiB download, 68.12 MiB unpacked):
  /nix/store/61s4l8b46g3bg9ymcji3bmjv8kzk2sms-readline-8.2p1
  /nix/store/bpz8rh8pnlm6zsh64xjk2c7hgc71f99p-gdbm-1.23
  /nix/store/f67hia73n1rf333qkwk294g44vkr6rn7-openssl-3.0.10
  /nix/store/fd7wwy1bzzpg68pd20gxb02rgwdaa41w-libxcrypt-4.4.36
  /nix/store/g5cm6iik6p4k39cj9k7a6sg2p09hl7wf-python3-3.10.12
error: unexpected end-of-file

The rosetta thing

List:

• https://xyno.space/post/nixos-utm-rosetta
• cctools ld segfaults when linking haskellPackages.Agda (2.6.2 only?) NixOS/nixpkgs#149692 (comment)
• https://evanrelf.com/building-x86-64-packages-with-nix-on-apple-silicon
• podman machine init Support emulating different architectures containers/podman#11458 (comment)

The qemu-user-static thing

• TODO: read the full article! https://codepyre.com/2019/12/arming-yourself/#required-packages-ubuntu-1804
• https://medium.com/@earielli/arm-emulation-on-x86-amazon-linux-2-1-centos-ubuntu-debian-a5a3cc4fba76

What name put here?

macOS to NixOS the Purely Functional Linux Distribution by Daniel Britten

Franz Pletz: NixOS, start=913&end=928, about homebrew vs nix

https://discourse.nixos.org/t/nixos-on-the-new-apple-macbook-m1/15715/2

• TODO: look that: https://www.reddit.com/r/NixOS/comments/rrhbkm/m1_dev_setup_using_a_nixos_virtual_machine/
• TODO: about home-manager Provide a convention for allowing MacOS Spotlight to index Applications without getting hacky NixOS/nix#7055 (comment)
• TODO: https://www.mathiaspolligkeit.com/dev/exploring-nix-on-macos/
• TODO: https://wickedchicken.github.io/post/macos-nix-setup/
• TODO: https://github.com/NixOS/nixpkgs/blob/cc5f7c2f3dd3a288c540c287c7788242a2b557c5/nixos/modules/virtualisation/qemu-vm.nix#L580-L591

Remember Mac uses zsh (since when?):

• https://stackoverflow.com/questions/61980734/deleting-or-renaming-one-of-two-instances-of-zsh-in-etc-shells
• https://stackoverflow.com/questions/76239476/how-to-send-variable-containing-password-to-subprocess#comment134446126_76239476

• It seems that upgrading macOS system will remove/reset the file /etc/zshrc that was updated by Nix during installation. https://stackoverflow.com/questions/70733236/nixos-installation-issue-command-not-found-nix#comment133743350_70822086

utmapp/UTM#2688 (comment)

MacOS Simple KVM

• macOS VM on Linux & Windows using Docker-OSX: How to run mac VMs (Monterey/Big Sur/Catalina)
• MacOS Virtual Machine Made Easy
• Fixes for NixOS and for better handling foxlet/macOS-Simple-KVM#530 (comment)

Part 1:

git clone https://github.com/adisbladis/macOS-Simple-KVM.git \
&& cd macOS-Simple-KVM \
&& git checkout de4114741e5dd0079d04a4d56d275659d0579c31 \
&& (direnv allow || nix-shell)
./jumpstart.sh --mojave \
&& test -f MyDisk.qcow2 || qemu-img create -f qcow2 MyDisk.qcow2 64G \
&& ./basic.sh

For some reason it is need to format the disk in the Disk Utility

?:

test -f MyDisk.qcow2 || qemu-img create -f qcow2 MyDisk.qcow2 64G

HEADLESS=1 MEM=1G CPUS=2 SYSTEM_DISK=MyDisk.qcow2 ./headless.sh 

About the installer

TODO: try to test that
NixOS/nix#3616 (comment)

qemu

TODO: it documents part of history: https://stackoverflow.com/a/66887947
TODO: https://www.willhaley.com/blog/debian-arm-qemu/

# nix flake metadata --no-write-lock-file 'github:Mic92/nixos-aarch64-images'
# github:Mic92/nixos-aarch64-images/c29e786f552a5c38e1dcdceb552412ad488e45d3
nix \
build \
--no-write-lock-file \
'github:Mic92/nixos-aarch64-images#aarch64Image'

nix \
build \
--no-write-lock-file \
--override-input nixpkgs github:nixos/nixpkgs/nixpkgs-unstable \
'github:Mic92/nixos-aarch64-images#aarch64Image'

nix profile install nixpkgs#qemu_full

curl -O http://ftp.us.debian.org/debian/dists/stable/main/installer-armhf/current/images/cdrom/initrd.gz \
  && curl -O http://ftp.us.debian.org/debian/dists/stable/main/installer-armhf/current/images/cdrom/vmlinuz

curl -O -L https://cdimage.debian.org/debian-cd/current/armhf/iso-dvd/debian-11.1.0-armhf-DVD-1.iso

qemu-img create -f qcow2 debian-arm.sda.qcow2 100G

qemu-system-arm \
  -m 4G \
  -machine type=virt \
  -cpu cortex-a7 \
  -smp 4 \
  -initrd "./initrd.gz" \
  -kernel "./vmlinuz" \
  -append "console=ttyAMA0" \
  -drive file="./debian-11.1.0-armhf-DVD-1.iso",id=cdrom,if=none,media=cdrom \
    -device virtio-scsi-device \
    -device scsi-cd,drive=cdrom \
  -drive file="./debian-arm.sda.qcow2",id=hd,if=none,media=disk \
    -device virtio-scsi-device \
    -device scsi-hd,drive=hd \
  -netdev user,id=net0,hostfwd=tcp::5555-:22 \
    -device virtio-net-device,netdev=net0 \
  -nographic

https://www.willhaley.com/blog/debian-arm-qemu/

file $(nix --system aarch64-darwin build --no-link --print-out-paths --print-build-logs github:NixOS/nixpkgs/nixpkgs-unstable#qemu)/bin/qemu-system-aarch64

nix --system aarch64-darwin run github:NixOS/nixpkgs/nixpkgs-unstable#qemu -- --version

qemu + NixOS => build-vm

Excelent: Running qemu-system-aarch64 with -accel hvf on aarch64-darwin

Copying files like .img .qcow2

https://serverfault.com/a/670227
https://stackoverflow.com/questions/41733410/remove-qcow2-image-files-holes

[PedroRegisPOAR]

The nixpkgs#darwin.builder

Main references:

• It is a bit old but really impressive, cross-compile a Linux kernel on Darwin using Nix
• Nixpkgs support for Linux builders running on macOS
• https://discourse.nixos.org/t/remote-building-flake-support-on-darwin/21918/6
• https://discourse.nixos.org/t/remote-building-flake-support-on-darwin/21918/7

nix show-derivation nixpkgs#darwin.builder

nix show-derivation --system aarch64-darwin nixpkgs#darwin.builder | jq '.[]."inputDrvs"' -r

nix build -L --system aarch64-darwin nixpkgs#darwin.builder

nix \
--option allowed-impure-host-deps \
'/bin/sh /usr/lib/libSystem.B.dylib /usr/lib/system/libunc.dylib /dev/zero /dev/random /dev/urandom' \
build \
-L \
--system aarch64-darwin \
nixpkgs#darwin.builder

file /bin/sh

/bin/sh: Mach-O universal binary with 2 architectures: [x86_64:Mach-O 64-bit executable x86_64] [arm64e:Mach-O 64-bit executable arm64e] 
/bin/sh (for architecture x86_64):      Mach-O 64-bit executable x86_64                                                                                      
/bin/sh (for architecture arm64e):      Mach-O 64-bit executable arm64e

TODO: Name it

nix profile install nixpkgs#qemu_kvm nixpkgs#podman nixpkgs#socat

podman machine stop; \
podman machine rm --force; \
podman --log-level=trace machine init --memory=3072 --cpus=8 \
&& podman --log-level=trace machine start

socat -u OPEN:/dev/null UNIX-CONNECT:"${XDG_RUNTIME_DIR}"/tmp/podman-run-1000/podman/podman-machine-default_ready.sock

curl -v -s -X GET --unix-socket /tmp/podman-run-1000/podman/podman-machine-default_ready.sock "http:///libpod/containers/json"

qemu-system-x86_64 \
-m 3072 \
-smp 4 \
-fw_cfg name=opt/com.coreos/config,file=/home/vagrant/.config/containers/podman/machine/qemu/podman-machine-default.ign \
-qmp unix:/tmp/podman-run-$(id -u)/podman/qmp_podman-machine-default.sock,server=on,wait=off \
-netdev socket,id=vlan,fd=3 \
-device virtio-net-pci,netdev=vlan,mac=5a:94:ef:e4:0c:ee \
-device virtio-serial \
-chardev socket,path=/tmp/podman-run-$(id -u)/podman/podman-machine-default_ready.sock,server=on,wait=off,id=apodman-machine-default_ready \
-device virtserialport,chardev=apodman-machine-default_ready,name=org.fedoraproject.port.0 \
-pidfile /tmp/podman-run-$(id -u)/podman/podman-machine-default_vm.pid \
-accel kvm \
-cpu host \
-virtfs local,path=/home/vagrant,mount_tag=vol0,security_model=none \
-drive if=virtio,file=/home/vagrant/.local/share/containers/podman/machine/qemu/podman-machine-default_fedora-coreos-37.20230303.2.0-qemu.x86_64.qcow2 \
 -nographic &

podman system service --time=0 unix:///tmp/podman.sock &

curl -s --unix-socket /tmp/podman.sock http://d/v1.0.0/libpod/info

Refs.:

• https://medium.com/devops-dudes/how-to-setup-root-less-podman-containers-efd109fa4e0d

NixOS build-vm

--memory=3072 --cpus=8

podman machine stop; \
podman machine rm --force; \
podman --log-level=trace machine init --image-path $(show-coreos-path) \
&& podman --log-level=trace machine start

Alpine

podman machine stop; \
podman machine rm --force; \
podman --log-level=trace machine init \
&& sleep 30 \
&& podman --log-level=trace machine start

podman machine stop; \
podman machine rm --force; \
podman --log-level=trace machine init --memory=3072 --cpus=8 \
&& sleep 30 \
&& podman --log-level=trace machine start

mkdir -p ~/.config/containers
cat << 'EOF' >> ~/.config/containers/policy.json
{
    "default": [
        {
            "type": "insecureAcceptAnything"
        }
    ],
    "transports":
        {
            "docker-daemon":
                {
                    "": [{"type":"insecureAcceptAnything"}]
                }
        }
}
EOF

mkdir -p ~/.config/containers
cat << 'EOF' >> ~/.config/containers/registries.conf
[registries.search]
registries = ['docker.io']
[registries.block]
registries = []
EOF

Troubleshooting

podman machine list

podman machine inspect

cat /home/nixuser/.config/containers/storage.conf