VM disk corruption with Apple Silicon

[EdwardMoyse]

Tip

EDIT by @AkihiroSuda

For --vm-type=vz, this issue seems to have been solved in Lima v0.19 (#2026)

---

Description

Lima version: 0.18.0
macOS: 14.0 (23A344)
VM: Almalinux9

I was trying to do a big compile, using a VM with the attached configuration (vz)

NAME           STATUS     SSH                VMTYPE    ARCH       CPUS    MEMORY    DISK      DIR
myalma9        Running    127.0.0.1:49434    vz        aarch64    4       16GiB     100GiB    ~/.lima/myalma9

The build aborted with:

from /Volumes/Lima/build/build/AthenaExternals/src/Geant4/source/processes/hadronic/models/lend/src/[xDataTOM_LegendreSeries.cc](http://xdatatom_legendreseries.cc/):7:
/usr/include/bits/types.h:142:10: fatal error: /usr/include/bits/time64.h: Input/output error

And afterwards, even in a different terminal, I see:

[emoyse@lima-myalma9 emoyse]$ ls
bash: /usr/bin/ls: Input/output error

I was also logged into a display, and there I saw e.g.

If I try to log in again with:

limactl shell myalma9

each time I see something like the following appear in the display window:

[56247.6427031] Core dump to l/usr/lib/systemd/systemd-coredump pipe failed

Edit: there has been a lot of discussion below, and the corruption can happen with both vz and qemu, and on external (to the VM) and internal disks. Some permutations seem more likely to provoke a corruption than others. I have reproduced my experiments in the table in the following comment below.

[EdwardMoyse]

In case it is relevant, I was compiling in a separate APFS (Case-sensitive) Volume as described here. This volume seems absolutely fine - so the corruption seems limited to the VM itself. I can't see how this could have happened with 100GB but I wonder if it's possible that it ran out of space? I could try increasing the disk size, but the whole point of using an external volume was that this would not be necessary.

[AkihiroSuda]

• Is this specific to AlmaLinux 9?
• Is this specific to --vm-type=vz?
• Do you see some meaningful message in dmesg?

[EdwardMoyse]

Hmm. I just tried again but compiling in /tmp rather than the case-sensitive volume, and this worked fine. A colleague has confirmed a similar experience - problems with /Volumes/Lima, but works fine in /tmp. So my best guess right now is it is some interaction with an APFS Volume and Lima (which might also explain the following "stuck VM" discussion : #1666)

Answering your other questions:

• I'm not sure if it is specific to AlmaLinux9 since my use-case requires that particular OS. But I will try doing a big compile on a different OS and in a Volume and to see if I can replicate.
• We have not replicated with qemu but this is so slow that this is very hard to do. I will try.
  I will also try again with dmesg running.

[afbjorklund]

You would get much better performance with a local filesystem, as well. If you want to keep it separate from the OS image, you could add a native Lima image using the limactl disk command. And copy the results, when the build is done.

EDIT: One potential feature could be to be able create disk images on an attached disk, instead of under LIMA_HOME.
You can probably use symlinks from _disks as a workaround, but would be better with some optional flag support...

[afbjorklund]

If you really need to access 100 GiB from the host, then we might have to add some more "boring" type like NFS...
It seems like sftpd and virtiofsd, and also their Linux clients, have some stability issues when put under pressure?

[EdwardMoyse]

EDIT: One potential feature could be to be able create disk images on an attached disk, instead of under LIMA_HOME.
You can probably use symlinks from _disks as a workaround, but would be better with some optional flag support...

This would, I think, really help us.

Our use-case is this - we want to be able to edit files from within macOS, but then compile inside Almalinux9. The codebase we are compiling is relatively large (>4 million lines of C++) and can take up to 400 GB of temporary compilation space. I was reluctant to make separate VMs with this much local storage, especially since a lot of us will be working on laptops. Ideally we would have a large build area (possibly on an external drive), accessible from several VMs, and with very fast disk io to the VM (since otherwise the build time can become unusably slow). We do NOT, in general, need to be able to access this build area from the host (at least, not with fast io - it would mainly be to examine compilation failures)

(I will get back to the other tests shortly - but I'm currently travelling with limited work time, and it seems very likely that the issue is related to compiling outside the VM)

[AkihiroSuda]

(I will get back to the other tests shortly - but I'm currently travelling with limited work time, and it seems very likely that the issue is related to compiling outside the VM)

I'm not sure how virtiofs affects the XFS disk, but maybe this issue should be reported to Apple?

[afbjorklund]

I was under the impression that the problem was with the /Volumes/Lima mount, but the logs say vda2...

  - location: /Volumes/Lima
    writable: true

So the remote filesystem is a separate topic*, from this ARM64 disk corruption. Sorry for the added noise.

Though I don't see how switching from remote /Volumes/Lima to local /tmp could have helped, then...

---

* should continue in a different discussion

Note that disk images cannot be shared...
(they can be unplugged and remounted)

[AkihiroSuda]

Is this relevant?

• Apple Virtual machine keeps setting itself as read-only utmapp/UTM#4840

(UTM uses vz too)

Looks like people began to hit this issue since September, so I wonder if Apple introduced a regression on that time?

I still can't repro the issue locally though.
(macOS 14.1 on Intel MacBookPro 2020, macOS 13.5.2 on EC2 mac2-m2pro)

[AkihiroSuda]

Can anybody confirm this rumor?

utmapp/UTM#4840 (comment)

Is it me or deactivating ballooning solves the problem?
I've deactivated it two weeks ago, and no problem since on my side.

Removing these lines will disable ballooning:

lima/pkg/vz/vm_darwin.go

Lines 598 to 604 in 7cb2b2e

	configuration, err := vz.NewVirtioTraditionalMemoryBalloonDeviceConfiguration()
	if err != nil {
	return err
	}
	vmConfig.SetMemoryBalloonDevicesVirtualMachineConfiguration([]vz.MemoryBalloonDeviceConfiguration{
	configuration,
	})

[wdormann]

For what it's worth, I believe I've narrowed down the problem that I've noticed in utmapp/UTM#4840 to having used an external SSD drive. I've not reproduced the corruption if the VM lives on my Mac's internal storage.

@EdwardMoyse Your separate APFS volume... is it on the same storage device that your Mac runs on, or is it a separate external device?

@AkihiroSuda I've not seen disabling the Balloon device to help with preventing corruption. At least, if I'm working with a QEMU-based VM that lives on my external SSD storage, it has Balloon Device un-checked by default, and the VM's filesystem will eventually corrupt under heavy disk load. So I believe this is a red herring.