Kernel Panic on Updating to CachyOS v7.05-2

[Neptune3013]

Issue: Kernel panic on updating CachyOS

(Kernel Panic report and update details linked)

Workaround: Switching to UKI (adding ENABLE_UKI=yes in /etc/default/limine)

Specs (after updating)

Operating System: CachyOS Linux
KDE Plasma Version: 6.6.5
KDE Frameworks Version: 6.26.0
Qt Version: 6.11.0
Kernel Version: 7.0.5-2-cachyos (64-bit)
Graphics Platform: Wayland
Processors: 12 × 13th Gen Intel® Core™ i5-13420H
Memory: 16 GiB of RAM (15.3 GiB usable)
Graphics Processor 1: Intel® Graphics
Graphics Processor 2: NVIDIA GeForce RTX 4050 Laptop GPU
Manufacturer: Acer
Product Name: Nitro ANV15-51
System Version: V1.26

Note: file checksum not disabled.

[Mintsuki]

Thanks for the report.

May I ask what makes you think that this is a Limine issue?

[Neptune3013]

I'll be honest with you, I am not sure whether it is a Limine or a CachyOS issue. I first raised the query on CachyOS subreddit. Was advised to also make a report on the Limine github so that you and the other devs are aware in case it is a Limine issue because the workaround involved switching to UKI.

[ptr1337]

Did -1 work? The only difference between -1 and -2 was that GCC was used for -2 and -1 used LLVM/Clang as compiler.

[Mintsuki]

From Limine's perspective, I see that the kernel boots until fairly late in the boot process, then fails to mount the root for some reason... I am not sure exactly how that could possibly be Limine-related, unless you have a very specific reproducer that can isolate the issue specifically to Limine.

[Zesko]

FYI:
If a whole line module_path: ... is removed manually, I see the exact same kernel panic, because the kernel does not know which root device should be used.

In this case, I suspect @Neptune3013's initramfs may not be loaded correctly when file hashes are enabled, like it gets skipped during boot.

Hopefully this helps narrow down why the initramfs is not loaded.

---

Edit:

@Neptune3013
Try downgrading mkinitcpio from v41 to 40. Does that fix the issue?

[Neptune3013]

Did -1 work? The only difference between -1 and -2 was that GCC was used for -2 and -1 used LLVM/Clang as compiler.

@ptr1337 I believe I updated directly from v7.0.3-1 to 7.0.5-2

Edit: I could be wrong, I did not check the version before updating, but this was the version of cachyos kernel on the latest snapshot I could boot into and restore.

[Neptune3013]

@Neptune3013 Try downgrading mkinitcpio from v41 to 40. Does that fix the issue?

@Zesko I downgraded it from v41-2 to v40-5 and removed the line"ENABLE_UKI=yes" from /etc/default/limine
on rebooting I was presented with a blue screen with a large qr underneath which was written:

KERNEL PANIC!
Please reboot your computer.
VFS: Unable to mount root fs on unknown-block(0,0)

Restored a previous snapshot to boot.

[Mintsuki]

Just to try to isolate things to Limine as much as possible, could you try with/without checksum? With/without secure boot, if applicable, etc.?

Because there is the suspicion that this may be in some way related to #559.

[Zesko]

Some Intel users reported a similar issue 3 months ago. It may be fixed by adding MODULES=(btrfs vmd) to /etc/mkinitcpio.conf.

[Neptune3013]

Just to try to isolate things to Limine as much as possible, could you try with/without checksum? With/without secure boot, if applicable, etc.?

Because there is the suspicion that this may be in some way related to #559.

@Mintsuki I faced no Kernel Panic on booting with ENABLE_VERIFICATION=no added to /etc/default/limine (I removed ENABLE_UKI=yes). Worked with both secure boot on and off.

Kernel: 7.0.5-2-cachyos
mkinitcpio 41-2
limine 11.4.1-1

---

To summarise, so far, adding ENABLE_UKI=yes OR ENABLE_VERIFICATION=no to /etc/default/limine has worked.

[Neptune3013]

Some Intel users reported a similar issue 3 months ago. It may be fixed by adding MODULES=(btrfs vmd) to /etc/mkinitcpio.conf.

@Zesko the link you referenced suggests the following:

edit: etc/mkinitcpio.conf: MODULES=(btrfs vmd)
run: sudo mkinitcpio -p linux-cachyos
edit: /boot/limine.conf: nvme_load=YES (into cmdline)
Activate Intel-VMD in BIOS and choose the edited boot option.

Are all the steps to be done?
Should the line nvme_load=YES be added in limine.conf or only following the first two steps are enough?

[Mintsuki]

@Neptune3013 thanks.

Do you think you could dump the actual Limine config (aka the limine.conf file somewhere in your ESP) and also send over a copy of the kernel+initramfs pair that fails to work with checksum verification enabled?

[Mintsuki]

Also please try with the latest Limine release, aka 12.2.0.

[Zesko]

Are all the steps to be done?
Should the line nvme_load=YES be added in limine.conf

Yes, I think that is related to Intel VMD. After applying the two changes, run limine-mkinitcpio

---

I found more reports with similar issues, they seem to be related to Intel CPUs and some Acer devices

https://discuss.cachyos.org/t/cant-boot-into-linux-using-stable-kernel-but-can-boot-using-lts-kernel/22707
It also looks like GRUB may be affected:

I am getting a kern/mm.c grub_memalign:552 out of memory error, then it asks me to press any key, and after that I get a blue screen.

Possible cause: an out-of-memory issue during boot or memory allocation?

Another similar report:
https://discuss.cachyos.org/t/vfs-cannot-open-root-device-cachyos-6-18-8-3-dont-see-nvme-drive/22575

That user said resetting BIOS to defaults fixed the issue.

[Neptune3013]

@Neptune3013 thanks.

Do you think you could dump the actual Limine config (aka the limine.conf file somewhere in your ESP) and also send over a copy of the kernel+initramfs pair that fails to work with checksum verification enabled?

limine.conf

Regarding Kernel and initramfs, could you specify what I need to send?

Regrding initramfs, i found a CPIO archive at /boot/<machine_id>/linux-cachyos/initramfs-linux-cachyos. Should I extract a copy and send?