CVE-2023-53618

In the Linux kernel, the following vulnerability has been resolved:

btrfs: reject invalid reloc tree root keys with stack dump

[BUG] Syzbot reported a crash that an ASSERT() got triggered inside preparetomerge().

That ASSERT() makes sure the reloc tree is properly pointed back by its subvolume tree.

[CAUSE] After more debugging output, it turns out we had an invalid reloc tree:

BTRFS error (device loop1): reloc tree mismatch, root 8 has no reloc root, expect reloc root key (-8, 132, 8) gen 17

Note the above root key is (TREERELOCOBJECTID, ROOTITEM, QUOTATREE_OBJECTID), meaning it's a reloc tree for quota tree.

But reloc trees can only exist for subvolumes, as for non-subvolume trees, we just COW the involved tree block, no need to create a reloc tree since those tree blocks won't be shared with other trees.

Only subvolumes tree can share tree blocks with other trees (thus they have BTRFSROOTSHAREABLE flag).

Thus this new debug output proves my previous assumption that corrupted on-disk data can trigger that ASSERT().

[FIX] Besides the dedicated fix and the graceful exit, also let tree-checker to check such root keys, to make sure reloc trees can only exist for subvolumes.