btrfs: reject invalid reloc tree root keys with stack dump

Vulnerability Description

In the Linux kernel, the following vulnerability has been resolved: btrfs: reject invalid reloc tree root keys with stack dump [BUG] Syzbot reported a crash that an ASSERT() got triggered inside prepare_to_merge(). That ASSERT() makes sure the reloc tree is properly pointed back by its subvolume tree. [CAUSE] After more debugging output, it turns out we had an invalid reloc tree: BTRFS error (device loop1): reloc tree mismatch, root 8 has no reloc root, expect reloc root key (-8, 132, 8) gen 17 Note the above root key is (TREE_RELOC_OBJECTID, ROOT_ITEM, QUOTA_TREE_OBJECTID), meaning it's a reloc tree for quota tree. But reloc trees can only exist for subvolumes, as for non-subvolume trees, we just COW the involved tree block, no need to create a reloc tree since those tree blocks won't be shared with other trees. Only subvolumes tree can share tree blocks with other trees (thus they have BTRFS_ROOT_SHAREABLE flag). Thus this new debug output proves my previous assumption that corrupted on-disk data can trigger that ASSERT(). [FIX] Besides the dedicated fix and the graceful exit, also let tree-checker to check such root keys, to make sure reloc trees can only exist for subvolumes.

References

• https://git.kernel.org/stable/c/314135b7bae9618a317874ae195272682cf2d5d4
• https://git.kernel.org/stable/c/3ae93b316ca4b8b3c33798ef1d210355f2fb9318
• https://git.kernel.org/stable/c/84256e00eeca73c529fc6196e478cc89b8098157
• https://git.kernel.org/stable/c/6ebcd021c92b8e4b904552e4d87283032100796d