AZL-70097

In the Linux kernel, the following vulnerability has been resolved:

btrfs: avoid potential out-of-bounds in btrfsencodefh()

The function btrfsencodefh() does not properly account for the three cases it handles.

Before writing to the file handle (fh), the function only returns to the user BTRFSFIDSIZENONCONNECTABLE (5 dwords, 20 bytes) or BTRFSFIDSIZE_CONNECTABLE (8 dwords, 32 bytes).

However, when a parent exists and the root ID of the parent and the inode are different, the function writes BTRFSFIDSIZECONNECTABLEROOT (10 dwords, 40 bytes).

If *maxlen is not large enough, this write goes out of bounds because BTRFSFIDSIZECONNECTABLEROOT is greater than BTRFSFIDSIZECONNECTABLE originally returned.

This results in an 8-byte out-of-bounds write at fid->parentrootobjectid = parentrootid.

A previous attempt to fix this issue was made but was lost.

https://lore.kernel.org/all/4CADAEEC020000780001B32C@vpn.id2.novell.com/

Although this issue does not seem to be easily triggerable, it is a potential memory corruption bug that should be fixed. This patch resolves the issue by ensuring the function returns the appropriate size for all three cases and validates that *max_len is large enough before writing any data.