In the Linux kernel, the following vulnerability has been resolved:

ksm: use range-walk function to jump over holes in scangetnextrmapitem

Currently, scangetnextrmapitem() walks every page address in a VMA to locate mergeable pages. This becomes highly inefficient when scanning large virtual memory areas that contain mostly unmapped regions, causing ksmd to use large amount of cpu without deduplicating much pages.

This patch replaces the per-address lookup with a range walk using walkpagerange(). The range walker allows KSM to skip over entire unmapped holes in a VMA, avoiding unnecessary lookups. This problem was previously discussed in [1].

Consider the following test program which creates a 32 TiB mapping in the virtual address space but only populates a single page:

include <unistd.h>

include <stdio.h>

include <sys/mman.h>

/* 32 TiB */ const size_t size = 32ul * 1024 * 1024 * 1024 * 1024;

int main() { char *area = mmap(NULL, size, PROTREAD | PROTWRITE, MAPNORESERVE | MAPPRIVATE | MAP_ANON, -1, 0);

    if (area == MAP_FAILED) {
            perror("mmap() failed\n");
            return -1;
    }

    /* Populate a single page such that we get an anon_vma. */
    *area = 0;

    /* Enable KSM. */
    madvise(area, size, MADV_MERGEABLE);
    pause();
    return 0;

}

$ ./ksm-sparse & $ echo 1 > /sys/kernel/mm/ksm/run

Without this patch ksmd uses 100% of the cpu for a long time (more then 1 hour in my test machine) scanning all the 32 TiB virtual address space that contain only one mapped page. This makes ksmd essentially deadlocked not able to deduplicate anything of value. With this patch ksmd walks only the one mapped page and skips the rest of the 32 TiB virtual address space, making the scan fast using little cpu.