[BIGSLEEP-434615384] cups-filters 1.x: out of bounds write in pdftoraster

Vulnerability Description

cups-filters contains backends, filters, and other software required to get the cups printing service working on operating systems other than macos. In cups-filters prior to 1.28.18, by crafting a PDF file with a large `MediaBox` value, an attacker can cause CUPS-Filter 1.x’s `pdftoraster` tool to write beyond the bounds of an array. First, a PDF with a large `MediaBox` width value causes `header.cupsWidth` to become large. Next, the calculation of `bytesPerLine = (header.cupsBitsPerPixel * header.cupsWidth + 7) / 8` overflows, resulting in a small value. Then, `lineBuf` is allocated with the small `bytesPerLine` size. Finally, `convertLineChunked` calls `writePixel8`, which attempts to write to `lineBuf` outside of its buffer size (out of bounds write). In libcupsfilters, the maintainers found the same `bytesPerLine` multiplication without overflow check, but the provided test case does not cause an overflow there, because the values are different. Commit 50d94ca0f2fa6177613c97c59791bde568631865 contains a patch, which is incorporated into cups-filters version 1.28.18.

References

• https://github.com/OpenPrinting/cups-filters/security/advisories/GHSA-893j-2wr2-wrh9
• https://github.com/OpenPrinting/cups-filters/commit/50d94ca0f2fa6177613c97c59791bde568631865
• https://github.com/OpenPrinting/cups-filters/blob/aea8d0db017e495b0204433ebdb0e86b4871094c/filter/pdftoraster.cxx#L1620
• https://github.com/OpenPrinting/cups-filters/blob/aea8d0db017e495b0204433ebdb0e86b4871094c/filter/pdftoraster.cxx#L1880
• https://github.com/OpenPrinting/libcupsfilters/blob/1dd86d835b27ed149b66aee1a4853d1db8a1f44c/cupsfilters/pdftoraster.cxx#L1790