These fonts can be huge.

A color font file is generally larger than a regular font file, and a lot more when the font embeds high-resolution bitmap characters. Whereas fonts usually weigh tens to a few hundred kilobytes, color vector fonts can reach hundreds of kilobytes to a couple megabytes depending on their visual complexity. Color bitmap fonts may range from a few megabytes to tens of megabytes, and sizes increase when multiple color font formats are embedded in a single file.

Support in different browsers.

When the browser doesn't support the file type, the font will fall back to B&W. Therefore, a fallback for color font is pretty important.

Sheets from colorfonts.wtf.
SVG is great for use in design apps, while COLR has wider browser support.

Colors are determined by the font maker.

The color scheme is hardcoded in the font. This might not be a big problem for emoji, but if you’re going to use a fancy colored font, you want it to fit the design and style of the rest of your website. One might see how Microsoft’s solution could perhaps be addressed from CSS (maybe with an extension to CSS like color-layer1: red;) — but that’s assuming the font shares one palette. If each character has its own set of colors, like emoji, this is going to get messy fast. And changing color of PNGs or SVGs seems nearly impossible. Some color fonts’ designers provide customized color choices or predesigned alternatives.

Color fonts with a customizable color palette

Render performance.

Getting that font to show up might be CPU intensive, especially on mobile. Implementations will surely be optimized over time.

Accessibility.

Color can spice up your design, and it can add meaning to text or UI elements — but only when done well. Multicolor fonts can reduce the legibility of text, especially when the human using your website is color blind. There is currently no way to force the multicolor font to use its non-color fallback.