MarkCleaner / Field notes

Remove Invisible Characters Safely: Why Normalization Is Not Removal

If you need to remove invisible characters, do not treat normalize("NFC") or normalize("NFKC") as a remover. Normalization standardizes defined Unicode equivalences; it is a different operation from filtering code points. Inspect the actual string, keep a copy, and remove only characters whose role you understand.

Quick check: paste a sample below and compare its code points before and after NFC/NFKC. In the included test, the woman-technologist sequence keeps U+200D under both forms. That is a reproducible example, not a claim that every character behaves identically.

Normalization and removal solve different problems

Unicode normalization maps text into one of four defined forms: NFC, NFD, NFKC, or NFKD. The forms handle canonical or compatibility equivalence; they are not a request to delete every character that has no visible glyph. NFKC can also collapse some compatibility distinctions, so applying it indiscriminately can change text you meant to preserve.

A zero-width character may be invisible and still carry a layout or joining function. The Unicode Standard describes U+200D ZERO WIDTH JOINER and U+200C ZERO WIDTH NON-JOINER as layout controls in relevant contexts. Therefore, “invisible” is a visual description, not enough evidence for safe deletion.

Try a local NFC/NFKC check

The sample starts as 👩‍💻. The demo runs in this page using JavaScript strings; it does not send the text to a server. Replace it with text you are allowed to inspect, then compare the code-point lists.

Sample text Press the button to display the original, NFC, and NFKC code points.

Reproduce the core check

const sample = "👩‍💻";
const codePoints = text =>
  Array.from(text, ch =>
    "U+" + ch.codePointAt(0).toString(16).toUpperCase().padStart(4, "0")
  );

for (const form of ["NFC", "NFKC"]) {
  const normalized = sample.normalize(form);
  console.log(form, normalized === sample, codePoints(normalized));
}

For this sample, the measured result is NFC true and NFKC true; each code-point sequence is U+1F469 U+200D U+1F4BB. The same check in Node.js produced that output. The code is included so readers can repeat it in a browser console or JavaScript runtime.

Use the result to choose the next action

What the comparison shows	Safer next step
NFC/NFKC output still contains the code point	Normalization did not remove it in this string. If removal is your goal, handle that code point separately and inspect its surrounding text first.
The visible text or code-point sequence changes	Keep the original. Decide whether that specific normalization change is acceptable for the field or document before using the output.
A joiner sits inside an emoji or a script-specific sequence	Do not blanket-delete it. Compare rendering and meaning in the destination after any deliberate edit.
You cannot identify why a character is present	Leave it in place or ask the text’s owner. A scan identifies code points; it does not identify who added them or why.

A small, reversible cleanup workflow

Save an untouched copy of the exact text.
List the code points around the suspicious position; do not rely on a visual blank.
Choose normalization only when canonical or compatibility normalization is what you intend.
If you remove a character, do it in a separate copy and compare the result in its destination.
For emoji, multilingual text, source code, or identifiers, verify rendering and behavior before replacing the original.