Quick Start Guide: Delimiters | Headers | Rules

Every data cleaning pipeline rests on three pillars: how records are separated (delimiters), how columns are identified (headers), and what happens to each value (rules). This guide explains each one so you can configure your first pipeline with confidence.

Delimiters

A delimiter is the character that separates one field from another inside each row of your file. The most common is the comma (,), but tabs, semicolons (;), and pipes (|) are also used — especially in European or legacy datasets where commas appear inside values.

In PipelineConfig, you declare your delimiter explicitly. At runtime, CleanTransform hands this value to an Apache Commons CSV CSVFormat builder that also enables quote handling and ignores surrounding spaces, so values like "Amazon.in, CO" parse correctly instead of splitting on the embedded comma.

If the delimiter field is left blank, the pipeline defaults to a comma to keep processing safe.

{
  "delimiter": ",",
  "header": "index,Order ID",
  "fields_config": [ ... ]
}

Headers

The header is the first line of your CSV that names each column — for example, index,Order ID,Amount. During parsing, this line must be skipped so it doesn’t get treated as a data row. In PipelineConfig, the header field holds the exact text to skip — when a line starts with that text, CleanTransform quietly ignores it.

Once past the header, the pipeline needs to know which column index maps to which field name. That mapping lives in FieldMetadata records, where each FieldConfig.name must match a FieldMetadata.name for rules to fire. Fields present in your config but absent from metadata are treated as dynamic fields and evaluated after all metadata fields, with access to every previously resolved value.

[
  { "name": "id",       "index": 1  },
  { "name": "amount",   "index": 15 },
  { "name": "category", "index": 8  }
]

Processing Rules

Every field can carry a list of rule actions. Each action has a sequence number — actions execute in ascending order regardless of whether they belong to the cleanser, validation, or transformation family. A failed validation rejects the entire row.

Rules fall into three categories, each backed by its own utility class:

Cleansers

Sanitization rules that clean field values. All methods are thread-safe and stateless for Beam distributed execution.

• toUpperCase — Converts the field value to uppercase using locale-neutral rules.
• toLowerCase — Converts the field value to lowercase using locale-neutral rules.
• toUTF8 — Re-encodes the field value from ISO-8859-1 to UTF-8.
• formatInteger — Extracts only numeric characters. Uses O(n) iteration with zero regex overhead.
• formatAmount — Single-pass numeric parsing that handles both US (1,234.56) and Latin American (1.234,56) formats, avoiding regex and stream overhead.
• addValueIfNullOrEmpty — Fills the field with a default value when the current value is blank.
• removeNonNumeric — Strips every non-digit character from the field value.
• removeSpecialCharacters — Removes symbols and special characters, keeping only alphanumeric characters and spaces.
• emptyIfRegexMatch — Sets the field to empty if its trimmed value matches a provided regex pattern.

Validations

High-performance validation rules optimized with pre-compiled patterns and formatters to minimize CPU overhead in distributed execution.

• isAlpha — Checks if the value contains only alphabetic characters (a–z, A–Z).
• matchesRegex — Validates the field value against a dynamic regex pattern.
• isNumeric — Validates that the value contains only digits (or a valid decimal number).
• isValidDate — Validates date integrity using a round-trip check, rejecting dates that a default resolver would silently adjust (e.g., day 32 → day 31).
• isMinLength — Validates that the string length is at least a specified minimum.
• isMaxLength — Validates that the string length does not exceed a specified maximum.
• isNotFutureDate — Asserts that the date is before the current system date.
• dateIsAfter — Asserts that the field date is after a user-provided reference date.

Transformations

Transformation rules that mutate field values using parameters. All methods are thread-safe and stateless for Beam distributed execution.

• sum — Adds a numeric parameter value to the field value.
• subtractValue — Subtracts a numeric parameter value from the field value.
• divideValue — Divides the field value by a provided divisor. Returns the original value if the divisor is zero.
• replaceRegex — Replaces content in the field value based on a regex pattern and replacement string.
• transformDate — Converts date strings between formats (e.g., yyyy-MM-dd to MM/dd/yyyy). Falls back to the original input if parsing fails.
• concatFields — Concatenates multiple field values using a given delimiter. Blank values are excluded from the result.
• sumFields — Adds the values of one or more additional fields to the current field value. Non-numeric values are silently skipped.
• subtractFields — Subtracts the values of one or more additional fields from the current field value. Non-numeric values are silently skipped.

Putting It All Together

A minimal configuration looks like this — one delimiter, one header line, and a field with two actions that fire in sequence:

{
  "delimiter": ",",
  "header": "index,Order ID",
  "fields_config": [
    {
      "name": "id",
      "required": true,
      "actions": [
        { "rule": "toUpperCase", "sequence": 1 },
        { "rule": "isAlpha",     "sequence": 2 }
      ]
    }
  ]
}

Here, the id field is first uppercased (cleanser), then validated for letters only (validation). If validation fails, the entire row is rejected and routed to the error log. This intermixing of rule types is what gives CleanTransform its flexibility — you control the order.