JSON audit plugin

Package: github.com/go-zoox/api-gateway/plugin/jsonaudit

The gateway registers the JSON audit plugin when top-level json_audit.enable is true or any route sets json_audit.enable (same idea as rate limiting). It buffers the incoming request body (bounded) before the upstream call, then after the upstream responds checks whether the response looks like JSON. Only then it emits one structured JSON log line containing both request and response payloads (masked or plain depending on redact settings and provider defaults), suitable for compliance / security audits.

Behaviour summary

Phase	What happens
OnRequest	If path rules and sampling allow, read the client request body up to max_body_bytes, restore Body for forwarding, stash metadata + body on ctx.Request.Context().
OnResponse	Read the upstream response body (same size cap), restore Body for the client. If the response is deemed JSON-like, marshal one JSON audit line and send it to json_audit.output (output.provider, default console → app logger info).

Important: Audit lines are emitted only when the upstream response qualifies as JSON-like (see below). Non-JSON responses produce no audit record for that request.

Configuration (json_audit)

YAML json_audit maps to config.JSONAudit / route.JSONAudit (see Config API). Use the root block for defaults and optional routes[].json_audit for overrides; the plugin resolves longest-prefix route settings per request, then falls back to the global block when enable is true — one struct type, no separate jsonaudit.Config file.

YAML keys use snake_case.

Field reference

Field	Required?	Default	Summary
enable	Yes*	false	Must be true for the plugin to register.
output	No	provider: console	Nested block: provider — console (default), file, http, or database (webhook / endpoint / api map to http; db / sql map to database). For file, set file.path. For http, set http (url required; optional method, headers, timeout_seconds). For database, configure dedicated output.database (DSN or structured fields), no top-level fallback. URL DSN (postgres://, mysql://, sqlite:///) can omit engine; structured fields or non-URL DSN should set engine. If provider=database but output.database is missing, startup panics. Database sink uses github.com/go-zoox/gormx and runs startup auto-migrate. HTTP/database failures also emit the same audit line via console info.
max_body_bytes	No	1048576	Max bytes read from request/response bodies for auditing.
sample_rate	No	1	Fraction of requests to audit after path filtering; ≤0 behaves like full sampling.
sniff_json	No	true	Allow json.Valid sniff when Content-Type is not JSON.
decompress_gzip	No	true	Try gzip decompress when Content-Encoding indicates gzip.
include_paths	No	(empty)	Prefix allow list; empty means all paths (minus excludes).
exclude_paths	No	(empty)	Prefix deny list evaluated after includes.
redact	No	enable depends on provider*	Nested block: enable — redaction on/off (explicit value wins). If omitted, redaction defaults to on only for provider=console; for file/http/database it defaults to off. keys — JSON/query keys to mask; empty uses built-ins when enabled. When enable: false, headers, query, and JSON bodies are logged without masking (including Authorization).

*The plugin registers when global json_audit.enable is true or any route enables json_audit.

**redact.enable explicit value wins; when omitted, only output.provider=console defaults redaction to on.

Detailed field notes

Each snippet below is valid inside the root json_audit: block unless noted. The same keys work under routes[].json_audit for route-specific overrides (matched by longest prefix).

enable

true loads the plugin when set on the root json_audit block or on any route (same rule as rate limiting). Use enable: true on a route when that route defines json_audit overrides for its prefix; effective options per request come from the longest matching route, then fall back to the root block.

json_audit:
  enable: true

output (nested)

output.provider selects where one NDJSON line per audit event goes:

provider	Behaviour
console (default)	Emit through the app logger at info (same pipeline as other gateway logs).
file	Append to output.file.path (newline after each JSON object).
http	POST (unless overridden) the JSON bytes to output.http.url with Content-Type: application/json.
database	Insert one structured row into the auto-migrated audit table using output.database (DSN or structured fields, via gormx).

Synonyms such as webhook, endpoint, or api for provider are treated like http; db / sql are treated like database. If provider is file, http, or database, startup validates required keys: file.path, http.url, or database config. For database, use either output.database.dsn or structured fields (engine, host, port, username, password, db). Structured fields win over dsn. URL DSN (postgres://..., mysql://..., sqlite:///...) can infer engine; non-URL DSN usually needs explicit engine. Database config is read only from output.database (no top-level fallback). If provider: database is selected without any database config, startup panics. The plugin connects with gormx, runs AutoMigrate, and stores structured columns (see below). On delivery/write failures, the same line is also logged at console info so audits are not silently dropped.

Default (console) — omit output or set provider only:

json_audit:
  enable: true
  output:
    provider: console

Write to a local NDJSON file:

json_audit:
  enable: true
  output:
    provider: file
    file:
      path: /var/log/api-gateway/json-audit.ndjson

Ship to an HTTP collector:

json_audit:
  enable: true
  output:
    provider: http
    http:
      url: https://logs.example.com/ingest/json-audit
      method: POST
      headers:
        Authorization: Bearer your-ingest-token
      timeout_seconds: 8

Write to database (PostgreSQL / MySQL / SQLite):

json_audit:
  enable: true
  output:
    provider: database
    database:
      dsn: postgres://postgres:secret@127.0.0.1:5432/apigw_audit?sslmode=disable

json_audit:
  enable: true
  output:
    provider: database
    database:
      dsn: mysql://root:secret@127.0.0.1:3306/apigw_audit?charset=utf8mb4&parseTime=True&loc=Local

json_audit:
  enable: true
  output:
    provider: database
    database:
      dsn: sqlite:///var/lib/api-gateway/json-audit.sqlite

Use URL DSN (postgres://..., mysql://..., sqlite:///...) when possible; URL DSN can omit engine.

Structured database fields (without dsn):

json_audit:
  enable: true
  output:
    provider: database
    database:
      engine: postgres
      host: 127.0.0.1
      port: 5432
      username: postgres
      password: secret
      db: apigw_audit

json_audit:
  enable: true
  output:
    provider: database
    database:
      engine: mysql
      host: 127.0.0.1
      port: 3306
      username: root
      password: secret
      db: apigw_audit

json_audit:
  enable: true
  output:
    provider: database
    database:
      engine: sqlite
      db: /var/lib/api-gateway/json-audit.sqlite

Note: json_audit.output.provider: database must configure connection info inside json_audit.output.database; top-level database is not used as fallback.

Database table (json_audit_records) stores structured columns and extracts auth metadata:

• username, password: from Authorization: Basic ... when decodable.
• token: from Authorization: Bearer ....
• authorization: raw Authorization header value.
• x_api_key: from X-API-Key.
• client_id, client_secret: prefer X-Client-ID / X-Client-Secret; fallback to query client_id / client_secret only when header is missing; body is not used.

Route-only file sink (this route overrides global output for matching paths):

routes:
  - path: /billing
    json_audit:
      enable: true
      output:
        provider: file
        file:
          path: /var/log/api-gateway/billing-audit.ndjson
    backend:
      service:
        name: billing

max_body_bytes

Hard cap on bytes read from the incoming request body and upstream response body for auditing. Larger bodies are truncated; request_truncated / response_truncated in the audit JSON reflect that. Tune down on memory-sensitive hosts; default is 1 MiB.

json_audit:
  enable: true
  max_body_bytes: 262144   # 256 KiB

sample_rate

After include / exclude path checks, each remaining request is audited with probability sample_rate in (0,1]. 1 means always; values ≤ 0 are normalized to full sampling behaviour in the plugin (see Sampling). Use 0.1 for roughly 10% of traffic.

json_audit:
  enable: true
  sample_rate: 0.1

sniff_json

When true (default), a response whose Content-Type is not JSON-like may still qualify if the trimmed body passes json.Valid. Set false if you only want audits when the response declares JSON via media type.

json_audit:
  enable: true
  sniff_json: false

decompress_gzip

When true (default), if Content-Encoding indicates gzip, the plugin decompresses up to max_body_bytes for JSON detection and for the logged response.body. Disable if you never gzip JSON responses or want to avoid CPU on compressed payloads.

json_audit:
  enable: true
  decompress_gzip: false

include_paths and exclude_paths

Prefix matching on the gateway request path (ctx.Path). If include_paths is non-empty, the path must start with one of the prefixes. exclude_paths runs after includes: any matching prefix skips auditing for that request. Prefer exclude_paths for /health, metrics, or large/binary routes.

json_audit:
  enable: true
  include_paths:
    - /api/
  exclude_paths:
    - /health
    - /metrics
    - /static/

redact (nested)

redact.enable turns masking on or off (explicit value wins). If omitted, masking defaults to on only for provider=console; for file/http/database it defaults to off. When false, audit records include plaintext sensitive headers/query/JSON values — use only in locked-down environments.

redact.keys lists case-insensitive JSON object keys and query parameter names to replace with "[REDACTED]". When redaction is on and keys is empty, built-in defaults apply (password, token, …).

json_audit:
  enable: true
  redact:
    keys:
      - password
      - national_id
      - bank_account

Disable masking entirely:

json_audit:
  enable: true
  redact:
    enable: false

When is a response JSON-like?

Either:

1. Content-Type indicates JSON — contains the substring json (covers application/json, application/problem+json, application/vnd.api+json, etc.), or
2. sniff_json is true (default) and the trimmed body passes Go’s json.Valid.

Empty response bodies never qualify.

gzip

If decompress_gzip is true (default) and Content-Encoding contains gzip, the plugin attempts one-shot gzip decompression (still bounded by max_body_bytes) for detection and logging. If decompression fails, the raw bytes are used instead (JSON detection may fail).

Path filtering

Evaluation order:

1. If include_paths is non-empty, the request path must start with at least one listed prefix (prefix match).
2. exclude_paths: if the path starts with any listed prefix, auditing is skipped for that request.

Use exclude_paths for health checks, binaries, streaming endpoints, or routes with very large payloads.

Sampling

sample_rate is the fraction of requests considered for auditing after path checks:

• 1.0 or greater — consider every eligible path (default).
• Between 0 and 1 — Bernoulli sample (e.g. 0.1 ≈ 10%).
• 0 or negative — treated like 1.0 internally (audit all qualifying paths).

Skipped requests carry no audit payload for that hop.

Redaction

When masking is enabled (redact.enable=true, or omitted with provider=console), bodies are parsed as JSON when possible; matched object keys (case-insensitive, any nesting depth) are replaced by "[REDACTED]" in the logged structures. If redact.keys is empty, the plugin uses built-in defaults, including:

password, passwd, secret, token, authorization, api_key, apikey, access_token, refresh_token.

Sensitive HTTP headers (Authorization, Cookie, …) are replaced with ["[REDACTED]"] when masking is on. Query parameter names matching redact.keys (or built-ins) are masked.

Non-JSON bodies are still logged under request.body / response.body as string fragments. With masking off (explicit off, or omitted with non-console provider), parsed JSON and headers are logged without masking.

Audit log schema

Each audit line is one JSON object. With output.provider: console (default), it is emitted via ctx.Logger.Infof (level info); file / http are described in Configuration (json_audit) above.

Top-level fields

Field	Meaning
type	Always json_audit.
time	UTC wall time as string (RFC3339Nano).
timestamp	Same instant as Unix milliseconds (int), for sorting / numeric pipelines.
method, path	Same as request.method / request.path (shortcut for indexing).
remote_addr	Client RemoteAddr.
request_id	First non-empty among X-Request-ID, X-Correlation-ID, X-Trace-ID.
user_agent	User-Agent header.
response_status	Same as response.status (shortcut).
content_type	Upstream response Content-Type.
request_truncated, response_truncated	Whether body capture hit max_body_bytes.
request	See below.
response	See below.

request object

Field	Meaning
method, path	HTTP method and routed path (ctx.Path).
headers	Request headers as map[string][]string; when masking is on, known sensitive headers become ["[REDACTED]"]; with masking off (explicit off or non-console default), values are copied verbatim.
query	URL query (map[string][]string); parameter names matching redact.keys (or built-ins) are redacted when masking is on.
params	Route parameters from ctx.Params().ToMap() (map[string]any), empty object if none.
body	Request body: parsed JSON after key redaction when valid (if masking on), else raw string; with masking off, parsed JSON is unchanged.

response object

Field	Meaning
status	HTTP status code from upstream.
body	Response body: parsed JSON after redaction when valid (if masking on), else a raw string.

Collect logs with your existing pipeline (stdout, shipper, SIEM) when using output.provider: console, or ingest NDJSON from output.file.path. Avoid logging truly secret environments in plaintext.

Example: configuration and log output

Scenario

A client calls POST /api/v1/login with a JSON body that contains credentials. The upstream returns 200 and a JSON body that includes a session token. The gateway has json_audit enabled and uses default redaction keys (password, token, …) when provider=console (or when redact.enable=true is set explicitly).

Auditing runs only if this request passes path filters and sampling (here we assume it does). The plugin writes one info-level log message whose message payload is a single JSON object (see implementation: ctx.Logger.Infof("%s", …)).

Your log collector may still prefix each line with severity, timestamp, or logger name depending on Zoox / deployment settings—the example below shows only the audit JSON payload.

Sample gateway YAML (excerpt)

Minimal toggle:

port: 8080

json_audit:
  enable: true

Typical production-style snippet (narrow paths + custom redaction):

port: 8080

json_audit:
  enable: true
  max_body_bytes: 1048576
  sample_rate: 1
  sniff_json: true
  decompress_gzip: true
  include_paths:
    - /api/v1/
  exclude_paths:
    - /health
    - /metrics
  redact:
    keys:
      - password
      - secret
      - national_id

# … routes / backend / cache etc. — unchanged by json_audit …

With include_paths: [/api/v1/], a request to /api/v1/login is audited (unless excluded); GET /health is not.

Sample audit log payload

Below is pretty-printed for readability. In production it is usually emitted as one compact line.

Client request body (conceptual): {"username":"alice","password":"secret123"}
Upstream response body (conceptual): {"ok":true,"token":"eyJhbG..."}

Recorded audit object (password / token redacted in bodies; Authorization redacted in headers):

{
  "type": "json_audit",
  "time": "2026-04-19T14:32:01.234567891Z",
  "timestamp": 1776609121234,
  "method": "POST",
  "path": "/api/v1/login",
  "remote_addr": "203.0.113.50:49152",
  "request_id": "req-7f91ac",
  "user_agent": "ExampleClient/1.0",
  "response_status": 200,
  "content_type": "application/json; charset=utf-8",
  "request_truncated": false,
  "response_truncated": false,
  "request": {
    "method": "POST",
    "path": "/api/v1/login",
    "headers": {
      "Accept": ["application/json"],
      "Authorization": ["[REDACTED]"],
      "Content-Type": ["application/json"],
      "User-Agent": ["ExampleClient/1.0"],
      "X-Request-ID": ["req-7f91ac"]
    },
    "query": {
      "source": ["web"]
    },
    "params": {},
    "body": {
      "username": "alice",
      "password": "[REDACTED]"
    }
  },
  "response": {
    "status": 200,
    "body": {
      "ok": true,
      "token": "[REDACTED]"
    }
  }
}

If the client sends X-Request-ID, X-Correlation-ID, or X-Trace-ID, the first non-empty value appears in request_id (and is still listed under request.headers when present).

If either body exceeds max_body_bytes, the captured bytes are truncated and request_truncated or response_truncated is true.

When JSON parsing fails but the response is still treated as JSON-like (for example Content-Type: application/json with invalid bytes), request.body / response.body may be a string instead of an object.

Limitations

• Buffered bodies: Request and response must be buffered in memory up to max_body_bytes; streaming / very large payloads can raise memory usage or truncate.
• application/json with invalid body: Still counted as JSON-like via media type — response.body / request.body may be plain strings without structured key redaction.
• Sampling is probabilistic — not guaranteed uniform over short windows.

Related

• Plugins overview
• Configuration