Kernel incident: Protect secrets and PII before logging

A log should explain why a sign-in failed. That needs an error code and a request ID. It does not need the API key sent with the request. Logging complete requests or configuration objects can quickly create another collection of credentials. At a logging provider, that collection can become another route to access.

The recent Kernel incident is a reason to check this boundary in your own code. Logging libraries already offer useful mechanisms. Teams should use them and test their protection against the actual output.

What Kernel has disclosed about the incident

Kernel reported the incident on 4 November. According to the company, a compromised API key provided access to its logging provider. The data there included further API keys and secrets. On 3 November, the same attacker used a compromised hosting credential to access four servers. Three ran browser sessions. Kernel reports that 0.28 percent of customers were affected and describes revoked access and rotated keys. Kernel's original report.

The following code examples are recommendations for your own applications. They do not reconstruct Kernel's implementation.

Mark sensitive data before the log call

A recommendation in a developer handbook does not prevent a new error object from containing a token tomorrow. Make the safe representation a property of the value instead.

In Go, a custom type can implement the slog.LogValuer interface. The JSON handler then resolves the value through its log representation. A short excerpt:

type Sensitive string

func (Sensitive) LogValue() slog.Value {
    return slog.StringValue("[REDACTED]")
}

logger.Info("request rejected",
    "request_id", requestID,
    "token", Sensitive(token),
)

The entry still contains the event and request ID. The token is replaced with [REDACTED].

This protection applies to that logging path. Converting the value with string(secret) first and inserting it into a message passes ordinary free text to the logger. The classification is then lost. A type like this does not automatically sanitise arbitrary nested objects either.

Central filters complement classification

A shared logger can remove known sensitive fields before output. Go provides HandlerOptions.ReplaceAttr for this. In .NET, data classifications can be handled through an enabled redaction pipeline. An attribute alone does not enable filtering.

For Node.js, Pino supports structured field paths. This example configuration removes the specified fields:

const logger = pino({
  redact: {
    paths: [
      "req.headers.authorization",
      "req.headers.cookie",
      "user.email",
      "apiKey"
    ],
    remove: true
  }
});

This is not a universal filter list. Pino's matching is case-sensitive. A field at request.headers.authorization does not match the configured path req.headers.authorization. These path rules do not detect a key embedded in the message text either.

Define the diagnostic fields allowed for each event. Add known exclusions rather than logging entire objects and hoping a later full-text search will find the secrets.

PII needs a separate decision

Personal data, often referred to as PII, is another category. An email address is not an API key. Its complete value is nevertheless often unnecessary for troubleshooting.

Decide which information each event needs. For example:

An internal identifier can still be linked to a person. Replacing a value therefore does not automatically anonymise it. The purpose, lawful basis, access rights and retention of personal data in logs need to be established. The GDPR requires data minimisation and data protection by default.

A filter for known field names can work alongside additional PII detection. Detection is another control, not a guarantee for arbitrary free text, attachments or new data formats.

Encrypt selectively when original values are needed

Some investigations have a justified need for the original value. A logger extension can encrypt selected fields with Vault Transit before output. The encrypted value is then held in the log store. This requires an integration; it does not automatically protect every logging library.

For a SIEM, we recommend separate roles:

If the SIEM decrypted everything on ingestion, raw data would again enter its indexes and exports. Depending on the purpose, internal references or separate HMAC values can support correlation. The Transit API supports HMAC, but HMAC values cannot be decrypted. Correlation must account for key versions and rotation. These values are not automatically anonymous either.

Plan for latency and failures. A synchronous Vault request for every log event creates another dependency. The Transit API supports batching; check individual item errors too. Decide whether the pipeline should buffer securely, omit sensitive fields or block the affected operation on failure. Where audit records are mandatory, the decision must account for their completeness. A silent plaintext fallback is not a safe solution.

Unnecessary passwords and access tokens should still be omitted. Encryption is for original data with a justified need; it does not replace data minimisation, retention rules or access controls.

Middleware and extensions must use the same protection

A logging wrapper, handler or processor is useful when it gives applications the same classification and output rules. Protection must apply before sensitive raw data reaches a local file, standard output or an external export. Masking values later in a search interface leaves already stored raw data untouched.

Include framework middleware, error services and plugins in the review. They may capture requests themselves or create their own logger. The Pino documentation for child loggers gives a concrete example: custom redact options override inherited rules. Setting redact: [] removes inherited protection.

A shared logger factory helps. Extensions also need clear requirements and checks that detect alternative output paths. This includes debug modes enabled only during an incident.

Protect every network hop carrying logs

Even sanitised logs may contain operational information or personal data. Unencrypted transport allows attackers with access to the network path to read it. We therefore recommend authenticated, encrypted connections for every network hop in the logging pipeline:

• Use HTTPS for HTTP-based log exports and the Vault API. Prefer TLS 1.3; any remaining TLS 1.2 use needs a configuration matching the applicable current cryptographic profile. Account for the IETF TLS recommendations and their updates.
• Validate the certificate chain and the identity of the destination. Disabling certificate or hostname verification undermines authentication of the receiver.
• Use a secure transport for syslog, such as syslog over TLS under RFC 5425, accounting for the cryptographic update in RFC 9662. Prevent automatic fallback to unencrypted UDP or TCP.
• Check the application, collector, relays and SIEM individually. After TLS termination, the next network hop needs protection too. mTLS can additionally authenticate the sending service by certificate.

Transport encryption protects the connection. Rules for log contents, access rights and storage are still required.

Test the output with synthetic data

Use clearly identifiable synthetic values for acceptance testing. Do not use real credentials or customer data.

1. Supply a test key and a synthetic email address as structured values.
2. Repeat the test through error messages, HTTP headers, URLs and bodies that your application actually processes.
3. Inspect every final output destination, including the console, files and external error or logging services.
4. Repeat the check with child loggers, new middleware and debug mode enabled.
5. Also check the remaining diagnostic value. Events, error codes and safe correlation data must still support troubleshooting.
6. Test connections to untrusted peers or destinations with mismatched names. They must be rejected; a certificate error must not trigger an unencrypted replacement transport.

This makes a failure reproducible. The mere presence of a filter configuration does not demonstrate that it works. The OWASP Logging Cheat Sheet covers both data to exclude and verification and protection of log processing.

Secrets already in logs need a response

A new filter does not remove old copies. If usable credentials are found in logs, the responsible teams need to revoke or rotate them, investigate possible access and account for other copies. These may include exports, tickets and backups. Evidence preservation and clean-up need to be coordinated.

For personal data, the responsible privacy and security teams also need to assess the incident. Finding the data alone does not establish whether notification is required. Our Incident Report helps structure findings, impacts and decisions.

Who fixes the gap and who verifies the result?

A redaction rule is implemented only when it works in the affected service. Assign the finding to an asset and an owner. Record which output paths were checked and how effectiveness was established.

For an external logging service, include access to that service in the vendor assessment.

In Kopexa, you can connect the finding with risks, measures and verification evidence. Technical filtering remains in your application and logging infrastructure.