Skip to content

Find, verify, and analyze leaked credentials

3.8 KB
Raw Blame

Logging in sources

These rules are for code in pkg/sources. Every source should follow them, so that debug logs look the same in every source and never leak anything sensitive. The general logging guidelines in CONTRIBUTING.md still apply.

Levels

Level Use it for
V(0) Things a user should see: scan start and end, totals, warnings.
V(1) Lifecycle: source init, enumeration start and end, config choices.
V(2) One line per unit: a repo, bucket, project, channel, or site.
V(3) One line per item in a unit: a file, object, commit, page, or message.
V(4) Decisions and API detail: skips and why, pages, retries, rate limits.
V(5) Debug: request and response summaries, internal state, timings.

Errors are logged with Error(err, ...), not with Info and an error field.

Never log

  • Secrets, tokens, passwords, or API keys.
  • Auth headers, or URLs with credentials in them.
  • Raw chunk data.

Logs are redacted for known secrets, but that is a safety net, not permission. Debug logs may include item names, paths, and URLs, so think about what a line could contain before adding it.

Use the logger from the context

  • Log with ctx.Logger() from the context passed into the current call.
  • Don't store a logr.Logger in a struct. A stored logger misses the fields and the log level that were set on the context, including per-source debug.
  • Don't create context.Background() or context.TODO() inside a source. Pass the context you were given.
  • Use key/value pairs, not format strings. Add fields that apply to a whole block of work with context.WithValues.
  • If building a debug line is expensive, check ctx.Logger().V(5).Enabled() first.

Per-source debug

Debug logging can be turned on for a single source while every other source keeps the normal log level. This works through the context: context.WithLogLevel(ctx, log.DebugLevel) returns a context whose logger writes everything up to V(5). Any code that logs through ctx.Logger() gets this for free, which is one more reason not to store loggers.

Users turn it on with debug: true on a source in the config file, see the configuration docs. The engine then passes a debug context to both Init and the scan.

The rules about stored loggers and new contexts are checked by semgrep (hack/semgrep-rules/sources.yaml). For now the findings are warnings. If a source really needs a new context, add a // nosemgrep comment that says why.

Testing

Use the logtest helpers to run a source and check what it logged. They check what must hold for every source, not the text or the number of lines, so renaming a log message never breaks a test:

ctx, logs := logtest.SourceContext(true) // false for a normal run
require.NoError(t, s.Init(ctx, "test source", 0, 0, false, conn, 1))
logtest.CheckInitLine(t, logs.All()) // Init logged a V(1) line

// scan with Chunks, Enumerate or ChunkUnit

logtest.CheckSourceLines(t, logs.All(), testSecret) // every line has the source fields, and no secret

With debug off, call logtest.CheckNoDebugLines to check that nothing below V(0) was written. A normal run may log nothing, and the debug run already checks its lines, so it needs no other check.

Give the source fake data that makes it log the lines support relies on, like an API error, a retry and a skipped item, so those lines get checked too. Check real behavior, such as a skipped item not being scanned, through the chunks the source sends, not through log lines.