Find, verify, and analyze leaked credentials
Logging in sources
These rules are for code in pkg/sources. Every source should follow them, so that debug logs look the same in
every source and never leak anything sensitive. The general logging guidelines in
CONTRIBUTING.md still apply.
Levels
| Level | Use it for |
|---|---|
V(0) |
Things a user should see: scan start and end, totals, warnings. |
V(1) |
Lifecycle: source init, enumeration start and end, config choices. |
V(2) |
One line per unit: a repo, bucket, project, channel, or site. |
V(3) |
One line per item in a unit: a file, object, commit, page, or message. |
V(4) |
Decisions and API detail: skips and why, pages, retries, rate limits. |
V(5) |
Debug: request and response summaries, internal state, timings. |
Errors are logged with Error(err, ...), not with Info and an error field.
Never log
- Secrets, tokens, passwords, or API keys.
- Auth headers, or URLs with credentials in them.
- Raw chunk data.
Logs are redacted for known secrets, but that is a safety net, not permission. Debug logs may include item names, paths, and URLs, so think about what a line could contain before adding it.
Use the logger from the context
- Log with
ctx.Logger()from the context passed into the current call. - Don't store a
logr.Loggerin a struct. A stored logger misses the fields and the log level that were set on the context, including per-source debug. - Don't create
context.Background()orcontext.TODO()inside a source. Pass the context you were given. - Use key/value pairs, not format strings. Add fields that apply to a whole block of work with
context.WithValues. - If building a debug line is expensive, check
ctx.Logger().V(5).Enabled()first.
Per-source debug
Debug logging can be turned on for a single source while every other source keeps the normal log level. This works
through the context: context.WithLogLevel(ctx, log.DebugLevel) returns a context whose logger writes everything up
to V(5). Any code that logs through ctx.Logger() gets this for free, which is one more reason not to store
loggers.
Users turn it on with debug: true on a source in the config file, see the
configuration docs. The engine then passes a debug context to both Init and the scan.
The rules about stored loggers and new contexts are checked by semgrep (hack/semgrep-rules/sources.yaml). For now
the findings are warnings. If a source really needs a new context, add a // nosemgrep comment that says why.
Testing
Use the logtest helpers to run a source and check what it logged. They check what must hold for every source, not
the text or the number of lines, so renaming a log message never breaks a test:
ctx, logs := logtest.SourceContext(true) // false for a normal run
require.NoError(t, s.Init(ctx, "test source", 0, 0, false, conn, 1))
logtest.CheckInitLine(t, logs.All()) // Init logged a V(1) line
// scan with Chunks, Enumerate or ChunkUnit
logtest.CheckSourceLines(t, logs.All(), testSecret) // every line has the source fields, and no secret
With debug off, call logtest.CheckNoDebugLines to check that nothing below V(0) was written. A normal run may
log nothing, and the debug run already checks its lines, so it needs no other check.
Give the source fake data that makes it log the lines support relies on, like an API error, a retry and a skipped item, so those lines get checked too. Check real behavior, such as a skipped item not being scanned, through the chunks the source sends, not through log lines.