Preproduction is where configuration errors, missing credentials, and unsafe test integrations should be found before release. This guide presents a repeatable workflow for managing preproduction secrets and configuration: choose an appropriate storage method, separate environment-specific values, validate changes in CI/CD, limit access, protect logs, and maintain a review cadence that keeps the environment trustworthy.
Overview
Preproduction should resemble production closely enough to provide meaningful release confidence, but it should not become a second copy of production data or credentials. The safest operating model treats configuration as a controlled dependency of the application, with clear ownership, defined access, and an auditable path from change request to deployment.
Start by separating three categories:
- Non-sensitive configuration: values such as feature flags, service URLs, timeouts, region names, and resource limits. These may be stored in version control when they contain no confidential information.
- Secrets: passwords, private keys, API tokens, signing keys, database credentials, and other values that could enable access or impersonation.
- Environment metadata: information about which services, queues, databases, certificates, and external integrations preproduction uses. This should be documented even when the underlying values are protected.
Use the environment strategy documented for your team to define how preproduction differs from staging and production. The goal is not identical values; it is predictable behavior with deliberate differences. A preproduction database may be separate, test data may be anonymized, and external integrations may use sandbox accounts, but those choices should be explicit and reviewed.
For broader release preparation, pair this workflow with the preproduction environment checklist and the guidance on staging versus preproduction versus production.
What to track
Choose the storage boundary
Different storage options solve different problems. Select based on access control, auditability, rotation, deployment integration, and the number of environments involved.
| Storage method | Good fit | Controls to require | Limitations to plan for |
|---|---|---|---|
| Encrypted repository secrets | Small teams and straightforward CI/CD credentials | Protected branches, limited maintainers, environment-specific scopes, and masked logs | Rotation and access review may depend heavily on repository administration |
| Cloud secret manager | Teams operating multiple services, accounts, or environments | Workload identity, least-privilege policies, audit records, versioning, and rotation procedures | Requires initial integration and clear ownership of identities and policies |
| Kubernetes Secrets | Values that workloads need inside a Kubernetes deployment | Restricted namespace access, encrypted storage where supported, carefully scoped service accounts, and controlled manifests | Base Kubernetes Secret objects are not a complete secrets-management strategy by themselves |
Do not commit plaintext secrets to application repositories, container images, build artifacts, tickets, or chat transcripts. Removing a value from the latest commit does not guarantee that it has disappeared from history or from logs. If a secret is exposed, treat it as compromised and rotate or revoke it according to the credential type.
Maintain a configuration inventory
A lightweight inventory makes ownership and review visible. Keep sensitive values out of the inventory itself; record references, not secret contents.
| Field | Example entry |
|---|---|
| Service | checkout-api |
| Variable or reference | PAYMENTS_API_TOKEN |
| Classification | Secret |
| Environment | Preproduction |
| Source | Cloud secret manager, preproduction path |
| Consumer | Deployment service account |
| Owner | Payments platform team |
| Rotation expectation | Defined by credential owner and provider requirements |
| Last verified | Date and reviewer |
| Failure or rollback note | Previous version retained for controlled rollback |
For each variable, document whether it is required, its expected format, its default behavior when absent, and whether a restart is needed after a change. This prevents a common failure mode in which a value exists but is malformed, loaded into the wrong process, or changed without restarting the consumer.
Cadence and checkpoints
Secret management is not a one-time setup task. Build checks into the normal CI/CD pipeline and schedule periodic reviews.
At pull-request review
- Confirm that no secret value, token-like string, private key, or sensitive connection string is included in code, test fixtures, manifests, or sample files.
- Review changes to configuration names, required variables, default values, and environment mappings.
- Require an owner to approve changes that affect access, external integrations, database connections, or deployment identity.
- Check that validation tests cover missing, malformed, and incompatible configuration.
At every preproduction deployment
- Resolve configuration from the intended environment rather than copying a production export.
- Verify that the deploying identity can read only the secrets required for that service and environment.
- Run a startup or configuration validation step before sending traffic to the new version.
- Confirm that logs, traces, error reports, and diagnostic output mask sensitive fields.
- Record the configuration version or reference used, without recording secret contents.
Monthly or quarterly
Use a recurring checkpoint to review the inventory, unused credentials, access grants, service accounts, certificate or token expiration dates, and rotation status. The exact interval should reflect credential sensitivity, organizational controls, and the ability to respond to changes. A quarterly review may be reasonable for stable low-risk integrations, while frequently changing or high-impact credentials may need more frequent attention.
Include a dry run of emergency access and rollback. The team should know who can revoke a credential, restore a known-good configuration reference, redeploy the prior application version, and communicate the impact. Review the configuration drift detection guide when comparing declared configuration with what environments actually use.
How to interpret changes
A configuration change is not automatically safe because the application starts successfully. Interpret changes through four questions:
- What changed? Identify the variable, secret version, policy, manifest, or external dependency that changed.
- Who or what can use it now? Check workload identity, namespace permissions, repository roles, and CI/CD environment access.
- How will failure appear? Define expected signals such as authentication errors, elevated latency, failed health checks, rejected messages, or missing feature behavior.
- How can it be reversed? Confirm whether rollback means restoring a previous secret version, reverting a configuration commit, changing a reference, or rotating a compromised credential.
Separate application defects from configuration defects during diagnosis. Compare the deployed configuration references, not secret values, between a healthy and unhealthy release. A deployment that fails before startup often points to missing variables, invalid formats, inaccessible secret paths, or identity-policy changes. A deployment that starts but fails under traffic may indicate an incorrect endpoint, incompatible credentials, a blocked network path, or a dependency configured for the wrong environment.
When a secret is suspected of appearing in logs, stop treating the log as safe evidence. Restrict access, preserve only the minimum diagnostic material, remove or redact exposed records where practical, and rotate the credential. Use the CI/CD failure troubleshooting decision tree for a log-first investigation without exposing sensitive values.
For Kubernetes, inspect references, events, pod status, service-account permissions, and rollout behavior. Avoid printing Secret objects or decoded values during troubleshooting. A failed rollout can be safer than a successful rollout using the wrong credential, so make configuration validation and health checks release gates rather than optional diagnostics. The Kubernetes deployment troubleshooting runbook provides a related operational path.
When to revisit
Revisit this workflow on a monthly or quarterly schedule and whenever the underlying system changes. The most important triggers are a new service, a new cloud account or cluster, a change in CI/CD provider, a migration to a different secret manager, a new external integration, a team or role change, or an incident involving credentials or configuration.
Also update the workflow when provider commands, identity-management features, Kubernetes behavior, encryption options, or vulnerability-prevention guidance changes. Verify examples against the versions and deployment model your team actually uses; avoid copying commands from an older runbook without checking their current effect.
Make the next review actionable:
- Export the configuration inventory without secret values.
- Mark each entry as current, unused, ownerless, expired, or requiring verification.
- Test one preproduction deployment using the documented retrieval path.
- Review access for humans, CI/CD identities, service accounts, and emergency operators.
- Confirm that logs and artifacts remain free of secret material.
- Run a controlled rollback or document why it cannot yet be run safely.
- Assign owners and due dates for every gap found.
Configuration management becomes reliable when it is observable, reviewable, and reversible. Treat preproduction secrets as operational dependencies rather than hidden implementation details, and the environment will provide stronger evidence before a release reaches production.