By Admin A container that passes every build-time security check can still be exploited after deployment. This is not a failure of build-time security. It is the fundamental nature of a threat landscape that continues to evolve after the image is built.
Build-time security handles what is known at build time: CVEs in the package database, misconfigurations detectable by static analysis, compliance controls verifiable against the image manifest. Runtime security handles what cannot be known at build time: new CVEs disclosed after deployment, zero-day exploits without signatures, insider threats, and attack patterns that emerge from the interaction of running services.
Build-time security alone is not a complete security posture. It is half of one.
What Build-Time Security Cannot Prevent?
Post-deployment CVE disclosures: A container deployed today with zero Critical CVEs may have one tomorrow if a Critical CVE is disclosed against a package in the image. The build-time scan was accurate at build time. It is no longer accurate at some point after deployment. Build-time scanning does not detect this change; it has no mechanism to observe a running container.
Novel attack techniques: Build-time security tools check for known vulnerabilities and known misconfigurations. A new exploit technique that does not correspond to a known CVE does not generate a build-time finding. Runtime monitoring that observes behavior can detect novel attack patterns by recognizing that the container is doing something it should not do, even if the specific technique has no known signature.
Supply chain compromises that bypass build gates: A sophisticated supply chain attack that injects malicious code into the build process itself may produce an image that passes all static security checks. Runtime behavioral monitoring detects the malicious behavior when the code executes, regardless of whether build-time tools flagged the image.
Insider threats and misconfiguration in production: A legitimate container that is misconfigured at deployment time (overly permissive RBAC, unintended network access) may not trigger build-time security alerts. Runtime monitoring detects the resulting anomalous behavior.
The Defense-in-Depth Model
Effective container security combines both layers:
Build-time (prevent known risks):
- Vulnerability scanning: eliminate known CVEs before deployment
- Hardening: remove unused packages to reduce attack surface
- SBOM generation: document component inventory for ongoing monitoring
- Configuration analysis: check Dockerfile and manifest security settings
Runtime (detect unexpected behavior):
- Behavioral baselining: establish what the container is supposed to do
- Anomaly detection: alert on deviations from the baseline
- Drift detection: identify when new code paths are executed post-deployment
- Continuous CVE matching: match running container SBOM against current CVE feeds
The combination covers threats that each layer cannot address alone.
The Role of Container Hardening in Runtime Monitoring
Container security software that hardens images before deployment makes runtime monitoring dramatically more effective by establishing a precise behavioral baseline.
An unminimized container with 400 packages has a large, ambiguous expected behavior set. Many packages may be invoked by other packages without explicit application code calling them. Runtime monitoring against this container generates alerts that may or may not be anomalous — the expected behavior set is too broad to distinguish genuine anomalies from normal package interactions.
A hardened container with 45 packages has a precisely defined expected behavior set. The RBOM documents every binary, system call, and network connection observed during profiling. Runtime monitoring against this container generates alerts for genuine deviations — behaviors that were not observed during profiling are definitively unexpected.
Container hardening is not just an attack surface reduction measure. It is the prerequisite for accurate runtime anomaly detection.
Implementation Patterns for Combined Build + Runtime Security
Pattern 1: Profile at build time, monitor at runtime
# Build pipeline
stages:
– build: docker build -t $IMAGE .
– profile: run-test-suite-with-profiling $IMAGE
– harden: harden-tool $IMAGE –output ${IMAGE}-hardened
– scan: trivy image ${IMAGE}-hardened
– push: docker push ${IMAGE}-hardened
# Runtime deployment with monitoring agent
containers:
– name: app
image: ${IMAGE}-hardened
– name: runtime-monitor
image: runtime-agent:latest
env:
– name: EXPECTED_PROFILE
value: “${IMAGE}-hardened” # Agent uses build-time RBOM as baseline
The runtime monitoring agent uses the RBOM from the build pipeline as its behavioral baseline. Deviations from the build-time profile generate alerts.
Pattern 2: Continuous drift detection
New CVEs disclosed against the image’s SBOM components are the most common source of post-deployment security risk. Continuous CVE matching that notifies the team when the deployed image has accumulated new Critical CVEs bridges the gap between build-time scan state and current threat landscape.
# Scheduled job: match running image SBOMs against current CVE feeds
for pod in get_production_pods():
sbom = fetch_image_sbom(pod.image_digest)
new_cves = match_against_current_cvefeed(sbom, since=pod.deploy_timestamp)
if has_critical_cves(new_cves):
alert_team(pod, new_cves)
trigger_rebuild_pipeline(pod.image)
Frequently Asked Questions
What is one of the challenges of DevOps in terms of security?
One of the primary security challenges in DevOps is that build-time scanning only captures vulnerabilities known at the moment an image is built. New CVEs are disclosed after deployment, novel attack techniques bypass signature-based detection, and supply chain compromises can inject malicious code that passes all static checks. Runtime protection addresses what build-time security structurally cannot: the threat landscape as it exists when containers are actually running.
What is fundamental to ensuring runtime protection for deployed containers?
Establishing an accurate behavioral baseline is fundamental to runtime protection. Container hardening that removes unused packages produces a precise expected behavior set — documented in an RBOM — which runtime monitoring uses to identify genuine anomalies. Without a tight baseline, monitoring agents generate too much noise to be actionable; with a hardened container’s narrow baseline, any deviation from expected behavior is definitively suspicious.
What are the security best practices in DevOps?
Security best practices in DevOps combine build-time and runtime controls. At build time: vulnerability scanning to eliminate known CVEs, container hardening to remove unused packages, SBOM generation for component inventory, and configuration analysis. At runtime: behavioral baselining, anomaly detection against that baseline, drift detection for new code paths executed post-deployment, and continuous CVE matching against the running container’s SBOM. Neither layer alone is sufficient.
What is the runtime protection mechanism?
Runtime protection works by establishing a behavioral baseline for each container during profiling, then monitoring running containers for deviations from that baseline. eBPF-based agents observe system calls, network connections, and file access patterns without significant overhead. When a container executes a system call, opens a network connection, or accesses a file path outside its normal profile, the runtime monitoring system generates an alert — detecting attacks even when no known CVE signature matches the technique.
Overhead Considerations for Production Runtime Security
Runtime security tools that add significant CPU or memory overhead cannot be deployed in production environments where SLOs depend on container performance baselines. A monitoring agent that adds 5% CPU overhead changes every SLO calculation and generates confusion about whether latency changes are caused by the security agent or application behavior.
The constraint for production runtime security: overhead must not be detectable in application performance metrics. For eBPF-based monitoring approaches, this is achievable — the overhead is typically well under 1% for behavioral observation workloads. For agents that intercept every system call with expensive userspace processing, overhead can be significant.
Selecting runtime security tools that can demonstrate sub-1% overhead through production workload testing is a prerequisite for deploying them alongside production services. The security value of a monitoring agent that cannot be deployed is zero.