Archiving TCP

In modern distributed systems, network visibility is often a binary choice: you either perform full packet capture (PCAP

In modern distributed systems, network visibility is often a binary choice: you either perform full packet capture (PCAP) or you rely on intermittent snapshots of the socket state. Full packet capture provides unparalleled depth but is prohibitively expensive in terms of storage and CPU overhead, making it unsustainable for long-term retention. Conversely, periodic snapshots via tools like ss provide a "point-in-time" view that often misses the ephemeral connections—the "micro-bursts" of short-lived TCP flows—that are frequently the root cause of latency spikes or connection failures.

TCP Archiving emerges as the middle ground. It is the practice of persisting the metadata, state transitions, and performance telemetry of the TCP lifecycle without the burden of storing application-layer payloads. By focusing on the "how" and "when" of a connection rather than the "what," TCP archiving provides a high-fidelity, low-cost audit trail for troubleshooting, security forensics, and capacity planning.

TCP Archiving vs. Traditional Approaches

To understand the value of archiving, we must compare it against the two industry standards:

  • Vs. Full Packet Capture (tcpdump/Wireshark): While PCAP captures every byte, it is data-intensive. TCP archiving focuses on the 5-tuple (source/destination IP and port, protocol) and performance metrics. This results in a significantly smaller data footprint, enabling much longer retention periods and faster analytical queries.
  • Vs. Snapshotting (ss//proc/net/tcp): Snapshots are "pull-based" and periodic. If a connection lasts only 50ms between two 60-second polling intervals, it is invisible to the snapshot. TCP archiving, particularly when event-driven, captures the complete trajectory of a connection from the initial SYN to the final FIN or RST.

In a mature production environment, these are not mutually exclusive. The best practice is a hybrid strategy: use full packet capture for short-term, deep-dive debugging of specific issues, and use TCP archiving for long-term observability and trend analysis.

Designing a Robust Data Model

A high-quality TCP archive must be structured to support both real-time alerting and historical batch analysis. A unified schema should be implemented to ensure consistency across different collection methods. We can categorize the essential fields into several logical groups:

1. Connection Identifiers

These fields allow you to uniquely identify a flow and correlate it across different layers of the stack.

  • 5-tuple: src_ip, src_port, dst_ip, dst_port, protocol.
  • Contextual Metadata: netns (network namespace) to support containerized environments.

2. Lifecycle and Temporal Data

To reconstruct the timeline of an event, you need precise timestamps.

  • Timestamps: start_ts, end_ts.
  • State & Duration: The final TCP state and the total duration_ms.

3. Traffic and Performance Telemetry

This is the core of capacity planning and performance tuning.

  • Throughput: tx_bytes, rx_bytes, tx_packets, rx_packets.
  • Congestion & Latency: retrans_segs (retransmission count), rtt_us (Round Trip Time), cwnd (congestion window), and ssthresh (slow start threshold).

4. Process Attribution and Security

In a microservices architecture, knowing which process owned a socket is critical.

  • Ownership: pid, comm (command name), uid, container_id, and cgroup_id.
  • Security/Termination Signals: syn_flags, rst_reason, and close_reason (to distinguish between a graceful shutdown and an abrupt reset).

For implementation, JSON Lines (JSONL) is the recommended format for the ingestion layer due to its append-only nature. For long-term analytical storage, these records should be converted to a columnar format like Parquet and loaded into OLAP engines such as ClickHouse or Elasticsearch.

Collection Methodologies

There are two primary ways to implement TCP archiving: snapshot-based and event-based collection.

Snapshot-based Collection (The Polling Approach)

This method involves periodically querying the kernel's networking stack to see what connections currently exist. It is easy to implement using standard tools like ss or by reading /proc/net/tcp.

Pros: Low complexity; minimal impact on kernel stability.
Cons: High risk of missing short-lived connections; provides only a "stale" view of the network.

A simple shell-based implementation might look like this:

# Archiving established connections every 60 seconds
while true; do
  ts=$(date +%s)
  # Using ss to extract connection details and process info
  ss -H -tanp -o state established | \
    awk -v ts="$ts" '{print ts, $1, $2, $3, $4, $5, $6}' >> tcp_archive.log
  sleep 60
done

Note: In production, adding the -e flag to ss is recommended to capture additional socket statistics.

Event-based Collection (The eBPF Approach)

The modern gold standard for TCP archiving is eBPF (extended Berkeley Packet Filter). Instead of polling, eBPF allows you to attach programs to specific kernel tracepoints or kprobes. This allows the system to "push" an archive record whenever a specific event occurs, such as a state change or a retransmission.

Pros: Extremely high precision; captures every connection regardless of duration; minimal overhead for high-frequency events.
Cons: Requires higher technical expertise and modern kernel support.

Using bpftrace, you can intercept state transitions in real-time:

# Example: Tracing TCP state changes via eBPF
bpftrace -e '
tracepoint:sock:inet_sock_set_state
{
  printf("%llu old=%d new=%d sport=%d dport=%d\n",
         nsecs, args->oldstate, args->newstate, 
         args->sport, args->dport);
}'

By hooking into functions like tcp_retransmit_skb or tcp_close, you can build a highly granular stream of connection telemetry that captures the exact moment a network degradation begins.

Conclusion

TCP archiving transforms network observability from a reactive "firefighting" activity into a proactive analytical discipline. By moving away from the heavy weight of full packet captures and the blind spots of periodic snapshots, organizations can build a sustainable, long-term record of their network health. Whether through simple polling for long-lived connections or sophisticated eBPF tracing for high-precision telemetry, archiving the TCP lifecycle is an essential step toward mastering complex, distributed environments.