Most of my coding agents ended up on remote servers — an agent runs for tens of minutes at a time, and a laptop that goes into meetings, sleeps and changes networks is the wrong place for that. The annoying part is managing them: open a terminal, SSH in, remember which directory this project lives in, start the agent. Once you have a few machines and a few projects you stop remembering who is running what, and anything started in the foreground dies with the connection.

That is what AgentMux is for. Every server, project and agent sits in one tree, colour-coded by run state, so a glance tells you what is running and how far along it is.

The important bit is that everything it starts runs inside a tmux session on the server; the desktop app never holds a process. Closing a tab is just a detach. Quitting the app, losing the network or shutting the lid does not stop the agent, and when you come back the terminal is where you left it, scrollback included.

Roughly what else it does:

• A real terminal — xterm.js attached to the agent's own pane, so you can take over typing without interrupting it first
• One instruction to many agents, with a separate delivery receipt from each
• An SFTP file browser; double-click a text file and it opens in Monaco, with atomic saves that keep the permission bits and warn you if something else changed the file
• Host metrics: CPU broken down and per core, memory, disk, network, temperature, plus utilisation and memory for NVIDIA GPUs
• An install panel that detects which agent CLIs a box is missing (Claude Code, Codex, Gemini CLI, OpenCode, Aider, Cursor CLI) and installs them inside tmux

Passwords and key passphrases are encrypted with AES-256-GCM before they touch disk, the master key lives in the OS keychain, and the frontend only ever learns whether a credential is set. It does not proxy the agent's model traffic and never touches API keys.

If you want to try it, the releases page has macOS, Windows and Linux builds (unsigned, so the first launch takes one extra click). Building from source needs Go 1.25+ and Node 20+; on the remote side, tmux and an SSH account is all it takes:

git clone https://github.com/tan-zhuo/AgentMux.git
cd AgentMux
cd frontend && npm install && npm run build && cd ..
go build -o agentmux .
./agentmux

Plenty is still missing: the orchestrator AI from the original design only has its plumbing in place, there are no split panes inside a tab, file transfers are one file at a time, and metrics history lives only in the open panel. The editor is Monaco, not VS Code, so extensions do not run — if you need them, running code-server on the box and letting AgentMux keep it alive in tmux is the better answer.

The code is at github.com/tan-zhuo/AgentMux — try it and file issues.

Why build this​

What I wanted was simple: a globe that spins the moment you open the browser, where clicking a country shows you how it's doing today — no client to install, no token to register, no paid API. The professional options (Cesium, say) are far heavier than this calls for, so I assembled a lightweight one myself.

What's inside​

Country info: click a country and the camera glides over, with altitude adapted to the country's size. The detail panel shows the flag, bilingual and official names, capital, region, area, population, languages, currency, GDP (nominal and per capita), development status, and system of government, plus an embedded Wikipedia history summary that follows the language setting.

Data layers: a GDP bar layer stands every country's latest GDP up as 3D columns (sqrt-normalized, colored by World Bank income group); a flag layer pins each flag at its country, sized by area; a shipping layer draws the world's major maritime trade corridors — animated arcs segmented by hub ports, with width and color encoding traffic level. Each layer toggles independently.

Global rankings: GDP, exports, and imports (live World Bank data), plus static snapshots of oil and gas production. Click a row to fly straight to that country.

Time-travel mode: the paleogeographic 3D evolution of 16 geological eras, from the supercontinent Rodinia 750 million years ago to the present, with a draggable timeline and autoplay (paleogeographic maps © C.R. Scotese, PALEOMAP Project). You can watch the continents drift, collide, and split apart.

The cosmic scale ladder: the top bar switches between five scales, each with a bilingual fact card — Earth (10⁷ m), the Moon (10⁸ m, lunar textures plus 10 historic landing sites from Apollo, Chang'e, and Luna), the Solar System (10¹³ m, eight planets orbiting), the Milky Way (10²¹ m, a procedural barred spiral of 50,000 particles, with the Solar System's position and Sagittarius A* marked), and the observable universe (10²⁷ m, a cosmic-web point cloud of galaxy clusters and filaments).

Where the data comes from​

All free, no API keys. Country basics come from mledoze/countries (the upstream dataset behind REST Countries), statically bundled at build time — zero requests at runtime. Population, GDP, and trade figures come from the World Bank Open Data API, fetched on demand when you click a country and cached in localStorage for 30 days. Flags come from flagcdn.com; country borders are Natural Earth 110m, self-hosted with the app; the Earth textures are the NASA ones bundled with three-globe; planet and Moon textures are from Solar System Scope (CC BY 4.0).

One gotcha: the REST Countries v3.1 API was deprecated in 2026 (the new version requires registration), so the project never depends on it at runtime — the upstream dataset is statically baked in at build time, and updating the data is a single script re-run.

Tech stack​

React 18 + TypeScript + Vite, with Three.js (via globe.gl) for the 3D — orders of magnitude lighter than Cesium, no Ion token, and country polygons, fly-to animation, and hover picking all work out of the box. Styling is Tailwind CSS v4, state is Zustand, i18n is react-i18next (browser-language detection plus localStorage memory). Pure frontend, statically deployable, with a right-hand panel on desktop and a bottom drawer on mobile.

What's next​

Still planned: country search, major exports and resources, multi-country comparison, and chart visualizations. The code is at github.com/tan-zhuo/earth — give it a spin and file issues.

There is one essential difference from the usual "neural network explainer animations": this is not an animation — it is real computation. At load time, every network is genuinely trained in your browser with deterministic seeds; every value you see — activations, attention weights, softmax probabilities — is the true result of the forward pass. Click any node and check the arithmetic yourself.

Why I built it​

I've seen plenty of diagrams and videos explaining neural networks, and most stop at the "illustration" level: arrows, boxes, gradient colors, made-up numbers, staged flows. I always wanted a version you could audit — every neuron's inputs, weights, bias, and activation laid out in front of you, where the Σ you add up is exactly the number glowing on screen. If you suspect it's lying, grab a calculator.

That became AI Room. Pure frontend, no backend, no runtime network calls — all training and inference happens in your browser.

What's inside​

MLP (multi-layer perceptron): classifies three Gaussian clusters. Data-flow particles run through the weighted connections layer by layer, and the animation order is the computation order.

CNN (convolutional network): classifies patterns (vertical / horizontal / diagonal / ring). The receptive-field window slides across the input exactly in computation order. Kernels come in two modes — hand-crafted Sobel-style edge detectors (interpretable; only the dense head trains), or kernels that start as random noise and are trained end-to-end through a hand-written backward pass. There's also a draw mode: paint your own pattern on the input grid and watch the whole network classify it live, stroke by stroke. Three sizes (S/M/L), and switching retrains the network on the spot.

Tiny Transformer (character-level): a structurally faithful Transformer block — tokenizer → embedding → sinusoidal positional encoding → multi-head causal attention (2 heads, with output projection) → residual + LayerNorm → feed-forward → residual + LayerNorm → output softmax. Both the forward and backward passes (including LayerNorm and residual gradients) are hand-written, trained in-browser on a small corpus in about 2.5 seconds. Both heads' 8×8 attention matrices light up row by row; open an Add & Norm cell and you'll see the full arithmetic down to μ, σ, γ, β. Hit "Generate" for true autoregressive decoding: the sampled character is appended to the context, the window slides, the whole pipeline re-runs, and the text streams out one character at a time — exactly how real LLMs write. A temperature slider (0.2–1.4) controls the sampling distribution live.

Language ID (an AI application): type anything; the text becomes 8 interpretable statistics (Latin %, CJK %, kana %, and so on) and a trained MLP decides whether it's 中文, English, or 日本語. It's also direct proof that the computation is real: your input, its numbers, its prediction — one chain you can follow end to end.

Every layer title can be clicked for an explanation: what it does, why the network needs it, and a plain-words analogy — in Chinese, English, and Japanese — while the layer glows in 3D and everything else dims.

It doesn't pretend to be GPT​

The README has a dedicated section on the differences from production models: a single Transformer block, 2 heads, d=12, character-level tokens; training is plain sample-by-sample SGD; the CNN has one conv+pool stage; language ID deliberately uses hand-crafted statistical features instead of modern embeddings. The math shown is real; the scale is not. I think honestly listing the simplifications is worth more than pretending "this is how ChatGPT works."

Stack​

Vite + React + TypeScript + React Three Fiber + Drei + Zustand. npm run sanity trains every network at every scale and verifies accuracy.

The code is at github.com/tan-zhuo/ai-room — give it a spin, and issues are welcome.

OpenClaw excels at gateway, tools, sessions, and local multi-agent routing, but falls short when it comes to large-scale collaboration, cross-node scheduling, and operator visibility. Meanwhile, golutra-style orchestration and trace capabilities offer a valuable complement.

Building on OpenClaw's gateway runtime, I propose an evolutionary architecture: treat Kubernetes as a solid foundation and build an AI-native control plane on top of it — call it the Kubernetes AI OS. It does not replace Kubernetes; it treats Kubernetes as the infrastructure layer and adds an agent runtime, execution scheduling, capability discovery, side-effect governance, and cross-node coordination — a genuinely operator-facing execution system.

Why This Architecture?​

OpenClaw is currently strong in channels, tools, sessions, and local multi-agent routing, but has clear gaps in large-scale collaboration, execution-trace auditing, and cross-node placement.

What we want to fuse together:

• OpenClaw's southbound integrations and agent runtime
• golutra-style execution traces and visualization
• Kubernetes's lifecycle management, scheduling, and failure recovery

The core idea fits in one sentence: execution must be visible, inspectable, and attributable. Without a clear execution ledger, a multi-agent system quickly becomes a black box.

Design Goals​

• Clear multi-agent collaboration within a single node
• Workflow-level scheduling across nodes
• Nodes that can automatically join, be quarantined, and recover
• Small models assist with role recommendation, but high-risk decisions must go through the policy plane
• Every meaningful action lands in a unified execution ledger
• Clean separation of control plane, execution plane, observability plane, and policy plane

Non-goals: no replacing Kubernetes-native scheduling, no unrestricted free-form chat between agents, and no letting the model directly decide on high-risk operations like privilege escalation or topology changes.

The Six-Layer Architecture at a Glance​

The system is divided into six layers with clear responsibility boundaries:

• Southbound Gateway Layer: reuses OpenClaw's channel, tool, and session integration capabilities
• Execution Kernel: the core execution layer, managing the full Run → Flow → Step → SideEffect lifecycle
• Cluster Mesh: handles node registration, capability publishing, step scheduling, and security isolation (free-form chat between nodes is discouraged)
• Observability Plane: the unified execution ledger, with visual views of runs, flows, steps, and side effects
• Cognition Plane: uses small models for workload identification, role recommendation, and state summarization
• Policy Plane: owns security approvals, secret scoping, side-effect governance, and trust checks

The control plane decides "what should run, where, and under which policies"; the execution plane does the actual work and produces the trace.

A Single-Node Multi-Agent Runtime​

Even with just one node, multiple roles should run in coordination. The recommended base roles:

• Planner
• Executor
• Reviewer
• Watcher
• Specialist (domain expert)

Every action is first reported to the local ledger and then aggregated by the control plane, keeping the execution trace clearly visible.

Cross-Node Scheduling: Steps, Not Chat​

The key to cross-node collaboration is scheduling at the granularity of Steps, rather than letting nodes forward messages freely. Only then are ownership, cancellation, and auditing guaranteed.

The scheduler weighs required tools, model availability, hardware resources, locality, trust level, and tenant affinity.

Node states include joining, ready, degraded, quarantined, and draining, with automatic quarantine of high-risk nodes.

The Core Primitive: the Execution Ledger​

This is the most important piece of the entire system. Every event — run, flow, step, side effect, policy, and so on — is bound to a runId, forming a unified, queryable ledger.

An operator should be able to easily answer:

• Which agent is executing?
• What side effects have occurred?
• Which step is it currently stuck on?
• Is this failure safe to retry?

The Relationship with Kubernetes​

Kubernetes continues to own the infrastructure capabilities: Pod scheduling, liveness, service discovery, storage, and RBAC.

The AI OS layer expresses objects like HiveRun, HiveFlow, HiveStepLease, and HiveNode through CustomResources (CRDs), with Operators implementing the reconciliation logic.

A Suggested Evolution Path​

Start small:

1. Phase 1: on a single node, build out the run ledger, flow graph, and side-effect visualization
2. Phase 2: implement the single-cluster step scheduler and node management
3. Phase 3: add lightweight role recommendation
4. Phase 4: implement adaptive cluster behavior

The first goal is not full autonomy, but a single-node AI runtime with first-class observability. Make execution visible first, then make side effects governable, and only then tackle cross-node scheduling.

Wrapping Up​

The goal of the Kubernetes AI OS is to make Kubernetes the true operating-system foundation of the AI era: unlocking multi-agent collaboration while staying safe and controllable. It doesn't reinvent the wheel — it stitches together OpenClaw's southbound integrations, golutra-style execution traces, and Kubernetes's scheduling and recovery. The entire design revolves around a single thread: execution must be visible, inspectable, and attributable. And you don't have to get there in one leap — start with a single-node execution ledger, then move step by step toward cross-node scheduling and adaptive clusters.

Overall Architecture: A Hub-and-Spoke, Layered "OS-Grade" Framework​

OpenClaw's core design philosophy is to "treat AI as an infrastructure problem, not a prompt-engineering problem" — the model only reasons, while the system owns state management, execution control, security, and multi-channel coordination. The architecture combines a hub-and-spoke topology with layered decoupling, keeping the core logic centralized and controllable while leaving individual components flexible.

Hub-and-Spoke: the Gateway as the Single Control Plane​

OpenClaw is centered on the Gateway, with every component — clients, channel adapters, agent executors — radiating out from it. The Gateway is the system's "traffic hub." It binds to the local loopback address (127.0.0.1:18789) by default, allowing access only from the local device and protecting data privacy at the source. Its core responsibilities include:

• Unified authentication and session isolation: sessions from different users or devices are managed independently, preventing data from leaking across sessions;
• Lane Queue: each session executes tasks serially by default, eliminating state conflicts and interleaved logs from concurrent tasks (parallel mode can be enabled manually);
• Message routing and dispatch: forwarding normalized messages precisely to the corresponding agent;
• Streaming interaction: pushing model output and "typing" status in real time to simulate a natural conversation.

Layered Architecture: Capabilities Unpacked from the Outside In​

OpenClaw's layers progress from user interaction down to infrastructure, each with clear responsibilities and independently extensible.

1. Client layer: multiple entry points

Users don't need a dedicated app — they can interact with OpenClaw through:

• The command-line tool (CLI): great for technical users and quick debugging;
• The Web UI: a visual control panel;
• The macOS menu bar app and iOS/Android Nodes: lightweight mobile entry points;
• Third-party chat platforms: 20+ popular apps including WhatsApp, Telegram, and Discord, so users can issue commands from tools they already know.

2. Access coordination layer: the Gateway's "nerve center"

As described above, the Gateway is the core of access coordination — it converts heterogeneous input into a format the system understands and manages session lifecycles.

3. Channel abstraction layer: breaking through platform protocol barriers

Message protocols differ wildly across chat platforms (WhatsApp's Baileys protocol, Telegram's MTProto protocol). Channel Adapters play the role of "translator": they normalize each platform's messages (text, images, voice, etc.) into OpenClaw's internal standard format, and handle attachment download, caching, and access control (such as "only allow designated contacts to send commands").

4. Core logic layer: the agent's "thinking and execution engine"

The core logic layer consists of the Agent Runner and the Memory System — the heart of an AI agent that gets things done:

• Agent Runner: each channel or group can map to an independent agent instance (multi-agent collaboration supported), invoking the Pi Agent Runtime via RPC to complete the full loop of "receive instruction → call model → execute tools → produce result";
• Memory System: owns context management, including session history, long-term user memory, and tool capability descriptions, providing the model with what it needs to decide.

5. Infrastructure layer: the foundation for local execution

The infrastructure layer provides the base capabilities the system runs on:

• Local persistence: all data stored as Markdown and .jsonl files (no cloud database dependency);
• WebSocket streaming: keeping the real-time connection between clients and the Gateway;
• Sandboxed execution: restricting tools' system permissions to prevent malicious operations;
• Cron and Webhooks: proactively triggered tasks (such as a daily morning report).

How the Core Components Work Together​

To make the architecture easier to grasp, the relationships between core components can be simplified as:

1. The Gateway acts as the dispatch center, receiving messages from clients or channel adapters and assigning them to the right agent;
2. The Agent Runner follows the Gateway's instructions, calls the model to generate decisions, and executes tools (reading/writing files, driving the browser);
3. Channel Adapters "translate" messages from third-party platforms so users can interact without switching tools;
4. Skills: plugins stored under ~/.openclaw/workspace/skills, which the agent can discover, install, and invoke on its own (skills can be shared via the ClawHub registry);
5. Nodes: device-side executors running on macOS, iOS, etc., exposing local hardware capabilities (voice input, camera, screen recording) over WebSocket;
6. Canvas: a visual workspace served on a dedicated port (18793), where the agent can generate HTML/A2UI interfaces (task progress bars, data dashboards).

Runtime Logic: the Agent Loop and Proactive Scheduling​

OpenClaw's core value is "letting AI actually do things," powered by a repeatable Agent Loop and a proactive scheduling mechanism — the former handles user-triggered tasks, the latter enables unattended autonomous operation.

The Typical Message Flow: from Input to Output​

Here is the full journey of a user instruction such as "organize the files on my desktop from the last 3 days and generate a list."

1. Message intake and normalization

The user sends the instruction via Telegram → the Channel Adapter converts the Telegram message into the system's standard format (text content, sender info, context) → the normalized message is placed into the corresponding Session Lane Queue in the Gateway (serial execution, no conflicts).

2. Routing and agent activation

Based on configuration (e.g. "the main session binds to the default agent," "non-main sessions require an @mention"), the Gateway routes the message to the target agent. For example, when the user @mentions OpenClaw in a family group, the Gateway activates the "family assistant" agent.

3. Context assembly: making sure the agent "remembers" what matters

The Memory System automatically assembles the following as the model's input context:

• The current session's message history (so the agent doesn't "forget");
• Long-term user memory (e.g. "the user likes organizing files in Markdown," "the desktop path is ~/Desktop");
• System prompts: AGENTS.md (the agent's behavioral baseline, e.g. "prefer local tools"), SOUL.md (personality, e.g. "concise and efficient"), TOOLS.md (tool capability descriptions, e.g. "file.read can read local files");
• Installed Skills (e.g. a "file classification" skill).

If the context grows too long, the system automatically compresses and summarizes it to keep the model's input within the context window.

4. Model invocation and decision generation

The Agent Runner calls the configured LLM (Claude, GPT, or a local model like Llama 3), and the model decides based on the context: does this need a tool?

• If not: it generates a natural-language reply directly (e.g. "Done — the list has been saved to your desktop");
• If so: it generates a tool call (e.g. {"name": "file.list", "parameters": {"path": "~/Desktop", "days": 3}}).

5. Tool execution: looping until the task is done

Tool execution runs as a loop:

1. The Agent Runner executes the tool in a sandbox (the main session has host permissions by default; non-main sessions can be forced into Docker sandbox isolation);
2. The tool returns its result (e.g. "found 5 files: report.pdf, photo.jpg...");
3. The result is written back into the context, and the Agent Runner calls the model again to decide whether further action is needed (e.g. "should the list be converted to Markdown?");
4. The steps repeat until the model judges the task complete.

6. Output and persistence

The final result is streamed back to the originating channel (e.g. Telegram) via the Gateway, while all interaction records, tool execution logs, and memory updates are written to local .jsonl and Markdown files — fully traceable, no cloud required.

Proactive Operation: Where the AI's "Initiative" Comes From​

Traditional chatbots wait for the user. OpenClaw supports proactive operation, letting the AI complete tasks without being asked. There are three mechanisms.

1. Scheduled tasks (Heartbeat Scheduler/Cron)

Users can configure recurring tasks, such as "every morning at 8, organize yesterday's email and produce a summary" or "every Sunday, back up the desktop to the external drive." The Gateway wakes the agent on schedule, and results are automatically delivered to the designated channel (e.g. the user's WhatsApp).

2. External event triggers (Webhook/Pub/Sub)

OpenClaw can receive external events via Webhooks — a new Gmail message, a GitHub issue being created, a smart-home device firing. For example, when an important work email arrives, OpenClaw can automatically extract the key information and alert the user.

3. Multi-agent collaboration

Agents can collaborate through built-in tools (sessions_list to see all sessions, sessions_send to message another agent). For instance, a "calendar agent" that notices a meeting tomorrow can ask the "file organizer agent" to prepare the meeting materials in advance.

Getting It Running: Deployment in a Few Steps​

OpenClaw's deployment is lightweight enough for everyday users to get started quickly:

# Install the latest version globally
npm install -g openclaw@latest

# Onboarding: guided setup of chat channels, LLM model, and workspace path
openclaw onboard

# Run persistently: register the Gateway as a system daemon for 24/7 operation
openclaw onboard --install-daemon

Once done, connect to the Gateway via the CLI, Web UI, or a chat app and start using it.

Key Design Highlights and Security: Local-First Privacy​

OpenClaw's design revolves around user control and privacy. Here are its core highlights.

Local-First: Your Data Stays with You​

All data — session history, memory, tool execution logs — is stored on the local device. The Gateway listens only on 127.0.0.1 (the local loopback address) by default and never sends data to the cloud. Even offline, OpenClaw can still execute local tasks (file operations, local model calls).

Security Isolation: Layered Protection​

OpenClaw protects the system through multiple mechanisms:

• DM pairing approval: direct-message commands from third-party chat platforms require manual user approval (e.g. after a Telegram user sends /pair, OpenClaw sends a verification code, and interaction is only allowed after verification);
• Sandboxed execution: tool execution in non-main sessions defaults to a Docker sandbox, restricting access to the host system;
• Tool blocklists: users can forbid the agent from calling certain dangerous tools (such as system.exec for running system commands);
• Explicit control: commands like /think high (deepen model reasoning) and /verbose on (show tool execution details) let users fine-tune agent behavior.

Extensibility: a Plugin Architecture for Customization​

Every core component of OpenClaw is hot-swappable:

• Channels: build your own Channel Adapter (WeChat, Signal, etc.);
• Models: plug in any LLM (just implement the standard interface);
• Skills: write custom Skills ("auto-sync Obsidian notes," "generate a slide outline") — the agent reads the Skill's contract file and invokes it on its own;
• Memory: swap the Memory System for a vector database (such as Pinecone) to improve long-term memory retrieval.

The Memory System: Four Layers That Make the Agent Smarter Over Time​

OpenClaw's memory system has four layers, ensuring the agent uses historical information efficiently:

1. SOUL layer: the agent's personality and values (e.g. "protect user privacy first," "answer concisely");
2. TOOLS layer: descriptions of what tools can do and how to use them;
3. USER vector memory: the user's long-term preferences (e.g. "prefers coffee over tea," "commonly used file paths"), stored as vectors for fast retrieval;
4. Session short-term memory: the current session's message history, automatically compressed to avoid redundancy.

The system periodically runs memory "compaction," keeping the essentials and dropping duplicates so the agent's memory stays efficient.

Wrapping Up​

OpenClaw answers the question of "how AI actually gets things done" with an OS-grade architecture: the Gateway serves as the single control plane for unified dispatch, and the layered design makes channels, models, and skills independently replaceable. The Agent Loop's multi-round tool cycle, combined with proactive mechanisms like cron and Webhooks, lets the AI both respond to commands and run unattended. Meanwhile, local-first storage, sandbox isolation, and pairing approval draw a clear line between capability and safety. For anyone looking to wire a large model into real workflows, there is plenty in this design worth borrowing.

Background: IDEs as Programming Tools​

For a long time, software development relied mainly on traditional IDEs like Visual Studio Code, IntelliJ IDEA, and Eclipse. Their core capabilities were code editing, syntax highlighting, autocompletion, and debugging tools—but the actual code logic still had to be written by the programmer.

It wasn't until large language models (LLMs) arrived that software development entered the era of AI programming. The earliest wave of AI coding tools included GitHub Copilot and Tabnine, which mainly tackled code completion. Cursor's goal, however, was never simple completion. It was:

Build an AI-native development environment.

The Birth of Cursor​

Cursor is an AI-native code editor (AI-Native IDE) developed by Anysphere. The company was founded in 2022 by four MIT graduates:

• Michael Truell
• Aman Sanger
• Sualeh Asif
• Arvid Lunnemark

Cursor was officially released in 2023. Technically, it is:

An AI IDE built as a fork of VS Code

On top of the VS Code foundation it added a large set of AI features: AI code generation, AI debugging, AI refactoring, AI codebase queries. Cursor's goal was crystal clear:

Let developers write code directly through natural language.

Cursor's Core Technical Design​

Cursor's design philosophy centers on three key points.

1) AI Code Generation​

Cursor generates code through large language models. A developer can simply type "implement a Redis distributed lock," and Cursor will generate the code, comments, and tests. This is completely different from traditional IDE code completion:

• Traditional IDE: the developer writes code, the IDE assists with completion
• Cursor: the developer describes the requirement, the AI generates the code

This mode of working has come to be known as Vibe Coding.

2) Codebase Understanding​

One of Cursor's core capabilities is understanding the entire code repository. It builds vector indexes of the code, along with file relationships and dependencies, so a developer can ask "where is the user login logic in this project?" and Cursor will locate the relevant code. This is similar to a RAG + code embedding approach.

3) AI Code Modification​

Cursor supports a capability called Smart Rewrite. For example, given "make this module asynchronous," Cursor can automatically modify multiple files, adjust functions, and update call sites. This is nearly impossible in a traditional IDE.

Cursor's Rapid Growth​

Cursor's growth has been astonishing. Less than two years after release:

• ARR exceeded $300 million
• Funding raised exceeded $1 billion
• Company valuation approached $30 billion

Cursor became one of the fastest-growing AI SaaS products ever. Many companies started using it—Nvidia, Stripe, Spotify—with Nvidia having over 30,000 engineers writing code with Cursor.

Cursor's Technical Challenges​

Despite Cursor's popularity, academic research has pointed out some issues: AI coding assistants speed up development, but may increase code complexity and the number of warnings.

One research paper noted:

After adopting AI coding assistants, project velocity improves in the short term, but code complexity rises as well.

Another study found:

AI code assistants are easily influenced by faulty code in their context, leading them to generate defective code.

Which is to say, AI programming still requires developers to review and stay in control.

Cursor's Future Direction​

Cursor's vision is aggressive: software development may eventually become "natural language → software system." The development workflow could turn into:

Requirements
↓
AI generates code
↓
AI generates tests
↓
AI deploys automatically

In other words, software development may shift from "writing code" to "describing requirements." Cursor has even articulated a vision:

One day, developers will only need to write 50 lines of pseudocode to generate 2,000 lines of complete code.

Wrapping Up​

By forking VS Code, Cursor quickly acquired a mature editor foundation, then built LLM capabilities into code generation, repository understanding, and cross-file modification—hitting the AI programming boom at exactly the right moment. The staggering growth numbers show the approach works, but the research is a reminder that AI-generated code brings new problems like rising complexity. No matter how powerful the tool, the developer remains ultimately responsible for code quality. That's unlikely to change for the foreseeable future.

Background​

PolarDB is a cloud-native database from Alibaba Cloud, and one of its core design principles is decoupling the compute layer from the storage layer. This architecture gives the database stronger elastic scalability and higher resource utilization.

Problems with the Traditional MySQL Architecture​

In a traditional MySQL setup, the database typically runs on a single server, with compute and storage on the same machine:

MySQL Server
 ├── CPU
 ├── Memory
 └── Local Disk      # data and compute bound to the same machine

This architecture was good enough in the early internet era, but as businesses grow, several obvious problems emerge.

1) Storage Is Hard to Scale​

Database data usually lives on local disks. When data volume grows, the only options are upgrading the disk or moving to a bigger machine—scalability is limited.

2) Read Scaling Is Expensive​

The common way to scale MySQL is primary-replica replication:

Master
 ├── Slave1      # each replica holds a full copy of the data
 ├── Slave2
 └── Slave3

Every read-only node needs a complete copy of the data. As read nodes multiply, storage costs grow rapidly, and replication pressure rises with them.

3) High Availability Is Complicated​

If the primary node fails, a failover—manual or automatic—is required. Replication lag may exist during the switch, and data consistency needs extra handling.

PolarDB's Storage-Compute Separated Architecture​

PolarDB's core design: compute nodes and storage nodes are fully decoupled. The architecture looks roughly like this:

          +--------------------+
          |    Compute Node    |
          | (MySQL / PG / ORA) |   # compute layer: SQL parsing, execution, transactions
          +---------+----------+
                    |
          +---------+----------+
          | Distributed Storage|   # storage layer: data, logs, replicas
          |     (PolarStore)   |
          +--------------------+

Compute nodes handle SQL parsing, query execution, and transaction processing; the storage layer handles data storage, log management, and replica replication. All compute nodes share a single distributed storage system—an approach known as Shared Storage Architecture.

What Storage-Compute Separation Solves​

This design addresses several core problems of traditional databases.

1) Elastic Scalability​

With a traditional database, adding compute capacity means upgrading the server. In PolarDB, you simply add compute nodes:

Storage Layer
   ↑
   │
Compute1      # new compute nodes need no data copy
Compute2
Compute3

All nodes share the same data, enabling read scaling, distributed computing, and scale-out in seconds.

2) Lower Storage Costs​

Traditional MySQL read nodes each need a full data copy. Under storage-compute separation, multiple compute nodes share one storage system—no full copies needed—cutting storage costs significantly.

3) Stronger High Availability​

PolarDB's storage layer typically uses a distributed replica mechanism:

Storage Node
 ├── Replica1      # multiple replicas guarantee durability
 ├── Replica2
 └── Replica3

When a node fails, the system can recover extremely quickly. This design enables automatic failure recovery, a highly available architecture, and durability guarantees.

4) Extreme Performance​

PolarDB's storage layer typically uses distributed log structures, parallel I/O, and SSD cloud storage, which dramatically improves database performance. The main bottleneck shifts from the storage layer to internal network I/O.

PolarDB's Drawbacks​

While storage-compute separation has many advantages, it also comes with challenges.

1) Network Latency​

Separating compute from storage means data access goes over the network, adding some latency compared with local disks.

2) High Architectural Complexity​

A storage-compute separated architecture requires designing a distributed storage system, network protocols, and data consistency mechanisms—far harder to build than a traditional database.

3) Heavy Dependence on Cloud Infrastructure​

This architecture depends heavily on high-performance networking, distributed storage, and cloud platform scheduling, so it generally only fits cloud environments.

PolarDB vs AWS Aurora​

PolarDB and Aurora are very close in overall design philosophy. Both are archetypal cloud-native database architectures, and both adopt compute + storage decoupling.

Aurora separates compute nodes from a distributed storage cluster, with multiple compute nodes sharing the same storage system. A sketch of the architecture:

          Writer Node
               |
       +-------+-------+
       |               |
   Reader1         Reader2         # readers share the underlying storage
       |               |
   +-----------------------+
   |   Distributed Storage |
   +-----------------------+

Aurora's distributed storage is typically deployed across multiple Availability Zones for higher availability. In their concrete implementations, however, the two still differ in important ways.

1) Storage Architecture Differences​

Aurora's core design is log-structured storage. In Aurora, compute nodes do not write database pages directly; instead they send the redo log to the storage nodes, and the storage layer reconstructs data pages from those logs. Simplified, the flow is:

Client
↓
Aurora Compute Node
↓
Redo Log                    # only logs are shipped, not full data pages
↓
Distributed Storage Nodes
↓
Storage nodes reconstruct data pages

Aurora's storage layer typically consists of 6 replicas spread across 3 Availability Zones (AZs):

AZ1 : Storage Node A / Storage Node B
AZ2 : Storage Node C / Storage Node D
AZ3 : Storage Node E / Storage Node F

Writes commit with just a 4/6 quorum. This design brings several advantages:

• Writes only ship logs, so network traffic is smaller
• No need to fully replicate database pages
• Faster failure recovery

PolarDB's storage implementation is somewhat different. PolarDB uses a shared distributed storage architecture: compute nodes access the underlying PolarStore / PolarFS distributed file system directly, and database pages are still stored as pages. Simplified structure:

Client
↓
Compute Node (MySQL / PG / Oracle)
↓
PolarFS / PolarStore        # distributed file system, page-based storage
↓
Distributed Storage Nodes

This design stays closer to how traditional databases store data, so the data page format remains compatible and the changes to the database kernel are relatively small. On the write path, however, the amount of data going over the network can be larger than with Aurora's log-shipping approach.

2) Read Node Architecture​

Aurora's read nodes are Reader Instances that generally handle reads only, with writes concentrated on the Writer Node. PolarDB supports a Reader Node + primary node with shared storage model, where multiple nodes access the shared storage directly, enabling more flexible read scaling.

3) Ecosystem and Compatibility Strategy​

Aurora is a database engine deeply reworked by AWS; although compatible with MySQL / PostgreSQL, its internals have changed substantially. PolarDB places more emphasis on native database compatibility:

• PolarDB MySQL
• PolarDB PostgreSQL
• PolarDB Oracle

As a result, migrating applications to PolarDB often costs less in changes.

Wrapping Up​

The essence of storage-compute separation is splitting a database's compute and storage into two layers that scale independently. Through shared distributed storage, PolarDB gains scale-out in seconds, low-cost read scaling, and stronger high availability—at the price of network latency and higher architectural complexity. Compared with Aurora, the two share a philosophy, but PolarDB takes the shared file system route, staying closer to traditional database storage with better compatibility, while Aurora uses log-structured storage to minimize write traffic. Understanding the trade-offs between these two routes is genuinely useful when evaluating cloud database options.

When I first started leading a team, I took some wrong turns. Early on, I believed that being technically strong meant I could solve most problems by doing them myself. But over time I found that once a team grows, one person can't carry it. What a tech lead really needs to do is raise the capability of the whole team—not become the busiest person on it. Over these years of engineering management, I've gradually distilled a few lessons that are simple but important.

Think It Through First, Then Let the Team Build​

When engineering teams are inefficient, it's usually not because the engineers aren't capable—it's because the problem itself wasn't thought through. I've been there: a requirement lands, everyone starts coding immediately, and halfway through development we discover the design doesn't hold up and the work has to be redone.

Over time I built a habit: before a project starts, make sure the design approach and overall plan are thought through. What features the system needs, roughly how it should be broken apart, how the core modules should be designed, where the bottlenecks might be. If those things aren't clear, the harder the team works, the further off course it may drift.

So now the first thing I do is this: understand the problem clearly, then assign the work. It's not just more efficient—it also eliminates a lot of unnecessary rework.

Ownership Must Be Clear​

One of the most common problems in engineering teams is unclear responsibility. Many systems look like everyone is working on them, but when something actually breaks, nobody knows who to go to.

I gradually realized that for a team to run smoothly, responsibilities must be clearly divided. Every system needs an owner—someone who doesn't have to write all the code, but who must understand the system's architecture, core logic, and how it's running.

I usually assign an Owner to every core system—a service, a module, or a foundational component. When something goes wrong, the team can find the responsible person quickly, and nobody gets to pass the buck. Once ownership is clear, team efficiency improves noticeably.

A Tech Lead Shouldn't Do Everything Themselves​

When I first led a team, I couldn't resist writing a lot of the code myself. Doing it myself was faster, and I trusted the result. But over time it becomes clear that this approach holds the team back. If the tech lead does all the critical work personally, team members can't grow, and in the long run the team becomes ever more dependent on one person.

So I gradually changed my approach. Now I lean toward involving team members in design and decision-making, not just execution. In system design discussions, for example, I have everyone participate rather than deciding alone. It may be slower at first, but over the long run, the team keeps getting stronger.

Process Isn't There to Constrain—It's There to Make You Faster​

Many engineers hear "process" and think "more overhead." In reality, a mature team needs basic processes. In our team, for instance, there are a few fixed stages:

1. Requirements review
2. Technical design
3. Code review
4. Testing and verification
5. Release

The purpose of these processes isn't added complexity—it's fewer problems. Code review catches potential bugs early, technical design prevents architectural mistakes, and the testing process reduces production incidents. Once the process stabilizes, collaboration becomes much smoother.

A Tech Lead Has to Watch the Long Term​

Teams handling requirements every day can easily end up focused only on what's in front of them. But if technical debt goes unattended for long, the system gets harder and harder to maintain. So I usually set aside time to think about the architecture:

• Does the system have performance bottlenecks?
• Does the architecture need adjusting?
• Should we introduce new technical components?

These things may show no results in the short term, but over the long run they have a large impact on system stability. A tech lead has to solve today's problems while also considering where the system is headed.

Team Atmosphere Really Matters​

An engineering team's effectiveness depends not only on technical ability but also on its atmosphere. Where trust is missing between team members, everything gets harder.

One thing I care about is making sure people can discuss technical problems freely. Different people bring different perspectives, and those discussions often produce better solutions. I'm also not a fan of overly strict management—engineering teams work better in a relatively open environment where people can focus on doing the work well.

Engineering Management Is Its Own Kind of Growth​

Going from engineer to tech lead is a big transition. Before, you only had to care about code; now you care about the team, the projects, and the system as a whole. It feels uncomfortable at first, but gradually you find that engineering management is a new kind of challenge in its own right. It takes not just technical skill, but thinking and communication skills too.

For me, the greatest reward of leading a team is watching team members keep growing while the system keeps improving.

Wrapping Up​

After these years of engineering management, my biggest takeaway is actually quite simple: managing an engineering team isn't complicated management theory—it's enabling a group of engineers to get things done well together. With clear goals, the team's effort isn't wasted; with clear ownership, someone is accountable when things break; with stable processes, collaboration doesn't depend on luck. Add mutual trust on top of that, and most things simply start to flow.

Why Microservices Emerged​

In the early days of software, most applications used a monolithic architecture. Every functional module ran inside a single application process — the user system, order system, payment system, inventory system, and so on were all packaged into one project and deployed together. When the system is small, this is simple and direct, and development moves fast.

But as the business keeps growing, the problems of a monolith start to surface. The codebase balloons — a single project may contain hundreds of thousands or even millions of lines of code, and every change requires rebuilding and redeploying the entire system. Modules become tightly coupled, so a failure in one module can easily take down the whole application. And when traffic grows, you can't scale just the hot module — you have to scale the entire system, which wastes resources.

Against this backdrop, microservice architecture gradually became the mainstream design for large systems. The core idea is to break a massive system into multiple independent services, each owning a well-defined business capability that can be developed, deployed, and scaled independently. This reduces system complexity while improving maintainability and scalability.

Core Design Principles of Microservices​

Microservice architecture is not just about chopping a system into many small services. What matters more is designing the system around business capabilities. Each service should have a clear boundary of responsibility and be able to run on its own.

Microservice design typically follows a few core principles:

• Single responsibility: each microservice owns one core business domain — a user service, an order service, an inventory service, and so on. This keeps responsibilities from blurring together and makes it easier to divide work across teams.
• Service autonomy: each microservice has its own database and data model; multiple services should never share a database. This reduces hard dependencies between services and keeps the system loosely coupled.
• Independent deployment: each service can be released and upgraded on its own without affecting others. This matters enormously for large teams and dramatically improves engineering velocity.
• Interface-driven communication: services talk to each other through APIs or messaging systems, never by reaching into each other's databases or sharing code libraries.

Together, these principles form the foundation of microservice architecture.

Overall Architecture of a Microservice System​

In practice, a complete microservice architecture is built from multiple foundational components — it's much more than just splitting up services.

At the outermost layer sits the API Gateway. As the unified entry point, the gateway handles client requests and provides authentication, rate limiting, logging, and routing. It hides the complexity of the backend services so that clients only ever face a single unified interface.

Behind the gateway is the business service layer, made up of the individual microservices — user service, order service, product service, payment service, and so on. Each service owns an independent business capability and interacts with others via API or RPC.

To let services find and talk to each other, systems typically introduce a service registry and discovery component such as etcd, Nacos, or Consul. Services register themselves on startup, and other services look up target addresses through the registry.

High-concurrency systems usually also introduce a message queue (MQ) for asynchronous communication — Kafka, RabbitMQ, or RocketMQ. Message queues help smooth out traffic spikes, decouple services, and build event-driven architectures.

On the data side, each microservice generally has its own database — MySQL, PostgreSQL, or a NoSQL store. For performance, Redis is commonly added as a caching layer to take pressure off the database.

Finally, the whole system typically runs on a container platform such as Kubernetes, using container orchestration for automated deployment, scaling, and failure recovery.

Key Design Questions in Microservice Architecture​

For all its benefits, microservice architecture introduces new complexity of its own. Several key questions deserve close attention during system design.

1) Service Granularity​

Split too coarsely, and you still end up with a large monolith; split too finely, and you drown in services and complexity. The usual advice is to draw boundaries along business domains, not technical modules.

2) Service Communication​

Synchronous calls usually go over HTTP or gRPC, while asynchronous communication goes through message queues. In high-concurrency systems, heavy synchronous calling can create tangled call chains between services, so communication patterns need to be designed deliberately.

3) Distributed Transactions​

In a monolith, database transactions guarantee data consistency. In a microservice architecture, each service has its own database, so traditional transactions no longer apply directly. The common solutions are eventual consistency, the Saga pattern, or a distributed transaction framework.

4) System Stability​

More services means more points of failure, so mechanisms like rate limiting, circuit breaking, and graceful degradation become necessary — implemented via a service governance framework or a Service Mesh for traffic control.

Scalability of Microservice Architecture​

One of the biggest advantages of microservices is elastic scalability.

In a traditional monolith, if one module becomes the bottleneck — say the order service is under heavy load — you have to scale the entire system. With microservices, you can scale just the order service, for example by adding instances behind a load balancer.

In cloud-native environments, Kubernetes can automatically scale service instances based on load: adding nodes during traffic peaks and scaling back down when traffic subsides, which greatly improves resource utilization. This on-demand elasticity makes microservices a natural fit for high-concurrency internet workloads.

The Challenges of Microservices​

Microservices solve many problems, but they are no silver bullet. Compared to a monolith, a microservice system is significantly more complex:

• Higher operational overhead: a system may contain dozens or even hundreds of services, each with its own deployment pipeline and monitoring, which demands a mature DevOps platform.
• Service governance: as the number of services grows, the call graph between them becomes very complex and requires dedicated governance tooling to manage.
• Inherent distributed-systems problems: network latency, data consistency, service dependencies, and so on. These barely exist in a monolith but become first-class concerns in a microservice system.

So when a system is still small, a monolith is often simpler and more efficient — microservice architecture is better suited to medium and large systems.

Wrapping Up​

At its core, microservice architecture reduces system complexity through service decomposition and autonomy, improving scalability and maintainability along the way. It lets the system grow flexibly with the business and enables teams to develop in parallel — which is exactly what large internet systems need.

But microservices are not a simple technology upgrade. They require supporting infrastructure: a service registry, an API gateway, a messaging system, monitoring, a container platform, and more. Only with this complete technical foundation in place can microservice architecture deliver its real value.

In real-world system design, architects need to weigh business scale, team capability, and system complexity to choose the architecture that fits — rather than chasing microservices for their own sake. Truly good architecture is not the most technically sophisticated option, but the one that strikes the right balance between complexity and payoff.

Why Multi-Level Caching​

In internet-scale systems, the database tends to become the performance bottleneck as the business grows. If a flood of requests hits the database directly, you get high latency, an overloaded database, and eventually exhausted connections and slow queries. That's why virtually every high-concurrency system introduces a cache to boost performance.

The most common setup uses Redis as a unified caching layer. Client requests hit Redis first; on a cache hit, the data is returned directly, and on a miss, the system queries the database and writes the result back into the cache. This alone can dramatically reduce database load.

But as the system keeps scaling, relying on Redis alone runs into new problems. Under high QPS, a huge number of requests hitting Redis simultaneously generates network overhead and puts pressure on Redis CPU. Meanwhile, certain hot keys get read constantly, and paying a network round trip to Redis every single time adds unnecessary latency.

To solve these problems, many large systems adopt a multi-level cache architecture, and the most common form is the L1 + L2 cache design.

The Basic L1/L2 Cache Architecture​

L1/L2 caching is essentially a tiered caching architecture, typically with two layers.

L1 Cache​

The L1 cache lives inside the application process and is also known as a local cache. For example:

• Go: bigcache / ristretto
• Java: Caffeine / Guava Cache
• Node: LRU Cache

Its defining characteristics:

• Extremely fast access (in-memory)
• No network communication
• Each service instance has its own copy

L2 Cache​

The L2 cache is typically a distributed caching system — most commonly Redis or Memcached — shared by all service instances.

Its defining characteristics:

• Shared across nodes
• Better data consistency
• Larger capacity
• Supports distributed deployment

The overall access flow: client request → service process → check L1 cache → on miss, check L2 cache → on miss again, query the database.

The L1/L2 Access Flow​

In a real system, a typical request flows like this.

First, the request enters the application service, which checks the L1 cache. If the data is there, it returns immediately — the lowest-latency path, usually on the order of microseconds.

On an L1 miss, the system queries the L2 cache (Redis). If Redis has the data, it returns the result and also writes it into the L1 cache, so subsequent requests can hit locally.

If Redis doesn't have it either, the cache has fully missed, and the system goes to the database. Once the database returns, the result is written into both Redis and the L1 cache, establishing fresh cache entries.

The whole flow boils down to:

L1 Cache → L2 Cache → Database

This way, the vast majority of requests are intercepted at the cache layers and never touch the database.

Why Two Tiers​

The two cache tiers each solve different problems, and combining them delivers a major performance boost.

Reducing Redis Load​

In a high-concurrency system, if every request goes to Redis, Redis itself can become the new bottleneck. With an L1 cache, hot data is served straight from application memory, cutting the number of Redis requests.

Cutting Network Overhead​

Talking to Redis means a network round trip. Even at 1ms of latency, that adds up under high QPS. The L1 cache is an in-process lookup — far faster than any network call.

Increasing Throughput​

With an L1 cache, the system can handle more requests without adding load to Redis or the database.

Faster Access to Hot Data​

In many business scenarios, a small set of hot data gets read constantly — popular products, user profiles, configuration data, recommendation results. The L1 cache serves these locally.

Pitfalls of L1 Cache Design​

The L1 cache performs beautifully, but it introduces problems of its own.

1) Data Consistency​

Since each service instance holds its own L1 cache, the copies across instances can drift apart. When data is updated, without a proper invalidation mechanism you may serve stale reads. The usual fix is a cache invalidation strategy — on update, notify all service instances to evict their local copies via a message queue or a pub/sub mechanism.

2) Capacity Control​

The L1 cache lives in application memory; if it grows unbounded, it can eat significant memory and hurt system stability. Eviction policies like LRU or LFU are typically used to cap its size.

3) Cache Penetration and Breakdown​

When a flood of requests targets nonexistent data, it can slam the database directly. Bloom filters or mutex-based locking are the usual defenses.

Cache Update Strategies​

In a multi-level cache system, the update strategy is critical. Common approaches:

1) Cache Aside​

The application checks the cache first; on a miss it queries the database and then populates the cache. This is the most common pattern.

2) Write Through​

The application updates the cache at the same time it writes to the database, keeping the two in sync.

3) Write Back​

The application writes only to the cache, and the cache system asynchronously flushes to the database. This offers higher performance but is more complex to implement.

Most internet-facing workloads go with Cache Aside, because it's simple to implement and reliable.

When L1/L2 Caching Fits​

Multi-level caching is a good fit for scenarios like:

• Read-heavy, high-concurrency systems: product detail pages, user profile lookups, config reads.
• Frequently accessed hot data: leaderboards, recommendation lists.
• Heavy database load: the cache layers can drastically cut database queries.

For systems with strict real-time consistency requirements — financial trading systems, for example — multi-level caching is usually avoided in favor of going straight to the database.

Wrapping Up​

The core idea of L1/L2 caching is layered interception: the local cache absorbs the hottest traffic, Redis holds the data shared across instances, and the database only handles true misses. This architecture significantly reduces latency and backend load — at the cost of multi-copy consistency, which must be designed alongside invalidation notifications, capacity control, and penetration defenses. Whether multi-level caching is worth it depends on your read/write ratio and consistency requirements — for consistency-sensitive scenarios, the simpler design is often the safer one.

• Service nodes need to share configuration
• The system needs service discovery
• Distributed locks need a coordination point
• The cluster needs a consistent state store

At their core, all of these problems call for a reliable distributed coordination system.

In the single-machine era, these needs could be met with a database table or even a plain file. But once you have multiple nodes, the question becomes: how do multiple replicas agree on the "current state"? Who has the final say? Can you still trust the data after a node dies? These are exactly the questions a distributed coordination system has to answer.

In the modern cloud-native world, the most widely used component for this is etcd.

For example:

• Kubernetes
• CoreDNS
• service mesh
• distributed configuration centers

All of these systems rely on etcd underneath.

What is etcd​

In short:

etcd is a highly reliable distributed key-value store.

But what sets it apart from an ordinary database is this:

It was designed specifically for distributed coordination.

That positioning dictates its trade-offs: it doesn't chase massive data volumes or extreme throughput; instead, it aims to make every piece of data you store trustworthy. So it's a great fit for small but critical data — metadata, configuration, state — and a poor fit as a general-purpose business database.

Its main features are:

• Strong consistency
• Distributed cluster support
• A Watch mechanism
• Transactions
• A Lease mechanism

The combination of these features is quite interesting:

1. Watch lets clients subscribe to changes on a key or prefix. When a config value changes, every node learns about it in real time — no polling required.

2. A Lease is essentially a heartbeat token with a TTL. Keys can be attached to a lease, and once the client stops renewing it, those keys are deleted automatically — this is exactly how service registration and failure eviction are implemented.

3. Transactions (Txn) provide compare-and-swap semantics: "write this key only if it doesn't exist" can be done atomically, which is the foundation of distributed locking.

Many distributed systems use etcd for:

• Service registries
• Configuration centers
• Distributed locks
• Leader election

Kubernetes, for example, stores the entire cluster state in etcd. kube-apiserver is the only component that reads and writes etcd directly; all other controllers observe state changes indirectly through the apiserver's Watch mechanism.

Strong consistency (the Raft consensus algorithm)​

Internally, etcd uses the Raft consensus algorithm.

The core idea of Raft is:

The cluster elects a single Leader node.

All write requests must go through the Leader.

The flow looks roughly like this:

Client → Leader → replicate to a majority of nodes → commit succeeds

Only when:

more than half of the nodes have acknowledged the write

is the data considered truly committed.

Let's break this down a bit further. In Raft, every node is in one of three roles: Leader, Follower, or Candidate. During normal operation there is exactly one Leader, which periodically sends heartbeats to all Followers. If a Follower doesn't receive a heartbeat within its timeout, it turns itself into a Candidate and starts an election; whichever node wins a majority of votes becomes the new Leader.

Writes go through log replication: the Leader first appends the write operation to its own log, then sends it to all Followers in parallel. Only when a majority (more than half) of nodes have persisted the log entry does the Leader mark it as committed and respond to the client.

The "majority" design is the key to strong consistency: any two majority sets must overlap, so even if the Leader crashes, the newly elected Leader is guaranteed to contain all committed data — no committed writes are lost.

This also explains why an odd number of etcd nodes (3, 5, 7) is recommended:

• 3 nodes tolerate 1 failure; 5 nodes tolerate 2
• 4 nodes also only tolerate 1 failure (a majority requires 3), so fault tolerance is the same as 3 nodes, with one extra replica's worth of sync overhead

Setting up an etcd cluster​

Deployment environment:

Operating system:

Debian 11.5.0

1 Download etcd​

Download the etcd binaries from GitHub:

https://github.com/etcd-io/etcd/releases

The release page provides archives per platform; just pick the one for your architecture (e.g. linux-amd64). etcd ships as a single static binary with no extra runtime dependencies — one of the reasons it's so easy to deploy.

2 Install etcd​

After extracting, place the following files into:

/usr/local/bin

Mainly these two:

etcd
etcdctl

The division of labor is clear: etcd is the server process, and etcdctl is the command-line client — all subsequent queries, writes, and maintenance go through it. Putting them in /usr/local/bin gets them onto the PATH so you can invoke them from any directory.

3 Set execute permissions​

# The extracted files may lack the executable bit; add it manually
sudo chmod +x /usr/local/bin/etcd
sudo chmod +x /usr/local/bin/etcdctl

At this point, the binaries for a single node are ready. In cluster mode, repeat the installation steps above on every node, then declare each node's name, listen addresses, and the initial cluster member list via startup flags. Nodes talk to each other over port 2380, and clients connect on port 2379. My environment has TLS enabled, with certificates stored under /opt/etcd/ssl, so the verification commands below all need certificate arguments.

4 Verify the cluster​

You can use etcdctl to inspect the data:

# List all keys in the cluster (keys only, no values)
# --cacert/--cert/--key: CA and client certificates for mutual TLS
# --endpoints: list all three nodes; the client load-balances and fails over automatically
ETCDCTL_API=3 etcdctl \
--cacert=/opt/etcd/ssl/ca.pem \
--cert=/opt/etcd/ssl/server.pem \
--key=/opt/etcd/ssl/server-key.pem \
--endpoints="https://172.24.93.151:2379,https://172.24.93.149:2379,https://172.24.93.150:2379" \
get / --prefix --keys-only

If the command returns results, the client-to-cluster path, the certificates, and quorum are all working. --prefix enables prefix matching; combined with / it effectively walks every key.

Delete keys:

# Delete by prefix; the empty prefix "" matches ALL keys — use with extreme caution in production
ETCDCTL_API=3 etcdctl \
--cacert=/opt/etcd/ssl/ca.pem \
--cert=/opt/etcd/ssl/server.pem \
--key=/opt/etcd/ssl/server-key.pem \
--endpoints="https://172.24.93.151:2379,https://172.24.93.149:2379,https://172.24.93.150:2379" \
del --prefix ""

Why etcd beats Redis for distributed coordination​

A common question:

Can't Redis do distributed locks too?

It can.

But Redis is an AP system, while etcd is a CP system.

Put simply:

Redis prioritizes availability and performance.

etcd prioritizes consistency and reliability.

For locking specifically, the difference shows up in failure behavior: Redis master-replica replication is asynchronous. If the master writes a lock and crashes before replicating it, the promoted replica has no record of the lock — it's simply "lost", and two clients can hold the lock at the same time. The Redlock algorithm tries to mitigate this, but its safety has long been debated in the community. etcd writes, on the other hand, must be confirmed by a majority before they succeed; a Leader failover never loses committed data, so the lock's semantics hold even under failure.

The cost, of course, is performance: every write requires a round of majority replication plus an fsync to disk, so latency and throughput are far behind Redis. So the conclusion isn't that one is better than the other — it's about matching the tool to the scenario. Use Redis for caching, counters, and lightweight locks that can tolerate occasional failure; use etcd for configuration, leader election, and locks that require strict mutual exclusion.

Pitfalls and notes​

A few things worth recording from the setup and usage process:

1. Don't forget the ETCDCTL_API=3 environment variable. Older versions of etcdctl default to the v2 API, and v2 and v3 data are completely isolated — using the wrong API version produces the illusion that "I wrote the data but can't read it back". Newer versions default to v3, but declaring it explicitly in scripts never hurts.

2. List all nodes in --endpoints. If you list only one node and it goes down, the client loses connectivity entirely; with all nodes listed, the client fails over to healthy nodes automatically.

3. Certificate paths and SANs must match. With TLS enabled, the IPs/domains in the certificate must cover the addresses actually used in the endpoints, or the handshake will be rejected.

4. Keep the node count odd. The Raft section above explains why: an even node count adds no fault tolerance, only synchronization cost.

5. etcd is sensitive to disk latency. Every Raft log commit requires an fsync; a slow disk directly inflates write latency and can even trigger Leader election churn. Use SSDs if you can.

Wrapping up​

etcd is a textbook case of trading performance for consistency: Raft's majority commit guarantees that any committed data you read is trustworthy at any moment, and Watch, Lease, and transactions turn that trustworthy state store into a general-purpose foundation for service discovery, leader election, and distributed locking. The setup itself is simple — two binaries and a set of startup flags. What actually deserves your attention is TLS certificates, node planning, and a clear-eyed judgment of where etcd fits. Figuring out the division of labor between etcd and Redis, and using each where it excels, is far more useful than arguing about which one is better.

From batch processing to stream processing​

Traditional data processing is mostly offline: run a job every night to aggregate user behavior, compute ad metrics, generate reports, and analyze business KPIs. This approach is called batch processing, with typical tools such as:

• Hadoop MapReduce
• Hive
• Spark

These systems are great at crunching massive volumes of historical data. But as the internet grew, more and more businesses started needing real-time data processing, for example:

• Real-time risk control
• Real-time recommendations
• Real-time monitoring
• Real-time ad bidding
• Real-time log analysis

These scenarios share one trait: data must be processed as it is produced — you can't wait until tomorrow. This gave rise to a new computation model: stream processing.

Why Flink was needed​

Early real-time systems mostly relied on Storm and Spark Streaming, but both had issues.

Storm is a genuine stream processor, but development is complex, fault tolerance is mediocre, and state management is hard. Spark Streaming is stable, but it is fundamentally micro-batch processing — in other words, not true streaming. For example:

Process one batch of data every second

For many scenarios, this still isn't real-time enough. So Flink emerged, with a very clear goal: build a true stream processing system.

What problems Flink solves​

1) Real-time data processing​

Flink delivers true millisecond-level real-time computation. Take a user clicking on a product:

Click → Kafka → Flink → real-time recommendation

The whole pipeline can complete in tens of milliseconds.

2) State management​

One crucial problem in stream processing is state. Take counting user clicks:

User A clicks: 1 time
User A clicks: 2 times
User A clicks: 3 times

The system has to remember what it saw before. Flink has built-in state management (Keyed State, Operator State) and supports both local state and RocksDB-backed state storage, which allows it to handle state at enormous scale.

3) Fault tolerance​

In distributed systems, node failures are the norm. Flink protects data through its checkpoint mechanism: the system periodically saves computation state, and if a node goes down, computation resumes from the checkpoint. This mechanism guarantees exactly-once processing — each record is processed exactly one time.

4) Event time​

Stream processing has another notoriously hard problem: event time. Log data may arrive late, and if you compute purely by system time, your statistics come out wrong.

Flink introduced the watermark mechanism. Watermarks help the system decide when data has "mostly all arrived", which makes it possible to correctly handle late data, window computations, and time-based aggregations.

Flink's core design ideas​

1) Stream First​

Flink's core philosophy is that everything is a stream — batch processing is just a bounded stream. As a result, Flink runs batch and streaming on the same engine.

2) State Driven​

Flink is a state-driven computation system: all computation revolves around state. User behavior analytics, real-time aggregation, real-time monitoring — all of them depend on state management.

3) Exactly Once​

Flink puts heavy emphasis on correctness of data processing. Through checkpoints, distributed snapshots, and two-phase commit, Flink achieves exactly-once semantics — a capability that many real-time systems consider essential.

Flink in real-world systems​

Flink is used extensively across internet companies, in scenarios such as real-time recommendation systems, user behavior analytics, ad click counting, real-time risk control, and real-time log analysis. A common data pipeline looks like:

User behavior → Kafka → Flink → ClickHouse

Flink sits in the middle, handling data cleaning, real-time computation, and real-time aggregation.

Wrapping up​

Flink exists, at its core, to do "true stream processing". It unifies batch and streaming on a single engine, supports large-scale stateful computation with built-in state management, buys exactly-once correctness with its checkpoint mechanism, and handles out-of-order and late data with watermarks. Once you understand these points, you understand the fundamental difference between Flink and earlier real-time systems — and the specific APIs and cluster deployment will make much more sense afterwards.

When I first picked up Netty, I went straight to its API and example code — and got more confused the more I read. Why does EventLoop exist? Why are connections and reads/writes handled by two separate thread groups? It took me a while to realize that these designs didn't appear out of thin air; they exist to solve inherent problems in traditional network programming. Learning Netty while skipping this history is like memorizing answers without reading the questions.

So this post holds off on code and first sorts out the design ideas behind the NIO networking model that Netty is built on. Once you grasp this main thread, every Netty component falls into place.

The problem with traditional network programming​

In early Java network programming, most programs used the BIO (Blocking IO) model.

BIO "blocks" in two places: accept() blocks while waiting for new connections, and read() blocks while waiting for data to arrive. Once a thread is blocked on a connection's read or write, it can't do anything else.

For example:

The server creates a new thread for every incoming client connection.

one connection = one thread

The upside of this model is that it's simple and intuitive to program — each thread only cares about its one connection, reading it from start to finish. With a small number of connections, it works perfectly fine.

But with many connections, say:

10,000 connections = 10,000 threads

several serious problems emerge:

Massive thread resource consumption​

Threads themselves cost memory and scheduling overhead. Every Java thread occupies its own stack memory; with tens of thousands of connections, the thread stacks alone can eat a huge amount of memory — before even counting the OS's overhead for maintaining thread structures.

Heavy context-switching overhead​

The CPU has to switch between threads constantly. Every switch means saving and restoring registers and flushing caches. When the thread count far exceeds the number of CPU cores, a significant share of CPU time goes into the switching itself rather than actual work.

Poor scalability​

As connections pile up, the system becomes prone to collapse. What's more awkward: most of those connections are actually idle (in long-lived connection scenarios, most clients aren't sending data most of the time), yet each one hogs a thread that just sits there waiting — resources wasted on waiting.

So traditional BIO is simply not suited for high-concurrency network services.

The core idea behind NIO​

To solve BIO's problems, Java introduced NIO (Non-Blocking IO).

The core idea of NIO is remarkably simple:

Manage a large number of connections with a small number of threads

The shift in thinking is this: since most connections are idle most of the time, don't have threads babysit connections. Instead, hand the job of watching "which connection has data" over to the operating system, and have threads step in only when an event actually occurs. This is the idea of IO multiplexing, which on Linux corresponds to system calls like select/poll/epoll.

It relies on three core components:

Channel
Buffer
Selector

Together they form the core architecture of NIO. Let's look at each in turn.

Channel​

A Channel can be understood as:

a pipe for data transfer

Unlike traditional IO:

Traditional IO:

input stream / output stream

A Channel, however:

is bidirectional

It can both read and write. A Channel is essentially an abstraction of a network connection.

In Java NIO, the common implementations are ServerSocketChannel (listens on a port and accepts connections) and SocketChannel (represents a specific TCP connection). A Channel can be set to non-blocking mode — the prerequisite for being managed by a Selector: when there's no data to read, it returns immediately instead of stalling the thread.

Buffer​

In NIO, all data must first pass through a Buffer.

Think of it as:

a temporary staging area for data

The data flow:

network -> Buffer -> program
program -> Buffer -> network

A Buffer is essentially a stateful region of memory, tracked internally by a few pointers — position, limit, capacity — that record "how far we've written, how far we can read". Switching between read and write modes requires methods like flip(), which is one of the reasons the raw NIO API is famously easy to get wrong. Netty later implemented its own ByteBuf largely to escape this awkward machinery.

Selector​

The Selector is the most central component of NIO.

Its job is:

managing multiple Channels with a single thread

That is:

one thread
monitoring many connections

It works roughly like:

event polling

Concretely, when a Channel registers with a Selector, it declares which event types it cares about: a new connection arriving (ACCEPT), data readable (READ), ready to write (WRITE), and so on. The thread calls select() and blocks; as soon as any Channel becomes ready, the call returns the ready set and the thread processes them one by one.

The flow:

Selector
   |
monitors multiple Channels
   |
which Channel has an event
   |
process that one

This way, the thread spends all of its time "handling ready events" rather than "waiting on a particular connection" — which is precisely the fundamental difference between NIO and BIO.

The Reactor threading model​

The Reactor threading model is a design pattern widely used in high-concurrency network servers. Its core idea: handle a large number of network connections with a small number of threads, driven by events. Instead of creating a dedicated thread per connection, threads listen for network events (connect, read, write, etc.) and process a connection only when data actually arrives on it. This avoids the resource consumption and context-switching problems of massive thread counts, dramatically improving a server's concurrent capacity.

The Reactor model typically involves several key roles:

1. Event monitoring (Reactor)​

Listens for network events, such as new connections arriving, data becoming readable, or the socket becoming writable.

2. Connection acceptance (Acceptor)​

When a new client connects, accepts the connection and registers it with the downstream processing threads.

3. Event dispatching (Dispatcher)​

Routes different network events to their corresponding handling logic.

4. Business processing (Handler)​

Executes the actual business logic: parsing the protocol, handling the request, returning the result.

The Reactor model itself has evolved through several forms: single Reactor, single thread (one thread does both monitoring and processing); single Reactor, multiple threads (monitoring on one thread, processing handed to a thread pool); and master-worker Reactor (connection acceptance and read/write processing handled by separate Reactor thread groups). The higher the connection volume, the more important it is to separate acceptance from read/write so they don't drag each other down.

In Netty, the Reactor model typically shows up as the BossGroup + WorkerGroup thread structure, which maps exactly to the master-worker Reactor form:

• Boss threads: accept client connections
• Worker threads: handle network reads/writes and business logic

Through this design, Netty can handle tens of thousands of connections with a small number of threads — the core reason it achieves high-performance network communication.

Pitfalls and notes​

A few things worth knowing up front when learning and using this model:

1. Never do slow work on a Worker thread. The Reactor model assumes event handling is fast. A Worker thread typically serves many connections; if a Handler runs a slow query or a synchronous remote call, every other connection on that thread gets stuck along with it. Time-consuming business logic belongs in a separate business thread pool.

2. Raw NIO APIs are very easy to get wrong. Buffer read/write mode switching and Selector event-handling details are full of traps — which is why real-world projects almost always choose Netty over hand-rolled NIO: it encapsulates all that complexity away.

3. NIO wins by saving threads, not by making individual IO faster. With very few connections, BIO isn't necessarily worse; the advantage of NIO/Reactor shows up when huge numbers of connections coexist. Choose your technology based on the scenario.

Wrapping up​

To recap the main thread of this evolution:

• BIO's problem is "one connection, one thread" — threads sit wasted on idle connections;
• NIO uses Channel + Buffer + Selector to achieve IO multiplexing, letting a few threads handle only ready events;
• The Reactor model defines, on top of that, the division of responsibilities for event monitoring, dispatching, and handling;
• Netty's BossGroup + WorkerGroup is the production-grade implementation of the master-worker Reactor.

With this thread in mind, Netty's concrete components — EventLoop, Pipeline, ByteBuf — no longer look like designs that came out of nowhere. I'll cover Netty's core components and hands-on usage separately in a follow-up.

For example:

• Users grabbing coupons
• Scheduled task execution
• Inventory deduction
• Order status updates

If the system runs as a single process, it's easy — a local lock (mutex) solves it. But with a microservice architecture or cluster deployment, the problem changes: the system may have 10 service instances, 100 workers, even multiple data centers. Local locks become completely useless, because different processes have no idea about each other's lock state.

This is where the concept of a distributed lock comes in. Its goal is simple:

In a distributed environment, guarantee that only one node can execute a given piece of logic at any moment.

Why can Redis serve as a distributed lock?​

When implementing a distributed lock, many people's first instinct is the database, for example:

-- Mutual exclusion via database row locks
select ... for update

But database locks have problems: poor performance, coarse granularity, and heavy pressure under high concurrency. So people looked for a system better suited to locking — and Redis fits very well, because it naturally has three advantages.

1) Single-threaded model​

Redis executes its core commands on a single thread — only one command runs at a time. So an operation like:

# NX: write only if the key doesn't exist — inherently mutually exclusive
SET key value NX

is an absolutely atomic operation, with no possibility of a race condition.

2) In-memory operations, extremely fast​

All Redis data lives in memory. A lock operation is usually just one SET and one DEL, with latency typically in the tens of microseconds — far faster than a database.

3) Built-in expiration​

Redis keys can carry a TTL, for example:

# NX: create only if absent; EX 10: auto-expire after 10 seconds
SET lock:order 123 NX EX 10

This way, even if the service crashes, the lock won't be stuck forever.

Releasing a Redis lock​

You can't just DEL to release a lock, because this can happen:

Thread A acquires the lock
Thread A runs past the timeout
The lock expires
Thread B acquires the lock
Thread A finishes and deletes the lock

Now thread A has mistakenly deleted thread B's lock. The correct approach is to only delete the lock you actually hold, usually via a Lua script:

-- Compare the lock's value (unique identifier) first; delete only on match,
-- ensuring you only release your own lock
if redis.call("GET",KEYS[1]) == ARGV[1] then
    return redis.call("DEL",KEYS[1])
else
    return 0
end

Lua scripts execute atomically as a whole in Redis — no other command can slip in between the GET and the DEL — which is what makes the release safe.

Classic problems with Redis distributed locks​

Although a Redis lock is simple, real systems hit plenty of pitfalls. The two most common are below.

1) Lock expiration​

If the business logic runs longer than the TTL, the lock is released early, and other nodes may enter concurrently. The fix is automatic renewal (WatchDog). For example:

• Set the lock TTL to 10 seconds
• When the remaining validity enters its last 30%, renew once automatically, with random jitter added so that renewal requests don't all hammer Redis at the same instant

As long as the task is still running, the lock never expires.

2) Redis as a single point of failure​

If Redis goes down, every lock is lost. Common solutions:

• Redis Sentinel
• Redis Cluster
• The RedLock algorithm

Redis's AP design versus the CP architecture of etcd and ZooKeeper​

Redis is fundamentally an AP (availability-first) system: during network partitions or node failures, Redis prioritizes keeping the service available over keeping data strictly consistent. This introduces a risk — in extreme cases, two clients can both believe they hold the lock.

For example:

Client A acquires the lock on the master node
The master hasn't yet replicated it to the replica
The master suddenly crashes
The replica is promoted to the new master

The new master has no idea A ever took the lock, so client B can acquire it again — and now two clients hold the lock simultaneously. This is the theoretical consistency risk of Redis distributed locks.

By contrast, systems like etcd and ZooKeeper are built on the CP model (strong consistency), using the Raft or ZAB protocol internally to commit data only after a majority of nodes confirm it. Once a lock is acquired, the entire cluster agrees on it, and multiple clients can never hold the lock at once.

Hence a common rule of thumb in practice: for business-level locks (inventory, task control, and the like), a Redis distributed lock is usually good enough; but for scenarios demanding strong consistency — financial transactions, global schedulers, distributed coordination services — many systems opt for strongly consistent lock implementations on etcd / ZooKeeper.

Wrapping up​

The core idea of a Redis distributed lock is dead simple: use Redis's atomic operations to achieve mutual exclusion. It comes down to three key points: atomic acquisition, safe release, and automatic expiration or renewal. In practice, a mature Redis lock also adds retry logic, WatchDog auto-renewal, Lua-based safe release, and a unique lock identifier. And you have to stay clear about its AP nature — good enough for business-level mutual exclusion, but for strong-consistency scenarios, look to etcd / ZooKeeper. Only with all of these pieces combined does it become a genuinely stable, reliable distributed locking system.

What Is an Algorithm?​

An algorithm is, at its core, a set of steps for solving a problem.

Anything that:

takes some input data
processes it according to defined rules
produces a result

—that whole process is an algorithm.

It doesn't have to be sophisticated. Looking up a word alphabetically in a dictionary is an algorithm; so is a recipe—both are explicit, repeatable sequences of steps. The only difference when writing programs is that the steps will be executed by a computer, so every step has to be precise, with no room for ambiguity.

Understanding Algorithms​

Why should programmers care about algorithms? Because for the same problem, different algorithms can differ enormously in efficiency. One approach might finish in seconds; another might run for minutes or longer. As the data grows, the gap widens—which is why the performance of many systems comes down largely to algorithm design.

This gap is usually described with "time complexity," the familiar big-O notation. It doesn't care how many milliseconds a particular run took; it cares how the amount of work grows as the data grows: a linear scan is O(n)—double the data, double the work; binary search is O(log n)—double the data, just one extra lookup; two nested loops give you O(n²)—double the data, four times the work. With small inputs everything is fast, but once you hit hundreds of thousands or millions of records, the difference in growth curves is the difference between "usable" and "unusable."

Algorithms rarely exist in isolation; they almost always work together with data structures. Data structures organize the data—arrays, linked lists, trees, hash tables—while algorithms operate on that data. Simply put, one stores the data, the other processes it, and only together do they make a program efficient.

The two choices influence each other. For "check whether an element exists," an array requires scanning from start to end, while a hash table locates it in essentially one step; for "frequent insertions in the middle," an array has to shift elements around, while a linked list only needs pointer updates. Often, switching to a more suitable data structure makes the algorithm fast on its own—how you model the problem determines which algorithms are even available afterward.

In real systems, algorithms are everywhere. Search engines rank web pages with algorithms, short-video platforms use them to recommend content, navigation apps use them to compute the shortest route, and e-commerce platforms use them for product recommendations and ranking. Much of the software we use every day has all kinds of algorithms working behind the scenes.

Learning algorithms isn't about how many problems you've solved—it's about developing a way of thinking about problems. When you face a problem, being able to quickly come up with several approaches and pick the more efficient one—that's the real value of algorithms. Many experienced engineers are, in essence, continuously refining how they solve problems.

What Should You Focus On When Learning Algorithms?​

Many people fall into the trap of grinding problem sets.

What actually matters more is understanding:

1. How to model the problem

Translate a fuzzy real-world requirement into clear inputs, outputs, and constraints. Get the model right and the problem is often half solved; get it wrong and no amount of fast coding afterward will help.

2. How to design the data structures

Choose structures based on access patterns: heavy lookups call for a hash table, ordered traversal suggests a tree, first-in-first-out means a queue. Pick the right structure and the code naturally becomes simpler.

3. How to optimize time complexity

First estimate whether the current approach can handle the target data volume, then decide whether to optimize. Not all code deserves optimization, but you should know where the bottleneck is.

4. How solutions evolve

Start from the brute-force solution, observe where computation is repeated, and improve step by step toward something better. That derivation process is far more valuable than memorizing the optimal solution.

Real algorithmic skill isn't about how many problems you've memorized. It's this: see a problem, quickly think of a way to solve it.

Pitfalls and Caveats​

• Don't chase the optimal solution right away. Write a working brute-force solution first to confirm you understand the problem, then talk about optimization. Often the brute-force approach is good enough at real-world data sizes.
• Complexity analysis has to account for data volume. When n is only a few hundred, the difference between O(n²) and O(n log n) is negligible; don't complicate simple code just to show off.
• If you only memorize conclusions from solutions without working through the reasoning, you'll forget them in two weeks. Close the solution and re-derive it yourself—that's when you've actually learned it.
• Don't ignore space complexity. Trading space for time is a common technique, but caches and hash tables eat memory too—know what the cost is.

Wrapping Up​

An algorithm is a set of steps for solving a problem; data structures organize the data; together they determine a program's efficiency. The core tool for judging an algorithm is complexity analysis, which describes how the amount of work grows as the data grows. The point of learning algorithms isn't the number of problems solved but a few key skills—modeling, choosing structures, deriving solutions. Practice those until they're second nature, and when a new problem shows up, you'll know where to start.

Stage One: The Birth of AI (1950–1980)​

In 1950, the British scientist Alan Turing posed a question:

"Can machines think like humans?"

He proposed a now-famous test (the Turing test). In simple terms: if a human chats with a machine and can't tell which one is the machine, the machine counts as "intelligent."

In 1956, a conference in the United States formally coined the term:

Artificial Intelligence

From then on, AI became an official field of research. But computers at the time were weak and data was scarce, so AI mostly stayed in the realm of theory.

Stage Two: AI Gets Somewhat Useful (1980–2010)​

As computers grew more powerful, some AI techniques started to land in the real world, such as:

• Speech recognition
• Machine translation
• Recommendation systems

Many internet products were using AI long before anyone called it that, for example:

• Taobao product recommendations
• Google search ranking
• Netflix movie recommendations

But AI back then was still fairly "dumb"—it could basically only handle a single task at a time.

Stage Three: The Era of Large Models (2010–Present)​

Three factors truly set AI off:

1) Compute got stronger​

GPU compute exploded.

2) Data got bigger​

The internet generated massive amounts of data.

3) Algorithms broke through​

Especially deep learning + the Transformer architecture.

In 2017, Google published a paper:

Attention Is All You Need

That paper directly gave rise to today's large models, and with them a wave of AI products:

• ChatGPT
• Claude
• Gemini
• DeepSeek

Almost overnight, AI went from a "tool" to an intelligent assistant that can converse, write, code, and draw.

Where Does AI Stand Today?​

Ten years ago, most people would have seen AI as a lab technology or a collection of amusing toys. That's no longer the case at all. Today's AI has gone from novelty to genuine productivity tool, and it's pulling its weight in more and more industries.

The most visible change is in software development. Many programmers now use AI to help write code day to day, with tools like Cursor, Copilot, ChatGPT, and Claude. AI can write functions and generate scripts, help hunt down bugs, explain code logic, and even scaffold a complete program from a requirement. Across many teams, adopting AI has lifted development productivity by 30–50%.

AI is also widely used in content creation. It can now write news articles, fiction, ad copy, and summary reports, and plenty of independent creators use AI to draft articles, organize material, or generate ideas. The final content usually still needs human editing, but AI handles a large share of the groundwork, dramatically speeding up the creative process.

In enterprise services, AI is starting to take over a lot of repetitive work—customer support being the obvious example. Many e-commerce platforms, banks, and telecom operators now run AI support systems for common questions. Compared with human agents, AI responds 24/7, handles huge request volumes simultaneously, and slashes labor costs.

Visual content production is advancing fast too. Tools like Midjourney and DALL·E can generate high-quality images from text, and video generation from Sora, Runway, and others is maturing. Short videos, ad design, and even film production are starting to use AI to help generate scenes, characters, and animation.

AI is playing a growing role in data analysis as well. Many companies use AI to analyze reports automatically, generate BI dashboards, and even offer trend assessments and decision recommendations. Some have started deploying "AI data analyst" assistants that help management understand data and make decisions faster.

Which Industries Has AI Already Reached?​

AI has quietly worked its way into a great many industries. Content recommendation on platforms like Douyin, Taobao, and Bilibili is essentially AI deciding what you see. Finance uses AI for risk control, fraud detection, and investment analysis—many banks have AI models running inside their core systems. In medicine, AI already assists doctors with diagnosis and medical imaging, reading X-rays and CT scans, sometimes faster and more accurately than humans. In manufacturing, AI powers factory automation, quality inspection, and predictive maintenance, making factories steadily smarter. In IT, AI is entering operations: automated monitoring, log analysis, automated troubleshooting and scaling—we may even see "AI ops assistants" helping engineers manage complex systems. AI has gone from a lab technology to foundational infrastructure behind many industries.

What Does AI Mean for Humanity?​

Many people worry about AI replacing humans. The more realistic picture is this: AI isn't here to replace people—it's here to amplify them.

Look back at the history of technology and you'll see humans have always used tools to amplify their own abilities. The steam engine gave us physical power far beyond our bodies; computers gave us extraordinary computational power; the internet made information travel with essentially no distance limits. What AI amplifies is our capacity for intellectual work.

Things that used to take a whole team can now be done by one person plus AI—writing code, analyzing data, producing content, doing design, drafting operations plans. AI rapidly generates ideas, organizes information, and grinds through the groundwork, while humans set the direction, make the decisions, and own the results.

Put another way, the working model of the future may well be:

One person + AI = what used to be a small team.

So over the long run, what AI really changes isn't "whether people lose their jobs" but how work gets done. Repetitive, standardized work will increasingly be handled by AI, while humans focus on creating, judging, deciding, and innovating.

AI isn't humanity's opponent—it's more like a new kind of tool. Just as computers and the internet reshaped the world, AI is becoming a new foundational capability, gradually woven into every industry.

Where Is AI Headed?​

Based on current trends, AI is likely to develop along a few lines. First, it will evolve from a "chat tool" into agents that can complete tasks on their own (AI Agents)—writing code, analyzing data, operating systems autonomously; you give it a goal and it works through the steps itself. Second, AI will increasingly enter the physical world, pairing with robots, autonomous driving, and smart factories, so machines don't just "think" but also "act." Third, AI will become a foundational capability, embedded in software and systems the way the internet and electricity are—soon nearly every application will ship with AI features. In short, future AI won't be just a tool but something closer to humanity's "intelligent assistant" and "productivity multiplier," helping individuals and companies get more done with fewer people.

Money and Cost: The State of AI Investment​

Globally, AI has become one of the most capital-intensive areas in tech, with governments, tech companies, and investors all doubling down. According to Stanford's AI Index 2025 report, global corporate investment in AI reached about $252.3 billion in 2024, an all-time high.

The United States remains the biggest investor: US private AI investment hit $109.1 billion in 2024, roughly 12 times China's (about $9.3 billion) and 24 times the UK's ($4.5 billion).

Within that, generative AI (large models) is the fastest-growing segment: global private investment in generative AI reached $33.9 billion in 2024, up about 18.7% year over year.

Meanwhile, enterprise adoption is climbing fast. Data shows about 78% of companies were using AI in their business in 2024, up sharply from 55% the year before—AI is shifting from "technology experiment" to standard business tooling.

In market terms, the AI industry is in a high-growth phase: the global AI market is around $391 billion in 2025 and projected to exceed $1.8 trillion by 2030, making it one of the fastest-growing tech industries ahead.

Simply put, today's AI boom is driven by massive combined investment in capital, compute, and enterprise adoption. Tech giants, investors, and governments keep pouring in resources, each hoping to lead the coming AI industry race.

Wrap-Up​

AI has traveled more than 70 years—from a thought experiment of Turing's, to single-task tools, to today's large models that converse, write code, and generate images. Behind it all is the long accumulation of compute, data, and algorithms, not some overnight explosion. AI has already permeated software development, content creation, finance, medicine, and many other industries, backed by ever-growing investment. For individuals, rather than worrying about being replaced, treat it as a tool that amplifies your own abilities—and learn to work with it sooner rather than later.

https://github.com/go-lynx

The headline feature: Plug-and-Play—genuinely works out of the box, turning complex microservice architecture into "building blocks."

It doesn't reinvent wheels from scratch; it stands on the shoulders of giants:

• Core runtime borrowed from Kratos (Bilibili's open-source framework)
• Service discovery/governance via Polaris (Tencent's cloud-native service mesh) and Nacos (open-sourced by Alibaba)
• Distributed transactions via Seata, DTM, and others
• On top of that, I added a plugin management system + event bus + control plane of my own, enabling true hot-plugging.

Versus Kratos: Kratos is the "foundation framework"; Lynx is "Kratos hardened for production + a plugin system + zero configuration." If you like Kratos but find its configuration tedious and its plugins inconsistent, this framework was born from exactly that pain.

Scaffolding​

Zero config + a CLI scaffold—one command generates a complete project:

# Install the lynx CLI
go install github.com/go-lynx/lynx/cmd/lynx@latest

# Generate a complete microservice project skeleton
lynx new my-service

The Plugin Ecosystem So Far​

• Existing plugins: gRPC, HTTP, Redis, PostgreSQL, Redis distributed lock, Swagger, Tracer, Seata, and more.
• Plugins communicate through a built-in Event Bus, enabling inter-plugin messaging and control.
• Service registration and discovery + health checks + multi-version + load balancing (Polaris)
• Traffic governance: rate limiting, circuit breaking, blue-green/canary, fallback
• Security: mutual TLS, JWT, OAuth2, RBAC/ABAC
• Distributed transactions: fully automatic via Seata
• Observability: Prometheus + OpenTelemetry + Zap (JSON logging)
• Graceful shutdown, retries, dead-letter queues, recovery manager

Config-Driven​

• Every plugin has a fixed confPrefix (e.g. lynx.grpc.service, lynx.grpc.client), with configuration loaded from the Lynx Runtime via rt.GetConfig().Value(confPrefix).Scan(...).
• Supports hot updates via Configure(c): the server side guards config pointer replacement with confMu, ensuring concurrency safety and allowing partial capabilities to be updated without a restart.

Example lynx project configuration (YAML)​

lynx:
  polaris: {namespace: default}   # Service registration and discovery
  http: {addr: ":8080"}           # HTTP listen address
  grpc: {addr: ":9090"}           # gRPC listen address
  # Ready-made templates exist for metrics/tracing/logging/tls/rate_limit too

The overall feel is similar to Spring Boot: configure only the plugins you use, and every plugin works out of the box.

Plugin Architecture​

• Unified plugin interface: plugins implement Lynx's plugins.Plugin interface, and embedding BasePlugin gives them common capabilities—ID, name, version, config prefix, weight, and so on.
• Factory registration: client plugins register in init() via factory.GlobalTypedFactory().RegisterPlugin(clientPluginName, "lynx.grpc.client", ...), and the framework loads them by config prefix and manages their lifecycle.
• Lifecycle: a standard three-phase model—InitializeResources (read and validate config), StartupTasks (start services/connections), CleanupTasks (graceful shutdown, resource release).
• Dependency injection: server-side plugins receive the application name, Logger, certificates, control plane, etc. via SetDependencies, avoiding global singletons and making testing and multi-instance setups easier.

Security Module Design​

To secure communication between internal microservices, Lynx has built-in automatic TLS certificate management and rotation. Certificates update while the service is running with no restart required—newly established connections automatically pick up the latest certificate, delivering true zero-downtime updates. The design centers on a unified CertificateProvider for certificate management: internal gRPC servers fetch the current certificate dynamically during the TLS handshake, while clients obtain the Root CA through the same mechanism with support for credential refresh, so certificate updates take effect automatically. The mechanism can also hook into file watchers, cloud certificate services, or key management systems for automated certificate rotation and enterprise-grade secure communication—improving security while substantially cutting operational overhead.

Wrap-Up​

The core idea of go-lynx is not to rebuild the wheel but to fill in the layer that production environments need on top of Kratos: a unified plugin system, a config-driven out-of-the-box experience, and hot-plugging powered by the event bus and control plane. Service governance, distributed transactions, observability, and automatic TLS rotation are all delivered as plugins—use what you need, configure only that. If you're on Kratos and have been burned by inconsistent configuration and plugins, take a look at go-lynx and see whether this design solves your pain points.

The reason for writing this is simple: after using chat-style AI long enough, you realize its ceiling isn't "how well it answers" but "what happens after the answer." It suggests the code, lists the commands—but actually running, verifying, and fixing things is still on you. Agent frameworks exist to close that last mile: turning "saying" into "doing." OpenClaw is one I've actually run for a stretch recently, so I'm writing down my observations and issues—partly for my own reference, partly for anyone evaluating options.

In short, you can think of OpenClaw as:

An AI engineering assistant that can write code, run commands, analyze projects, and iterate on tasks automatically.

Unlike plain ChatGPT or Claude, OpenClaw is designed to:

• Give AI the ability to execute tasks
• Break down complex goals
• Automatically call tools and run commands
• Keep iterating until the task is done

That makes it something closer to an AI software engineer.

OpenClaw — Personal AI Assistant

OpenClaw's Core Capabilities​

OpenClaw's capabilities boil down to three things: task planning + autonomous execution + tool calling.

These map directly onto the classic agent loop: the model produces a plan from the goal (planning), acts on the real environment through tools (execution), then feeds the results—command output, error messages, test results—back to itself as observations (feedback) and adjusts its next move accordingly. The loop repeats until the model decides the task is complete. Once you understand this plan–execute–observe loop, OpenClaw's behavior becomes fairly predictable.

1) Task Planning​

When you give OpenClaw a goal, for example:

"Add a Redis cache to this project"

OpenClaw automatically breaks the task down, for example:

1. Analyze the project structure
2. Locate the database access code
3. Design a caching strategy
4. Modify the code
5. Add tests
6. Run verification

The process is essentially an AI generating a development plan on its own.

Worth emphasizing: the plan isn't generated once and then executed blindly to the end. After each step, the model re-examines the remaining steps—say it analyzes the project structure and discovers it uses MyBatis rather than JPA, then the subsequent changes adapt accordingly. The plan is alive, and that is the essential difference between an agent and "asking the model to output a script in one shot."

2) Autonomous Task Execution​

OpenClaw doesn't just give advice—it can:

• Modify code
• Create files
• Run shell commands
• Install dependencies
• Execute scripts

For example:

# Clone the target project locally
git clone project
# Install project dependencies
npm install
# Run the tests to verify the change didn't break anything
npm run test

The AI executes these operations directly.

The key to execution is result feedback: each command's stdout, stderr, and exit code flow into the next round of context. If tests fail, the model sees exactly which case broke and what the error was, then goes and fixes it itself—that is what "iterating until the task is done" looks like in practice.

3) Tool Calling​

OpenClaw can invoke different capabilities through tools, such as:

• Filesystem
• Shell
• Git
• HTTP APIs
• Build tools
• Test frameworks

A tool is essentially a described interface exposed to the model; the model decides on its own which one to call and with what arguments, based on the task. The mechanism is extensible: wrap an internal system in an interface and register it, and the agent can operate it. The capability boundary isn't set by the framework itself but by how many tools you're willing to plug in—of course, the more you plug in, the more careful you need to be about access control.

OpenClaw's Weaknesses​

As capable as OpenClaw is, it still has some obvious problems.

1) Heavily Dependent on the Model​

OpenClaw's capability is highly dependent on the underlying model.

If the model isn't strong enough:

• Task planning degrades
• Code quality drops
• Execution goes wrong more often

So in many cases:

Agent capability ≈ model capability

With a weak model, OpenClaw becomes pretty dumb too.

This problem gets amplified in agent scenarios: in a normal conversation, if the model gets one answer wrong, a human glances at it and corrects it; in a multi-step execution loop, a mistake in the first planning step is inherited by every subsequent step—errors compound. Prompt engineering at the framework level can absorb some of this, but it can't paper over the model's own reasoning gaps.

2) Token Cost​

Agent systems typically require:

• Multiple rounds of reasoning
• Many model calls
• Persistent context

Token consumption is very high—every call carries a large amount of context. Tuning only trims part of the overhead, and swapping in a cheaper model brings back the problem above: OpenClaw becomes completely unreliable and uncontrollable.

The reason isn't hard to see: every round of calls has to carry the task goal, action history, file contents, and command output, so context grows roughly linearly with the number of iterations, and each round is a full model call. A multi-step task can easily cost tens of times more than a single conversation. Cost and capability form an unavoidable tension here—save money with a weaker model, and the first problem comes knocking immediately.

Pitfalls and Caveats​

Given the two weaknesses above, a few practical suggestions:

• Don't run it directly in production. The agent really does execute commands—deleted files and changed configs are deleted and changed for real. Run it in a container or an isolated working directory with minimal permissions.
• Keep tasks small. The bigger the goal, the more likely planning drifts off course, and the faster the context balloons. "Add a cache" is far more reliable than "refactor the entire data layer."
• Watch the execution. Given the opacity problem mentioned earlier, without a companion monitoring platform you should at least keep complete operation logs, so you can trace afterward what it actually touched.
• Cap the cost. Token consumption across multiple iterations can spiral easily, especially when the model keeps retrying the same mistake. Set a round limit or a budget cap so you don't wake up to an exploded bill.

Overall​

OpenClaw is an AI agent framework with a lot of potential.

Strengths:

• Automatic task planning
• Autonomous execution
• An extensible agent architecture

Weaknesses:

• Heavily model-dependent
• High token cost (model calls are very frequent; capability on complex tasks depends entirely on the model)
• Opaque execution (a serious issue—the execution process is completely opaque and needs a control platform for monitoring)

For now it's a better fit for:

• AI-driven development automation
• DevOps automation
• AI agent experiments

But combined with:

• A strong model
• A visualization system
• An agent management platform

it could eventually become:

Core infrastructure for AI-automated teams.

Wrapping Up​

The value of an agent framework is connecting model output to the real environment in a closed execution loop, and OpenClaw covers that path completely: planning, execution, and tool calling are all there. But its ceiling is locked by model capability, and its floor is dragged down by cost and opacity. My current usage pattern: small tasks, isolated environments, eyes on the logs. Once the monitoring and management ecosystem around it matures, I'll consider putting it into more important workflows.

Background​

Early ops was extremely primitive. When servers arrived, you racked them yourself, installed the OS, configured the network, and edited config files. When a machine had a problem, you SSH'd in to troubleshoot, often relying on experience and memory. Ops back then was, in essence, "humans watching machines."

But as server counts grew, this approach quickly broke down. There were two main problems: repetitive labor and configuration chaos.

Repetitive labor is easy to understand—deploying the same service to dozens of machines meant manually running a pile of commands every time; at scale, ops people spent every day doing these repetitive chores. In the SRE world this kind of work has a dedicated name: toil (work that is automatable, repetitive, and low in long-term value).

The other problem is configuration drift. You tweak a parameter on one machine today; six months later nobody remembers; configurations across machines gradually diverge, and the system becomes harder and harder to maintain.

To solve these problems, ops moved toward automation. First came shell scripts of all kinds, which later evolved into configuration management tools (Puppet, Chef, Ansible). Then a more important idea arrived: Infrastructure as Code (IaC).

Simply put, you write server configuration as code and manage it in Git. Every change is recorded, can be rolled back, and can be replayed. Ops is no longer "go change things on the machine" but "change the code."

Virtualization and Cloud Computing​

Then came a truly pivotal shift: virtualization and cloud computing.

Buying servers used to be a heavyweight affair—procurement, racking, cabling. With virtualization, one physical server could run many virtual machines; servers stopped being hardware and became a "resource." In the cloud era, you don't even need to buy servers—a few clicks in a console provisions the resources. The ops role began shifting from "the person who manages machines" to "the person who schedules resources."

Containerization​

What truly changed how software is delivered was Docker and Kubernetes.

Docker solved an old problem: software behaving differently across environments. It runs on the dev machine, fails on the test machine, and production is different again. Docker packages the application together with its runtime environment into an image—like a boxed lunch you can take anywhere and it just works.

But Docker alone isn't enough. Once you have dozens, hundreds, or thousands of containers, you need a system to manage them. Enter Kubernetes.

The core idea of Kubernetes is actually simple: you just tell the system "here is what I want the system to look like," and the system keeps pulling reality back toward that desired state.

For example, you declare "I want 3 instances of this service." If one dies, Kubernetes automatically starts another; if a machine fails, containers are rescheduled onto other machines. This mechanism is called the control loop, and it's the key to how Kubernetes automates the management of large clusters.

So from a broader vantage point, ops has gone through three big leaps:

• Configuration management / IaC: making server configuration versionable
• Virtualization / cloud computing: turning servers into resources
• Docker + Kubernetes: letting entire clusters run themselves

After Kubernetes, Where Next​

Kubernetes is not the endpoint—it just solved the "cluster management" problem reasonably well. After Kubernetes, ops development is concentrated in three directions: more automation, more observability, less server-related mental overhead.

1) GitOps​

The GitOps idea is simple: Git is the single source of truth for the system.

All configuration lives in Git, and the system automatically reconciles the live environment with Git's state. To change a config, you no longer touch a server—you open a PR. Once the code merges, the system deploys automatically.

The biggest wins of this model: changes are auditable, rollbacks are easy, and the process is clearer.

2) Observability and Service Governance​

Once a system becomes microservices, the real complexity usually isn't any single service but the call relationships between services. A request may pass through a dozen services, and if any link slows down, the whole system suffers.

Hence technologies like Service Mesh, Prometheus, and OpenTelemetry. Their core goal: make the system's runtime state more transparent so problems are easier to find.

3) Serverless and Edge Computing​

The serverless idea: developers just write code, and the platform manages the servers. A function, for instance, runs only when a request arrives and consumes nothing otherwise. For many scenarios, this model can dramatically reduce operational cost.

At the same time, some computation is moving toward the "edge"—retail store devices, factory equipment, in-vehicle systems. To fit these scenarios, lightweight Kubernetes distributions (like K3s) and cloud-edge coordination technologies have emerged.

How AI Is Changing Ops​

In recent years, AI has started making its way into operations.

Many traditional monitoring systems are based on fixed thresholds—alert when CPU exceeds 80%, say. But in complex systems, many anomalies are hard to describe with thresholds. So more and more platforms use machine learning for anomaly detection.

For example:

• AWS DevOps Guru analyzes metric data, automatically surfaces anomalies, and offers recommendations
• Azure's smart detection automatically identifies performance degradation or rising error rates
• Elastic's anomaly detection first learns the system's normal behavior, then flags abnormal patterns

There are also research efforts and products that apply deep learning to log analysis, finding anomalous patterns in log sequences. The goal of all this is simple: cut alert noise so ops can find the real problem faster.

Where the Ops Profession Goes From Here​

Seeing all this automation and AI, many people worry that ops will disappear. The reality is closer to: the way ops works is changing.

In the past, an ops engineer's value often came down to "can you handle problems by hand"—logging into machines, changing configs, restarting services, troubleshooting incidents. But the capabilities that will matter more going forward are:

• Turning experience into automation
• Turning processes into platforms
• Understanding system problems through business metrics
• Designing observability and reliability systems

The way I see it, the ops engineer of the future is not the person who fixes machines, but the person who builds systems that fix themselves, observe themselves, and carry out construction automatically from human-language instructions.

Wrapping Up​

Across thirty-plus years, the evolution of ops has one central thread: freeing humans from repetitive labor. Configuration management and IaC made changes versionable; virtualization and cloud computing turned servers into resources; Docker and Kubernetes let clusters run themselves. Looking ahead, GitOps, observability, and serverless are the high-certainty directions, while AI is gradually taking over anomaly detection and incident localization. Ops won't disappear, but the value of "handling problems by hand" will keep declining—and the people who can design and build automated systems will keep becoming more valuable.

The reason this topic deserves its own post is that "Redis is single-threaded" comes up constantly in interviews and everyday discussion, yet most people's understanding stays at the literal level: either they think the entire Redis process has only one thread, or they think command execution went multi-threaded after Redis 6. Both are wrong. And this model directly shapes how we use Redis—which commands you shouldn't run casually, why big keys are dangerous, why pipelines work—all of it traces back to the threading model.

This post covers Redis's single-threaded design, the event-driven model, IO multiplexing, and the multi-threading improvements from Redis 6 onward.

Why Redis Chose a Single Thread​

In early versions (before Redis 6), the core execution model was single-threaded command processing.

That means:

• All client requests
• All command execution
• All data reads and writes

are handled by one main thread.

This design yields a crucially important property: only one command executes at any given moment. Individual Redis commands are therefore naturally atomic—no locks needed, and no possibility of two commands interleaving on the same key. Commands like INCR and SETNX can be used directly as concurrency primitives; that's a byproduct of this model.

But note:

Redis's single-threadedness refers only to command execution—it doesn't mean the whole Redis process has a single thread. For example:

• RDB persistence
• AOF rewrite
• Asynchronous deletion
• BIO threads

are all background threads.

Take asynchronous deletion: in the main thread, the UNLINK command does just one thing—remove the key from the dictionary. The actual memory reclamation is handed to a background thread to do at its own pace. Even deleting a very large object won't stall the main thread. RDB persistence goes further: it forks a child process to write the snapshot, using the OS's copy-on-write to avoid blocking the main thread.

The core Redis thread is responsible only for:

• Network IO
• Command parsing
• Command execution
• Returning results

Why a Single Thread Is Still Fast​

A common puzzle: how can a single thread sustain hundreds of thousands of QPS?

Redis's performance comes from a few main sources.

In-memory database: all Redis data lives in memory, avoiding disk IO. Memory and disk speeds aren't even in the same order of magnitude. For an ordinary GET, the actual execution is just one hash table lookup—microseconds or even nanoseconds. The bottleneck simply isn't CPU computation.

No thread-switching overhead: single-threaded execution means no locks, no contention, no complex concurrency control. In a multi-threaded model, context switches, lock contention, and cache invalidation aren't cheap; when each request needs only a tiny amount of processing, the cost of that concurrency machinery can exceed the benefit. Choosing a single thread is essentially Redis making a judgment call: command execution isn't the bottleneck, network IO is—so solve concurrency at the IO layer and keep the execution layer as simple as possible.

On top of that, Redis's internal data structures serve this model: SDS strings, skip lists, ziplists, and so on are all optimized for memory access, keeping individual operation paths very short.

IO Multiplexing​

Redis doesn't handle one connection at a time—it uses an IO multiplexing model.

IO multiplexing means one thread monitors many sockets at once, with the operating system telling you "these connections have data ready to read/write," so the thread handles only the ready ones. Compared with the traditional one-thread-per-connection model, it avoids paying each connection's thread memory and scheduling cost, letting a single thread support a massive number of connections.

The IO models Redis uses include:

• epoll (Linux)
• kqueue (macOS / BSD)
• select
• evport

At compile time Redis automatically picks the best implementation for the platform—on Linux that's epoll. In the source this abstraction layer is called ae (A simple Event driven programming library), which exposes a unified event interface to the layers above.

Redis listens on all client connections through its event loop:

             ┌───────────────┐
Client 1 ───▶│               │
Client 2 ───▶│               │
Client 3 ───▶│  IO Multiplex │
Client N ───▶│               │
             └───────┬───────┘
                     │
                     ▼
               Redis EventLoop
                     │
                     ▼
               Command Execute

The processing flow:

1. Monitor many client sockets
2. Detect which socket has data ready
3. Read the request
4. Execute the command
5. Return the result

Each turn of the event loop pulls the ready events from the multiplexing interface and processes them one by one: for read events, parse and execute the command; for write events, send results back to the client. Because each event is handled quickly, the loop spins fast enough that clients never notice they're "waiting in line."

That's how Redis can handle a huge number of connections on a single thread.

Redis 6's Multi-Threading Improvements​

As hardware advanced and CPU core counts grew, the single-threaded model hit a ceiling in certain scenarios—and that ceiling is mainly network IO.

For example:

• Very large numbers of client connections
• Massive volumes of network traffic

Command execution itself is pure in-memory work and blindingly fast; but reading requests out of kernel buffers and writing responses back—those read/write system calls have real cost. Past a certain traffic level, the main thread spends most of its time moving data, and actual command execution becomes a small fraction—that is the ceiling of the single-threaded model.

So Redis 6 introduced IO threads.

The execution flow becomes:

Client Request
       │
       ▼
IO Threads (read)
       │
       ▼
Main Thread Execute Command
       │
       ▼
IO Threads (write)

The idea is clean: hand the "porter" work—reading/parsing requests and writing back responses—to multiple IO threads in parallel, while command execution still converges on the main thread and runs serially. This exploits multiple cores to boost network throughput while fully preserving the atomicity semantics of single-threaded execution—application code doesn't need to change at all.

Therefore:

• Command execution remains single-threaded
• IO operations can run in parallel

Note that IO threads are disabled by default and enabled via configuration:

# redis.conf
# Number of IO threads; keep it below the CPU core count
io-threads 4
# By default only writes use IO threads; enable this to parallelize reads/parsing too
io-threads-do-reads yes

It only makes sense to enable this on instances where network traffic is genuinely the bottleneck; in low-traffic scenarios it just adds a layer of thread coordination overhead.

Practical Advice for the Single-Threaded Model​

Because command execution is single-threaded, keep the following in mind. One slow command blocks every client's requests—this is the part of the single-threaded model that deserves the most respect.

Avoid Slow Commands​

For example:

# Scans the entire keyspace, O(N), blocking all requests while it runs
KEYS *

Commands like this block Redis.

Instead, use:

# Cursor-based batched iteration—processes a small chunk each time, no long blocking
SCAN

Similar offenders include full-read commands on large collections such as SMEMBERS, HGETALL, and LRANGE 0 -1—all of them essentially stuff O(N) work into the single thread.

Keep Big Keys Under Control​

Big keys cause:

• Network congestion
• Longer CPU execution time

Moreover, deleting and expiring big keys also happens on the main thread (unless you use UNLINK or enable lazy freeing)—when a key of several hundred MB expires, it can produce a visible latency spike.

Recommendations:

• Split the data up
• Use hashes / sets

Use Pipelines​

Pipelines reduce network round trips:

client -> redis -> client

Without a pipeline, each command waits for the previous response before the next one is sent, so throughput is capped by network RTT. A pipeline batches a group of commands, sends them in one go, and collects all responses at once, compressing N round trips into one.

This improves overall throughput.

Pitfalls and Caveats​

A few points that are easy to overlook in practice, given the threading model:

1. Lua scripts also block the main thread. While a script runs, Redis processes no other commands; a script looping over a large amount of data is effectively one giant slow command.

2. Don't interpret Redis 6's IO threads as "concurrent command execution." The atomicity semantics of transactions, Lua, and INCR are completely unchanged—no extra locking needed. Likewise, don't expect IO threads to fix slow commands: if the slowness is in the execution layer, multi-threaded IO can't help.

3. When investigating stalls, check the slow log first. SLOWLOG GET records command execution time (excluding network) and is the first stop for diagnosing "Redis occasionally hiccups"; combined with the latency family of commands, you can further distinguish slow commands from slow forks or slow disks.

4. A single instance not saturating multiple cores is normal. Command execution uses only one core. To utilize a whole multi-core machine, the usual approach is multiple instances per host or Redis Cluster—not waiting for Redis to go multi-threaded on execution.

Summary​

Through in-memory operations, IO multiplexing, an event-driven architecture (event loop), and single-threaded command execution, Redis achieves extremely high concurrency while avoiding lock contention and thread-switching overhead. Redis 6's IO threads only parallelize network send/receive—command execution remains serial. That core hasn't changed, so the usage principles built around it (avoid slow commands, control big keys, use pipelines wisely) remain valid. Once you understand this, many Redis best practices stop being memorized checklist items and become inevitable conclusions derived from the model.

A binary tree is a common data structure made up of nodes, where each node has at most two children:

• Left child
• Right child

Hence the name binary tree.

The "at most two" constraint looks simple, but it brings an important property: the tree's shape can be defined recursively—every binary tree consists of a root node, a left subtree, and a right subtree, and the left and right subtrees are themselves binary trees. Nearly every binary tree algorithm that follows (traversal, search, insertion) is built around this recursive structure.

Each node typically has three parts:

Node {
    value   // the data stored in the node
    left    // reference to the left child, or null if none
    right   // reference to the right child, or null if none
}

In other words, a binary tree doesn't require contiguous storage in memory; nodes are connected by references (pointers), much like a linked list—except each node goes from "one successor" to "at most two children."

A simple example:

        A
       / \
      B   C
     / \   \
    D   E   F

Here:

• A is the root
• B and C are A's children
• D and E are B's children

Nodes with no children (like D, E, F) are called leaf nodes. The number of levels from the root down to a leaf is the tree's height, which directly determines the cost of most operations on the tree.

Based on shape, binary trees come in a few common special forms—let's go through them one by one.

Full Binary Tree​

If every node in the tree has either two children or no children, it's a full binary tree.

        A
       / \
      B   C
     / \ / \
    D  E F  G

Characteristics:

• Every node has either 0 children
• or 2 children

In other words, a full binary tree has no node with "only one child." This is the most densely packed shape—for a given number of levels, it holds the most nodes.

Complete Binary Tree​

A complete binary tree requires:

• Every level except the last is completely filled
• Nodes in the last level are packed contiguously from left to right

Example:

        1
       / \
      2   3
     / \  /
    4  5 6

This structure is perfect for array storage, which is why the heap is a complete binary tree.

Why is a complete binary tree so well suited to arrays? Because numbering nodes "top to bottom, left to right" leaves no gaps in the middle: put the root at index 0, and the node at index i has its left child at 2i + 1, right child at 2i + 2, and parent at (i - 1) / 2 (rounded down). No pointers need to be stored—index arithmetic alone lets you jump between parent and child, which saves memory and is cache-friendly. Heapsort and priority queues exploit exactly this.

Binary Search Tree (BST)​

The two categories above are about "shape"; the binary search tree is about "how node values are arranged." A binary search tree satisfies:

left subtree < root < right subtree

Note that this constraint applies recursively: it's not just the immediate children—every value in the entire left subtree is less than the root, and every value in the entire right subtree is greater.

Example:

        8
       / \
      3   10
     / \    \
    1   6    14

Characteristics:

• Fast lookups
• Average time complexity: O(log n)

Searching works just like binary search: start at the root, go left if the target is smaller than the current node, go right if it's larger, and each level down eliminates roughly half the candidates. Insertion works the same way—follow the search path to an empty slot and hang the new node there.

But if the tree degenerates into a linked list:

1
 \
  2
   \
    3

the time complexity degrades to:

O(n)

The classic trigger for degeneration is inserting values in sorted order: each new node is larger than all previous ones, so it can only keep hanging off the right side. The tree "grows crooked" into a chain, each level of a search eliminates only one node, and the BST's advantage vanishes entirely. This is exactly why self-balancing binary search trees like AVL trees and red-black trees exist—they restructure via rotations on insert and delete, keeping the tree height at O(log n).

Traversals​

Visiting all the nodes of a binary tree is called traversal. There are four common kinds:

1. Preorder: root → left subtree → right subtree; often used for copying or serializing a tree
2. Inorder: left subtree → root → right subtree; for a BST, an inorder traversal yields the values in ascending order
3. Postorder: left subtree → right subtree → root; suited to "handle the children before yourself" scenarios, like freeing an entire tree
4. Level-order: visit level by level, left to right, implemented with a queue—breadth-first search on a tree

The first three are very natural to write recursively, which confirms the point from earlier: a binary tree is itself a recursively defined structure.

Pitfalls and Caveats​

A few things that trip people up in practice and in coding interviews:

• Don't assume a BST is balanced. When analyzing complexity, distinguish "average O(log n)" from "worst-case O(n)"—the follow-up question in interviews is usually about the degenerate case.
• Validate a BST with an inorder traversal. To check whether a tree is a valid BST, comparing each node with just its immediate children isn't enough—the entire subtree must satisfy the ordering. Checking that the inorder sequence is strictly increasing is the least error-prone approach.
• Recursion depth. When the tree degenerates into a chain, the call stack of a recursive traversal is also O(n), which can overflow the stack on large inputs; switch to an iterative version with an explicit stack when necessary.
• Distinguish "full" from "complete". The two definitions are easy to mix up: a full binary tree forbids single-child nodes, while a complete binary tree requires the last level to be left-aligned. True/false questions love pitting these two concepts against each other.

Summary​

The binary tree is one of the most fundamental and important data structures. Its core traits:

• Each node has at most two children
• Supports efficient search
• Extends into many advanced data structures

Common variants include:

• Binary search tree (BST)
• AVL tree
• Red-black tree
• Heap

Understanding binary trees is an essential foundation for learning algorithms and data structures. Once you've internalized the recursive definition, the differences between the common shapes, and the traversal patterns, you'll find that balanced trees and heaps—those "advanced" structures—are really just binary trees with extra constraints layered on.

Background and Problem​

In a microservice architecture, the system typically exposes a unified entry point through the Istio IngressGateway. When traffic spikes (a promotion, crawlers, endpoint abuse) and there's no rate limiting in place, the backend can run into problems like:

• Endpoints hit at high frequency, overloading service CPU or the database
• Traffic bursts causing system meltdown or cascading failures
• Core APIs abused by malicious callers, crowding out legitimate users

Implementing rate limiting inside each service works, but the rules end up scattered and inconsistent—and by the time a request is rejected, it has already entered the business process and consumed resources. The better approach is unified rate limiting at the gateway layer, so traffic is governed before it ever reaches the backend.

Istio's data plane is Envoy Proxy, which has this capability built in: with the Envoy RateLimit filter + a standalone ratelimit service + Redis, you get global rate limiting shared across gateway replicas, with per-path access frequency control.

How It Works​

There are three actors on the path. Once you understand how they relate, the YAML below is easy to read:

1. The Envoy RateLimit filter. It sits on the IngressGateway's HTTP filter chain. For every request passing through, it builds a set of descriptors from the actions configured on the route, then asks the ratelimit service over gRPC—along with the domain—whether the request should be allowed.

2. The ratelimit service (envoyproxy/ratelimit). A standalone gRPC service that loads quota rules from a ConfigMap. When it receives descriptors, it looks up the rules, decrements the quota, and returns OK or OVER_LIMIT.

3. Redis. The storage for quota counters. Because the counters live in Redis, all gateway replicas share the same counts—that's what makes this "global" rate limiting, as opposed to local rate limiting, where each Envoy instance counts on its own.

The key to matching: the descriptor combination generated by the route's actions must correspond exactly to the key/value hierarchy of descriptors in the ConfigMap, and the domain on both sides must match. If any link is off, rate limiting fails silently and every request passes through.

When You'd Use This​

The typical candidates are low-traffic but sensitive endpoints where abuse is expensive:

/api/login
/api/send-code
/api/payment

A hammered login endpoint triggers a flood of password checks and risk-control computation; an abused verification-code endpoint burns SMS fees directly; the payment endpoint touches downstream resources. These endpoints have low QPS by nature, so a very small quota blocks the vast majority of malicious traffic with minimal collateral damage.

Full Configuration​

Here's a configuration you can apply directly, in four parts: a Secret for the Redis password, a ConfigMap defining quotas, the ratelimit service itself, and two EnvoyFilters—one to mount the filter, one to attach descriptors.

# --------------------------------------------
# 0) Redis password in a Secret (more secure)
#    Use a Secret for sensitive values (like the Redis password) so they never
#    appear in plaintext in images or Pod env var history
# --------------------------------------------
apiVersion: v1
kind: Secret
metadata:
  name: ratelimit-redis-secret          # Secret name, referenced by the Deployment via secretKeyRef
  namespace: istio-system               # Same namespace as the ratelimit service, for easy reference and management
type: Opaque
stringData:
  REDIS_AUTH: "password"       # K8s handles base64 encoding of the plaintext

---
# --------------------------------------------
# 1) Ratelimit quota config (runtime config for the Envoy Ratelimit Server)
#    - This ConfigMap is mounted read-only into the container at /data/ratelimit/config
#    - domain must exactly match the domain in the Envoy HTTP filter, or nothing matches
#    - descriptors define the "dimension combinations" for limiting (formed by combined actions)
# --------------------------------------------
apiVersion: v1
kind: ConfigMap
metadata:
  name: ratelimit-config
  namespace: istio-system
data:
  config.yaml: |
    domain: ingress-ratelimit          # Must match the domain in the RateLimit filter
    descriptors:
      - key: header_match              # Top-level key: matches header_value_match in HTTP_ROUTE (action 1/2)
        value: path-api         # Top-level value: from descriptor_value in the route's actions
        descriptors:                   # Second-level descriptors: further refinement (combined into a unique limit key)
          - key: generic_key           # Matches generic_key in HTTP_ROUTE (action 2/2)
            value: istio-limit-v1      # Matches the descriptor_value set on the route
            rate_limit:
              unit: second             # Time unit: second / minute / hour / day
              requests_per_unit: 5     # Quota: requests allowed per unit of time
---
# --------------------------------------------
# 2) Ratelimit Service (connects to your existing standalone Redis)
#    - Quota service implementing global sliding-window/token-bucket limits via envoyproxy/ratelimit
#    - replicas=1: single replica for testing; in production run at least 2-3 replicas with HA Redis
#    - readiness/liveness: health checks on port 6070
#    - Key env vars: REDIS_URL / REDIS_AUTH / RUNTIME_ROOT / RUNTIME_SUBDIRECTORY
# --------------------------------------------
apiVersion: v1
kind: Service
metadata:
  name: ratelimit                         # Resolved by Envoy via cluster name (outbound|8081||ratelimit.istio-system.svc.cluster.local)
  namespace: istio-system
spec:
  selector: { app: ratelimit }            # Matches the Deployment's labels
  ports:
    - name: grpc
      port: 8081                          # gRPC service port (the Envoy RateLimit filter connects here)
      targetPort: 8081
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ratelimit
  namespace: istio-system
spec:
  replicas: 1                              # 1 for testing; in production use >=3 plus HPA/PodDisruptionBudget
  selector:
    matchLabels: { app: ratelimit }
  template:
    metadata:
      labels: { app: ratelimit }
    spec:
      containers:
        - name: ratelimit
          image: envoyproxy/ratelimit:875d418c   # Pin a commit/tag for reproducibility; consider upgrading to a newer official tag
          imagePullPolicy: IfNotPresent
          command: ["/bin/ratelimit"]            # Entry point
          ports:
            - containerPort: 8080                # HTTP Admin (optional: metrics/debugging)
            - containerPort: 8081                # gRPC main service port
            - containerPort: 6070                # Health check port (/healthcheck)
          env:
            - name: REDIS_SOCKET_TYPE
              value: tcp                         # Connect to Redis over TCP
            - name: REDIS_URL
              value: "redis://1.1.1.1:6379"# Your Redis address (in production, use an internal DNS name + sentinel or cluster)
            - name: REDIS_POOL_SIZE
              value: "20"                        # Connection pool size; tune for concurrency/replica count
            - name: REDIS_AUTH
              valueFrom:
                secretKeyRef:
                  name: ratelimit-redis-secret   # Inject the password from the Secret, no plaintext
                  key: REDIS_AUTH
            - name: USE_STATSD
              value: "false"                     # Enable if you have a statsd/Prometheus sidecar
            - name: RUNTIME_ROOT
              value: /data                       # Root of the mount point
            - name: RUNTIME_SUBDIRECTORY
              value: ratelimit                   # Subdirectory; final config path is /data/ratelimit/config
            - name: RUNTIME_IGNOREDOTFILES
              value: "true"                      # Ignore dotfiles
          volumeMounts:
            - name: config
              mountPath: /data/ratelimit/config  # Must match the runtime path above so config lands in the config subdirectory
          readinessProbe:
            httpGet:
              path: /healthcheck
              port: 6070
            initialDelaySeconds: 2               # Initial delay to avoid false negatives during cold start
            periodSeconds: 5                     # Probe interval
          livenessProbe:
            httpGet:
              path: /healthcheck
              port: 6070
            initialDelaySeconds: 10
            periodSeconds: 10
      volumes:
        - name: config
          configMap:
            name: ratelimit-config               # Mounts the ConfigMap above

---
# --------------------------------------------
# 3) Inject the global RateLimit filter into the IngressGateway + point it at the ratelimit CLUSTER
#    - Uses an EnvoyFilter to insert the global ratelimit HTTP filter before the HTTP Router
#    - workloadSelector: limits scope to the gateway workload (label: istio=ingressgateway)
#    - rate_limit_service: connects to the ratelimit gRPC service defined above (via cluster name/authority)
#    - Also appends a Lua response filter: rewrites backend 429s into a unified JSON body
# --------------------------------------------
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: gateway-global-limit-filter
  namespace: istio-system
spec:
  workloadSelector:
    labels:
      istio: ingressgateway                 # Only applies to IngressGateway instances
  configPatches:
    # 3.1 Insert the ratelimit HTTP filter before the router (applies globally)
    - applyTo: HTTP_FILTER
      match:
        context: GATEWAY
        listener:
          filterChain:
            filter:
              name: envoy.filters.network.http_connection_manager    # HTTP connection manager
              subFilter:
                name: envoy.filters.http.router                      # Insert before the Router
      patch:
        operation: INSERT_BEFORE
        value:
          name: envoy.filters.http.ratelimit
          typed_config:
            "@type": type.googleapis.com/envoy.extensions.filters.http.ratelimit.v3.RateLimit
            domain: ingress-ratelimit          # Must match the domain in the ConfigMap
            failure_mode_deny: false           # Whether to reject requests when the ratelimit service is down; false=allow (recommended)
            timeout: 10s                       # Timeout for calls to the ratelimit service
            rate_limit_service:
              grpc_service:
                envoy_grpc:
                  cluster_name: outbound|8081||ratelimit.istio-system.svc.cluster.local  # Cluster name auto-generated by Istio
                  authority: ratelimit.istio-system.svc.cluster.local                     # HTTP/2 Host (SNI), optional
              transport_api_version: V3         # Use the v3 API (recommended)
    # 3.2 Insert the Lua response filter on the port 1035 listener (optional: applies only to that listener)
    - applyTo: HTTP_FILTER
      match:
        context: GATEWAY
        listener:
          portNumber: 1035                      # Only applies to the listener on port 1035 (your gateway exposes this port)
          filterChain:
            filter:
              name: envoy.filters.network.http_connection_manager
              subFilter:
                name: envoy.filters.http.router
      patch:
        operation: INSERT_BEFORE
        value:
          name: envoy.filters.http.lua
          typed_config:
            "@type": type.googleapis.com/envoy.extensions.filters.http.lua.v3.Lua
            inlineCode: |
              function envoy_on_response(handle)
                local s = tonumber(handle:headers():get(":status") or "0")
                if s == 429 then
                  local body = '{"code":429,"message":"Too many requests. Please try again later."}'
                  handle:headers():replace("content-type", "application/json")
                  handle:headers():replace("content-length", tostring(#body))
                  handle:body(true):setBytes(body)
                end
              end
   # Notes:
   # - If your gateway doesn't listen on 1035 (or uses multiple ports), consider widening this filter to all ports (drop the portNumber condition)
   # - Alternatively, local rate limiting can generate the 429 locally on rejection, and this Lua rewrites it uniformly
---
# --------------------------------------------
# 4) Inject per-route rate limit actions into the VirtualHost
#    - Configure actions on a specific Route (or the whole vhost) to generate descriptors that match the ConfigMap
#    - Two actions are used here:
#        a) header_value_match: when :path matches the /api prefix, emit descriptor_value=path-api
#        b) generic_key: additionally emit descriptor_value=istio-limit-v1
#    - Combined => (key=header_match,value=path-api) + (key=generic_key,value=istio-limit-v1)
#      which matches the ConfigMap exactly, activating the quota
# --------------------------------------------
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: gateway-limit-descriptor-svc
  namespace: istio-system
spec:
  workloadSelector:
    labels:
      istio: ingressgateway                       # Scope limited to the gateway
  configPatches:
    - applyTo: HTTP_ROUTE
      match:
        context: GATEWAY
        routeConfiguration:
          vhost:
            name: host:port       # Watch this vhost name: usually "<host>:<port>"
      patch:
        operation: MERGE
        value:
          route:
            rate_limits:
              - actions:
                - header_value_match:
                    headers:
                      - name: ":path"             # Match the :path pseudo-header, prefix-based
                        prefix_match: "/api"
                    expect_match: true            # Only add the descriptor when the match succeeds
                    descriptor_value: "path-api"   # Matches the ConfigMap's top-level key/value
                - generic_key:
                    descriptor_value: "istio-limit-v1"    # Matches the ConfigMap's second-level key/value

A few points worth unpacking:

1. Both EnvoyFilters are required. The first one only mounts the ratelimit filter onto the filter chain and tells it where to find the ratelimit service. What actually decides "which requests carry which descriptors to the limiter" is the second one—without rate_limits.actions on the route, the filter does nothing at all for a request.

2. failure_mode_deny: false. When the ratelimit service or Redis goes down, allow requests through instead of rejecting them. Rate limiting is a protective measure; it shouldn't become a new single point of failure. Unless an endpoint is so sensitive that you'd rather have it unavailable than abused, keep this false.

3. The Lua filter is a UX refinement. When Envoy's rate limit trips, it returns a bare 429 by default, which frontends can't handle uniformly. This Lua snippet rewrites the 429 body into a consistent JSON structure at the response stage—much friendlier to clients.

Pitfalls and Caveats​

• A mismatched domain makes rate limiting fail silently. The domain in the filter and the domain in the ConfigMap must match character for character. When they don't, there's no error—the ratelimit service simply finds no rules and returns OK for everything.
• Descriptor hierarchy must correspond exactly. The two-level structure in the ConfigMap—header_match/path-api nesting generic_key/istio-limit-v1—maps to the order of the two actions on the route. One missing action, a wrong value, or a flipped hierarchy, and nothing matches.
• The vhost name isn't arbitrary. routeConfiguration.vhost.name is usually of the form "<host>:<port>" and must match the vhost name in the route config Envoy actually generates. Confirm it from the gateway's config_dump; if it's wrong, the MERGE simply won't take effect.
• Mind the port your EnvoyFilter matches. The Lua filter above is scoped to portNumber: 1035; if your gateway listens on a different port or multiple ports, adjust or drop that condition, or the 429 rewrite will only apply to part of your traffic.
• Availability of the ratelimit service itself. A single replica is fine for testing; run multiple replicas in production and plan for Redis high availability too. Tune REDIS_POOL_SIZE for your concurrency and replica count.
• EnvoyFilter is a fairly low-level extension mechanism. It patches Envoy config directly and is version-coupled to Envoy's internal APIs. Before a major Istio upgrade, verify in a test cluster that both filters still take effect.

Wrap-Up​

Global rate limiting at the gateway boils down to three things: the actions on the route tag requests with descriptors, the ratelimit service checks quotas against the rules in the ConfigMap, and Redis lets all gateway replicas share the counts. The configuration itself isn't complicated—the difficulty is concentrated in domain and descriptor matching, where mistakes produce no errors, just silent no-ops. Once the matching chain is clear in your head, the rest is just tuning quotas to your business needs.

Pain Points​

1. The underlying database can't handle massive data volumes. As a business grows, 10 TB, 100 TB, or even PB- and EB-scale data no longer fits, and fast query response, data analysis, and data mining become impossible to support.

2. Real-time compute performance falls short. You need a reliable compute engine for millisecond-level real-time processing, with data quality that is dependable, controllable, and observable.

3. Data model changes are too slow. Without the flexibility to restructure data models quickly, it's hard to keep up with reporting demands from the business.

Use Cases​

• Real-time risk control with Flink and a rules engine
• Building a real-time anomaly detection system with Realtime Compute (Flink) and Gaussian models
• Building a simple real-time recommendation system with Realtime Compute (Flink)

The Real-Time Data Warehouse​

Overall data development workflow:

The pipeline boils down to: data ingestion -> data buffering -> real-time compute -> sink to storage.

Component Selection​

Flink​

Alibaba Cloud Realtime Compute for Apache Flink

A fully managed Flink real-time compute engine from Alibaba Cloud, sparing you from building Yarn and Flink nodes yourself and from all the tedious configuration work. Its job is to process business data flowing through the DataHub message bus and route computed results downstream according to each data stream's business scenario. Being part of the Alibaba Cloud ecosystem also makes metadata management much more convenient.

DataHub​

DataHub message bus

DataHub serves as the data buffering layer, providing high-performance storage and streaming transport for real-time data. Intermediate DWD and DWS layer data produced during Flink computation, as well as downstream data, all pass through it for task forwarding and storage—it acts as the message bus. A single topic can handle terabytes of reads and writes per day, with efficient and stable data flow. This component is also fully managed, so you never think about node counts.

Hologres​

Hologres real-time data warehouse: unified interactive analytics and serving for big data

After Flink finishes computing, output for each layer sinks into the Hologres warehouse. Hologres supports row-oriented storage, column-oriented storage, and field-based partitioned storage, solving the massive-data storage problem and delivering sub-second analytics on PB-scale data. Under the hood, Hologres stores data on Pangu, Alibaba's in-house distributed file system (similar to HDFS). Architecturally it follows big data warehouse design principles with full Storage Disaggregation—compute nodes scale out horizontally to increase parallelism and keep queries over massive data fast.

Processing Pipeline​

Warehouse Layers​

In the big data world, ODS, DWD, DWS, and ADS are common data layering concepts used to organize and manage data, each playing a distinct role in how data is processed and consumed.

1. ODS (Operational Data Store): a storage layer for operational business data, typically holding raw data from source systems while preserving its completeness and detail. The ODS layer primarily supports real-time or near-real-time operational needs, such as online transaction processing and real-time queries. Data at this layer usually undergoes little heavy processing or computation—it serves operational workloads directly.
2. DWD (Data Warehouse Detail): the detail layer of the data warehouse, storing data that has been cleansed, integrated, and transformed—typically subject-oriented, queryable data. DWD data has been through ETL (extract, transform, load) and supports analytics, reporting, and decision support. It offers high quality and consistency, usually organized as fact tables and dimension tables.
3. DWS (Data Warehouse Summary): stores aggregated, summarized, and precomputed data, providing higher-level data summaries and better analytical performance. DWS data is typically organized as summary tables or precomputed metrics, supporting complex analytics, data mining, and business intelligence needs.
4. ADS (Application Data Service): an application-facing data storage layer holding data that has been further processed, computed, and optimized. ADS data is usually application-specific views or datasets, adapted and tuned to deliver faster data access and processing.

To sum up: the ODS layer serves operational workloads and preserves raw data intact; the DWD layer provides cleansed, integrated, transformed data ready to query; the DWS layer trades summarization and precomputation for better query performance; and the ADS layer serves specific applications with processed, optimized data.

This layered design brings several benefits:

1. Separation of data processing and management: splitting data into layers also splits the responsibilities. The DWD layer handles extraction, cleansing, and integration of raw data; the DWS layer handles aggregation and summarization; the ADS layer provides application-facing data services. Data processing becomes modular and more flexible.
2. Data quality and consistency: the DWD layer preserves the completeness and detail of raw data, useful for data quality analysis and lineage tracing; the DWS layer offers aggregated data with better query performance; the ADS layer guarantees consistency and accuracy across different application scenarios.
3. Query performance and development efficiency: through summarization and precomputation, the DWS layer speeds up analytics and decision-making; the ADS layer exposes application-oriented data services and interfaces, simplifying application development and boosting productivity.

Wrap-Up​

The core idea of this architecture is to hand cluster operations over to the cloud vendor: DataHub buffers and routes data, Flink handles real-time compute, and Hologres takes care of storage and analytics—the whole pipeline is managed, with no nodes of your own to maintain. Combined with the ODS/DWD/DWS/ADS layered design, each layer has a clear responsibility, and both query performance and reporting productivity are well covered. For teams that don't want to sink headcount into a self-hosted big data cluster, this is a pragmatic path.

Some background first. gRPC defaults to plaintext HTTP/2, which means request bodies between services can be captured and read directly on the internal network. In production, even an internal network cannot be assumed absolutely trustworthy — once a single machine is compromised, sniffing lateral traffic costs the attacker very little. Adding TLS to gRPC is the most direct hardening measure. And for internal services there's no need (and often no way) to request certificates from a public CA; the more common approach is to build your own root certificate and issue a certificate for each service. The whole process can be done with OpenSSL, but there are plenty of details — especially Go's validation rules for certificate fields — worth writing down in full.

Below are the steps to generate self-signed certificates with the OpenSSL command-line tool on Linux or macOS. The process creates a new root certificate (CA), then uses that root certificate to sign a new server certificate.

The certificate hierarchy here has two tiers: the root certificate signs itself and serves as the trust anchor; service certificates are issued by the root. As long as a client trusts the root certificate up front, it can verify any service certificate the root has signed. That's also why only ca.crt needs to be distributed to clients, rather than every individual service certificate.

A quick outline of the overall flow:

Root certificate:

1. Generate the root private key

2. Generate the root certificate from the root private key

Service certificate:

1. Generate the service private key

2. Generate a service CSR from the service private key

3. Sign the service .csr file with the root certificate + root certificate key to produce the service certificate .crt file

Generating the Root Certificate's Private Key​

# Generate a 2048-bit RSA private key to serve as the root CA's key
openssl genrsa -out ca.key 2048

This command generates a new 2048-bit RSA private key, saved to a file named ca.key.

This key is the root of the entire trust chain — whoever holds it can issue certificates trusted by every client. Guard it carefully and never commit it to a code repository.

Generating the Root Certificate​

# Self-sign an X.509 root certificate with the root key, valid for 365 days
openssl req -new -x509 -days 365 -key ca.key -out ca.crt

This command generates a new X.509 root certificate, signed with SHA-256 and valid for 365 days (adjust as you like). The certificate is saved to a file named ca.crt.

The -x509 flag means output a self-signed certificate directly rather than a signing request — this is exactly what "the root certificate signs itself" means.

While running this command, OpenSSL will prompt you for some information to be embedded in the certificate. In most cases you can just press Enter to accept the defaults.

However, the Subject Name or Common Name should be filled in with the name of your microservice.

With the root certificate ready, the next step is generating certificates for individual services. Just repeat the three steps below for each service.

Generating the Server Certificate's Private Key​

# Generate a 2048-bit RSA private key for the service
openssl genrsa -out [service-name].key 2048

Similar to the first command; the generated private key is saved to a file named [service-name].key, where [service-name] is your own service's name.

Generating the Server Certificate Signing Request (CSR)​

# Generate a certificate signing request (CSR) from the service private key
openssl req -new -key [service-name].key -out [service-name].csr

This command generates a new certificate signing request (CSR) containing the server certificate's public key and some additional information, saved to a file named server.csr.

A CSR is not itself a certificate — it's an application form stating "who I am and what my public key is," which only becomes a usable certificate after the CA signs it.

Again, OpenSSL will prompt you for information to be included in the CSR.

Signing the Server Certificate with the Root Certificate​

# Sign the CSR with the root certificate + root key to produce the service certificate
openssl x509 -req -in [service-name].csr -CA ca.crt -CAkey ca.key -CAcreateserial -out [service-name].crt -days 365

This command signs the server certificate with the root certificate; the resulting certificate is saved to a file named server.crt.

The -CAcreateserial flag creates a .srl serial-number file that records the serial numbers of certificates this CA has issued; on subsequent signings it increments automatically with no manual handling needed.

The rootCA.crt, server.crt, and server.key produced by the steps above are now ready for encrypted gRPC communication — configure them on the corresponding server and client sides. The server loads its own certificate and private key; the client loads the root certificate to verify the server's identity.

Custom Fields​

In newer versions of Go, the validation and retrieval of certain certificate fields has changed. (Since Go 1.15, the alt_names field of the certificate is used for service name verification.) So we need to specify some custom fields when generating certificates.

In other words, putting the service name in the Common Name alone is no longer enough: Go's TLS client takes the connection's target name during the handshake and matches it against the certificate's SAN (Subject Alternative Name) field — no match, and the handshake fails outright. The certificate must therefore explicitly carry the SAN extension. Since the interactive command line is awkward for filling in these extension fields, we switch to a configuration file.

Create a san.cnf file with the following contents:

[req]
default_bits = 2048
prompt = no
default_md = sha256
distinguished_name = dn

# Certificate subject info, replacing interactive input
[dn]
C = CN
ST = Beijing
L = Beijing
O = 'lynx'
OU = 'lynx'
emailAddress = 'lynx'
CN = [service-name]

# Extensions used at the CSR stage: declare the SAN
[req_ext]
subjectAltName = @alt_names

# The SAN list — these DNS names are what the Go client validates against
[alt_names]
DNS.1 = [service-name]

# Extensions used at the certificate-issuing stage
[v3_ext]
authorityKeyIdentifier=keyid,issuer:always
basicConstraints=CA:FALSE
keyUsage=keyEncipherment,dataEncipherment,digitalSignature
extendedKeyUsage=serverAuth,clientAuth
subjectAltName=@alt_names

In this config, prompt = no tells OpenSSL to skip the interactive questions and read the [dn] section directly; extendedKeyUsage declares both serverAuth and clientAuth, meaning this certificate can be used on the server side and also as a client certificate in mutual-TLS scenarios.

Then bring the config into the certificate-generation steps above.

When generating the service CSR, add the -extensions req_ext -config san.cnf options:

# Reference the config file when generating the CSR, carrying the SAN declared in the req_ext section
openssl req -new -key [service-name].key -out [service-name].csr -extensions req_ext -config san.cnf

When issuing the service certificate, likewise add the -extensions v3_ext -extfile san.cnf options:

# The v3_ext extensions must also be applied at signing time, or the SAN won't make it into the final certificate
openssl x509 -req -in [service-name].csr -CA ca.crt -CAkey ca.key -CAcreateserial -out [service-name].crt -days 365 -extensions v3_ext -extfile san.cnf

Certificates generated this way contain the fields we specified, along with some certificate description information.

Verifying Certificate Contents​

You can check whether the generated certificate really contains the expected fields with:

# Print the certificate in text form to inspect the SAN and other extensions
openssl x509 -in [service-name].crt -text -noout

In the output, focus on whether the X509v3 Subject Alternative Name section contains the expected DNS names, and whether the validity period and issuer are correct. It's worth making this check a fixed step of the process — far easier than debugging a handshake failure after the service is already live.

Pitfalls and Caveats​

1. The target name the client connects with must match a DNS name in the certificate's SAN. If you connect directly by IP, the SAN needs an IP entry declared as IP.1 = ... — a DNS entry will not match.

2. Once a certificate expires, the gRPC handshake fails outright, and this class of failure tends to surface only on expiry day. Nobody reminds you to renew self-signed internal certificates, so record the expiry dates and rotate ahead of time.

3. ca.key and each service's .key file are sensitive material. Watch the file permissions and never ship them in images or repositories; clients only need ca.crt.

4. After changing san.cnf, both the CSR and the signing step must be redone. Re-signing without regenerating the CSR may carry stale subject information.

Wrapping Up​

The whole process boils down to one chain of trust: the root key produces the root certificate, the service key produces a CSR, and the root certificate signs the CSR into a service certificate. In Go's gRPC scenario, the key points are writing the SAN field into the certificate via san.cnf, explicitly specifying the extensions at both the CSR and signing stages, and finally verifying with openssl x509 -text. Once these steps are captured in a script, issuing a certificate for a new service is a one-minute job.

Anyone doing real-time computing has probably felt this: Flink itself is powerful enough, but the development and operations experience around it is not friendly. Finished SQL has to be packaged and submitted, debugging means digging through logs, multiple clusters are each managed on their own — and once the number of jobs grows, things get messy. While researching tooling for a real-time data warehouse recently, I came across Dinky. These notes cover which of those problems it solves.

Dinky is an open-source platform for developing and managing Flink jobs. Its design is lightweight and easy to use, and it can manage multiple Flink clusters from one place. Developers can write and debug Flink SQL online through the web console, then submit it to a chosen Flink cluster with one click, dramatically lowering the development and operations cost of real-time computing tasks.

In other words, it pulls the entire chain of "write SQL, debug, submit, monitor" into a single browser page. Compare that with the native Flink workflow — write code locally, build a jar, submit from the command line, then check status in the Flink Web UI — and this one-stop experience is a huge time-saver for SQL-first real-time job development.

Project Home​

https://github.com/DataLinkDC/dinky

Features​

Dinky ships with built-in whole-database synchronization, which can sync database tables from microservice systems into a real-time data warehouse — an effective answer to the data-silo problem in microservice architectures.

This point deserves elaboration. Under a microservice architecture, every service has its own database, and analytics often needs to bring together tens or hundreds of tables. With native Flink CDC, you typically write one source definition per table — tedious at scale, and it eats up a lot of database connections. The idea behind whole-database sync is: one job reads the change log for the entire database (e.g., MySQL's binlog), then splits the stream by table inside the job and writes to downstream targets. Both the configuration burden and the resource footprint are much smaller.

Dinky also supports developing and managing Flink SQL across multiple Flink versions, and provides data lineage analysis to help developers see data flows and dependencies clearly, which makes troubleshooting and system maintenance easier.

The value of lineage analysis only becomes obvious once job counts grow. When a result table shows anomalous data, you can trace up the lineage graph to the intermediate and source tables it depends on, instead of relying on memory or reading through code. Conversely, before decommissioning or changing a table, you can first see exactly which downstream jobs would be affected.

Overall, Dinky is quite complete across real-time data development, job management, and data governance. It is a genuinely practical Flink development platform and well worth adopting in real-time data warehouse projects.

Building a Real-Time Data Warehouse​

With Flink SQL plus Dinky, you can quickly build real-time warehouse jobs, for example:

• Real-time user behavior analysis
• Real-time transaction statistics
• Real-time risk control systems

What these scenarios have in common: data keeps flowing in, and the business wants results at minute or even second granularity. A traditional offline warehouse running daily or hourly batches can't meet that freshness bar. Writing the logic as streaming jobs in Flink SQL and letting Dinky manage the lifecycle brings the development rhythm very close to writing offline SQL.

Data Synchronization and CDC​

Using Dinky's whole-database sync capability, you can implement Change Data Capture (CDC) and stream data from business databases into a data warehouse or message queue in real time.

In a nutshell, CDC works by subscribing to the database's transaction log: every INSERT / UPDATE / DELETE leaves a record in the log, and the CDC tool parses those records into a stream of change events. Compared to periodic full-table scans, it is less intrusive on the source database, has lower latency, and can even capture deletes. A typical Flink SQL CDC source table definition looks roughly like this:

-- Declare a CDC source table that captures changes to the MySQL orders table in real time
CREATE TABLE orders_source (
  order_id BIGINT,
  user_id BIGINT,
  amount DECIMAL(10, 2),
  PRIMARY KEY (order_id) NOT ENFORCED
) WITH (
  'connector' = 'mysql-cdc',   -- Use the MySQL CDC connector to read the binlog
  'hostname' = '...',
  'database-name' = '...',
  'table-name' = 'orders'
);

In Dinky, this SQL can be debugged online and its results previewed; once confirmed, submit it to the cluster — no more repeated local package-and-verify cycles.

A Real-Time Data Development Platform​

Dinky can also serve as a company's unified real-time computing development platform, giving data engineers a standardized development environment.

The point of a unified platform is not just convenience: SQL scripts are hosted centrally with version history, and jobs are submitted and operated the same way for everyone, so new team members don't have to first untangle everyone's personal submission scripts and environment quirks. For team collaboration, that often matters more than any single feature.

Pitfalls and Caveats​

• Dinky itself does not run compute workloads — it is the entry point for submission and management. The Flink cluster still needs to be deployed and maintained separately, and version compatibility between the two should be confirmed in advance.
• Before using CDC whole-database sync, verify that the source database has its change log enabled (e.g., MySQL's binlog) and that the sync account has been granted the necessary privileges, or the job won't start.
• Online debugging is convenient, but the debug environment differs from the production cluster in resources and data volume. For jobs with complex logic, it's still wise to validate in a production-spec environment before going live.

Wrapping Up​

What Dinky fills in is the "developer experience" gap in the Flink ecosystem: online SQL editing, one-click submission, whole-database sync, and lineage analysis cover most of the daily work of real-time warehouse development. If your team's real-time jobs are mostly Flink SQL and you've been missing a unified place to develop and manage them, it deserves a spot on your shortlist.

Code that runs is not necessarily good code. These are my notes on SonarQube: what problems it can catch, how it works under the hood, and how to use it in CI/CD.

Why does this topic deserve its own post? Because code quality problems have a peculiar trait: invisible when you write them, expensive when they blow up. Manual code review catches some of it, but it's limited by reviewers' energy and experience, and a lot of the repetitive work — checking for null pointers, unclosed resources, hard-coded passwords — can be handed to a tool entirely. Once a team grows, without a unified, automated quality standard, the codebase quickly fragments into everyone's personal style.

SonarQube is an open-source platform for code quality management that helps developers continuously detect potential problems during development. Through static code analysis, coverage statistics, and multi-dimensional quality metrics, SonarQube can surface defects, security vulnerabilities, and performance risks before code reaches production, improving overall system stability and reducing long-term maintenance cost.

In team and continuous-integration settings, SonarQube typically serves as the code quality gatekeeper (Quality Gate), making quality checks automated and standardized.

How It Works​

A quick look at how it operates — once the mechanism is clear, the configuration that follows is straightforward.

SonarQube's analysis is static analysis: it never runs the code. Instead, it parses the source into a syntax tree and runs a rules engine over it to match known problem patterns. The system splits into two roles:

1. Scanner: runs on the build machine; it parses source code, executes the rules, collects coverage reports, and uploads the results to the server. There are Scanners for Maven, Gradle, and the command line.
2. Server: stores historical data, renders the web reports, and evaluates the Quality Gate. Every scan's results are compared against previous versions, which is how you get quality trends.

A Quality Gate is essentially a set of threshold assertions — for instance "zero bugs in new code," or "new code coverage no lower than some percentage." After a scan, the server evaluates each condition; if any one fails, the overall status is a failure — and the CI pipeline can use that status to decide whether to halt a release. This is the key difference from tools that merely "run and hand you a report to browse": the result can actually block the process.

Another concept worth mentioning is "New Code": for legacy projects, fixing every historical issue at once is unrealistic, so SonarQube's default strategy is to apply strict standards only to added and modified code, while the backlog gets paid down gradually. This lets old projects onboard smoothly.

Improving Code Quality​

SonarQube automatically detects potential problems in code, such as:

• Bugs
• Security Vulnerabilities
• Code Smells
• Potential performance issues

These categories descend in severity. Bugs are logic that will very likely go wrong — conditions that are always true, possible null-pointer dereferences. Security vulnerabilities cover risks like injection and hard-coded credentials. Code smells don't affect correctness but make the code progressively harder to change — overly long methods, deeply nested logic. SonarQube tags each issue with a severity level and an estimated fix time, which helps with prioritization.

Improving Maintainability​

SonarQube analyzes code across multiple dimensions, for example:

• Complexity
• Duplicated code
• Test coverage
• Technical Debt

Through these metrics, developers get a much clearer picture of the system's health and can refactor and optimize with focus.

Technical debt here is a very intuitive measure: sum the estimated effort of all outstanding issues, and you get the time needed to "pay off the debt." It may not be precise, but the trend is meaningful — steadily rising technical debt means the team is borrowing against its future development velocity.

Developers can find and fix these problems before release, reducing production incidents.

Beyond detection itself, SonarQube brings several engineering-practice advantages:

1. A rich plugin ecosystem: SonarQube supports many languages and frameworks, and its extensive plugin ecosystem lets you pick the right plugins to extend functionality for your project.
2. Continuous integration and deployment: SonarQube integrates with CI and CD tools to automate code checking and quality measurement, ensuring every build maintains good code quality.
3. Better team collaboration: SonarQube provides a centralized platform where team members can view code quality and metrics in one unified place, improving collaboration efficiency.
4. Better code maintenance: by analyzing quality metrics, developers gain a clearer understanding of the code's health and can maintain and optimize it with more precision.

Getting Started​

Example setup: Suppose we have a Java project. First add SonarQube's Maven plugin to the project, then run the command below to trigger static analysis:

<!-- Maven plugin configuration -->
<build>
    <plugins>
        <plugin>
            <!-- The official Maven Scanner plugin from SonarQube -->
            <groupId>org.sonarsource.scanner.maven</groupId>
            <artifactId>sonar-maven-plugin</artifactId>
            <version>3.9.1.2184</version>
            <executions>
                <execution>
                    <!-- Bind to the verify phase: scan after unit tests, before install/deploy -->
                    <phase>verify</phase>
                    <goals>
                        <goal>sonar-check</goal>
                    </goals>
                </execution>
            </executions>
        </plugin>
    </plugins>
</build>

Besides the plugin itself, the Scanner needs to know where to send results: the server address (sonar.host.url) and the authentication token (sonar.login or a token) usually go in Maven's settings.xml or in CI environment variables. Don't hard-code them in pom.xml and commit them to the repository.

After running mvn clean verify, SonarQube automatically analyzes the project's source code and generates a report on the SonarQube server. Developers can view the report through the server's web interface and optimize the code based on its findings.

Using It in a DevOps Pipeline​

Running a scan on a single machine is only the starting point — SonarQube's real value shows up inside a pipeline. In an actual DevOps workflow, SonarQube is typically paired with the CI/CD toolchain, for example:

1. Developers push code to the Git repository
2. The CI system (e.g., Jenkins) triggers a build
3. The build runs the SonarQube scan
4. The Quality Gate decides whether the release may proceed
5. Scan results are pushed to the team via DingTalk / WeCom / email

Step 4 is the heart of the whole chain: a failed Quality Gate halts the pipeline, blocking quality problems before merge or release instead of digging through reports afterwards. Combined with the platform's Pull Request analysis, issues can even be annotated directly in the review interface, letting code review focus on design while the mechanical checks go to the tool.

Pitfalls and Caveats​

A few things trip people up in real deployments:

1. The server's resource requirements are not small. SonarQube embeds Elasticsearch for indexing, which has real memory demands; on Linux you usually also need to raise kernel parameters like vm.max_map_count, or the service won't start. Small teams can start with a single-node Docker deployment, but don't put it on an under-provisioned machine.
2. SonarQube does not compute coverage. It only reads coverage reports (in the Java ecosystem, typically generated by JaCoCo). If the build doesn't generate a report first, the coverage on the dashboard stays at 0 — the single most common source of confusion for newcomers.
3. Tailor the rules to your team. The default rule set is comprehensive; enabling everything wholesale produces a flood of warnings, and the team quickly becomes "immune" to the results. Start from the default Quality Profile, disable rules that clearly don't apply, and make every warning worth acting on.
4. For legacy projects, start with "New Code." Don't try to zero out the backlog in one go. Put the Quality Gate's constraints on new code, and pay down the existing debt on a long-term plan.

Wrapping Up​

The core problem SonarQube solves is turning code quality from "people keeping watch" into "process guarantees." Static analysis finds bugs, vulnerabilities, and code smells; multi-dimensional metrics quantify maintainability; and the Quality Gate wires those standards into CI/CD, blocking anything below the bar. Onboarding is cheap — one Maven plugin and one command gets it running. The hard part is tuning the rules and thresholds to fit your team afterwards, so that it becomes a gatekeeper people genuinely trust rather than a red warning everyone ignores.

When building business systems, you frequently run into this requirement: a row changes in the database, and downstream caches, search engines, and data warehouses all need to change with it. Dual writes in application code are easy to miss; periodic full-table comparisons are far too heavy. There is a more elegant answer to this class of problems — subscribing directly to the database's change log — and that's why I put together these notes on Canal.

Canal is an open-source Alibaba component for real-time data subscription and consumption based on MySQL Binlog, commonly used to implement Change Data Capture (CDC).

In real-world business scenarios, database changes often need to be propagated beyond ordinary reads and writes, for example:

• Real-time data synchronization (database → data warehouse)
• Pushing data changes to an MQ (Kafka / RocketMQ)
• Building real-time analytics or monitoring systems
• Syncing data to a search engine (such as Elasticsearch)

What these scenarios share is that downstream systems don't care about "what the database looks like right now" — they care about "what changed." If application code has to manually notify downstream systems after every write, it's not only invasive but also very hard to keep consistent with the database transaction: if the write succeeds but the notification fails, the data quietly drifts out of sync. The Binlog, on the other hand, is the log MySQL itself guarantees to write, so using it as the data source sidesteps the dual-write problem entirely.

To make this work, Canal subscribes to the Binlog by impersonating a MySQL slave, parses out the insert/update/delete operations, and pushes these change events to downstream consumers in real time, achieving near-real-time data synchronization.

How It Works​

Canal borrows the mechanics of MySQL master-slave replication. In normal replication, the slave sends a dump request to the master, the master continuously streams Binlog events to the slave, and the slave replays those events to stay in sync.

What Canal does is "disguise" itself as a slave:

1. It sends a dump protocol request to MySQL, using exactly the same interaction as a real replica;
2. MySQL streams Binlog events to Canal, which parses the binary log and reconstructs the changes for every row;
3. The parsed, structured events are handed to downstream consumers — either pulled by clients over TCP, or delivered directly to a message queue such as Kafka or RocketMQ.

From MySQL's point of view, Canal is just an ordinary replica. No plugin needs to be installed on the database side, which is why this approach is so low-touch.

For Canal to subscribe successfully, MySQL needs to meet a few prerequisites:

# Key settings in my.cnf
log-bin=mysql-bin        # Enable Binlog
binlog-format=ROW        # Must be ROW mode to capture the full before/after image of every row
server_id=1              # Unique ID for each node in the replication topology; must not clash with Canal's slaveId

The Binlog format must be ROW. STATEMENT mode records the SQL statements themselves, from which the concrete per-row changes cannot be reconstructed; only ROW mode records row-level before-and-after images, which is what makes CDC meaningful. In addition, the account Canal uses to connect to MySQL needs the REPLICATION SLAVE and REPLICATION CLIENT privileges — the same requirements as a real replica.

Comparing Common CDC Components​

The CDC components you'll commonly see today are:

• Canal
• Debezium
• Flink CDC

Canal currently supports only MySQL, versions 5.x and 8.x.

GitHub - alibaba/canal: Alibaba's MySQL binlog incremental subscription & consumption component

The core mechanism of all these tools is essentially the same: parse the database's Binlog to obtain the changed data along with the operation type (INSERT / UPDATE / DELETE), and turn those changes into structured events for downstream systems to consume.

Where they differ is mainly in ecosystem positioning. Debezium is built on Kafka Connect, supports MySQL, PostgreSQL, and several other databases, and suits teams that already run Kafka. Flink CDC embeds change capture directly into Flink stream processing, so capture and transformation happen in the same job. Canal is more lightweight and focuses exclusively on MySQL.

Note that Canal currently supports mainly MySQL (versions 5.x and 8.x), which is why it is so widely used to build real-time synchronization and data distribution systems within the MySQL ecosystem. If your stack is pure MySQL and you don't want to bring in a heavy dependency like Kafka Connect or Flink, Canal is a pragmatic choice.

High Availability​

For high availability, Canal offers a cluster deployment mode. In a cluster, each Server node manages multiple synchronization task instances (Instances), and a task-distribution mechanism spreads the load.

Cluster mode relies on ZooKeeper for coordination: multiple Server nodes compete for the right to run a given Instance, only one node actually runs it at any moment, and the rest stand by. When the running node fails, a standby node detects this via ZooKeeper and takes over the task. The consumption position (i.e., the Binlog consumption progress) is also stored in ZooKeeper, so the node taking over can resume from the last recorded position, avoiding data loss or large-scale duplication.

In practice, though, network hiccups, database connection failures, or resource limits can occasionally cause synchronization instances to terminate. In production, Canal is therefore usually paired with an automatic restart mechanism or an ops monitoring system so that tasks recover automatically after failures.

Pitfalls and Caveats​

Building on the mechanics above, a few points are worth thinking through before rollout:

1. Binlog format and retention. The upstream database must run in ROW mode, and Binlog retention needs to be long enough. Otherwise, if Canal is down for a while and the logs at its saved position have already been purged, the task cannot resume and must be re-initialized from scratch.

2. The delivery semantics are at-least-once. After a failover or restart, a small window of change events may be delivered again. Downstream consumers must deduplicate by primary key and never assume each change arrives exactly once.

3. Ordering depends on the partitioning strategy. When delivering to an MQ, if changes to multiple tables or primary keys get scattered across different partitions, the consumption order may diverge from the Binlog order. For order-sensitive scenarios, route partitions by table name or primary key.

4. Zombie Instances. Task instances sometimes don't exit cleanly — they stall at a position and stop advancing. Monitoring can't just check process liveness; it must also verify that the position keeps moving forward and track the lag against the database's current Binlog position.

Wrapping Up​

Canal's idea is not complicated: impersonate a MySQL replica, subscribe to the Binlog over the standard replication protocol, and reconstruct row-level changes into structured events for downstream consumption. It's zero-intrusion on the database and lightweight to deploy, making it a great fit for real-time sync and data distribution in a pure-MySQL stack. What deserves engineering attention are the ROW format and Binlog retention policy, idempotent consumption downstream, and monitoring based on position advancement — get those solid, and the pipeline's stability follows.

Official site: PolarisMesh

Microservice Governance​

For traffic management, PolarisMesh supports rate limiting, circuit breaking with degradation, retries, and canary releases. When a service misbehaves or its response time climbs, the system can automatically trip the circuit breaker to keep the failure from spreading. Meanwhile, flexible traffic-control policies let you route traffic by percentage, by label, or by version, making version upgrades and feature rollouts safer and more controlled.

High-Performance Proxy​

PolarisMesh ships with a high-performance proxy component, Polaris Sidecar, which can serve as the data plane in a service mesh. Built on a high-performance network model, the proxy handles request forwarding and response processing efficiently, keeping network overhead as low as possible while providing full governance capabilities. With the proxy model, applications gain complete service governance without any intrusion into business code, dramatically lowering the cost of adoption.

PolarisMesh also supports a Proxyless mode, meaning services can adopt governance capabilities without a sidecar proxy. Developers can integrate directly via SDK or Java Agent, choosing whichever integration style fits their architecture best.

Multi-Protocol Support​

PolarisMesh supports the major communication protocols — HTTP, gRPC, TCP, and more — covering the communication needs of different kinds of microservices. It also offers SDKs for multiple languages, including Java, Go, Python, and C++, and supports common microservice frameworks such as Spring Cloud and Dubbo, so developers can bring existing systems onto PolarisMesh quickly without large-scale changes to business code.

Service Configuration Management​

Beyond governance, PolarisMesh includes a configuration center for centrally managing the configuration a microservice needs at runtime. Developers can manage service parameters in one place, with support for dynamic configuration updates: applications load new configuration without restarting, improving both operational efficiency and flexibility.

Observability​

In a microservice architecture, observability is essential to keeping the system running smoothly. PolarisMesh provides comprehensive monitoring and logging, collecting key data in real time: call traces, request success rates, latency metrics, and more. With these metrics, operators can quickly pinpoint bottlenecks and misbehaving services, speeding up troubleshooting.

PolarisMesh also offers an event center and operation auditing, recording significant system events and user operations to support both operations and security audits.

Extensibility and Flexibility​

PolarisMesh is built on a cloud-native architecture, with native support for Kubernetes, container environments, and hybrid-cloud deployments. Its components communicate through loose coupling, so users can extend individual modules as business needs dictate. It can also be used together with a Service Mesh architecture, giving microservice systems stronger governance capabilities without giving up performance.

PolarisMesh vs. Nacos​

PolarisMesh and Nacos are both common service governance components in microservice architectures, but they differ in positioning and emphasis.

In terms of positioning, Nacos leans toward being a service registry and configuration center, focused primarily on service discovery and configuration management, whereas PolarisMesh positions itself as a complete microservice governance platform — it provides registration and configuration management, plus much richer traffic and service governance capabilities.

In terms of architectural capability, PolarisMesh offers a more complete, integrated feature set across traffic management, service governance, and observability — routing policies, rate limiting, circuit breaking, canary releases, and so on — while Nacos concentrates on service registration and configuration management themselves.

Wrapping Up​

PolarisMesh consolidates registration and discovery, traffic management, fault tolerance, configuration management, and observability into a single platform, and its two integration modes — Sidecar and Proxyless — give teams with different architectures room to choose. If all you need is a registry plus a configuration center, Nacos is pleasantly lightweight; but once governance needs like rate limiting, circuit breaking, and canary releases surface, the all-in-one PolarisMesh saves you the cost of stitching multiple components together. Before choosing, it's worth mapping out your actual governance requirements and then deciding which layer of capability to bring in.

While evaluating our team's containerization options recently, one question was unavoidable: should the K8S cluster be self-hosted, managed, or should we skip nodes altogether? ASK, a fully serverless flavor of cluster, sits squarely in that last category, and it's worth writing up separately.

For small teams, the hidden cost of self-hosting K8S is routinely underestimated: the control plane needs high availability, etcd needs backups, versions need upgrading, nodes need patching. None of this work produces business value, yet none of it can be skipped. Cloud vendors' managed offerings target exactly this pain, and ASK goes one step further by removing the node layer from the user's view entirely.

Alibaba Cloud ASK (Alibaba Cloud ACK) is Alibaba Cloud's managed Kubernetes service. Built on the open-source Kubernetes project, it gives users an easy-to-use, elastically scalable, highly available, and secure platform for deploying and managing containers — essentially serverless Kubernetes. You can deploy containerized applications directly without purchasing nodes, with no node maintenance or capacity planning for the cluster, and you pay on demand based on the CPU and memory resources your applications are configured with. ASK clusters offer full Kubernetes compatibility while lowering the barrier to using Kubernetes, letting you focus on your applications rather than managing the underlying infrastructure.

The "SK" in ASK stands for Serverless Kubernetes — the official docs now also call it ACK Serverless. The core difference from standard managed ACK: the managed version runs the control plane for you, but the worker nodes are still your own ECS instances; ASK does away with the concept of worker nodes entirely, and the smallest resource unit you deal with is the Pod itself.

What is ACK Serverless — Container Service for Kubernetes (ACK) — Alibaba Cloud Help Center

Pods in an ASK cluster run on Alibaba Cloud Elastic Container Instance (ECI) in a securely isolated container runtime environment. Each Pod instance is fully and strongly isolated underneath by a lightweight virtualized security sandbox, so container instances cannot affect one another.

Let me unpack that mechanism a bit. In traditional K8S, multiple Pods share the host machine's kernel, and isolation relies on namespaces and cgroups — soft isolation. If a container-escape vulnerability appears, other Pods on the same node can be affected. ECI's approach wraps every Pod in a lightweight VM sandbox: Pods don't share a kernel, so the isolation level approaches that of a virtual machine, while startup time and resource overhead stay far below a traditional VM. To the user, the sandbox is transparent — you still submit standard K8S resources like Deployments and Services, and the scheduling layer lands your Pods on ECI.

And precisely because billing is based on the CPU and memory declared by each Pod, resource requests/limits in ASK are no longer just scheduling parameters — they directly determine the numbers on your bill, which is a very different mindset from running your own cluster.

Advantages of ASK Clusters​

1. Easy to use: ASK provides an intuitive graphical interface and command-line tools, making it easy to create, deploy, and manage Kubernetes clusters.
2. Elastic scaling: ASK clusters automatically scale out and in based on actual demand, keeping applications performant and adapting flexibly to traffic changes.
3. High availability: ASK clusters use a multi-availability-zone architecture with automatic fault tolerance and recovery, keeping applications available during failures and minimizing business interruption.
4. Secure and reliable: ASK clusters offer multiple layers of security — network isolation, access control, data encryption, and more — to protect applications and data.
5. Ecosystem integration: ASK integrates seamlessly with other Alibaba Cloud products and services, such as Container Registry, Log Service, and CloudMonitor, providing a comprehensive solution.

Of these, I find elastic scaling the most practically meaningful. In a self-hosted cluster, scaling out Pods first requires schedulable nodes to exist; node scale-out alone takes minutes at minimum, plus cluster-autoscaler tuning. Once ASK removes the node layer, scaling out reduces to a single action — launching Pods — so the elasticity path is much shorter. For workloads with pronounced traffic peaks and troughs, this difference shows up directly in both responsiveness and cost.

Some Downsides​

1. Relatively high price: ASK carries service fees, which may matter for budget-constrained users. Compared with an ACK cluster on annual/monthly subscription, pay-as-you-go can incur extra costs if resource allocation is uneven.
2. Steep learning curve: users who have never worked with Kubernetes will need time to learn and adjust to ASK's concepts and workflows.
3. Cloud provider lock-in: using ASK means deploying your applications on Alibaba Cloud, so you may face constraints and dependencies tied to that specific provider.

One more note on price: pay-as-you-go unit prices are usually higher than subscription ECS, so ASK only pays off when your workload is genuinely elastic — always-on services running at full load 24/7 are actually cheaper on reserved resources. Before choosing, it's worth estimating how much of your workload is "steady-state" versus "elastic."

Pitfalls and Caveats​

1. Configure resource specs carefully. ASK bills by the resources each Pod declares: casually oversized requests inflate the bill, while undersized ones risk OOM kills. Before going live, base your configuration on load-test results that reflect real usage.

2. Watch for behavioral differences from standard K8S. Having no real nodes means node-dependent features — DaemonSet, hostPath, hostNetwork — behave differently from a normal cluster. Verify each item before migrating existing applications, and treat the official docs as the source of truth for what's supported.

3. Cold starts aren't free. Spinning up a Pod on ECI takes extra instance-creation time compared with scheduling onto an existing node. For latency-sensitive bursty traffic, consider pre-warming or keeping some always-on replicas.

Conclusion​

Compared with building and maintaining your own K8S cluster — heavy time investment, high maintenance cost, long-term upkeep, tedious upgrades, security patching you have to guarantee yourself, fiddly configuration, and real demands on your team's ops capability — using ASK spares you all of that and lets you pour more energy into product development. And when traffic grows later, you simply scale out Pod replicas, without managing the physical limits of underlying node resources.

My take: for a small team without dedicated ops staff, a serverless cluster like ASK is a pragmatic starting point. Once the business scales up and cost sensitivity rises, it's not too late to reevaluate moving back to managed or self-hosted clusters.

Extensibility​

A system should be designed with good extensibility, so it can support future business growth and changing requirements. This usually means adopting modular, layered architectures, so that new features, larger data volumes, or more users can be added without jeopardizing overall stability.

Reliability​

A system needs a high degree of reliability — it must stay stable and available over long-term operation. Architecture design typically has to account for failure recovery mechanisms, fault tolerance, and data backup strategies, so that the business can recover quickly when something goes wrong.

A highly reliable system generally needs to be able to:

• Detect failures automatically
• Repair problems automatically
• Switch over automatically (failover)

These mechanisms minimize the business impact of system failures.

Extreme Performance​

Architecture also has to address performance, ensuring the system responds to user requests quickly and handles heavy concurrent load. Common performance optimization techniques include:

• Well-chosen data structures and algorithms
• Caching
• Load balancing
• Concurrency control

In practice, performance is usually measured in TPS (Transactions Per Second). System performance and concurrent user count are not simply positively correlated. In general:

• A system's maximum TPS is fixed within a certain range
• Concurrent user count can be regulated through queuing, rate limiting, and similar techniques

Performance testing usually means simulating the worst case and stress-testing the servers. For example:

• Large systems: 10,000 – 50,000 concurrent users
• Small to mid-sized systems: around 5,000 concurrent users

System throughput is calculated as:

Throughput (TPS) = concurrency / average response time

Security​

A system must be designed with solid security to protect user data and system resources. Common security measures include:

• Authentication
• Access control (authorization)
• Data encryption
• Security auditing

Layered security mechanisms effectively guard against data leaks, unauthorized access, and potential attacks.

Maintainability​

A well-designed architecture should be highly maintainable, so the system can be modified, extended, and serviced easily down the road. This usually requires:

• A clear code structure
• Complete documentation
• Consistent code comments
• Ease of debugging and testing

All of this helps developers understand the system quickly and iterate on it.

Scalability​

A system also needs to scale well, expanding or shrinking resources dynamically with the size of the business. Common approaches include:

• Vertical scaling (scale up): adding resources to a single machine, e.g. more CPU or memory
• Horizontal scaling (scale out): adding server nodes and spreading the load across a distributed architecture

Modern architectures generally favor horizontal scaling to support large-scale business growth.

High Availability​

High availability is one of the key goals for large systems: the system must keep serving even when failures occur. Common techniques include:

• Multi-node deployment
• Load balancing
• Automatic service failover
• Multiple data replicas

Together these mechanisms minimize the impact of downtime on users.

Testability​

A system should be designed for testability, so it can support automated testing and continuous integration. This usually comes from modular, decoupled design that makes it easy to run:

• Unit tests
• Integration tests
• System tests

Good testability meaningfully improves both software quality and development velocity.

Portability​

Architecture also needs good portability, so the system can be deployed and run easily in different environments. For example:

• Supporting different operating systems or cloud platforms
• Using standardized interfaces
• Reducing dependencies on any specific platform

This lowers the cost of future migrations and upgrades.

Cost​

In small projects, cost usually isn't a primary concern. But as a system grows, cost becomes an important metric in architecture design.

One thing to keep in mind:

Low cost, high performance, and high availability tend to conflict with each other.

So in architecture design, low cost usually isn't the top priority — it's a factor to be balanced after performance and availability requirements are met.

How an Architect Should Think​

In real engineering work, an architect does more than design system structure — what matters more is the ability to think holistically. A mature architect weighs system design across multiple dimensions instead of fixating on any single technical point.

1. A global perspective. An architect needs to think about the system as a whole — business trajectory, technology choices, team capability, and future room to grow — not just the technical problem at hand.

2. The ability to make trade-offs. In real system design, a "perfect solution" rarely exists. Performance, cost, complexity, and maintainability constantly pull against each other. One of an architect's core skills is finding the most reasonable balance among them.

3. Long-term thinking. Architecture shouldn't only solve today's problems; it should account for how the system evolves over three years, five years, or longer. Good architecture supports continuous business evolution instead of needing a rewrite every so often.

4. Decomposing complex problems. Large systems are often extremely complex. An architect needs to break them down into clearly defined modules so the team can divide the work and iterate continuously.

5. Technical judgment. In a constantly shifting technology landscape, an architect must judge which technologies genuinely fit the business and which are just passing fads, and make stable, dependable technology decisions accordingly.

Put simply, an architect isn't just a technical designer — they're the planner of the system's long-term evolution.

Wrapping Up​

There is no silver bullet in architecture design, and these principles are inherently in tension: extreme performance conflicts with low cost, high availability conflicts with architectural simplicity. In practice, first identify what the business truly needs at its current stage, then make trade-offs around that — rather than trying to max out every dimension at once. For an architect, the ability to keep balancing these dimensions matters more than mastery of any specific technology.

Background​

Ent started as an internal Facebook project, designed to tackle the problems of complex data models, dense entity relationships, and hard-to-maintain data access layers in large systems. As Facebook's internal systems kept growing, traditional ORMs increasingly showed their limits in type safety, query maintainability, and modeling complex relationships — so Facebook built a brand-new ORM: Ent.

Ent's core philosophy is Schema First + Code Generation. You first define your data model schema in Go code, then run a code generator that produces the complete data access layer: entity structs, query builders, relationship operations, and CRUD methods. Because most of the logic is generated and validated at compile time, Ent delivers stronger type safety and better runtime performance without sacrificing development speed.

After going open source, Ent quickly drew broad attention in the Go community and has become one of the most distinctive ORM frameworks in the Go ecosystem, particularly well suited to large systems with complex data relationships — social platforms, e-commerce systems, and core data services in microservice architectures.

GitHub - ent/ent: An entity framework for Go

Strengths​

• Performance: Ent builds its data access layer through code generation. Where traditional ORMs use reflection at runtime to inspect structs and produce SQL, Ent generates the complete data operation code at compile time, so runtime overhead is minimal and performance approaches hand-written SQL. This design lets Ent hold up well even under high concurrency.
• Ease of use: Ent's query builder is a strongly typed DSL (Domain Specific Language). All query fields, relationships, and operations are produced by the code generator, so mistakes surface at compile time.
• Extensibility: Ent offers rich extension mechanisms that developers can customize to their needs, making it straightforward to integrate Ent deeply with existing systems.
• Type safety: Ent leans on Go's type system to guarantee code correctness, reducing the risk of runtime errors.

Weaknesses​

Compared with other ORM frameworks, Ent has a smaller community and relatively sparse documentation and learning resources. In certain specific scenarios its performance can hit limits and require targeted optimization.

A Brief History of Ent​

• Early 2018: Ent is started as an internal project at Facebook.
• Early 2019: Facebook open-sources Ent. It attracts wide attention and adoption, and the community steadily grows.
• Mid 2019: A series of releases bring performance optimizations, bug fixes, and new features, making the framework more stable and efficient.
• Early 2020: Ent ships version 1.0, adding multi-database support, custom types, and other new capabilities. The 1.0 release marks Ent as mature, stable, and ready for production.
• Mid 2020 to now: As the framework continues to evolve, more and more developers adopt Ent and contribute code and documentation.

Ent vs. Other Go ORM Frameworks​

The common ORM frameworks in the Go ecosystem are GORM, XORM, SQLBoiler, and Ent, and their design philosophies differ noticeably.

From a design-pattern perspective:

• GORM / XORM: Runtime ORMs that use reflection to inspect structs and generate SQL; easy to pick up.
• SQLBoiler: Generates code from the database schema, with good type safety and performance.
• Ent: You define models via Go schemas, then generate the data access code automatically, with an emphasis on type safety and complex relationship modeling.

In short:

• Small projects or rapid development: GORM is the better fit
• Complex data relationships or large systems: Ent has the edge

Wrapping Up​

Ent's core competitive advantage is the strong typing that comes from Schema First plus code generation: query errors surface at compile time instead of blowing up in production. The trade-off is a community and body of documentation that don't match GORM's, and a workflow that requires you to buy into code generation first. If your project has complex data relationships and demands strong type safety, Ent deserves serious consideration; for a small project or quick prototype, a runtime ORM like GORM is simply less hassle.

Official docs: Kratos

Framework Strengths​

1. High performance: Kratos leverages Go's goroutines and efficient concurrency model to handle high-concurrency workloads with low per-request overhead.
2. Simple and approachable: the framework ships with thorough documentation and examples, and its clean design means developers don't need to master a pile of concepts and technologies — the barrier to entry is low.
3. Modular design: built-in modules for configuration, logging, database access, and more make it quick to stand up a microservice, and plugin-based development lets you extend the framework as needed.
4. Highly customizable: rich configuration options let you tailor it to your requirements, and support for custom middleware, interceptors, and the like means developers can extend it themselves.

Core Features​

1. Service registration and discovery: supports etcd, Consul, and other registries.
2. Configuration management: supports multiple configuration sources, such as local files and etcd.
3. Logging: supports multiple log outputs, such as files, console, and Kafka.
4. Database access: supports MySQL, PostgreSQL, Redis, and other stores.
5. HTTP services: supports HTTP and HTTPS.
6. gRPC services: gRPC is supported natively.
7. Security: provides security middleware, with support for JWT authentication, TLS, and more.

Comparison with Other Microservice Frameworks​

1. vs. Spring Cloud: Spring Cloud is a microservice framework in the Java ecosystem. Compared with Kratos it has a steeper learning curve, requires mastering more concepts and technologies, and is generally more heavyweight, with higher deployment and operating costs.
2. vs. Dubbo: Dubbo is a Java microservice framework open-sourced by Alibaba, with no Go support. Kratos's documentation and examples are much closer to how Go developers work, making it cheaper to learn and use.
3. vs. gRPC: gRPC is Google's open-source cross-language RPC framework with excellent performance, but it only solves the communication layer — it lacks the service-governance middleware, interceptors, and supporting pieces, and has a fairly steep learning curve of its own.
4. vs. Go-Micro: Go-Micro is also a Go microservice framework, with a more flexible design, support for multiple protocols and registries, and multi-language support — but a steeper learning curve, requiring more concepts and technologies.
5. vs. Gin: Gin is a Go web framework — even lighter, and great for quickly building simple web services, but it lacks microservice-framework capabilities like service registration and discovery and RPC calls.
6. vs. Micro: Micro is another Go microservice framework, flexible in design with multiple protocols and registries, but it involves more concepts and is harder to get started with than Kratos.
7. vs. Beego: Beego is a Go web framework positioned similarly to Gin — good for quickly building web services, and likewise missing microservice capabilities such as service registration and discovery and RPC calls.

Wrapping Up​

Among Go microservice frameworks, Kratos strikes a good balance between performance, ease of use, and feature completeness. It bakes in the microservice fundamentals — service discovery, configuration management, logging, dual gRPC/HTTP protocols — while staying lightweight and customizable. Compared with pure web frameworks like Gin and Beego, it fills in the service-governance capabilities; compared with Go-Micro and Micro, it has fewer concepts and a faster ramp-up. Add Bilibili's ongoing maintenance and the wealth of documentation and examples, and Kratos is an option worth putting at the top of the list for any team bringing microservices to Go.

Once your business logic starts requiring multi-level relationship queries, joins in a relational database get unwieldy. These notes cover Neo4j, a graph database: what it is, how it's licensed, how its data is organized, and when it's worth adopting.

Neo4j is a high-performance graph database that stores and processes data using the graph data model. Compared with traditional relational databases, Neo4j is far better at handling complex, interconnected data, making it a natural fit for relationship-heavy workloads.

In many business systems, data is riddled with relationships, for example:

• Friend relationships in social networks
• User-product relationships in recommendation systems
• Entity relationships in knowledge graphs
• Attack path analysis in network security

In these scenarios, querying with a traditional relational database usually requires heavy join operations, and both query complexity and performance degrade quickly. A graph database can traverse relationships directly, making it much more efficient at handling complex connections.

Why do joins become the bottleneck? In a relational database, "relationships" are not first-class citizens — they're expressed through foreign keys and junction tables. Every level of join forces the database to do an index lookup to match rows across two tables, and the deeper the joins and the larger the data, the faster that matching cost grows. The classic example is a "friends of friends of friends" query: in MySQL that means self-joining the same relationship table three times, while in a graph database it's just two more hops along pointers.

Licensing​

Official site: Neo4j Graph Database & Analytics – The Leader in Graph Databases

Neo4j uses a dual open-source/commercial licensing model, so get clear on the boundary between the two editions before committing.

Community Edition​

• Open source and free
• Suitable for personal projects and small-to-medium applications
• Single-instance deployment only — no clustering or online hot backup

Enterprise Edition​

• Commercially licensed
• Provides clustering, high availability, security and authentication, and other advanced features
• Aimed at production environments with hard requirements on availability and access control

In short: for feature validation, internal tools, and workloads with manageable data volumes, the Community Edition is enough. Once you need high availability or fine-grained access control, there's no way around the Enterprise Edition — factor that licensing cost into your plan early.

Graph Database Data Structure​

A graph database is built on three constructs:

Node Relationship Property

The division of labor is clean: nodes represent entities (people, products, devices); relationships represent connections between entities and must have a direction and a type; properties are key-value pairs attached to nodes or relationships. Mapping to relational terms: a node roughly corresponds to a table row, and a relationship roughly corresponds to a foreign key — except a relationship can carry its own properties (a "friend" relationship can store when it was established, for instance), which a foreign key cannot.

For example:

(A)-[FRIEND]->(B)
(B)-[FRIEND]->(C)

Relationships are stored as direct pointers between nodes. The database simply follows the relationship pointers during traversal — no table joins required — which is why relationship queries dramatically outperform relational databases. This design has a name: index-free adjacency. Finding a node's neighbors doesn't go through a global index; it starts directly from the node's own pointers, so the cost of one traversal step depends only on how many relationships that node has, not on the size of the entire database.

Queries use the Cypher language, whose syntax mirrors the graph notation above almost exactly:

// Find A's friends of friends (a two-hop traversal)
MATCH (a:Person {name: 'A'})-[:FRIEND]->()-[:FRIEND]->(fof)
RETURN fof.name

You essentially draw the pattern and it becomes the query — one reason graph databases are so intuitive to model with.

Main Features​

Graph data storage and querying: Neo4j uses the graph data model to store data, making complex relationship data easy to work with.

Efficient query performance: thanks to the graph data model, Neo4j handles deep queries and complex relationship queries with ease and delivers higher query performance. Query time depends mainly on the size of the subgraph being traversed, not the size of the whole database.

Scalability: Neo4j scales comfortably to tens of billions of nodes and relationships.

ACID transaction support: Neo4j supports ACID transactions, ensuring data consistency and reliability. This sets it apart from the many graph computation frameworks that only do offline analytics — it is a database capable of taking online writes, not just an analysis tool.

Through the graph data model, Neo4j makes data relationships far more intuitive to express and holds a clear performance advantage in complex relationship queries. In traditional business systems, relational databases remain the core data store. But in relationship-heavy scenarios (social graphs, recommendation systems, knowledge graphs, and the like), a graph database can serve as a relationship analysis engine that complements the traditional database.

Pitfalls and Caveats​

1. Don't use a graph database as a replacement for a relational one. For aggregations, batch reporting, and other column-scan workloads, the graph model offers no advantage; the sensible architecture is usually an RDB for the master data and Neo4j for the relationship subset.

2. Change your modeling mindset. Graph modeling starts with "what are the nodes, what are the relationships called, which way do they point" — not with designing table schemas. Get a relationship's direction or type wrong and every subsequent query will feel awkward.

3. Bound your traversal depth. Cypher supports variable-length path matching; a query without a depth limit or node labels can easily degenerate into a full-graph scan and drag down the entire instance.

Wrapping Up​

A graph database solves exactly one core problem: making "relationships" first-class citizens and replacing joins with pointer traversal. The Neo4j Community Edition is plenty for validating ideas; production high availability means budgeting for the Enterprise license. It isn't a replacement for relational databases but a complementary engine for relationship-heavy workloads — confirm that your query patterns really are deep traversals before bringing it in.

By Data Model​

Relational Databases (RDBMS)​

Relational databases use the table as their core data structure, organizing data into rows and columns and using SQL (Structured Query Language) for querying and management. Tables are linked to one another through primary keys and foreign keys, enabling complex cross-table queries.

Their characteristics:

• Clear, strongly constrained data structures
• Support for complex queries (joins, aggregations, etc.)
• Transactions and ACID guarantees
• Well suited to structured data

Typical databases: MySQL, PostgreSQL, Oracle, SQL Server, and others.

Non-Relational Databases (NoSQL)​

NoSQL (Not Only SQL) databases drop the traditional relational table structure in favor of more flexible data models, built to handle large-scale data and high-concurrency access.

Their characteristics:

• Flexible data structures
• Strong scalability
• Good performance under high concurrency
• Usually weak or no transaction support

By Processing Style​

• Transactional databases: support transactions and the ACID properties (atomicity, consistency, isolation, durability).
• Non-transactional databases: no transactions or ACID guarantees, but higher performance and scalability in return.

By Deployment​

• Single-node databases: deployed on one server, with all data and computation on a single node.
• Distributed databases: spread data across multiple servers that coordinate over the network, achieving greater scalability and availability.

By Storage Medium​

• In-memory databases: keep data in RAM, so reads and writes are extremely fast — e.g. Redis, Memcached.
• Disk-based databases: persist data to disk or SSD — e.g. MySQL, PostgreSQL, Oracle.

By Access Pattern​

• OLTP (Online Transaction Processing) databases serve online transactional systems.
• OLAP (Online Analytical Processing) databases serve analytics and decision support — e.g. ClickHouse, Doris, StarRocks, Snowflake.

Typical Use Cases​

Relational Databases (RDBMS)​

• Applications handling large volumes of structured data, such as finance, e-commerce, and logistics.
• Applications requiring transactions and ACID guarantees, such as banking, telecom, and aviation.
• Applications requiring complex queries and analysis, such as data warehouses and business intelligence.

Non-Relational Databases (NoSQL)​

• Document databases: for semi-structured data, such as blogs, news, and comments.
• Key-value databases: for simple key-value data, such as caching and session management.
• Columnar databases: for large volumes of structured data, such as logs, events, and time-series data.
• Graph databases: for complex, interconnected data, such as social networks, recommendation systems, bioinformatics, and network security.
• Object databases: for applications that need object-oriented programming support, such as game development and multimedia applications.

In-Memory Databases​

• Applications needing fast response times, such as high-concurrency transaction processing and real-time analytics.
• Applications needing fast reads and writes, such as caching and session management.
• Applications needing real-time computation and analysis, such as real-time risk control and real-time recommendations.

Distributed Databases​

• Applications needing high availability and fault tolerance, such as distributed computing and distributed storage.
• Applications needing large-scale data processing and analysis, such as big data and machine learning.
• Applications needing multi-region deployment and data sharing, such as cloud computing and IoT.

Wrapping Up​

Databases can be sliced along many dimensions, and a single product often belongs to several categories at once — MySQL, for example, is a relational database, a disk-based database, and a transactional database all at the same time. When choosing one, don't obsess over the taxonomy itself; instead start from your data structure, consistency requirements, concurrency scale, and query patterns, and find the category that matches the business. Combining multiple databases in one system is also common in practice — Redis for caching, MySQL for business data, ClickHouse for analytical queries. Understanding what each category is for is what lets you make sound trade-offs in architecture design.

Why This Problem Is Worth Solving​

The business consequences of duplicate submissions are usually worse than they look: an order endpoint processed twice means two orders, and a payment request retried twice means a double charge. Disabling the submit button on the frontend only stops accidental clicks — it can't stop automatic retries at the network layer, let alone maliciously crafted requests. That's why this protection ultimately has to live on the server.

A CSRF attack works by tricking the user's browser into sending a request to the target site, and it succeeds because the browser automatically attaches cookies. The token, however, is not stored in a cookie, so a third-party site has no way to obtain it — which is why a single token mechanism can defend against both classes of problems at once.

How It Works​

The token mechanism is a common way for web applications to prevent duplicate submissions and CSRF attacks. The basic idea: each time a form is served, the server generates a token and stores it in Redis, then passes it to the client as a hidden form field or URL parameter. When the client submits the form, it sends the token back along with it. The server checks that the token is valid while processing the form, and deletes the token once processing is done.

The two protection goals rely on different aspects of the mechanism. Duplicate-submission protection depends on "delete immediately after validation" — by the time the same token arrives a second time, it no longer exists and the request is rejected. CSRF protection depends on the token only being issued to legitimate pages — a request forged by an attacker simply doesn't contain it. Redis plays two roles here: centralized storage, so any instance in a multi-instance deployment can perform the validation; and expiry control, using TTLs to automatically clean up tokens that were never consumed.

Implementation Steps​

1. Generate a token on the server and store it in Redis. The token can be a random number, a UUID, or anything else — as long as it is sufficiently random and unique. Set an expiry while writing it:

# NX ensures an existing token is never overwritten; EX sets the expiry in seconds
# so stale tokens don't pile up
SET submit:token:<tokenValue> 1 EX 600 NX

2. Pass the token to the client. For a form submission, include it as a hidden field; for an AJAX request, send it as an HTTP header.

3. When the client submits the form, it sends the token back to the server. For a form submission, a bit of JavaScript in the submit handler can add the token as a hidden field; for an AJAX request, attach the token as an HTTP header before sending.

4. While processing the form, the server checks whether the token is valid. If it is, this is a legitimate request and processing can continue; if not, this is a duplicate submission or a CSRF attack and the request should be rejected.

5. After processing the form, the server deletes the token from Redis so it can't be reused.

The "validate" and "delete" in steps 4 and 5 must happen as a single atomic operation. If you do GET first and DEL second as two separate steps, two concurrent requests can both pass validation and duplicate-submission protection falls apart. A short Lua script lets Redis perform validate-and-delete atomically:

-- Validate and delete: if the token exists, delete it and return 1; otherwise return 0
-- The whole script runs atomically inside Redis, so only one concurrent request can succeed
if redis.call('GET', KEYS[1]) then
  return redis.call('DEL', KEYS[1])
else
  return 0
end

Security Considerations​

The token mechanism is an effective defense against duplicate submissions and CSRF, but the token itself must be protected, or an attacker can turn it against you. Specifically:

The token must be sufficiently random and unique, or an attacker may guess it or reuse it.

The token must be protected in transit, or an attacker may intercept or tamper with it. Always use HTTPS, and never put the token in a URL that will end up in logs.

The token must have an expiry, or an attacker may reuse it.

The token must be protected while stored in Redis, or an attacker may steal or tamper with it. The Redis instance should not be exposed to the public internet, and authentication should be enabled.

Pitfalls and Caveats​

The expiry should be set based on how long users actually take to fill out the form. Too short, and the token expires while the user is still working through a long form, guaranteeing a failed submission; too long, and stale tokens pile up in Redis.

When validation fails, the frontend must give clear feedback and guide the user to refresh the page to obtain a new token before resubmitting — not silently drop the request. Otherwise users just see "nothing happened when I clicked" and keep hammering the button.

Also think through when to delete: whether to return the token when business processing fails, allowing the user to retry directly, depends on the idempotency requirements of the specific business — there is no one-size-fits-all answer.

Wrapping Up​

The heart of the token mechanism is the "one-time credential": issued by the server, stored in Redis, validated on submission, deleted after use — one flow that solves both duplicate submissions and CSRF. When rolling it out, keep an eye on four things: token randomness, security in transit and at rest, a sensible expiry, and atomic validate-and-delete. Get those right and you have a low-cost, high-value protection you can apply broadly.

Internal Architecture Components​

The Docker Daemon and Docker Client communicate over a RESTful API: the Client sends commands to the Daemon, the Daemon receives and handles them, and finally returns the result to the Client.

When the Docker Daemon receives a command to create a container, it first checks whether the required image already exists locally. If not, the Daemon pulls it from the Docker Registry; if it does, the Daemon creates a container from that image and runs it inside its own isolated namespace.

While a container is running, the Daemon routes its output and error messages to stdout and stderr, and users can view the container's output through the Client. When the container is no longer needed, the user sends a stop command via the Client. The Daemon sends SIGTERM to the processes inside the container so they can exit gracefully; if they haven't exited within a certain window, the Daemon sends SIGKILL to force-kill them.

How Containers Are Implemented​

Docker containers rest on a few mechanisms:

1. Namespaces: Docker uses namespaces to isolate processes, networking, file systems, users, and other resources between containers. Each container has its own namespaces, keeping containers independent of and invisible to one another.
2. Control Groups (Cgroups): Docker uses Cgroups to limit the CPU, memory, network bandwidth, and other resources a container can use. Each container gets a resource quota, so containers share the host's resources fairly.
3. Union File System: Docker uses a union file system to isolate container file systems. Each container has its own file system, yet can share parts of the host's. This keeps container file systems extremely lightweight, so they can be created and destroyed quickly.
4. Container images: containers are created from images — read-only file systems containing the application, dependency libraries, and configuration. Docker uses the union file system to store images in layers, so images can be shared and reused, dramatically cutting storage space and download time.
5. Container runtime: Docker uses a container runtime to start and manage containers. The runtime creates the container's namespaces, Cgroups, and union file system, then launches the processes inside. It also monitors container state and resource usage, adjusting resource quotas as needed.

Docker's operating model, then, is the interaction between the Docker Daemon and the Docker Client to create, start, stop, and delete containers. Images are the foundation of containers and can be built from Dockerfiles; containers run in isolated namespaces, providing isolation and security.

Underlying Storage​

Docker's storage model comes down to two things: container images and data volumes.

Image storage​

Docker uses a union file system to store images in layers. An image is made up of multiple read-only layers, each of which can be shared and reused. When a container starts, Docker unions those read-only layers together and stacks a writable layer on top, forming the container's file system. This keeps images very lightweight and containers fast to create and destroy.

By default Docker uses AUFS (Advanced Multi-Layered Unification File System) as the union file system, which supports multi-layer unioning, fast creation and teardown, and read-only layer sharing. Docker also supports other union file systems, such as OverlayFS, DeviceMapper, and Btrfs.

Data volume storage​

A data volume is a directory or file a container can read and write, used for sharing data between containers, persistent storage, and so on. Docker supports several volume types, including host bind mounts, named volumes, and anonymous volumes.

A host bind mount maps a directory or file on the host into the container, which can then read and write it. A named volume is managed by Docker and can be given a name, making it easy to manage and share. An anonymous volume is generated automatically by Docker for a container's temporary data.

Volume storage is built on Linux's file system mount mechanism. When a container starts, Docker mounts the volume at the specified path inside the container; data written to that path lands on the volume. When the container stops, Docker unmounts the volume, and the data stays on it — that's how persistence works.

Wrapping Up​

Docker's architecture isn't complicated: the Client sends instructions to the Daemon over a RESTful API, and the Daemon handles image pulls and the full container lifecycle. Containers themselves rely on Linux Namespaces for isolation and Cgroups for resource limits, while the union file system underpins layered image storage and the container's writable layer. Persistence is delegated to data volumes, whose mount mechanism keeps data independent of the container's lifecycle. Once you understand these mechanisms, the "lightweight, portable, reusable" qualities of containers pretty much explain themselves.

Background​

The most basic approach to data synchronization is scheduled batch jobs: pull a full copy of the business database every night and write it into the warehouse. It's simple to build, but latency is measured in hours or even days, and once data volumes grow, the load on the source database is significant. The moment the business asks for "real-time dashboards," batch jobs can't keep up. CDC is the technical path that emerged to solve this, and Flink CDC is one of the most thorough integrations of CDC capability with a stream processing engine.

Flink CDC (Change Data Capture) is an important component in the Apache Flink ecosystem, used to capture database changes in real time and turn them into data streams for processing. In modern data architectures, CDC is widely used to build real-time data synchronization, real-time data warehouses, and event-driven systems.

Compared with traditional synchronization approaches (such as scheduled full syncs), CDC captures data changes by reading the database's change logs—Binlog, WAL, and the like—achieving low-latency, low-intrusion synchronization.

What Is CDC​

CDC stands for Change Data Capture. The core idea: whenever data in the database undergoes an INSERT / UPDATE / DELETE, record those changes and deliver them to downstream systems in real time.

It's worth unpacking the implementation mechanics a bit. CDC comes in two broad flavors. One is query-based: it polls timestamp or auto-increment key columns to discover new data—simple to implement, but it can't catch DELETEs and easily misses records modified multiple times between two polls. The other is log-based: it consumes the change log the database already maintains for primary-replica replication—MySQL's Binlog and PostgreSQL's WAL both fall into this category. The log naturally records the complete before and after state of every row change, so log-based CDC captures the full set of insert/update/delete events with virtually no extra query load on the source database. Flink CDC takes the log-based path.

In real data architectures, CDC is commonly used for:

• Database → data warehouse (real-time warehousing)
• Database → Kafka (event streams)
• Database → search engines (e.g. Elasticsearch)
• Database → caching systems (e.g. Redis)

CDC is therefore one of the key foundational capabilities of a modern real-time data platform.

Flink CDC works by capturing the source system's change logs, converting them into data streams, and processing them in real time. This happens without affecting the source system, because the CDC component only reads the source system's logs and never writes to it. From the database's point of view, a CDC client is essentially indistinguishable from an ordinary replica—it masquerades as a replication client, subscribes to the change log from the primary, and parses log events into structured change records for Flink to process.

A strength of Flink CDC is its support for many data sources, including relational databases (MySQL, Oracle, etc.) and NoSQL databases (MongoDB, Cassandra, etc.). It also supports multiple data formats, such as JSON and CSV.

Another easily underrated strength is how deeply it integrates with Flink itself. Once the change stream enters Flink, it's just an ordinary data stream: Flink's windowing, multi-stream joins, state management, and checkpoint-based fault tolerance all apply directly. In other words, "capturing changes" and "processing changes" happen in the same engine—no separate Kafka Connect cluster to maintain as a relay, a shorter pipeline, and fewer moving parts to operate.

With Flink CDC you get real-time data synchronization and stream processing in one. For example, you can use the CDC component to sync data from a MySQL database into Kafka, then process it in real time with Flink—achieving real-time synchronization and processing that improve both the timeliness and the accuracy of your data.

Flink CDC Components​

At a high level, the division of labor in a Flink CDC pipeline is: the Source handles "in," the Sink handles "out," and Flink's operators and state mechanisms do the computation in between. Flink CDC ships with the following built-in components:

1. Source: reads change logs from the data source and converts them into a Flink data stream. A CDC Source typically takes a snapshot of the existing data first, then seamlessly hands off to incremental log consumption, so downstream consumers receive the complete "full plus incremental" dataset.
2. Debezium Connector: an open source CDC tool that connects to many data sources (MySQL, PostgreSQL, MongoDB, etc.) and captures their change logs. Flink CDC reuses Debezium's log-parsing capability under the hood, running it embedded inside the Flink job and eliminating the cost of deploying a standalone Debezium service.
3. Sink: writes the Flink data stream into a target system, such as Kafka, HDFS, or Elasticsearch. Working with Flink's checkpoint mechanism, the Sink avoids data loss when a job fails and restarts.
4. State: handles stateful stream processing—for example, merging two data streams requires the State component to store intermediate state. State is especially important in CDC scenarios: dimension-table joins, deduplication by primary key, and reconstructing the latest image from UPDATE events all depend on state storage.
5. Table API: provides a SQL-like API for convenient stream processing and querying. For most synchronization needs, declaring a CDC source table and a target table in Flink SQL and writing a single INSERT INTO gets the whole pipeline running—no Java code required.

Pitfalls and Caveats​

A few things worth checking up front in practice:

1. Confirm the source database's log configuration first. Taking MySQL as an example, Binlog must be enabled with format ROW, or you won't get row-level before/after images; the connection account also needs the replication privileges required to read the Binlog.

2. Don't set the log retention too short. If the job is stopped for a while and the Binlog for that period has already been purged by the database, the job can't find its resume position and has to redo a full snapshot.

3. The downstream must be able to handle UPDATE and DELETE. A CDC stream is a changelog stream with retraction semantics; if the Sink side (say, an append-only store) can't digest deletes and updates, the pipeline design needs extra handling.

Wrap-up​

CDC answers the question of how changes inside a database can flow out with low latency and low intrusion, and the log-based approach is today's mainstream path. Flink CDC's value is pulling log capture (via Debezium) and stream computation (the Flink engine itself) into a single framework: the Source reads changes, the Sink writes to targets, State powers stateful computation, and the Table API lowers the barrier to entry. For teams building a real-time warehouse or syncing heterogeneous data, it's a short-pipeline, few-component option well worth considering first.

The Symptom​

Say the Java side has 2022-01-01 12:34:56.789. After writing it into a plain DATETIME column, it reads back as 2022-01-01 12:34:57 — the milliseconds got rounded up into the seconds.

The discrepancy is easy to miss day-to-day, but it surfaces in a few scenarios:

1. A unit test saves an entity, reads it back, and compares the time field with assertEquals — and fails intermittently. The carry only happens when the milliseconds are 500 or above, so the problem comes and goes.

2. Cursor pagination or range queries on a creation-time column count a boundary record twice, or skip it entirely.

3. The timestamp in your logs and the one in the database differ by a second, and nothing lines up when you're tracing a request.

Why It Happens​

Since MySQL 5.6.4, DATETIME, TIMESTAMP, and TIME all support fractional seconds. The precision is set by the fsp (fractional seconds precision) in the type definition, ranging from 0 to 6, with a default of 0 — seconds only.

The key behavior: when an inserted value has higher precision than the column, MySQL rounds per the SQL standard rather than truncating, and by default emits no warning at all. So .789 carries into the seconds, while .400 is silently dropped — either way, the stored value no longer matches what Java has in memory.

On the Java side, LocalDateTime stores time internally in nanoseconds, and the JDBC driver passes the fractional seconds along to the server, so the rounding happens on MySQL's end. java.sql.Timestamp also has nanosecond precision — switching the Java type to it doesn't avoid the database-side rounding. The root of the problem is the column's precision definition.

Fixes​

1) Raise the column precision​

If your use case needs millisecond precision (enough for the vast majority of created/updated timestamp scenarios), just define the column as DATETIME(3):

ALTER TABLE t_order
  MODIFY create_time DATETIME(3) NOT NULL;

Use DATETIME(6) if you need microseconds. Note that the matching default-value functions need a precision argument too: NOW() only goes to seconds, so write NOW(3) or CURRENT_TIMESTAMP(3), otherwise the default value itself has no fractional seconds.

2) Truncate ahead of time on the Java side​

If you'd rather not touch the schema, align the precision to seconds before writing:

// Drop everything below seconds, matching DATETIME(0) precision
LocalDateTime now = LocalDateTime.now().truncatedTo(ChronoUnit.SECONDS);

Now the in-memory value and the stored value are identical, and comparisons and assertions never drift. Handling this once in a shared entity base class or audit-field populator is enough.

Gotchas​

• The rounding carries up rather than zeroing out. Many people instinctively assume extra precision gets chopped off; it's actually rounded, which is exactly where the "time gained a second" weirdness comes from. MySQL 8.0 offers the TIME_TRUNCATE_FRACTIONAL sql_mode to switch the behavior to truncation, but it affects behavior globally — evaluate existing workloads before flipping it.
• After changing to DATETIME(3), all existing rows have .000 fractional seconds. Keep that in mind when sorting a mix of old and new data.
• Higher fsp costs more storage. Pick what you need — there's no reason to default everything to DATETIME(6).

Wrapping Up​

The "extra second" is fundamentally a precision mismatch: Java sends nanoseconds, the MySQL column only stores seconds, and the excess gets rounded. Two ways out — either change the column to DATETIME(3)/DATETIME(6) so the database can hold it, or truncate on the Java side with truncatedTo(ChronoUnit.SECONDS) up front. There's only one principle: make the precision in memory match the precision in the column definition, so what you write is what you read.

While reviewing big-data storage options recently, there was no getting around the term "data lake". Business data long ago stopped being just tables in a relational database: event logs, JSON payloads, images, audio, and video all need somewhere to land. Traditional data warehouses struggle more and more with these workloads, and a wave of open-source projects built around the data lake concept fills exactly that gap. This post sorts out the difference between data lakes and warehouses, and takes a look at Iceberg, one of the most representative projects in the space — my own study notes.

Data Lakes vs. Data Warehouses​

A data lake is a centralized repository for storing both structured and unstructured data. It can hold data of every type: traditional relational data, semi-structured data, unstructured data, and more. Its defining traits are flexibility and scalability — data can be stored and processed with minimal friction.

A core idea behind data lakes is "store first, manage later" (schema-on-read): data is written in its raw format to cheap distributed storage (HDFS or object storage), and its structure is only parsed when it's actually analyzed. This is the exact opposite of the warehouse approach of "model first, then load" (schema-on-write), and it's why lakes can accommodate data in any shape.

Traditional data warehouses, by contrast, are built mainly for structured data and typically use relational databases for storage and management. They are highly normalized and structured, which guarantees data accuracy and consistency. The downside is that warehouses are inflexible: they can't store or process unstructured or semi-structured data, and they're hard to scale and upgrade.

The main advantages of a data lake over a data warehouse:

1. Unrestricted storage: a data lake can hold every kind of data — structured, semi-structured, unstructured. That makes it flexible enough for storage needs of any type and scale.
2. Faster processing: data lakes typically use distributed storage and compute, enabling high-speed processing and analysis. Traditional warehouses often need multiple rounds of data transformation and computation, which slows things down.
3. Lower cost: data lakes are usually built on open-source technology like Hadoop and Spark, keeping costs relatively low, whereas traditional warehouses require commercial database software.
4. More flexibility: a data lake makes it easy to store and process data however you need, while a traditional warehouse's repeated transformation and computation steps make the process complex and cumbersome.

Of course, flexibility has a price. A bare data lake is just a pile of file directories — no transactions, no schema enforcement. If writers and readers aren't careful, they trample each other, and over time the lake easily degrades into a "data swamp". The table format layer of abstraction exists to solve exactly this: it defines "table" semantics on top of the files. Iceberg is one of those projects.

Apache Iceberg​

Apache Iceberg

Iceberg is an open-source table format and processing library designed to solve the problems of managing and processing tables in a data lake. It was developed at Netflix and has since become a top-level project of the Apache Software Foundation.

Iceberg addresses several pain points of traditional data lakes, such as table versioning, table metadata management, partition management, and table snapshot management. It works with a range of storage and processing systems, including Hadoop, Spark, Presto, and Flink.

Its working mechanism deserves a closer look. Iceberg itself doesn't store data — the data still lives in columnar files like Parquet and ORC; what Iceberg manages is the metadata layer on top. The metadata has roughly three levels:

1. Snapshots: the complete state of the table at a point in time. Every committed write produces a new snapshot, and old snapshots remain readable.
2. Manifest lists and manifest files: they record which data files a snapshot contains, along with each file's partition ranges, row counts, column-level statistics, and so on.
3. Data files: the actual columnar files. Metadata only references them, never modifies them.

On write, the new metadata is persisted first, and the commit completes with a single atomic swap of the table pointer — giving you database-like ACID semantics on top of a file system. On read, the engine consults the metadata first and uses its statistics to skip irrelevant files, instead of listing an entire directory the way Hive does — on object storage, that makes a huge difference to performance.

Iceberg's Key Features​

Iceberg's main features:

1. Table versioning: Iceberg supports version control for tables. You can easily roll back to a historical version, or merge different versions of a table.
2. Table metadata management: Iceberg provides a metadata mechanism that makes tables easy to manage and query. Through the metadata you can inspect a table's structure, partitions, data statistics, and more.
3. Partition management: Iceberg supports multiple partitioning schemes, including hash partitioning, range partitioning, and hybrid hash-range partitioning. Pick whichever fits your needs.
4. Table snapshot management: Iceberg supports snapshots of tables, making backup and recovery straightforward.

One more detail on partitioning: Iceberg's partitions are hidden partitions. Partition values are derived automatically from column values through transform functions (say, truncating a timestamp to the day), so queries just filter on the original columns and the engine applies partition pruning automatically. Whoever writes the query doesn't need to know how the table is partitioned — much friendlier than Hive's manually maintained partition columns. The partition spec itself can also evolve, and changing it doesn't require rewriting historical data.

Schema evolution is another strong suit: adding, dropping, and renaming columns only touch metadata. Columns are tracked by unique IDs rather than by name or position, which avoids the classic old-table-format problem where renaming a column could make you read the wrong data.

Gotchas​

1. The small-files problem still exists. High-frequency streaming writes produce lots of small data files and metadata files; you need to run compaction regularly, or query performance keeps degrading.
2. Snapshots aren't cleaned up automatically. Every commit leaves a snapshot behind, and files referenced by historical snapshots are never deleted. Skip snapshot expiration for long enough and storage just keeps piling up.
3. Think through your catalog choice first. Iceberg's table pointer needs a catalog to manage it (Hive Metastore, JDBC, and others all work). For different engines to access the same table they must share the same catalog, and migrating later is painful.
4. Delta Lake and Apache Hudi are peer projects — similar positioning, different ecosystem emphasis. Before choosing, compare them against the compute engine you actually use most; don't just follow the crowd.

Wrapping Up​

Data lakes solve the "store anything" problem, but storage without management turns a lake into a swamp fast. Table formats like Iceberg fill in transactions, versioning, partitioning, and schema management on top of the files, giving the lake table semantics close to a warehouse's while keeping the advantages of open formats and low cost. For me, the key to understanding it is that three-level metadata structure — once you see how snapshots and manifest files are organized, rollback, time travel, and partition pruning all fall into place naturally. When I get the chance, I'll write up the hands-on experience of reading and writing Iceberg tables from Flink / Spark separately.

The reason the stack deserves its own post is that it's just about the simplest data structure there is, yet it's everywhere: function calls, exception stack traces, the browser's back button, an editor's undo — all the same model underneath. Plenty of problems that look complicated (bracket matching, expression evaluation) become instantly clear once you think of using a stack. Understanding the stack is also a prerequisite for understanding recursion and call-stack overflow issues.

Basic Concepts​

In a stack, insertion and deletion are usually called push and pop. When an element is inserted, it goes on top of the stack; when an element is removed, it comes off the top. The top of the stack holds the most recently added element; the bottom holds the earliest.

Think of a stack as a pile of plates: a new plate can only go on top, and you can only take plates from the top. That "single opening" constraint is exactly where the stack's value lies — it naturally remembers "who came in last", making it perfect for anything that needs to "retrace its steps".

Stacks are used everywhere. For instance, function calls and recursive calls in a computer are implemented with a stack. When a function is called, its arguments, return address, and local variables are pushed onto the stack; when the function returns, that information is popped back off.

Basic Operations​

The basic operations of a stack:

push(element): push an element onto the top of the stack.

pop(): pop an element off the top of the stack.

top(): return the top element without modifying the stack.

isEmpty(): check whether the stack is empty.

size(): return the number of elements in the stack.

Note the difference between pop and top: pop removes the top element and returns it, while top (called peek in some implementations) just "takes a look" and leaves the stack unchanged. Mixing these two up is a common source of bugs.

Implementing a stack with an array is very direct — just maintain an index pointing at the top:

// A simple array-based stack, core logic only
public class ArrayStack {
    private int[] data;
    private int top = -1; // index of the top element; -1 means empty

    public ArrayStack(int capacity) {
        data = new int[capacity];
    }

    public void push(int element) {
        // In real code, grow the array or throw a stack-full exception here
        data[++top] = element; // advance the top pointer first, then write
    }

    public int pop() {
        // Should throw on an empty stack; check omitted here
        return data[top--]; // return the top element and move the pointer back
    }

    public int top() {
        return data[top]; // read only, no modification
    }

    public boolean isEmpty() {
        return top == -1;
    }

    public int size() {
        return top + 1;
    }
}

A linked-list implementation treats the list head as the top of the stack: push is a head insertion, pop deletes the head node. The advantage is that you don't need to pre-allocate capacity.

Complexity Analysis​

A stack's time complexity is O(1), because every operation happens at the top. Its space complexity, however, is O(n), since all elements have to be stored.

Put another way: the stack's efficiency comes precisely from its restrictions. Because insertion and deletion in the middle aren't allowed, every operation reduces to a single read or write at the top — no shifting of other elements, no traversal. The one exception in the array implementation is the copy during resizing, but amortized, push is still constant time.

Classic Applications​

In computing, stacks are widely used in function calls, expression evaluation, compilers, operating systems, and more.

1. Function calls: when a function is called, its arguments, return address, and local variables are pushed onto the stack, and popped off when the function returns. This is the function call stack, the foundation of how function calls work.
2. Expression evaluation: when a computer evaluates an expression, a stack is typically involved. For example, converting an infix expression to postfix requires a stack to hold operators so the expression's value is computed correctly.
3. Compilers: while translating source code into target code, compilers use stacks for parsing and code generation. In a compiler, stacks hold information about variables, functions, statements, and so on.
4. Operating systems: process scheduling, interrupt handling, and similar OS features also rely on stacks. When a process is interrupted, the OS pushes the current process's context (register values, program counter, etc.) onto a stack, then runs the interrupt handler. When the handler finishes, the OS pops the context back off and resumes the process.

These four scenarios share one thing: a nested "enter — process — exit in reverse order" structure. Nested function calls, nested brackets, nested interrupts — they're fundamentally the same class of problem, which is why they all map onto the stack model. Bracket matching, monotonic stacks, and the iterative form of depth-first traversal you meet in coding exercises are all extensions of the same idea.

Gotchas​

1. Empty-stack operations: calling pop or top on an empty stack is either undefined behavior or throws outright. Either check with isEmpty first or explicitly rely on exception handling — don't just assume.

2. Stack overflow: recursion is fundamentally consuming the call stack, and going too deep overflows it (StackOverflowError in Java). For recursion of uncontrolled depth, rewrite it as an iterative version with an explicit stack and a loop, moving the data onto the heap.

3. Confusing pop with top: calling pop when you only meant to read the top silently drops an element — a class of bug that's especially hard to track down inside a loop.

Wrapping Up​

The stack is the textbook case of trading constraints for efficiency: expose only one opening at the top, and in exchange every operation costs O(1), while naturally fitting anything nested or backtracking-shaped. Once you've internalized push/pop/top and the LIFO mental model, the function call stack, expression evaluation, and the rest are just the same model applied at different levels.

Why Ingress Matters​

By default, services in a cluster can only reach each other from inside the cluster. The most direct ways to expose a service externally are NodePort or LoadBalancer Services, but both have limitations: NodePort has a restricted port range and makes domain and path management awkward, while LoadBalancer consumes one load balancer per exposed service—which on a cloud provider means real money.

Ingress solves exactly this: a single unified entry point receives all external HTTP/HTTPS traffic and dispatches it to the cluster's Services by hostname and path. Which Ingress controller you choose directly shapes the day-to-day experience of routing configuration, TLS management, and layer-4 exposure, so it's worth lining up the mainstream options side by side at selection time.

How It Works​

Ingress works by mapping HTTP and HTTPS requests onto Services. In Kubernetes, a Service is the resource object that routes traffic to Pods. When Ingress receives an external HTTP or HTTPS request, it matches the hostname and path in the request against the Ingress rules and routes the request to the corresponding Service.

Here's a point that trips people up: the Ingress resource itself is just a declarative routing rule—writing one does not make anything happen on its own. The real work is done by the Ingress controller. It runs in the cluster as Pods, continuously watches the API Server for changes to Ingress, Service, Endpoint, and other resources, translates those rules into configuration for its underlying proxy (Nginx, Envoy, or Traefik's built-in proxy, for example), and hot-reloads it into effect. In other words:

1. Ingress resource: declares "which path of which hostname forwards to which Service";
2. Ingress controller: watches for rule changes, generates and loads the proxy configuration;
3. Underlying proxy: actually receives external requests and forwards them.

When matching a request, the controller first matches the virtual host by Host, then matches routing rules by Path, and finally forwards the request to the Pods behind the corresponding Service. TLS certificates are typically attached to Ingress rules as Secrets, and the controller handles TLS termination at the edge.

Common Ingress Controllers​

To use Ingress, you must install an Ingress controller first. Kubernetes does not ship a default one, but there are many third-party controllers to choose from:

A quick take on the trade-offs of the common options:

1. Nginx Ingress Controller: the community's most common choice, built on Nginx as a reverse proxy, with rich documentation and case studies. Much of its configuration is done via annotations. A solid default option;
2. Traefik: ships with a dashboard, applies configuration dynamically without an Nginx-style reload, and integrates smoothly with Let's Encrypt for automatic certificate issuance;
3. Istio Ingress Gateway: if the cluster is already running the Istio service mesh, using its Gateway as the entry point ties in with in-mesh traffic governance (canary rollouts, circuit breaking, telemetry)—but pulling in Istio just for an entry point is far too heavy;
4. Others like HAProxy and Kong each lean into performance and API gateway capabilities respectively.

The selection logic can be plain and simple: with no special requirements, go with Nginx Ingress; if you want dynamic configuration and a more modern operations experience, look at Traefik; if you're already on a service mesh, just use the mesh's own Gateway.

Exposing Layer-4 Services​

We often want to expose services outside the cluster beyond layer-7 HTTP—some run on layer-4 protocols, such as MySQL, Redis, and MongoDB.

However, the Kubernetes Ingress specification itself only supports HTTP and HTTPS, so default Ingress controllers only support these two protocols. Some third-party Ingress controllers, though, can be extended to expose TCP and UDP.

1. Traefik: Traefik supports exposing TCP by specifying traefik.tcp.routers and traefik.tcp.services in the routing rules to configure TCP services and routes.
2. Istio: Istio is a service mesh framework whose Ingress Gateway component supports exposing TCP. You configure TCP services and routes by defining a VirtualService against the Ingress Gateway.

One addition: the Nginx Ingress Controller can also expose TCP/UDP services. The approach is to maintain two ConfigMaps, tcp-services and udp-services, containing the "external port → namespace/Service:port" mappings, then open the corresponding ports on the controller's Service. This bypasses the Ingress resource entirely—it's the controller's own extension mechanism.

Layer-4 exposure differs fundamentally from layer-7: TCP has no notion of Host or Path, so you cannot distinguish backends by hostname the way HTTP does—only by port number. Every exposed TCP service therefore consumes one port at the entry point, and port management gets fiddly as services multiply.

Pitfalls and Caveats​

1. If you create an Ingress resource and nothing happens, most likely no controller is installed in the cluster, or the ingressClassName on the Ingress doesn't match the controller—an especially easy trap when multiple controllers run in one cluster;
2. Be careful exposing database services (MySQL, Redis, etc.) to the public internet through Ingress: layer-4 forwarding adds no authentication hardening. Safer options are the internal network, a VPN, or a bastion host;
3. Annotations are not portable across controllers. When migrating from Nginx to Traefik, every rewrite, rate-limit, and timeout previously configured via annotations must be translated one by one;
4. For long-lived layer-4 connections, watch the proxy's idle timeout: the default is often shorter than what database client connection pools expect, and connections can get cut off mid-flight.

Wrap-up​

Ingress is the unified entry point for north-south traffic in the cluster: rules are declared by Ingress resources, and the actual forwarding is done by the controller. For layer-7 scenarios, any of the mainstream controllers will do the job—the differences lie mainly in configuration style and ecosystem. Layer-4 exposure goes beyond the Ingress spec itself and relies on each vendor's own extension mechanism, whether Traefik, Istio Gateway, or Nginx Ingress's ConfigMaps. Choosing based on your team's existing stack and operational habits is more practical than chasing feature lists.

Introduction​

The discussion revolves around three points:

1. The limitations of microservices
2. What Service Mesh does and how it is implemented
3. Serverless platforms built on top of Service Mesh today

The stacks we know best—Java, Spring Cloud, Spring Cloud Alibaba, and the like—belong to the traditional microservice architecture. This design suffers from tight SDK coupling: every update to the underlying SDK version can easily destabilize services at the business layer. When the SDK is tightly coupled to the code environment, many constraints follow—you cannot fully leverage the different languages and their ecosystems in the domains where they excel. For example: the nimble Go and Python languages with their goroutines, web scraping, and neural network libraries; or the rigorous Rust language with no garbage collection overhead, better suited to enterprise middleware scenarios that demand high efficiency and a low memory footprint. Every language has its own strengths and ecosystem, and looking further ahead, it is a clear trend that large industry leaders will spread their business deployments across different languages as they iterate.

The traditional microservice stack:

Moreover, the cross-OS portability the JVM once prided itself on has been superseded by container technologies and ecosystems such as K8S, Docker, Containerd, CRI-O, and Kata. Advanced features like garbage collection, dynamic proxies, agents, and ZGC have also been borrowed by other languages. What props up Java's position today is the ecosystem accumulated over the years (open source communities from Apache, Spring, Alibaba, Huawei, and others keep injecting new middleware—the well-known Kafka, Hadoop, HBase, RocketMQ, Flink, Seata, and so on—plus whatever comes next to keep the ecosystem alive). But over time, in the long run (20–30 years or more—a subjective prediction, take it for what it's worth), these advantages may be claimed by rising newcomers, and the next generation of microservice architecture is bound to iterate on the strengths of many languages.

A personal take: Java is now owned by Oracle, and Oracle is not a technologically aggressive company, so it may well constrain Java's development going forward. There is an open source edition, but open source efforts are less concentrated than corporate ones. Meanwhile Google's Go language has risen rapidly and already taken a slice of Java's market—likely one of the reasons companies like ByteDance, Bilibili, Tencent, Baidu, JD.com, and Xiaomi have adopted it.

Service Mesh​

Overview​

Back to our topic: Service Mesh. Its dominant implementation approach today is the Sidecar pattern. In K8S the Pod is the basic deployment unit; the Sidecar approach deploys a Proxy container inside the same Pod as the business Server container (Pod container injection), and the Proxy container intercepts all of the Server container's traffic.

The big-picture architecture of Istio's Service Mesh implementation:

citadel: the core security component responsible for identity authentication and certificate management.

galley: the configuration management component—it validates the format and content of configuration files and feeds that configuration to pilot and mixer.

pilot: the control hub, covering service discovery plus rule translation and distribution.

proxy: implemented by the C++-based Envoy together with Pilot-agent, providing dynamic service discovery, load balancing, TLS, circuit breaking, health checks, traffic splitting, canary rollouts, and more. It also produces telemetry data, giving microservices observability.

Ingressgateway: the gateway at the entrance—traffic from outside the mesh reaches services inside the mesh through this gateway.

With the Proxy container, business code is fully decoupled from its SDK, and you gain a long list of capabilities: traffic mirroring, canary releases, service registration and discovery, remote invocation, circuit breaking, degradation, distributed tracing, control plane management, and encrypted in-cluster transport (distributed transactions have no dedicated solution here, since transactions strictly speaking do not belong to the service governance layer). All of this is completely transparent to programmers: developers do not need to know the deployment details and can code almost exactly as they would for a single machine, with no language restrictions and no dependency on any microservice SDK. Companies are then free to develop different functional modules in different languages (which may bring higher code-management and communication costs, and a broader spread of languages the team must master).

The Official Demo​

The official demo, the Book-Info microservice app, can be deployed to a K8S environment with one Helm command so you can experience these features firsthand (Book-Info architecture diagram):

The Bookinfo microservices are deployed with Istio sidecars and are written in 4 different languages, and the Java-based Reviews service ships 3 different versions of the application—an excellent showcase of Istio's capabilities in multi-version control, traffic management, traffic migration, and integration. Deploy it yourself and you will get a very direct feel for how powerful Service Mesh is and how flexible the configuration can be (all changes take effect within seconds, with imperceptible traffic migration).

The Most Common Implementations​

Istio​

Istio is an open source Service Mesh framework initiated by Google, IBM, and Lyft. The project launched in 2017 and released version 1.0 in July 2018; as of 2022-12-06 the current version is 1.16.0. Istio is the flagship Service Mesh implementation today. If the Sidecar is the data plane of the whole Service Mesh, Istio's main improvements are on the control plane. Istio uses Envoy as the sidecar, and the entire control plane is written in Golang, bringing a significant performance boost. Istio is first and foremost a service mesh, but it is more than that: on top of classic service meshes like Linkerd and Envoy, Istio provides a complete solution that delivers behavioral insight and operational control over the entire mesh, meeting the diverse needs of microservice applications.

Istio

Linkerd2​

Linkerd is a high-performance network proxy open-sourced by Buoyant in 2016 and the industry's first Service Mesh framework, built to address the problems services face when communicating in distributed environments—unreliable and insecure networks, latency, packet loss, and so on. Linkerd was originally written in Scala; its successor, Linkerd2, was rewritten in Go and Rust.

The world’s lightest, fastest service mesh.

Conduit​

Conduit, released in December 2017, is another open source project sponsored by Buoyant after Linkerd, positioned as a standalone Kubernetes-focused edition of Linkerd. Conduit aims to radically simplify the complexity of running a service mesh on Kubernetes and improve the user experience, rather than optimizing for every platform the way Linkerd did. Its main goals are to be lightweight, high-performance, secure, and very easy to understand and use. Like Linkerd and Istio, Conduit consists of a data plane and a control plane: the data plane is written in Rust, which lets Conduit run on very little memory, while the control plane is written in Go.

Buoyant. All of the service mesh. None of the service mess.

Architecture Comparison​

Recap​

Kubernetes has already solved most operational problems—container deployment, high availability, multiple replicas, container migration, elastic scaling, rolling updates, image version management, liveness probing, compute resource allocation, node monitoring—with an excellent automated operations machinery. But K8S does not provide monitoring and traffic management for application-layer Service containers, nor inter-container invocation, monitoring, registration, configuration, or traffic governance. Service Mesh technology emerged to fill that gap and extend K8S in this area, so that microservices no longer worry about specific configuration or environment details, focus on implementing business features, and are freed from language constraints.

Serverless​

Overview​

As the microservice philosophy took hold, more and more companies gradually moved their applications from monoliths to microservice architectures, and the arrival of container technology accelerated that transition. Containers effectively solved the many runtime-environment discrepancies between services, but as the number of services grew, container orchestration and management became the new problem. Kubernetes then answered the challenge of orchestrating and deploying microservice containers at scale, making the industry realize that PaaS could actually land in practice. And as the container count in a microservice system kept climbing, service governance and traffic management became unavoidable—hence Istio, which, through its strategy of separating the network proxy from control, enables effective and sensible management of service control policies.

At this point the architectural evolution seems to have reached a lovely place:

• Microservices: solve the problem of bloated, over-coupled applications.
• Containers: solve runtime-environment discrepancies and deployment.
• Kubernetes: solves orchestration and "aggregate" deployment of huge numbers of micro-containers.
• Istio: solves the traffic, release, and governance problems services face once they go live.

At first glance, building a container cloud now has a complete chain and solution—everything looks so "perfect."

But step back and dig deeper: does service interaction under the microservice model still have problems? First, whether over HTTP or RPC, it is fundamentally one service remotely calling another; within application code, services cannot be truly transparent to each other. The consequence: no matter how "finely" the business is decomposed into microservices, the business units still cannot run and evolve independently—they are never fully decoupled—and when branching into different development domains, you cannot pick the most suitable implementation for each. So we want to standardize development environments through "templates" and "configuration," and provide an "event" mechanism to reduce the coupling of service-to-service interaction to a minimum. Second, there is the problem of dynamic scaling in production: elasticity in today's Kubernetes environments requires customers to collect monitoring data and implement scaling manually, but we would rather have production services scale automatically and intelligently.

Finally there is service standardization. We want the internal service model to be standard—quickly replicable, quickly buildable; service communication to be standard—standard protocols, standard formats; and the runtime environment to be standard—fast to deploy, fast to migrate.

Knative arrived to solve exactly these problems: direct remote invocation, automatic online scaling, version snapshots, and a whole series of standardization issues.

Knative​

Knative is the weapon Google unveiled for the Serverless world in 2018. It brings serverless architecture to any public or private cloud, letting users adopt serverless programming techniques. The main participating companies are Google, Pivotal, IBM, and Red Hat; it was announced on July 24, 2018 and is still evolving rapidly. Unlike Kubernetes, which must keep at least one Pod instance running to serve an application, Knative can scale down to zero (cold/warm start technology). Only when a client request hits your application does Knative actually spin up the application's Pods. This can save a great deal of the money spent keeping applications running year-round. For instance, infrequently accessed, low-traffic functional modules are good candidates for cold-started applications to save memory and compute (container startup time can be optimized to reduce cold-start latency—down to milliseconds at best; in China, Alibaba Cloud is probably the leader here).

Official sites:

Documentation at knative.dev/docs and github.com/knative

Knative's goal is to support the entire development lifecycle on top of Kubernetes. Concretely: first, it lets you as a developer write code in whatever language and style you want; second, it helps you build and package the application; and finally it helps you run and scale it. Knative is built around three core components—Build, Serving, and Eventing—and it is these three that drive the great Serverless ship of Knative forward.

Build​

• In-cluster builds: builds run inside Kubernetes, compiling functions and shipping them quickly, tightly integrated with the whole Kubernetes ecosystem.
• Standardization: it aims to be a general, standardized build component usable as part of larger systems; deployment scripts are standardized and structured, making services easier to migrate and deploy.

Serving​

• Fast deployment: rapidly deploy serverless containers.
• Scale on demand: automatic scaling up and down, including scale-to-zero.
• Routing policies: routing and network programming built on Istio.
• Version snapshots: deployment snapshots (production container snapshots that can be kept long-term and restored to any snapshot version at will).

Eventing​

Source, Channel, Subscription

The eventing system makes producing and consuming events easy: it abstracts event sources and lets operators use the messaging layer of their choice. The heart of Serverless is event-based triggering—when something happens, a particular function fires. The event concept decouples functions from their callers: a deployed function does not care who will call it, and an event source does not care who will handle it. In short, our code no longer names the concrete callee Service; it only concerns itself with emitting events and handling events (anyone familiar with MQ queues will feel at home here, though Eventing's system is a bit more complete, with several event-processing patterns). When our Service makes a remote RPC or HTTP call, it needs neither knowledge of a concrete service instance nor a subscription to a registry—just emit the event source, and Eventing handles all of the returned data. This achieves true transparency between services and solves the coupling problem. (As the saying goes, coupling and decoupling are often just one indirection layer apart.)

Overall Strengths​

Convenience: Knative uses Kubernetes as its underlying framework, so any Kubernetes cluster—online or on-premises, a managed cloud service or self-built—can become a Serverless platform quickly by installing the Knative add-on.

Standardization: together with the CNCF, Knative standardized all events into CloudEvents, giving events cross-platform portability while decoupling functions from their concrete callers.

Inter-service decoupling: with Knative, applications are no longer hard-bound to underlying dependent services and can interoperate across clouds.

A mature ecosystem: Knative is built on the Kubernetes stack and integrates tightly with its ecosystem.

Autoscaling: it monitors application requests and scales automatically, and—thanks to Istio (or ambassador, gloo, etc.)—natively supports blue-green deployments and rollbacks, streamlining the release process.

Application monitoring: log collection, search, and analysis are supported, along with metrics dashboards and call-chain tracing.

Snapshot deployment: every release is recorded as a snapshot that can be retained long-term, and you can seamlessly restore to any snapshot version at any point in time.

Summary​

Serverless has become the new technology hotspot, evolved on top of traditional container technology and the Service Mesh. Serverless cloud functions free developers from server deployment and operations: develop just the core business logic or function and it is live, automatically gaining distributed disaster tolerance and load-based autoscaling. On public clouds you are billed by actual invocation count, execution duration, and compute consumed—no wasted resources, and better cost savings for the business.

Serverless broadly falls into two categories: BaaS (Backend as a Service) and FaaS (Function as a Service).

BaaS: "serverless" was first used to describe applications that significantly or fully incorporate third-party, cloud-hosted applications and services to manage server-side logic and state. These are typically "rich client" applications—single-page web apps or mobile apps—that use the vast ecosystem of cloud-accessible databases (e.g. Parse, Firebase), authentication services (e.g. Auth0, AWS Cognito), and so on. These kinds of services were previously described as "Backend as a Service"—think of the containers and service middleware we commonly use on the backend: Spring Boot, Tomcat, Dubbo, WebLogic, Gin, Flask.

FaaS: serverless can also mean applications where server-side logic is still written by the application developer. Unlike traditional architectures, it runs in stateless compute containers that are event-triggered—you implement just a function without worrying about the environment—ephemeral (possibly lasting a single invocation), and fully managed by a third party. One way to think of it is "functions as a service," or FaaS. Abroad, AWS Lambda is one of the most popular FaaS platforms; in China, Alibaba Cloud's Function Compute (FC) offers FaaS today.

In short, serverless architecture did not appear to replace traditional applications. Starting from its highly flexible usage model and event-driven nature, it helps us reduce deployment effort, improve scalability, and shrink the burden of maintaining the infrastructure behind the code—giving us more options to pick the deployment approach that best fits the business.

Further Questions​

1. How will deployment architectures evolve? How will the 3H traits—high availability, high performance, high concurrency—spawn new technology stacks? How does an enterprise private cloud reach 6 to 12 nines of availability? (6 nines: (1-99.9999%)*365*24*60*60 = 31 seconds—no more than 31 seconds of downtime per year. But 12 nines is quite a challenge; from a cost perspective it is unnecessary and perhaps impossible: annual downtime must not exceed (1-99.9999999999%)*365*24*60*60 = 0.00003 seconds.) Combined with multi-region active-active architectures, how many new middleware technologies might that spawn?

2. Does this architecture really have only upsides and no downsides? Are its overhead, footprint, and controllability suitable for every company? Is it sound for a company to jump straight to Serverless without considering its business scenarios?

3. Have you noticed that container ecosystems in software and hyper-converged infrastructure in hardware are strikingly similar? Both aim for elasticity, migration, cost savings, flexibility, agility, and greater reliability.

Corrections and additions are welcome—feel free to leave a comment.

Why the Default Keymap Is Worth Learning​

IDEA supports custom keymaps, and you can switch to Eclipse or VS Code style with one click. But I recommend sticking with the defaults, for a simple reason: when you switch machines, hop onto a coworker's computer, or follow official docs and video tutorials, the default keymap works everywhere, and your muscle memory never goes to waste.

The shortcuts below follow the default Windows/Linux keymap. On macOS, swapping Ctrl for Cmd and Alt for Option usually gets you there; the few exceptions are called out separately.

Search and Navigation​

These are the most frequently used—practice them first.

• Double Shift: Search Everywhere—globally search classes, files, symbols, and settings. When in doubt, use this.
• Ctrl + N: find a class by name; Ctrl + Shift + N: find a file; Ctrl + Alt + Shift + N: find a symbol (methods, fields).
• Ctrl + Shift + F: project-wide text search; Ctrl + Shift + R: project-wide replace.
• Ctrl + B (or Ctrl + left click): jump to definition; Ctrl + Alt + B: jump to implementation—more useful than jumping to the definition when reading interfaces.
• Alt + F7: Find Usages—see who calls a method or references a field.
• Ctrl + E: recently opened files; Ctrl + Shift + E: recently edited files.
• Ctrl + G: go to a specific line number—handy when chasing stack traces in logs.

Editing and Code Generation​

• Alt + Enter: context-aware quick fix—importing packages, adding try-catch, creating missing methods all rely on it. Arguably the soul of IDEA.
• Alt + Insert (Cmd + N on macOS): Generate menu—constructors, getters/setters, equals/hashCode, method overrides.
• Ctrl + D: duplicate the current line; Ctrl + Y: delete the current line. Note that Ctrl + Y is redo in many editors but deletes a line in IDEA—easy to hit by mistake at first.
• Ctrl + W / Ctrl + Shift + W: expand/shrink selection by syntax structure—great for selecting an expression or a code block.
• Ctrl + Alt + L: format code; Ctrl + Alt + O: optimize imports.
• Ctrl + /: line comment; Ctrl + Shift + /: block comment.
• Shift + Alt + Up/Down: move a line up or down; Ctrl + Shift + Up/Down: move by syntax block, so statements never leave the method body.

Refactoring​

The value of refactoring shortcuts is safety: IDEA updates every reference in sync, far more reliable than manual search-and-replace.

• Shift + F6: rename—works for classes, methods, and variables alike.
• Ctrl + Alt + M: Extract Method; Ctrl + Alt + V: extract variable.
• Ctrl + Alt + C: extract constant; Ctrl + Alt + F: extract field.
• Ctrl + Alt + Shift + T: Refactor This—pops up every refactoring available at the current position. When you forget the ones above, this is your fallback.

Running and Debugging​

• Shift + F10: run the current configuration; Shift + F9: run with the debugger.
• F8: step over; F7: step into; Shift + F8: step out of the current method.
• F9: resume execution to the next breakpoint.
• Ctrl + F8: toggle a breakpoint on the current line.
• Alt + F8: open Evaluate Expression while debugging to run an ad-hoc expression and inspect the result.

Pitfalls and Caveats​

When a shortcut stops working, don't rush to reinstall—another program has most likely grabbed it:

1. On Windows, resident software like input methods, NetEase Cloud Music, and QQ register global hotkeys, and combos like Ctrl + Alt + L are the most commonly hijacked. Disable the global hotkey in the offending app.

2. Linux desktop environments (GNOME/KDE) ship window-management shortcuts that can also conflict with IDEA; rebind them on the system side in your settings.

3. In IDEA, press Ctrl + Shift + A (Find Action) and type a feature name to see the key currently bound to any action—the universal entry point for troubleshooting conflicts and discovering obscure features.

Wrap-up​

Shortcuts pay compound interest: each use only saves a second or two, but you search, navigate, and refactor hundreds of times a day. The default keymap travels across machines, and with Ctrl + Shift + A available for lookups, there's barely any memorization burden. I'll keep coming back to this list myself whenever I forget one.

CPU-Bound​

Heavy computation, high CPU usage, CPU loading at 90-100%. Aside from the CPU there is some I/O to read/write (disk/memory), but that I/O completes in very little time — most of the work is the CPU crunching large amounts of data and doing math. Think data analytics or stream processing; programs like these typically run at very high CPU utilization.

Suppose you're on a single-core CPU with 6 threads in the pool. Being single-core, only one thread can run at a time, and once you factor in the cost of context switching between threads, it's actually less efficient than a single thread. So on a single-core CPU handling CPU-bound work, skip multithreading altogether.

With a 6-core CPU and 6 threads, throughput can in theory improve 6x (in practice it won't quite get there — threads contend with each other and there's room for optimization). Every thread has a CPU to run on, so no thread waits for a time slice and there's no switching overhead. Multi-core CPUs are the right fit for CPU-bound programs, potentially with no thread context switching at all in between (one core per thread — the usual configuration doesn't exceed CPU count + 1).

IO-Bound​

The opposite of CPU-bound: the work is mostly disk and network IO, CPU usage is low, and most of the time the CPU is waiting on IO responses — threads block while waiting, the CPU sits idle, and the pressure falls on disk or network throughput instead. In this scenario, the crude "CPU cores × 2" rule of thumb isn't really rigorous; IO-bound workloads have a proper formula to apply. Take a typical web application backend: most of the work is CRUD against a database or cache, and much of each endpoint's latency goes to disk and network IO. When sizing the pool's core threads, you can plug the server's allocated CPU resources into the formula.

The formula from Java Concurrency in Practice:

Nthreads = Ncpu × Ucpu × (1 + W/C)

• Ncpu: number of CPU cores
• Ucpu: target CPU utilization
• W/C: wait time / compute time

The formula does yield an expected thread count, but in a real program it's usually very hard to get accurate wait and compute times, because programs are complex — they're not just "computation." A stretch of code mixes memory reads/writes, computation, I/O, and other compound operations, so measuring those two metrics precisely is difficult; sizing threads by formula alone is too idealistic. What we can do is take the formula's result as a baseline and then fine-tune the core thread count through load testing until we hit the best efficiency we can expect.

Mixed​

Some applications compute over data while simultaneously moving it over disk or network. How do you configure thread pools for maximum performance in such programs? The usual approach is to split by the server's CPU core count and create two pools: one for the compute portion, one for the IO portion — a fairly sound configuration. There's also another formula floating around online:

Core threads = (thread wait time / thread CPU time + 1) × CPU cores

Thread Counts in Real Programs​

So in an actual program — say, a typical Java business system — what's the right number of threads (thread pool size)?

The short answer: there is no fixed answer. Set expectations first — target CPU utilization, load, GC frequency, and similar metrics — then use the formula to set an initial core thread count, and keep adjusting through testing until you land on a reasonable number.

The formula only gets you a ballpark figure; real-world interference comes from all directions, especially contention for the server's compute resources. If your application runs directly on a server with nothing else competing, iterating on load tests will find the right thread count. But many applications are containerized nowadays: in a K8S environment, containers are spread across worker nodes, and a single worker node may run many containers. Without resource limits on the containers, compute contention is almost guaranteed. You need to set proper CPU, memory, disk, and network limits for each container, or actual efficiency can end up far below expectations (with other containers eating your compute).

Summary​

There is no universally correct core thread count: CPU-bound workloads should hug the CPU core count, IO-bound workloads can be estimated with Nthreads = Ncpu × Ucpu × (1 + W/C), and mixed workloads are worth splitting into separate compute and IO pools. The formula is only a starting point — wait and compute times are hard to measure precisely in real programs, and containerized environments add resource contention on top. So the sensible approach is: pick an initial value from the formula, define your expected metrics for CPU utilization and load, then load-test and adjust step by step until the numbers meet expectations.

Official BufferPool documentation:

MySQL :: MySQL 8.0 Reference Manual :: 15.5.1 Buffer Pool

Why design a BufferPool at all? Anyone with a computer science background knows memory and disk are worlds apart in speed — from the CPU's perspective, disk transfer and response times are painfully slow. So MySQL designed the BufferPool to mirror the real data on disk, managing and manipulating data entirely in memory, which makes DML execution take off. As for how the data in the BufferPool survives crashes and power loss (physical damage is another story — if the disk itself is toast, nothing can help), I covered that in an earlier post:

The MySQL Execution Pipeline

Buffer Pool Overview​

InnoDB accesses data in units of Pages, each 16KB by default, and the Buffer Pool exists to manage and cache these Pages. InnoDB carves out a contiguous block of memory for the Buffer Pool and splits it into multiple Buffer Pool Instances for better management. Each Instance is the same size, and an algorithm guarantees that a given Page always lands in one specific Instance. This multi-Instance design improves the Buffer Pool's concurrency.

Each Buffer Pool Instance maintains its own set of management structures. InnoDB reads data from disk files into memory in 16KB Pages and caches them via an LRU List: frequently accessed Pages sit near the front of the LRU List, rarely accessed ones near the back. When accessing a Page, InnoDB first looks in the Buffer Pool; on a miss it reads the disk data file, then puts the fetched Page into the LRU List. When an Instance has no free Pages left, Pages on the LRU List get evicted.

The Buffer Pool also contains a fair amount of Page compression logic — compressing the actual 16KB Pages down to 8KB, 4KB, 2KB, or 1KB. We'll skip that here and stick with the default 16KB Pages to trace the main logic.

So the BufferPool is a layered structure:

• The BufferPool consists of multiple Buffer Pool Instances.
• Each Buffer Pool Instance consists of multiple Buffer Chunks (128MB each by default); a Buffer Chunk manages a contiguous memory region called Buffer Chunk Memory.
• Each Buffer Chunk consists of multiple Buffer Pages (Pages for short, 16KB by default).

This layering exists to reduce mutex contention under high concurrency.

This post focuses on the BufferPool's data structures, how it manages pages, and its eviction and flushing strategies. Start with the official LRU data-caching diagram:

The Buffer Pool's default total size is 128M, adjustable via innodb-buffer-pool-size = xxxxxxx as needed (the official recommendation for a dedicated database server is to give it most of the physical memory). It is divided into a New Sublist and an Old Sublist — sound like the JVM's young and old generations? The idea is indeed similar: the New Sublist holds the most active data, the Old Sublist holds infrequently accessed data, and the Old Sublist takes up 3/8 of the BufferPool by default.

Since memory is scarce, you can't load every table on disk into memory — an eviction strategy is required. The BufferPool uses the LRU algorithm plus MySQL's own variant logic to evict in-memory data, also known as hot/cold data handling: frequently accessed data drifts toward the New Sublist, infrequently accessed data toward the Old Sublist, and when memory pressure is high, data in the Old Sublist is evicted first. A newly read page is placed at the Head of the Old Sublist; if it gets accessed often it moves into the New Sublist, otherwise it stays in the Old Sublist.

The BufferPool also contains a region called the ChangeBuffer, as shown:

The ChangeBuffer caches DML changes whose target pages are not in the BufferPool, sparing every modification from first pulling the page off disk. When the corresponding page is later read into memory, the buffered changes get merged in, reducing random IO. By default the ChangeBuffer takes 25% of the BufferPool, adjustable up to a maximum of 50%.

A fairly complete picture of the BufferPool's structure (a few lists are omitted, such as the compression lists):

Before we start, a few prerequisite concepts:

• Data page: a page that has been loaded with data from disk.
• Dirty page: a page that has been modified and needs to be flushed back to disk.
• Blank page: a page holding no data yet.
• Control block: the BufferPool page pointer stored in nodes of the various management lists.
• Buffer page: the general term for any page in the BufferPool (all pages are 16KB).

Configuring Multiple Buffer Pool Instances​

When the MySQL server starts up, it requests the BufferPool's memory from the operating system. Under multithreading, every list requires locking, and when the BufferPool is very large and concurrent access is very heavy, a single BufferPool becomes a throughput bottleneck. So the BufferPool is split into several smaller BufferPools, called Buffer Pool Instances, each independently allocating memory and managing its own lists, so concurrent threads don't interfere with each other. The instance count is configurable:

# Create 2 buffer pool instances
innodb_buffer_pool_instances = 2

# Memory per instance = total size divided by instance count
# innodb_buffer_pool_size / innodb_buffer_pool_instances

Since creating and managing multiple instances carries its own overhead, MySQL stipulates: when innodb_buffer_pool_size is below 1G, there is only one instance by default and setting more has no effect; only above 1G is configuring multiple instances encouraged.

How the Buffer Pool Manages Pages​

Data pages are not scattered haphazardly through the BufferPool — they are organized by a hash table and several lists to support the operations described earlier. The nodes in these lists hold the Page Descriptor:

1. page hash (fast access): all pages are organized by a hash table called the page hash, used to quickly locate a page in the buffer pool; the page hash key is the page id.
2. free list (page allocation): when loading a page into the buffer pool, a buf_block_t must be allocated as its page descriptor. How do you find a free, unoccupied buf_block_t? They all hang on the free list.
3. LRU list (page eviction): buffer pool space is finite; when there is no room for pages loaded from disk, some existing pages must be evicted, using the LRU algorithm.
4. flush list (dirty page writeback): when a page becomes dirty, the page cleaner thread must periodically write it back to disk; all dirty pages live on the flush list.

Managing the Free List​

When the MySQL server first starts, the BufferPool must be initialized: memory for the BufferPool is requested from the OS and divided into pairs of control blocks and buffer pages. At this point no real disk pages have been cached in the BufferPool (nothing has needed them yet); as the program runs, pages from disk get cached in over time. Which raises a question: when reading a page from disk into the BufferPool, which buffer page slot should it go into? Put differently, how do we tell which buffer pages in the BufferPool are free and which are in use?

Best to record somewhere which buffer pages are available — and this is where each buffer page's control block earns its keep: the control blocks of all free buffer pages are linked into a list as nodes, called the Free list. In a freshly initialized BufferPool, every buffer page is free, so every buffer page's control block joins the Free list.

To manage the Free list better, a dedicated base node is defined, containing the list's head node address, tail node address, and the current node count, among other information. Note that the base node's memory is not part of the big contiguous allocation made for the BufferPool — it is a separately allocated chunk of memory.

With the Free list in place, things get easy: whenever a page needs to be loaded from disk into the BufferPool, take a free buffer page from the Free list, fill in its control block's information (the tablespace the page belongs to, the page number, and so on), and remove that control block from the Free list, marking the buffer page as in use. Be clear that what we actually take from the list is a control block — the real page is reached through the control block. Likewise, "iterating over the buffer pages in the BufferPool" really means "iterating over the buffer pages' control blocks."

In short: manage the Free list well and you always know which pages in the BufferPool are free.

Page Hash: Hashing the Buffer Pages​

When data in a page needs to be accessed, the page is loaded from disk into the BufferPool; if it's already there, it can simply be used. But how do we know whether the page is in the BufferPool? Surely not by walking through every buffer page one by one — with that many buffer pages, a full scan would be exhausting.

Think about it: we actually locate a page by tablespace number + page number, so tablespace number + page number acts as a Key, and the buffer page's control block is the corresponding Value. Finding a Value quickly by Key — that's naturally a hash table.

So a hash table is built with tablespace number + page number as the Key and the buffer page control block's address as the Value. When a page's data is needed, first look up the hash table by tablespace number + page number: if a corresponding buffer page exists, use it directly; if not, pick a free buffer page from the Free list and load the corresponding page from disk into that slot.

Managing the LRU List​

The LRU list is a way of managing the BufferPool's cache space. Think about it: if you keep loading data pages into the cache, sooner or later it fills up, and there must be a mechanism to release the pages nobody needs — that mechanism is the LRU (least recently used) algorithm.

You can picture the algorithm as a linked list split into two parts (MySQL's variant): one part stores very frequently used buffer pages — the hot data — called the Young region (New Sublist); the other stores infrequently used buffer pages — the cold data — called the Old region (Old Sublist). InnoDB splits the LRU list proportionally, and you can check the Old region's share via the innodb_old_blocks_pct parameter.

With these two regions, InnoDB's designers could optimize for BufferPool hit-rate scenarios:

• Optimizing for read-ahead pages that may never be accessed again. The rule is that when a page from disk is first loaded into a buffer page, its control block goes to the head of the Old region. That way, pages read ahead into the BufferPool but never subsequently accessed drift out of the Old region without disturbing the frequently used buffer pages in the Young region.
• Optimizing for full table scans that access a burst of low-frequency pages in a short time. During a full table scan, pages loaded for the first time do land at the head of the Old region, but they get accessed again immediately afterward, and each access would move the page to the head of the Young region — still squeezing out the genuinely hot pages. The designers reasoned that during a full table scan, even if a page holds many rows and each row read counts as one page access, the whole process takes very little time. So the rule is: on the first access to a buffer page in the Old region, record the access time in its control block; if subsequent accesses fall within a certain interval of that first access, the page does not move from the Old region to the head of the Young region — only accesses outside the interval move it. This interval is controlled by the system variable innodb_old_blocks_time.

Managing the Flush List​

If you modify the data in a buffer page, it no longer matches the page on disk — such a buffer page is called a dirty page. You could of course flush it to the corresponding disk page immediately after every modification, but writing to disk that frequently would kill performance. So each modification does not trigger an immediate flush; the page is flushed at some point in the future instead.

But if you don't flush immediately, how do you later know which pages in the BufferPool are dirty and which have never been modified? Hence another list has to be created to hold dirty pages: the control block of every modified buffer page joins this list as a node. Since all these buffer pages need to be flushed to disk, it is called the Flush list. Its construction is much like the Free list. Also note: a free buffer page is definitely not dirty, and a dirty page is definitely not free — meaning a given buffer page's control block cannot be a node of both the Free list and the Flush list at once; it belongs to at most one of them.

In short: the Free list is simply the list of all free pages in the cache, while the Flush list is the list of all modified pages (dirty pages).

The Dirty Page Flushing Mechanism​

Dirty pages are flushed back to disk in the following situations:

1. The Redo Log is full. As the durability guarantee, the Redo Log must record every update. If it runs out of space, pages must be flushed to sync data to disk, after which the Redo Log's CheckPoint pointer moves forward to free up space. If memory is not under pressure at that point, the flushed pages don't need to be evicted.

2. Memory can't hold any more data pages. Queries read data pages from disk into memory before returning; if memory runs out, the least recently used pages must be evicted, and if an evicted page is dirty, it must be flushed to disk first.

3. The database is idle. When the database considers itself idle, it flushes dirty pages; if memory is not under pressure, flushed pages are not evicted.

4. The database shuts down normally. On shutdown, pages are flushed as well, persisting prior modifications to disk.

Wrapping Up​

The BufferPool is InnoDB's core design for trading memory for IO: data is cached in memory in 16KB Pages, and DML operations modify memory first, then land on disk asynchronously. To reduce lock contention under concurrency it is split into multiple Instances; internally, the Page Hash provides fast lookup, while the Free, LRU, and Flush lists handle free page allocation, hot/cold data eviction, and dirty page writeback respectively. The LRU's New/Old partitioning plus the innodb_old_blocks_time window solves the problem of read-ahead and full table scans washing out hot data. Once you understand how these lists cooperate, the BufferPool's overall machinery becomes clear.

From Connector to Executor​

When a client submits a SQL statement to the MySQL server, a connection must be established first. The connector handles the client connection and verifies the user's identity and privileges. If the check passes, the SQL parser performs semantic analysis on the statement (checking whether the SQL is valid and can be parsed). After parsing, the execution plan is optimized — for example, stripping useless conditions like 1=1, and choosing the best index and ordering of WHERE clause columns.

The statement is then handed to the executor, which runs it against whatever storage engine the database uses. MySQL was designed as two layers: the Server layer and the storage engine layer. The benefit is decoupling — you can pick the storage engine that fits each scenario.

Buffer Pool​

InnoDB, the most commonly used storage engine, manages storage in units of pages (16KB). Every insert, delete, update, or query ultimately operates on a whole page: the entire page is loaded into the Buffer Pool, and all data operations happen inside the Buffer Pool. The Buffer Pool is typically configured to use around 70-80% of the server's memory, so data operations run directly in memory, which dramatically improves throughput. (If every DML statement required disk IO, the database's disk would quickly become the bottleneck — modifying one row would mean reading from disk and then flushing back, i.e. 2 IOs, one read and one write.)

An example makes this concrete. Assume there is no Buffer Pool, the user table has a single row with age = 1, and we need to run three statements:

Transaction A: update user set age = 2 (one read + one write, 2 IOs) Transaction B: update user set age = 3 (one read + one write, 2 IOs) Transaction C: update user set age = 4 (one read + one write, 2 IOs)

Each one has to read the data from disk into memory, modify it, then flush it back to disk — 6 IOs in total.

Now with a Buffer Pool:

Transaction A: update user set age = 2 (read into the Buffer Pool and modify, 1 read IO) Transaction B: update user set age = 3 (modify the data in the Buffer Pool, 0 IOs) Transaction C: update user set age = 4 (modify the data in the Buffer Pool and flush to disk, 1 write IO)

Only the first statement needs to load the data page; subsequent operations happen entirely in memory. With the Buffer Pool, the total drops to 2 IOs. (This is a simplified illustration — the real optimization logic isn't quite this simple — but the big picture holds: the Buffer Pool maps data from disk into memory for manipulation, reducing disk IO. The catch is that modifications happen in memory, so a crash or power loss could easily lose data.)

Inspecting Buffer Pool information:

-- The output includes Buffer Pool size, hit rate, dirty page count, and more
SHOW ENGINE INNODB STATUS

This is where several crucial logs come in to help: the Redo Log, the Bin Log (binary log), and the Undo Log (rollback log).

Redo Log​

After data is modified, the new value is first recorded in the Redo Log on disk. Even if power is suddenly cut and everything in the Buffer Pool is lost, the Buffer Pool can be rebuilt from the Redo Log when power comes back. This keeps the Buffer Pool's in-memory efficiency while guaranteeing no data loss. The Redo Log has three flush strategies:

• Set to 0: no flush on transaction commit (by default the Master Thread syncs the redo log once per second)
• Set to 1: a synchronous flush on every transaction commit — the safest option, because if a crash happens at that point, the transaction never committed successfully, so there is nothing to recover (the default)
• Set to 2: on every commit, the Redo Log Buffer contents are only written to the Page Cache without syncing; the filesystem (OS) decides when to sync to disk

Redo Log flushing here is sequential-write WAL (Write-ahead logging). Compared with random disk writes, this is vastly more efficient — message queues like Kafka and RocketMQ also use sequential log writes to boost disk write throughput. Sequential disk writes are already fast, but still nowhere near memory speed. To push Redo Log efficiency further, InnoDB introduced the Change Buffer in memory (16MB by default, located inside the Buffer Pool, with a configurable percentage) to record changes in memory, only performing the sequential disk write when the transaction commits.

The Change Buffer is divided into many Blocks of 512KB each, and all the Redo Log produced by one transaction is called a Group.

Bin Log (Binary Log)​

The Bin Log is a binary log recording all table schema changes (CREATE, ALTER TABLE, ...) and table data modifications (INSERT, UPDATE, DELETE, ...). It does not record operations like SELECT and SHOW, since those don't modify data — though you can check the general query log to see every statement MySQL has executed.

One thing to note: even an update that changes nothing still gets written to the Bin Log.

Like the Redo Log, the Bin Log has its own flush strategy, controlled by the sync_binlog parameter:

• Set to 0: before each commit, the Bin Log is written to the OS Cache, and the operating system decides when to flush it to disk
• Set to 1: the Bin Log is written to disk synchronously, bypassing the OS Cache
• Set to n: after every n transaction commits, Fsync is called once to force the Bin Log in the OS Cache to disk

The Bin Log has two common use cases:

• Replication: MySQL Replication enables the Bin Log on the Master, which ships its binary log to Slaves to keep Master and Slave data consistent.
• Data recovery: restoring data via the mysqlbinlog tool.

Which raises a question: both the Bin Log and the Redo Log record post-modification values — what's the difference? If we have the Redo Log, why do we still need the Bin Log?

Bin Log vs. Redo Log​

• The Bin Log belongs to the MySQL Server layer; the Redo Log belongs to the Engine layer
• The Bin Log works with all engines; the Redo Log is InnoDB-specific
• The Bin Log records logical operations; the Redo Log records the updated content
• The Bin Log is append-only across multiple files; the Redo Log is a fixed-size set of files written in a circular fashion
• During transaction execution, operations are continuously written to the Redo Log; the Bin Log is only written at commit time

Undo Log (Rollback Log)​

When data is modified, alongside the Redo Log a corresponding Undo Log is also recorded. If the transaction fails or is rolled back for any reason, the Undo Log makes the rollback possible. Undo Logs are recorded in segments — each Undo Log operation occupies one Undo Log Segment. Their purpose: preserving a version of the data as it was before the transaction, usable both for rollback and for reads under multi-version concurrency control. Note that the Undo Log lives in the global tablespace by default — you can loosely think of the Undo Log as being stored in a MySQL table of its own, so inserting an Undo Log entry is similar to inserting an ordinary row. Which means that writing the Undo Log itself also involves writing the Redo Log.

The Execution Pipeline​

With the groundwork above in place, here is the full execution pipeline of a SQL statement:

0. We skip the connector, parser, and optimizer steps here (described above).

1. First, the Undo Log records the rollback information for this transaction (transaction ID, rollback pointer), enabling rollback and the MVCC transaction isolation that depends on it.

2. Next, the data in the Buffer Pool is modified, and the change information is simultaneously added to the Change Buffer. If the target data page is not yet in memory, then — provided data consistency is not affected — InnoDB caches these updates in the Change Buffer first, avoiding the need to read the page from disk. The next time a query needs that page, the page is read into memory and the Change Buffer operations related to it are applied. (In some cases, though, the page must be read into memory before the operation can proceed.)

3. After the change information lands in the Change Buffer, it must also be flushed to the Redo Log for durable storage, so a crash cannot lose the operations. (Flushing here only happens after the transaction commits — flushing an uncommitted transaction is pointless. MySQL defaults to synchronous Redo Log flushing; if you switch to asynchronous flushing, a sudden crash or power loss will lose data, including everything recorded in the in-memory Change Buffer.)

4. The concrete SQL operation is also added to the Bin Log Cache — but the Bin Log Cache records only logical operation information (the actual DML, DDL, and DQL statements executed), not data change details. The Bin Log Cache is a per-thread private memory area (32K by default) sized to hold the Bin Log Events generated by an entire transaction; a large transaction can easily exceed this setting, in which case a temporary Bin Log file is used for storage.

5. When the client commits the transaction, two-phase commit begins. First the Redo Log records are flushed to disk and marked with the prepare state; then the Bin Log records are flushed, and once the Bin Log flush succeeds, the corresponding Redo Log state is changed from prepare to commit. The two logs must be kept consistent — if they diverge, you get master-slave inconsistencies and all sorts of problems. At this point the client can be told the commit succeeded, but the data (dirty pages) has not yet hit disk; it still lives in the Buffer Pool. Only the corresponding log records have fully succeeded.

6. The MySQL server asynchronously flushes the dirty pages in the Buffer Pool to disk (at the right flushing moments). Only once the dirty pages are written to disk is the data truly persisted. If a crash or power loss happens during this asynchronous flushing, InnoDB can replay the Redo Log on startup to rebuild the previous Buffer Pool state.

The Right Moments to Flush Pages​

1. The Redo Log is full. As the durability guarantee, the Redo Log must record every update. If it runs out of space, pages must be flushed to sync data to disk, after which the Redo Log's CheckPoint pointer moves forward to free up space. If memory is not under pressure at that point, the flushed pages don't need to be evicted.

2. Memory can't hold any more data pages. Queries read data pages from disk into memory before returning results. If memory runs out, the least recently used pages must be evicted — and if an evicted page is dirty, it must be flushed to disk before eviction.

3. The database is idle. When the database considers itself idle, it flushes dirty pages; if memory is not under pressure, flushed pages are not evicted.

4. The database shuts down normally. On shutdown, pages are flushed as well, persisting prior modifications to disk.

Wrapping Up​

MySQL improves durability performance through sequential log writes, and the Buffer Pool maps disk data into memory for DML operations, dramatically boosting throughput. Asynchronous flushing avoids the poor performance of random disk writes caused by modifications scattered across different tables and rows. The Redo Log guarantees recovery after a crash, the Bin Log powers replication and data recovery, and the Undo Log handles rollback and MVCC — together with two-phase commit, the three logs strike the balance between performance and data consistency.

In the 1960s, the U.S. Department of Defense's Advanced Research Projects Agency (ARPA) built ARPANET, widely regarded as the origin of the internet. In the 1970s, drawing on their experience with ARPANET, researchers invented the famous TCP/IP protocol suite. With its clean layered design and stable performance, it made its way into the UNIX kernel in the mid-1980s, bringing many more computers onto the network.

In 1989, Dr. Tim Berners-Lee published a paper proposing a system of hyperlinked documents built on top of the internet. In it he established three key technologies: URI, HTML, and HTTP.

With these three technologies, a hypertext system could run beautifully on the internet, and Berners-Lee named it the "World Wide Web". HTTP began to spread across the internet — today the most common versions are HTTP/1.1 and HTTP/2.0.

Every generation of HTTP has been an optimization built on the shortcomings of the one before it.

HTTP/0.9​

HTTP (HyperText Transfer Protocol) was officially born in 1991, at version 0.9. As the name suggests, its job was to transfer hypertext content — HTML. The protocol defined a communication model where the client sends a request and the server responds. The request message was a single line: GET plus the path of the requested file. With no way to distinguish message types, it could only transfer text.

HTTP/1.0​

As the internet grew and browsers appeared, plain text was no longer enough. Browsers wanted HTTP to carry scripts, stylesheets, images, audio, video — files of all kinds. So the 1.0 version, released in 1996, introduced the following:

• New methods such as HEAD and POST
• Response status codes to indicate possible error causes
• The concept of a protocol version number
• The concept of HTTP headers, making request and response handling much more flexible
• Transferred data no longer limited to text

The most fundamental change was the header mechanism, with header content expressed as key-value pairs. The request headers use the Accept field to tell the server which file types the client can receive, and the response headers use Content-Type to tell the browser what kind of file is being returned. Header fields solved more than content-type negotiation — they enabled caching, authentication, and many other features.

HTTP/1.1​

With the internet's rapid growth, HTTP/1.0 soon fell short too, and the core issue was connections. Every HTTP/1.0 exchange had to go through three phases: establish the connection, transfer the data, and tear down the connection. When a page referenced many external files, all that connecting and disconnecting added substantial network overhead.

HTTP/1.1 made Connection: keep-alive the default, avoiding the cost of repeatedly setting up and tearing down connections. But the server had to return results strictly in the order the client sent its requests, so the client could match each response to its request — using header fields to determine whether the current response had been fully received, with no two responses allowed in flight at once.

To address HTTP/1.0's problems, HTTP/1.1, released in 1999, brought the following:

• Persistent connections (Connection: keep-alive): TCP connection reuse — a TCP connection stays open by default and can be reused across multiple requests
• Concurrent connections: multiple persistent connections allowed per domain (easing the head-of-line blocking inherent to persistent connections)
• Pipelining, letting one TCP connection send multiple requests at once (responses must come back in request order, so it saw little use)
• New methods: PUT, DELETE, OPTIONS, PATCH
• New caching fields (Cache-Control, ETag)
• The Range request header, enabling resumable downloads
• Chunked responses, useful for transferring large files
• A mandatory Host header, making virtual hosting on the internet possible

HTTP/2.0​

Persistent connections in HTTP/1.1 eliminated much of the cost of creating and tearing down connections, but its concurrency remained limited in two ways:

• With a persistent connection, HTTP/1.1 can only process one request at a time per connection. Until the current request finishes, all other requests are stuck waiting — the situation known as head-of-line blocking
• To ease server load, browsers cap the number of HTTP connections per domain, typically at 6 to 8

HTTP/2, officially released in 2015, dropped ASCII-encoded transmission by default in favor of binary data for better efficiency.

When sending requests, the client packages each request's content into numbered binary frames and sends them all to the server together. On receipt, the server reassembles frames with the same number into the complete request. The same framing and reassembly applies in the other direction, when the server returns results to the client.

With binary framing, a client needs only a single connection per domain to handle all its communication. Sending multiple requests over one connection this way is called multiplexing, and each of these lanes is called a stream.

The main changes in HTTP/2.0:

• Data transmitted as a binary protocol rather than plain text
• Multiplexing, replacing 1.1's pipelining
• Header compression with a dedicated algorithm, reducing data volume
• Frame prioritization, letting the server handle certain requests first
• Server push, allowing the server to proactively send data to the client
• All header fields lowercase; pseudo-headers introduced, appearing before regular header fields and starting with a colon
• Stronger security, with encrypted communication (TLS) required "in practice"

Although HTTP/2.0 has been out for years, HTTP/1.1 has proven so entrenched and dominant that HTTP/2.0 adoption remains modest — plenty of websites still run HTTP/1.1.

HTTP/3.0​

Of course, HTTP/2 isn't perfect either. If packets are lost during communication, or either side's network drops, the entire TCP connection stalls.

Because HTTP/2 multiplexes via binary framing, it typically uses just one TCP connection — so when packets are lost or the network hiccups, everything behind the loss gets blocked. HTTP/1.1, by contrast, can open multiple TCP connections; if one fails, the others carry on transferring data unaffected. In this scenario HTTP/2 actually performs worse than HTTP/1.

In 2018, HTTP/3 swapped the underlying TCP for UDP, eliminating the problem at its root. UDP's biggest difference from TCP is that it needs no connection setup to send data and can fire off multiple packets simultaneously — very efficient, but with no acknowledgment mechanism to guarantee delivery. So Google started from scratch and built the UDP-based QUIC protocol, which HTTP/3 adopted. Many browsers and hardware vendors now support HTTP/3.0.

The QUIC Protocol​

Baidu Baike: QUIC_百度百科 (baidu.com)

QUIC sounds like "quick", and that's precisely the point — fast. It's a UDP-based transport protocol proposed by Google, hence its other name: "Quick UDP Internet Connections". QUIC's defining trait is speed. So why is it fast, and where exactly does the speed come from?

1) Faster Connection Establishment​

As we know, HTTP uses TCP at the transport layer, and HTTPS and HTTP/2.0 add TLS for encryption on top — which means handshake-induced connection latency: the TCP three-way handshake (one RTT) plus the TLS handshake (two RTTs), as shown below.

For short-lived connections this handshake latency hurts, and it can't be eliminated. RTT is, after all, humanity's eternal battle with efficiency.

QUIC's handshake is faster because it uses UDP as its transport, cutting out the three-way-handshake delay. Its encryption uses the newest version of TLS, TLS 1.3. Compared with TLS 1.1–1.2, TLS 1.3 lets the client start sending application data without waiting for the TLS handshake to complete, supporting 1-RTT and even 0-RTT — achieving fast connection establishment.

2) No Head-of-Line Blocking​

As noted earlier, HTTP/2.0 solved head-of-line blocking at the application layer, but since its connections still ride on TCP, blocking at the transport layer remained.