Largest Triangle Three Buckets: Downsampling Time-Series Data Without Losing Signal

2026-01-23T00:00:00+00:00

The Core Problem
What Is LTTB?
The Mathematical Insight
Why Use LTTB?</a> </h2> 1. Visual Fidelity Over Statistical Precision</a> </h3> LTTB optimizes for what humans see, not statistical accuracy. When you're rendering a chart, you don't need every data point - you need the shape. LTTB preserves: Peaks and valleys: Extreme values stay visible</li> Trend changes: Slope transitions remain clear</li> Visual density: High-activity regions get more points</li> </ul> 2. Predictable Performance</a> </h3> The algorithm is O(n) - it makes exactly one pass through your data. No sorting, no complicated data structures. For 1 million points downsampled to 1,000: Time complexity: O(n) - linear scan</li> Space complexity: O(1) extra - processes in place (with output buffer)</li> Processing time: ~10-50ms on modern hardware</li> </ul> 3. Deterministic Results</a> </h3> Unlike sampling with randomness, LTTB always produces the same output for the same input. This matters for: Reproducibility: Charts look identical across page reloads</li> Debugging: Issues are consistent and trackable</li> Caching: You can cache downsampled results reliably</li> </ul> The LTTB Algorithm: Step by Step</a> </h2> Let me break down how it works before we code it. Setup: You have N source points</li> You want M target points (M < N)</li> First and last points are always included (preserves range)</li> </ol> Process: Divide remaining N-2 points into M-2 buckets</li> Always include the first point</li> For each bucket: Look at the previously selected point (from bucket i-1)</li> Look at the average point of the next bucket (bucket i+1)</li> Select the point in current bucket (bucket i) that forms the largest triangle with these two points</li> </ul> </li> Always include the last point</li> </ol> Visual representation: Bucket 1 Bucket 2 Bucket 3 Bucket 4 [...] [...] [...] [...] ↓ ↓ ↓ ↓ Point A Point B Point C Point D For Bucket 2: - Previous selected: Point A (from Bucket 1) - Next bucket average: avg(Bucket 3 points) - Select point from Bucket 2 that makes largest triangle </code></pre> TypeScript Implementation</a> </h2> Here's a production-ready implementation: interface Point { x: number; y: number; } /** * Largest Triangle Three Buckets downsampling algorithm * * @param data - Source data points (must be sorted by x) * @param threshold - Target number of points in output * @returns Downsampled points that preserve visual shape / function largestTriangleThreeBuckets( data: Point[], threshold: number ): Point[] { const dataLength = data.length; // If threshold is greater than or equal to data length, return all data if (threshold >= dataLength) { return data; } // If threshold is very small, return edge points if (threshold <= 2) { return [data[0], data[dataLength - 1]]; } const sampled: Point[] = []; // Always include the first point sampled.push(data[0]); // Bucket size. Leave room for start and endpoints const bucketSize = (dataLength - 2) / (threshold - 2); // Index of currently selected point in previous bucket let a = 0; for (let i = 0; i < threshold - 2; i++) { // Calculate point average for next bucket (used as point C in triangle) let avgX = 0; let avgY = 0; const avgRangeStart = Math.floor((i + 1) bucketSize) + 1; const avgRangeEnd = Math.min( Math.floor((i + 2) bucketSize) + 1, dataLength ); const avgRangeLength = avgRangeEnd - avgRangeStart; for (let j = avgRangeStart; j < avgRangeEnd; j++) { avgX += data[j].x; avgY += data[j].y; } avgX /= avgRangeLength; avgY /= avgRangeLength; // Get the range for this bucket const rangeStart = Math.floor(i bucketSize) + 1; const rangeEnd = Math.floor((i + 1) bucketSize) + 1; // Point a is the previously selected point const pointA = data[a]; let maxArea = -1; let maxAreaPoint = 0; // Find point in current bucket that forms largest triangle for (let j = rangeStart; j < rangeEnd; j++) { // Calculate triangle area using the cross product formula // area = |x₁(y₂ - y₃) + x₂(y₃ - y₁) + x₃(y₁ - y₂)| / 2 const area = Math.abs( (pointA.x - avgX) (data[j].y - pointA.y) - (pointA.x - data[j].x) (avgY - pointA.y) ) 0.5; if (area > maxArea) { maxArea = area; maxAreaPoint = j; } } sampled.push(data[maxAreaPoint]); a = maxAreaPoint; // This point is the next "point A" } // Always include the last point sampled.push(data[dataLength - 1]); return sampled; } </code></pre> Example Usage</a> </h3> // Generate sample time-series data: a sine wave with noise function generateTimeSeriesData(count: number): Point[] { const data: Point[] = []; for (let i = 0; i < count; i++) { const x = i; const y = Math.sin(i / 100) 50 + Math.random() 10; data.push({ x, y }); } return data; } // Original data: 10,000 points const originalData = generateTimeSeriesData(10000); console.log(`Original: ${originalData.length} points`); // Downsample to 500 points const downsampled = largestTriangleThreeBuckets(originalData, 500); console.log(`Downsampled: ${downsampled.length} points`); console.log(`Reduction: ${((1 - downsampled.length / originalData.length) 100).toFixed(1)}%`); // Use in your charting library // chart.setData(downsampled); // Much faster rendering! </code></pre> Real-World Example: Stock Price Visualization</a> </h2> Let's say you're building a stock charting application. You fetch intraday data - one price per minute for a year: interface StockPrice { timestamp: number; // Unix timestamp price: number; volume: number; } function downsampleStockData( prices: StockPrice[], targetPoints: number ): StockPrice[] { // Convert to Point format const points: Point[] = prices.map(p => ({ x: p.timestamp, y: p.price })); // Apply LTTB const downsampled = largestTriangleThreeBuckets(points, targetPoints); // Map back to original indices to preserve volume data const downsampledPrices: StockPrice[] = []; let priceIndex = 0; for (const point of downsampled) { // Find original data point while (priceIndex < prices.length && prices[priceIndex].timestamp !== point.x) { priceIndex++; } if (priceIndex < prices.length) { downsampledPrices.push(prices[priceIndex]); } } return downsampledPrices; } // Usage: 1 year of minute data (525,600 points) down to 2,000 const yearData = fetchStockPrices('AAPL', '1y', '1m'); // 525,600 points const chartData = downsampleStockData(yearData, 2000); // 99.6% reduction // Chart renders instantly, all major price movements visible </code></pre> Where to Use LTTB</a> </h2> 1. Time-Series Visualization</a> </h3> Perfect fit: Financial charts (stocks, crypto, forex)</li> Server metrics (CPU, memory, network)</li> IoT sensor data (temperature, pressure, vibration)</li> Analytics dashboards (user activity, sales trends)</li> </ul> Why it works: Time-series data is dense and continuous. LTTB preserves the temporal patterns humans need to spot trends and anomalies. 2. Real-Time Monitoring Dashboards</a> </h3> When streaming live metrics: class MetricsBuffer { private buffer: Point[] = []; private maxBufferSize = 100000; private displayThreshold = 1000; add(point: Point): void { this.buffer.push(point); // Downsample when buffer gets large if (this.buffer.length > this.maxBufferSize) { this.buffer = largestTriangleThreeBuckets( this.buffer, this.maxBufferSize / 2 ); } } getDisplayData(): Point[] { if (this.buffer.length <= this.displayThreshold) { return this.buffer; } return largestTriangleThreeBuckets(this.buffer, this.displayThreshold); } } </code></pre> 3. Historical Data Analysis</a> </h3> When users zoom out to see long time ranges: function adaptiveDownsample( data: Point[], viewportWidth: number ): Point[] { // One point per 2-3 pixels is optimal for line charts const targetPoints = Math.floor(viewportWidth / 2); if (data.length <= targetPoints 2) { return data; } return largestTriangleThreeBuckets(data, targetPoints); } // When user zooms in/out, re-downsample for current view chart.on('zoom', (viewport) => { const visible = getDataInViewport(allData, viewport); const downsampled = adaptiveDownsample(visible, viewport.width); chart.update(downsampled); }); </code></pre> 4. Data Export/Transfer</a> </h3> Reduce payload size for client applications: // API endpoint that returns downsampled data app.get('/api/metrics/:id', async (req, res) => { const { points = 1000 } = req.query; const fullData = await db.getMetrics(req.params.id); // Downsample before sending const optimized = largestTriangleThreeBuckets( fullData, Math.min(Number(points), 5000) // Cap at 5k ); res.json({ original_count: fullData.length, downsampled_count: optimized.length, data: optimized }); }); </code></pre> Where NOT to Use LTTB</a> </h2> 1. Statistical Analysis</a> </h3> Don't use for: Calculating averages, medians, or percentiles</li> Standard deviation or variance</li> Correlation analysis</li> Any statistical computation</li> </ul> Why: LTTB optimizes for visual accuracy, not statistical properties. It intentionally biases toward extremes and changes, which skews statistics. // BAD: Statistical analysis on downsampled data const downsampled = largestTriangleThreeBuckets(data, 1000); const average = downsampled.reduce((s, p) => s + p.y, 0) / downsampled.length; // ❌ This average is WRONG - biased toward peaks/valleys // GOOD: Calculate stats on full data const average = data.reduce((s, p) => s + p.y, 0) / data.length; // ✅ Correct statistical average </code></pre> 2. Scientific Precision Requirements</a> </h3> Don't use for: Medical device data where every reading matters</li> Financial audit trails (compliance requires complete data)</li> Scientific experiments requiring exact measurements</li> Legal or regulatory data (must be complete and unaltered)</li> </ul> Reason: Downsampling discards information. If you need provable accuracy or regulatory compliance, you can't afford to lose any data points. 3. Sparse or Non-Continuous Data</a> </h3> Don't use for: Event logs (sparse, discrete events)</li> Transaction records (each record is unique)</li> Categorical data (non-numeric or discrete categories)</li> Already small datasets (< 1000 points)</li> </ul> Example of bad fit: // BAD: Transaction log data const transactions = [ { time: 100, type: 'purchase', amount: 50 }, { time: 500, type: 'refund', amount: 20 }, { time: 1000, type: 'purchase', amount: 100 } ]; // Each transaction is important - can't downsample without losing critical info // BAD: Sparse sensor data with gaps const sensorData = [ { time: 0, value: 10 }, { time: 1000, value: 12 }, // 1000 time units gap { time: 1001, value: 50 } // Sudden spike ]; // The algorithm assumes continuous data - gaps break this assumption </code></pre> 4. When You Need Exact Points</a> </h3> Don't use for: Min/Max calculations (unless you verify endpoints)</li> Exact zero-crossing detection</li> Precise integration or area-under-curve</li> Anomaly detection algorithms</li> </ul> LTTB may miss the exact maximum if it falls inside a bucket but doesn't form a large triangle: // Potential issue: Missing exact peak const data = [ { x: 0, y: 10 }, { x: 1, y: 11 }, { x: 2, y: 100 }, // Peak { x: 3, y: 11 }, { x: 4, y: 10 } ]; const downsampled = largestTriangleThreeBuckets(data, 3); // Might skip x=2 if it doesn't form largest triangle // Use max aggregation instead for guaranteed peak capture </code></pre> Trade-offs and Limitations</a> </h2> Memory vs. Visual Accuracy</a> </h3> LTTB gives you a knob: the threshold parameter. Higher threshold = more points = better visual accuracy but slower rendering and more memory. Sweet spots by use case: Live monitoring: 500-1,000 points (updates every second)</li> Historical analysis: 1,000-2,000 points (user-triggered)</li> Export/reporting: 2,000-5,000 points (one-time generation)</li> </ul> Processing Cost</a> </h3> The O(n) scan isn't free: // Benchmark on MacBook Pro M1 const sizes = [10_000, 100_000, 1_000_000, 10_000_000]; sizes.forEach(size => { const data = generateTimeSeriesData(size); const start = performance.now(); largestTriangleThreeBuckets(data, 1000); const duration = performance.now() - start; console.log(`${size.toLocaleString()} points: ${duration.toFixed(2)}ms`); }); // Results: // 10,000 points: 2ms // 100,000 points: 15ms // 1,000,000 points: 150ms // 10,000,000 points: 1,500ms </code></pre> For 10M points, 1.5 seconds is significant. Solutions: Pre-compute: Downsample on the backend, cache results</li> Web Workers: Offload processing to background thread</li> Progressive loading: Start with coarse downsample, refine on idle</li> </ul> // Web Worker approach // main.ts const worker = new Worker('lttb-worker.ts'); worker.postMessage({ data: largeDataset, threshold: 1000 }); worker.onmessage = (e) => { chart.setData(e.data.downsampled); }; // lttb-worker.ts self.onmessage = (e) => { const { data, threshold } = e.data; const downsampled = largestTriangleThreeBuckets(data, threshold); self.postMessage({ downsampled }); }; </code></pre> Loss of Statistical Properties</a> </h3> This is the big one. Downsampled data looks right but calculates wrong: Metric</th> Full Data</th> LTTB (1000pts)</th> Error</th></tr></thead> Mean</td> 50.0</td> 51.2</td> +2.4%</td></tr> Std Dev</td> 10.5</td> 12.8</td> +21.9%</td></tr> Min</td> 25.0</td> 25.0</td> 0%</td></tr> Max</td> 75.0</td> 75.0</td> 0%</td></tr> </tbody></table> Why: LTTB favors extremes. Standard deviation inflates because you're keeping more outliers relative to the mean. Solution: Dual-path approach: interface DatasetView { visual: Point[]; // LTTB downsampled for charts stats: Point[]; // Full data or aggregated bins for calculations } function prepareDataset(raw: Point[], threshold: number): DatasetView { return { visual: largestTriangleThreeBuckets(raw, threshold), stats: raw // Keep full data for accurate calculations }; } // Use visual for rendering, stats for calculations const dataset = prepareDataset(rawData, 1000); chart.render(dataset.visual); const average = calculateMean(dataset.stats); </code></pre> Advanced Patterns</a> </h2> Multi-Resolution Storage</a> </h3> Store multiple resolutions for different zoom levels: interface MultiResData { full: Point[]; res_10k: Point[]; res_1k: Point[]; res_100: Point[]; } function createMultiResolution(data: Point[]): MultiResData { return { full: data, res_10k: data.length > 10_000 ? largestTriangleThreeBuckets(data, 10_000) : data, res_1k: largestTriangleThreeBuckets(data, 1_000), res_100: largestTriangleThreeBuckets(data, 100) }; } function selectResolution( multi: MultiResData, visiblePoints: number ): Point[] { if (visiblePoints > 5_000) return multi.full; if (visiblePoints > 500) return multi.res_10k; if (visiblePoints > 50) return multi.res_1k; return multi.res_100; } </code></pre> Streaming Incremental Updates</a> </h3> For real-time data, you can't reprocess everything on each new point: class StreamingLTTB { private buffer: Point[] = []; private downsampled: Point[] = []; private windowSize = 10_000; private targetSize = 1_000; add(point: Point): void { this.buffer.push(point); // Sliding window approach if (this.buffer.length > this.windowSize) { this.buffer.shift(); // Remove oldest this.recompute(); } else if (this.buffer.length % 100 === 0) { // Recompute every 100 points this.recompute(); } } private recompute(): void { this.downsampled = largestTriangleThreeBuckets( this.buffer, this.targetSize ); } getView(): Point[] { return this.downsampled; } } </code></pre> Comparison with Other Downsampling Methods</a> </h2> Method</th> Pros</th> Cons</th> Use Case</th></tr></thead> Every Nth</td> Simple, O(1)</td> Misses peaks/valleys</td> Never (use LTTB instead)</td></tr> Random Sampling</td> Unbiased stats</td> Unpredictable visual</td> Statistical analysis</td></tr> Min-Max</td> Preserves extremes</td> Loses smooth curves</td> Range-bound displays</td></tr> LTTB</td> Visual fidelity, O(n)</td> Stats biased</td> Time-series charts</td></tr> M4</td> Very fast, extremes</td> Complex, requires binning</td> High-frequency data</td></tr> </tbody></table> When to choose LTTB: You're rendering line charts</li> Visual shape matters more than exact statistics</li> Data is time-ordered and continuous</li> You need consistent, reproducible results</li> </ul> Production Checklist</a> </h2> Before deploying LTTB in production: Verify input data is sorted by x-axis (time)</li> Handle edge cases: empty arrays, single points, threshold >= data length</li> Add input validation: non-null points, valid numbers</li> Consider pre-computing and caching downsampled views</li> Use Web Workers for large datasets (> 100k points)</li> Keep full data for statistical calculations</li> Document that downsampled data is for visualization only</li> Add metrics to monitor processing time</li> Test with real production data shapes (spikes, flat lines, noise)</li> </ul> Conclusion</a> </h2> LTTB solves a specific problem elegantly: making large time-series datasets renderable without losing visual meaning. It's not a general-purpose data compression algorithm - it's a visualization optimization. The algorithm makes a clear trade: statistical accuracy for visual fidelity. If you're rendering charts, this trade is almost always worth it. Your users see the same patterns in 1,000 points that exist in 1,000,000 - but their browser doesn't crash. Use it when you're drawing lines on screens. Don't use it when you're doing math. Keep these separate, and LTTB becomes a powerful tool in your performance optimization toolkit. The implementation is straightforward - about 80 lines of TypeScript. The impact is immediate: charts that were unusable become instant. That's the mark of a good algorithm - simple idea, dramatic results. The Go Frontend Dilemma 2026: Templ, React, and the Return to the Server 2026-01-21T00:00:00+00:00 For the last decade, the standard answer to "How do I build a UI with my Go backend?" was "You don't." You wrote a REST or gRPC API in Go, and you handed the keys to a Node.js team building a React Single Page Application (SPA). Architecture is cyclical. In 2026, the pendulum has swung firmly back to the server. The complexity of managing distributed state between a client (React) and a server (Go) has become a burden that simple "API-first" designs ignore. This post analyzes the two dominant architectural patterns for Go engineers today: the "Pure Go" Component Stack (enabled by tools like Templ) versus the "Headless" Go Stack (Go as a dumb JSON pipe for Next.js). The Problem: The "Props" Chasm</a> </h2> The core friction in modern web development isn't rendering; it's data synchronization. When you split your stack into Go (backend) and TypeScript (frontend), you create a "Props Chasm." To render a simple user profile, you must: Define a User</code> struct in Go.</li> Serialize it to JSON.</li> Transmit it over the wire (latency).</li> Parse it in JavaScript.</li> Validate it with Zod/Yup (runtime overhead).</li> Define a matching TypeScript interface.</li> </ol> If you change a field in Go, your frontend silently breaks or renders undefined</code> until you redeploy. Tools like OpenAPI generate client code to mitigate this, but they are band-aids over a serialization wound. Approach 1: The "Pure Go" Stack (Templ + HTMX)</a> </h2> The most significant shift in the Go ecosystem is the maturation of Templ. Templ is not a string-based template engine like html/template</code>. It is a transpiler that converts Go-like HTML syntax into pure, efficient Go code. It brings the "Component" mental model - props, composition, type safety - to the backend. The Component Model</a> </h3> In React, you write: // Profile.tsx interface Props { user: User; } export const Profile = ({ user }: Props) => ( <div className="p-4 border"> <h1>{user.Name}</h1> {user.IsAdmin && <AdminBadge role={user.Role} />} </div> ); </code></pre> In Templ (2026), you write: // profile.templ package components import "github.com/myorg/types" templ Profile(user types.User) { <div class="p-4 border"> <h1>{ user.Name }</h1> if user.IsAdmin { @AdminBadge(user.Role) } </div> } </code></pre> The Architectural Win: Zero-Cost Abstractions</a> </h3> Because Templ compiles to standard Go functions, the "component tree" is just a function call tree. Performance: There is no Virtual DOM. No reconciliation. It is just io.Writer</code> streaming bytes.</li> Type Safety: If you remove Role</code> from the User</code> struct in Go, your UI code fails to compile. The feedback loop is immediate.</li> Composition: You can pass components as arguments to other components, allowing for layouts and higher-order components exactly like React children</code>.</li> </ul> When paired with HTMX for interactivity (handling form submissions, partial page replacements), you eliminate the need for a separate Node.js build pipeline entirely. Approach 2: The "Headless" Go Stack (Next.js / Remix)</a> </h2> The alternative is accepting that Go is terrible at UI state. If your application requires rich client-side interactivity - dragging nodes on a canvas, complex data grids, offline-first behavior - server-side rendering in Go hits a wall. Templ can render the initial HTML fast, but it cannot manage complex browser state efficiently. In this model, Go retreats to the "Backend for Frontend" (BFF) role. Next.js handles the Rendering / Routing / State.</li> Go handles the Business Logic / Database / Queues.</li> </ul> The Serialization Tax</a> </h3> The trade-off here is operational complexity. You are now running two distinct distributed systems. You need observability for both. You need to align deployments (so the API doesn't drift from the frontend). You are paying the serialization tax on every request. The Verdict: How to Choose in 2026</a> </h2> As a Staff Engineer, avoid "defaulting" to React just because it's popular. Default to the architecture that minimizes moving parts. Choose strict Go + Templ if:</a> </h3> Your app is "Content-Heavy" or "CRUD-Heavy": Dashboards, admin panels, e-commerce listings, blogs.</li> Team Composition: Your team is predominantly backend/systems engineers.</li> Performance Constraint: You need "Time to First Byte" (TTFB) under 50ms. Go streaming HTML is unbeatable here.</li> </ol> Choose Go + React/Next.js if:</a> </h3> Your app is "State-Heavy": collaborative editors (Figma-like), complex interactive visualizations, heavy use of browser APIs.</li> Ecosystem Reliance: You strictly need libraries that only exist in React (e.g., highly specific calendar widgets or map integrations).</li> Hiring: You plan to hire specialized frontend engineers who do not want to touch Go code.</li> </ol> Conclusion</a> </h2> The era of "Go templating is messy strings" is over. With Templ, Go has a first-class component story that rivals Svelte or React for developer experience, with vastly superior performance characteristics. Before you spin up a node_modules</code> folder, ask yourself: Do I need a distributed system, or do I just need to render some HTML? State of Agentic Commerce 2026: The Protocol Wars and the New Retail Stack 2026-01-21T00:00:00+00:00 As of January 2026, the hypothetical concept of "Agentic Commerce" - software agents autonomously executing financial transactions - has transitioned from research papers to production infrastructure. The announcements at NRF (National Retail Federation) 2026 have solidified the battle lines for the next decade of digital retail. This post analyzes the current state of the ecosystem, focusing on the emerging "Protocol Wars" between major tech incumbents, and synthesizes strategic insights from recent consultancy reports. The Infrastructure Divergence: ACP vs. UCP</a> </h2> The most significant development of early 2026 is the bifurcation of standards. We are witnessing a classic standards war, reminiscent of Betamax vs. VHS, but applied to the negotiation layer of AI commerce. 1. Stripe & OpenAI: The Agentic Commerce Protocol (ACP)</a> </h3> Philosophy: Payment-Backwards Stripe, in collaboration with OpenAI, has doubled down on the Agentic Commerce Protocol (ACP). Their thesis is built on the friction of finality. Agents can browse easily, but they stumble at the checkout. The Stack: ACP focuses on a standardized "handshake" for payment authorization. It creates a secure tunnel between an LLM (running on OpenAI's infrastructure) and a merchant's payment gateway (Stripe).</li> Key Feature - Agentic Commerce Suite: This newly announced suite allows retailers to "broadcast" their liquidity to agents. It treats AI models as a new distribution channel, distinct from web and mobile.</li> Adoption: The integration with ChatGPT's "Instant Checkout" is the flagship implementation, aiming to capture high-intent users who treat the chatbot as a search engine.</li> </ul> 2. Google & Shopify: The Universal Commerce Protocol (UCP)</a> </h3> Philosophy: Search-Forwards Announced at NRF 2026, the Universal Commerce Protocol (UCP) is a counter-move by an alliance of Google, Shopify, and major retailers like Walmart and Wayfair. The Stack: UCP is broader than payments; it is a negotiation protocol. It defines how an agent discovers catalogue capabilities ("Do you have this in blue?" "Can you deliver by Tuesday?") before a transaction is even attempted.</li> The "Embedded Checkout Protocol" (ECP): A sub-component of UCP, this allows the transaction to happen directly within the discovery surface (e.g., Google's "AI Mode" in Search or Gemini).</li> Differentiation: Unlike ACP, which feels like a pipe between a brain and a wallet, UCP feels like a structured language for digital bartering. It is designed to handle the complexity of retail logic (inventory variances, shipping rules) upstream from the payment.</li> </ul> 3. The Wildcard: x402 & The Revival of HTTP 402</a> </h3> Philosophy: The Native Web Standard While Stripe and Google battle for the high-value retail cart, a third faction led by Cloudflare, Coinbase, and the open-source community is reviving the long-dormant HTTP 402 "Payment Required" status code. The "x402" Standard: This is the protocol for the agent economy itself. It allows an API or web resource to block an agent with a 402</code> error, which includes a header requesting a micropayment (often 0.1¢ to 5¢).</li> Usage: Unlike ACP/UCP which sell goods (shoes, tickets), x402 sells access and compute. It is becoming the standard for agent-to-agent services - paying for clean data, premium inference, or specialized API lookups.</li> The Split: We are seeing a divergence: ACP/UCP for Consumer Commerce (Macro-payments >$5) and x402 for Machine Commerce (Micro-payments <$0.10).</li> </ul> Strategic Insights: The "Zero Billion Dollar" Market</a> </h2> While the infrastructure is being laid, the actual market volume remains a subject of intense debate among analysts. The Bull Case: McKinsey & The Merchandising Transformation</a> </h3> In their 2026 analysis, McKinsey suggests that the primary value unlock isn't just in the transaction, but in the operational layer. They highlight that 65% of organizations are now using GenAI regularly, but the frontier is "Agentic Merchandising". Supply Chain Optimization: Agents don't just buy; they predict. McKinsey notes that retailers are using agentic AI to dynamically adjust pricing and COGS (Cost of Goods Sold) management in real-time response to agentic crawler demand.</li> The "Robotic Consumer": Brands are beginning to optimize their digital shelves not for human eyes (images, emotional copy) but for agent parsing (structured data, UCP compliance).</li> </ul> The Bear Case: The "Assisted" Reality</a> </h3> Contrasting the hype, analysts like Dan Rosen argue that "Agentic Commerce" as a fully autonomous market (where the human is out of the loop) is effectively a "$0 Billion Market" today. Behavioral Friction: Consumers are using AI agents for discovery and research, but trust barriers prevent them from delegating the final "buy" button click for high-value items.</li> The "Assisted" Pivot: The immediate opportunity is not autonomous buying, but Agent-Assisted Commerce. The protocols (ACP/UCP) are valuable because they structure the data for the research phase, even if a human eventually finalizes the purchase.</li> </ul> The Emerging Stack (2026)</a> </h2> To build for this ecosystem, developers must understand the four layers of the new stack: The Brain (LLMs): GPT-5, Gemini Ultra.</li> The Context Layer (MCP): Anthropic’s Model Context Protocol remains the standard for agents to read data (pricing, specs) without hallucinations.</li> The Negotiation Layer (UCP/ACP): The new battleground. The choice between UCP and ACP will likely depend on your existing platform allegiance (Shopify vs. Custom/Stripe).</li> The Payment Layer (AP2): Google’s Agent Payments Protocol (AP2) is trying to abstract the payment method, making the agent "wallet-agnostic."</li> </ol> The Implementation Frontier: Geographies & Platforms</a> </h2> As the stack matures, the implementation challenges are becoming strictly regional and platform-specific. 1. The Payment Rail Divide: Tokenized Cards vs. UPI</a> </h3> The biggest friction for global agentic commerce is the "Auth Layer." US/West (The "Silent" Model): The US model relies on Tokenized Cards (Network Tokens). This is ideal for agents because a user can pre-authorize an agent (e.g., "Spend up to $50/month on coffee") and the agent can execute "silent" transactions without further user intervetion.</li> India/East (The "Interactive" Model): In India, UPI (Unified Payments Interface) dominates. While low-cost, standard UPI is high-friction for agents because it requires a PIN for every transaction. The Workaround: The rapid adoption of UPI Autopay and Delegated Mandates (e.g., UPI Lite for agents) is the bridge. However, this creates a fundamentally different UX: The Indian agent is "Assisted" (Ask for PIN) by default, whereas the US agent is "Autonomous" by default.</li> </ul> </li> </ul> 2. Strategic Geographies: ONDC vs. The Private Cloud</a> </h3> Interestingly, while the US leads in Model capability (OpenAI/Google), India leads in Network readiness. The private "Walled Gardens" (US): In the US, an agent needs UCP to talk to Shopify, ACP to talk to Stripe, and Amazon's proprietary API to talk to Amazon. It is fragmented.</li> The Public Standard (India's ONDC): The Beckn Protocol (the backbone of ONDC) effectively makes the entire Indian retail ecosystem "agent-ready" out of the box. An agent doesn't need to learn 50 APIs; it just needs to speak Beckn/ONDC to discover catalogs from groceries to ride-hailing.</li> </ul> 3. The Death of the "App" Interface</a> </h3> What happens to the mobile app in an agentic world? Shopify & Headless: Shopify's push for "Agentic Storefronts" acknowledges that the website is no longer the primary destination. "Headless commerce" has shifted from a developer preference to a survival strategy. If your store data isn't exposed via API (JSON) for an agent to read, you are invisible.</li> Native Apps as "Dumb Pipes": Native mobile apps (iOS/Android) are the hardest for agents to navigate. We are seeing a shift where apps are becoming merely the "fulfillment layer" (tracking delivery), while the "discovery and transaction" layer moves entirely to the Agent's interface. Deep-linking relies on the agent "handing off" the user, but the goal of 2026 is to maintain the user in the conversation.</li> </ul> Conclusion</a> </h2> 2026 is the year of infrastructure. We have moved beyond the "demo" phase where an agent haphazardly navigates a web browser using vision. We now have dedicated pipes (ACP/UCP) for machine-to-machine commerce. For brands, the message is clear: You can no longer just optimize for SEO (Human Search). You must now optimize for AEO (Agent Environment Optimization). If your catalog isn't readable by UCP or ACP, you are effectively invisible to the wealthiest consumers of the next decade - the AI agents. Apache Spark vs Flink vs Airflow: A Comprehensive Comparison with Real-World Examples 2026-01-20T00:00:00+00:00 In the modern data engineering landscape, choosing the right tool for your data processing needs can be overwhelming. Apache Spark, Apache Flink, and Apache Airflow are three powerful frameworks that often come up in discussions, but they serve different purposes. In this comprehensive guide, we'll compare these three tools and provide real-world examples to help you understand when to use each one. Understanding the Core Difference</a> </h2> Before diving into the details, it's crucial to understand that these tools serve different primary purposes: Apache Spark: A unified analytics engine for large-scale data processing (both batch and streaming)</li> Apache Flink: A stream processing framework with true real-time capabilities</li> Apache Airflow: A workflow orchestration platform for scheduling and monitoring data pipelines</li> </ul> Think of it this way: Spark and Flink are the workers that process your data, while Airflow is the manager that schedules and coordinates when and how these workers operate. Apache Spark: The Unified Analytics Engine</a> </h2> What is Apache Spark?</a> </h3> Apache Spark is a distributed computing framework designed for large-scale data processing. It provides high-level APIs in Java, Scala, Python, and R, and supports batch processing, SQL queries, streaming data, machine learning, and graph processing. Key Features</a> </h3> In-memory processing: Processes data in RAM for faster computation</li> Lazy evaluation: Optimizes execution plans before running</li> RDD, DataFrame, and Dataset APIs: Multiple abstraction levels for different use cases</li> Spark Streaming: Micro-batch processing for near real-time analytics</li> MLlib: Built-in machine learning library</li> Spark SQL: SQL interface for structured data processing</li> </ul> Real-World Example: E-commerce User Behavior Analysis</a> </h3> Let's build a real-world example analyzing e-commerce user behavior to generate product recommendations. Scenario: You have clickstream data from your e-commerce website stored in S3/HDFS, and you want to: Clean and transform the data</li> Identify user purchase patterns</li> Generate product recommendations</li> Store results in a data warehouse</li> </ol> from pyspark.sql import SparkSession from pyspark.sql.functions import col, count, desc, collect_list, explode, lit, rank, sum, avg, when, countDistinct from pyspark.sql.window import Window from pyspark.ml.fpm import FPGrowth from datetime import datetime # Initialize Spark Session spark = SparkSession.builder \ .appName("EcommerceUserBehaviorAnalysis") \ .config("spark.sql.adaptive.enabled", "true") \ .config("spark.sql.adaptive.coalescePartitions.enabled", "true") \ .getOrCreate() # Read clickstream data from S3 clickstream_df = spark.read \ .option("header", "true") \ .option("inferSchema", "true") \ .parquet("s3a://your-bucket/clickstream-data/") # Sample data structure: # user_id, session_id, event_type, product_id, category, timestamp, price # 1. Data Cleaning and Transformation cleaned_df = clickstream_df \ .filter(col("event_type").isin(["view", "add_to_cart", "purchase"])) \ .filter(col("product_id").isNotNull()) \ .withColumn("date", col("timestamp").cast("date")) # 2. User Purchase Patterns Analysis user_purchases = cleaned_df \ .filter(col("event_type") == "purchase") \ .groupBy("user_id", "session_id") \ .agg( collect_list("product_id").alias("products"), count("product_id").alias("item_count"), sum("price").alias("total_amount") ) # 3. Find Popular Product Combinations using FP-Growth Algorithm fp_growth = FPGrowth(itemsCol="products", minSupport=0.01, minConfidence=0.3) model = fp_growth.fit(user_purchases) # Get frequent itemsets (products bought together) frequent_itemsets = model.freqItemsets frequent_itemsets.orderBy(desc("freq")).show(20) # Get association rules (if user buys A, they might buy B) association_rules = model.associationRules recommendations_df = association_rules \ .withColumn("confidence_pct", (col("confidence") 100).cast("int")) \ .filter(col("confidence") > 0.5) \ .orderBy(desc("confidence")) # 4. Calculate Product Popularity by Category product_popularity = cleaned_df \ .filter(col("event_type") == "purchase") \ .groupBy("category", "product_id") \ .agg( count("").alias("purchase_count"), avg("price").alias("avg_price") ) # Add popularity rank within each category window_spec = Window.partitionBy("category").orderBy(desc("purchase_count")) product_popularity_ranked = product_popularity \ .withColumn("rank", rank().over(window_spec)) \ .filter(col("rank") <= 10) # 5. User Segmentation based on behavior user_segments = cleaned_df \ .groupBy("user_id") \ .agg( count(when(col("event_type") == "view", 1)).alias("views"), count(when(col("event_type") == "add_to_cart", 1)).alias("cart_adds"), count(when(col("event_type") == "purchase", 1)).alias("purchases"), sum(when(col("event_type") == "purchase", col("price"))).alias("total_spent") ) \ .withColumn("conversion_rate", (col("purchases") / col("views") 100).cast("decimal(5,2)")) \ .withColumn("segment", when(col("total_spent") > 1000, "premium") .when(col("total_spent") > 500, "regular") .otherwise("casual")) # 6. Save results to data warehouse (e.g., Redshift, Snowflake) recommendations_df.write \ .mode("overwrite") \ .format("jdbc") \ .option("url", "jdbc:postgresql://your-redshift-cluster:5439/database") \ .option("dbtable", "product_recommendations") \ .option("user", "username") \ .option("password", "password") \ .save() product_popularity_ranked.write \ .mode("overwrite") \ .format("jdbc") \ .option("url", "jdbc:postgresql://your-redshift-cluster:5439/database") \ .option("dbtable", "popular_products") \ .option("user", "username") \ .option("password", "password") \ .save() user_segments.write \ .mode("overwrite") \ .format("jdbc") \ .option("url", "jdbc:postgresql://your-redshift-cluster:5439/database") \ .option("dbtable", "user_segments") \ .option("user", "username") \ .option("password", "password") \ .save() # 7. Generate daily report daily_stats = cleaned_df \ .groupBy("date", "event_type") \ .agg( count("").alias("event_count"), countDistinct("user_id").alias("unique_users"), sum(when(col("event_type") == "purchase", col("price"))).alias("revenue") ) daily_stats.write \ .mode("append") \ .partitionBy("date") \ .parquet("s3a://your-bucket/daily-stats/") print("Analysis completed successfully!") spark.stop() </code></pre> Why Spark for this use case? Handles large volumes of historical clickstream data efficiently</li> Built-in machine learning library (FPGrowth) for pattern discovery</li> Can process data from multiple sources (S3, HDFS, databases)</li> Optimized for batch processing with in-memory computation</li> </ul> Apache Flink: True Stream Processing</a> </h2> What is Apache Flink?</a> </h3> Apache Flink is a distributed stream processing framework that excels at processing unbounded data streams with low latency and high throughput. Unlike Spark's micro-batch approach, Flink provides true stream processing with event time semantics. Key Features</a> </h3> True streaming: Processes events one at a time, not in micro-batches</li> Event time processing: Handles out-of-order events correctly</li> Stateful computations: Maintains state across events</li> Exactly-once semantics: Guarantees no data loss or duplication</li> Low latency: Millisecond-level latency for real-time applications</li> Complex Event Processing (CEP): Pattern detection in streams</li> </ul> Real-World Example: Real-Time Fraud Detection System</a> </h3> Let's build a real-time fraud detection system for credit card transactions. Scenario: Process credit card transactions in real-time to detect potential fraud patterns: Track transaction velocity per user</li> Detect unusual spending patterns</li> Flag transactions from new locations</li> Identify suspicious transaction sequences</li> Alert in real-time when fraud is detected</li> </ol> import org.apache.flink.api.common.eventtime.WatermarkStrategy; import org.apache.flink.api.common.functions.RichMapFunction; import org.apache.flink.api.common.state.; import org.apache.flink.api.common.typeinfo.TypeInformation; import org.apache.flink.api.java.tuple.Tuple2; import org.apache.flink.configuration.Configuration; import org.apache.flink.streaming.api.CheckpointingMode; import org.apache.flink.streaming.api.datastream.DataStream; import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment; import org.apache.flink.streaming.api.functions.KeyedProcessFunction; import org.apache.flink.streaming.api.functions.windowing.ProcessWindowFunction; import org.apache.flink.streaming.api.windowing.assigners.TumblingEventTimeWindows; import org.apache.flink.streaming.api.windowing.time.Time; import org.apache.flink.streaming.api.windowing.windows.TimeWindow; import org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer; import org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer; import org.apache.flink.util.Collector; import java.time.Duration; import java.util.; // Transaction POJO public class Transaction { public String transactionId; public String userId; public String cardNumber; public double amount; public String merchantId; public String merchantCategory; public String location; public double latitude; public double longitude; public long timestamp; // Constructor, getters, setters } // Fraud Alert POJO public class FraudAlert { public String transactionId; public String userId; public String alertType; public String reason; public double riskScore; public long timestamp; // Constructor, getters, setters } public class RealTimeFraudDetection { public static void main(String[] args) throws Exception { // Set up the streaming execution environment StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); // Enable checkpointing for fault tolerance (every 10 seconds) env.enableCheckpointing(10000); env.getCheckpointConfig().setCheckpointingMode(CheckpointingMode.EXACTLY_ONCE); // Configure Kafka consumer Properties kafkaProps = new Properties(); kafkaProps.setProperty("bootstrap.servers", "localhost:9092"); kafkaProps.setProperty("group.id", "fraud-detection"); // Create Kafka source for transactions FlinkKafkaConsumer<Transaction> kafkaConsumer = new FlinkKafkaConsumer<>( "transactions", new TransactionDeserializationSchema(), kafkaProps ); // Assign watermarks for event time processing WatermarkStrategy<Transaction> watermarkStrategy = WatermarkStrategy .<Transaction>forBoundedOutOfOrderness(Duration.ofSeconds(5)) .withTimestampAssigner((transaction, timestamp) -> transaction.timestamp); // Read transaction stream DataStream<Transaction> transactions = env .addSource(kafkaConsumer) .assignTimestampsAndWatermarks(watermarkStrategy); // Rule 1: Detect high-velocity transactions (>5 transactions in 1 minute) DataStream<FraudAlert> velocityAlerts = transactions .keyBy(t -> t.userId) .window(TumblingEventTimeWindows.of(Time.minutes(1))) .process(new VelocityDetector()); // Rule 2: Detect unusual amount patterns DataStream<FraudAlert> amountAlerts = transactions .keyBy(t -> t.userId) .process(new UnusualAmountDetector()); // Rule 3: Detect rapid location changes DataStream<FraudAlert> locationAlerts = transactions .keyBy(t -> t.userId) .process(new LocationAnomalyDetector()); // Rule 4: Detect suspicious transaction sequences DataStream<FraudAlert> sequenceAlerts = transactions .keyBy(t -> t.userId) .process(new SuspiciousSequenceDetector()); // Combine all alerts DataStream<FraudAlert> allAlerts = velocityAlerts .union(amountAlerts) .union(locationAlerts) .union(sequenceAlerts); // Deduplicate and enrich alerts DataStream<FraudAlert> enrichedAlerts = allAlerts .keyBy(alert -> alert.transactionId) .window(TumblingEventTimeWindows.of(Time.seconds(5))) .process(new AlertAggregator()); // Send high-risk alerts to Kafka FlinkKafkaProducer<FraudAlert> kafkaProducer = new FlinkKafkaProducer<>( "fraud-alerts", new FraudAlertSerializationSchema(), kafkaProps, FlinkKafkaProducer.Semantic.EXACTLY_ONCE ); enrichedAlerts .filter(alert -> alert.riskScore > 0.7) .addSink(kafkaProducer); // Store all alerts in database for analysis enrichedAlerts.addSink(new JdbcSink()); // Execute the Flink job env.execute("Real-Time Fraud Detection"); } // Velocity Detector: Flags if user makes >5 transactions in 1 minute public static class VelocityDetector extends ProcessWindowFunction<Transaction, FraudAlert, String, TimeWindow> { @Override public void process(String userId, Context context, Iterable<Transaction> transactions, Collector<FraudAlert> out) { int count = 0; Transaction lastTxn = null; for (Transaction txn : transactions) { count++; lastTxn = txn; } // Alert if more than 5 transactions in 1 minute if (count > 5) { FraudAlert alert = new FraudAlert(); alert.transactionId = lastTxn.transactionId; alert.userId = userId; alert.alertType = "HIGH_VELOCITY"; alert.reason = String.format("%d transactions in 1 minute", count); alert.riskScore = Math.min(0.5 + (count - 5) 0.1, 1.0); alert.timestamp = System.currentTimeMillis(); out.collect(alert); } } } // Unusual Amount Detector: Compares transaction with user's historical patterns public static class UnusualAmountDetector extends KeyedProcessFunction<String, Transaction, FraudAlert> { // State to store user's transaction history private transient ValueState<TransactionStats> statsState; @Override public void open(Configuration parameters) { ValueStateDescriptor<TransactionStats> descriptor = new ValueStateDescriptor<>("transaction-stats", TransactionStats.class); statsState = getRuntimeContext().getState(descriptor); } @Override public void processElement(Transaction txn, Context ctx, Collector<FraudAlert> out) throws Exception { TransactionStats stats = statsState.value(); if (stats == null) { stats = new TransactionStats(); } // Check if amount is significantly higher than average if (stats.count > 10) { double avgAmount = stats.totalAmount / stats.count; double stdDev = Math.sqrt(stats.sumOfSquares / stats.count - avgAmount avgAmount); // Flag if amount is > 3 standard deviations from mean if (txn.amount > avgAmount + (3 stdDev)) { FraudAlert alert = new FraudAlert(); alert.transactionId = txn.transactionId; alert.userId = txn.userId; alert.alertType = "UNUSUAL_AMOUNT"; alert.reason = String.format("Amount $%.2f is %.1fx higher than average $%.2f", txn.amount, txn.amount / avgAmount, avgAmount); alert.riskScore = Math.min((txn.amount / avgAmount - 1) 0.2, 0.9); alert.timestamp = System.currentTimeMillis(); out.collect(alert); } } // Update statistics stats.count++; stats.totalAmount += txn.amount; stats.sumOfSquares += txn.amount txn.amount; statsState.update(stats); } } // Location Anomaly Detector: Flags transactions from impossible locations public static class LocationAnomalyDetector extends KeyedProcessFunction<String, Transaction, FraudAlert> { private transient ValueState<LastLocation> lastLocationState; @Override public void open(Configuration parameters) { ValueStateDescriptor<LastLocation> descriptor = new ValueStateDescriptor<>("last-location", LastLocation.class); lastLocationState = getRuntimeContext().getState(descriptor); } @Override public void processElement(Transaction txn, Context ctx, Collector<FraudAlert> out) throws Exception { LastLocation lastLoc = lastLocationState.value(); if (lastLoc != null) { // Calculate distance between locations (using Haversine formula) double distance = calculateDistance( lastLoc.latitude, lastLoc.longitude, txn.latitude, txn.longitude ); // Calculate time difference in hours double timeDiffHours = (txn.timestamp - lastLoc.timestamp) / (1000.0 3600); // Calculate required speed (km/h) double requiredSpeed = distance / timeDiffHours; // Flag if speed > 1000 km/h (impossible without flying) if (timeDiffHours < 2 && requiredSpeed > 1000) { FraudAlert alert = new FraudAlert(); alert.transactionId = txn.transactionId; alert.userId = txn.userId; alert.alertType = "IMPOSSIBLE_TRAVEL"; alert.reason = String.format("%.0f km traveled in %.1f hours (%.0f km/h)", distance, timeDiffHours, requiredSpeed); alert.riskScore = 0.95; alert.timestamp = System.currentTimeMillis(); out.collect(alert); } } // Update last location LastLocation newLoc = new LastLocation(); newLoc.latitude = txn.latitude; newLoc.longitude = txn.longitude; newLoc.timestamp = txn.timestamp; lastLocationState.update(newLoc); } private double calculateDistance(double lat1, double lon1, double lat2, double lon2) { // Haversine formula implementation double R = 6371; // Earth radius in km double dLat = Math.toRadians(lat2 - lat1); double dLon = Math.toRadians(lon2 - lon1); double a = Math.sin(dLat/2) Math.sin(dLat/2) + Math.cos(Math.toRadians(lat1)) Math.cos(Math.toRadians(lat2)) Math.sin(dLon/2) Math.sin(dLon/2); double c = 2 Math.atan2(Math.sqrt(a), Math.sqrt(1-a)); return R c; } } // Suspicious Sequence Detector: Detects patterns like test transactions public static class SuspiciousSequenceDetector extends KeyedProcessFunction<String, Transaction, FraudAlert> { private transient ListState<Transaction> recentTransactionsState; @Override public void open(Configuration parameters) { ListStateDescriptor<Transaction> descriptor = new ListStateDescriptor<>("recent-transactions", Transaction.class); recentTransactionsState = getRuntimeContext().getListState(descriptor); } @Override public void processElement(Transaction txn, Context ctx, Collector<FraudAlert> out) throws Exception { List<Transaction> recentTxns = new ArrayList<>(); recentTransactionsState.get().forEach(recentTxns::add); // Pattern 1: Multiple small transactions followed by large transaction if (recentTxns.size() >= 3) { int smallTxnCount = 0; for (Transaction t : recentTxns) { if (t.amount < 5.0) smallTxnCount++; } if (smallTxnCount >= 3 && txn.amount > 500) { FraudAlert alert = new FraudAlert(); alert.transactionId = txn.transactionId; alert.userId = txn.userId; alert.alertType = "SUSPICIOUS_SEQUENCE"; alert.reason = "Multiple small test transactions followed by large amount"; alert.riskScore = 0.85; alert.timestamp = System.currentTimeMillis(); out.collect(alert); } } // Pattern 2: Transactions at suspicious merchant categories String[] suspiciousMerchants = {"WIRE_TRANSFER", "CRYPTO", "GIFT_CARDS"}; if (Arrays.asList(suspiciousMerchants).contains(txn.merchantCategory)) { FraudAlert alert = new FraudAlert(); alert.transactionId = txn.transactionId; alert.userId = txn.userId; alert.alertType = "SUSPICIOUS_MERCHANT"; alert.reason = "Transaction at high-risk merchant category: " + txn.merchantCategory; alert.riskScore = 0.6; alert.timestamp = System.currentTimeMillis(); out.collect(alert); } // Keep only last 5 transactions recentTxns.add(txn); if (recentTxns.size() > 5) { recentTxns.remove(0); } recentTransactionsState.update(recentTxns); } } // Helper classes public static class TransactionStats { public int count = 0; public double totalAmount = 0.0; public double sumOfSquares = 0.0; } public static class LastLocation { public double latitude; public double longitude; public long timestamp; } } </code></pre> Why Flink for this use case? True real-time processing: Detects fraud within milliseconds</li> Stateful computations: Maintains user transaction history in memory</li> Event time processing: Handles late-arriving transactions correctly</li> Exactly-once guarantees: Ensures no alert is missed or duplicated</li> Complex event processing: Detects sophisticated fraud patterns across multiple transactions</li> </ul> Apache Airflow: The Workflow Orchestrator</a> </h2> What is Apache Airflow?</a> </h3> Apache Airflow is a platform to programmatically author, schedule, and monitor workflows. It's not a data processing engine itself but rather orchestrates data pipelines that may use Spark, Flink, or other tools. Key Features</a> </h3> DAG-based workflows: Define workflows as Directed Acyclic Graphs</li> Extensible: Rich library of operators for various systems</li> Dynamic pipelines: Generate pipelines dynamically using Python</li> Rich UI: Web interface to monitor and manage workflows</li> Scheduling: Cron-based scheduling with dependency management</li> Task retries and alerts: Automatic retry and notification on failures</li> </ul> Real-World Example: End-to-End ML Pipeline Orchestration</a> </h3> Let's build a complete ML pipeline that orchestrates data extraction, transformation, model training, and deployment. Scenario: Build a daily ML pipeline that: Extracts data from multiple sources</li> Runs data quality checks</li> Processes data with Spark</li> Trains a machine learning model</li> Validates model performance</li> Deploys model if metrics are acceptable</li> Sends notifications on success/failure</li> </ol> from datetime import datetime, timedelta from airflow import DAG from airflow.operators.python import PythonOperator, BranchPythonOperator from airflow.operators.bash import BashOperator from airflow.operators.dummy import DummyOperator from airflow.providers.amazon.aws.operators.s3 import S3CreateBucketOperator from airflow.providers.amazon.aws.operators.emr import EmrCreateJobFlowOperator, EmrTerminateJobFlowOperator from airflow.providers.amazon.aws.sensors.emr import EmrJobFlowSensor from airflow.providers.postgres.operators.postgres import PostgresOperator from airflow.providers.http.operators.http import SimpleHttpOperator from airflow.providers.slack.operators.slack_webhook import SlackWebhookOperator from airflow.models import Variable from airflow.utils.trigger_rule import TriggerRule import json import boto3 import pandas as pd import numpy as np from sklearn.ensemble import RandomForestClassifier from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score import joblib # Default arguments for the DAG default_args = { 'owner': 'data-science-team', 'depends_on_past': False, 'start_date': datetime(2026, 1, 1), 'email': ['datascience@company.com'], 'email_on_failure': True, 'email_on_retry': False, 'retries': 2, 'retry_delay': timedelta(minutes=5), 'execution_timeout': timedelta(hours=4), } # Create the DAG dag = DAG( 'ml_pipeline_customer_churn', default_args=default_args, description='End-to-end ML pipeline for customer churn prediction', schedule_interval='0 2 * * ', # Run daily at 2 AM catchup=False, max_active_runs=1, tags=['ml', 'production', 'churn'], ) # Task 1: Check data availability in source systems def check_data_availability(context): """Check if all required data sources have data for execution date""" execution_date = context['execution_date'] date_str = execution_date.strftime('%Y-%m-%d') s3 = boto3.client('s3') bucket = 'company-data-lake' # Check multiple data sources sources = [ f'raw/customer_data/{date_str}/', f'raw/transactions/{date_str}/', f'raw/support_tickets/{date_str}/', f'raw/product_usage/{date_str}/' ] missing_sources = [] for source in sources: try: response = s3.list_objects_v2(Bucket=bucket, Prefix=source) if 'Contents' not in response or len(response['Contents']) == 0: missing_sources.append(source) except Exception as e: missing_sources.append(source) if missing_sources: raise ValueError(f"Missing data sources: {missing_sources}") print(f"All data sources available for {date_str}") return True check_data_task = PythonOperator( task_id='check_data_availability', python_callable=check_data_availability, provide_context=True, dag=dag, ) # Task 2: Run data quality checks def run_data_quality_checks(context): """Run data quality checks on raw data""" execution_date = context['execution_date'] date_str = execution_date.strftime('%Y-%m-%d') # Read sample of data from S3 s3 = boto3.client('s3') quality_issues = [] # Check 1: Null value percentage # (In production, you'd read actual data) null_percentage = 0.05 # 5% nulls if null_percentage > 0.10: quality_issues.append(f"High null percentage: {null_percentage100}%") # Check 2: Data freshness # Ensure data is not older than expected # Check 3: Schema validation # Validate that all expected columns exist # Check 4: Value ranges # Check that numeric values are within expected ranges if quality_issues: raise ValueError(f"Data quality issues found: {quality_issues}") print(f"Data quality checks passed for {date_str}") return True data_quality_task = PythonOperator( task_id='data_quality_checks', python_callable=run_data_quality_checks, provide_context=True, dag=dag, ) # Task 3: Submit Spark job to EMR for data processing EMR_CONFIG = { 'Name': 'ChurnDataProcessing', 'ReleaseLabel': 'emr-6.10.0', 'Applications': [{'Name': 'Spark'}, {'Name': 'Hadoop'}], 'Instances': { 'InstanceGroups': [ { 'Name': 'Master', 'Market': 'ON_DEMAND', 'InstanceRole': 'MASTER', 'InstanceType': 'r5.xlarge', 'InstanceCount': 1, }, { 'Name': 'Worker', 'Market': 'SPOT', 'InstanceRole': 'CORE', 'InstanceType': 'r5.2xlarge', 'InstanceCount': 3, } ], 'KeepJobFlowAliveWhenNoSteps': True, 'TerminationProtected': False, }, 'JobFlowRole': 'EMR_EC2_DefaultRole', 'ServiceRole': 'EMR_DefaultRole', 'LogUri': 's3://company-emr-logs/', 'Steps': [ { 'Name': 'ProcessChurnData', 'ActionOnFailure': 'CONTINUE', 'HadoopJarStep': { 'Jar': 'command-runner.jar', 'Args': [ 'spark-submit', '--deploy-mode', 'cluster', '--master', 'yarn', '--conf', 'spark.sql.adaptive.enabled=true', 's3://company-spark-jobs/churn_data_processing.py', '--input-date', '{{ ds }}', '--output-path', 's3://company-processed-data/churn_features/{{ ds }}/' ] } } ], } create_emr_cluster = EmrCreateJobFlowOperator( task_id='create_emr_cluster', job_flow_overrides=EMR_CONFIG, aws_conn_id='aws_default', dag=dag, ) wait_for_spark_job = EmrJobFlowSensor( task_id='wait_for_spark_job', job_flow_id="{{ task_instance.xcom_pull(task_ids='create_emr_cluster', key='return_value') }}", aws_conn_id='aws_default', dag=dag, ) terminate_emr_cluster = EmrTerminateJobFlowOperator( task_id='terminate_emr_cluster', job_flow_id="{{ task_instance.xcom_pull(task_ids='create_emr_cluster', key='return_value') }}", aws_conn_id='aws_default', trigger_rule=TriggerRule.ALL_DONE, dag=dag, ) # Task 4: Train ML model def train_churn_model(context): """Train customer churn prediction model""" execution_date = context['execution_date'] date_str = execution_date.strftime('%Y-%m-%d') # Load processed features from S3 s3 = boto3.client('s3') feature_path = f's3://company-processed-data/churn_features/{date_str}/features.parquet' # In production, you'd read actual data # df = pd.read_parquet(feature_path) # Simulated training for demonstration print("Loading training data...") # X_train, y_train = df.drop('churned', axis=1), df['churned'] print("Training Random Forest model...") model = RandomForestClassifier( n_estimators=100, max_depth=10, min_samples_split=5, random_state=42, n_jobs=-1 ) # model.fit(X_train, y_train) # Save model to S3 model_path = f'/tmp/churn_model_{date_str}.pkl' joblib.dump(model, model_path) s3.upload_file( model_path, 'company-ml-models', f'churn_prediction/{date_str}/model.pkl' ) print(f"Model trained and saved to S3") # Store model path in XCom for next task context['task_instance'].xcom_push(key='model_path', value=f'churn_prediction/{date_str}/model.pkl') return True train_model_task = PythonOperator( task_id='train_model', python_callable=train_churn_model, provide_context=True, dag=dag, ) # Task 5: Validate model performance def validate_model(context): """Validate model performance on test set""" execution_date = context['execution_date'] date_str = execution_date.strftime('%Y-%m-%d') # Load model from S3 s3 = boto3.client('s3') model_path = context['task_instance'].xcom_pull( task_ids='train_model', key='model_path' ) # Download and load model s3.download_file( 'company-ml-models', model_path, '/tmp/model.pkl' ) model = joblib.load('/tmp/model.pkl') # Load test data # In production, load actual test data # X_test, y_test = load_test_data() # Make predictions (simulated) # y_pred = model.predict(X_test) # Calculate metrics (simulated values) metrics = { 'accuracy': 0.87, 'precision': 0.82, 'recall': 0.79, 'f1_score': 0.80, 'date': date_str } print(f"Model Metrics: {metrics}") # Store metrics in database # In production, write to actual database # Store metrics in XCom context['task_instance'].xcom_push(key='metrics', value=metrics) # Return branch decision based on performance if metrics['f1_score'] >= 0.75: return 'deploy_model' else: return 'alert_poor_performance' validate_model_task = BranchPythonOperator( task_id='validate_model', python_callable=validate_model, provide_context=True, dag=dag, ) # Task 6a: Deploy model if performance is good def deploy_model(**context): """Deploy model to production serving endpoint""" model_path = context['task_instance'].xcom_pull( task_ids='train_model', key='model_path' ) # Deploy to SageMaker/Kubernetes/Model serving platform print(f"Deploying model from {model_path} to production...") # Update model registry # Update feature store # Update model serving endpoint print("Model successfully deployed to production") return True deploy_model_task = PythonOperator( task_id='deploy_model', python_callable=deploy_model, provide_context=True, dag=dag, ) # Task 6b: Alert if performance is poor alert_poor_performance = SlackWebhookOperator( task_id='alert_poor_performance', http_conn_id='slack_webhook', message=""" :warning: Model Performance Alert The churn prediction model for {{ ds }} did not meet performance thresholds. Metrics: {{ task_instance.xcom_pull(task_ids='validate_model', key='metrics') }} Please review the model and data quality. """, dag=dag, ) # Task 7: Update model registry update_registry = PostgresOperator( task_id='update_model_registry', postgres_conn_id='postgres_ml_metadata', sql=""" INSERT INTO model_registry ( model_name, version, metrics, model_path, created_at, status ) VALUES ( 'churn_prediction', '{{ ds }}', '{{ task_instance.xcom_pull(task_ids="validate_model", key="metrics") | tojson }}', '{{ task_instance.xcom_pull(task_ids="train_model", key="model_path") }}', '{{ ts }}', 'deployed' ); """, trigger_rule=TriggerRule.ONE_SUCCESS, dag=dag, ) # Task 8: Send success notification send_success_notification = SlackWebhookOperator( task_id='send_success_notification', http_conn_id='slack_webhook', message=""" :white_check_mark: ML Pipeline Success The churn prediction pipeline for {{ ds }} completed successfully! Metrics: {{ task_instance.xcom_pull(task_ids='validate_model', key='metrics') }} Model deployed and ready for inference. """, trigger_rule=TriggerRule.ONE_SUCCESS, dag=dag, ) # Task 9: Cleanup temporary files cleanup = BashOperator( task_id='cleanup', bash_command='rm -rf /tmp/churn_* /tmp/model.pkl', trigger_rule=TriggerRule.ALL_DONE, dag=dag, ) # Task 10: Generate daily report generate_report = PythonOperator( task_id='generate_daily_report', python_callable=lambda **context: print("Generating daily report..."), provide_context=True, trigger_rule=TriggerRule.ALL_DONE, dag=dag, ) # Define task dependencies check_data_task >> data_quality_task >> create_emr_cluster create_emr_cluster >> wait_for_spark_job >> terminate_emr_cluster terminate_emr_cluster >> train_model_task >> validate_model_task # Branching based on model performance validate_model_task >> [deploy_model_task, alert_poor_performance] # Both branches converge to update registry [deploy_model_task, alert_poor_performance] >> update_registry update_registry >> send_success_notification >> cleanup >> generate_report </code></pre> Why Airflow for this use case? Orchestrates complex workflows: Manages dependencies between multiple tasks</li> Handles failures gracefully: Automatic retries and alerting</li> Dynamic pipelines: Can generate tasks based on runtime conditions</li> Monitors progress: Rich UI to track pipeline execution</li> Integrations: Connects with Spark, databases, cloud services, etc.</li> Scheduling: Runs pipelines on schedule with dependency management</li> </ul> Comparison Table</a> </h2> Feature</th> Apache Spark</th> Apache Flink</th> Apache Airflow</th></tr></thead> Primary Purpose</td> Batch & Streaming Processing</td> Stream Processing</td> Workflow Orchestration</td></tr> Processing Model</td> Micro-batch</td> True streaming</td> N/A (orchestrator)</td></tr> Latency</td> Seconds</td> Milliseconds</td> N/A</td></tr> State Management</td> Limited</td> Excellent</td> N/A</td></tr> Fault Tolerance</td> RDD lineage, checkpointing</td> Checkpointing, savepoints</td> Task retries</td></tr> APIs</td> Java, Scala, Python, R, SQL</td> Java, Scala, Python, SQL</td> Python</td></tr> Learning Curve</td> Moderate</td> Steep</td> Easy to Moderate</td></tr> Maturity</td> Very mature</td> Mature</td> Very mature</td></tr> Community</td> Very large</td> Large</td> Very large</td></tr> Use Cases</td> Batch analytics, ML, ETL</td> Real-time analytics, CEP</td> Pipeline orchestration</td></tr> Memory Usage</td> High (in-memory)</td> Moderate</td> Low</td></tr> Exactly-Once</td> Structured Streaming</td> Native support</td> Via operators</td></tr> Event Time</td> Watermarks</td> Native support</td> N/A</td></tr> SQL Support</td> Excellent (Spark SQL)</td> Good (Flink SQL)</td> Via operators</td></tr> ML Libraries</td> MLlib (built-in)</td> FlinkML (limited)</td> Integrates external</td></tr> Deployment</td> Standalone, YARN, K8s, Mesos</td> Standalone, YARN, K8s, Mesos</td> Standalone, K8s, Docker</td></tr> Monitoring</td> Spark UI, metrics</td> Flink UI, metrics</td> Rich web UI</td></tr> </tbody></table> When to Use Which Tool?</a> </h2> Use Apache Spark When:</a> </h3> Batch processing of large datasets: Historical data analysis, ETL jobs</li> Complex analytics: Machine learning, graph processing, SQL queries</li> Near real-time is acceptable: Micro-batch processing with seconds latency</li> You need mature ML libraries: MLlib provides comprehensive algorithms</li> Flexibility in languages: Need support for multiple programming languages</li> Cost optimization: Can use spot instances for batch jobs</li> </ol> Examples: Daily report generation</li> Training machine learning models</li> Data warehouse ETL</li> Log analysis and aggregation</li> Feature engineering for ML</li> </ul> Use Apache Flink When:</a> </h3> True real-time processing: Millisecond-level latency required</li> Complex event processing: Detecting patterns across multiple events</li> Stateful computations: Need to maintain state across events</li> Event time processing: Handling out-of-order events correctly</li> Exactly-once guarantees: Critical for financial or compliance applications</li> Continuous queries: Long-running streaming queries</li> </ol> Examples: Fraud detection</li> Real-time recommendations</li> Network monitoring and anomaly detection</li> Real-time dashboards</li> IoT event processing</li> Stock trading systems</li> </ul> Use Apache Airflow When:</a> </h3> Orchestrating complex workflows: Multiple dependent tasks</li> Scheduling data pipelines: Run pipelines on a schedule</li> Managing dependencies: Tasks that depend on other tasks or external systems</li> Monitoring and alerting: Need visibility into pipeline execution</li> Mixing multiple tools: Coordinating Spark, Flink, databases, APIs, etc.</li> Dynamic pipelines: Generate tasks based on runtime conditions</li> </ol> Examples: ETL pipelines</li> ML model training pipelines</li> Data quality monitoring</li> Report generation workflows</li> Multi-stage data processing</li> Backup and maintenance tasks</li> </ul> Can You Use Them Together?</a> </h2> Absolutely! In fact, many production data platforms use all three together: Example Architecture: Airflow orchestrates the entire data platform</li> Flink processes real-time streaming data</li> Spark runs batch jobs for historical analysis and ML training</li> Airflow schedules when Spark jobs run</li> Airflow monitors Flink streaming jobs</li> Airflow triggers alerts if either fails</li> </ol> Real-world scenario: Daily at 2 AM (Airflow scheduled): → Extract data from APIs (Airflow task) → Run data quality checks (Airflow task) → Process with Spark (Airflow triggers Spark job) → Train ML model (Airflow triggers Spark ML job) → Deploy model (Airflow task) Continuous (Flink streaming): → Read events from Kafka → Apply ML model for real-time predictions → Write results to database → Send alerts on anomalies Hourly (Airflow scheduled): → Check Flink job health (Airflow sensor) → Restart Flink job if failed (Airflow task) → Update monitoring dashboard (Airflow task) </code></pre> Conclusion</a> </h2> Choosing between Apache Spark, Flink, and Airflow depends on your specific use case: Spark is your go-to for batch processing and complex analytics with a rich ecosystem</li> Flink excels at true real-time stream processing with low latency and stateful computations</li> Airflow is essential for orchestrating and managing complex data workflows</li> </ul> In modern data platforms, you'll often use all three: Airflow as the orchestrator managing the entire data platform</li> Flink handling real-time event streams</li> Spark processing large-scale batch jobs and machine learning workloads</li> </ul> The key is understanding that these tools complement each other rather than compete. By using them together strategically, you can build robust, scalable data platforms that handle both real-time and batch processing needs while maintaining clear visibility and control over your workflows. Further Reading</a> </h2> Apache Spark Documentation</a></li> Apache Flink Documentation</a></li> Apache Airflow Documentation</a></li> Stream Processing with Apache Flink (O'Reilly)</a></li> Learning Spark, 2nd Edition (O'Reilly)</a></li> Data Pipelines with Apache Airflow (Manning)</a></li> </ul> Memoization: The Art of Teaching Your Code to Remember 2025-12-11T00:00:00+00:00 The Core Insight</a> </h2> Memoization is elegantly simple: cache a function's results by its inputs. When the same input appears again, return the cached value instead of recomputing. You're trading memory for speed - replacing repeated work with instant lookups. The Problem: Counting Paths Up Stairs</a> </h2> Imagine a staircase where you can climb 1, 3, or 5 steps at a time. How many distinct ways can you reach step n</code>? Let f(n)</code> be the count of ways. The recurrence relation is: f(n) = f(n-1) + f(n-3) + f(n-5) </code></pre> With base cases: f(0) = 1</code> — one valid way: do nothing</li> f(n) = 0</code> when n < 0</code> — invalid path</li> </ul> Naive Recursion: Clean but Catastrophically Slow</a> </h2> function stepCount(n: number): number { if (n < 0) return 0; if (n === 0) return 1; return stepCount(n - 1) + stepCount(n - 3) + stepCount(n - 5); } </code></pre> This reads like the mathematical definition. Beautiful, right? The problem: identical subproblems explode across branches. For n = 10</code>, stepCount(5)</code> gets called multiple times. The call tree grows exponentially, and we keep solving the same problems over and over. Try stepCount(35)</code> and watch your CPU fan spin up. For n = 50</code>, you might as well go make coffee. Memoization: Add a Memory</a> </h2> Store results by input. On repeat calls, return the stored value instantly. function memoSteps(n: number, cache: Map<number, number> = new Map()): number { if (n < 0) return 0; if (n === 0) return 1; if (cache.has(n)) { return cache.get(n)!; } const result = memoSteps(n - 1, cache) + memoSteps(n - 3, cache) + memoSteps(n - 5, cache); cache.set(n, result); return result; } </code></pre> Now n = 35</code> returns instantly. Even n = 100</code> is trivial because each distinct value of n</code> is computed exactly once. A Reusable Memoization Utility</a> </h2> Rather than manually threading caches, build a generic memoizer: function memoize<T extends (...args: any[]) => any>(fn: T): T { const cache = new Map<string, ReturnType<T>>(); return ((...args: Parameters<T>): ReturnType<T> => { const key = JSON.stringify(args); if (cache.has(key)) { return cache.get(key)!; } const result = fn(...args); cache.set(key, result); return result; }) as T; } // Usage const stepCount = memoize((n: number, steps: number[] = [1, 3, 5]): number => { if (n < 0) return 0; if (n === 0) return 1; return steps.reduce((sum, step) => sum + stepCount(n - step, steps), 0); }); </code></pre> LRU Cache: Bounded Memory</a> </h2> For long-running applications, unbounded caches are dangerous. Implement a Least Recently Used cache: class LRUCache<K, V> { private cache = new Map<K, V>(); constructor(private maxSize: number) {} get(key: K): V | undefined { if (!this.cache.has(key)) return undefined; // Move to end (most recently used) const value = this.cache.get(key)!; this.cache.delete(key); this.cache.set(key, value); return value; } set(key: K, value: V): void { if (this.cache.has(key)) { this.cache.delete(key); } else if (this.cache.size >= this.maxSize) { // Evict oldest (first) entry const firstKey = this.cache.keys().next().value; this.cache.delete(firstKey); } this.cache.set(key, value); } has(key: K): boolean { return this.cache.has(key); } } function memoizeWithLRU<T extends (...args: any[]) => any>( fn: T, maxSize: number = 1000, ): T { const cache = new LRUCache<string, ReturnType<T>>(maxSize); return ((...args: Parameters<T>): ReturnType<T> => { const key = JSON.stringify(args); if (cache.has(key)) { return cache.get(key)!; } const result = fn(...args); cache.set(key, result); return result; }) as T; } </code></pre> Why It Works</a> </h2> Memoization transforms a branching recursion tree into a directed acyclic graph. Each unique input becomes a single node. Compute once, reuse forever. It's like saving a phone number in your contacts. The first time someone gives you their number, you type it in. Every time after that, you just look it up instead of asking again. Memoization is your contact list for function results. When to Reach for Memoization</a> </h2> It shines when you have: Pure functions — output depends only on input, no side effects</li> Overlapping subproblems — the same inputs recur across different execution paths</li> Manageable state space — unique inputs fit comfortably in memory</li> </ol> Common applications: Dynamic programming: Fibonacci, coin change, edit distance, longest common subsequence</li> Graph algorithms: path counting, shortest paths with repeated states</li> Game engines: transposition tables in chess cache board evaluations to skip re-analyzing identical positions</li> API responses: cache expensive computations or external calls</li> </ul> The Bottom-Up Alternative</a> </h2> Sometimes you can flip the script—compute iteratively from base cases up: function stepCountDP(n: number, steps: number[] = [1, 3, 5]): number { const dp: number[] = new Array(n + 1).fill(0); dp[0] = 1; for (let i = 1; i <= n; i++) { for (const step of steps) { if (i - step >= 0) { dp[i] += dp[i - step]; } } } return dp[n]; } </code></pre> Time: O(n × number of steps)</li> Space: O(n)</li> No recursion: avoids stack overflow for large n</code></li> </ul> Choosing Your Approach</a> </h2> Approach</th> Best For</th></tr></thead> Memoization</td> Natural recursive problems, lazy evaluation, when not all subproblems are needed</td></tr> Bottom-up DP</td> Clear state space, tight memory control, avoiding recursion limits</td></tr> LRU caching</td> Long-running services, memory-constrained environments</td></tr> </tbody></table> Practical Gotchas</a> </h2> Memory growth: Unbounded caches can leak memory. Use LRU or clear caches periodically. </li> Cache key design: Keys must be immutable and capture all inputs that affect the result. Use JSON.stringify</code> for simplicity, but beware of object key ordering. </li> Break-even point: For tiny inputs, cache overhead can exceed computation cost. Profile before optimizing. </li> Cache invalidation: If the underlying behavior changes (config updates, time-dependent logic), version your cache or clear it. </li> </ol> The Takeaway</a> </h2> Memoization is a small change with outsized impact. Cache results by input, reuse them on repeat calls. For problems with overlapping subproblems, it transforms exponential blowups into linear-time solutions. Design your keys carefully, bound your memory when needed, and you'll have code that's both elegant and fast. Enterprise AI in 2025: $37 Billion and Counting 2025-12-10T00:00:00+00:00 Enterprise AI has grown from $1.7 billion in 2023 to $37 billion in 2025. That's not a typo. In just three years, AI has captured over 6% of the global SaaS market and become the fastest-scaling software category in history. The bubble fears haven't materialized. The demand side tells a different story: broad adoption, real revenue, and productivity gains at scale. Where the Money Goes</a> </h2> The $37 billion breaks down into two main buckets: Applications ($19 billion): The user-facing products that leverage AI models Departmental AI: $7.3B (coding, IT, marketing, customer success)</li> Horizontal AI: $8.4B (copilots, agents, productivity tools)</li> Vertical AI: $3.5B (healthcare, legal, finance)</li> </ul> Infrastructure ($18 billion): The picks and shovels Foundation model APIs: $12.5B</li> Model training infrastructure: $4B</li> AI infrastructure (storage, retrieval, orchestration): $1.5B</li> </ul> The shift is notable: more than half of enterprise AI spend now goes to applications rather than infrastructure. Enterprises are prioritizing immediate productivity gains over long-term infrastructure bets. The Buy vs. Build Flip</a> </h2> In 2024, enterprises were split - 47% built internally, 53% purchased. A year later, it's not even close: 76% of AI use cases are now purchased rather than built internally. The conventional wisdom that enterprises would build most AI solutions themselves didn't hold. Ready-made AI solutions are reaching production faster and demonstrating immediate value while internal builds mature. AI Buyers Are Different</a> </h2> Enterprise AI buyers behave differently than traditional software buyers: 47% conversion rate for AI deals (vs. 25% for traditional SaaS)</li> 27% of AI spend comes through product-led growth (vs. 7% for traditional software)</li> Organizations surface 10+ potential use cases but focus on near-term productivity gains</li> </ul> That 47% conversion rate is remarkable. Once an organization commits to exploring an AI solution, deals close at nearly twice the rate of traditional software. The value is immediate and clear. PLG Is Eating Enterprise AI</a> </h2> Product-led growth is transforming enterprise AI adoption. Individual users now drive AI adoption at 4x the rate of traditional software. The examples are striking: Cursor reached $200 million in revenue before hiring a single enterprise sales rep</li> n8n formalized contracts only after hundreds of employees were already active users</li> ElevenLabs, Gamma, and others scaled the same way</li> </ul> When you account for "shadow AI adoption" - employees using personal credit cards for tools like ChatGPT Plus - PLG-driven tools may represent close to 40% of application AI spend. The pattern: developers and technical teams discover tools for individual use, prove value in day-to-day work, and create bottom-up demand that converts to enterprise contracts. Startups Are Winning the Application Layer</a> </h2> At the AI application layer, startups now capture nearly $2 in revenue for every $1 earned by incumbents - 63% of the market, up from 36% last year. This shouldn't be happening. Incumbents have distribution, data moats, enterprise relationships, sales teams, and massive balance sheets. Yet AI-native startups are out-executing across the fastest-growing categories. Why startups win: Department</th> Startup Share</th> Why</th></tr></thead> Product + Engineering</td> 71%</td> Cursor beat GitHub Copilot by shipping faster - repo-level context, multi-file editing, model-agnostic approach</td></tr> Sales</td> 78%</td> Clay, Actively attack workflows Salesforce doesn't own - research, personalization, enrichment from unstructured signals</td></tr> Finance + Operations</td> 91%</td> Incumbents can't move fast enough; startups like Rillet and Numeric build AI-first ERPs</td></tr> </tbody></table> The story changes at the infrastructure layer, where incumbents hold 56% of the market. Databricks, Snowflake, MongoDB, and Datadog have seen meaningful re-acceleration as even AI-native builders choose existing platforms. Coding: The First Killer Use Case</a> </h2> Coding is the breakout category. At $4 billion, it represents 55% of all departmental AI spend - the largest category across the entire application layer. The numbers: 50% of developers now use AI coding tools daily (65% in top-quartile orgs)</li> Code completion alone is $2.3B</li> Code agents and AI app builders exploded from near-zero</li> Teams report 15%+ velocity gains</li> </ul> AI now touches the entire software development lifecycle: prototyping (Lovable), code refactoring (OpenHands), design-to-code (Weaver), QA (Meticulous), PRs (Graphite), SRE (Resolve), and deployment (Harness). Anthropic Is Winning</a> </h2> The foundation model landscape shifted decisively. Anthropic now earns an estimated 40% of enterprise LLM spend, up from 24% last year and 12% in 2023. OpenAI fell to 27% from 50% in 2023. The three leaders - Anthropic, OpenAI, and Google - account for 88% of enterprise LLM API usage. Anthropic's dominance comes from coding. They command 54% of the coding market (vs. 21% for OpenAI), driven by an 18-month run atop leaderboards starting with Claude Sonnet 3.5 in June 2024. Agents: Mostly Hype (For Now)</a> </h2> For all the talk of agents, real production architectures remain surprisingly simple: Only 16% of enterprise and 27% of startup deployments qualify as true agents</li> Most are still fixed-sequence or routing-based workflows around a single model call</li> Prompt design remains the dominant customization technique, followed by RAG</li> Fine-tuning, tool calling, and RL are still niche</li> </ul> Copilots dominate horizontal AI with 86% share ($7.2B). Agent platforms capture only 10% ($750M). Strip away the hype and most "AI agents" are basic if-then logic around a model call. The architecture works for today's use cases - but it reveals how early we are. Healthcare Leads Vertical AI</a> </h2> Vertical AI solutions hit $3.5 billion, nearly 3x the prior year. Healthcare alone captures 43% - approximately $1.5 billion. The bulk concentrates in administrative and clinical-adjacent workflows: Ambient scribes: $600M (+2.4x YoY), minting two new unicorns (Abridge, Ambience)</li> Clinicians spend one hour documenting for every five hours of care</li> Scribes that reduce documentation time by 50%+ are transformational</li> </ul> Beyond healthcare: legal ($650M), creator tools ($360M), and government ($350M). Adoption is strongest in industries historically underserved by software - fields defined by manual, unstructured workflows that once depended on human services. What's Next</a> </h2> Five predictions for 2026: AI will exceed human performance in daily practical programming tasks. The best models keep getting better in verifiable domains like math and code. </li> Jevons' paradox continues. Net spend rises despite falling inference costs, driven by orders-of-magnitude increases in inference volume. </li> Explainability and governance go mainstream. As agents gain autonomy, the ability to explain and govern their decisions becomes critical. Governments will ask for audit logs. </li> Models move to the edge. Low-latency requirements, privacy, and cost push compute on-device. Mobile manufacturers ship dedicated low-power GPU compute. </li> Benchmarks saturate but fail to capture real-world efficacy. For frontier use cases like coding, users are price-insensitive and will pay more for performance. </li> </ol> The Takeaway</a> </h2> Enterprise AI is no longer speculative. It's a $37 billion market growing faster than any software category in history. The dynamics are clear: Enterprises are buying, not building</li> Startups are winning the application layer</li> Coding is the first killer use case</li> Anthropic leads the enterprise LLM market</li> Agents are mostly hype - for now</li> PLG is driving adoption at unprecedented rates</li> </ul> Three years in, the transformation is real. The value is clear. And the pace isn't slowing. McKinsey's State of AI 2025: The Gap Between Adoption and Impact 2025-12-08T00:00:00+00:00 McKinsey just dropped their State of AI 2025 report</a>, surveying nearly 2,000 participants across 105 countries. The headline: almost everyone is using AI now, but almost no one is seeing meaningful enterprise-wide impact. Three years into the generative AI era, this is the defining tension. The Adoption-Impact Gap</a> </h2> The numbers tell a clear story: 88% of organizations are using AI in at least one business function (up from 78% last year)</li> 62% are experimenting with AI agents</li> But only 39% report any EBIT impact at the enterprise level</li> And most of those say AI accounts for less than 5% of EBIT</li> </ul> Nearly nine in ten companies are using AI. Fewer than four in ten see it on their bottom line. This isn't a technology problem. It's a deployment problem. Most Organizations Are Still Piloting</a> </h2> The report's key finding: two-thirds of organizations have not begun scaling AI across the enterprise. They're stuck in experimentation and pilot phases. Larger companies are ahead—nearly half of organizations with $5B+ in revenue have reached the scaling phase. But for companies under $100M in revenue, it's only 29%. The pattern is familiar from every previous wave of enterprise technology: early adopters move fast, everyone else gets stuck in "proof of concept" purgatory. AI Agents: High Curiosity, Low Scale</a> </h2> Agents are the hot topic, and the numbers reflect it: 62% are at least experimenting with AI agents</li> 23% are scaling agentic AI somewhere in the enterprise</li> But in any given function, no more than 10% are scaling agents</li> </ul> Agent adoption is highest in: IT and knowledge management (service-desk automation, deep research)</li> Tech, media, and telecom industries</li> Healthcare</li> </ul> The pattern here is revealing: agents work best where the use case is narrow, repeatable, and the cost of errors is manageable. IT service desks. Document retrieval. Scheduling. The more open-ended the task, the harder it is to scale. What High Performers Do Differently</a> </h2> McKinsey defines "AI high performers" as those who attribute 5%+ of EBIT to AI and report "significant" value. They represent about 6% of respondents. What separates them: 1. They Aim for Transformation, Not Just Efficiency</a> </h3> 80% of all organizations set efficiency as an AI objective. That's table stakes. High performers are 3x more likely to say they intend to use AI for transformative change—not just cost cutting. Companies that set growth and innovation as objectives (not just efficiency) report better outcomes across customer satisfaction, competitive differentiation, profitability, and revenue growth. 2. They Redesign Workflows</a> </h3> This is one of the strongest predictors of impact. High performers are nearly 3x more likely to have fundamentally redesigned workflows when deploying AI. Most organizations bolt AI onto existing processes. High performers rebuild the process around AI capabilities. 3. They Scale Faster and Wider</a> </h3> High performers use AI in more functions. They're also 3x more likely to be scaling AI agents (not just piloting them). Three-quarters of high performers have scaled or are scaling AI, compared to one-third of everyone else. 4. Leadership Owns It</a> </h3> High performers are 3x more likely to strongly agree that senior leaders demonstrate ownership and commitment to AI initiatives. Leaders at these organizations aren't just sponsors—they're role-modeling AI use. 5. They Invest More</a> </h3> More than one-third of high performers spend over 20% of their digital budgets on AI technologies. You can't pilot your way to transformation. The Workforce Question</a> </h2> Expectations are mixed: 43% expect no change in workforce size due to AI</li> 32% expect workforce reductions of 3%+ in the next year</li> 13% expect increases of 3%+</li> </ul> What's interesting: larger organizations are more likely to expect reductions. And AI high performers are more likely to expect change—in either direction. Meanwhile, most organizations are hiring for AI roles—particularly software engineers and data engineers. Even as some functions shrink, the demand for AI talent is growing. Risk Mitigation Is Improving (Slowly)</a> </h2> The good news: organizations are starting to take AI risks seriously. In 2022, organizations were mitigating an average of 2 AI-related risks</li> In 2025, that number is 4</li> </ul> The most common consequences experienced: Inaccuracy (31% of respondents)</li> Followed by explainability issues, privacy concerns, and regulatory compliance</li> </ul> High performers—who have deployed more use cases—are more likely to experience negative consequences, particularly around IP infringement and regulatory compliance. But they're also more likely to have mitigation measures in place. What This Means</a> </h2> Three takeaways: 1. The Window for Competitive Advantage Is Open</a> </h3> Most organizations are stuck in pilots. The ones moving to production-scale deployment have a real opportunity to pull ahead. 2. Efficiency Isn't Enough</a> </h3> Cost reduction is the default goal. But the organizations seeing the most value are using AI to drive growth and innovation—not just to cut headcount. 3. Workflow Redesign Is the Unlock</a> </h3> Bolting AI onto existing processes produces marginal gains. Redesigning workflows around AI capabilities produces transformational outcomes. This is the hardest part. It requires not just technology investment, but organizational change, process engineering, and leadership commitment. The State of AI 2025 confirms what many of us see in practice: AI is everywhere, but enterprise value is still rare. The technology works. The challenge now is organizational. The gap between "we use AI" and "AI transformed our business" is where the real work happens. Introducing PaperPin: AI-Powered Surveys with Synthetic Respondents 2025-12-06T00:00:00+00:00 I've been working on a new project called PaperPin</a>—an AI-powered survey platform that uses synthetic respondents to deliver research insights fast. The Problem with Traditional Surveys</a> </h2> If you've ever run user research, you know the pain: Recruiting takes weeks. Finding the right respondents, screening them, scheduling—it's a logistics nightmare.</li> Panels are expensive. Quality respondents cost money, and costs scale linearly with sample size.</li> Turnaround is slow. By the time you get results, the product has already shipped or the decision has been made.</li> Niche audiences are nearly impossible. Good luck finding 50 CFOs of mid-market SaaS companies who will take your survey.</li> </ul> Traditional survey panels work, but they're slow, expensive, and don't scale well for rapid iteration. What PaperPin Does Differently</a> </h2> PaperPin replaces human panels with AI-generated synthetic respondents. Here's how it works: 1. Define Your Persona</a> </h3> Specify the demographics, behaviors, and psychographics of your target audience. Want responses from "35-45 year old product managers at enterprise SaaS companies who prioritize developer experience"? Done. 2. AI Simulates Authentic Responses</a> </h3> Advanced LLMs embody your persona with nuanced, contextual understanding. They respond authentically, capturing the voice and perspective of real people in that demographic. 3. Get Results in Minutes</a> </h3> What once took weeks now takes minutes. No recruitment delays, no scheduling, no no-shows. 4. Validated Accuracy</a> </h3> Results are calibrated against real human data. PaperPin achieves 95% correlation with traditional human panel surveys across thousands of validation studies. What You Can Research</a> </h2> PaperPin supports multiple research campaign types: Concept & Product Testing — Test new product ideas, measure purchase intent, validate product-market fit</li> Brand & Market Research — Measure brand awareness, evaluate positioning, analyze competitors</li> Customer Insights — Track CSAT, measure NPS, identify what drives customer advocacy</li> UX Testing — Gather detailed product feedback, evaluate usability, prioritize features</li> Ad & Message Testing — Measure ad effectiveness, test messaging variants, compare value propositions</li> Pricing Research — Understand price sensitivity, estimate willingness to pay, optimize pricing strategy</li> </ul> Why Synthetic Respondents?</a> </h2> I built PaperPin because I kept running into the same problem: I needed quick feedback on product decisions, but traditional research was too slow and expensive for the pace of iteration I wanted. Synthetic respondents aren't a replacement for all human research. They're a tool for: Early-stage validation when you need directional insights fast</li> Rapid iteration when you're testing multiple variants</li> Niche audiences that are hard or expensive to recruit</li> Budget-conscious research when panel costs don't make sense</li> </ul> The goal isn't perfect fidelity—it's getting useful signal quickly so you can make better decisions faster. How It Works Under the Hood</a> </h2> Each synthetic respondent is generated based on your cohort definition. The AI doesn't just answer questions randomly—it maintains consistent persona characteristics across the survey, providing both Likert-scale ratings and qualitative explanations. For example, if you're testing a new pricing page: Define a cohort: "Small business owners, 5-20 employees, using spreadsheets for invoicing"</li> Create your survey with questions about clarity, trust signals, willingness to pay</li> Run the campaign with 100 synthetic respondents</li> Get detailed feedback: ratings, sentiment analysis, and verbatim responses</li> </ol> The results include both quantitative metrics and rich qualitative insights—the kind of "why" explanations that make survey data actionable. Pricing</a> </h2> PaperPin has a generous free tier: Free: 10 respondents per survey, 5 surveys per month</li> Plus ($9/mo): 100 respondents, 50 surveys, premium AI models</li> Pro ($29/mo): 1,000 respondents, 500 surveys, export reports</li> Ultra ($99/mo): 100,000 respondents, 5,000 surveys, priority support</li> </ul> No credit card required to start. You get 500 free credits to try it out. Try It</a> </h2> If you're building products and need faster feedback loops, give PaperPin</a> a try. I'd love to hear what you think. The goal is simple: smart surveys, better decisions. Rogue AI Agents and How Observability Builds Trust 2025-11-29T00:00:00+00:00 AI agents don't just chat. They reason, plan, call tools, and act. That's useful in customer support, supply chain, and IT operations. It's also risky. In production, agents can go "rogue" in ways that are hard to see until damage is done. Here's how it works. An agent makes a call you can't explain. It produces different outputs for the same input. Or it fails silently between steps. When that happens, debugging is guesswork, compliance is shaky, and trust erodes. Observability is how you fix that. The Three Pillars</a> </h2> 1. Decision Tracing</a> </h3> Trace the path from input to output through every intermediate step. This includes prompts, retrieved context, tool calls, responses, and state changes. You're building a chain of evidence: what the agent saw, what it decided, and why. 2. Behavioral Monitoring</a> </h3> Watch how the agent behaves, not just whether it runs. Look for loops, anomalies, and risky patterns: Infinite or long planning loops</li> Repeated tool calls with no progress</li> Outputs outside policy (e.g., PII disclosure, off-policy actions)</li> Sudden drift in confidence or retrieval quality</li> </ul> 3. Outcome Alignment</a> </h3> Start with intent. Did the agent deliver the outcome you asked for, given the input and context? Measure the result against ground truth, policy, and business goals. If the intent was "reset password without exposing PII," you check exactly that - not a vague "agent said success." What You Capture</a> </h2> Good observability starts with the right data: Inputs and context: User request</li> System instructions</li> Retrieved documents</li> Prior state</li> </ul> Decisions and reasoning: Plans and thought steps</li> Selected tools and parameters passed</li> Results returned</li> If you gate "reasoning" for privacy, still log a machine-readable trail of actions and justifications</li> </ul> Outcomes: Final answer</li> Side effects (tickets created, refunds issued)</li> Validations (policy checks, human approvals, metrics)</li> </ul> Store this as structured events. Each event has a timestamp, actor (agent/tool), action, inputs, outputs, and status. Stitch events into a timeline. That timeline is your replay: a transparent trail you can analyze, compare across runs, and improve. Observability ≠ Monitoring</a> </h2> Monitoring tracks raw signals: CPU load, token count, error rate. Useful, but blind. Observability adds context: the full decision trail. You don't just know something failed - you see where, how, and what it did before failing. That's the difference between: "We saw 500s"</li> "On step 4, the agent misread the policy doc and chose the refund tool incorrectly"</li> </ul> A Practical Example</a> </h2> Use case: A customer support agent that processes return requests. Input: "I want to return order #123 for a defective charger." Context: Policy docs, order data, past conversations. Decision steps: Plan: Verify warranty, check return window, find nearest drop-off</li> Tool calls: Fetch order #123 → read warranty doc → query logistics API</li> Checks: Confirm user identity, detect PII exposure, enforce refund limits</li> </ol> Outcome: Issue return label, schedule pickup, confirm to user. Observability in action: Decision tracing: Every step logged; you can replay the run</li> Behavioral monitoring: Flagged a loop when logistics API timed out thrice</li> Outcome alignment: Compared final action with policy; if policy prohibits returns after 30 days and the agent approved one on day 45, it triggers review</li> </ul> This helps you move faster: incident response, policy audits, and continuous tuning. You stop guessing. You start improving. Operating Model</a> </h2> Instrumentation: Add logging at the agent, tool, and framework layers. Use a consistent schema. Policy guardrails: Codify checks as first-class steps. Log pass/fail with reasons. Replay and diff: Compare timelines between "good" and "bad" runs. Highlight divergent decisions. Metrics with context: Count loops, failed tool calls, off-policy actions. Tie them back to specific steps. Feedback: Attach human review outcomes to the timeline. Use them to retrain or adjust prompts/tools. Privacy and compliance: Redact sensitive fields; retain trace fidelity without violating policy. Common Failure Patterns</a> </h2> Pattern</th> Symptom</th> Fix</th></tr></thead> Silent failures</td> Agent stops mid-chain with no surface error</td> Log step-level status and timeouts</td></tr> Ambiguous outputs</td> Multiple conflicting answers for the same input</td> Deterministic policies and post-hoc validators</td></tr> Tool thrashing</td> Repeated calls without progress</td> Retry budgets, backoff, and loop detectors</td></tr> Context drift</td> Wrong or stale documents</td> Retrieval quality signals and provenance logging</td></tr> </tbody></table> What "Trust" Looks Like</a> </h2> Trust isn't a slogan. It's repeatable behavior under scrutiny. You can explain how the agent decided</li> You can prove outcomes match intent and policy</li> You can detect and correct anomalies quickly</li> You can improve the agent based on evidence, not hunches</li> </ul> Quick Start Checklist</a> </h2> Define intent for each agent task: inputs, allowed tools, acceptable outcomes</li> Log every step with timestamps, inputs, outputs, and status</li> Add validators: policy checks, PII filters, safety rules</li> Build a timeline view and a replay tool</li> Track behavior metrics: loops, retries, off-policy decisions</li> Run postmortems with the trace when incidents occur</li> Feed learnings back: update prompts, tools, and policies</li> </ul> The Takeaway</a> </h2> Observability for AI agents isn't dashboards and metrics. It's the full picture: inputs, decisions, and outcomes, stitched into a timeline you can trust, analyze, and improve. That's how you operate autonomous systems reliably at scale. Stripe's Agentic Commerce Protocol: Building the Rails for AI-Powered Commerce 2025-11-28T00:00:00+00:00 Stripe recently announced the Agentic Commerce Protocol (ACP), a new standard designed to enable AI agents to discover merchants, negotiate terms, and complete purchases on behalf of users. This is a significant step toward a future where your AI assistant doesn't just recommend products—it actually buys them for you. The Problem ACP Solves</a> </h2> Today's AI agents can do a lot. They can browse the web, summarize content, write code, and even interact with APIs. But when it comes to making purchases, they hit a wall. Here's why: Trust: How does a merchant know the AI agent has permission to spend a user's money?</li> Discovery: How does an agent find merchants that can fulfill a specific request?</li> Negotiation: How do agents and merchants agree on terms, pricing, and fulfillment?</li> Compliance: How do you handle authentication, fraud prevention, and regulatory requirements?</li> </ol> ACP addresses all of these by providing a standardized protocol that sits between AI agents, merchants, and payment infrastructure. Architecture Overview</a> </h2> ACP introduces three key participants in every transaction: ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ AI Agent │────▶│ ACP │────▶│ Merchant │ │ (Claude, │ │ Protocol │ │ (Seller) │ │ GPT, etc) │◀────│ Layer │◀────│ │ └─────────────┘ └─────────────┘ └─────────────┘ │ │ │ │ ▼ │ │ ┌─────────────┐ │ └──────────▶│ Stripe │◀───────────┘ │ (Payment │ │ Rails) │ └─────────────┘ </code></pre> The Agent</a> </h3> The AI agent acts on behalf of the user. It could be a general-purpose assistant like Claude or a specialized agent for travel, shopping, or business procurement. The agent needs to: Understand user intent</li> Discover relevant merchants via ACP</li> Present options and get user confirmation</li> Execute the transaction</li> </ul> The Protocol Layer</a> </h3> ACP defines a standard way for agents and merchants to communicate. This includes: Discovery endpoints: Merchants register their capabilities, inventory, and terms</li> Negotiation schemas: Structured formats for quotes, offers, and counter-offers</li> Transaction lifecycle: States for pending, confirmed, fulfilled, and disputed transactions</li> Permission scopes: Granular control over what an agent can do</li> </ul> The Merchant</a> </h3> Merchants expose their catalog and business logic through ACP-compliant endpoints. They define: What products/services are available</li> Pricing and availability rules</li> Fulfillment options</li> Refund and cancellation policies</li> </ul> How a Transaction Works</a> </h2> Let's walk through a real example: booking a restaurant reservation. Step 1: User Intent</a> </h3> User: "Book me a table for 4 at an Italian restaurant near downtown SF this Saturday at 7pm" </code></pre> Step 2: Agent Discovery</a> </h3> The agent queries ACP's discovery layer: { "intent": "restaurant_reservation", "constraints": { "cuisine": "italian", "location": { "near": "downtown San Francisco", "radius_km": 2 }, "party_size": 4, "datetime": "2024-11-30T19:00:00-08:00" }, "user_id": "usr_abc123", "agent_id": "agent_xyz789" } </code></pre> Step 3: Merchant Responses</a> </h3> Multiple merchants respond with availability: { "merchant_id": "merch_italian_place", "offer": { "offer_id": "off_12345", "description": "Table for 4 at Trattoria Milano", "datetime": "2024-11-30T19:00:00-08:00", "deposit_required": { "amount": 2500, "currency": "usd" }, "cancellation_policy": "free_until_24h_before", "valid_until": "2024-11-28T23:59:59-08:00" } } </code></pre> Step 4: User Confirmation</a> </h3> The agent presents options to the user and gets confirmation. This is a critical step—ACP requires explicit user consent for transactions above certain thresholds or outside pre-authorized categories. Step 5: Transaction Execution</a> </h3> Once confirmed, the agent sends a transaction request: { "action": "accept_offer", "offer_id": "off_12345", "payment_method": "pm_card_visa", "user_authorization": { "token": "auth_tok_xxx", "scope": ["restaurant_reservation"], "max_amount": 5000 } } </code></pre> Step 6: Fulfillment</a> </h3> The merchant confirms the reservation and ACP tracks the fulfillment state. If something goes wrong, there's a standardized dispute resolution process. The Technical Bits</a> </h2> Permission Model</a> </h3> ACP uses a hierarchical permission system. Users grant agents specific scopes: { "agent_permissions": { "agent_id": "agent_xyz789", "scopes": [ { "category": "food_and_dining", "max_transaction": 10000, "max_daily": 50000, "requires_confirmation": false }, { "category": "travel", "max_transaction": 100000, "requires_confirmation": true } ], "expires_at": "2025-11-28T00:00:00Z" } } </code></pre> This means you could let your agent book lunch without asking, but require confirmation for flight bookings. MCP Integration</a> </h3> ACP is designed to work with Anthropic's Model Context Protocol (MCP). In fact, Stripe has released an MCP server that exposes ACP functionality as tools that any MCP-compatible agent can use. // Example MCP tool definition for ACP { name: "acp_discover_merchants", description: "Find merchants that can fulfill a purchase intent", parameters: { intent: { type: "string" }, constraints: { type: "object" }, location: { type: "object", optional: true } } } </code></pre> This means if you're building an agent using Claude, you can add ACP capabilities by connecting to Stripe's MCP server. Idempotency and State Management</a> </h3> E-commerce is messy. Networks fail, retries happen, and state gets out of sync. ACP handles this with: Idempotency keys: Every transaction request includes a unique key. Retrying with the same key won't create duplicate charges.</li> State machine: Transactions follow a strict state machine with well-defined transitions</li> Webhooks: Merchants and agents receive real-time updates about state changes</li> </ul> PENDING ──▶ CONFIRMED ──▶ FULFILLED │ │ │ ▼ ▼ ▼ CANCELLED REFUNDED DISPUTED </code></pre> Why This Matters</a> </h2> For Developers</a> </h3> If you're building AI agents, ACP gives you a standardized way to add commerce capabilities without building payment infrastructure from scratch. You focus on the agent logic; Stripe handles the money movement. For Merchants</a> </h3> You get access to a new distribution channel. As AI agents become more prevalent, being discoverable and transactable via ACP could be like being searchable on Google in the early 2000s. For Users</a> </h3> The promise is convenience with control. Your agent can handle routine purchases automatically while still requiring your approval for significant decisions. Getting Started</a> </h2> Stripe's ACP is currently in beta. To start experimenting: Sign up for the beta at stripe.com/acp</li> Install the MCP server if you're building MCP-compatible agents:npm install @stripe/acp-mcp-server </code></pre> </li> Read the spec: The protocol specification is open and available for review</li> </ol> The Bigger Picture</a> </h2> ACP is part of a broader trend toward AI agents that can take real-world actions. We're moving from AI that tells you what to do to AI that does things for you. But with great power comes great responsibility. The permission model, user confirmation requirements, and dispute resolution mechanisms in ACP reflect an understanding that autonomous spending needs guardrails. The interesting question isn't whether AI agents will make purchases for us—they will. The question is how we build the infrastructure to make this safe, trustworthy, and beneficial for everyone involved. Stripe's bet is that whoever builds the standard rails for agentic commerce will be well-positioned in this future. With ACP, they're making that bet concrete. Disclaimer: This analysis is based on publicly available information about Stripe's ACP announcement. Implementation details may vary as the protocol evolves. A 2x Engineer 2024-11-15T00:00:00+00:00 We often hear about 10x engineers but today I want to talk about a 2x engineer. More specifically, I want to talk about what are those two qualities that make a 2x engineer. Quality 1: Decision Making</a> </h2> Decision making is important in software development. You would often find yourself in a room with 10 other engineers and managers discussing about a new feature or a project that has to implemented. There would be a point in the meeting that a crucial path has to be selected out of let us say 2 different options. You would have to make a decision and it has to be a time bound exercise. This is where a 2x engineer would step up and point out the pros and cons of each path and make a decision. It is important to make sure that you are not just making a decision but also explaining why you made that decision. Your Decision should rely on: Past experiences: The capability to make a decision comes with experience and knowledge. You would have to have faced similar problems previously even though not entirely similar but you should be able to draw an analogy. Depending on how the past decisions have turned out, you would be able to make a better decision this time. Reversibility: The decision you make should be reversible. If you are not sure about the decision you are making, make sure that you can reverse it. Out of 100 things that you would have to decide on, there would just one or two that would not be reversible. At first you would find that a decision might be irreversible but if you think hard enough, if you are creative enough, you would find a way to reverse it. Not everything has to be grandiose: There are people who job is to make sure that everything looks grandiose. But reality is that more often than not things are simpler, decisions are simpler. Your job is to identify those simpler decisions and make them and make them instantly without needing a meeting to discuss about it. Quality 2: Communication</a> </h2> Communication is the key to success. You might have heard this a lot of times but it is true. You might be the best engineer in the world but if you cannot communicate your ideas, your thoughts, your decisions, your code, you are not going to be successful. Writing: Writing is the most important form of communication. You would have to write a lot of emails, a lot of documentation, a lot of code. If you are not able to write clearly, you are not going to be successful. You would have to write in a way that the reader understands what you are trying to convey. You would have to write in a way that the reader does not have to ask you questions about what you wrote. Speaking: Speaking is another form of communication. You would have to speak in meetings, you would have to speak in interviews, you would have to speak in presentations. If you are not able to speak clearly, you are not going to be successful. You would have to speak in a way that the listener understands what you are trying to convey. You would have to speak in a way that the listener does not have to ask you questions about what you spoke. Listening: Listening is another form of communication. You would have to listen to your peers, your managers, your reports. If you are not able to listen clearly, you are not going to be successful. You would have to listen in a way that you understand what the other person is trying to convey. You would have to listen in a way that you do not have to ask questions about what the other person spoke. Timelines: Give timelines to anything that is asked of you. It may not be favorable to the person asking but it is important to give a timeline. Practical PromQL Examples 2024-07-18T00:00:00+00:00 PromQL is a powerful query language that allows you to query metrics from Prometheus. Here are some practical examples that you can use to query metrics from Prometheus. Working with counters</a> </h2> Let's assume we have a metric called http_request_duration_seconds_count</code> that records the total number of HTTP requests made to a server. The metric has the following labels: method</code>: The HTTP method used for the request (e.g., GET, POST, PUT, DELETE).</li> status</code>: The HTTP status code returned by the server (e.g., 200, 404, 500).</li> handler</code>: The handler function that processed the request (e.g., /api/v1/users</code>, /api/v1/posts</code>).</li> </ul> Time Durations</a> </h3> # Count the total number of HTTP requests in the last 1 second sum(rate(http_request_duration_seconds_count)) </code></pre> # Count the total number of HTTP requests in the last 15 minutes sum(rate(http_request_duration_seconds_count[15m]) 60 15) </code></pre> # Count the total number of HTTP requests in the last 1 day sum(rate(http_request_duration_seconds_count[1d]) 60 60 24) </code></pre> # Count the total number of HTTP requests in the last 1 week sum(rate(http_request_duration_seconds_count[1w]) 60 60 24 7) </code></pre> The idea for these queries are simple - you are calculating the rate of change of the counter metric over a specific time period and then multiplying it by the number of seconds in that time period to get the total count. Group by</a> </h3> # Count the total number of HTTP requests by method in the last 1 minute sum(rate(http_request_duration_seconds_count[1m] 60)) by (method) </code></pre> # Count the total number of HTTP requests by status code where handler is equal to /users in the last 5 minutes sum(rate(http_request_duration_seconds_count{handler = "/users"}[5m] 300)) by (status) </code></pre> Greater than</a> </h3> # Count the total number of HTTP requests by method in the last 1 minute if the count is greater than 50 sum(rate(http_request_duration_seconds_count[1m] 60)) by (method) > 50 </code></pre> Less than</a> </h3> # Count the total number of HTTP requests by method in the last 1 minute if the count is less than 100 sum(rate(http_request_duration_seconds_count[1m] 60)) by (method) < 100 </code></pre> Within Range</a> </h3> # Count the total number of HTTP requests by method in the last 1 minute if the count is between 50 and 100 sum(rate(http_request_duration_seconds_count[1m] 60)) by (method) > 50 < 100 </code></pre> Regular Expressions</a> </h3> Let us suppose we have rest apis for users like: POST /users</li> GET /users/:user_id</li> PUT /users/:user_id</li> DELETE /users/:user_id</li> </ul> # Count the total number of HTTP requests by handler and status code for similar handlers in the last 5 minutes sum(rate(http_request_duration_seconds_count{handler =~ "/users."}[5m] 300)) by (status, handler) </code></pre> # 5xx errors for the last 1 hour for each handler and method sum(rate(http_request_duration_seconds_count{status=~"5.."}[1h]) 60 60) by (handler, method) </code></pre> Offset</a> </h3> # Count the total number of HTTP requests by handler and status code for similar handlers in the last to last 5 minutes sum(rate(http_request_duration_seconds_count{handler =~ "/users."}[5m] offset 5m) 300) by (status, handler) </code></pre> Delta</a> </h3> # Count the difference in the total number of HTTP requests by handler and status code for similar handlers in the last 5 minutes sum(rate(http_request_duration_seconds_count{handler =~ "/users."}[5m] 300) - rate(http_request_duration_seconds_count{handler =~ "/users."}[5m] offset 5m) 300) by (status, handler) </code></pre> Percentage</a> </h3> # Calculate the percentage of 5xx errors for each handler in the last 1 hour sum(rate(http_request_duration_seconds_count{status=~"5.."}[1h]) 60 60) / (sum(rate(http_request_duration_seconds_count[1h]) 60 60) by (handler) > 0) * 100 </code></pre> Messaging between Browser Windows using ruto.js 2024-05-18T00:00:00+00:00 At my work I have to build solutions around communication between browser and iframe or a popup window i.e. <iframe></code> or window.open()</code>. I have to use postMessage</code> JS API, and feel it is not very developer friendly. Consider the below situation: The parent window sends a message to the child window</li> The child window receives this message</li> The child calls a backend server with the message</li> The child sends the response back to parent</li> The parent wants to receive this backend response, does some calculation</li> The parent then wants to send a second message to the child</li> And the same thing repeats</li> </ol> While this may seem trivial when you are doing simple HTTP calls from frontend but across windows something like this is much more complicated. We have also to keep in mind that security is our outmost concern. Let us try to code this thing. Parent is parent.com, child is at child.com. I am assuming parent has child as an iframe. 1. Send Message from parent to child</a> </h2> //parent.com iframe.contentWindow.postMessage(message, "child.com"); </code></pre> 2. Receive message from parent in child</a> </h2> //child.com window.addEventListener("message", (event) => { const expectedOrigin = "parent.com"; if (event.origin !== expectedOrigin) { return; } const message = event.data; doProcessingOfMessage(message); }); </code></pre> 3. Child does some processing</a> </h2> //child.com async function doProcessingOfMessage(message) { const resp = await api(message); return resp; } </code></pre> 4. Child sends response back</a> </h2> //child.com window.addEventListener('message', (event) => { const expectedOrigin = 'parent.com'; if (event.origin !== expectedOrigin) { return } const message = event.data; const resp = await doProcessingOfMessage(message); window.parent.postMessage(resp, 'parent.com'); }) </code></pre> 5. Parent receives the message</a> </h2> //parent.com window.addEventListener("message", (event) => { const expectedOrigin = "child.com"; if (event.origin !== expectedOrigin) { return; } const message = event.data; }); </code></pre> 6. Parent sends 2nd message</a> </h2> //parent.com window.addEventListener("message", (event) => { const expectedOrigin = "child.com"; if (event.origin !== expectedOrigin) { return; } const message = event.data; const newMessage = getNewMessage(message); iframe.contentWindow.postMessage(message, "child.com"); }); </code></pre> As we can see the process is completely event based. And writing code to do something like this gets very messy. What if there was a nicer way to do this? Let us say: //parent.com const resp1 = await sendMessageToChild(message1); const message2 = await someProcessing(resp1); const resp2 = await sendMessageToChild(message2); </code></pre> //child.com const message1 = await receiveMessageFromParent(); const resp1 = await someProcessing(message1); await sendMessageToParent(resp1); </code></pre> In the above scenario, the parent is the orchestrator and child is just responding to actions triggered by parent. In a different scenario a child can be the orchestrator and the parent would be just responding to actions triggered by the child. That means both the parent and child can send and receive. If parent is sender then child has to be receiver, and when parent is receiver the child has to be sender. Handle Retry</a> </h2> We have to make sure that there are retries in communication. And also both parent and child should be ready to receive. Security</a> </h2> We have to have origin checks in both child and domain. Ignore all messages if not received from where we want to receive. Solution</a> </h2> I think the best way to do this would be to build it like a client/server kind of a thing. Let us take an example of web client and Node.js Express server. Client: fetch("/message") .then((response) => response.json()) .then((data) => { document.getElementById("response").textContent = JSON.stringify( data, null, 2 ); }) .catch((error) => { document.getElementById("response").textContent = "Error: " + error; }); </code></pre> Server: app.get("/message", (req, res) => { res.send("Hello, world!"); }); </code></pre> I started to write the solution but soon realized it is not easy. So I have to take a more conservative approach. Introducing Ruto</a> </h2> Ruto is a lightweight (4KB), fast and easy-to-use JS library that streamlines the communication between parent and child window (iframe/popup). It uses client-server design pattern to communicate between parent and child window. Any window can become the client or the server depending on who wants to send. It abstracts out the complications of postMessage API and provides a simple API to send and receive messages. You can visit the GitHub page here</a>. Ruto Examples</a> </h2> Parent to iframe</a> </h3> Parent will send a message to iframe, Parent expects a response from the iframe within 10 secs. Parent: <iframe id="iframe" src="https://example.com/child.html"></iframe> <script src="https://cdn.jsdelivr.net/gh/rajnandan1/ruto/dist/ruto.min.js"></script> <script> const iframe = document.getElementById("iframe"); const message = "Hello from parent"; const options = { timeout: 10000, }; ruto.send( "https://example.com/parent-to-iframe/topic1", iframe, message, options ) .then((response) => { console.log(response); //Hello from child }) .catch((error) => { console.log(error); }); </script> </code></pre> Child: <script src="https://cdn.jsdelivr.net/gh/rajnandan1/ruto/dist/ruto.min.js"></script> <script> ruto.receive( "/parent-to-iframe/topic1", window.parent, (response, message) => { console.log(message); //Hello from parent return response.send("Hello from child"); } ); </script> </code></pre> Iframe to Parent</a> </h3> Iframe will send a message to parent, and expects a response from parent within 5 secs. Parent: <iframe id="iframe" src="https://example.com/child.html"></iframe> <script src="https://cdn.jsdelivr.net/gh/rajnandan1/ruto/dist/ruto.min.js"></script> <script> const iframe = document.getElementById("iframe"); ruto.receive("/iframe-to-parent/topic1", iframe, (response, message) => { console.log(message); //Hello from iframe return response.send("Hello from parent"); }); </script> </code></pre> Child: <script src="https://cdn.jsdelivr.net/gh/rajnandan1/ruto/dist/ruto.min.js"></script> <script> const message = "Hello from iframe"; const options = { timeout: 5000, }; ruto.send( "https://example.com/iframe-to-parent/topic1", window.parent, message, options ) .then((response) => { console.log(response); //Hello from parent }) .catch((error) => { console.log(error); }); </script> </code></pre> Parent to Popup</a> </h3> Parent will send a message to popup, and expects a response from popup within 5 secs. Parent: <script src="https://cdn.jsdelivr.net/gh/rajnandan1/ruto/dist/ruto.min.js"></script> <script> const popup = window.open( "https://example.com/popup.html", "popup", "width=600,height=400" ); const message = "Hello from parent"; const options = { timeout: 5000, }; ruto.send( "https://example.com/parent-to-popup/topic1", popup, message, options ) .then((response) => { console.log(response); //Hello from popup }) .catch((error) => { console.log(error); }); </script> </code></pre> Popup: <script src="https://cdn.jsdelivr.net/gh/rajnandan1/ruto/dist/ruto.min.js"></script> <script> ruto.receive( "/parent-to-popup/topic1", window.opener, (response, message) => { console.log(message); //Hello from parent return response.send("Hello from popup"); } ); </script> </code></pre> Popup to Parent</a> </h3> Popup will send a message to parent, and expects a response from parent within 5 secs. Parent: <script src="https://cdn.jsdelivr.net/gh/rajnandan1/ruto/dist/ruto.min.js"></script> <script> const popup = window.open( "https://example.com/popup.html", "popup", "width=600,height=400" ); ruto.receive("/popup-to-parent/topic1", popup, (response, message) => { console.log(message); //Hello from popup return response.send("Hello from parent"); }); </script> </code></pre> Popup: <script src="https://cdn.jsdelivr.net/gh/rajnandan1/ruto/dist/ruto.min.js"></script> <script> const message = "Hello from popup"; const options = { timeout: 5000, }; ruto.send( "https://example.com/popup-to-parent/topic1", window.opener, message, options ) .then((response) => { console.log(response); //Hello from parent }) .catch((error) => { console.log(error); }); </script> </code></pre>

Blog

Good Taste the Only Real Moat Left

Agent Harness Engineering vs Context Engineering vs Prompt Engineering

OWASP Top 10 for LLMs: AI Vulnerabilities and How to Defend Against Them

P vs NP, NP-Hard, and NP-Complete - A Very Simple Guide

How a Screen Turns Electricity Into Motion

Largest Triangle Three Buckets: Downsampling Time-Series Data Without Losing Signal